Overview and details of the sessions of this conference. Please select a date or location to show only sessions at that day or location. Please select a single session for detailed view (with abstracts and downloads if available).
A5: Measurement in Mobile Web Surveys
Mobile-Friendly Grid Questions: The Accordion Grid as an Alternative to the Traditional Grid
GfK Custom Research, United States of America
Relevance & Research Question:
Grid questions are used routinely in online surveys when researchers want to assess multiple items using the same response format. Grids typically present elements to be evaluated in rows with responses arrayed in the columns. Grids can be problematic with smaller screen sizes such as those of smartphones. Horizontal (left-right) scrolling is often required to view all response options, making it impossible to view both the item text and all response options at the same time and raising the possibility that respondents will be less likely to select response options not visible on the screen. In the accordion grid format, respondents see the item text listed vertically, then can click on each item to reveal the response scale in a standard single response format. Using this approach, respondents can see the entire response scale for each item, along with the response items on a single screen. However, vertical response presentation has been associated with an increased likelihood of response order effects. Is the accordion grid an effective alternative to the traditional grid format?
Methods & Data:
We conducted two experiments each using over 1,000 cases from KnowledgePanel®, GfK’s U.S. probability-based online panel. With each study, respondents were randomly assigned to complete either an accordion or traditional grid.
The time it took for respondents to complete each grid type was comparable, and we found few differences in results across the two types of grids for both dichotomous grids and unipolar scales with three or more response options. We found similar concurrent validity for both grid types when correlating the results with behavioral data and little evidence of response order effects. Respondents provided similar ratings of the ease and accuracy with which they could respond to the traditional and accordion grid scales.
Taken together, the results suggest that the accordion grid is a promising alternative to the traditional grid design, especially given the increase in the proportion of online survey respondents who complete surveys using smartphones and other mobile devices.
Adapting Questionnaires for Smartphones: An Experiment on Grid Format Questions
Kantar Public, United Kingdom
Relevance & research question:
As ownership and use of smartphones grows, we need to adapt survey design to enable people to complete questionnaires on their chosen device, without impacting negatively on respondent experience or data quality.
Previous research highlights issues with grid questions on smartphones (e.g. McClain & Crawford, 2013). The ‘traditional’ grid format can appear cluttered on a smartphone screen, and this in turn can cause respondent burden and risks miscoding responses and higher drop out rates.
In this paper we consider alternatives to traditional grid formats and present experimental data to show the relationship between grid format and data quality.
Methods & data:
We present the results of an experiment comparing traditional grids with three alternatives: item by item scrolling, item by item paging and dynamic grids.
The experiment was run on the Kantar TNS online omnibus. The achieved sample size for each format was c. 1,200, including c. 250 respondents completing on a smartphone in each cell.
Alongside this experiment we have conducted usability testing with respondents to provide qualitative feedback on ease of use and perceptions of different grid formats.
We compared results from the formats over a number of analysis dimensions, including substantive responses, missing response levels, ‘Don’t know’ rates, question timings, flatlining, and respondent assessments. Results are also compared across device types and screen dimensions.
The results were broadly similar over the four formats, suggesting that for this sample/questionnaire the format did not substantially affect responses. However, there were differences in relation to question timings and missing response levels, with dynamic grids performing positively.
In the usability testing respondents found the dynamic grids to be engaging and intuitive, with many expressing a preference for this format.
As social studies increasingly move online we need to ensure that questionnaires are optimised for mobile devices. This paper adds important evidence to one of the challenges associated with this shift: how to deal with grids. Through our experiment and usability testing we assess the pros and cons of alternative formats, including an interactive dynamic grids approach, and consider implications for adapting existing surveys.
The Effects on Data Quality of Horizontal and Vertical Question Orientation and Scales of Different Length for Respondents Using Smartphones, Tablets and PCs
1University of Gothenburg, Sweden; 2University of East Anglia, UK
Questionnaire design, Mobile surveys, Response order, Scale direction, Scale length
Relevance & Research Question:
The increasingly mixed technical devices respondents use to answer online surveys cause concern among survey researchers. Despite efforts to adjust online surveys to smartphones knowledge is still limited concerning how different response scales work on different devices. The main concern is that respondents who use a smartphone might treat response scales different than PC or laptop users. This paper analyzes response order effects (e.g. primacy effects) and uses large sample sizes that enable us to analyze whether these effects are conditional on scale length, question orientation (horizontal vs vertical) and the type of response device participants are using.
Methods & Data:
This paper reports four experiments where response option order, scale visual orientation (horizontal or vertical) also scale length are varied randomly. These experiments examine effects for dichotomous scales, 3-point scales, 5-point scales, 7-point scales and 11-point scales for political attitude questions. The data was collected by the Laboratory of Opinion Research (LORE) at the University of Gothenburg, Sweden, from their online panel (The Swedish Citizen Panel) between May 2014 and June 2015.
Results indicate that vertical response scales suffer more from primacy effects than horizontal response scales, and that this is even more so for smartphone users. Moreover, we find an effect of scale orientation in itself apart from response order effects: vertically oriented scales produce more responses in the beginning of the scale (topmost) than horizontally oriented scales do (leftmost). This means that comparability between smartphone and PC data is threatened when questions are not shown in the same way to respondents on different devices. However, scale length clearly matters in that these effects are substantially larger when scales are 5 points or longer.
This paper demonstrates the importance of question orientation (horizontal vs vertical) and that these should be identical on smartphones and PCs to ensure comparability. Since each of the four experiments contain at least 10,000 participants and the share of smartphone users is substantial (between 20 and 30 percent) these can be analyzed with sufficient statistical power.
Contact and Legal Notice · Contact Address:
Conference: GOR 17
|Conference Software -
ConfTool Pro 2.6.96
© 2001 - 2016 by H. Weinreich, Hamburg, Germany