Relevance & Research Question: Online survey participants increasingly complete questionnaires on their smartphones. However, a common finding in survey research is that survey respondents using mobile devices break off more often than participants using a computer. Previous research has revealed numerous aspects that potentially affect the breakoff rates. These aspects can be divided into two sections: layout features and survey related conditions. Layout features are, e.g., screen-optimized designs, ecessities to scroll, and matrix questions. The survey related conditions involve the invitation mode, reminders, compulsion for a certain device, etc. So far, the literature shows heterogeneous influences of these effects on breakoff rates. This brings us to our research question: How effective are different measures of optimizing surveys for smartphones to reduce breakoff rates of smartphone respondents?

Methods & Data: To answer this question, we collected research results regarding measurement on smartphone optimization and device effects from more than 50 papers and a variety of conference presentations published between 2007 and August 2019. By conducting a systematic review and a meta-analysis, we tested which of these predictors lower the breakoff rates in mobile web surveys. We hypothesize that mobile optimized surveys are more user-friendly, which in turn increases survey enjoyment and lowers survey burden. Consequently, lowering the survey burden leads to lower breakoff rates. We aim to examine which measures are helpful to optimize surveys for mobile devices.

Results & added Value: Based on our findings, we will present best practices from the current state of research to sustainably reduce breakoff rates in mobile web surveys. We build upon earlier findings of a meta-analysis from Mavletova and Couper (2015), add new empirical evidence, and expand their analytical framework. Our preliminary results so far show that a smartphone-optimized layout decreases breakoff rates. The final results will be available at the beginning of 2020.

Relevance & Research Question:

Running an experience sampling study via smartphones is a complex undertaking. Scheduling and sending mobile notifications to participants is a tricky task because it requires the use of native mobile applications. In addition, the existing software solutions often restrict the number of possible question types. To solve these problems, we have developed a free web application that runs in any browser and can be installed on mobile phones. Using the application, researchers can create their studies, schedule notifications, and monitor users' reactions. The content of notifications is fully customizable and may include links to studies created with external survey services.

Methods & Data:

We have conducted several empirical studies to test the application and its features, such as creating different types of notifications schedules and logging participants’ interactions with notifications. First pilot testing was carried out in student projects that conducted different surveys (e.g. happiness, stress, sleep quality, dreaming) with a schedule from several days up to one week. The second study was our own experience sampling survey with a university sample that was completed during one week with notifications sent seven times a day in the two-hours intervals. We also plan a third study with online samples, the results of which will be presented at the conference.

Results:

In the first pilot study (8 projects, n = 63), we analyzed the response rate of the participants based on the logging of interactions with notifications. In addition, the design and functionality of the web application was improved following a usability survey with application users. In the second study (n = 23) we analyzed how the type of participant’s device (i.e., mobile phone) is related to the response rate. Additionally, we investigated the relationship between the interaction with notifications and the response rate in the experience sampling survey. In the third study, we plan to repeat the analysis for the sample recruited online.

Added Value:

Our application provides a direct and easy way to run experience sampling studies. It has an open-source code and is available at https://samply.js.org.

Relevance & Research:

As the majority of online surveys nowadays are mixed-device studies of personal desktop computers (PC) and smartphones, the layout needs to be adapted to both device types. A lot of well-established constructs are usually presented in the matrix format. However, matrixes are not recommended for the use in smartphone surveys. Therefore, matrix questions are a challenge for all mixed-device studies. So far, the majority of studies that investigate the effects of layout and device on data quality have focused on indicators such as nonresponse and satisficing strategies. In our experimental study we focus on the combined effect of devices and layouts on measurement invariance.

Methods & Data:

In an experimental study we assessed the comparability of different constructs across device and layout combinations. We varied the two factors device (desktop vs. mobile device) and layout (optimized for desktop vs. optimized for smartphones vs. build-in adaptive layout), resulting in six groups of layout-device combinations. We included 5 well-established constructs with different numbers of scale points that are usually presented in a matrix format.

In October 2018 respondents from an online access panel in Germany were randomly invited to one of the six experimental groups. We applied quota sampling regarding age, sex, and education. Overall 3096 respondents finished the survey.

The experimental design allows us to examine whether the different layout settings have an impact on the perceived range of response scales and the presentation of multiple question as one conceptional unit. We evaluate whether layout and device have an impact on mean levels and whether the latent constructs are comparable across groups by the means of structural equation modelling.

Results: We find that layout and device do not impact mean levels of the constructs and we find a high level of comparability across experimental groups (scalar invariance).

Added Value:

This study provides evidence on the effect of layout choices on measurement invariance, depending on the device used. Furthermore, it offers information about comparability of results in mixed-device studies and practical guidance for designing mixed-device studies.

Relevance & Research Question: As a self-administered survey mode, Web surveys allow respondents to temporarily leave the survey page and switch to another window or browser tab. This form of sequential multitasking has the potential to disrupt the response process and may reduce data quality due to respondents' distraction (Krosnick 1991). Browser data indicating respondents leaving the survey page allow to non-reactively measure respondents’ multitasking. To investigate whether page-switching respondents produce lower data quality, one has to consider how to identify and delimit this group based on the time they do not spent on the survey page. Given that very short page-switching events might occur due to slips or unintentional behavior they might not be harmful to the response process. According, the aim of this paper is to discuss the adequate time threshold to classify respondents as multitaskers.

Methods & Data: For analyses reported in this paper, two Web surveys among members of a non-probability online panel (n=1,653; n=1,148) and a Web survey among university applicants (n=1,125) conducted in 2018 were used. To measure multitasking the JavaScript tool SurveyFocus (Höhne & Schlosser 2018) was implement. The prevalence of page-switching is computed using different time thresholds (< 2 sec, < 5 sec, < 10 sec). Item-nonresponse, degree of differentiation in matrix questions and characters to open-ended questions serve as measures of data quality.

Results: Preliminary analyses indicate that 15 to 33 percent of respondents multitask at least once in the survey. Previous results on all page switchers also indicate that these respondents do not produce lower data quality. However, so far we did not differentiate between respondents with short or long time absent. The analyses presented in this paper will show whether these results change when different time thresholds are applied. Furthermore, we will investigate whether page-switching respondents differ in their characteristics, their device used and completion time depending on the time they spent absent.

Added Value: Paradata on page-switching provides an opportunity to measure respondents’ multitasking unobtrusively. This paper addresses the challenge to identify multitasking respondents based upon this data to investigate the relationship of multitasking and data quality.

Relevance & Research Question: Economists/governments are deeply interested in the income distribution, the level of movement across the income distribution, and how observable characteristics predict someone's position on the distribution. These topics are answered in different countries using a combination of cross-sectional surveys, panel studies, and administrative data. Australia has been well served by sample surveys on the income distribution, but these are limited for relatively small population groups or for precise points on the distribution. Australian researchers have made limited use of administrative data. Not because the administrative data doesn't exist, but because of privacy and practical challenges with linking individuals and making that data available to external researchers. In this paper, we apply machine learning and standard econometric techniques to develop synthetic estimates of the Australian income distribution, validate this data against high quality survey data, use this administrative dataset to measure movement across the income distribution longitudinally, and measure ethnic disparities (by Indigeneity and ancestry)

Methods & Data: The dataset used in this paper has at its core individually linked medical, cros-sectional Census (i.e. survey), social security and tax data for 6 financial years. None of this data alone is complete for all parts of the income distribution, but combined can generate high quality estimates. Broadly, we generate a continuous cross-sectional income estimate from Census bands in 2011, test various machine learning algorithms to predict income using observed tax and social security data in 2011, use parameter estimates from the algorithms to estimate income in the following 5 financial years (based on demographic, tax and social security data for those years), validate against survey data, and then analyse.

Results: We show that certain algorithms perform far better than others, and that we are able to generate highly accurate predictions that match survey data at the national level. We then derive new insights into income inequality in Australia.

Added Value: We outline a methodology and set of techniques for when income data needs to be combined across multiple sources, demonstrate a productive link between ML and econometric techniques, and shed new light on the Australian income distribution.

Relevance & Research Question: Reliable market research is the basis for making the right decisions. Market researchers understand customer interests or the perception of existing products. However, the question of how and where potential customers can be reached is difficult to answer precisely. To solve this problem, Civey has developed Small Area Estimation through multilevel regression with poststratification in a live system. Thus, customers recognize potential leads even in the smallest geographical areas such as districts (“Landkreise”).

Methods & Data: The basis for this is a MRP model (Multilevel Regression with Poststratification), which Civey has implemented for real-time calculations. Data is collected online on over 25,000 websites. This way, over fifteen million opinions are collected each month. With one million verified and active users monthly, Civey has established Germany's largest open access panel.

Based on a two-stage process developed by Civey, which combines hierarchical logistic regression models and poststratification with variable selection by LASSO, real-time applications of MRP are possible to provide Small Area Estimations. In addition to the user-based information, the model also accounts for publicly available auxiliary information on district level.

Results: The model can be used to predict the probability that a certain person will give a particular answer for any combination of sociodemographic information. The model "learns" based on all information available. This model-based approach enables fast valid results even in the smallest geographical areas.

Added Value: After a brief introduction to the methodology, Civey provides unique insights into their results. This includes interesting evaluations of potential customers in the automotive market, but also amusing examples to show the variety and depth of data that this innovation allows.

Relevance & Research Question: The central idea of crowd-science is to engage a wide base of potential contributors who are not professional scientists into the process of conducting and/ or analyzing research data (z.B. Franzoni & Sauermann, 2014). Crowd science carries the potential to lift data treasures or to analyze data that is too large for a small research team, but at the same time too unstandardized for computational research methods. While such approaches have been used successfully in the natural sciences and the digital humanities, they are rare in the social sciences. Hence, we know only very little about the particular challenges of this approach, its fit to certain research questions or types of data (Scheliga et al 2018).

Methods & Data: In this talk, we report and reflect about our crowd-science approach that we used to utilize data on the social relationships among entrepreneurial groups (Ruef 2010). Starting from a core data set based on administrative data (Weinhardt and Stamm 2019), we designed a crowd science task that asks participants to research information on company websites and in news articles on predefined cases of entrepreneurs in order to enrich our overall data set. To implement this task, we set up our own crowd science platform that moderated task distribution and the collection of the researched information. In order to qualify the crowd, in our case students in the social sciences across Germany, for this task we offered a 45 min online training on the methodology of process-generated data. After completion, participating students could engage in the research task, and by doing so, collect points and win prizes.

Results: We discuss the methodological challenges, from extracting and combining the information from the different sources as well as pragmatic challenges from setting up a multi-purpose online platform to finding and motivating participants.

Added Value: These insights and reflections advance the methodological discussion on crowd science as digital method and initiate a discourse on the potentials and shortcomings of combining data sources via platform moderated data collection.

Relevance & Research Question:

In Germany, the utilization of trains in the long-distance traffic has risen in the last 10 years from about 44% (2008) to 55% (2018). Further demand growth is stipulated by the German government for the coming years. The goal is to double the number of passengers by 2030. While demand has so far primarily been controlled by a Revenue Management system (saver fare and super saver fare), the question arises whether controlling and smoothing demand is also possible through non-price measures.

Methods & Data:

Based on forecast data, capacity utilization for each journey is estimated. Using these data, a display system was developed (4 icons), which provides customer information on the expected utilization of a single train connection on bahn.de. After a concept phase, qualitative research as well as A/B testing was performed. Finally, in April 2019, the display system was introduced on all major distribution channels. Recently, ticket buyers have been surveyed: here, one study focused on ticket buyers ( Jan.-Oct 2019, n=>10.000), the other study surveyed visitors of bahn.de who did not buy a train ticket (Oct. 2019, n=2.000).

Results:

By using a multi-source multi-method approach, there are clear and consistent indicators for several positive effects of the utilization forecast icons: first, there is a shift in demand towards less utilized trains (thus achieving the goal of demand smoothing), secondly, seat reservation quota is increased and thirdly, the information leads to a comfort improvement for the travelers. However, it can also be seen that in time windows with overall high train utilization, sometimes a loss of customers takes place.

Added Value:

On the one hand, the combination of big data, experimental design and online surveys generates the database for displaying icons (load forcast) at the same level as train connections and fares on bahn.de, while on the other hand, during the period of market introduction (as of May 2019), key information can be obtained leading to a 360-degree perspective, generating deep insights into the effects for Deutsche Bahn as well as for railway customers. Furthermore, starting points for optimizing the displayed icons are identified.

Relevance & Research Question: Keywords: E-government, ethic affiliation, social inequality, trust in government.

Studies in the field of digital government have established the existence of a two-way association between e-government use and trust in government. Yet to date, no study has examined the interactive effect of ethnic affiliation and e-government use on trust in government or the interactive effect of ethnic belonging and trust in government on e-government use. The current study investigated these effects by means of social inequality approaches outlined in Internet sociology studies.

Methods & Data: Keywords: Social survey, categorical regression.

This study has used the data from the 2017 Israel Social Survey. The findings were received from the multivariate categorical (logistic and ordinal) regression models.

Results: Keywords: Ethnic minorities, trust-use interaction.

The study found that Arabs from small localities with varying levels of trust in government (except for those with the highest level) are less likely to use e-government than Israeli Jews with the same levels of trust, yet they are more likely than Israeli Jews to have some degree of trust in government. Arabs from large localities differ from Israeli Jews in terms of e-government use only when they have some degree of trust in government, but they do not differ from Israeli Jews regarding the trust itself. Except for variations in predicted probabilities, no differences were found between the two Arab groups with respect to either of the criteria.

Added Value: Keywords: Locality size, social stratification.

The results provide support for the social stratification approach and in general provide justification for treating disadvantaged minorities according to the size of their residential localities.

Relevance & Research Question:

In the recent decade, social media has been identified as an important source of digital trace data, reflecting real world behaviour in an online environment. Many researchers have analyzed social media data, often text messages, to make inferences about peoples attitudes and opinions. Yet many such opinions and attitudes are not saliently expressed, but remain implicit. One example are gender role attitudes, that are hard to measure using textual data. In this regard, images posted on social media such as Instagram may be better suited to analyze the phenomenon. Existing research has shown that men and women differ in how they portray themselves when being photographed (Goffman 1979, Götz & Becker, 2019, Tortajada et al., 2013). Our study is concerned with the question how images from social media containing gender self-portrayal can be harnessed as a measure of gender role attitudes.

Methods & Data:

We rely on about 800,000 images collected from Instagram in 2018. We present a new approach to quantify gender portrayal using automated image processing. We use a body pose detection algorithm to identify the 2-dimensional skeletons of persons within images. We then cluster these skeletons based on the similarity of their body pose.

Results:

As a result we obtain a number of clusters which can be identified as gender typical poses. Examples of typical female body poses include S-shaped body poses reflecting sexual appeal, the feminine touch (touching the own body or hair) implying insecurity, or asymmetric body posture representing fragility. Typical male body poses include the upper body facing the camera square to show strength, or a view aimed into the distance signifying pensiveness.

Added Value:

The (self)-portrayal of women and men has been an active field of research across various disciplines including sociology, psychology and media studies, but has usually been analyzed by qualitative means using small, manually labeled data sets. We provide an automated approach that allows for a quantitative measurement of gender role attitudes within pictures by examining gender portrayal via body poses. Our results contribute to a better understanding of online/social media gender reproduction mechanisms.

Relevance & Research Question: The importance of deliberative processes to democracy has been studied for a long time now. As people discuss actual issues, share ideas, and try to change their minds in a friendly, open-minded environment, they become active, aware citizens. The flourishing of Social networking sites has encouraged scholars to examine their potential contribution to deliberative processes, as they enable an abundance of opportunities to deliberate. The current study has examined the inner dynamic of closed Israeli women's groups A quantitative content analysis was conducted (coders reliability = 0.73) to examine 1070 random posts and analysis of the profile of the original post contributors (including some indicators that measured the posts’ entire threads) that were written during December 2017-January 2018. All posts derive from a large and well- known closed Israeli women's group on Facebook (with over 100, 000 members). on Facebook to identify deliberative processes among them.

Methods & Data:

Results: An overwhelming majority of posts (89%) included dialogical elements. Furthermore, in most (94%) of the posts, authors' names, profile pictures, and Facebook's full profile were overt. A positive correlation was found between the level of personal exposure and the depth of discourse that followed the user's initial post (r = .214, p <.001). Although most popular topics of the posts were health (15%), motherhood (13%), relationships with partners (12%), and sexuality (9%), many posts were dedicated to political issues. In these posts, group members were freely discussing actual-political issues in a non-judgmental environment, opening themselves to other ideas and points of view

Added ValueThis study highlights the vital role that closed women's groups on Facebook may play in their members' lives, not only in social and psychological aspects, but also in the sense of practicing deliberative interactions, and therefore strengthening the vital sense of being empowered citizens.:

Relevance & Research Question

As a way to reduce survey length, or even to replace surveys, we have been involved in passive data (web navigation) collection for a couple of years. Passive data is relevant to the description of online behaviour, but declarative data is still needed to interpret and explain behavior; as one could hope to directly infer individual attitudes from internet behavior. Due to the recent advances in natural language, such automated analysis of contents and attitudes is no longer an elusive dream. As an experiment, we have thus used BERT (Google's deep learning based language model) and further proprietary deep learning techniques in order to try and analyze opinions, based on online media consumption.

More precisely, is it possible to instantly, without asking anything, combine passive data and BERT and get a deep understanding of the audience of any website? For example, is it possible to get the specific attitude of visitors to Audi.com towards ecology? Our talk will be based on the presentation of a prototype of an online tool/dashboard. Its objective will be to share the promises and the challenges of this usage of AI.

Methods and Data

The data we use is based on 7000 respondents (from France, Germany, UK) who agreed to install a tracking software. For 347 days on average, we continuously collected the navigation data (urls visited and / or apps used) for each of them. Data is analyzed via BERT properly trained. Realtime vizualisation of the results powered by Tableau.

Results

The quality of results relies on the ability of our neutral network to accurately categorize words in a consistent semantic field. Some results are pretty impressive: without having been trained on these particular fields, “Ronaldo” is associated to football and “parenting” is related to family life. But some are disappointing: “psoriasis” is associated to medicine in general (not even to dermatology only). We will discuss these results and will try to explain them.

Added Value

It is a work in progress, at this stage, the main benefit is to present and discuss concrete applications of AI in market research.

We were wondering: If we find out, how many people have been looking for a specific thing (or basically just words) on the internet in the past, would we be able to calculate their interest in the future, too?

The use of Artificial Intelligence in combination with existing technologies has been repeatedly discussed lately. We were interested in combining AI with traditional trend research and developed Pythia, a tool which forecasts culture and consumer trends. How? By examining Google search data and other sources on new trends, evaluating and structuring them individually. The neural networks analyze huge amounts of data and are trained on search data from the past decade. The trend research tool was created to obtain insights that could be used to find new products, improve them and present them more effectively to have a positive impact on product development by using trend forecasts.

We know what you will be needing to sell and how to interact with your customer in the future.

As of today, Pythia can forecast the latest culture and consumer trends of the next 18 months with 95 percent probability in over 50 countries.

The results and experiences with cooperating companies have supported our initial goal to successfully AI with traditional trend research. In an early cooperation with our Co-Founder Rossmann, Pythia suggested “CBD”, "Ingwer Shots" and many more trending topics in Germany. CBD products have been strong performers in their online shop ever since. The tool also proved to be useful for enhancing polls: 10 days prior to the election of the SPD federal chairman, Pythia predicted the correct result.

Want to know what's going to happen within your business? Ask Pythia.

When we talk about predictive analytics, we should not leave aside a method that has been around for what feels like ages (i.e. at times when the term predictive analytics was not even born yet...), but whose predictive power is still one of the best that the market research toolbox has to offer: conjoint analysis. Its value can be seen simply from the fact that it is still one of the most relevant methods of price and product research and is used globally.

In contrast to what is commonly known as predictive analytics, however, conjoint is not based on existing data, but on data collected in decision-making experiments within the framework of surveys.

As an indirect method, it is free of inflation of pretensions and scale effects, and as a reflection of a real decision situation, it is also able to cover behavioural economics effects.

If we assume that there are essentially three variants of predictive analytics (predictive models, descriptive models and decision models), conjoint analysis even includes all three.

But Conjoint not only helps us to develop better products, but can also help to determine the pricing strategy and improve communication and marketing.

No man is an island and no online user is alone. All human activity is embedded in social context and structure and the rise of social media has made this fact more pertinent to online research. On the one hand, the size and composition of the group individuals interact in, the structure of interactions, and collective or other-based incentives affect individual perceptions, behavior, and outcomes. On the other hand, beyond individual outcomes, group outcomes such as segregation and the unequal distribution of resources matter too. However, analyzing social interactions and groups involves a new set of methodological challenges related to gathering data, reducing data heterogeneity, and addressing the non-independence of observations. In this talk, I will present recent work that uses online surveys, experiments, and digital trace data to study social perception, social interactions, and group effects in context as diverse as social media, wikis, online gaming, and crowdsourced contests.

Milena Tsvetkova is an Assistant Professor in the Department of Methodology at the London School of Economics and Political Science. She completed her PhD in Sociology at Cornell University in 2015. Prior to joining LSE, she was a Postdoctoral Researcher in Computational Social Science at the Oxford Internet Institute, University of Oxford. Milena’s research interests lie in the fields of computational and experimental social science. In her research, she uses large-scale web-based social interaction experiments, network analysis of online data, and agent-based modeling to investigate fundamental social phenomena such as cooperation, social contagion, segregation, and inequality. Her work has been sponsored by the US National Science Foundation and Germany’s Volkswagen Foundation, published in high-impact disciplinary and general science journals such as New Media and Society, Nature Scientific Reports, and Science Advances, and covered by The New York Times, The Guardian, and Science, among others.

Relevance & Research Question: There is an ongoing debate in the survey research literature about whether and when probability and nonprobability sample surveys produce accurate estimates of a larger population. Statistical theory provides a justification for confidence in probability sampling, whereas inferences based on nonprobability sampling are entirely dependent on models for validity. This presentation systematically reviews the current debate and answers the following research question: Are probability sample surveys really (still) more accurate than nonprobability sample surveys?

Methods & Data: To examine the current empirical evidence on the accuracy of probability and nonprobability sample surveys, we collected results from more than 30 published primary research studies that compared around 100 probability and nonprobability sample surveys to external benchmarks. These studies cover results from more than ten years of research into the accuracy of probability and nonprobability sample surveys from across the world. We synthesize the results from these studies, taking into account potential moderator variables.

Results: Overall, the majority of the studies in our research overview found that probability sample surveys were more accurate than nonprobability sample surveys. None of the studies found the opposite. The remaining studies led to mixed results: for example, probability sample surveys were more accurate than some but not all examined nonprobability sample surveys. In addition, the majority of the studies found that weighting did not sufficiently reduce the bias in nonprobability sample surveys. Furthermore, neither the survey mode nor the participation propensity seems to moderate the difference in accuracy between probability and nonprobability sample surveys.

Added Value: Our research overview contributes to the ongoing discussion on probability and nonprobability sample surveys by synthesizing the existing published empirical evidence on this topic. We show that common claims about the rising quality of nonprobability sample surveys for drawing inferences to the general population have little foundation in empirical evidence. Instead, we show that it is still advisable to rely on probability sample surveys when aiming for accurate results.

Relevance

With the German Emigration and Remigration Panel Study (GERPS) we established a new and unique longitudinal data set to investigate consequences of international migration from a life course perspective. This task is challenging, as internationally mobile individuals are hard to survey for different reasons (e.g. sampling design and approach, contact strategy, panel maintenance).

Data

GERPS is funded by the German Research Foundation (DFG) and surveys international mobile German citizens (recently emigrated abroad or recently re-migrated to Germany) in four consecutive waves within a push- to- web online panel design. Based on a probability sample, GERPS elucidates the individual consequences of cross-border mobility and concentrates on representative longitudinal individual data.

Research question

This paper introduces the aim, scope and design of this unique push-to-web online panel study which has the potential for analyzing the individual consequences of international migration along four key dimensions of social inequality: employment and income, well-being and life satisfaction, family and partnership as well as social integration.

Results

We will mainly reflect the effectiveness of our innovative study design (register-based sampling, contacting individuals all over the world and motivate them to follow a stepwise push-to-web panel approach). Up to now we successfully conducted two waves (W1: N=12.059; W2: N=7.438) and our 3rd wave is currently in the field. Due to the information available in the population registers, in W1 we had to recruit our respondents postally, aiming to “push” them to a web survey. However, during the following waves we had been able to manage GERPS as online-only panel.

Added Value

These results can be very helpful to international researchers in the context of surveying mobile populations or researchers aiming to implement a push- to- web survey.

Relevance & Research Question: Millennials (born between 1982 and 2003) witnessed events during their lives that differentiate them from older age cohorts (Generation X, Boomers, and Silents). Thus, one can also expect that Millennials’ web survey participation differs from that of older cohorts. The goal of this study is to compare Millennials to older cohorts on different aspects that are related to web survey participation: participation rates, break-off rates, smartphone participation rate, survey evaluation, and data quality.

Methods & Data: We use data from two probability-based online panels covering four countries: 1) the CROss-National Online Survey (CRONOS) panel in Estonia, Slovenia, and the UK and 2) the German Internet Panel (GIP). We use descriptive and regression analyses to compare Millennials and older age cohorts regarding participation rates, break-off rates, rates of surveys completed with a smartphone, survey evaluation (using two indicators: rate of difficult surveys and rate of enjoyed/liked surveys) and data quality (using two indicators: rate of non-substantive responses and rate of selecting the first answer category).

Results: We find a significantly lower participation rate for Millennials than for older cohorts and a higher break-off rate for Millennials than for older cohorts in two countries. Smartphone participation is significantly higher for Millennials than for Generation X and Boomers in three countries. Comparing Millennials and Silents, we find that Millennials’ smartphone participation is significantly higher in two countries. There are almost no differences regarding survey evaluation and data quality across age cohorts in the descriptive analyses. However, we find some age cohort effects in the regression analyses. These results suggest that it is important to develop tailored strategies to encourage Millennials’ participation in online panels.

Added Value: While ample research exists that posits age as a potential explanatory variable for survey participation and break-off, only a small portion of this research focuses on online panels and even less consider age cohorts. This study builds on Bosch et al. (2018), testing some of their hypotheses on Millennials and older cohorts, but it also extends their research by testing new hypotheses and addressing some of their methodological limitations.

Relevance & Research Question:

The Department for Education (in England) sought to understand whether survey estimates for the Childcare and early years survey of parents (CEYSP), a random probability face-to-face survey of around 6,000 parents per year, and an Official Statistic, could be collected using a push-to-web methodology.

Methods & Data:

The face-to-face questionnaire was adapted to follow "Mobile First" principles, using cognitive and usability testing with parents. Three features of the push-to-web survey were experimentally manipulated to explore the optimal design: incentivisation (a £5 gift voucher conditional on completion, vs a tote bag enclosed in the invitation mailing, vs no incentive); provision of a leaflet in the invitation mailing (leaflet, vs no leaflet); and survey length (15 vs 20 minutes).

Survey materials were designed following the Tailored Design Method, using an invitation letter, a reminder letter, and a final reminder postcard.

Results:

The overall response rate to the push-to-web survey was 15.2%, which compares with 50.9% for the most recent face-to-face CEYSP. Of the three experimental treatments, only incentivisation had a significant impact on response: the tote bag increased the response rate by 4.4 percentage points vs no incentive, and the £5 gift voucher increased the response rate by 9.3 percentage points vs no incentive.

A comparison of the responding push-to-web sample profile against that of the most recent face-to-face survey found the push-to-web sample to be biased in certain ways. Parents responding to the push-to-web survey were more highly educated, with higher incomes and levels of employment, lived more often in couple (vs lone parent) families, and lived in less deprived areas of the country. The offer of a £5 gift voucher tended to reduce these biases, whereas the provision of the tote bag tended to exacerbate these biases.

Despite these biases, the push-to-web survey produced similar estimates to the most recent face-to-face survey for certain simple, factual questions. However, greater differences arose for questions relating to parents’ attitudes and intentions.

Added Value:

The survey contributes to our understanding of expected response rates to Government-sponsored push-to-web surveys, and the extent and nature of non-response bias in such surveys.

Relevance & Research Question: Due to rising nonresponse rates and costs, self-administered modes seem a viable alternative to traditional survey modes. However, when planning such a survey in Germany, we identified a lack of evidence on effective incentive strategies and mode choice sequence. Setting up adequate pre-testing was not viable due its costs.

Responsive survey designs (RSD) promise a solution by collecting data across multiple phases. Knowledge gained in prior phases of a survey is used to adjust the survey design in later phases to optimize outcomes and efficiency. Yet, there is a research gap on practical applications of RSD and especially on whether RSD outperform the use of static design (SD) that does not adjust. We address this research gap by comparing outcomes and costs between a RSD and several SDs.

Methods & Data: We drew on a self-administered mixed-mode survey with a RSD that was conducted as part of the German EVS (N~3,200). In the first phase, incentives (5€ prepaid vs. 10€ postpaid) and mode choice sequence (sequential vs. simultaneous) were experimentally varied (2x2). In the second phase, the survey was conducted in the best performing design (5€ prepaid, simultaneous). Our probability sample was randomized across phases and experimental groups. Based on the experiments, we calculated what response rates, risk of nonresponse bias, and survey costs would have been when using SDs instead of a RSD.

Results: Our RSD helped mitigate risks of design decisions: response rate was 10%-points higher and survey costs 13%-points lower compared to the worst SDs. However, because the RSD included four experimental groups that varied in outcomes it did not outperform all SDs. The RSD’s response rate was 4%-points lower and its costs 2%-points higher compared to the best SDs.

Added Value: Our study adds to the sparse knowledge about the feasibility of running RSDs in practice. We show how RSD can be used to conduct a survey under uncertain outcome conditions. Moreover, we highlight that RSDs are faced with an optimizing problem when keeping the learning phases as small as possible but large enough to gain insights.

Relevance & Research Question:

In response to declining survey response rates and a focus on increasing inclusivity, the push-to-web mixed-mode methodology is emerging as a high-quality alternative to postal surveys. Through the NHS Adult Inpatient Survey, part of the English NHS Patient Survey Programme owned by the Care Quality Commission, we are conducting a pilot testing the feasibility of moving a postal (paper-only) survey online through push-to-web methods, and the impact on non-response bias. The pilot will provide insight about the comparability of these methods, through testing a classic postal survey approach alongside a sequential push-to-web, mixed-mode approach (involving paper and SMS reminders).

Methods & Data:

Through the NHS Adult Inpatient Survey, a sample of eligible patients were invited to take part in a non-incentivised survey. Patients were randomly assigned to one of three conditions:

1. Control group (n = 5,221) receive three paper mailings with questionnaires included, as in the current survey design.

2. Experimental group 1 (n = 3,480) receive four paper mailings (with a paper questionnaire included in the third and fourth mailings), and an SMS reminder after each mailing without a paper questionnaire.

3. Experimental group 2 (n = 3,480) receive four paper mailings (with a paper questionnaire included only in the third mailing), and an SMS reminder after each mailing without a paper questionnaire.

Analysis will review overall response rate, percentage completing online, representativeness by key demographic groups and responses to key survey questions for each group. This will provide insight into the cost implications and feasibility of maintaining trends following a move to mixed-methods.

Results:

Fieldwork is ongoing and final results will be available in January 2020. However, preliminary results are encouraging and suggest relatively similar response rates between the control and the experiment groups.

Added Value:

Although previous studies have shown the effectiveness of push-to-web approaches, this pilot provides direct comparability between a non-incentivised, multi-mode contact, push-to-web approach and a classic postal approach on a large-scale survey. The pilot will also provide insight into the feasibility of moving a paper survey online and consider the potential impact on trends and cost effectiveness.

Relevance and Research Question:

Using 360-degree images in research studies presents a lot of benefits for researchers, clients, as well as consumers: it allows us to present more realistic concepts and products for evaluation – and it gives respondents the ability to examine products and concepts in more detail, and in a more realistic context, and thus hopefully increase respondent engagement.

Methods & Data:

In an experimental design, we compared the responses and behavior of respondent being exposed to traditional images (i.e. static/front-facing) vs. 360-degree concepts (N=600 completes). We focused on engagement metrics (direct engagement and passive in survey measures) and measured the possible impact of 360-degree images on the overall survey data.

Results

We will show that unsurprisingly, respondents showed a positive reaction on the new way of displaying concepts / products; in particular, we will highlight how engagement measures increased. We will also discuss the impact on data we observed, and we will present our recommendations on whether or not to we believe replacing traditional images with 360-degree images would impact benchmarks or trends.

Added Value:

This research is examining the impact of the new 360-degree technology on survey data and gives an outlook on how it can be adapted to serve market research needs.

Relevance & Research Question: Taking and uploading images may provide richer and more objective information than text-based answers to open-ended survey questions. Thus, recent research started to explore the use of images to answer web survey questions. However, very little is known yet about the use of images to answer web survey questions and its impact on four aspects: break-off, item nonresponse, completion time, and question evaluation. Besides, no research has explored the effect of adding a specific motivational message encouraging participants to upload images, nor of the device used to participate, on these four aspects. This study addresses three research questions: 1. What is the effect of answering web survey questions with images instead of text on these four aspects? 2. What is the effect of including a motivational message on these four aspects? 3. How PCs and smartphones differ on these four aspects?

Methods & Data: We conducted a web survey experiment (N = 3,043) in Germany using an opt-in access online panel. Our target population was the general German population aged between 18-70 years living in Germany. Half of the sample was required to answer with smartphones and the other half with PCs. Within each device group, respondents were randomly assigned to 1) a control group answering open-ended questions with text, 2) a first treatment group answering open-ended questions with images, and 3) a second treatment group answering with images but prompted with a motivational message.

Results: Overall, results show higher break-off and item nonresponse rates, as well as lower question evaluation for participants answering with images. Motivational messages slightly reduce item nonresponse. Finally, participants completing the survey with a PC present lower break-off rates but higher item nonresponse.

Added Value: To our knowledge, this is the first study that experimentally investigates the impact on break-off, item nonresponse, completion time, and question evaluation of asking respondents to answer open-ended questions with images instead of text. We also go one step further by exploring 1) how motivational messages may improve respondent’s engagement with the survey and 2) the effect of the device used to answer on these four aspects.

Relevance & Research Question:

Today, most recent available voice assistants talk with non-emotional tone. However, with technology becoming more humanoid, this is about to change. From a marketing perspective, this is especially interesting, as the voice assistant’s emotional tone may affect consumers’ emotions which play an important role while shopping. For example, happy consumers tend to seek more variety in product choice and are more likely to engage in impulse buying. Against this background, we explore how the tone of a voice assistant impacts consumers’ shopping behavior.

Methods & Data:

We develop a deep learning model to synthesize speech in German with three different emotional tones: excited, happy and uninvolved. Listening tests with two experts, 120 university students, and 224 crowd workers are performed to ensure that people perceive the synthesized emotional tone. Afterwards, we conduct lab experiments, where we ask 210 participants to interact with a prototypical voice shopping interface talking in different emotional tones and we measure their emotion and shopping behavior.

Results:

Listening tests confirmed very good quality of synthesized emotional speech. Experts recognized the emotion category with almost perfect accuracy of 98%, university students with 90% and crowd workers without any German skills still achieved an accuracy of 71%. The lab experiment shows that the tone of voice impacts participants’ valence and arousal which in turn impact their trust, product satisfaction, shop satisfaction and impulsiveness of buying.

Added Value:

In human-human-interaction, people often catch the emotion of other people. With the increasing use of voice assistants, the question arises whether people also catch the expressed emotion of voice assistants. Several studies manipulating voices found that the same social mechanisms prevalent in human-human-interactions also exist in human-computer-interactions. However, there is also research showing that people interact differently with computers than humans. For example, they are more likely to accept unfair offers by computers than by humans. Considering the contradicting evidence, this study aims to shed light on emotional contagion in the interaction between voice assistants and consumers. This is especially important since voice assistants may potentially reach and impact a huge number of consumers in contrast to one single human shop assistant.

The relationship between user research (user research as part of the user experience design) on the one hand and market research on the other hand has been repeatedly discussed lately. While one part of the community tends to emphasize differences, the other one clearly sees overlaps. We were interested in the subjective reality of professionals with market-research background and user experience research background and how members of each group see their contributions to each phase in a standard development process (early idea gathering, conceptualisation, implementation, market entry and operations). In each of those phases different questions must be answered to ensure the success of the to-be-product/service with the customers. The hypothesis was that each group does not regard its contribution at the same level to each phase, but that they complement each other depending on the phase.

As of today, we collected 37 answers from two groups (user researchers or market researchers) with a qualitative online 10 item questionnaire with open and closed questions. The data collection and processing is ongoing.

The primary results support our assumption. The contribution of the both groups qualifies to be complementing, i.e. when one group sees its contribution as high, the other group regard its contribution as low, and vice-versa. This might be interpreted as if both groups see its contribution as very distinct. Nevertheless, the other answers show that both groups share similar methods, where user research is often more qualitative and market research more quantitative, but not exclusively.

From the results, we propose a structured combination of market research methods (often quantitative) and user research methods (often qualitative) depending on the phases in the product development. Based on the findings we urge every product team to ensure an approach with mixed methods (this should be a no-brainer today) and a mixed team of heterogeneous mindsets, i.e. people coming more from market research and people from user research. The results could be interpreted in a different way too: the transition from market researcher to user researcher and vice-versa might only be a question of the mindset. This is future research.

Mobile diary apps are one of the latest developments in the field of research diaries. They allow participants to provide spontaneous and in-the-moment feedback with their smartphones. By utilising mobiles for digital qualitative research, diary apps represent the next step in closing the gap between participant and researcher.

By providing the opportunity to gather valuable insights on physical and digital product and service usage, smartphone diary apps are especially promising for user research (as part of UX). They allow to identify users’ needs and desires, their usage patterns as well to collect installation, setup and usage feedback over the course of time.

Like other feedback channels, the response behaviour and input quality of smartphone diaries rely heavily on consumers’ motivation to participate and contribute. Engaging participants in a digital diary without the social glue of a community, with little moderator response and without reactions by community peers to their contributions can be quite challenging, especially on the long run. In fact, there appears to be little evidence on how qualitative smartphone diary studies perform as long-term projects with regard to participant engagement.

We will present first results on participant motivation and satisfaction derived from a long-term User experience smartphone diary study with a duration of more than 12 months. We observed relatively high satisfaction rates with the feedback process and the research mode across the user base, as well as low drop-out and non-response rates over the course of the study.

The results show that it is possible to engage consumers and collect insights over a longer period of time in a digital app diary model – making smartphone diaries an interesting alternative to conduct and accompany In-home-Use-Tests and other product and service research models for Market and User experience research the like.

One of the most relevant differences between User Experience (UX) research and market research for us is the creative involvement of the participants in the design of the study settings. UX research is usually the beginning of a user centered innovation approach and provides essential inputs to the future design process. CoCreation methods are an essential part of this research phase to collect sticky information, to uncover user needs & ideas and to consider these results in the further creative process.

Co-Creation can be considered to be a subset or contemporary form of Participatory Design (PD) while using tools and techniques that engender people’s creativity, which is in part motivated by a belief in the value of democracy to civic, educational, and commercial settings.

New technologies (such as virtual reality, artificial intelligent, robotics, etc.) make it possible to reduce the cost of carrying out CoCreation and, moreover, they offer easy access for a broad group of users for collaboration. The focus is particularly on virtual and augmented reality technologies for the implementation of these studies and these technologies provides new possibilities to transform CoCreation into an engaging digital playground for serious collaboration. For example, participants could meet from any point in the world in a virtual workshop setting and work together on topics. Enhanced interaction methods combined with simulation or AI empower non-experts to work on a professional design level for resolving complex challenges. Furthermore, the results of the CoCreation process is immediately available and editable in the virtual world and could be shared on the internet for widespread user involvement.

We will present innovative approaches from ongoing research projects and how virtual CoCreation methods could be used and what we can expect from the future technologies and possibilities. On the one hand we will introduce our H2020 Research Project SHOTPROS and the involvement of Law Enforcement Agencies in the user centered design process. And on the other hand, we will show ideas and approaches with virtual reality in the domain of architecture and urban planning. Especially VR for participatory planning offer a completely new medium to walk through virtual worlds providing a high level of immersion and presence. This will completely change participation: Citizens no longer look at content but become part of the virtual world which is perceived as real and enable people to interact with and feel connected to the world.

In this talk, Frauke Kreuter and Katherine Morris will discuss the worldwide COVID-19 Symptom Tracker Survey that Facebook has launched in the spring of 2020. The presentation will cover methodological aspects of conducting world-wide rapid surveys, challenges in adapting instruments to different cultural and language contexts, and opportunities to combine the data collected in the COVID-19 Symptom Tracker Survey with other COVID-19 data resources. This presentation will focus on the topics such as: (a) why this kind of survey might provide better data quality than the existing snowball samples, (b) what it takes to field a global survey, (c) how to organize data transmission, and (d) envisioned use of the Survey results.

The YouGov COVID-19 Monitor is one of the first globally syndicated, dedicated data trackers for the pandemic, launching in February 2020. The tracker itself covers 26 countries across MENA, APAC, Europe, and the American countries. The aims of the Monitor are threefold: to provide freely accessible data to academics and health organisations; to inform the public through visualisation tools; and lastly, providing consumer behaviour and economic recovery data to clients via our own reporting tool, Crunch.

The following presentation will cover the methodology of the YouGov COVID-19 Monitor, looking at the advantages that online research provides, and considerations for the field that this project has highlighted. In particular, the presentation will focus on the issue of data quality by demonstrating three key needs: the need for a maintained, international array of panels; the need for a regular, fast-paced fieldwork cadence via online survey methods; and the need for a centralised team with connections to local researchers in each region. The presentation will also briefly discuss the relationship between data quality and the narratives that can be drawn from international data.

Relevance and Research Question: We live in a digital age with high level of use of technologies. Surveys have also started adopting technologies for data collection. There is a move towards online data collection across the world due to falling response rates and pressure to reduce survey costs. Evidence is needed to demonstrate that the online data collection strategy will work and produce reliable data which can be confidently used for policy decisions. No research has been conducted so far to assess data quality in the UK NatCen probability-based online panel. This paper is timely and fills this gap in knowledge. This paper aims to compare data quality in NatCen probability-based online panel and non-probability-based panels (YouGov, Populus and Panelbase). It also compares NatCen online panel to the British Social Attitude (BSA) probability-based survey on the back of which NatCen panel was created and which collects data using face-to-face interviews.

Methods and Data: The following surveys are used for the analysis: NatCen online panel, BSA Wave 18 data as well as data from YouGov, Populus and Panelbse non-probability-based online panels.

Various absolute and relative measures of differences will be used for the analysis such as mean average difference and Duncan dissimilarity Index among others. This analysis will help us to investigate how sample quality might impact on differences in point estimates between probability and non-probability samples.

Results: The preliminary results suggest that there are differences in point estimates between probability- and non-probability-based samples.

Added value: This paper compares data quality between “gold standard” probability-based survey which collects data using face-to-face interviewing, probability-based online panel and non-probability-based online panels. Recommendations will be provided for future waves of data collection and new probability-based as well as non-probability-based online panels.

Relevance & Research Question:

The purpose of this paper is to describe the building of a new random sample mixed-mode panel in the UK ('Public Voice'), focusing on its various design features and how each component influenced the final composition of the panel.

Methods & Data:

The Public Voice panel has been built via a combination of two recruitment methods: (i) face-to-face interviewing, and (ii) web/paper surveying. So far as possible, measurement methods have been unified, including the use of a self-completion section within the face-to-face interview for collecting initial opinion and (potentially) sensitive data. The same address sample frame was used for both methods. For this initial phase, the objective was to recruit to the panel c.2,400 individuals, split evenly by method.

Results:

The response rates to the two recruitment survey methods were aligned with expectations (c.40% for the interview survey, c.8% for the web/paper survey) as were the observable biases. Presenting the panel up front (an experimental manipulation) did not lower the web/paper recruitment survey response rate compared to introducing it at the end of the survey. Respondent agreement to join the panel was much higher than expected in the web/paper survey (>90%). Contact details were of generally high quality in the face-to-face and web modes but less so in the paper mode. [More results to come]

Added Value:

This paper adds to the evidence base for what works when building survey panels with a probabilistic sample base. In particular, the use of a dual-design recruitment method is novel.

Relevance & Research: Even with a growing number of Internet users in Germany, a substantial proportion of respondents with Internet access still chose to participate in the mail mode, when given a choice. We know little about the characteristics of those reluctant respondents, as most survey designs do not allow to measure potential predictors of the mode choice process before individuals make a decision. This study aims to fill this gap by investigating which personal characteristics of respondents in a mixed-mode panel are related to their willingness to respond via the web mode.

Methods & Data: We use data from multiple waves of the GESIS Panel, a probability-based mixed-mode panel in Germany (N=5,700). In October/November 2018, a web-push intervention motivated around 20 percent of 1,896 panelists previously using the mail mode to complete the survey via the web mode. We measured potential predictors of mode choice in waves before the intervention. These predictors include indicators of web-skills, web usage, attitudes to the Internet, and privacy concerns. Our study design allows us to investigate how those predictors are associated with mode choice of panelists who switched to the web and those who refused to do so.

Results: Preliminary results suggest that web-skills and web usage are important predictors of mode choice. In contrast, general privacy concerns do not seem to affect the decision to respond via the web mode, but attitudes towards the Internet do.

Added Value: This study will provide new insights into how the characteristics of respondents predict their decision to participate in web surveys. Learning more about the mode choice process and response propensities of web surveys is important to develop effective web-push methods for cross-sectional and longitudinal studies.

Relevance & Research Question:

There can be multiple reasons why data collected in online surveys may differ from the “truth”. Surveys which do not collect data from smartphones for example will include bias from a skewed sample that does not reflect the modern world. The way that individual questions are asked may be subject to inherent biases and some respondents may find survey experience itself frustrating or confusing which will impact their willingness to answer truthfully.

Methods & Data:

This paper will discuss key psychological motivations for respondents to answer surveys truthfully even when this requires them to make more of an effort for the same financial incentive. What drives individuals to tell the truth and how can survey design help to reward such honesty. We will look at number of questioning techniques that reflect real-life decision making and make it easier to for respondents to answer truthfully. Conversely, we will also examine methods for validating data to reduce overclaim from aspirational respondents.

Results:

By conducting a number of research-on-research surveys on the Kantar panel we have seen the direct impact of asking questions across a range of subjects and countries to encourage honesty in data collection and also to validate or trap respondents who are prepared to answer dishonestly. We will present the results of this research and provide some key learnings which can be used directly in online questionnaires.

Added Value:

Many research companies and end-clients use the results of online research as an import part of their insight generation process or tracking studies. By using the techniques that will be presented in this paper they should be assured that we will be collecting higher quality and more honesty respondents from more engaged respondents. This is something that we would encourage anyone involved in the design of online surveys to take some consideration of.

Relevance & Research Question: Researchers have conducted cognitive testing online through the use of web-based probing. However, Lenzer and Neuert (2017) mention that, of several possible cognitive interviewing techniques, they applied only one technique: verbal probing. They also suggest that given the technical feasibility of creating an audio and screen recording of a web respondent’s answering process, future studies should look into whether web respondents can be motivated to perform think-aloud tasks while answering an online questionnaire. Using an online instrument to guide the process, this study demonstrated that unmoderated, think-aloud cognitive interviewing could be successfully conducted online, and that the use of retrospective probes after the think-aloud portion was completed resulted in additional insights.

Methods & Data: Think-aloud cognitive interviewing, immediately followed by the use of retrospective web-based probing, was conducted online using a commercially available online testing platform and separate software for displaying survey instructions and questions. Twenty-five participants tested 9 questions dealing with the cognitive demands of occupations. Videos lasting a maximum of 20 minutes captured screen activity and each test participant’s think-aloud narration. A trained coder used the video recordings to code the think-aloud narration and participants’ answers to the retrospective web-based probing questions.

Results: 25 cognitive interviews were successfully conducted. A total of 41 potential problems were uncovered, with 78% (32) identified in the think-aloud section, and an additional 22% (9) problems identified in the retrospective, web-based probing section. The types of problems identified dealt mostly with comprehension and response-selection issues. Findings agreed with results from a field test of the interviewer-administered questions, with findings from both studies used to revise the survey questions.

Added Value: A think-aloud online test proved successful at identifying problems with survey questions. Moreover, it was easier, faster, and less expensive to conduct the online think-aloud testing and retrospective web-based probing. Online and field testing yielded similar results. However, online testing had the advantage that respondent problems could be shared using videos. And online results had the additional advantage of providing clearer examples of respondent problems, which were then available for use as examples in interviewer training and manuals.

Relevance & Research Question: Online surveys are an established data collection mode that use written language to provide information. The written language is accompanied by visual elements, such as presentation forms and shapes. However, research has shown that visual elements influence response behavior because respondents sometimes use interpretive heuristics to make sense of the visual elements. One such heuristic is the “left and top means first” (LTMF) heuristic, which suggests that respondents tend to expect that a response scale consistently runs from left to right or from top to bottom.

Methods & Data: In this study, we build on the experiment on “order of the response options” by Tourangeau, Couper, and Conrad (2004) and extend it by investigating the consequences for response behavior and data quality when response scales violate the LTMF heuristic. We conducted an experiment in the probability-based German Internet Panel in July 2019 and randomly assigned respondents to one of the following two groups: the first group (n = 2,346) received options that followed in a consistent order (agree strongly, agree, it depends, disagree, disagree strongly). The second group (n = 2,341) received options that followed in an inconsistent order (it depends, agree strongly, disagree strongly, agree, disagree).

Results: The results reveal significantly different response distributions between the two experimental groups. We also found that inconsistently ordered response options significantly increase response times and decrease data quality in terms of criterion validity. These findings indicate that order discrepancies confuse respondents and increase the overall response effort in terms of response times. They also affect response distributions reducing data quality.

Added Value: We recommend presenting response options in a consistent order and in line with the design strategies of the LTMF heuristic. Otherwise, this may affect the outcomes of survey measures and thus the conclusions that are drawn from these measures.

Relevance & Research Question: This study compares the effectiveness of individually targeted and standardized interventions in reducing panel attrition. We propose the use of machine learning to identify sample members with high risk of attrition and to target interventions on an individual level. Attrition is a major concern in longitudinal surveys since it can affect the precision and bias of survey estimates and costs. Various efforts have been made to reduce attrition, such as using different contact protocols or incentives. Most often, these approaches have been standardized, treating all sample members in the same way. More recently, this standardization has been challenged in favor of survey designs in which features are targeted to different sample members. Our research question is: Can personalized interventions make survey operations more effective?

Methods & Data: We use data from the German Internet Panel, a probability-based online panel of the general population in Germany, which interviews respondents every two months. They receive study invitations via email and a 4€ incentive per survey completed. To evaluate the effectiveness of different interventions on attrition, we implemented an experiment in 2018 using a standardized procedure. N = 4,710 sample members were randomly allocated to one of three experimental groups, and within each group were treated in the same way: Group 1 received an additional 10€ incentive, Group 2 received an additional postcard invitation while Group 3 served as control group.

Results: Preliminary results suggest that the standardized interventions were only effective for sample members interviewed for the first time (postcard significantly reduced the attrition rate by 3%-points; incentive no effect), but not for those in subsequent waves. In a further analysis, we conduct a counterfactual simulation investigating the effect of these interventions if 1) only people with high attrition propensities were targeted, and 2) these people received the treatment that was predicted to be most effective for them.

Added Value: This study provides novel evidence on the effectiveness of using personalized interventions in reducing attrition. In 2020, we will develop prescriptive models in addition to the predictive models for actually targeting panel members during fieldwork under a cost-benefit framework.

Relevance & Research Question:

Keeping respondents who have a high likelihood to attrite from a panel in the sample is a central task for (online) probability panel data infrastructures. This is especially important when respondents at risk of dropping out are notably different from other respondents. Hence, it is key to identify those respondents and prevent them from dropping out. Previous research has shown that response behavior in previous waves, e.g., response or nonresponse, is a good predictor of next wave’s response. However, response behavior can be described in more detail, by, for example, taking paradata such as time until survey return into account. Until now, time until survey return has mostly been researched in cross-sectional contexts, which offer no opportunity to study panel attrition. In this innovative study, we investigate whether (a) respondents who return their survey late more often than others and (b) respondents who show changes in their response behavior over time are more likely to attrite from a panel survey.

Methods & Data:

Our study relies on data from the GESIS Panel which is a German bi-monthly probability-based mixed-mode panel (n = 5,000). The GESIS Panel includes data collected in web and mail mode. We calculated the days respondents required to return the survey from online and postal time stamps. Based on this information, we distinguish early, late and nonresponse. Further, we identify individual response patterns by combining this information across multiple waves. We calculated the relative frequency of late responses and the changes in a response pattern.

Results:

Preliminary results show that the likelihood to attrite increases by 0.16 percentage points for respondents who always return their survey late compared to those who always reply early. Further, respondents who change their response timing each wave are 0.43 percentage points more likely to attrite.

Added Value:

The time until survey return is an easily available paradata. We show that the frequency of late responses as well as the changes in response time patterns predict attrition just as good as previously used models that include survey evaluation or available time, which might not always be available.

Relevance & Research Question: For many years, online panels have been struggling with every time lower response rates and shorter membership durations in average. The responses to this threatening challenge are manifold. Simply put, panels either have to lower the quality standards to sustain a high recruitment volume or they have to increase the loyalty and activity rate of its then costlier recruited members. We decided to invest into the longevity of our member’s base by relaunching out panel in 18 European countries and introducing game mechanics to our panelists.

Methods & Data: We have strictly followed a research-based process to identify the motivation and pain-points of our panel members. With the help of focus groups and iterative user testing, we successively developed a panelist centric platform that included a new visual design, new functions for the user and game mechanics to better engage with our members. An integral part of the whole project was (and still is) accompanying research. Among the KPIs we continuously monitor over time are panel composition (i.e. demographics), panel performance (e.g. churn rate, response rate) and panelist satisfaction.

Results: Our first results are very promising. We see that all target groups increased their activity and loyalty level. To our satisfaction, especially hard to reach segments (i.e. young men) experienced a significant boost. As a result, our panel has become more balanced and better performing than before.

The evaluation of this transition is ongoing, especially as we are still introducing new features and making smaller adjustments to existing functions. We are planning to share the current status of this long-term project with the audience of the conference.

Added Value: While comparability of data is a very high value in research, the dynamic nature of digitalization requires us to adapt the method from time to time. Our case shows that research methodology can evolve without compromising its quality standards. We believe that this is partly because the whole process was based on and accompanied by research.