Relevance and Research Question:

Satisficing response behavior (i.e., taking short-cuts in the response process) is a threat to data quality. Previous research provides mixed-evidence on whether satisficing increases across time in a panel study impairing the quality of survey responses in later waves (e.g., Schonlau & Toepoel 2015; Sun et al. 2019). However, these studies were non-experimental so little is known about what accounts for possible increases. Specifically, past research did not distinguish between the effects of general survey experience (process learning) or the familiarity with specific questions (content learning).

Methods and Data:

Participants of a non-probability German online access panel (n=882) were randomly assigned to two groups. The experimental group received target questions in all six panel waves, whereas the control group received these questions only in the last wave. The target questions included six between-subject question design experiments, manipulating (1) the response order, (2) whether the question included a ‘don’t know’ option, and (3) whether someone received a question in the agree/disagree or the construct specific response format. Our design, in which all respondents have the same survey experience (process learning) allows us to test the hypothesis whether respondents increasingly employ satisficing response strategies when answering identical questions repeatedly (content learning).

Results:

Since the study will be finished by end of March 2021, we conducted preliminary analyses using within-subject comparisons of the first three waves of the experimental group. The question design experiments provide evidence for the appearance of all three forms of satisficing (i.e., primacy effects, acquiescence, and saying ‘don’t know’) in each of the three waves of the study. These response effects have an average magnitude of 10 to 15 percentage points. However, there seems to be no clear pattern of increase or decrease in satisficing over time, disconfirming the content learning hypothesis.

Added value:

Currently, it is unclear how process and content learning affect satisficing response behavior across waves in longitudinal studies. Our findings contribute to the understanding of whether there are unwanted learning effects when asking respondents to complete identical survey questions repeatedly, which is a critical study design to monitor social change.

Relevance & Research Question: Straightlining is one of the important indicators of poor data quality. Straighlining can be identified when respondents give answers to batteries of attitudinal questions. Previous research suggests that the likelihood of straightlining is higher in online mode of data collection when compared to face-to-face interviews and there is a difference in the likelihood of straightlining depending on the choice of device respondents use in mixed-device online surveys. As many social surveys nowadays move to either mixed-mode designs with online mode available for some respondents or even to online data collection as a single mode, it is important to address various data quality issues in longitudinal context. When different batteries of questions are asked in different waves of longitudinal surveys, it is possible to identify whether some individuals consistently choose straightlining as a response style behaviour across waves. This paper addresses the research question of whether there is consistency in straightlining behaviour within individuals across waves in online component of a longitudinal survey? And if yes, what their characteristics are.

Methods & Data: The project uses online components of Understanding Society Survey Waves 8-10. These data provide a unique opportunity to study straighlining across time in an online mixed-device longitudinal survey in the UK context. In Wave 8 around 40% of households responded in online mode and in consecutive waves the proportions were even higher. Longitudinal data analysis is used to address the main research question.

Results: Preliminary results are already available, the final results will become available in June 2021.

Added Value: This project addresses an important issue of data quality in longitudinal mixed-device online surveys. When the individuals who consistently choose straighlining response behaviour across waves are identified, they can be targeted during survey data collection either through real-time data quality evaluation or by using the information about data quality from a previous wave in the current wave. Tailored treatment can then be employed to improve quality of data from these respondents.

Relevance & Research Question:

Comprehending survey questions is an essential step in the cognitive response process that respondents go through when answering questions. Respondents who have difficulties understanding survey questions may not answer at all, drop out of the survey, give random answers, or take shortcuts in the cognitive response process – all of which can decrease data quality. Comprehension problems are especially likely among respondents with low literacy skills. We investigate whether the use of ‘Simple Language’ in terms of clear, concise, and uncomplicated language for survey questions helps mitigating comprehension problems and thus increase data quality. ‘Simple Language’ is a linguistically simplified version of standard language and is characterized by short and succinct sentences with a simple syntax avoiding foreign words, metaphors, or abstract concepts.

Methods & Data:

To investigate the impact of ‘Simple Language’ on data quality, we conducted a web survey of 10 minutes length among 4,000 respondents of an online access panel in Germany in December 2020. Respondents were randomly assigned to a questionnaire in ‘Standard Language’ or to a version of the questionnaire that had been translated into ‘Simple Language’. We compared both groups with respect to various measures of data quality, including item nonresponse, nondifferentiation, and speeding. In addition, we investigate several aspects of respondents’ survey assessment.

Results:

Our findings to date are based on preliminary analyses. We found an effect of language on item nonresponse: Respondents who received the questionnaire in ‘Simple Language’ were more likely to provide substantive answers compared with respondents who received a questionnaire in ‘Standard Language’. The findings regarding other quality indicators seem to be mixed and need further investigation.

Added Value:

The study contributes to a deeper understanding of the benefits of ‘Simple Language’ for question comprehension and data quality in web surveys. In addition, our findings should provide useful insights for improving the survey experience. These insights may be particularly helpful for low literate respondents who are frequently underrepresented in social science surveys.

Relevance & Research Question: In social science research, survey questions with rating scales are a commonly used method in measuring respondents’ attitudes and opinions. Compared to other rating scale characteristics, rating scale direction and its effects on response behavior has not received much attention in previous research. In addition, a large part of research on scale direction effects has solely focused on differences on the observational level. To contribute to the current state of research, we investigate the size of scale direction effects across five- and seven-point rating scales by analyzing observed and latent response distributions. We also investigate latent means and the equidistance between scale points.

Methods & Data: For this purpose, we conducted a survey experiment in the probability-based German Internet Panel (N = 4,676) in July 2019 and randomly assigned respondents to one out of four experimental groups defined by scale direction (decremental or incremental) and scale length (five- and seven-point). All four experimental groups received identical questions on achievement motivation with end-labeled, vertically aligned scales and no numeric values. We used a single question presentation with one question per page.

Results: The results reveal substantial direction differences between five- and seven-point rating scales. Five-point scales seem to be relatively robust against scale direction effects, whereas seven-point scales seem to be prone to scale direction effects. These findings are supported by both the observed and latent response distributions. However, equidistance between scale points is (somewhat) better for seven- than five-point scales.

Added Value: Our results indicate that researchers should keep the direction of rating scales in mind because it can affect response behavior of respondents. This similarly applies to the scale length. Overall, there is a trade-off between direction effects and equidistance when it comes to five- and seven-point rating scales.

Relevance & Research Question:

Implicit Association Testing (IAT) after Greenwald et al. is established for decades now. The first experimental designs using the keyboard to track respondent's answers are still in practice (see project implicit.harvard.edu). Some companies transferred the mechanism from the desktop into the mobile environment without specific adaptation respecting the opportunities of touch screen interactions.

The idea of this new approach is to adapt the established swiping mechanism inspired by the dating app Tinder together with a background time measurement as a means of implicit measurement in brand research.

Method & Data:

The work of C.G. Jung's archetypes serves as a framework to measure the brand relationship strength towards several pharmaceutical vaccine brands related to the fight against COVID-19 on an implicit level with an implicit single association test (SAT).

The online representative sample (n>1.000) drawn from a professional panel in Germany allows the manipulation of several experimental conditions in the mobile only survey approach.

The data collection approach aims to compare the established mechanism of clicking with the approach of swiping answers (Tinder style answers). Contentwise the study is dealing with COVID-19 vaccination brands.

Results:

The analysis shows differences in the answer patterns of those technically deviant approaches. The authors discuss the question of validity of the data collection on mobile devices. Additionally paradata about respondent's behaviour is discussed, as the swipe approach may be a good option to keep respondent's motivation up during an intense interview, resulting in lower cost and effort for the digital researcher.

Added Value:

The study is meant to inspire researchers to adopt their established methodological setting to the world of mobile research. The very serious measurement approach turns out to be even fun for some of the respondents. In an overfished environment of respondents this seems to open a door to even more sustainable research with less fatigue and a higher willingsness to participate. The constribution shows that Serious Tinder Research is more than just a joke (even though it started as a fun experiment).

Background:

When applying multi-item rating scales in web surveys, a key design choice is to decide the number of items that are presented on a single screen. Research suggests that it may be preferable to restrict the number of items that are presented on a single screen and instead increase the number of pages (Grady, Greenspan & Liu 2018, Roßmann, Gummer, & Silber, 2017, Toepoel et al., 2009). In the case of mixed-device web survey, multi-item rating scales are typically presented in a matrix format for large screens such as PCs and a vertical item-by-item format for small screens such as smartphones (Revilla, Toninelli & Ochoa, 2017). For PC respondents, splitting up a matrix over several pages is expected to counteract respondents using cognitive shortcuts (satisficing behaviour) due to a lower visual load as compared with a one large matrix on a single screen. Smartphone respondents who receive the item-by-item format do not experience a high visual load even if all items are on a single screen as only a few items are visible at the same time. However, they have to undergo more extensive scrolling that is supposed to come with a higher amount of fatigue as compared to the presentation of fever items on more screens.

Method:

To investigate the effects of the number of items per screen we will field a survey panel members of the non-probability online panel of respondi in the spring of 2021. Respondents will be randomly assigned to a device type to use for survey completion three experimental conditions that vary the presentation of several rating scales.

Results:

Results will be reported for response times, drop-out rates, item missing data, straightlining, and non-differentiation.

Added value:

This paper contributes to the research on the optimal presentation of rating scales with multiple items in mixed-device web surveys. The results will inform as to whether decreasing the number of items per screen at the expense of more survey pages is beneficial for both the matrix format on a PC and the item-by-item format on a smartphone.

Relevance & Research Question:

Ad spending in social media is projected to reach US$110,628m in 2021. In this context, the smartphone is by far the tool with which people spend the most time on social media. In Germany, the top social media smartphone apps for 2020 were Instagram (23min), YouTube (22min) and Facebook (14min). However, getting access to real and independent performance data for ads shown to specific target groups is technically, and from a data privacy point of view, a huge challenge

Methods & Data:

We have built a new method to access in-app social media advertising data and interaction data on smartphones. By voluntarily installing an app for study purposes, participants passively provide information for all in-app advertisements they see and interact with on Facebook, YouTube, Instagram. To detect and process the data we use machine learning methods and smartphone-sensing technology. Data is only used for study purposes, in compliance with GDPR and German Market and Social Research standards. In a first test study with respondi, we have looked at 50 Facebook-app users who participated for 45 days on average in Feb-May 2021. We saw over 91.000 Facebook ads in total from more than 8.000 publishers – top ad publishers were Amazon and Wish.

Results:

Our methods provide granular data about deployment and success of social media ads from all industries and competitors. They also reveal which target groups are exposed to which ads, e.g. by company and product category. With natural language processing and machine learning algorithms it is possible to improve ad-targeting and ad-content based on real-world ad-performance data: What are most successful ads (i.e. language, text length, emojis), which target group(s) are they served to, and in which frequency. Interaction data from participants (e.g. ad clicks) reveals the viral potential of individual ad campaigns.

Added Value:

Our method offers an easy to use, GDPR-compliant way to analyze real social media ads on smartphones. The app is easy to download and install from the Google Playstore. After installation, it runs in the background without any need for further user-interaction, which minimizes attention bias.

Keywords: Surveys, Social media, Twitter, Data linkage, Consent, Ethics, Cognitive testing

Relevance & Research Question:

Linking survey and social media data can enhance both. For example, survey data can benefit from additional data covering areas not included in the original questionnaire, while social media data can benefit from survey data’s structure and direction.

A key methodological challenge is collecting informed consent. Striking a balance between providing enough information that consent is ‘informed’ while not overwhelming participants is difficult. In addition, previous research has found consent rates to be low, particularly in web surveys, potentially reducing the usefulness of a linked dataset.

Consulting the public can help to ensure protocols developed for asking consent are ethical and effective. This study looks at how can we encourage informed consent to link social media, specifically Twitter, and survey data.

Methods & Data:

This study develops methods previously used for understanding consent to link survey data and administrative records. A total of 25 interviews will be conducted with a purposive sample of British adults using a mixture of cognitive and depth interviewing techniques. Participants will initially be asked to complete a short questionnaire, including a question asking for their consent to link their survey and Twitter data, during which they will be encouraged to ‘think aloud’. Following this, cognitive probes will be used to explore the participants’ decision making process and understanding of the consent question, before opening up into a wider discussion of their attitudes to data linkage of survey and social media data.

Results:

Fieldwork is underway at the time of submission. We expect results to provide insight into people’s understanding of the consent question (and therefore the extent to which any consent decision is informed), and what may be encouraging or discouraging people from consenting.

Added Value:

Findings from this study will help to inform the future design of consent questions, with the goal of improving informed consent rates and therefore data quality. It will also provide evidence of the public acceptability of this approach and how protocols developed for collecting, analysing, archiving and sharing data can best address any concerns.

Relevance & Research Question: Computational social science research on socioeconomic inequality has been constrained by the lack of individual-level socioeconomic status (SES) measures in digital trace data. Even for the most researched social media platform, Twitter, there is an inadequate number of existing studies on estimating the SES of individual users, and most of them have methodological limitations. To fill the gap, we propose an unsupervised learning method that is firmly embedded in sociological theory.

Methods & Data: Following Bourdieu, we argue that the commercial and entertainment brands that Twitter users follow reflect their economic and cultural capital and hence, these followings can be used to infer the users’ SES. Our method parallels an established political science approach to estimate Twitter users’ political ideology from the political actors they follow. We start with the official Twitter accounts of popular brands and employ correspondence analysis to project the brands and their followers onto a linear SES scale. Using this method, we estimate the SES of 3,484,521 Twitter users who follow the Twitter accounts of 342 brands in the United States.

Results: The results show reasonable correlations between our SES estimates and the standard proxies for SES. We validate the measure for 50 common job titles, identifying 61,091 users who state one of the titles in their profile description and find significant correlations between median estimated SES and income (ρ = 0.668, p < 0.001) and median estimated SES and occupational class (ρ = 0.653, p < 0.001). We further use audience estimation data from the Facebook Marketing API to verify that the brands’ estimated SES is significantly associated with their audience’s educational level.

Added Value: Compared to the existing approaches, our method requires less data, fewer steps, and simpler statistical procedures while, at the same time, returns estimates for a larger set of users. The method provides SES estimates on a continuous scale that are operationally easy to use and theoretically interpretable. Social scientists could combine these SES estimates with digital trace data on behaviours, communication patterns, and social interactions to study inequality, health, and political engagement, among other topics.

Relevance & Research Question: Online intermediaries such as social network sites (e.g., Facebook) or search engines (e.g., Google) are playing an increasingly important role in citizens' information diets. With their algorithmically and socially driven recommender systems, these platforms are assumed to cater to the predispositions of users who are - by and large - not primarily interested in news and politics. Yet recent research indicates that intermediaries also foster incidental, i.e., non-intentional exposure to news. We therefore ask: do online intermediaries indeed drive away citizens from news? Or do they actually foster - non-political and political - news exposure? And what is the role of personal characteristics such as education and political interest?

Methods & Data: We recruited 7,775 study participants from online access panels with a continuous web tracking in six countries: France, Germany, Italy, Spain, UK and US. We combine observed data on web browsing behavior for 3 months with the complementary advantages of surveys of the same set of participants. A machine learning model trained on the crawled text of newspaper articles is used to automatically identify political news articles.

Results: The results from random-effects within-between models that separate daily variation from stable behavior show that across countries and personal characteristics, using online intermediaries increases the number of newspaper articles and sources of news consumption. These effects are stable across personal characteristics and countries as well as political and non-political news.

Added Value: The big online platforms counteract societal fragmentation tendencies and news avoidance. As such, the findings have implications for scholarly and popular debates on the dangers to democracy posed by digital high-choice media environments.

Relevance & Research Question: We examine the prevalence and origins of populist alternative news use and the relationship with voting for populist parties in Germany. Empirical evidence of exposure to populist alternative news use in Germany is scarce and is mostly based on inaccurate self-reported survey data.

Methods & Data: We draw from two combined data sets of web-tracking and survey data which were collected during the 2017 German Bundestag campaign (1,523 participants) and the 2019 European Parliamentary election campaign in Germany (1,009 participants). We investigate the relationships between exposure to populist alternative news and political preferences using two-component count data regression modeling.

Results: Results indicate that while populist alternative news draw more interest during first-order election campaigns (Bundestagswahl), they do not reach large user groups. Moreover, most users visit their websites rather seldom. Nonetheless, our data suggest alternative news exposure is strongly linked to voting for populist parties. Our data also shed light on the role of platforms in referring users to populist alternative news. About 40% of website visits originated from Facebook alone in both data sets, another third of visits from search engines.

Added Value: We provide novel insights into the prevalence and origins of populist alternative news use in Germany using fine-granular web tracking data. The large share of populist alternative news use originating from social media platforms fuels debates of algorithmic accountability.

Relevance & Research Question:

Political polls are at the same time questionable and irreplaceable. Elections after elections they show their limits but no other method has yet been proven to be more accurate or reliable.

In this paper, we would like to present the experiment we are conducting about the 2021 german federal elections whose objective is trying to improve opinion monitoring thanks to web navigation data. More precisely, our goal is to enrich insights and improve predictions about voting intention thanks to a combination of survey results and news consumption on the internet.

Methods & Data:

For now more than 5 years, respondi has been involved in combining survey data with passive data. In Germany we operate a nat rep panel (sample size obviously changes every month, as we have to deal with churn, cleansing operation, but we keep it close to n=2500 over time. Panel size is currently n=2541) whose members have accepted to equip (at least) one connected device of theirs with a software which monitors (among other things) which website they visit.

The design of the experiment is the following: we survey these people about their voting intention every week (we plan to conduct 8 waves of interrogation, each of them collecting around 350 completes), and in the mean time we collect all the news (related to the elections or not) they read online (based on our previous observations, we collect around 30k articles per month in Germany).

News articles are classified and summarized using a deep learning language model based on Google’s BERT and fine-tuned for topic detection. We will thus be able to identify patterns of news consumption which are associated with changes in opinion.

Results:

Displayed on a live dashboard powered by Tableau.

Obviously no results are available yet. Our intention is to associate change in opinion with the content read : which message did trigger a change in opinion for which audience ?

Added Value:

Ultimately, if this experiment works, it leads to a new type of election monitoring: real time measurement of change in opinion, and granular explanations of the changes.

For data journalism, the covid pandemic is both an enormous challenge and an encouragement. Hardly ever before has the analysis of current data sets been so relevant for readers, but at the same time sources are far from optimal. Much of the data is not collected in the required depth, is made available in unwieldy formats and is, moreover, only of limited value for a comprehensive assessment of the current situation. Data journalists have responded to this - with innovative formats, new processes and investigations that shed light into the black box that is the pandemic. In this lecture, these developments will be introduced, illustrated with examples and discussed in a broader context.

Relevance & Research Question: Measures of political knowledge are crucial in various fields to determine and explain public and political phenomena. Depending on the research question, researchers are interested in capturing declarative (knowing information) and/or procedural memory (knowing where and how to find information). In web surveys, respondents can look up information easily, thus, confounding a measure of declarative memory with procedural memory. Our study advances existing research on looking up answers to political knowledge questions in important aspects. First, we investigate whether instructions can be used to discourage or even encourage looking up answers. Second, we compare the use respondents’ self-reports of looking up answers and paradata on window switching behavior.

Methods & Data: We implemented a survey experiment in wave 51 of the probability-based German Internet Panel which was fielded in January 2021. We used a between-subject design and randomly assigned respondents to four experimental groups. Group 1 (control group) received three political knowledge questions. Group 2 received an additional instruction encouraging them to look up answers. Group 3 received an instruction discouraging them to look up answers. Group 4 were asked for their commitment to not looking up answers. We captured lookups via self-reports by respondents, paradata on window switching, and a combined measure integrating self-report and paradata.

Results: Preliminary analyses show that providing respondents with instructions significantly affects their behavior. Encouraging instructions resulted in a higher share of lookups compared to the control group. Similar, discouraging them and asking for their commitment reduced the share of lookups compared to the control group. We found these effects across all three measures of looking up answers. Yet, we also found significant differences between the three measures with self-reports indicating the lowest number of lookups and the combined measure indicating the highest number.

Added Value: Our study provides evidence on the use of instructions to encourage or discourage respondents from looking up answers to political knowledge questions. Consequently, instructions can be used to reduce bias. Moreover, we provide insights on the use of paradata to supplement self-reported measures of looking up answers.

Relevance & Research Question: Using reminders is an established practice in survey methodology to increase response rates. Nevertheless, concern is widespread that "late respondents" are less motivated to provide survey data of high quality, e.g., item nonresponse, satisficing. There is evidence that late and early respondents differ in sociodemographic characteristics as well as relevant study outcomes (e.g., attitudinal or behavioural measures). In the continuum resistance model it is assumed that late respondents are similar to nonrespondents, hence, serving as a proxy for nonrespondents. Because the last review on time of responding by Olson (2013) did not address mode differences systematically and because web surveys were not included, we here provide an up-to-date systematic review. With this review we want to answer the question whether late responding varies for the different self-administered survey modes.

Methods & Data: After a comprehensive literature search our preliminary sample consists of 122 published and non-published studies, covering several fields, e.g., health, marketing, political science. We considered studies in English and German from 1980 to 2021. All studies included a comparison between early and late respondents in mail or web surveys and reported either sociodemographic or data quality or study outcome differences. We collected for each study features of publication (e.g., year, type of publication) and study (e.g., sample size, effect sizes, response rate, operationalization of late respondents, number of reminders) via two independent coders.

Results: Our systematic review describes late responding in detail in relation to publication and study features. Hence, our review provides results on the relevance of late responding and different study features with a special focus on the survey mode and its impact on data quality.

Added Value: Our review provides deeper insights into which (web) survey practices lead to which consequences in the trade-off between measurement error and nonresponse bias and on the effect of late responding on data quality.

Literature

Olson, K. (2013). Do non-response follow-ups improve or reduce data quality? A review of the existing literature. Journal of the Royal Statistical Society: Series A (Statistics in Society), 176(1), 129–145. https://doi.org/10.1111/j.1467-985X.2012.01042.

Relevance & Research Question: Questionnaire length has been identified as a key factor affecting response quality. A larger number of questions and the associated respondent burden are deemed to lower the respondents’ motivation to thoroughly process the questions. Thus, respondent burden increasing with each additional question respondents have to work through is likely to lower response quality. However, only little is known so far about the relationship between actual and perceived respondent burden, how this relationship may change over the course of questionnaire completion, and how response quality is affected by this depending on the relative position of the question within the questionnaire.

Methods & Data: A web survey was conducted among respondents of an online access panel using a questionnaire of 25-29 minutes length. The question order was fully randomized, allowing the effects of question position on response quality to be disentangled from the effects of content, format, and difficulty of individual questions. Among these randomly ordered survey questions, a block of evaluation questions on self-reported burden was asked several times. Due to complete randomization of the survey questions and by repeatedly asking the evaluation questions, changes in the actual and perceived respondent burden over the course of questionnaire completion and its effect on response quality could systematically be examined. Several indicators of response quality were taken into account; among others, don’t know responses, nondifferentiation, attention check failure, and length of answers to open-ended questions.

Results: We found only minor effects of actual respondent burden on response quality, whereas higher perceived respondent burden is associated with poorer response quality in a variety of different question types.

Added Value: This study provides evidence of how the actual and perceived respondent burden evolves over the course of the questionnaire and how both affect response quality in web surveys. In this respect, the present study contributes to a better understanding of previous evidence on lower data quality in later parts of questionnaires.

Relevance & Research Question:

The GP Patient Survey (GPPS) is a large-scale, non-incentivised, postal survey with an online completion option. As with other surveys, GPPS has experienced declining response rates and increasing postage costs. A new sampling frame provided access to mobile numbers in 2020, allowing us to experimentally test several push-to-web strategies with multi-mode contact. One experiment tested the feasibility of replacing a postcard reminder with SMS contact.

Methods & Data:

GPPS uses stratified random sampling of all adult patients registered with a GP practice in England. A control group was obtained from the existing sample frame, before selecting an experiment group from the new sample frame using the same criteria. Fieldwork took place simultaneously and tested the following:

1. Control (n=2,257,809) received three mailings with online survey log-in details and paper questionnaires, and a postcard reminder after mailing one.

2. Experiment 1 (n=5,982) received three mailings with online survey log-in details and paper questionnaires, and an SMS after mailing one. Experiment 2 (n=5,978) also received a second SMS after mailing two. Both SMS reminders included unique online survey links.

Results:

Where one SMS replaced the postcard (experiment 1), participants were pushed online compared with the control (27.2% vs. 19.4%) but the response rate was lower (30.4% vs. 31.9%). Sending two SMS reminders (experiment 2) pushed participants online (29.2%) with no significant impact on response rate (31.6%).

Neither demographics nor survey responses were impacted for the experiment group, suggesting no impact on trends. There was some evidence of impact on data quality: non-response increased for questions with long response scales for those completing via SMS (compared with via the GPPS website or letter URL).

The experiment also provided significant cost savings: SMS is cheaper than postal contact, and maintaining the response rate with more online completes reduced return postage and scanning costs.

Added Value:

Although previous studies have trialled SMS reminders, this provides direct comparability between postcard and SMS contact using a large-scale, non-incentivised, general population survey. The results provide insight into the impact on online completion, response rate, trends, non-response bias and cost-effectiveness.

Relevance & Research Question:

With increasing cost of data collection for phone, mail and in-person modes, the need for robust on-line data collection methodologies has never been greater. Text message surveys have a unique advantage in conducting short, quick turn-around surveys in a cost-effective manner. Text message surveys can also be quite effective in reaching harder-to-reach populations. To-date, the use of this methodology has been limited due to concerns of low participation rates and representativeness of text message-based surveys. Also, majority of Text message-based survey research to-date has been conducted via opt-in panels. SSRS has launched the first TCPA-compliant nationally representative probability-based text message-based panel. This paper explores the effectiveness of probability-based Text message survey panel as a data collection methodology.

Methods & Data:

Data collection was conducted via an interactive text message survey (as opposed to sending a web survey link via Text message). The advantage of this methodology is that the survey can be administered via smart phone or other phones thereby improving coverage. No internet service is required as the Text messages are sent via mobile service. To evaluate the effectiveness of Text message-based survey methodology in generating projectable population-based estimates, we conducted Text message survey and a parallel survey fielded via RDD phone.

Results:

In this paper, we provide demographic and substantive comparison of RDD phone and text message-based survey methodology. Our findings suggest that Text message surveys produce very similar results to time-tested RDD phone methodology.

Added Value:

In addition to providing methodological guidance on implementing Text message surveys, this paper also provides best practices guidance in implementation of Text message-based surveys.

Life in Australia™ is Australia’s only probability-based online panel, in operation since 2017. The panel was initially recruited in 2016 using dual-frame random digit dialling (RDD), topped up in 2018 using cell phone RDD as a single frame, expanded in 2019 using address-based sampling (ABS), and topped up in late 2020 using a combination of ABS, interactive voice response (IVR) calls to cell phones, and SMS push-to-web (i.e. invitations using only SMS), noting that a different regulatory regime to the TCPA prevails in Australia, which allows for automated dialling of cell phones and sending SMS without prior consent.

We present our findings with respect to recruitment and profile rates, retention, and completion rates. We also present the demographic profile of panel members and compare it to Census 2016 benchmarks with respect to age, gender, education, and nativity. We supplement our respondent profile findings with results of trials we ran on IVR and SMS as modes of invitation.

The yields from IVR and SMS push-to-web sample were below that of ABS, however the costs for IVR and SMS push-to-web were well below those of ABS and the less expensive modes actually delivered a more desirable panel member profile with respect to age and nativity, though not education. Our research raises interesting questions as to the trade-off between bias, cost and face validity in the form of response rates.

This paper contributes to the international body of research on recruitment methods for probability-based online panels (see, e.g., Bertoni 2019; Bilgen, Dennis, and Ganesh 2018; Blom, Gathmann, and Krieger 2015; Bosnjak et al. 2018; Jessop 2018; Knoef and de Vos 2009; Meekins, Fries and Fink 2019; Pedlow and Zhao 2016; Pew Research Center 2015, 2019; Pollard and Baird 2017; Scherpenzeel and Toepoel 2012; Stern 2015; Vaithianathan 2017; Ventura et al. 2017).

Relevance & Research Question: In 1981, Sherwin Rosen theorized that some

markets, that are showing extreme inequalities in their distributions of income, do this

due to technologies allowing joint consumption and due to poor substitutabilities of

high-quality services by low-quality services. In 2021, artificial intelligence allows us to

investigate if this theory also applies to online-communication and if the reasons for

inequality are the ones described by Rosen. The research question of the present

article is therefore twofold: In a first part we will check if the superstar-phenomen is

also present digitally. In a second step we will see if the reaons for the existence of

the superstar-effect Rosen has given can be confirmed.

Methods & Data: Using a big data-technology called „Social Listening“, roughly 30

million text fragments regarding more than 5,000 german companies were collected

online. Using artificial intelligence, this data was categorized into different event types

and different tonalities (negative, neutral, positive). Gini coefficients as measures of

concentration were then used to get an overview of the inequalites of online-

communication. After that regression analysis was conducted to find evidence to

support or disprove Rosen’s theory.

Results: The data quite clearly show that online-communication is characterized by

quite strong inequalities. This statement is valid for the total number of fragments, all

five topics that are discussed and all tonalities (corrected Gini-coefficient > 0,9). Also,

there is limited evidence supporting Rosen’s theory of superstars (in particular his

explanation for the reasons of superstardom) in the world of online-communication.

Added Value: The results are important for market researchers and marketing

managers as they show the strength of superstardom in online communication. They

also somewhat show the validity and to some degree the limitations of Rosen’s theory.

In addition to that, the study can serve as an example of how big data can be used to

empirically verify the validity of theoretical work. It also hints at the fact that the debate

about the meaning of extreme inequalities of online-communication still needs to be

made.

Relevance & Research Question: Before making changes to a product portfolio or pricing strategy, the brightest minds in any business put effort in assessing the expected impact of such changes on profit or market share. One of the methods in assessing these changes is conjoint. The resulting simulation tool can identify the most optimal product / pricing scenario which promises to maximize value / volume. However, due to certain limitations of the methodology, conjoint gives directional information about the market share only but struggles to consider certain ‘real-life’ circumstances. On the other hand, time series forecasting can be used to predict market share using past ‘real-life’ data such as sales, distribution, and promotion. However, due to its dependency on history, this technique also has its shortcomings: it cannot predict any changes in the market or to a product that never happened before. The problem of using each method in isolation is that one cannot rely only on stated preferences or only on historical data to make an accurate prediction on sales. Can the insights be elevated when combining both data in one model?

Methods & Data: We show an approach to perform a data fusion between the key results of conjoint analysis and time series forecasting. We built one model that is fed with the switching matrix and price elasticities from a conjoint and complemented by time series data of sales, price and distribution. Through parallel optimization a period-based market simulator engine was built.

Results: We can show that this ‘new’ simulator is more suitable for planning yearly pricing strategies since its predictions are more accurate than looking at conjoint or time-series data in isolation. By adding historical data, the impact of promotions and seasonality become visible and lead to more accurate outcomes and insights.

Added Value: A model that takes the best of both worlds – conjoint results and time-series data – provides companies with the possibility to play with all relevant factors in one tool while having a more stable model. In consequence business decisions can be made with greater certainty and decrease the risk of making a wrong decision.

Relevance & Research Question: Within the last decade, research on natural language processing has seen great progress, mainly through the introduction and extension of the word embedding framework. Recently the introduction of models like BERT have led to even greater improvements in the field (Devlin et al. 2018). However, these advancements come at a cost: Word embedding models store the biases present within the training corpora (cf. Bender et al. 2021, Bolukbasi et al. 2016). Other researchers have shown that these biases can also be harnessed to generate valuable e.g., sociological, or psychological insights (e.g., Garg et al. 2018, Kozlowski et al. 2019, Charlesworth et al. 2021). I argue that there are even greater benefits if the contextualization of word embeddings is grounded in triangulation with other data types. I use word embedding models, contextualized with semi-structured interviews, to analyze how street renaming initiatives are perceived as a form of historic reappraisal of Germanys colonial past.

Methods & Data: For this project, two Skip-gram models were trained. The first one on 8.5 years of German national weekly newspaper articles (about 60,000 articles from about 450 issues), the second one was trained on approximately 730 million German tweets posted between October 8th, 2018 and August 15th, 2020. Additionally, semi-structured interviews were conducted and used during the method triangulation. The method developed by Kozlowski et al. (2019) was used to project terms within Skip-gram models onto socially constructed dimensions.

Results: Similar definitions of how colonialism is understood within the research field can be found in both data types. Most interview participants saw value in street renaming initiatives as tool to initiate a public discourse about Germany’s colonial past, to collectively process and reflect on Germany’s colonial heritage. The analysis of the text corpora in conjunction with word embeddings has shown that such a discourse is continuous, but not very prevalent and mostly negatively connotated.

Added Value: Triangulation of Skip-gram model analysis with semi-structured interviews offers additional insights that one of these methods alone would not. If both data types are interpreted with a coherent methodology, this enables new research perspectives and insights.

Relevance & Research Question: In 2020, we saw a broader awakening to the continued systemic racism throughout all aspects of our society and heard renewed calls for racial justice. For the survey and market research industries, this has raised questions about how well our industry does to ensure that our public opinion research captures the full set of diverse voices that make up the United States. These questions were reinforced in the wake of the 2020 election with the scrutiny faced by the polling industry and the role that voters of color played in the election. Given the differential impact of COVID on people of color in the US and the volume of surveys working to understand vaccine hesitancy, the stakes could not be higher for us as an industry to get this right.

Methods & Data: We conducted a study to assess how well online samples represent communities of color and their diversity. While past studies have found lower bias in probability-based samples with online panels compared to opt-in samples (MacInnis et al., 2018; Yeager et al., 2011) there has been little investigation into representativeness among subgroups of interest. In Sept. 2020, we fielded parallel studies on Ipsos’ probability-based KnowledgePanel which is designed to be representative of the US and on opt-in nonprobability online sample with approximately 3,000 completes from each sample source. The questionnaire included a number of measures that could be benchmarked against gold standard surveys such at the Current Population Survey, the American Community Survey, and the National Health Interview Survey.

Results: We found that across all race/ethnicity groups KnowledgePanel had lower bias than opt-in sample. However, in both sample sources, we found that bias was lowest among white respondents and higher among Black and Hispanic respondents. We highlight areas where it appears online samples underrepresent some of the diversity within communities of color.

Added Value: We provide recommendations to improve representativeness with online samples.

Relevance & Research Question: The paper aims at exploring inequality patterns in electoral participation and in different modes of non-institutionalized political participation among young adults in Greece. The main research question is whether youth political participation inequality patterns are shaped by both individual level determinants and the wider socio-economic conditions prevailing in Greece during the recent recession.

Methods & Data: The data derive from the EU-funded Horizon 2020 research project “Reinventing Democracy in Europe: Youth Doing Politics in Times of Increasing Inequalities” (EURYKA). The analysis uses youth-over sampled CAWI survey data of respondents under the age of 35 in Greece. Binary logistic regressions are used to predict Greek young adults’ electoral participation, non-institutionalized protest oriented participation (including demonstrations, strikes and occupations) and non-institutionalized individualized political participation (including boycotting, buycotting and signing petitions).

Results: The inequalities in young adults’ political engagement become most evident for non-institutionalized individualized acts and are less clear for non-institutionalized protest oriented acts and electoral participation. The non-institutionalized individualized modes of political involvement are not so widespread among diverse socio-economic groups among the Greek young population. However, social class determinants are less clearly related to young adults’ protest oriented political participation. Young adults from a broad range of social strata engage in the massive protests, demonstrations and the occupation movements emerged during the recent Greek recession. Similarly, a heterogeneous segment of young adults voted in the parliamentary elections of 2015, which was a landmark in the political scene, as the bipolar political system which dominated the country since the restoration of democracy in 1974, collapsed.

Added Value: Research on youth political participation inequalities is of great importance as young generations represent the emerging political and civic cultures in modern democracies. Political behaviors heavily depend on both the characteristics of individuals as well as of the environments in which they live. Although individual determinants are important in understanding potential inequalities in youth political participation, contextual conditions associated with the recent economic crisis might be decisive in mobilizing a more heterogeneous young population to claim their rights through non-institutionalized protest oriented acts and electoral politics in Greece.

Relevance and research question: In the last decade, internet-enabled mobile devices have become nearly universal, domesticated, and habitual. This paper examines how smartphone use influences broader patterns of internet use and outcomes. Combining domestication theory with media system dependency theory

Methods and data: We use the 2019 Oxford Internet Survey (OxIS), a random sample of the British population (N = 1818), collected using face-to-face in-home interviews. We use principal components analysis to derive three types of dependency on smartphones. Factor scores are used as dependent variables in OLS to identify the characteristics of people who are dependent in each of these three ways. Other regressions show how the three types contribute to ability to benefit from Internet use

Results: We identify three ways in which people have domesticated smartphones: orientation, play, and escape dependency. Each type of dependency has been developed by users with different characteristics. Orientation dependency is characteristic of people who are highly skilled and use their phone for instrumental purposes. Play dependency occurs among people who are less educated. Escape dependency is strong in people who are non-white, in large households and live in urban areas. All three dependencies are major contributors to amount and variety of use. All three also shape internet outcomes: orientation dependency has a positive influence and play and escape dependencies a negative influence on the extent to which they benefit from Internet use.

Added value: The results show that, in addition to demographic and internet skills variables, the ways in which people incorporate mobile devices into their lives has a strong influence on how they use the entire internet and whether they enjoy its benefits. The negative effect of play and escape dependency demonstrates a digital division in which socially excluded individuals tend to domesticate internet technologies in ways that do not give them certain key benefits of the internet. Depending on the internet for play or escape does not improve their ability to find a job, participate in the political system, save money or find information.

Daten aus probabilistischen und nicht-probabilistischen Online Panel-Stichproben sind in der aktuellen Zeit omnipräsent und haben unter anderem eine prominente Rolle in der Erforschung der Auswirkungen der COVID-19-Pandemie eingenommen. Die Daten aus diesen Online Panel Stichproben werden häufig als „(bevölkerungs-)repräsentativ“ und/oder „fit-for-purpose“ bezeichnet. Aber was bedeutet das eigentlich? Und unter welchen Voraussetzungen kann man dies als zutreffend erachten, bzw. annehmen? Und wie kann man „Repräsentativität“ und „Fitness-for-purpose“ eigentlich messen und kommunizieren? Basierend auf existierenden theoretischen Konzepten und dem aktuellen Stand der empirischen Evidenz soll dieser Vortrag dazu beitragen, neue Impulse in die Diskussion um Datenqualität in (nicht-)probabilistischen Online Panel Daten zu geben, und eine transparente und (teil-)standardisierte Kommunikation über Datengüte zu fördern.

In vielen Ländern und Kontexten sehen sich Umfrageforscher mit sinkenden Antwortquoten und steigenden Umfragekosten konfrontiert. Die Datenerhebung ist sogar noch komplexer und teurer, wenn seltene oder schwer zu erreichende Bevölkerungsgruppen befragt werden sollen. In diesen Fällen sind in der Regel alternative Stichproben- und Rekrutierungsverfahren erforderlich, darunter Non-Probability- und Online-Convenience-Stichproben. Ein recht neuer Ansatz zur Rekrutierung seltener Bevölkerungsgruppen für die Teilnahme an Online- und mobilen Umfragen ist die Werbung in sozialen Medien. Social-Media-Plattformen bieten einen relativ kostengünstigen Zugang zu einer großen Anzahl potenzieller Befragter und ermöglichen es, ansonsten schwer zu erreichende Bevölkerungsgruppen zu identifizieren und anzusprechen. Diese Rekrutierung birgt jedoch eine Reihe von Herausforderungen, darunter Untererfassung und Selbstselektion, Betrug und gefälschte Interviews sowie Probleme bei der Gewichtung der Umfragedaten, um unverzerrte Schätzungen zu ermöglichen.

Dieser Vortrag gibt Einblicke in die Möglichkeiten und Hürden, die Social Media Plattformen für die Umfrageforschung bieten. Es werden zwei Social-Media-Stichproben von seltenen Bevölkerungsgruppen vorgestellt und diskutiert. Darüber hinaus werden durch den Vergleich einer Social-Media-Stichprobe mit einer gleichzeitig erhobenen Face-to-Face-Wahrscheinlichkeitsumfrage die Möglichkeiten, die Angemessenheit und die Grenzen der Social-Media-Rekrutierung im Vergleich zu traditionellen Stichprobenverfahren bewertet.

Das Modell zur Informationssystem-Erfolgsmessung von DeLone und McLean ist die dominante theoretische Grundlage der Literatur zum Thema. In das Modell fließen System-, Dienst- und Informationsqualität als abhängige Variablen ein. Weiterhin werden quantitative Nutzung und qualitative Daten in der Erfolgsmessung zusammen und konzertiert eingesetzt. Für professionelle Praxisgemeinschaften, wie sie vor allem durch Etienne Wenger einführt wurden, wird über die Festlegung von Erfolgsfaktoren und Messung der Erfolgskriterien ein Erfolgsmodell erzeugt, wobei auf die kontinuierliche, langfristige und möglichst automatische Erhebung von Daten fokussiert wird. Zu diesem Zweck haben wir mit MobSOS ein Rahmenwerk und eine technische Plattform geschaffen, die es uns erlauben, Erfolgsmodelle kollaborativ zu erzeugen, zu pflegen, Erfolgsfaktoren aus Katalogen zu wählen, mit Messungen zu instrumentieren, Ergebnisse zu visualisieren und durch die Analyse im zugrundeliegenden Informationssystem und der Erfolgsmodellierung gegebenenfalls zu intervenieren. Durch die Lenkung der Aufmerksamkeit von professionellen Praxisgemeinschaften auf Erfolgsmodelle und die dadurch ermöglichte Reflexion werden die Grundlagen geschaffen für das soziale Lernen der Gemeinschaft zur Erhaltung oder gar Steigerung der Handlungsmacht trotz sich wandelnder Praxen und Systeme. Beispiele aus realen Praxen und laufenden Forschungsprojekten werden den Vortrag illustrieren.

Relevance & Research Question: Paid advertisements running on platforms such as Facebook and Instagram offer a unique opportunity for researchers, who need quick and cost-effective access to a pool of online survey participants. However, scholars using Facebook paid advertisements need to pay special attention to the issues of sample biases and cost-effectiveness. Our research explores how Facebook tools and advertisement content can be customized to reach cost-effective and balanced samples across the world.

Methods & Data: In this paper, we are presenting the findings of three online surveys conducted in the United Kingdom, Turkey, and Spain during February and March 2021. In these studies, we explored how two tools offered by Facebook, the choice of campaign objectives and the use of demographic targeting, affected the recruitment process. Campaign objectives affect the menu of optimization strategies available to the advertiser. We compare the performances of three campaign objectives in this study: traffic, reach, and conversion. Facebook also allows researchers to target specific demographic groups for their advertisements. We compare the effects of two ways of demographic targeting, targeting several demographic characteristics at once and targeting only one demographic property at each of our advertisements, along with no targeting.

Results: Our studies reveal a series of important findings. First of all, we were able to collect high-quality samples in each of these countries with low costs. Secondly, we found that while traffic campaigns produce more clicks to our Facebook advertisements, it is conversion campaigns that recruit higher quality surveys for a cheaper price. Our study also demonstrated that the use of demographic targeting is necessary to produce balanced samples, although it might cause smaller sample sizes overall.

Added Value: We believe that our study will help researchers planning to use online surveys for comparative research. Most social scientists conventionally use traffic in their Facebook campaigns. We demonstrate that it is actually conversion campaigns that return cheaper and more high-quality samples. We demonstrate the benefits of demographic targeting, and we also discuss under what conditions demographic targeting becomes most effective for researchers.

Relevance & Research Question: The rise of social media platforms and the increasing proportion of people active on such platforms provides researchers with new opportunities to recruit study participants. Targeted advertisements can be used to quickly and cost-effectively reach large numbers of potential survey participants – even if these are considered rare population members. Although a growing number of researchers use these new methods, so far, the particular danger points for sample validity and data quality associated with the recruitment of participants via social media have largely remained unaddressed. This presentation addresses a problem that regularly arises when recruiting research participants via social media: fake interviews by trolls and bots.

Methods & Data: To address this issue, data from a number of social science surveys – each to recruit rare population members into an online, convenience sample with the help of ads on the social media platforms Facebook and Instagram – are compiled. Using previous findings from the field of online survey research (e.g., Teichert et al. 2015; Bauermeister et al. 2012) as well as extensions for the specific field of social media generated samples, the completed surveys were reviewed for evidence of fraudulent indications. Fraudulent or at least implausible indications, as well as suspicious information in the metadata, were flagged, and thus a fraud index was formed for each survey participation.

Results: Preliminary results indicate that more than 20 percent of the completed interviews could be classified as at least suspicious. Particularly in the case of socially polarizing survey topics, there appears to be a comparatively high proportion of people who deliberately provide false information in order to falsify study results.

Added Value: All insights derived from the various social media fieldwork campaigns are condensed into a best practice guide to handle and minimize issues due to trolls, bots, and fake interviews in social media recruited samples. This guide adds to the knowledge of how to improve the data quality of survey data generated via social media recruitment.

Relevance, Research Question:

Recruiting respondents by placing ads on social networks sites (SNS) such as Facebook or Instagram is a fairly new, non-probabilistic method that provides cost advantages and offers a larger, more targeted sampling frame than existing convenience access panels employ. By using social networks, hard-to-reach groups, such as migrants (Pötzschke & Weiß 2020), LGBTQ individuals (Kühne 2020) can be reached. However, self-recruitment via ads might lead to systematic sample bias. In our study, we employ SNS advertisements to recruit individuals working in the health care sector into an online survey on SARS-CoV-2. This group is difficult to reach with established recruitment methods due to their small number in the overall population. To test the effects of different targeting strategies, three ad campaign designs are compared in an experimental way. The subject of the research is (1) the detailed analyses of self-selection bias and (2) the evaluation of different methodological choices within SNS-based recruitment.

Methods, Data:

To test how well health sector workers can be targeted using the database and algorithm provided by Facebook/Instagram, three recruitment variants will be tested (about 500 respondents per variant): Variant 1: Specifying the industry "health" in the Facebook/Instagram profile; Variant 2: Specifying health as an "interest" in the profile; Variant 3: Recruiting from the total population as a control group. The control group is a critical reference variable to test whether recruitment via statements in the profile is beneficial.

Results:

The study will be fielded in March/April 2021. We will compare the different recruitment strategies and other methodological aspects (e.g., effect of different pictures in the ads) against each other. Further, we will compare the characteristics of respondents recruited with the different recruitment variants against benchmarks from the general population (e.g., gender and age distribution).

Added Value:

The results will add to the sparse empirical evidence and provide recommendations regarding this relatively new methodology. Specifically, three ways of targeting respondents will be experimentally compared. In addition, we will provide evidence on selection bias and compare five different add versions with respect to effectivity of recruiting respondents of the target population.

The method of web probing integrates open-ended questions (probes) into online surveys to evaluate questions. When asking multiple probes, they can either be asked on one subsequent survey page (scrolling design) or on separate subsequent pages (paging design). The first design requires respondents to scroll down the page to see and answer all questions, but they are presented together and independently of the survey question. The latter design presents each probe separately and respondents only see how many and what sorts of probes they will receive by navigating successive survey pages. A third alternative is to implement the probes on the same page as the question being tested (embedded design). This might have the advantage that the probes are directly related to the survey question and the answer process is still available in respondents’ memory. On the negative side, this makes the response task more complex and might affect how respondents answer the survey question presented on the same page.

In this paper, we examine whether multiple probes should be presented on the same page as the question being tested, on a subsequent page that requires respondents to scroll down, or on separate, consecutive questionnaire pages.

Based on a sample of 2,200 German panelists from an online access panel, we conducted a web experiment in which we varied both presentation format and probe order to investigate which format produced the highest data quality and the lowest drop-out rate. Respondents were randomly assigned to three conditions: an embedded design, a paging design, a scrolling design. The study was fielded in November 2020.

We expect the embedded design and the scrolling design to make the response task more complex, resulting in lower data quality compared to the paging design.

We will use the following data-quality indicators: amount of probe nonresponse, number of uninterpretable answers, number of dropouts, number of words per probe, and survey satisfaction. However, research is still work in progress, and therefore results are not available, yet.

The results will provide information on how (multiple) open-ended questions should be implemented to achieve the best possible response quality.

Relevance & Research Question: There is debate about using “since the Coronavirus pandemic began” as a time reference for survey questions. We present an analysis of three open-ended web probes to examine the timeframe respondents had in mind when presented with this phrase, as well as “when the Coronavirus pandemic first began to affect” their lives and why. The following research questions are addressed: How consistently do people understand when “the Coronavirus pandemic began”? To what extent does this align with when the pandemic began affecting their lives? Methodologically, what is the quality of responses to the open-ended probes and how might this differ by key socio-demographics?

Methods & Data: Data are from Round 1 of the Research and Development Survey (RANDS)during Covid-19 developed by researchers at the United States’ National Center for Health Statistics (NCHS). The National Opinion Research Center (NORC) at the University of Chicago collected the data on behalf of NCHS from June 9, 2020 to July 6, 2020 using their AmeriSpeak® Panel. AmeriSpeak® is a probability-based panel representative of the US adult English-speaking non-institutionalized, household population. The data for all three probes is open text. A rules-based machine learning approach was developed to automate the data cleaning for the two probes about timeframes. In combination with hand review, topic modeling and other computer-assisted approaches were used to examine the content and quality of responses to the third probe.

Results: Results show respondents do not have a uniform understanding of when the pandemic began and there is little alignment between when people think the pandemic began and when it began affecting their lives. Preliminary data quality findings indicate most respondents gave valid answers to the two date probes, but a wider range in response quality and variation among key population subgroups is observed for the third probe.

Added Value: This analysis sheds light on use of the phrase “since the Coronavirus pandemic began” as a time reference and helps us understand when and how the pandemic began affecting peoples’ lives. Methodologically, we implemented new and innovative data science approaches for the analysis of open-ended web probes.

Relevance & Research Questions:

Increasing costs of data collection and the issue of non-response in social surveys has led to a proliferation of mixed-mode and self-administered web surveys. In this context, understanding how the design and content of survey invitations influences propensities to participate could prove beneficial to survey organisations. Reducing respondent burden with efficient invitation design may increase the number of early responders, the number of overall responses and reduce non-response bias.

Methods & Data:

This study implemented a randomised experiment where two design features thought to be associated with respondent burden were randomly manipulated: the length of the text and the location of the survey invitation link. The experiment was carried out in a sequential mixed-mode survey among young adults (18-35-year-old) in Iceland.

Results:

Results show that participants were more likely to participate in the initial web survey when they receive shorter survey invitation letters and when the survey link is in the middle of the letter, although further contacts by other modes mitigate these differences for the full survey results. Additionally, short letters with links in the middle perform well compared to other letter types in terms of non-response bias and mean squared error for those characteristics available in the National Register.

Added Value:

These findings indicate that the concept of respondent burden can be extended to mailed survey invitations to web surveys. Design choices for survey invitations, such as length and placement of participation instructions, can affect propensities to respond to the web survey, resulting in cost-savings for survey organisations.

Key words: Surveys, Online panels, Recruitment

Relevance & Research Question:

The recruitment stage is a key step in the set-up of a probability-based panel study: a lower recruitment rate risks introducing bias and limits what subsequent interventions to minimise non-response can achieve. This paper looks at how positioning the recruitment question relative to the offer of an incentive for participating in the recruitment survey, and how staggering information about joining a Panel and allowing participants to say they are ‘not sure’ affects recruitment and participation rates.

Methods & Data:

A split-sample experiment was implemented in the 2020 British Social Attitudes survey, a probability-based push-to-web survey in which participants were invited to join the NatCen Panel. Of 3,964 participants a random half were asked if they would like to join the Panel immediately before being asked what type of incentive they would like, and the other half were asked immediately after.

In addition, a random half were presented with all information about joining the panel up-front, while the other half were presented basic information, but given the option to ask for more information. This group was then provided with more information and asked again but were allowed to say they were still unsure.

Results:

There was no significant difference in the proportion of people agreeing to join the panel or taking part in the first panel survey by the positioning of the recruitment question. In contrast, participants that were allowed to say they were ‘not sure’ were more likely to agree to join the panel, although this difference was no longer significant when looking at the proportion that took part in the first survey wave.

Added Value:

Findings from this study will inform the future design of recruitment questions for panel studies. More generally, it provides evidence on the use of an ‘unsure’ option in consent questions, and how moving away from a binary, ‘in the moment’, approach might affect data collection.

Relevance & Research Question: Technological advancements now allow exploring the potential of voice recordings for open-ended questions in smartphone surveys (e.g., Revilla & Couper 2019). Voice-recordings may also be useful for web surveys covering the general population. It is unclear whether and which respondents show a preference to provide voice-recordings and which respondents prefer to type responses to open-ended questions.

Methods & Data: We report on an experiment that was implemented in the LISS panel in December 2020. Respondents in this experiment were randomly assigned to a voice-recording only, a text-recording only, or a group in which they could select between voice and text recording. We will report who shows preferences for voice-recordings and which factors influence these preferences (e.g., perception of anonymity of data, potential by-standers during data collection).

Results: Preliminary analyses indicate that respondents show strong preferences to provide written instead of voice-recordings. We expect that respondents who are concerned about the anonymity of their data and who had by-standers during data collection are even less willing to provide voice-recordings.

Added Value: This research provides important insights whether voice-recording is a viable alternative for data collection of open-ended questions in general social surveys. The results also reveals factors that need to be addressed to increase the willingness of respondents to provide such data.

Relevance & Research Question: Technological advancements and changes in online survey participation pave the way for new ways of data collection. Particularly, the increasing smartphone rate in online surveys facilitates a re-consideration of prevailing communication channels to naturalize the communication process between researchers and respondents and to collect high-quality data. For example, if respondents participate in online surveys via a smartphone, it is possible to employ pre-recorded audio files and allow respondents to have the questions read out loud to them (audio channel). Moreover, in this survey setting, respondents’ answers can be collected using the voice recording function of smartphones (voice channel). So far, there is a lack of information on whether respondents are willing to undergo this kind of change in communication channels. In this study, we therefore investigate respondents’ willingness to participate in online surveys with a smartphone to have the survey questions read aloud and to give oral answers via voice input functions.

Methods & Data: We conducted a survey with 2,146 respondents recruited from an online access panel. Respondents received two willingness questions – one on the audio channel and one on the voice channel – each followed-up by an open question asking for the reasons of respondents’ (non)willingness to use these communication channels. The study was fielded in Germany in November 2020.

Results: The data are currently still being analyzed. The results of this study will be reported as follows: we first analyze how many respondents reported to be (un)willing to use the audio and/or voice channel when answering a survey. Next, we examine the reasons they provided for their (non)willingness. Finally, we examine which respondent characteristics (e.g., gender, age, educational level, professional qualification, usage of internet-enabled devices, self-reported internet and smartphone skills, and affinity for technology) are associated with higher levels of willingness.

Added Value: This study adds to the scarce literature on respondents’ (non)willingness to answer surveys using the audio play and voice recording functions of their smartphones. To our knowledge, it is the first study to examine the reasons of respondents’ (non)willingness by means of open-ended questions.

Relevance & Research Question: Under the web survey design principles, one of the most cited considerations is reducing respondent burden. This consideration is mostly due to self-administration characteristic of web surveys and respondent owned technology. Often reduced respondent burden is hypothesized to be related to lower nonresponse and measurement errors. One of the operationalizations of reducing respondent burden is adapting widely used technology for other tasks to survey taking. Recently, a widely used technology is noted as digital voice assistants and its adaptation has a potential to improve nonresponse and measurement qualities in web survey data. Pew Research Center published that 42% of the U.S. adults use digital voice assistants on the smartphones (Pew Research Center, 2021).

Methods & Data: This study presents results from a randomized experiment that notes respondents can use voice-to-text instead of typing in 6 open-ended follow-ups in the experimental arm. The application is only presented in the smartphone version of the layout of an address based sampling web survey of the U.S. adult population.

Results: We will report (1) completion rates by device and initiation type (typing, QR code, email link), (2) item nonresponse rates, (3) codeable and noncodeable response rates, and (4) mean number of words in open-ended responses by two experimental arms.

Added Value: Our monthly data since January 2017 show an increase in the completion rates by smartphones and this study will be a baseline study to further understand the general population’s survey taking behavior in smartphones.

Relevance & Research Question: The coronavirus SARS-CoV-2 outbreak has stimulated numerous online surveys that are mainly based on online convenience samples where participants select themselves. The results are, nevertheless, often generalized to the general population. Based upon a probability-based sample that includes online and mail-mode respondents, we will tackle the following research questions assuming that the sample of online respondents mimics respondents of an online convenience survey: (1) Do online (CAWI) respondents systematically differ from offline (PAPI) respondents with respect to COVID-19-related outcomes (e.g., pandemic-related attitudes or behavior) and health characteristics (e.g., preconditions, risk group)? (2) Do internet users (in the CAWI and the PAPI mode) systematically differ from non-internet users with respect to COVID-19-related outcomes and health characteristics?

Methods & Data: The analyses utilize data from the German GESIS Panel, a probability-based mixed-mode access panel that includes about 5,000 online and mail-mode respondents. Upon recruitment, respondents’ preferred mode, i.e., CAWI or PAPI, was determined via a sequential mixed-mode design. The GESIS Panel was among the first surveys in Germany that started in March 2020, collecting data on the coronavirus outbreak. Since then, five additional waves have been fielded, allowing cross-sectional and longitudinal comparisons between the two survey modes (CAWI vs. PAPI) and groups (internet vs. non-internet users), respectively. Statistical analyses address mode and group comparisons regarding COVID-19-related outcomes such as pandemic-related attitudes or behavior as well as health characteristics.

Results: Preliminary analyses reveal only small differences with respect to some behavioral and attitudinal pandemic-related outcomes among the two modes/groups. However, larger systematic differences regarding mode can be reported for health characteristics (e.g., “belong to a risk group”). Further analyses will be conducted focusing on differences among internet vs. non-internet users.

Added Value: With a focus on the current COVID-19 pandemic, the results of this study add to the existing literature that cautions against the use of self-selected online surveys for population inference and policy measures.

Relevance & Research Question:

Commercial nonprobability online panels have grown in popularity in recent years due to their relatively low cost and easy availability. However, a number of studies have shown that probability-based surveys lead to more accurate univariate estimates than nonprobability surveys. Some researchers claim that while they do not produce accurate univariate estimates, nonprobability surveys are “fit for purpose” when conducting bivariate and multivariate analyses. Very little research to date has investigated these claims, which is an important gap we aim to fill with this study.

Methods & Data:

We investigate the accuracy of bivariate and multivariate estimates in probability-based and nonprobability online panels using data from a large-scale comparison study that included data collection in two academic probability-based online panels and eight commercial nonprobability online panels in Germany with identical questionnaires and field periods. For each of the online panels, we calculate bivariate associations as well as multivariate models and compare the results to the expected outcomes based on theory and gold-standard benchmarks, examining whether the direction and statistical significance of the coefficients accurately reflect the expected outcomes.

Results:

Preliminary results on key political variables (e.g., voter turnout) indicate a high variability in the findings gained from the different online panels. While the results from some panels are mostly in line with the expected results based on theory and findings from gold-standard survey benchmarks, others diverge a lot. For example, contrary to expectations, some panel results indicate that older people are less likely to vote conservative than younger people. Further analyses will extend these comparisons to health-related items (subjective health, BMI) and psychological indicators (Big 5, need for cognition).

Added Value:

Research on the accuracy of bivariate and multivariate estimates in probability-based and nonprobability online panels is so far very sparse. However, the growing popularity of online panels as a whole and nonprobability online access panels in particular warrant deeper investigation into the accuracy of the results obtained from these panels and into the question of whether nonprobability panels are indeed “fit for purpose” for such analyses.

Relevance & Research Question: Online surveys, a comparatively low-cost and low-effort medium, have become more and more common in international survey research projects as internet access continues to expand. At the same time, conventional probabilistic sample design is often impossible when utilizing commercial online panels. Especially in regions with comparatively low internet penetration, this poses the question of how well nonprobabilistic approaches can approximate best practice offline methodologies, and what a best practice for online sampling should look like when parts of the population are excluded by default from the sampling frame.

Methods & Data: For this study, we investigated one well-established approach to generating as-good-as-possible nonprobability samples from online panels, sample-matching, in three Latin American countries. In each country, we collected samples of at least 1000 responses each through the standard commercial “black box” approach as well as an original sample-matching approach. This experiment-based approach permits a comparison of matched samples to samples resulting from a panel provider's standard approach, as well as to census extracts and representative population surveys. To assess the quality of each sample, we assess mean average errors for the categories of benchmark questions and of standard demographic indicators, calculated between samples and reference populations.

Results: The results show that the sample-matching approach yields better reproduction of benchmark questions not employed in sample design, compared to the standard one. To conclude, the paper discusses the benefits and drawbacks of choosing a custom sampling approach as opposed to a standard one.

Added Value: We demonstrate that fully transparent and reproducible sampling approaches are possible, if not common, in nonprobabilistic commercial online surveys, and that they can measurably improve the quality of online samples. We also illuminate the possible practical drawbacks in deploying such a custom-made sampling method, adding a useful reference for those wishing to apply such an “outside the black box” approach to drawing samples from online panels provided to the survey research community by commercial firms.