Determinants of Item Nonresponse in the German Internet Panel
University of Mannheim, Germany
Relevance & Research Question
Accurate collection and processing of survey data plays an important role in today's data-driven society. The prevention and treatment of measurement error that may occur during these processes have so far been in the focus of several survey methodologists. Item nonresponse (INR) as one source of nonobservation error occurs if participants of a survey leave selected questions unanswered and should not be neglected as it might reduce overall data quality. In my bachelor thesis, I investigated the impact of respondent characteristics and question types - complex, sensitive and neutral ones - on the occurrence of INR using data of the German Internet Panel. Furthermore, applying a multi-level logistic regression, I examined potential interaction effects between both, sociodemographic characteristics and question types. As in online surveys respondents do not face any presence of an interviewer, their overall response behavior might somehow deviate compared to rather traditional sorts of interviews (e.g. CATI, CAPI or PAPI). In particular, older participants might be less familiar with the internet which may be reflected in the answers they provide. The profound analysis of nonsubstantive responses patterns may contribute to a better understanding of its underlying mechanisms and thus help preventing future INR in online surveys.
Methods & Data
In order to investigate the impact of question types, all questions of the questionnaire were evaluated with regard to potential complexity or sensitivity. Therefore, a coding scheme was created, examining each single question of the questionnaire. To detect correlates of INR, two multivariate regression models were developed. The first one included the isolated impact of sociodemographic characteristics and question types on INR, whereas in the second model potential interaction effects were taken into account. Due to the nested data structure, where responses are nested within respondents, a multi-level analysis was applied. As the outcome variable of interest was considered dichotomous, logistic regression was used.
The data were taken from wave 16 of the German Internet Panel. In general, the survey contains questions concerning economic and political attitudes and is obtained via online, self-administered questionnaires.
The findings provide some interesting insights into the patterns of nonsubstantive response behavior. Most independent variables showed to have a significant effect, with complex questions having the highest chance to produce INR. In terms of interaction effects, it could be found that people of lower cognitive abilities happen to struggle more with complex questions than people of higher abilities. In this context, also a gender gap could be proven. With regard to sensitive questions, there exists a weak but still significant connection to rising age.
Conclusion & Added Value
The findings suggest, that on the one hand, item nonresponse is mainly a function of specific respondent characteristics such as cognitive abilities and gender, respectively question characteristics. On the other hand, cross-level interactions between both, respondent and question characteristics, seem to play a crucial role. These results have important implications in terms of the overall questionnaire design, especially order and content of questions. As the application of any subsequent processing is still costly and time-consuming, potential error sources should always be prevented in advance. With my research on item nonresponse, I seek to contribute to the sustainable improvement of data collection through surveys and thus to enable a solid base for further empirical social research.
Browsing vs. Searching – Exploring the influence of consumers’ goal directedness on website evaluation
Westfälische Wilhelms-Universität Münster, Germany
Relevance & research questions: The increased flexibility users experience through different devices and mobile Internet is changing the way they consume content from the Internet. In 2015, a representative study of the German population found that users do not only go online to fulfill specific goals such as finding a piece of information (76 % of the participants), but they also browse the Internet for recreational purposes such as for reading articles (59%) or watching videos (53%) (ARD/ZDF-Onlinestudie, 2015). While searching for information can be seen as a goal-directed task, Internet visits for the purpose of recreation do not require the user to undertake specific tasks. For instance, when killing time users may visit the Internet with no specific goal in mind. They explore and browse. In research this differentiation between goal-directed and exploratory online behavior has been widely recognized (e.g. Hoffmann & Novak, 1996; Hassenzahl, 2005; Wang, Wang, & Farn, 2007). Although it is often assumed that during website interaction the type of task may change users’ perception (e.g. Van Schaik & Ling, 2009), very few studies experimentally manipulate the given tasks to assess possible interactional effects. This thesis aimed to fill this gap by investigating how goal-directed (searching) or exploratory (browsing) interactions performed on a website moderate the impact of key website attributes on users’ judgements. Such key elements, namely content, perceived usability, and aesthetics have already been identified by various researchers (e.g. Thielsch, Blotenberg, & Jaron, 2014). Furthermore, the current study distinguished between three post-use judgements. Besides the overall impression of a website, the intention to revisit or recommend a website was assessed. Based on psychological models following moderating effects were expected. The influences of content and usability perception were assumed to increase in searching tasks, while aesthetics was hypothesized to be of greater importance when browsing.
Methods & data: The survey was conducted online in English and German. In total, 481 participants (21.8 % female) took part, of whom 319 were German and 163 were English speaking. Ages ranged from 14 to 69 years with a mean age of 34.84 (SD = 10.68). Two different websites of competing brands in the premium and luxury fashion segment were used as stimuli. Therefore, familiarity to the online shopping context was given. Participants were randomly assigned to one of the two conditions: free exploration (browsing) or fulfillment of specific tasks (searching). In the searching condition participants were instructed to compare prices of two products and to answer questions about product materials. Both tasks were followed by items measuring the predictor variables: Perceived website usability was measured using the English questionnaire PWU (Perceived Website Usability; Flavián et al., 2006) and its German version WWU (Moshagen, Musch, & Göritz, 2009; Thielsch, 2008). For measurement of aesthetics the VisAWI (Visual Aesthetics of Websites Inventory; Moshagen & Thielsch, 2010), was employed. The Web-CLIC served to measure the subjective perception of content facets (Thielsch & Hirschfeld, under review). Finally, participants rated their intention to revisit or recommend the website and provided their overall impression.
Results: To assess the predictive strength of content, perceived usability, and aesthetics, hierarchical and stepwise multiple linear regressions were conducted using overall impression, recommendation, and revisit intention as outcome variables. Overall, content and aesthetics contributed significantly to revisit and recommendation intention as well as overall impression. Content had the greatest impact on revisit and recommendation intention, whereas aesthetics was the most influential for overall impression. Perceived usability only contributed to overall impression. Moderation analyses further analyzed the strength of those relationships for searching and browsing tasks. Interestingly, perceived usability was not influenced by the type of task in any of the calculated models. However, contrary to the hypotheses the predictive strength of aesthetics was stronger when searching. Yet, this change did not show significance in the moderation analysis. Only the influence of content was, in some models, significantly stronger when browsing. Taken all together, browsing or searching did not consistently change the way users were influenced by content, aesthetics or usability.
In additional analyses, differences in main effects were found for English and German participants, and for a company’s registered and non-registered customers.
Added value: Three main conclusions for researchers and professionals can be drawn from this study. First, the proposed assumptions regarding the influence of browsing and searching tasks cannot be supported: The perception of neither usability nor content showed stronger influences on users’ judgements in searching tasks. Aesthetics was not of greater importance when browsing. In fact, to some degree, opposite effects were found. For professionals, this may lead to important conclusions that contrast intuition. For instance, when dealing with highly goal-directed users (e.g. searching for information, comparing different products) perception of usability does not become overly important. Contrary it might be as important to aim for high aesthetical appeal as communicating product information. In sum, results imply that the importance of each construct may not strongly depend on the type of task, but rather differs for spontaneous and more complex evaluations. Content seems to be the strongest driver for building more complex decisions such as revisit or recommendation intention whereas aesthetics is highly influential for spontaneous and overall evaluations of a website. This results in a second implication. Being valued for its simplicity a great number of companies use a measurement of customer satisfaction called the Net Promotor Score (NPS), which consists of one question asking for users' intention to recommend a website (e.g. Reichheld, 2003). However, the current survey suggests that the intention to recommend is highly influenced by content, but not as much by website facets like usability and aesthetics. The two latter constructs seem to be important rather for building an overall impression. Thus, professionals should not only focus on intention to recommend a website but also consider overall impression evaluations such as giving the website a specific grade when measuring success of websites. Third and last, the reported individual differences require further investigations. Website designers should consider adapting their content or aesthetics levels in order to enhance user experience within different cultural backgrounds or for loyal customers.
Attention Dynamics of Scientists on the Web
1GESIS; 2University of Koblenz-Landau
Relevance & Research Question
Over the last years, the estimation of the scientific contribution of scholars got a lot of attention from the academic community. Prize awarding committees and a variety of non-scientific organizations use information about the academic impact to finance the most promising researches, to honor a scientist with a prize, or to hire the best experts in a field. For many years, the citation index has been used as an instrument to estimate the academic contribution of a scientist. Recently, many new methodologies such as Social Science Citation Index and Google Citation Index were developed. Nevertheless, the question of which tool provides the most complete set of citing literature often depend on the subject and publication year of a given article. Therefore, alternative indicators to measure scientific success were introduced. These new measures are often based on online social media. In the meantime, it is not clear how social media react to external shocks such as highly-publicized discoveries and big scientific awards. This research studies collective attention dynamics towards scientists who have and have not been awarded with an important scientific prize. We also examine how online public interest to scientific topics changes as researchers are awarded with some of the most prestigious prizes in their fields. For this, the following questions are considered:
1. Is the success of a scientist determined by the field he or she is working in or is the popularity of the field influenced by the scientist?
2. How does the public react to the success of a scientist?
3. Can we predict the future success of a scientist based on the dynamics of the public attention towards him/her?
Methods & Data
We use Wikipedia page views as a proxy for measuring online attention to scientists. We select a multidisciplinary group of scientists who received the most prestigious awards in their fields and compare it to a group of influential scientists from the same disciplines who did not receive an award. We construct lists of topics associated with the work of each scientist. Then we analyze time lags between creation of Wikipedia articles on these topics and the scientists for both groups. We also analyze whether the trends of online public attention to these topics differ between groups. We perform time series clustering analysis over the group of awarded scientists to understand how the attention dynamics vary between scientists from the different disciplines and prizes. Finally, we perform time series clustering analysis over the dataset of Nobel Prize winners and candidates (represented by Thomson Reuters Citation Laureates) to explore if we can predict Nobel Prize winners based on online attention dynamics.
Both awarded and non-awarded scientists datasets contain the same number of academics from different fields. All together, there are 262 unique researchers in each dataset.
The dataset of awarded scientists focuses on scientists whose work was acknowledged through some of the most prestigious academic prizes. We consider the awards between 2008 and 2015. We have combined a list of winners of the following prizes and awards: Nobel Prize, Abel Prize, Fields Medal, Turing Award, IEEE Medal of Honor, International Prize for Biology, Thomson Reuters Citation Laureates. We manually mapped these winners to the corresponding Wikipedia articles in the English edition.
For a fair comparison with the Awarded Scientists dataset, we combine a dataset of highly cited scientists who worked at the same time and in the same scientific fields as the winners, but who received no prestigious awards. To select these researchers, we turn to Thomson Reuters database on Highly Cited Scientists which provides yearly accounts on the most influential academics in a variety of fields.
For every selected researcher we analyze his or her Wikipedia article to construct a list of scientific topics related to the scientist. For that, we extract all in- and out-links for the articles about scientists in the English Wikipedia. Then, we go through its in- and out-linked articles, collected their category lists, and filter them using a self-created set of stop words. This way, we eliminate links that do not refer to scientific topics. We retrieved a total of 1,911 topics related to the scientists from the Awarded dataset, and 1,070 topics from the Non-Awarded dataset.
1. Is the success of a scientist determined by the field?
We discovered that articles about research topics were created closer to the articles of prize winners than to scientists who did not receive a prize. One explanation could be that the research topics are more closely related to the scientist who got an award. This supports that scientists who received the prize have introduced the topics to the public. It was observed that after a page about a scientist was created, research topics of prize winners received more attention than the topics of scientists who did not receive a prize. Therefore, one can conclude that the popularity of the topics is affected by the popularity of the scientists.
2. How does the public react to the success of a scientist?
The clustering algorithm grouped Nobel Prize, Abel Prize, and Fields Medal winners into a separate cluster based on the attention dynamics. The following trend analysis of the attention dynamics towards the Nobel Prize winners and Nobel Prize candidates showed that the candidates more often demonstrate an increasing trend in attention before the prize was awarded in comparison to the winners. Moreover, Nobel candidates more often show increasing attention dynamics after the award announcement, whereas Nobel Prize winners demonstrate a decreasing trend. The results suggest that the decision of the Nobel committee does not necessarily reflect the current attention towards the scientist. One can say that Nobel Prize winners were not expected by the public and lose their popularity after receiving the award.
3. Can we predict the future success?
The time series clustering analysis of the Wikipedia page views of the articles about Nobel Prize winners and candidates showed that it is difficult to predict the Nobel Prize winners based on the attention dynamics reflected by the Wikipedia page views of the articles about the scientists.
The contribution of the research is twofold. First, the research revealed the interrelation between the success of a scientist and success of the field he is working in. The presented methods can be generalized to investigate how an information network reacts on an event and how the attention spreads from the original subject to the related topics.
Second, our work is relevant for the Altmetrics community, and more generally, contributes to the studies of collective attention and information consumption on the Web.