Conference Agenda

Overview and details of the sessions of this conference. Please select a date or room to show only sessions at that day or location. Please select a single session for detailed view (with abstracts and downloads if available).

Session Overview
A 8: Innovative Topics in Web Surveys
Friday, 20/Mar/2015:
10:15 - 11:15

Session Chair: Henner Förstel, MANUFACTS Research & Dialog GmbH
Location: Room 248
Fachhochschule Köln/ Cologne University of Applied Sciences
Claudiusstr. 1, 50678 Cologne


Teacher judgments in an online virtual classroom environment: the impact of gender-stereotyping on judgment validity

Markus Hörmann

PH Freiburg, Germany

While diagnostic competence is an important part of teachers' expertise, there is still an unsatisfying lack of research outcomes (e.g. Artelt & Gräsel, 2009). Previous results report deviations in the validity of teacher judgments in the classroom of about 20% from standardized assessments (Südkamp & Möller, 2009), a general central tendency (Fiedler, Walther, Freytag & Plessner, 2002) and a more accurate ranking of pupils’ performance compared to absolute measurements (Schrader & Helmke, 2002). Little is known about the reasons for inaccurate judgments as the personality trait of a student is a hardly controllable bias. This experimental online study prevents that bias by an online research paradigm. On the theoretical framework of dual-process models (e.g. Chen & Chaiken, 1999), this study investigates the impact of gender-stereotypes on the evaluation of the student’s aptitude for university studies, motivation and ability.

A web based, virtual classroom using PHP/AJAX-technology was developed based on an offline method (e.g. Fielder, Wöllert, Tauber, Hess, 2013). In this virtual, simulated classroom environment 59 female and 22 male participants interacted in the role of a teacher with three male and three female, virtual students who all had identical average ability (probability of correct answers) and average motivation (probability of raising hands). The participants asked predefined questions related to math or German. Based on the presented video answers, the participants gave evaluations regarding ability, motivation and aptitude of each of the virtual students. For measuring implicit gender-stereotypes as associative strengths, a web port of the Sorting-Paired-Feature-Task (SPF; Bar-Anan, Nosek & Vianello, 2009) was developed.

Significant gender differences following a traditional gender-stereotype regarding the aptitude for engineering (d-Cohen=0.44) and philology (d-Cohen=0.29) appeared. The independence of implicit and explicit gender-stereotypes was proofed.

Besides showing theory consistent results regarding implicit gender-stereotyping and judgments, an online tool to easily examine a large amount of research questions concerning educational psychology was developed. The virtual, online classroom was proofed to be fully suitable for online research. By porting the SPF another important assessment tool in cognitive research is now available to easily conduct online studies. Results of both developed instruments are comparable to offline alternatives.
Hörmann-Teacher judgments in an online virtual classroom environment-192.pdf

Using video to re-invigorate the open question

Pete Cape

SSI, United Kingdom

Relevance & Research Question: Open responses in online surveys are universally awful. Yet many people have a camera and microphone on the very device they’re using to take the survey, what happens if we ask them to video their response?

Methods & Data: An online research project using an “off the shelf” video capture and transcription service. Random assignment to video and non-video treatments.

Results: Our experiments reveal we get more response, more individual answers from each person and more depth, colour and emotion to work with. Judged on character count we obtain up to 6 times more data. These characters are additional substantive content as well as providing rich description and colour. Contrast: A video response: “Hi, I like gas because I like to cook with it, with a range top. It's fast, it's easy, you know it's on, you know what the temperature is based on how big the flame is. I like gas because it's quiet. I like gas because you don't smell it. I think it's cleaner than oil. I know it's less expensive than oil. I like gas, it's maintained better than having to call the oil company to come get it” A written response: “Clean, no smell, easy” Also a change in emphasis on the relative importance of issues; issues that seemed to only affect a minority in a written short answer become more important when people take time to speak their full mind. Data differences are substantial. E.g. “a reliable source” was mentioned by 8% of people who did not make a video, putting it 6th in rank. Almost 3 times as many people in the video group mentioned this (21%) making it a substantial, not a minority, answer.

Added Value: We don’t get videos from everybody so who chooses to answer? Our first experiment suggested that “extremists” want to speak out and so bias reduction is the topic for our second experiment. We conclude with some practical remarks and suggestions: How difficult is it to implement? How can you maximise response? How much does it cost, in Euros and in time?
Cape-Using video to re-invigorate the open question-122.pptx

New survey recruiting strategies: Online Panel vs. Mobile Adverting

Holger Lütters

HTW Berlin, Germany

Relevance & Research Question:

The idea of recruiting participants into online studies is about to radically change.

While the majority of the online market research industry is still inviting people over the email channel, some new approaches of using advertising networks to recruit people literally pop-up.

Besides the largely discussed approach of Google consumer surveys, using the Google advertising eco-system on desktop and mobile devices, a new and independent approach has been developed by a Berlin start-up company. This approach allows inviting unknown people over an advertising partner’s website to take a survey.

The market researcher’s question at this point of technology development is, whether this approach can be considered to create valid data compared to the classic panel approach. In order to find out about the opportunities and problems of this recruiting procedure the standard panel approach was used in competition with the advertising recruiting approach.

Methods & Data:

The comparison of the competing data collection approaches was supervised by independent academic researchers and supported by the commercial online panel GapFish and the advertising recruiting approach as developed by Dalia Research. Overall more than 3.000 people were contacted to create an online sample of more than 800 respondents for each approach equally covering desktop, laptop, tablet and smartphone devices allowing to compare the results.

The device agnostic questionnaire was dealing with respondent’s behaviour when answering questionnaires and their expectations in compensation for taking questionnaires in the classic and the mobile environment.


Overall this first experiment showed comparable results in the data of both approaches. The ad targeting approach can already be considered being an additional option for the researcher. The contribution will explain the effects of both approaches on the data by looking at:

• Different participation patterns on device categories, quota management and drop-outs

• Non-reactive data of the 8 minutes survey

• Reactive data of expectations of the participants in compensation for market research studies

Added Value: The new approach of advertising sampling will be explained during the presentation.
Lütters-New survey recruiting strategies-116.pdf

Contact and Legal Notice · Contact Address:
Conference: GOR 15
Conference Software - ConfTool Pro 2.6.76
© 2001 - 2014 by H. Weinreich, Hamburg, Germany