Relevance & Research Question:

The Department for Education (in England) sought to understand whether survey estimates for the Childcare and early years survey of parents (CEYSP), a random probability face-to-face survey of around 6,000 parents per year, and an Official Statistic, could be collected using a push-to-web methodology.

Methods & Data:

The face-to-face questionnaire was adapted to follow "Mobile First" principles, using cognitive and usability testing with parents. Three features of the push-to-web survey were experimentally manipulated to explore the optimal design: incentivisation (a £5 gift voucher conditional on completion, vs a tote bag enclosed in the invitation mailing, vs no incentive); provision of a leaflet in the invitation mailing (leaflet, vs no leaflet); and survey length (15 vs 20 minutes).

Survey materials were designed following the Tailored Design Method, using an invitation letter, a reminder letter, and a final reminder postcard.

Results:

The overall response rate to the push-to-web survey was 15.2%, which compares with 50.9% for the most recent face-to-face CEYSP. Of the three experimental treatments, only incentivisation had a significant impact on response: the tote bag increased the response rate by 4.4 percentage points vs no incentive, and the £5 gift voucher increased the response rate by 9.3 percentage points vs no incentive.

A comparison of the responding push-to-web sample profile against that of the most recent face-to-face survey found the push-to-web sample to be biased in certain ways. Parents responding to the push-to-web survey were more highly educated, with higher incomes and levels of employment, lived more often in couple (vs lone parent) families, and lived in less deprived areas of the country. The offer of a £5 gift voucher tended to reduce these biases, whereas the provision of the tote bag tended to exacerbate these biases.

Despite these biases, the push-to-web survey produced similar estimates to the most recent face-to-face survey for certain simple, factual questions. However, greater differences arose for questions relating to parents’ attitudes and intentions.

Added Value:

The survey contributes to our understanding of expected response rates to Government-sponsored push-to-web surveys, and the extent and nature of non-response bias in such surveys.

Relevance & Research Question: Due to rising nonresponse rates and costs, self-administered modes seem a viable alternative to traditional survey modes. However, when planning such a survey in Germany, we identified a lack of evidence on effective incentive strategies and mode choice sequence. Setting up adequate pre-testing was not viable due its costs.

Responsive survey designs (RSD) promise a solution by collecting data across multiple phases. Knowledge gained in prior phases of a survey is used to adjust the survey design in later phases to optimize outcomes and efficiency. Yet, there is a research gap on practical applications of RSD and especially on whether RSD outperform the use of static design (SD) that does not adjust. We address this research gap by comparing outcomes and costs between a RSD and several SDs.

Methods & Data: We drew on a self-administered mixed-mode survey with a RSD that was conducted as part of the German EVS (N~3,200). In the first phase, incentives (5€ prepaid vs. 10€ postpaid) and mode choice sequence (sequential vs. simultaneous) were experimentally varied (2x2). In the second phase, the survey was conducted in the best performing design (5€ prepaid, simultaneous). Our probability sample was randomized across phases and experimental groups. Based on the experiments, we calculated what response rates, risk of nonresponse bias, and survey costs would have been when using SDs instead of a RSD.

Results: Our RSD helped mitigate risks of design decisions: response rate was 10%-points higher and survey costs 13%-points lower compared to the worst SDs. However, because the RSD included four experimental groups that varied in outcomes it did not outperform all SDs. The RSD’s response rate was 4%-points lower and its costs 2%-points higher compared to the best SDs.

Added Value: Our study adds to the sparse knowledge about the feasibility of running RSDs in practice. We show how RSD can be used to conduct a survey under uncertain outcome conditions. Moreover, we highlight that RSDs are faced with an optimizing problem when keeping the learning phases as small as possible but large enough to gain insights.

Relevance & Research Question:

In response to declining survey response rates and a focus on increasing inclusivity, the push-to-web mixed-mode methodology is emerging as a high-quality alternative to postal surveys. Through the NHS Adult Inpatient Survey, part of the English NHS Patient Survey Programme owned by the Care Quality Commission, we are conducting a pilot testing the feasibility of moving a postal (paper-only) survey online through push-to-web methods, and the impact on non-response bias. The pilot will provide insight about the comparability of these methods, through testing a classic postal survey approach alongside a sequential push-to-web, mixed-mode approach (involving paper and SMS reminders).

Methods & Data:

Through the NHS Adult Inpatient Survey, a sample of eligible patients were invited to take part in a non-incentivised survey. Patients were randomly assigned to one of three conditions:

1. Control group (n = 5,221) receive three paper mailings with questionnaires included, as in the current survey design.

2. Experimental group 1 (n = 3,480) receive four paper mailings (with a paper questionnaire included in the third and fourth mailings), and an SMS reminder after each mailing without a paper questionnaire.

3. Experimental group 2 (n = 3,480) receive four paper mailings (with a paper questionnaire included only in the third mailing), and an SMS reminder after each mailing without a paper questionnaire.

Analysis will review overall response rate, percentage completing online, representativeness by key demographic groups and responses to key survey questions for each group. This will provide insight into the cost implications and feasibility of maintaining trends following a move to mixed-methods.

Results:

Fieldwork is ongoing and final results will be available in January 2020. However, preliminary results are encouraging and suggest relatively similar response rates between the control and the experiment groups.

Added Value:

Although previous studies have shown the effectiveness of push-to-web approaches, this pilot provides direct comparability between a non-incentivised, multi-mode contact, push-to-web approach and a classic postal approach on a large-scale survey. The pilot will also provide insight into the feasibility of moving a paper survey online and consider the potential impact on trends and cost effectiveness.