Sizeable majorities of both the app and web groups agreed to take part in the study, and majorities of those who signed up actually participated in one or more of the individual surveys. The web group outperformed the app group in terms of cooperation rates and responses rates, both overall and to each of the 14 surveys. This is perhaps due to the fact that panelists assigned to the web treatment knew they could and did respond via smartphone, tablet, laptop or desktop while the app panelists were restricted to a mobile device.
The experimental web group was more likely to agree to participate in the follow up surveys than the app group. The agreement rate was 90% for the experimental web group and 76% for the app group. Of all panelists in the app group, 16% did not agree to participate using the app but said yes to using the web (APP2WEB). Overall 91% of those invited to participate in the follow up surveys agreed to do so; however this includes all of the app, experimental web, non-experimental web and APP2WEB groups.
In total, 80% of those who agreed to participate using an app actually downloaded the app. This corresponds to 61% of those invited.
The experimental web group was also more likely to respond to individual survey requests. Of those invited, the overall experimental web response rate (one or more completions) was 84% vs. 58% for the app group. Additionally 53% of the experimental web group vs. 40% of the app group completed 10 or more surveys, and 15% vs. 9% completing all 14 surveys, respectively. Of those who agreed to participate the overall completion rate was 94% for the experimental web group vs. 76% for the app group. Looking at those who completed 10 or more surveys these numbers drop to 59% and 53% respectively and when we look at who participated in all 14 surveys they decrease further to 17% and 12%, respectively.
Looking at the surveys across the field period, the app response rate was lower than the experimental web completion rate at every day and time. Additionally, we see a lower response rate on the first day than the second day for both modes, except when comparing the afternoon web surveys of the first two days. Of the subsequent days, each day is not statistically significantly higher or lower than the day before.
Combining all completed surveys by treatment across all days and times, we looked at when each completed survey was initially accessed within the two hour survey window. We find that a plurality of both the app and experimental web surveys was initially accessed within the first two minutes after the invite was dispatched, but a higher percent of the app surveys were accessed within that time frame compared to the experimental web group. Perhaps the push notifications the app group received, in addition to the text messages and emails both groups received, accounts for some of this increase. It could also be related to the demographics of the app group, discussed below.
The original research plan called for capturing the device used to complete each survey, for each panelist regardless of experimental treatment. However, due to an oversight, the device information for the experimental web group was captured for only the respondent’s last survey. Keep in mind that for some respondents their last survey could have been the second survey and for others the 14th survey. While the vast majority (80%) completed their last survey on a mobile phone, 7% used a tablet and 14% used a desktop or laptop computer.
The app group’s smartphone or tablet operating system was captured for every survey. iOS users (iPhone, iPad) outnumbered Android (smartphone, tablet) users 57% to 42% in the initial survey, a ratio that gradually narrowed to 51% to 48% by survey 14.
The unweighted demographic composition of the study’s participants and the weighted full sample of smartphone users from the October wave of the panel were very similar, suggesting that either experimental group could have been used to represent all smartphone users. However, app respondents represented all smartphone users slightly better than did experimental web respondents, in the sense that they were more likely to be young and not registered to vote. There were some relatively modest demographic differences between the app and web groups. For example, in comparing app vs. web for both those who agreed to participate and those who actually responded, the profile of the app group was younger, more likely to be from the Midwest and less likely to be registered to vote compared to those who agreed to participate and those who responded in the experimental web group.
One possible factor that could have contributed to the age difference between the app and web groups was that numerous panelists had technical difficulties in trying to download the app. Approximately 100 panelists emailed or called the panel help line reporting problems such as not knowing what the Google Play or Apple App Store were, not having their accounts configured to use said stores, not knowing how to download an app or even trying to download the app on their computer. If we assume that these less tech savvy panelists may perhaps have been older, this would skew the demographic composition of the app treatment towards younger panelists who were able to download the app.
In addition to tabular comparisons of cooperating and non-cooperating groups, researchers conducted a multivariate analysis of the factors predicting whether an individual responded as a way to test whether panelists invited to the app treatment were less likely to respond regardless of demographics. The dependent variable was whether an invited panelist responded to any of the 14 follow up surveys or not. The base is thus all invited, not just those who agreed to participate.
The model controlled for age, gender, education, race and ethnicity, census region, marital status, religiously affiliated or not, registered to vote, political party affiliation, treatment group (app vs. experimental web) and two-way interactions between treatment and select demographics (age, education and gender). The interactions were not significant and thus were dropped from the model.
The analysis found that the app group was less likely to respond than the web group even when controlling for demographics. Of the demographics, age was a significant predictor of responding with younger panelists more likely to respond regardless of treatment (app vs. web). Those from the Midwest were more than to respond than those from the West and those who were married were more likely to respond than those who had never been married.
Of greatest interest, of course, are the substantive questions in the study. Do the app and web versions yield different results for the questions at the heart of the research? The answer is no. The weighted1 substantive results from the app and the web groups do not differ. The app and the web groups were each separately weighted to targets for all smartphone users. Responses were compared for which activities respondents did at least once over the week, in which location, for what purpose and how it made them feel. On only one item out of 34, “taking pictures or videos,” was the app group significantly different from the experimental web group (app respondents were less likely to report having done so). Thus, this study suggests the choice of using an app or a web implementation of the survey would likely not affect the results of a future study attempting to represent all smartphone users at least on this topic.
