Pew Research Center for the People & the Press
The Impact of "Cell-Onlys" on Public Opinion Polls
Ways of Coping with a Growing Population Segment
The proportion of Americans who rely solely on a cell phone for their telephone service continues to grow, as does the share who still have a landline phone but do most of their calling on their cell phone. With these changes, there is an increased concern that polls conducted only on landline telephones may not accurately measure public opinion. A new Pew Research Center study finds that, while different demographically, Americans who mostly or exclusively rely on cell phones are not substantially different from the landline population in their basic political attitudes and preferences.
On key political measures such as presidential approval, Iraq policy, presidential primary voter preference, and party affiliation, respondents reached on cell phones hold attitudes that are very similar to those reached on landline telephones. Analysis of two separate nationwide studies shows that including interviews conducted by cell phone does not substantially change any key survey findings.
These findings are based on two surveys of adults, conducted Oct. 17-23 and Dec. 19-30, 2007 by the Pew Research Center for the People & the Press. The surveys included interviews with a total of 2,596 adults reached in a conventional landline sample, as well as 841 adults interviewed on their cell phones, using a sample drawn from a nationally representative cell telephone number database. Of those reached on a cell phone, 312 people (or 37%) reported that their cell phone is their only phone.
When data from both samples are combined and weighted to match the U.S. population on key demographic measures, the results are virtually identical to those from the landline survey alone. Across more than 100 political and attitudinal questions on the surveys, including cell phone interviews does not change the results by more than two percentage points in the vast majority of comparisons, and in only one comparison is the difference as large as 4 points.
In particular, there is no evidence that the polling in the Democratic and Republican nomination contests is biased by the fact that most polls rely only on landline interviews. In the December national poll, support for no candidate in the landline sample changed by more than two points when the preferences of cell phone respondents were blended in. The same was true in the October national poll.
There is no doubt that Americans who rely solely on cell phones differ from the rest of the public in some key respects. However, in most cases these differences are the result of their demographic characteristics, particularly the fact they tend to be very young. Since adjustments for age are made in standard landline surveys, adding the cell-only component to the survey substantially increases the raw number of younger people surveyed, but does not alter the overall weight of younger respondents in the final estimates.
In most respects, the political attitudes and behaviors of younger people who are cell-only do not differ substantially from younger people surveys do reach on landlines, meaning that the overall results are virtually identical to those from the landline survey alone.
However, on some non-political topics, and in surveys of certain groups in addition to young people, studies have shown that the inclusion of cell phones in the sample design makes a difference in the combined results. An earlier study by the Pew Research Center for the People & the Press found that blending landline and cell phone samples resulted in higher estimates of young people ages 18 to 25 using new technologies. In addition, small but significant differences were found for lifestyle measures such as attending church and alcohol consumption. In another study from the National Health Interview Survey, Blumberg and Luke (2007) found that for surveys of low-income adults and young adults, the estimates for health risk behaviors, HIV testing, exercise and obesity were all changed when cell phones were included in the sample.
In addition to testing the impact of cell phone sampling, the October and December Pew studies demonstrate the feasibility of including cell phones in telephone surveys. The response rates for the cell and landline samples were virtually identical in both studies, and there is no evidence that the quality of data gathered from cell phone surveys is lower than in landline surveys. Including cell phones, however, is very costly. On average, a cell phone interview costs approximately three times as much as a comparable landline interview.
Although the inclusion of cell phone samples is very costly, and may make little difference in the substantive conclusions one would draw from political surveys, other aspects of the dual frame design provide particular benefits that may argue for the adoption of this type of sampling frame design. Chief among these benefits is the improved demographic representation for certain groups and the attendant increase in the sizes of the samples of these groups for further analysis. This is because it is easier to reach by cell phone than by landline certain groups of respondents who have both types of service.
The inclusion of a cell phone sample may be essential in surveys of population groups that have high rates of cell-only households. More generally, with an estimated 14% of Americans relying solely on cell phones, their exclusion from opinion surveys may call into question the credibility of polls in the mind of the public.
Overview of Differences
Results from both the December and October polls show that the cell-only respondents have somewhat different attitudes and behaviors from those reached on landline telephones. In the December survey, which focused on the public's campaign news sources, cell-only respondents were significantly less likely to say they have watched a presidential debate on television, but more likely to have seen debate video online. This reflects a more general pattern: cell-only Americans are somewhat less likely to rely on newspapers and network evening news for campaign information, but more apt to get campaign news from the internet, late night comedy shows, and to use social networking sites. Not surprisingly, these behaviors are characteristic of younger respondents in general - whether cell-only or not - and the blended results for none of these measures change by no more than two percentage points.
The October survey included questions that asked registered voters about the importance of 16 issues to their vote. There were a few significant differences between the landline respondents and those who were cell-only: the latter group was 14 points less likely to say Social Security would be important to their vote, and somewhat more likely to say immigration would be important. Again, these differences are understandable, given the fact that cell-only respondents are younger (and thus less concerned about Social Security) and more likely to be Hispanic (who are more concerned about immigration). When the cell-only respondents were combined with the landline respondents, none of the overall survey estimates changed by more than one percentage point.
While including interviews conducted by cell phone in a national sample does not substantially affect survey findings, it does improve the overall representativeness of the sample by reaching more respondents in otherwise hard to reach subpopulations. This reduces sampling error for these groups, and may also mean that the survey requires less statistical adjustment to match the demographic profile of the population. Less clear is whether adding cell phone interviews is the most efficient use of resources. Cell phone interviews cost approximately three times as much as landline interviews, and the sample sizes of underrepresented groups can be boosted more cheaply by simply expanding the overall sample size of the landline survey.
Profile of Cell-Only Respondents
One of the most striking differences between cell-only respondents and people reached on a landline telephone is their age. Nearly half of the cell-only respondents (46%) are under age 30 compared to only 12% in the landline sample. Related to their younger age, only 26% of cell-only respondents are married, compared with 57% percent of those in the landline sample. Similarly, about half of cell-only respondents have never been married (51%), compared with only 16% in the landline sample.
In addition, the landline sample includes a higher proportion of college graduates than the cell-only group (38% vs. 26%), which may also reflect the greater use of cell phones among young people who are still in college. The income distribution also is quite different for the landline and cell-only groups; 29% of people in the landline sample have household incomes of at least $75,000 annually, compared with just 16% in the cell-only group. Similarly, nearly twice as many cell-only than landline respondents earn less than $30,000 a year (41% vs. 21%).
Overall, the landline sample includes more whites (82% vs. 68%) than the cell-only group while the cell-only group includes a greater proportion of minorities. In the cell-only group, there are more African-Americans (19% vs. 11%), Hispanics (13% vs. 6%), and Asians (5% vs. 1%) compared with the landline sample. The cell-only group also includes a larger percentage of males than the landline group (61% vs. 48%). Finally, more cell-only respondents than landline respondents are religiously unaffiliated (27% vs. 14%).
Read the full report.
Notes
1Scott Keeter, Courtney Kennedy, April Clark, Trevor Tompson, and Mike Mokrzycki. 2007. "What's Missing from National Landline RDD Surveys? The Impact of the Growing Cell-Only Population." Public Opinion Quarterly 71(5): 772-792.
2Stephen J. Blumberg and JulianV. Luke. 2007. "Coverage Bias in Traditional Telephone Surveys of Low-Income and Young Adults." Public Opinion Quarterly 71(5): 734-749.