By Courtney Kennedy, Scott Keeter, Andrew Mercer, Nick Hatley, Nick Bertoni and Arnold Lau
Five months after Donald Trump’s victory in the 2016 election, public opinion polls find him with the lowest approval ratings of any president in modern history at this stage of a presidency. Given the criticism of election polling in 2016, it is reasonable to ask if public opinion polls are understating support for Trump and his policies.
Survey watchers have posited several theories for why election polls, and by extension post-election surveys, might underestimate backing for Trump. This report tackles one of the most common of these: the supposition that some people may be reluctant to tell an interviewer that they approve of Trump’s job performance or policy proposals due to the fact that he remains a polarizing figure. If this is true, telephone surveys that use live interviewers would understate public support for the president’s agenda. But there is another way to poll Americans: via the inherently more private method of web surveys. Self-administered online surveys should be less affected by any reluctance to reveal actual preferences.1
To examine the impact of interview mode on expressions of opinion about policies and political leaders, Pew Research Center conducted a large-scale survey experiment that compared responses to 27 questions fielded on both a telephone and a web survey. Of these questions, 13 addressed attitudes about Trump or policies closely associated with him; four measured impressions of other political figures (Vice President Mike Pence, former President Barack Obama, Sen. Elizabeth Warren and Russian President Vladimir Putin); and 10 tapped core political values. For this analysis, the pro-Trump or conservative response option of the question was used to compare estimates from the web versus phone groups.2 The findings would be virtually unchanged if the anti-Trump or liberal response option had been used.
Overall, the survey experiment did not find significant mode differences in overall opinion about Trump or many of his signature policy positions. The web mode yielded estimates that were 1.8 percentage points more supportive of Trump and his policy positions, on average, than estimates from the phone mode.3 The differences ranged from 0 to 8 percentage points. In total, four of the 27 differences observed were statistically significant using conventional testing for opinion polls.4 And not all differences were in the direction one might expect based on the theory that people conceal their support for Trump and his policies in telephone interviews.
The two items that showed the largest difference between web and phone both focused on the policy treatment of undocumented immigrants. On two questions about this issue, web respondents expressed more support for deportation of undocumented immigrants than those interviewed by telephone.
That said, the big picture about where the public stands on Trump’s signature policies is not dependent upon whether the poll was conducted online or by phone. Both modes tell the same story. Both find minority support for a wall along the entire Mexican border, minority support for the travel restrictions on certain majority-Muslim countries, minority support for a national law enforcement effort to deport immigrants living in the U.S. illegally and an essentially even split on attitudes toward the 2010 health care bill.
A look into which parts of the public were more or less affected by the mode shows that – on the questions examined – mode generally had a more pronounced effect on the responses from Republicans than on the responses from Democrats. It seems likely that this finding is specific to the policy and favorability questions asked in this study. There is no reason to conclude that mode effects are always going to be larger among people identifying with one particular political party.
This survey was conducted Feb. 28-March 12, 2017, using Pew Research Center’s nationally representative American Trends Panel (ATP). Panelists were randomly assigned to either the phone mode (N=1,778 completed by phone) or the web mode (N=2,066 completed on the web). Phone mode respondents were called by professional interviewing staff following Pew Research Center’s standard dialing protocol, in which up to seven attempts were made in order to complete the interview. The phone respondents and web respondents were independently weighted to be representative of the U.S. public. As a result, the web and phone groups were nearly identical with respect to partisanship, 2016 presidential vote and key demographic characteristics such as race, education, age, sex, urban-rural residence and region. This helps to ensure that any observed differences in estimates from the web versus phone groups are a consequence of the mode of interview itself and not because of underlying differences between the two sets of respondents.
No real mode differences in opinions about Donald Trump
The study finds no significant difference by mode of interview on any of four key questions about Trump. Approval of the president’s job performance differs by only 2 percentage points across the two survey modes (44% approve and 56% disapprove on the web; 42% approve and 55% disapprove on the phone), a difference that is not statistically significant. On the other three questions specifically about Trump, the mode difference goes in the other direction, with the phone estimates being slightly more positive about Trump than the web estimates (though, again, differences are not statistically significant).
Overall, 39% of those interviewed by phone said Trump is trustworthy, while 37% of those interviewed on the web did so. And on the question of whether Trump is a strong leader, 49% of those interviewed by phone and 45% of those interviewed online said that he is. These results are not consistent with the narrative that a substantial share of the public simply does not want to tell interviewers that they support Trump. Instead, they align with a previous Pew Research Center finding that people tend to express more positive views of political figures when responding on the phone than on the web.
Few differences by mode on signature Trump policies, except for deportation
Questions about immigration policy and immigrants displayed the largest mode differences in the expected direction (more support for the Trump administration’s positions on the web), but not all such questions displayed mode of interview effects. While on both the web and phone a minority of adults say they favor a national law enforcement effort to deport all immigrants who are now living in the U.S. illegally, the share endorsing that position was significantly higher on the web (40%) than on the phone (32%). Similarly, while a minority of adults in both modes say that undocumented immigrants should not be allowed to stay in the country legally, there was greater support for that view when measured online (22%) than by phone (16%). Both of these differences are statistically significant.
At the same time, several other immigration and security-related items showed no appreciable differences between those who took the web survey and those who answered via phone. For example, there was only a 3-percentage-point mode difference in support for the proposed wall along the Mexican border and a similarly negligible gap on a question about whether Muslims living in the U.S. should be subject to additional anti-terrorism scrutiny.
One other issue that touches on immigration is the Trump administration’s executive order aimed at preventing people from a number of majority-Muslim countries from entering the U.S. The public overall is divided on this question, and there are large partisan divisions. But again, there is no difference in survey responses by mode of interview.
There were also no mode effects on Trump administration policy priorities on health care reform and trade agreements. Disapproval of the Affordable Care Act differed by only 1 percentage point between modes. And belief that free trade agreements have been a bad thing for the U.S. also had a 1-point difference by mode.
On the long-standing poll question about the way things are going in the country, the study found no difference between web and phone responses. Just under a third of adults responding online (32%) and by phone (31%) reported being “satisfied” with how things are going nationally.
Mode influences ratings of Obama and the intensity of responses
The study also examined whether web versus phone administration influences favorability ratings of prominent political figures. In most cases the answer was no. The public’s overall favorability ratings of Donald Trump, Mike Pence and Vladimir Putin were essentially the same when measured online or by phone. Ratings of Barack Obama, however, were significantly more positive on the phone than online (63% versus 57%, respectively, in overall favorability).
At first glance, the study also appears to show an effect from mode on ratings of Elizabeth Warren, whose favorability rating is 8 percentage points higher on the web than on the phone (45% versus 37%, respectively). A closer look, however, reveals that the result actually stems from a mode-based effect on the share providing a rating. Web respondents were presented with just four response options (ranging from very favorable to very unfavorable), and 8% of respondents proceeded to the next question without offering a rating. On the phone, respondents were read the same four answer choices but their responses were counted as “don’t know” if they indicated that they had never heard of Warren, could not rate her or refused to answer. As a result, 18% of phone respondents did not provide a rating – more than double the rate on the web. If the analysis is restricted to adults providing a rating of Warren, then the mode effect essentially disappears (49% favorable on the web versus 46% on the phone).
While a look at the net favorable ratings shows no effect from mode, there is a small but consistent effect when it comes to unfavorable ratings. Across all of the political figures, respondents were slightly more prone to expressing unfavorable views online than on the phone.
The battery of favorability questions also yielded another subtle pattern – adults responding online tended to express more intense ratings (both positive and negative) than those responding by phone. Averaging results across the five politicians, the share of adults expressing a “very unfavorable” opinion was 3 percentage points higher for the web group than the phone group (31% versus 28%, respectively). The study yielded the exact same result at the other end of the favorability spectrum. Across the five politicians, the average share expressing a “very favorable” opinion was 3 points higher for the web group than the phone group (17% versus 14%, respectively). Other mode-related work conducted by the Center found a similar result for very unfavorable ratings of six different political figures but no mode difference in very favorable ratings.
Mostly minor differences between web and phone results on political values questions
Pew Research Center has been measuring a set of important political values among Americans for the past three decades. This study measured 10 values that touch on views of government, the social safety net, regulation, the environment, immigration, business, race, homosexuality and the use of military force. These 10 values questions, which ask respondents to choose between alternative statements, were included in the mode experiment. (It’s important to note that respondents are asked to choose which alternative comes closer to their own views, “even if neither is exactly right.”)
Across the set of political values items, the average difference between the web and phone estimates on the conservative response option was 2.3 percentage points. Only one question showed a statistically significant difference when measured by web versus by telephone. Web respondents were 5 percentage points more likely than phone respondents to say that immigrants burden rather than strengthen the country.
A handful of other items showed smaller, non-statistically significant differences in the same direction. Those interviewed on the web were 5 points more likely than those interviewed on the phone to say that government is almost always wasteful and inefficient. And web respondents were 4 points more likely than phone respondents to say that the best way to ensure peace is through military strength.
The other political values items yielded virtually the same results when measured online or by phone. This included questions about environmental laws, corporate profits, government assistance to the poor and government regulation of businesses.
Mode had a more pronounced effect on responses from Republicans than responses from Democrats on questions in this study
For questions that break heavily along party lines (e.g., opinion of the 2010 health care law), it is possible that an overall finding of no discernable mode effect masks important differences within the political parties.
Indeed, the study finds that the impact of the mode of interview differed somewhat by party on a number of estimates, including approval of Trump’s job performance, personal impressions of Trump and Obama, opinion about the impact of immigrants on the country and opinion about the Affordable Care Act.
Among Republicans, including independents who lean Republican, Trump’s job approval was 7 percentage points higher on the web than the phone. By contrast, his approval was little different on the web (11%) than the phone (14%) among Democrats and Democratic-leaning independents.
With respect to overall favorability – that is, the share who report holding either a “very favorable” or “somewhat favorable” opinion of Trump – there is little difference by mode for either Democrats or Republicans. However, the intensity of favorability differs considerably. Republicans on the web were 8 points more likely than those on the phone to offer a “very favorable” opinion of Trump (37% versus 29%, respectively); there was only a minor mode difference among Democrats on this category. On the other end of the scale, Democrats on the web were 9 points more likely than those on the phone to say they have a “very unfavorable” opinion of Trump (68% versus 59%, respectively); Republicans differed little by mode of interview.
A similar mode difference by party – in the opposite direction – occurred on favorability of Obama. Democrats interviewed on the web were 9 points more likely than those on the phone to say they had a “very favorable” impression of the former president, while there was little difference by mode among Republicans (very few held very favorable views of Obama). Conversely, Republicans on the web were 14 points more apt than those on the phone to say they have a “very unfavorable” opinion of Obama, while Democrats on the web and phone differed little on this category.
While three questions about immigration and immigrants produced significantly more conservative overall results on the web than the phone, only one of them showed that the mode of interview clearly differed between Republicans and Democrats. Republicans interviewed on the web were more likely than those interviewed on the phone to agree with the statement that “immigrants today are a burden on our country because they take our jobs, housing and health care” (58% on the web, 46% on the phone); responses from Democrats did not differ by mode on this question.
A similar partisan pattern appeared on the question about the Affordable Care Act. Republicans on the web were 7 points more likely than those on the phone to say they disapprove of the law, while there is little mode difference in opinion among Democrats.
More generally, mode of interview appears to contribute to a slightly more polarized portrait of opinion about Democratic and Republican political figures, especially with respect to the expression of strong opinions. Mode effects on policy questions were more complicated. While the difference in opinion between Democrats and Republicans was larger among web respondents than phone respondents on a few questions, partisans were rarely pushed in opposite directions by the mode of interview.
Implications for polling
While no single study is definitive, these results strongly suggest that live interviewer telephone polls obtain results that are substantively equivalent to those in self-administered online polls on measures of opinion about Trump and his key policy positions – from the travel ban and health care to trade and the environment. This assumes that other factors, like sample source, are held constant. Policy toward undocumented immigrants is a notable exception, though even on this subject both telephone and online measurement produce similar conclusions about the broad contours of public opinion (that a minority of adults express conservative views on the issue).
While this study was not designed to address the question of whether telephone surveys in the 2016 general election or 2016 primaries understated Trump support because of respondents’ reluctance to admit support for him, it does provide relevant evidence. Across four questions about the president, there was no consistent pattern of difference in overall responses by mode of interview, though Republicans on the web were somewhat more positive toward the president and his policies on immigration. But the timing of the study – conducted more than one month after Trump took office – and the fact that it was not focused on presidential vote means that it only indirectly speaks to the possibility of a “Shy Trump” phenomenon last year.
