When three different polling organizations conducted surveys last weekend to gauge public reaction to the news about the Department of Justice’s subpoenas of reporters’ phone records, their findings were quite different – a case study in the challenges pollsters face in a breaking news environment when public attention and information is relatively limited.
While the Pew Research Center, CNN/ORC and Washington Post/ABC News pollsters all took a similar approach in asking whether people felt the Department of Justice was right or wrong to subpoena the Associated Press reporters’ phone records, there were multiple differences in the phrasing, structure and context of the questions that help to explain the different findings.
Comparing Three Questions
Let’s start with the Pew Research Center’s survey, which found more disapproval than approval of the DOJ’s actions by a slim 44%-36% margin. Earlier in the survey, Pew Research asked how closely people had followed the news story. Barely a third reported following it very closely (16%) or fairly closely (19%), while most (64%) said they had not followed too closely or at all.
Pew Research then went on to ask if respondents approved or disapproved of the Justice Department’s decision to subpoena the phone records of AP journalists as part of an investigation into the disclosure of classified information and found slightly more disapproving than approving.
By comparison, the Washington Post/ABC News survey described the situation as follows: The AP reported classified information about U.S. anti-terrorism efforts and prosecutors have obtained AP’s phone records through a court order. Do you think this action by federal prosecutors is or is not justified? The survey found substantially more saying the actions were justified, by a 52%-33% margin. The difference is likely attributable to three differences in the way the surveys described the situation. First, Pew Research referred to the decision to subpoena phone records while the WP/ABC survey said the records were obtained… through a court order, which may make the actions seem more legitimate by highlighting a court’s involvement. Second, what Pew Research described as classified information the WP/ABC poll described as classified information about U.S. anti-terrorism efforts; past research suggests the public is willing to accept broader government actions in the defense against terrorism. Third, Pew Research mentioned the Justice Department, while WP/ABC mentioned federal prosecutors. It is likely that the DOJ, as a more political institution associated with the administration, is viewed less favorably than are prosecutors. All three of these differences likely contribute to the remarkably different survey results.
A third CNN/ORC survey conducted Friday and Saturday (the other two surveys were in the field Thursday through Sunday) used different language as well. Their line of questioning mentions that it was classified information about U.S. anti-terrorism efforts, as did the WP/ABC question. It also mentioned that it was the Justice Department, as did Pew Research. But CNN/ORC differed from the other polls in saying the phone records were secretly collected, rather than just subpoenaed or obtained through a court order. All in all, this question found more saying the DOJ’s actions were unacceptable than acceptable by a 52% to 43% margin, similar to the balance of opinion in the Pew Research survey.
There are two other differences that stand out in the CNN/ORC survey. First, prior to asking about whether the actions were acceptable, respondents were first asked how important the issue was, with the vast majority saying it was very important (53%) or somewhat important (34%), and few saying not too (8%) or not at all (4%) important. In this context, far more respondents offered an opinion when then probed about the appropriateness of the DOJ’s actions – just 5% offered no opinion on the CNN/ORC survey, compared with 15% on the WP/ABC survey, and 20% on the Pew Research survey.
Moreover, the CNN/ORC survey is the one out of these three that found a substantial difference of opinion along partisan lines: Republicans were about half as likely as Democrats to say the DOJ’s actions were acceptable (30% vs. 57%). In the Pew Research Poll, there was only a slight partisan difference (36% of Republicans and 43% of Democrats approved of the DOJ’s actions), and there was virtually no difference in the WP/ABC poll (54% of Democrats and 51% of Republicans said the actions were justified).
A Later Entry
Late in the day on Tuesday, Fox News released yet a fourth survey question along the same lines, with the most lopsided balance of opinion of the questioning so far. It is possible that opinions shifted over time, and the Fox News survey began two days later and ended one day later than those conducted by Pew Research and WP/ABC. It’s also worth noting that Fox News surveys are restricted to registered voters rather than the general public, a narrower pool that is generally more Republican in its political leanings.
But it is also likely that the language and context of the question are substantial factors. Where the first three polls said the DOJ subpoenaed, obtained through a court order, or secretly collected the phone records, the Fox News question used the phrase secretly seized to describe the DOJ’s actions. The Fox News survey also pointed out that the records were extensive, included both work and personal phones, and that they did so without giving the news organization prior notice, as is customary. By two-to-one (60% vs. 31%) more said this action by the government went too far rather than was probably justified.
An additional factor that likely affected the responses to the Fox News question was the context in which it was presented. Immediately prior to asking about the legitimacy of the DOJ’s actions, Fox News asked: Does it feel like the federal government has gotten out of control and is threatening the basic civil liberties of Americans, or doesn’t it feel this way to you? There is a long track record in the survey industry of how framing a question in a particular context can affect responses, and there is little doubt that raising this broad criticism of government – which roughly two-thirds (68%) of respondents agreed with – prior to the DOJ question had some effect.
The Lesson?
In an ideal world, understanding the impact of question wording differences on surveys is straightforward – respondents are randomly assigned to groups, each of which is asked the same question in the same context with only the language of interest modified. But in the real world, it’s rarely so simple. In the above examples, each polling organization made good-faith efforts to describe the facts of the situation as accurately as possible, but the word choices and context make it impossible to identify one particular phrase or concept that tipped the public’s thinking.
The fact that the language and context made a difference in how respondents reacted to the questions is a challenge of polling in a low-information situation. In this case, the DOJ news story was quickly developing, some facts were still unclear (such as how the department obtained the phone records), and the subject matter was somewhat complex. As the Pew Research survey showed, most Americans paid little or no attention to the story through Sunday – far less than was paid to the Benghazi and IRS news coverage. The Pew Research survey found that those who paid at least a fair amount of attention took a far more negative view of the Justice Department’s actions. Sometimes the views of attentive respondents can be a leading indicator of the direction of public opinion if and when a story grows. But sometimes the pattern of responses is more partisan. In this case, it was the more attentive Republicans who expressed particular disapproval of the DOJ’s actions, which may reflect that they were simply more attuned to the political implications of the scandal for the Obama administration.
Overall, while wide differences across polls suggest that public reactions may be only loosely formed, there is important information to glean from situations like these. There is likely a core of Americans who support the DOJ’s actions irrespective of the survey language (the 31% who say justified even in the Fox News survey). And another core who concretely oppose it (the 33% who say not justified in the WP/ABC survey). And for the rest, there are clues as to the types of considerations that may affect their views if and when attention grows and information becomes more widespread.
