As with any research project, we could not test every methodological question we had about measuring news consumption on surveys. For the sake of transparency, and in hopes that it may help other researchers studying this subject, this appendix provides additional details on decision points that we did not opt to test formally using a representative national sample. In some cases, this is because the implications were already clear from non-representative testing (using SurveyMonkey’s Audience survey panel). In other cases, it was the result of data coming out of the cognitive interviews or previous work – and some were purely the result of discussions with experts around the Center. In addition, some of these decision points are specific to the way Pew Research Center has measured news consumption over the past few years and so may not be as broadly applicable. Nevertheless, they are presented here as a potential resource to others.
Top-of-mind elicitation on news use
We wanted to take a breaking news event – in this case, Robert Mueller’s testimony before Congress on July 24, 2019 – and ask, immediately afterward, how people had heard about it. This would, in theory, be a lower cognitive burden than trying to ask a question about how respondents get news more generally, or about getting news from different types of organizations, since it would be tied to a specific event in the past day. As such, on July 25, we asked SurveyMonkey respondents who’d heard about the testimony, in an open-ended question, how they first heard about it. Some said a specific news organization, some provided a format (“TV”), while others provided a verb such as “watching” or “listening.” Still others gave some version of “word of mouth.” This suggests that respondents have different top-of-mind responses when asked about news consumption and that asking exclusively about platforms or providers may not capture the full scope of how they get news.
Defining ‘news’
There are many different definitions of what is meant by “news”: Just breaking news? Only political news? Anything produced by the news media? Or just any information about the world beyond our immediate circle? We wanted to test whether different definitions of news would produce different estimates of news consumption. Using five different SurveyMonkey custom audiences of approximately 500 respondents each, we tested the following conditions:
- Control: No definition
- A broad definition: “information about events and issues that involve more than just your friends and family”
- A definition that primed different topics of news: “any kind of news, including sports, traffic, weather, business, politics, health, or any other topic”
- A story that primed different “levels” of news: “local, national or international news”
- A definition that primed “hard news” considerations: “stories about major events in the U.S. and around the world from journalists or news organizations”
Respondents were then asked about their overall news use and their use of different platforms. There was no systematic variation among the definitions; that is, there was no definition that produced consistently higher or lower results relative to the control condition, where there was no definition offered.
Longer response scales
Our main response scale for the news consumption items is a four-point scale: often, sometimes, rarely or never. We wondered if this may be artificially inflating the portion who end up at the top of the scale and so tested it on SurveyMonkey with a six-point scale instead (daily, several times a week, weekly, several times a month, once a month, less often). This did not notably reduce the portion giving the highest response.
Other tests and refinements
- When asking about individual news organizations, we will sometimes show respondents icons of each brand to aid in comprehension. We experimented with using icons to represent different types of news consumption, but this proved impractical for several categories (e.g., there are no obvious abstract icons that could differentiate among local TV news, network TV news and cable TV news).
- Following the way we asked about news consumption in our local news study, we decided early on to have separate batteries for platforms (the ways people get news) and providers (the types of organizations producing that news).
- We added examples to “network TV” and “cable TV” to help alleviate the confusion that cognitive interview respondents expressed about not knowing the difference between the two.
- We chose not to ask about digital-native news organizations in the providers battery. In the cognitive interviews, when respondents were asked to name specific digital-native news organizations they get news from, they frequently gave examples of TV or newspaper providers (such as latimes.com or cnn.com), suggesting strongly that survey respondents have difficulty separating online news organizations from legacy ones.
- In addition, we made some structural changes to our existing measures. This includes asking about individual online platforms for news (such as social media and podcasts) as a follow-up to our main platform battery; adding search, podcasts and email newsletters into this digital follow-up; and adding talk radio and public radio to the providers battery we launched in the local news study.
