This report examined the use of sources and social media in online news coverage of Pope Francis’ visit to the United States. The study focused on coverage from 9 a.m. Eastern Standard Time on Sept. 22, 2015, (the day the pope arrived in the United States), until 9 a.m. on Sept. 26, 2015 (the day he left New York City), a time period that allowed researchers to study the majority of the visit coverage. Coding and data analysis were conducted by researchers Mike Barthel, Jeffrey Gottfried, Katerina Eva Matsa, Kristine Lu, Elisa Shearer, Galen Stocking and Nancy Vogt.
Sample Design
Twelve news websites were studied. As of August 2015, these sites are all included in the top 25 entities of comScore’s news and information category. Researchers pulled the initial list of the top 25 digital properties and then removed those that are entirely devoted to weather (for example, weather.com), those which do not regularly produce original reporting around daily news (for example, about.com), and those that are non-U.S. (such as BBC and the Guardian). For listings that appear at a company level but have several distinct news entities such as Gannett, we “unrolled” the data to traffic levels of each individual news site (i.e. USA Today). From those, the 12 sites with the highest number of unique visitors for the month of August were selected for this study, ranging from newspapers to online-only outlets to broadcast TV.
The following are the news outlets included in this analysis:
Yahoo News ABC News The Huffington Post CNN USA Today NBC News Buzzfeed CBS News The New York Times Fox News The Washington Post The Los Angeles Times
Note: The Washington Post, with both national and local elements in their coverage, produced substantially more stories than other outlets: 148 in total. The Huffington Post also produced more stories than most other outlets: 74 in total. The other outlets produced an average of 29 stories.
Story Selection
The unit of analysis in this study was the story. Stories were included in the study only if 50% or more of their content was about the pope’s visit to the United States (a small number of stories also addressed the pope’s visit to Cuba, which preceded his arrival to the United States). To find the relevant stories in all websites, researchers used a specific set of keyword searches to collect potential stories.
Researchers searched Google News for each news outlet’s main domain with the following terms to find potential stories:
Pope OR Francis OR Bergoglio OR “Holy Father” OR Papal
Every effort was made to collect the entirety of the content from each provider in its primary form.
Researchers manually captured and saved digital versions of the websites each morning between 9 a.m. and 12 p.m. EST. In some cases where sites produced a large number of stories, content capture for a given day occurred over a longer period of time. Duplicate stories were removed, as well as stories that were not in English. Also excluded were stories that appeared in a different domain or URL. For example, researchers coded all related stories found in foxnews.com, but did not code those that appeared under foxbusiness.com.
Researchers then read each story to see if 50% or more of the story’s content was about the pope’s visit. The study did not include articles that were entirely standalone video, audio, slideshow, straight transcripts or quizzes that did not offer any content by the author of the piece. (Video, audio, transcript or slideshows embedded in a story were included in the study.)
Human Coding of Stories
The data in this study were created by a team of six experienced coders who were trained specifically for this project.
The two central variables in this study were source type and social media reference. Source type refers to a person, group, organization or publication cited in a story. Citations included direct or indirect quotes, interviews or labels, or references accompanying factual information. The goal was to identify the different types of sources that were present in the stories and not to provide a count of the total number of sources. With this in mind, researchers coded for the presence of 17 different source types. Within each story, a source type was coded only once, even if it was cited repeatedly. A source type could be present both as a social media reference and a non-social media reference and therefore both cases were captured as one source type from two different platforms. The 17 different source types were:
- The Vatican/Pope Francis
- Clergy/Religious group (non-Vatican)
- Issue-based and/or interest groups (non-religion)
- Quotes from religious texts
- Experts, non-clergy
- Polls
- State/local official
- Obama/Obama administration
- Presidential candidate
- Congress as institution or member of the Congress (D & R)
- Political & government figure not in office (GOP, Democratic or other partisan)
- Member of the public
- Other media (includes social media companies, as distinguished from users)
- Celebrity
- International political & government figures
- Business
- Other
The social media reference variable indicates whether the source was taken from a social networking site. The social networking sites tracked in this study were: Twitter, Facebook, Instagram and YouTube, with all others coded as “other social networking site.”
Coders were given two sets of news stories to evaluate during the training period. Once internal agreement on how to code the variables was established, a first round of five stories were coded by all members of the coding team. The percent of agreement on each variable was:
Presence of sources cited — 89% or higher
Social Networking Site — 92% or higher
Finally, a second round of 25 stories were coded at the end of the study by all members of the coding team. The percent of agreement on each variable was:
Presence of sources cited — 82% or higher
Social Networking Site — 96% or higher
Throughout the coding process, staff discussed questions as they arose and arrived at decisions under supervision of the content analysis team leader.
