The Color of News: How Different Media Have Covered the General Election is based on the aggregated data collected as part of the News Coverage Index from September 8 through October 16, 2008. This timeframe begins the Monday following the conclusion of the Republican National Convention and runs through the day after the final presidential debate between John McCain and Barack Obama.
For an accurate account of the chronology of the general election, the period studied has been broken into six distinct phases. These phases have been dictated by the timing of campaign events. Therefore, some of these phases do not correspond with a typical calendar week.
The complete methodology of the PEJ’s News Coverage Index is available here.
In addition to the coding already conducted as part of PEJ’s weekly reports, researchers conducted secondary coding of many of the campaign-focused stories for tone. Details of that process are below.
PEJ’s News Coverage Index
Examining the news agenda of 48 different outlets in five media sectors, including newspapers, online, network TV, cable TV, and radio, the NCI is designed to provide news consumers, journalists and researchers with hard data about what stories and topics the media are covering, the trajectories of major stories and differences among news platforms.
Following a rotation system, PEJ monitors 48 different news outlets each week: 34 or 35 outlets each weekday as well as 7 newspapers each Sunday.
The list of outlets and rotation is as follows:
Newspapers (13 in all, Sun-Fri)
The New York Times every day
Coded two out of these four every day
The Washington Post
Los Angeles Times
USA Today
The Wall Street Journal
Coded two out of these four every day
Philadelphia Inquirer
Chicago Tribune
Arkansas Democrat-Gazette
San Francisco Chronicle
Coded 2 out of these 4 every day
New Hampshire Union-Leader
MetroWest Daily News
The Gazette (Colorado Springs)
Modesto Bee
Web sites (Five in all, Mon-Fri)
CNN.com
Yahoo News
MSNBC.com
Google News
AOL News
Network TV (Seven in all, Mon-Fri)
Morning shows
ABC – Good Morning America
CBS – Early Show
NBC – Today
Evening news
ABC – World News Tonight
CBS – CBS Evening News
NBC – NBC Nightly News
PBS – Newshour with Jim Lehrer (Rotated daily between the first 30 minutes and the second 30 minutes)
Cable TV (Fifteen in all, Mon-Fri)
Daytime (2:00 to 2:30 pm) coded 2 out of 3 every day
CNN
Fox News
MSNBC
Nighttime CNN – coded 3 out of the 4 every day
Situation Room (6 pm)
Lou Dobbs Tonight
CNN Election Center
Anderson Cooper 360
Nighttime Fox News – coded 3 out of the 4 every day
Special Report w/ Brit Hume
Fox Report w/ Shepard Smith
O’Reilly Factor
Hannity & Colmes
Nighttime MSNBC – coded 2 out of the 4 every day
Race for the White House
Hardball (7 pm)
Countdown w/ Keith Olbermann
Rachel Maddow
Radio (Six in all, Mon-Fri)
ABC Radio headlines at 9am and 5pm
CBS Radio headlines at 9am and 5pm
NPR Morning Edition every day (Rotated daily between the first 30 minutes of the first hour and first 30 minutes of the second hour)
Talk Radio
Rush Limbaugh every other day
One out of two additional conservatives each day
Sean Hannity
Michael Savage
One out of two liberals each day
Ed Schultz
Randi Rhodes
From that content, PEJ analyzes all stories with a national or international focus that appear as follows:
- On the front page of newspapers
- In the entirety of commercial network evening newscasts and syndicated radio headlines
- During the first 30 minutes of network morning news, all cable programs, and talk radio programs
- During a thirty minute segment (rotated daily) of the PBS evening news and NPR’s Morning Edition
- As one of the top 5 stories on each Web site at the time of capture
Capture and Retrieval
All outlets included in the weekly index are captured and included in PEJ’s media archive.
For newspapers that are available in print in the Washington, D.C. area, we have hard copies delivered to our office each day. For newspapers that are not available for delivery, digital editions of the paper are retrieved either through the newspaper’s own Web site, or through the use of digital delivery services such as pressdisplay.com and newsstand.com. When necessary, the text of articles are supplemented by the archives available in the LexisNexis computer database.
Radio programs are captured through online streams of the shows. Using automated software, we record several local affiliates that air the program in various markets throughout the country. The purpose of this method is to ensure that we have a version of the program in case one of the streams is unavailable on a particular day, and so that we record the show in a manner that represents the way a typical listener would hear the program with commercials and newsbreaks.
Online websites are captured manually by a member of PEJ’s staff. The capture time is rotated daily between 9 am ET and 4 pm ET. The home pages and pages with the top articles for all five sites are saved so that when we reference the material, the format is the same as it appeared online at the time of capture.
Finally, all television shows are recorded digitally and archived for coding purposes. PEJ is a subscriber to DirectTV satellite service and all programs are recorded onto multiple TiVo recording units before being burned onto DVDs for archival purposes.
All television and radio programs are then coded by a member of PEJ’s staff who watches or listens to the archived version of the program.
Coding Team & Process for Weekly Index Coding
The data in this study derived from PEJ’s regular Index coding was created by PEJ’s team of 14 trained coders. We have tested all of the variables derived from the regular weekly Index coding and all the variables reached a level of agreement of 80% or higher. For specific information about those tests, see the methodology section for the NCI.
A majority of the codes and variables used in this study come out of the coding protocol created for the weekly Campaign Coverage Index which PEJ has been issuing throughout 2008. For the variables of frame/campaign topic, significant presence, and lead newsmaker, the data came from all campaign stories that appeared in PEJ’s weekly coding. The specific description of those variables can be found here.
Additional Coding of Campaign Stories for Tone
Between September 8 and October 16, the PEJ’s overall Index included 2,412 campaign stories. To measure the tone of the campaign coverage of the major presidential and vice presidential candidates, PEJ’s researchers conducted additional coding on a sample of the campaign stories that appeared in our weekly indexes.
Sample Selection
Stories in the NCI are considered to be about the presidential election if 50% or more of the story was devoted to discussion of the on-going presidential campaign. To analyze stories about tone about a given candidate, only stories in which at least one of the four major presidential and vice presidential candidates (John McCain, Barack Obama, Sarah Palin, and Joe Biden) were a lead newsmaker were included. For a candidate to be considered a lead newsmaker, they must be featured in at least 50% of the time or space that makes up that story.
Two categories of stories were excluded from the sample. Talk radio stories, which are part of PEJ’s regular NCI, were not included in this campaign study of tone. Broadcast stories that were 30 seconds or less were also excluded.
PEJ conducted further sampling on the selected stories. This was done by arranging the stories from each outlet in chronological order and randomly selecting a first story. We then selected every-other story within each outlet to arrive at the final sample.
This process resulted in 857 total stories from 43 different media outlets. These included 72 newspaper stories, 78 stories from news websites, 213 stories from network TV, 448 from cable TV, and 46 from radio programs.
Coding Design
The specific analysis of tone was conducted on the subset of campaign stories that was described above. The method of measuring tone was the same that had been used in previous PEJ studies.
Unit of Analysis
The unit of analysis for this study was the story. Each story was coded for tone for each of the four politicians followed in the study. If a candidate did not appear in at least 25% of the story, they were not considered a significant figure in the story and where therefore coded as “n/a” for not having a significant presence.
Coders then determined the tone of the story as a whole for each candidate who was a significant presence.
Tone Variable
The tone variable measures whether a story’s tone is constructed in a way, via use of quotes, assertions, or innuendo, which results in positive, neutral, or negative coverage for the primary figure as it relates to the topic of the story. While reading or listening to a story, coders tallied up all the comments that have either a negative or positive tone to the reporting. Direct and indirect quotes were counted along with assertions made by journalists themselves.
In order for a story to be coded as either “positive” or “negative,” it must have either 1.5 times the amount of positive comments to negative comments, or 1.5 times the amount of negative comments to positive comments (with an exception for 2 to 3, which is coded as “neutral”). If the headline or lead has a positive or negative tone, it was counted twice into the total value. Also counted twice for tone were the first three paragraphs or first four sentences, whichever came first.
Any story where the ratio of positive to negative comments was less than 1.5 to 1 was considered a “neutral” story.
In some previous studies, PEJ used a ratio of 2 to 1 instead of 1.5 to 1 in determining the overall tone of news stories.
The 2:1 ratio makes sets the bar even higher for a story to be coded as either positive or negative overall. As we entered the 2008 election campaign, PEJ reviewed and retested both the 2:1 ratio and the 1:5 to 1 ratio. We also consulted with academics of content analysis. First, we found only minor shifts in the overall outcome of stories. Indeed, in past content studies where we coded using both ratios, the overall relationship of positive to negative stories changed very little. The bigger difference was in an increase in mixed or neutral stories. In our pre-tests in 2007, the Project felt that the 1.5 to 1 ratio more precisely represented the overall tone of the stories. The academics consulted concurred.
Still, in making comparisons to previous years, it is important to note the different measures used. The 1.5 to 1 ratio was used in our previous 2007 study about the Invisible Primary season. The 2 to 1 ratio was used in PEJ’s 2004 report called The Debate Effect and the 2001 report on coverage of George W. Bush’s first 100 days in office.
Coding Team & Process for Specific Campaign-related Tone Coding
A team of five of PEJ’s experienced coders worked with a coding administrator in order to complete the specific tone coding for the campaign stories. Of the five coders, all but one had previously coded for tone in a previous PEJ campaign study.
The previous study that PEJ conducted in October 2007 using the same process for determining tone had a rate of agreement for intercoder reliability of 86%.
For this study, each of the five coders were trained (or re-trained) on the tone coding methodology and then were given the same set of 40 stories to code for tone for each of the four candidates. The rate of intercoder reliability agreement was 81%.
