Since 2018, Pew Research Center’s primary U.S. public opinion source has been online surveys conducted on the nationally representative American Trends Panel. Previously, telephone surveys were the Center’s primary U.S. public opinion source.
This transition from telephone to online surveys was necessary for a variety of reasons, including declining cooperation rates by telephone and increasing costs associated with reaching people on their cellphones. But it created a new challenge of determining how well these different survey modes align when studying long-term trends in public opinion (these differences across mode are known as “mode effects”).
Directly comparing answers from online and telephone surveys is complex because there are differences in how questions are asked of respondents and in how respondents answer those questions. These changes affected Pew Research Center’s measurements of party identification, making it necessary to adjust telephone trends for leaned party identification to account for mode effects in order to allow for direct comparisons over time.
Measuring party identification across survey modes
On both telephone and online surveys, party identification is measured by asking two questions. The first measures straight identification with the Republican or Democratic Party, while the second asks those who do not identify with one of the parties if they “lean” toward a party.
Telephone questions
The questions used on the telephone to measure the party identification offered three explicit responses: Republican, Democrat or Independent, but they allowed for a number of volunteered response options. These responses were not read by interviewers, but they could mark them as respondents’ answers if respondents said they did not identify with or lean toward either party.
ASK ALL:
PARTY In politics TODAY, do you consider yourself a Republican, Democrat, or independent?
1 Republican
2 Democrat
3 Independent
4 No preference (VOL.)
5 Other party (VOL.)
9 Don't know/Refused (VOL.)
ASK IF INDEP/NO PREF/OTHER/DK/REF (PARTY=3,4,5,9):
PARTYLN As of today do you lean more to the Republican Party or more to the Democratic Party?
1 Republican
2 Democrat
9 Other/Don't know/Refused (VOL.)Online questions
The transition to online surveys meant adjusting the questions. Because online surveys are not administered by an interviewer (that is, respondents see the question and answer the question without having to interact with an interviewer), it is not possible to replicate the “volunteered” response option that is available when there is an interviewer involved. Researchers had to decide whether or not to include explicit options for the previously volunteered response options.
After testing different approaches, a single additional explicit response option (“something else”) was added to the first question, but the second question only offers the original explicit options. This is because respondents tend to select explicit responses at far higher rates than they would have volunteered a similar response to an interviewer on the telephone.
FIRST SCREEN:
ASK ALL:
PARTY In politics today, do you consider yourself a…
1 Republican
2 Democrat
3 Independent
4 Something else
SECOND SCREEN:
ASK IF INDEP/SOMETHING ELSE (PARTY=3, 4 OR REFUSED):
PARTYLN As of today do you lean more to…
1 The Republican Party
2 The Democratic Party
Adjusting telephone survey trends
As a result of these differences in administering phone and online surveys, partisan identification measurements differ between the two modes. One way to think about this is that if the same set of respondents was given an otherwise identical survey on the phone and online, the results would differ solely as a result of the way in which the survey was conducted. The biggest difference between the two modes is that there are fewer independents who do not lean toward a party in the online survey data than there are in the phone data.
To account for mode effects between the phone and web modes, a universal adjustment was applied to the party identification measures from Pew Research Center telephone surveys conducted between 1994 and 2018. This adjustment makes data from the two modes directly comparable.
Researchers determined the size of the adjustment using a statistical analysis of data from an experiment that directly tested the impact of telephone vs. web modes on responses to the party identification questions. Here are the details of this experiment and the statistical analysis:
In 2014, Pew Research Center conducted an experiment to assess the impact of answering a survey online versus by telephone on commonly measured public attitudes, including party identification. This survey was conducted July 7-Aug. 4, 2014, among 3,351 respondents on the Center’s American Trends Panel. The survey randomly assigned half of its respondents to take the survey online, resulting in 1,509 online-mode completed interviews. The other half took the survey via a telephone interview, resulting in 1,494 phone-mode completed interviews. (The remaining completed interviews were completed by mail among respondents without web access and are excluded from this analysis.)
All respondents in the experimental survey answered questions about their party identification, allowing us to estimate the impact the different modes had. We fit a set of logistic regression models to the data from 2014 to estimate the impact of survey mode on party identification. The dependent variables in the models are five levels of partisanship (with the categories: Republican, Lean Republican, No lean/Don’t know/No answer, Lean Democrat, and Democrat). The predictive variables in the model are survey mode (phone or online), gender, education, age category, race and ethnicity, and a measure of partisanship from a previous telephone survey that all respondents completed.3
The regressions return an adjustment parameter (the coefficient on the “mode” variable in the regression) that is applied to all phone estimates of party identification. The shares of leaned party identifiers for all phone surveys is converted to the logit scale (z = log(p/(1 – p))), the coefficient from the regression is added to that share and the share is converted back to a probability (p = 1/(1 + exp(-z))).
The adjustment method uses a single set of parameters to adjust all telephone surveys from 1994 to 2018. This requires some key assumptions: First, the strong assumption that the mode effect is constant over time (which is unavoidable because we do not have data back to the 1990s that allows for comparing these survey modes). And second, this approach assumes that the mode effect is constant for all subgroups (a second adjustment strategy relaxed this assumption and produced nearly identical adjustments; details are below).
As a result of these adjustments, estimates of leaned party identification in previously published Pew Research Center reports differ from what is reported here. There was nothing inaccurate about the previous reports; they accurately discussed trends in party identification as measured using telephone surveys. Previously reported leaned party identification trends and the new adjusted trends tell essentially the same stories about how American voters’ partisanship has changed over the past 30 years.
The biggest difference between the two modes is there are fewer independents who do not lean toward a party in the online survey data than there are in the phone data, thus the primary impact of the adjustment is increasing the shares of leaned partisanship for both parties. On average, the shares of Republicans and Republican leaners are about 3.8 percentage points higher in the adjusted data, while the share of Democrats and Democratic leaners are about 3.1 points higher in the adjusted data. For example, in 1994, the phone survey data (unadjusted) had leaned partisan identification at 46% Republican/lean Republican, 44% Democrat/lean Democratic. After adjustment, the 1994 data is 51% Republican/lean Republican, 47% Democrat/lean Democratic. The relative balance of partisan identification and the overall patterns of partisan identification are the same in both approaches.
Please visit the full report from the 2014 mode experiment for more on mode effects between telephone and online surveys.
Mode-adjustment validation
We validated the regression-based mode adjustments with a second approach known as multiple imputation. The two approaches produce nearly identical results.
The multiple-imputation approach relied on the fact that prior to 2019, the American Trends Panel recruitment surveys were conducted via telephone. Any panelists recruited during that time were first asked their party identification over the phone in their initial recruitment survey and then asked again online when they took their first survey on the ATP. For individuals who were only ever asked about their party identification over the phone, this method filled in likely values for what each person would have said if they had also been asked the question online. This was done using a statistical model based on those individuals for which both a phone and online measurement were available. This process was then repeated multiple times on different random subsamples of the data in order to account for statistical uncertainty in the statistical modeling.
