As polling takes center stage again this election year, understanding its strengths and weaknesses is critical. At Pew Research Center, we’ve looked hard at the advantages and limitations of our methods. We’ve also contributed to the survey profession’s understanding of data quality problems in polls that use online “opt-in” sampling – an approach where people are not selected randomly but are instead recruited from a variety of online sources like banner ads or social media.
In this analysis, we’ll show how some recent findings from an opt-in poll may have unintentionally misled the public about the sensitive issue of Holocaust denial among young Americans.
Pew Research Center conducted this analysis to see if findings related to Holocaust denial among young Americans – drawn from a December 2023 online opt-in survey – would be repeated when data was collected using more rigorous, probability-based methods.
To find out, we repeated five questions from the December poll on a survey of 5,140 U.S. adults that we conducted Jan. 16-21, 2024. The questions were asked with the same wording and format as in the original survey to ensure that any differences in the results could not be attributed to differences in the wording.
Everyone who completed the January survey is a member of the Center’s American Trends Panel (ATP), an online survey panel that is recruited through national, random sampling of residential addresses. This way nearly all U.S. adults have a chance of selection. The survey is weighted to be representative of the U.S. adult population by gender, race, ethnicity, partisan affiliation, education and other categories. Read more about the ATP’s methodology.
Here are the questions we used for this survey experiment. For other details about the Pew Research Center survey, read our survey’s methodology.
Crosstabs and methodological details for the December opt-in poll can be found here.
Opt-in polls can struggle to measure the attitudes of young people, Hispanics
Online opt-in polls have become increasingly popular. And for some purposes, such as election polling, they can perform similarly to more traditional survey approaches.
There is evidence, however, that the online environment in which they operate is somewhat unstable.
In particular, several recent studies have documented large errors in online opt-in surveys due to the presence of so-called “bogus respondents.” These respondents do not answer questions sincerely; instead, they attempt to complete surveys with as little effort as possible to earn money or other rewards.
Studies have shown that bogus respondents can cause opt-in surveys to overestimate rare attitudes and behaviors, such as ingesting bleach to protect against COVID-19, belief in conspiracies like Pizzagate or support for political violence.
At Pew Research Center, we’ve found that this type of overreporting tends to be especially concentrated in estimates for adults under 30, as well as Hispanic adults. Bogus respondents may be identifying this way in order to bypass screening questions that might otherwise prevent them from receiving a reward, though the precise reasons are difficult to pin down. Whatever the underlying cause, the result can be unreliable estimates for those groups.
For example, in a February 2022 survey experiment, we asked opt-in respondents if they were licensed to operate a class SSGN (nuclear) submarine. In the opt-in survey, 12% of adults under 30 claimed this qualification, significantly higher than the share among older respondents. In reality, the share of Americans with this type of submarine license rounds to 0%.
The problem was even worse for Hispanic estimates. About a quarter (24%) of opt-in cases claiming to be Hispanic said they were licensed to operate a nuclear sub, versus 2% of non-Hispanics.
Another Pew Research Center study comparing three online opt-in samples and three probability-based panels had similar findings, though on different topics. We asked respondents 16 yes/no questions on topics ranging from smoking and hypertension to collecting Social Security and receiving workers’ compensation. Young and/or Hispanic opt-in respondents were significantly more likely than other respondents to answer “yes” to 10 or more of these questions, claiming combinations of characteristics that are virtually nonexistent in reality. Critically, on the more rigorous probability-based surveys, very few respondents in any age group gave the same kind of implausible answers to these questions.
Recent opt-in poll made headlines on Holocaust denial among young adults
In the context of what we’ve learned from our investigations into opt-in polls, we took particular notice of a recent online opt-in survey that had a startling finding about Holocaust denial among young Americans. The survey, fielded in December 2023, reported that 20% of U.S. adults under 30 agree with the statement, “The Holocaust is a myth.” This alarming finding received widespread attention from the news media and on social networks.
From a survey science perspective, the finding deserved a closer look. It raised both of the red flags in the research literature about bogus respondents: It focused on a rare attitude (Holocaust denial), and it involved a subgroup frequently “infiltrated” by bogus respondents (young adults).
Other questions asked in that December opt-in poll also pointed to a need for scrutiny. In the same poll, about half of adults under 30 (48%) expressed opposition to legal abortion. This result is dramatically at odds with rigorous polling from multiple survey organizations that consistently finds the rate of opposition among young adults to be much lower.
In an April 2023 Pew Research Center survey, for instance, 26% of U.S. adults under 30 said abortion should be illegal in all or most cases. This was 13 points lower than the share among older Americans (39%). Our estimate for young adults was similar to ones from other, more recent probability-based surveys, such as an AP-NORC survey from June 2023 (27%) and a KFF survey from November 2023 (28%).
Opt-in findings on Holocaust denial, abortion do not replicate
We attempted to replicate the opt-in poll’s findings in our own survey, fielded in mid-January 2024 on Pew Research Center’s American Trends Panel.
Unlike the December opt-in survey, our survey panel is recruited by mail – rather than online – using probability-based sampling. And in fact, our findings were quite different.
Rather than 20%, we found that 3% of adults under 30 agree with the statement “The Holocaust is a myth.” (This percentage is the same for every other age group as well.) Had this been the original result, it is unlikely that it would have generated the same kind of media attention on one of the most sensitive possible topics.
Likewise, our survey found substantial differences from the December poll on support for legal abortion. In the opt-in survey, roughly half of young adults (48%) said abortion should always be illegal or should only be legal in special circumstances, such as when the life of the mother is in danger. In our survey, 23% said so.
These differences in estimates for young adults are what we would expect to see – based on past studies – if there were a large number of bogus respondents in the opt-in poll claiming to be under the age of 30. These respondents likely were not answering the questions based on their true opinions.
How do these findings relate to attitudes about the Middle East?
Measuring antisemitism and Holocaust denial is an important, though sensitive, issue. For the purposes of this methodological analysis, we felt it was key to replicate the exact wording of the opt-in survey that drove headlines last year, but we would not have asked the question this way if designing the work ourselves.
First, questions structured with “agree/disagree” options have a long history of what’s called “acquiescence bias,” or the human tendency to give a positive answer if they have no opinion or feel confused by or uncertain about a question. Secondly, using a phrase like “is a myth” is potentially ambiguous and unclear to many respondents. Thirdly, the issue of Holocaust denial is so extremely sensitive that there are reasons respondents might take offense or offer expressive answers that don’t reflect their actual opinions.
While issues like this are difficult to address, Pew Research Center has done extensive work on Holocaust awareness that can be found on our website.
The takeaway from our recent survey experiment is not that Holocaust denial in the United States is nonexistent or that younger and older Americans all have the same opinions when it comes to antisemitism or the Middle East. For example, our survey experiment found that young adults in the U.S. are less likely than older ones to say the state of Israel has the right to exist. This is broadly consistent with other rigorous polling showing that young people are somewhat less supportive of Israel – and more supportive of Palestinians – than older Americans.
Rather, the takeaway is that reporting on complex and sensitive matters such as these requires the use of rigorous survey methods to avoid inadvertently misleading the public, particularly when studying the attitudes of young people.
Note: Here are the questions we used for this survey experiment. For other details about the Pew Research Center survey, read our survey’s methodology.
