Over the past five years, Pew Research Center has conducted surveys in the United States using our American Trends Panel, a randomly selected, probability-based sample of U.S. adults ages 18 and older. We recruit members of the panel offline, but once recruited, participants fill out our survey questionnaires online. (Those who don’t have internet access can do so on internet-enabled tablets we provide to them.)
The initial American Trends Panel was recruited in 2014 and had a little over 5,300 participants. After additional recruitments in 2015, 2017 and most recently in 2018, the panel has grown to more than 13,500 participants. It is now the Center’s primary vehicle for conducting surveys in the U.S., with some important advantages over traditional telephone surveys, as a new report explains in detail.
In this Q&A, Nick Bertoni, the manager of the American Trends Panel, shares more about how the panel works and, more broadly, what its recent expansion means for Pew Research Center’s U.S. survey work.
How does the American Trends Panel differ from other kinds of polling?
The panel comprises an online probability sample, which means that participants take surveys online and that panel members are randomly chosen using publicly available lists of all U.S. phone numbers or mailing addresses. Essentially, all adults in the U.S. have a known chance of being selected for our panel. That’s different from online nonprobability surveys – sometimes called online opt-in samples – which are not probability-based because members are not randomly chosen from a list that covers the entire population.
Online probability samples like the American Trends Panel have become more common in the survey research world in recent years, particularly as response rates to traditional telephone surveys have declined. Our latest research shows that telephone response rates at Pew Research Center have fallen to just 6%, which means that it’s become more difficult – and more expensive – to carry out traditional, random-digit-dial phone surveys.
But the decline in telephone response rates isn’t the only reason for using an online probability sample. Using the American Trends Panel also lets us survey the same Americans more than once, rather than calling different people for each survey. That allows us to do certain kinds of longitudinal research – that is, study how the attitudes of the same survey respondents change over time. Last year, for example, we examined how views of Donald Trump had changed (or, in most cases, had not changed) among those who voted for him.
Another advantage of the panel is that people sometimes give more honest answers to certain questions – especially sensitive questions on subjects like the respondent’s personal finances – when they’re answering them in self-administered, online surveys, as opposed to when they’re talking to another person over the phone. That’s what’s known as a “mode effect.”
Why did you decide to increase the size of the panel?
A larger panel means we can produce more precise estimates of the views and experiences of small groups in society because we have more of those kinds of people in the sample. Nick Bertoni
There were many reasons. We knew we wanted the panel to play a bigger role in the Center’s data collection and it needed to be bigger to do that well. Growing it not only allowed us to replace the people who drop out, but to add new people who haven’t had the experience of doing a lot of interviews. A larger panel also means we can produce more precise estimates of the views and experiences of small groups in society because we have more of those kinds of people in the sample.
Having a bigger panel also means we can do surveys more frequently without contacting the same people too often. Many of our surveys have typically been based on interviews with around 1,500-2,500 people, so having a panel with more than 13,000 people means we can do surveys of that size more often without the same respondents feeling bombarded with frequent requests to be interviewed.
How did you go about expanding the panel?
We’ve recruited new panelists a total of four times, including the first recruitment when we created the panel in 2014. The second was a phone recruitment effort in 2015 in which – as with the first recruitment – we invited Americans to join the panel at the end of a random-digit-dial telephone survey that dealt primarily with politics. We added about 3,000 new people in that recruitment.
The third recruitment took place in 2017, again over the phone. But this time, we invited respondents at the end of a survey that had little political content. That’s because we were concerned that again using a political survey might recruit proportionately too many people who are interested in politics. We added around 1,600 new participants during the third recruitment.
For the 2018 recruitment, we shifted gears and recruited panelists using mailed invitations for the first time. We used what’s known as address-based sampling, which just means drawing a random, representative sample of residential addresses included in a U.S. Postal Service database that covers about 97% of the U.S. population. Not only was the cost per panelist lower using this approach, the sign-up rate was much better: 94% of the people who completed our initial screening survey after receiving a mailed invitation joined the panel, compared with an average of about half of those who were recruited over the phone during our initial recruitment and two phone-based recruitments. This address-based recruitment brought more than 8,700 new people onto the panel.
Overall, from our initial creation of the panel to our three subsequent recruitments – and with some panelists dropping out over time, as you might expect – we now have more than 13,500 active participants.
Does the expansion of the American Trends Panel mean that Pew Research Center is no longer conducting telephone surveys?
No. The Center still conducts random-digit-dial phone surveys and we continue to have confidence in the data yielded by these surveys, despite the long-term decline in response rates. Our own research has shown, in fact, that there’s little relationship between telephone response rates and the accuracy of these surveys on most of the topics we study.
That said, as the public’s communication preferences and habits evolve, there are real challenges associated with telephone survey work – most notably, the twin challenges of declining response rates and rising costs. It takes a lot more effort to get a completed interview by phone than it did just a few years ago. That makes it logical for the Center to rely more heavily on the American Trends Panel, particularly as we’ve ramped it up in recent years and added so many more participants.
Related:
Main report: Growing and Improving Pew Research Center’s American Trends Panel
What our transition to online polling means for decades of phone survey trends
Response rates in telephone surveys have resumed their decline
