The answer to almost any survey question depends on who you ask.
At Pew Research Center, we conduct surveys in the United States and dozens of other countries on topics ranging from politics and religion to science and technology. Given the wide range of people we speak to for our polls – and the many issues we ask them about – it’s important to be as clear as possible in our writing about exactly who says what.
In research circles, this practice is sometimes called “defining the universe” – that is, clearly identifying the population whose attitudes we’re studying, whether those people are police officers in the U.S., Christians in Western Europe or some other specific group. This kind of clarification can go a long way toward ensuring that readers interpret survey results correctly.
In many cases, the universe of survey respondents we’re talking about is relatively straightforward. If we say that nearly two-thirds of U.S. adults think the national economic situation is good, we’re referring to the views of Americans ages 18 and older. But in other cases, additional clarification and context may be needed, particularly for readers who aren’t schooled in the art of interpreting poll results.
Consider one of our recent survey findings about Facebook users. The survey found that 74% of adult Facebook users in the U.S. have taken at least one of the following three steps in the past 12 months: adjusted their privacy settings, taken a break from checking the platform for several weeks or more, or deleted the app from their phone. The survey drew a lot of attention because it suggested – correctly – that adult Facebook users in the U.S. have a complex relationship with one of the major social platforms in their lives.
But there are some caveats to bear in mind when considering this finding. First, not all U.S. adults use Facebook. Pew Research Center’s most recent estimate is that 68% of American adults use the platform. In other words, it’s not accurate to say that 74% of all U.S. adults have adjusted their Facebook privacy settings, taken a break from the platform or deleted the app from their phone in the past 12 months. It is accurate to say that 74% of U.S. adults who currently use Facebook – that is, 74% of that 68% – have taken one of these three steps in the past year.
It’s also crucial to remember that this finding only refers to adult Facebook users in one country. Worldwide, Facebook claims more than 2 billion monthly active users, and our survey did not include respondents from countries outside the U.S. It also doesn’t reflect U.S. Facebook users who are younger than 18 – a substantial population. How substantial? We know from a separate Pew Research Center survey earlier this year that 51% of U.S. teens ages 13 to 17 use Facebook. And there may be many other U.S. Facebook users who are younger than 13.
Another example of a survey universe that’s not always easily understood concerns U.S. politics. Pew Research Center and other polling organizations frequently conduct polls to find out Americans’ views on political topics. But these views can vary depending on whether the universe of interest is all U.S. adults, registered voters, or – in some cases – an even more narrowly defined group, likely voters.
Only about two-thirds of adults are registered to vote, and these adults tend to be more politically engaged than those who are not registered. That can translate into differences in attitudes.
Some polling organizations also report on the subset of the public they deem likely to vote, particularly for surveys conducted in the run-up to an election. This determination is complicated, but might be made by asking respondents questions about their voting behavior and intentions, or by using people’s past records of voting, among other things.
Since different polling organizations use different definitions of the term “likely voters,” it’s important to understand who is and isn’t included in the definition being used. Only a third of U.S. adults cast ballots in the last midterm election, so a report based on likely midterm voters is aimed at understanding the views of that subset of the adult population.
Survey data can be tricky for non-experts to follow, as we’ve noted in prior blog posts. It can be confusing for some readers to understand why 51% may not represent a “majority,” or how a survey of 1,000 people can tell you what an entire country thinks. One way to help readers interpret the many poll findings they encounter is to be as clear as possible about the group of people whose answers are being described.
