After the 2020 U.S. Presidential election, an experienced pollster named David Miller thought he knew why the polls underestimated support for Donald Trump. He described his view in the Washington Post, writing that:
“I conducted tracking polls in the weeks leading up to the presidential election. To complete 1,510 interviews over several weeks, we had to call 136,688 voters. In hard-to-interview Florida, only 1 in 90-odd voters would speak with our interviewers. Most calls to voters went unanswered or rolled over to answering machines or voicemail, never to be interviewed despite multiple attempts.”
In summarizing the problem, Miller wrote, “… we no longer have truly random samples that support claims that poll results accurately represent opinions of the electorate. Instead, we have samples of “the willing,” what researchers call a “convenience sample” of those consenting to give us their time and opinions.”
What Miller diagnosed was non-response bias—a problem that arises when the people who respond to your poll or survey are different than those who do not. Although there are many reasons why some people may respond to a survey and others don’t, the problem that non-response bias creates is that the people researchers end up studying are systematically different than those who are not represented in the study.
Pollsters have known for decades that people are increasingly unwilling to participate in polls and about the problems this can create. In 2016, for example, White voters without a college degree were especially unlikely to participate in polls. The result was that many forecasts seriously underestimated support for President Trump. Even though pollsters thought they fixed this problem heading into 2020, polls in 2020 also underestimated support for President Trump, likely because voters low in social trust avoided the polls. As the last two Presidential cycles show, correcting for one form of non-response bias doesn’t necessarily protect polls from other forms of systematic non-response.
It’s easy to see how non-response bias affects traditional polling. Because polling relies on random sampling, anything that systematically affects who participates in the poll can lead to biased results. Online polling has many challenges too.
But most behavioral research is not conducted with random sampling. In fact, with most online surveys, researchers know they are drawing from convenience samples. Studies that aim to make frequency claims about a population, like a poll, are the most sensitive to non-response bias (see an overview of issues with online polling here).
However, when researchers are interested in learning about the association between variables (are people with higher incomes interested in a new product?) or testing the effect of experimental manipulations (does webpage layout A or B cause people to spend more time on the page?) non-response bias is less of a threat. And, for any type of online research study, there are multiple ways to reduce non-response bias.
None of the actions above are a perfect remedy for non-response bias, and some actions apply to some types of research more than others. Nevertheless, taking the time to think about whether each action above applies to your research project is worth the effort because minimizing non-response bias leads to more accurate and precise research findings.