Survey research ain’t what it used to be.
Back in 2020, the Harvard Business Review summarized the changes that have diminished polling accuracy. The article described the industry as “living on borrowed time,” and predicted that its increasing errors would not be soon–or easily–corrected.
The basic problem is low response rates. Thanks to caller ID, fewer Americans pick up the phone when a pollster calls, so it takes more calls to reach enough respondents to make a valid sample. It also means that Americans are screening themselves before they pick up the phone.
So even as our ability to analyze data has gotten better and better, thanks to advanced computing and an increase in the amount of data available to analysts, our ability to collect data has gotten worse. And if the inputs are bad, the analysis won’t be any good either.
It now takes 40+ calls to reach just one respondent. And there really is no reliable way to assess how those who do respond differ from those who don’t. (I know my own children do not answer calls if they don’t recognize the phone number–are they representative of an age group? An educational or partisan cohort? I have no idea–and neither do the pollsters.) There are also concerns that those who do respond are disproportionately rural.
These things matter.
A sample is only valid to the extent that the individuals reached are a random sample of the overall population of interest. It’s not at all problematic for some people to refuse to pick up the phone, as long as their refusal is driven by a random process. If it’s random, the people who do pick up the phone will still be a representative sample of the overall population, and the pollster will just have to make more calls.
Similarly, it’s not a serious problem for pollsters if people refuse to answer the phone according to known characteristics. For instance, pollsters know that African-Americans are less likely to answer a survey than white Americans and that men are less likely to pick up the phone than women. Thanks to the U.S. Census, we know what proportion of these groups are supposed to be in our sample, so when the proportion of men, or African-Americans, falls short in the sample, pollsters can make use of weighting techniques to correct for the shortfall.
The real problem comes when potential respondents to a poll are systematically refusing to pick up the phone according to characteristics that pollsters aren’t measuring…. if a group like evangelicals or conservatives systematically exclude themselves from polls at higher rates than other groups, there’s no easy way to fix the problem.
As the article notes, with response rates to modern polls below 15%, it becomes extremely difficult to determine whether systematic nonresponse problems are even happening.
These problems go from nagging to consequential when the characteristics that are leading people to exclude themselves from polls are correlated with the major outcome that the poll is trying to measure. For instance, if Donald Trump voters were more likely to decide not to participate in polls because they’re rigged, and did so in a way that wasn’t correlated with known characteristics like race and gender, pollsters would have no way of knowing.
Then there’s the failure of likely voter models.
People tend to say they’re going to vote even when they won’t. Every major pollster has its own approach to a “likely voter” screen, but they all include a respondent’s previous voting behavior. As long as that behavior stays stable, these models work. But when something generates turnout among voters who have previously been absent, all bets are off. That happened when the Obama campaign energized previously apathetic voters, and since the Dobbs decision overturning Roe v. Wade, we’ve seen evidence of significantly increased registration and turnout among women who hadn’t previously voted.
As the Harvard article noted,
It may be the case that standard sampling and weighting techniques are able to correct for sampling problems in a normal election — one in which voter turnout patterns remain predictable — but fail when the polls are missing portions of the electorate who are likely to turn out in one election but not in previous ones. Imagine that there’s a group of voters who don’t generally vote and are systematically less likely to respond to a survey. So long as they continue to not vote, there isn’t a problem. But if a candidate activates these voters, the polls will systematically underestimate support for the candidate.