Survey research ain’t what it used to be.
Back in 2020, the Harvard Business Review summarized the changes that have diminished polling accuracy. The article described the industry as “living on borrowed time,” and predicted that its increasing errors would not be soon–or easily–corrected.
The basic problem is low response rates. Thanks to caller ID, fewer Americans pick up the phone when a pollster calls, so it takes more calls to reach enough respondents to make a valid sample. It also means that Americans are screening themselves before they pick up the phone.
So even as our ability to analyze data has gotten better and better, thanks to advanced computing and an increase in the amount of data available to analysts, our ability to collect data has gotten worse. And if the inputs are bad, the analysis won’t be any good either.
It now takes 40+ calls to reach just one respondent. And there really is no reliable way to assess how those who do respond differ from those who don’t. (I know my own children do not answer calls if they don’t recognize the phone number–are they representative of an age group? An educational or partisan cohort? I have no idea–and neither do the pollsters.) There are also concerns that those who do respond are disproportionately rural.
These things matter.
A sample is only valid to the extent that the individuals reached are a random sample of the overall population of interest. It’s not at all problematic for some people to refuse to pick up the phone, as long as their refusal is driven by a random process. If it’s random, the people who do pick up the phone will still be a representative sample of the overall population, and the pollster will just have to make more calls.
Similarly, it’s not a serious problem for pollsters if people refuse to answer the phone according to known characteristics. For instance, pollsters know that African-Americans are less likely to answer a survey than white Americans and that men are less likely to pick up the phone than women. Thanks to the U.S. Census, we know what proportion of these groups are supposed to be in our sample, so when the proportion of men, or African-Americans, falls short in the sample, pollsters can make use of weighting techniques to correct for the shortfall.
The real problem comes when potential respondents to a poll are systematically refusing to pick up the phone according to characteristics that pollsters aren’t measuring…. if a group like evangelicals or conservatives systematically exclude themselves from polls at higher rates than other groups, there’s no easy way to fix the problem.
As the article notes, with response rates to modern polls below 15%, it becomes extremely difficult to determine whether systematic nonresponse problems are even happening.
These problems go from nagging to consequential when the characteristics that are leading people to exclude themselves from polls are correlated with the major outcome that the poll is trying to measure. For instance, if Donald Trump voters were more likely to decide not to participate in polls because they’re rigged, and did so in a way that wasn’t correlated with known characteristics like race and gender, pollsters would have no way of knowing.
Then there’s the failure of likely voter models.
People tend to say they’re going to vote even when they won’t. Every major pollster has its own approach to a “likely voter” screen, but they all include a respondent’s previous voting behavior. As long as that behavior stays stable, these models work. But when something generates turnout among voters who have previously been absent, all bets are off. That happened when the Obama campaign energized previously apathetic voters, and since the Dobbs decision overturning Roe v. Wade, we’ve seen evidence of significantly increased registration and turnout among women who hadn’t previously voted.
As the Harvard article noted,
It may be the case that standard sampling and weighting techniques are able to correct for sampling problems in a normal election — one in which voter turnout patterns remain predictable — but fail when the polls are missing portions of the electorate who are likely to turn out in one election but not in previous ones. Imagine that there’s a group of voters who don’t generally vote and are systematically less likely to respond to a survey. So long as they continue to not vote, there isn’t a problem. But if a candidate activates these voters, the polls will systematically underestimate support for the candidate.
I am one of those who do not answer my phone if I do not recognize the number.
While I occasionally find a glimmer of hope when I read a poll that gives Biden the advantage, and then fall into despair when they say otherwise, what I am concerned about is the complacency of the Democrat voter. In that way, I think that the polls that show MAGA in ascendency is better motivation for me to participate in grass-root efforts and make donations to candidates.
“Polling is broken, and we need to stop hyperventilating about their results.”
A few years ago, questioning poll results, I began researching the source of the polling numbers. Most at that time were a relative few hundred people for their view on issues effecting the entire nation. Even the few with numbers in the low thousands were of little use nationally or even locally on local issues.
Our source of information and the polls and surveys are on our increasing number of electronic devices we are expected to know how to install, set up and use once we remove them from the box they come in. The fact that they are electronically powered makes our source of that power more important than ever before. Last week, AGAIN only on my side of East 19th Street in this small development (approximately 200 residential homes) we had a 7 hour power outage; this has gone on on this side of 19th Street the 19 years I have lived here and long before per original home owners. Weather not an issue and the same area transformer repaired repeatedly only to fail again and again. Last August 29th to September 3rd a major power outages in many areas of Indianapolis after a major storm; not only loss of news sources and poll results but hundreds of thousands, if not millions of dollars in foods hauled out for trash pickup resulted. If we are to use electronic devices to run our lives as well as for information sources, we need to have electrical power we can rely on 24/7.
I again reported to AES the repeated problem and asked for answers regarding this small area; the result was that they found no reports of such outages in their records for years. AES replaced IPL as our power source long before changing the name of the monopoly provider; in addition to power outages there are billing problems with their new system resulting in some people billed thousands on their monthly statements. How much information is lost with the outages and what is the solution to running our lives on electrical power which cannot be maintained…with required electronic vehicles as our future transportation?
While talking with friends just yesterday regarding this problem in my area, and theirs on their farm; I reached the conclusion that after the invention of electronic dishwashers, washing machines and dryers they should have stopped inventing electronic devices to “improve our lives”. Can we get a poll on views of this situation?
I don’t think I get calls from pollsters anymore. I have a nasty problem that drives them crazy. I give them a response that they don’t like. When asked a question I will say something like “I know the question was designed to elicit this response, but the issue is more complicated than that.” Then I go on to tell them how that is. They really hate that. It takes twice as long as it should, but I love to do it. My hope is that the young person (usually a college student) at the other end of the line will think about I said and maybe learn something. It’s one of the hazards of working with so many researchers for years.
The polls support the election horse race coverage in our media. They all feel like they possess the pulse of America. But, as we all know, they do not. There is no way that 1,200 citizens represents the pulse of America when you consider how polarized we are in this country.
Yesterday, there was a report that Biden’s campaign committee told him to stop worrying about young people because they wouldn’t vote for Trump no matter what. These must be the same yahoos who managed Clinton in 2016. Even the folks who comment on this blog know that not casting a ballot is a vote for Republicans.
I suspect our youth and many others will be supporting third party candidates for POTUS. Nobody HAS to vote for GenocideJoe or CriminalDon if their conscience won’t allow it. The vote shamers are already out in full force but these are just party hacks who are scared that one candidate over the other will “bring down democracy.”
In case nobody was watching, the Democrats saved the House Speakers job just a few weeks ago. They are all owned by the MIC which is part of the oligarchy. They are all voting in lockstep with war profiteering and the Zionists in Israel. They are all voting for protectionism against China and Russia even at our own expense.
Washington sold us out years ago. The people serving, mainly sociopaths, have realized it’s the quickest way to become a millionaire in this country with great lifetime benefits. These are not moral leaders. They’re in it for the money which is the antithesis of morality. I don’t need a poll to tell me that my vote has become worthless on a state and federal level. Any changes I want to see in this corrupt manifestation will come from actions taken outside of voting for our two corrupt political parties.
Interesting comment, but the job remains the same: Get out the blue vote. Maybe we will overachieve, but we do not want a close vote. We want blues with majorities in both Congressional houses and the White House. Then we can start to restore the Republic for which we stand.
All of these polls can make future voters decide to not vote. I pay no attention to the polls as they are slanted by either the poll solicitor or the media in presenting any results. I will make up my own mind before I enter the voting booth, but I will vote. I hope more people are becoming cognizant of the fact that the only thing that does matter is going to the polls and voting.
Scammers and robo sales calls stop most of us from answering calls or responding to texts from unknown numbers. If it’s an important call they will leave a message.
A couple months ago I received multiple texts from political surveyors. Some claimed to be from the national Democrat party and others claimed to be from out of state candidates. They may have obtained my phone number from the state’s registered voters list or from each political party’s voting data sources. I refused to respond to the surveys, marked the texts as spam and blocked their numbers from contacting me in the future.
Scammers are now so technologically sophisticated that most of us don’t trust any form of contact from unknown numbers.
I got a call from a surveyor just the other day, and given the difficulty of understanding her, on top of what I saw as an annoyance factor, I just hung up.
It does seem that the polls are becoming increasingly inaccurate.
If I were teaching statistics in the fall, this would be required reading.
Scarey polls are good for bringing in political donations but little else. I read them with a so-called “grain of salt.” As Sheila notes > The only poll that counts is the one taken on Election Day. I’m informed that the polity doesn’t pay much attention to present day polls or even politics in general until Labor Day, and that’s months away, so let’s give our jangled nerves a rest.
Thank you for this. I’ve felt that the polls showing Trump leading Biden must be wrong but haven’t been able to defend that thought with evidence. I just found it hard to believe that tfg could be gaining more support than he was losing since 2020. I have even entertained the idea that the polls were being deliberately manipulated but I didn’t want to fall prey to my own conspiracy theory or wishful thinking. I feel better now.
I never answer a phone call if I don’t recognize the person or number. And I block many calls. If someone needs to talk to me, he or she can always leave a message.
I don’t answer the phone for unknown numbers anymore.
Something you didn’t touch on, any valid and accurate polling becomes increasingly expensive. Two years ago the Red Wave was predicted on a large number of low budget unscientific polls. It might be possible that given the new reality that to conduct a scientifically valid poll, the budget needs to be higher.
Something else that was reported on early on in the current primaries, is that Trump consistently polled 10% higher than actual vote counts. So one thing that recent polling has proven is whatever method they are using, the polls are skewed by at least 10% in Trumps favor.
Every time I start to read a news article and see that it revolves around what the polls say, I just say to myself, bullshit, bullshit, bullshit, and move on to something else. I even wrote a letter to the Washington Post complaining about their paying so much attention to the polls, but of course they didn’t print it.
I have read polls that show either Trump or Biden is way ahead, which I ignore. I will start paying more attention to such “polls” come October. My guess is that today Biden is ahead but, like today’s pollsters, I can’t prove it, and even if he is, it is no barometer of where either he or Trump will be come the big November poll.