Six Reasons Why You Can’t Trust the Polls – Especially Now
We are told that election polls these days are highly scientific. Therefore, to suggest that they could be off by more than the statistical margin of error is just wishful thinking.
Most of the polls have said all along that Hillary Clinton will win the presidential election with ease. However, the recent announcement by FBI Director James Comey that the investigation into Clinton’s email is being reopened raises the possibility that support for Clinton could erode. It will be interesting to see if polls conducted after Comey’s announcement show Trump pulling ahead or merely predict that Clinton will win by a narrower margin.
Here are six key reasons why you shouldn’t trust the polls, particularly at this critical moment in history:
The polling industry today is in crisis -- Though polls accurately predicted several recent U.S. elections, the industry has serious problems. As Nate Silver (who engages in the dubious practice of handicapping elections as well as sporting events) admits:
“The polls have managed to produce high-quality output (pretty good forecasts of election outcomes) with worse and worse input (fewer and fewer people responding to them). It’s something of a paradox.”
The biggest problem is indeed that fewer people are responding to surveys. On average, pollsters have to contact ten people to get one response. This introduces what people in the polling industry call “non-response bias.” While there have always been some people who refuse to answer surveys, there has been a dramatic shift from landlines to cellphones in the last 15 years, and an equally dramatic upgrade from basic cellphones to smartphones in the last eight years. People today have more and better ways to communicate with family, friends, and colleagues. And now they can easily screen out calls from telemarketers and pollsters.
This is particularly a problem in the final days before an election. Pollsters are under pressure to produce results in a short period of time, but it takes longer to reach the same number of people.
Polls employ risky shortcuts to try to get accurate results at lower cost. With fewer people responding, pollsters are under pressure to either employ automated techniques or go with smaller samples.
Pollsters believe, with good reason, that live contact between poll-takers and respondents is more reliable than automated techniques that rely on keypad input or voice recognition. Plus, “robocalls” are subject to Federal Communications Commission restrictions and are even illegal in some states.
The alternative is to employ techniques to help ensure that a small sample is representative of the larger, target population. A good way to do that is to ensure that the demographics of the sample are consistent with the overall population. Ensuring that the sample doesn’t significantly under- or over- represent people in terms of race, highest level of education completed, and so forth isn’t very controversial. However, pollsters also rely heavily on pre-cooking the percentages of Democrats, Republicans, and Independents.
Pre-cooking the percentages based on political affiliation may work well when the parties select relatively uncontroversial candidates. But does it work well when large numbers of Democrats feel their candidate was treated unfairly, and large numbers of Republican leaders declined to publicly endorse their party’s nominee?
Saying the polls are “scientific” gives them more credit than they deserve. This is just an attempt to bamboozle people. It’s not fundamentally different from saying that “studies” show conservatives are more susceptible to authoritarian behavior or that “experts” say that liberals are more creative thinkers.
Today, people often say that something is “scientific,” or that the overwhelming majority of scientists believe something, as a way to intimidate and silence opponents. Historically, most great scientific discoveries contradicted what the overwhelming majority of scientists (or “natural philosophers” as they were once called) believed at the time. Physicist Richard Feynman argued that good scientists are always questioning and doubting their own results.
The fact is that polls are subject to many sources of error, some that are well understood, and others that may not yet be recognized. Measurement errors can be reduced, but they can never be eliminated. Today’s polls rely heavily on assumptions that could work in some cases and not in others.
It’s particularly unwise to trust polls for being “scientific” at a time when there are so many last-minute surprises.
Polls may be used either to measure attitudes or to shape public opinion. Just as many “studies” are actually performed to promote an idea rather than test it, early polls may be conducted to shape public opinion. Rush Limbaugh’s theory is that most pollsters don’t really try to be accurate until the last week, because those polls are the only ones that will be closely scrutinized when the election is over.
What if Limbaugh’s theory does not work this year? What if pollsters fear that a poll showing Trump winning by a slim margin puts their ability to attract future funding at risk? Throwing out a few dozen responses as unreliable could flip the result in a poll of 2,000 likely voters.
While polls were very accurate in recent U.S. elections, recent polls in the UK and Israel were not. Ten of the thirteen polls conducted during the last week before the British referendum on EU membership said “remain” would win. Worse, they showed “remain” winning by up to 10 points, while the three that showed “leave” winning predicted only a 1-2% margin of victory and were online polls (generally considered less reliable).
Likewise, polls for the Israeli election in 2015 showed Netanyahu’s Likud Party losing to the Labor Party when, in fact, Likud ended up with more seats in the Knesset than anyone expected. And while polls correctly predicted that Scotland would vote to stay in the UK, they underestimated the size of the victory by about six percentage points.
There is no guarantee that U.S. polls will continue their winning streak. If recent overseas polls are an indication, the problems confronting today’s pollsters are finally breaking through to the surface.
Given widespread condemnation of Trump, people may be reluctant to tell strangers that they support him. This issue has been beaten to death, but keep in mind that now about 80% of U.S. voters have smartphones that make it easier than ever for them to avoid telephone surveys.
If election polls were as accurate and reliable as some would have us believe, then many people would not bother to vote. Fortunately, a poll is nothing more than an attempt to predict an outcome based on what a relatively small sample of people say they are going to do. File most poll results under “entertainment.”