Don't Be a Poll-Hater: Election Accuracy Remains Unchanged over Past 20 Years

With all the attention generated by the 2016 election, political spectators obsess over the steady diet of polls served up by the media. Poll numbers are hailed by victors, criticized by losers. Some critics are from campaigns, some criticisms come from political celebrities on social media. Do these criticisms hold any merit?

No, reminds the American Association for Public Opinion Research (AAPOR), the leading professional association of public opinion researchers. The professional association is leading an outreach effort to educate and demystify issues surrounding political polling:

An important part of this effort is to tell an old story again … and again. Despite headline-grabbing claims to the contrary, studies continue to show that the average error rate in candidate preference polls by the leading survey organizations was less than 3 percentage points in 2014.

Here’s the real headline: The accuracy of election surveys has been virtually unchanged over the past 20 years.

Now, some polls are poorly constructed and poorly executed. Some are “online” polls which are not scientific — they aren’t based on a random sample, and no “special sauce” will make them so. People who participate in an online poll (really, a panel) are compensated. The whole thing is more like a giant focus group.

AAPOR says:

The proliferation of pre-election polls using a wide variety of methodologies, some less rigorous than others, makes it expected that a few high-profile mistakes will occur. We urge the media and the public to not lose focus on the usefulness of pre-election polling and to ask questions about methodology when they disseminate and consider poll data.

Perhaps the first test of the 2016 political season will be the results tonight from the Iowa caucus. And it seems especially relevant to point out that polling for a caucus is a different breed than polling for a primary: it’s much harder.

First, we have the issue of turnout in caucuses.

In a primary, voters have options like early voting, voting by mail, or showing up on election day and voting in their precinct. But in a caucus, a voter has to commit to showing up at the right place, and cannot just drop an envelope in a post office box. It’s an hours-long commitment.

Caucus voters are surrounded by electioneering, which is not permitted at polling places for a primary election. A poll respondent will be confronted by all sorts of attempts to dissuade them from their choice at a caucus, but in a primary that’s much less likely. Maybe you told a pollster one thing, but after a few hours of the caucus, you change your mind. It’s difficult to measure that precisely.

When you ask people on the phone if they are planning to show up for a caucus, they like to say “yes” — even if they aren’t going to attend. This is “social desirability bias.”

And then there’s the various campaign ground games. Some candidates benefit from low turnout, some benefit from high turnout. Are new people being approached to participate, armed with information to show up and caucus? Will there be an avalanche of new participants at the Iowa caucus? We don’t know (2012 exit polls say 38% of Iowa caucusers were first-timers), especially this year, when the conventional wisdom suggests there’s a group of politically alienated people energized by Donald Trump.

How many of them should be added to the mix? It’s hard to say. Are they being effectively “turned-out,” or just saying so on the phone? Also hard to know.

Tonight, of course, we will have a better idea of how accurate the pollsters have been. But one thing’s for sure — the media won’t be giving up their obsession with polls and their accuracy anytime soon, so be on alert.