It isn’t unusual for the state of the polls to be a big issue this close to an election, but this week has been different from any previous campaign I remember — it’s not who’s ahead in the polls, it’s the polls themselves that are the big topic of discussion. My article “Skewed and Unskewed Polls” got picked up by both Drudge Report and Rush Limbaugh’s Stack O’ Stuff and — along with plenty of other contributions — made the topic of how polls are being performed into a national one.
The problem is that the discussion (as happens only too often) is now being led by people who don’t understand the whole topic very well. So let’s just talk about this a bit. I promise to almost completely eliminate the math; believe it or not, people can learn to reason about statistics without learning the central limit theorem.
Imagine a future day when, through technology and psychic powers, we can at any moment take an instant poll, checking what people mean in their heart of hearts to do when they vote in November. (And let’s not think what else could be done with that technology — this is a thought experiment.)
Secretary Dumbledore, Minister of Polling, goes into the office and pushes the appropriate button, and the poll is taken, click. It’s 51,267,303 for Romney and 49,109,941 for Obama, with 2,007,007 undecided. Does this tell us how the election is going to really come out? No, because it’s still 40 days (and nights) into the future. People may change their minds. Some people will die unexpectedly. And those two million-odd undecideds will have to either decide how to vote or decide not to vote at all.
And that’s the way it would work if we had this Perfect Magical Polling Wizardry. That would be a perfect poll.
Of course, we don’t. So this is what real polling companies do:
– First, they get a whole bunch of phone numbers. (When I was a kid, they actually sent pollsters door to door, but that’s pretty much died out.)
– Then, they establish some rules: people who are qualified to answer are adults, or adults who are registered to vote. They want to make sure to sample enough of important demographics (woman, minorities, and so forth) and they don’t want to sample too many people because it costs them between $10 and $100 per person they call. So they set some goal for the number of respondents they want, say 1000.
– They start to make calls, and at each call they ask some questions. Let’s say they call me, and I tell them I’m a male, mixed-race, Buddhist, registered Republican. If by chance they’ve already called the other one, they may say “okay, thanks, that’s all we need.” Or, they may go ahead and ask their questions.
– They repeat this thousands of times. Yes, thousands, plural: some people don’t answer, some people hang up, some say “Mommy’s in the bathroom and I’ve got a new kitty, want to hear him?” Eventually, they get a big enough sample.
(How, you may ask, do they know what “big enough” is? There’s a mathematical way of estimating what the probable error will be for a given population and sample size. You can look it up — look for “standard error” and “confidence intervals.” The main thing to remember is that it quickly leads to diminishing returns. You’ve got to get lots more respondents to improve the accuracy very much. This is why nearly every national poll has more or less the same margin of error: that’s as much accuracy as the press will pay for.)