Polling Theory: When Polling Is Useless

We figure we can get away with a three percent margin of error, so we dip up a cupful of BBs, and being careful not to pick one color over the other, we count out 1,000 of the BBs, and discover there are 530 red BBs versus 470 blue BBs. So we can report to the Boss that there are 53 percent red, plus or minus 3 percent.

If we repeat this experiment 20 times -- we should hire an intern for that -- we should get somewhere between 500 and 560 red BBs about 19 of those 20 times.

Of course, this is really just what a polling company does: they ask some number of people who they plan to vote for, and count the results. If they say the margin of error is 3 percent, that means they talked to roughly a thousand people. Every time. This also explains why the margin of error is usually between, say, 2 and 4 percent: a reasonable sample size is somewhere between 500 and 2000.

Now, the polling companies could get narrower margins of error, but if you look back at the table, you can see that to reduce the margin from 3 percent to 1 percent you have to ask ten times as many people their opinion. When you figure that every polling call costs between $2 and $20, you can see that can get expensive quickly.

But now, let's extend our thought experiment -- sounds much more impressive in German, by the way, "Gedankenexperiment" -- and assume that the paint on the red BBs wasn't completely dry. It's a little sticky, so when we're mixing the BBs in the cement mixer they tend to stick to each other and to the walls of the mixer bucket. A lot of the red BBs are taken out of circulation. That means when we dig out a handful of BBs, they're not really well-mixed, and our sample isn't really random. We get more blue BBs because the red ones are stuck to the mixer.

Now, we count out our 1000 BBs, but we get 490 red BBs and 510 blue BBs. This is what is called, in statistics, a systematic error or systematic bias. Here, the term bias doesn't mean there is some conspiracy -- it's just that there is something about our method that systematically undercounts the red BBs.

So that's the end of our experiment, and I'll leave it as an exercise to decide why we're mixing colorful BBs in our cement mixer, and how we're going to get the stuck-on red BBs out before we return the mixer to the rental place. Instead, let's look at Zombie's recent piece. He (she? It? Does it matter to a zombie?) pointed out some interesting facts. Polling companies are now reporting that their telephone results are working out like this:

38% could not be reached

53% were contacted but actively refused to answer

9% cooperated and answered the polling questions

In round figures, that means out of 10 phone numbers tried, 4 don't answer at all, 5 answer and hang up, and only about 1 out of 10 actually cooperates.