Polling Theory: When Polling Is Useless
Do current conditions make polling meaningless?
October 5, 2012 - 12:00 am
<- Prev Page 3 of 3 View as Single Page
Think about our BBs. We only needed 40 red BBs out of a thousand to “refuse to be counted” to reverse out little “poll” completely. And here, we’re seeing that by the time our pollsters have counted 1,000 BBs, 9,000 of them refuse to be counted.
What effect does this have on the polls? It’s hard to say: the one thing we don’t know is what the uncounted people think. But the one thing we can be pretty certain of is that the polls, with these kinds of conditions, are nearly useless.
<- Prev Page 3 of 3 View as Single Page
Excellent article. The one thing I would add is that the use of statistics and probability are indications that we are ignorant, to an extent, of the underlying process. For example, the probability of a rolling a die and getting a three is one in six. If we knew all of the forces and the initial conditions under which the die was thrown, theoretically we could predict, using the laws of physics, what number would turn up. Perfect knowledge should allow perfect predictions, with some exceptions. All polling is based on assumptions that we know at least a little of the underlying process of voting, which is proving to be less and less the case as people become more protective about giving away information to strangers.
In fact, it is possible to practice trick-shots to consistently win at Craps.
People also lie… I know that what I’ve told pollsters doesn’t have any bearing on my true belief. It’s my little effort to gum up the works and I know others who have done the same.
Well, dang it, we just don’t know If that 9% that are answering are representative of the voting public. The 38% that actively won’t cooperate could well be people who are generally annoyed at the culture of polling as part of media culture. I’m a lifelong student of the media and certainly public attitudes are changing. But, of course attitudes toward polls are not necessarily connected to attitudes toward media. They are also likely connected to anger at telemarketing! I understand polls well enough to not take any one poll seriously, but the RCP average has been pretty spot on in the last two presidential elections. A participation rate of 9% makes me wonder if the tried and true methods of social science still apply. We may see a 1948 style surprise.
Well, if I’m a smart political operative, and some of them are, I know that people think the RCP is a good indicator so I go to elaborate lengths with the help of my media running dogs to game the RCP.
as well as every other factor that might affect public opinion.
Hence, you rarely see a woman via the biased media who favors conservative thought. Every minority you see who espouses conservatism is mocked and marginalized. Movies consistently champion extreme liberal positions. And on and on.
Anybody who doesn’t view EVERYTHING nowadays with a critical eye and the fore-knowledge that leftists have infiltrated every information channel is a damned fool.
Case in point, we learn today that more jobs were added last week than in any week going back about 30 years, and lo and behold, unemployment is suddently below 8%. Why, it’s a virtual miracle. Here we have, just 4 weeks before the election, and just 2 days after a horrific debate performance, a once in 2 generation event, and it favors the president, the guy who controls the bean counters. Who could have possibly imagined that? obama couldn’t possibly encounter the same incredible streak of luck that he had in Sept/Oct of 2008, could he?
We are being gaslighted.
Probably not, if George Soros rigs another economic implosion it wouldn’t help Obama this time.
What can’t be polled: Of the 9% who is lying.
Now why would people lie? hmmmm.
I do in my small (and probably ultimately fruitless) quest to ruin polls as a news item.
I generally answer with my opinions accurately enough, since the feeling seems to be that there’s some advantage to a candidate to score well in polls. But my age, race, and party? Forget it. For the polls, I’m not an old white Republican, but a young Latino Democrat. (Who’s voting for Romney.)
Polling has been compromised and has been rendered worthless due to the widespread mendacity and venality of the MSM in support of their favored political party, the Marxocrats. People today know that polling has become nothing more than a propaganda tool used by the MSM to attempt to drive public opinion, rather than to record it. The low participation rates reflect this. Why waste your time answering irritating questions when you can’t be sure your answers will be rendered accurately or that they will likely be distorted to achieve some nefarious end?
Another big issue is the huge amount of unlisted cell phones that generally belong to younger people who have no land line. This means leaving out a big part of the voting public. I get the pollster’s calls all of the time and just hang up on them. Imagine what that does to their poll results if my actions are repeated thousands of times per poll. Even exit polls have shown to be useless and have caused great distress and had a negative influence on voters who have not voted yet. In today’s hostile political climate I think polls show us very little in the way of actual voter sentiment. The only believable poll will be on Nov. 6th.
Actually, younger people as a group tend to vote less frequently than do older demographics, so one would not miss much while polling.
The need now is for Republican pundits to raise their game – not to the same level as Romney, as that is an impossibility – but to at least an acceptable level. When the next polls come out look at the internals and point out that given current party identification and enthusiasm a D+ turnout model of any amount is a Democrat wet dream. This election will be an R+ election. Look at the internals and unskew the dishonest bias in favor of the Democrat candidate.
Pundits should also look at the other internals, such as age bias. I saw one the other day which had a turnout model where the 18-29 cohort matched the over 60 cohort. The polls will now show Romney ahead. If they had been adjusted to show an even R and D turnout, Romney was ahead all along. The panic was unnecessary.
Attack the amateur who can’t even be bothered to learn the financial facts. Praise the highly accomplished CEO who had a 3.97 GPA in combined MBA/JD. America is lucky that Romney has offered his services. Stop attacking him.
By my calculations, I just wasted about 7 minutes of my valuable time reading and commenting on this article.
Cheap at twice the price.
This is an awesome summary.
Thanks!
The hidden variable is an old problem in sampling.
All you need is one very, very small variable that makes supporters of X more likely to answer a poll than supporters of Y, and poof, without any intended bias at all, you’ve slanted the poll in favor of X.
Could be anything. Maybe GOP supporters are just busier, and have less patience for telephone interruptions.
I know lots of people who are eager to answer polls. Some of them actually talk to every telemarketer too. Is that really representative of a typical voter? I have caller ID, and I don’t talk to anybody whose number I don’t know. If it’s important, they can leave a message. Am I a typical voter? Who knows? But the polls will always reflect the phone answerer and not me.
I’m not too worried about the so-called losing perception and voter despair because my candidate is behind in the polls. I think this is highly exaggerated. We didn’t sue for peace with Japan because Tokyo Rose told us every day that they had sunk 5 American ships and killed 5,000 GIs.
“the so-called losing perception and voter despair because my candidate is behind”
How about:
“I’ve lost hope on my candidate to make changes that he promised last time, but I don’t want the rich guy to win. My candidate is ahead, I may as well stay home, and not have to own up to what I did on Nov. 6 when I lost my job just before Christmas. As a matter of fact, it’s quite embarassing to be fooled by that HopeAndChange thingie, but I’m always blue because my 1/32 Cherokee grannie was blue, what could I do?”
or
“My candidate is behind, I’d better get my butt off that couch and do my part on Nov. 6, so I can complain when that Hopey-Changey guy led us off the financial cliff.”
This is exactly the issue. The 9% issue, in and of itself, is meaningless. However, if Romney supporters (or Obama supporters) are, for some reason, more apt to either self-include (I’ve known people who tried to seek out polsters; they were weird people) or self-exclude (that would be me), then the sample gets skewed, making the result less representative. If there’s no selection bias operating, then it should all wash out. Having said that, the fact that the 9% issue exists at all raises a red flag that something is happening and that something very well might affect the result.
Mmmm, yes and no. IF there’s no selection bias happening, you’re right. But the fact is that ten out of eleven possible respondents is self-selecting out. That kind of level of self-selection is inherently troubling.
I never trusted polling since Ronald Reagan kicked Jimmy Carter’s behind in 1980. EVERYBODY said that Carter had the edge over Reagan on Election Day and that it was going to be close. Reagan won in a landslide. And remember all that “exit polling” that was done in 2004 that said Kerry was “definitely” going to defeat Bush? Well, how did that work out for them? Best thing to do is keep fighting hard for Romney and hope for the best on November 6. I think we will win, regardless of what any poll says.
I vote in every election and I never participate in polls. So I am significant in terms of the election, but never a component of polls. How many of me are out there? How many non-responders will vote? How will they vote? If you can’t make those determinations, by the time you get to 20% non-participants, using your poll to represent actual voting is meaningless.
Polls can be useful for trending. Say you use the same polling procedures and candidate A has steadily been losing points in the polls, you can realistically say that he is losing support. Or you might say that candidate B is likely to win if the polls consistently show him with a 10 point lead. But using poll numbers to predict the actual and numerical results, when the point spread is within the margin of error, is completely ridiculous. As it is with Romney vs. Obama.
No that 20% thing is wrong. The whole of statistical theory is based on the sample being an infinitesimaly tiny part of the population. It’s not what proprtion you sample that matters, its the absolute number of individuals.
Well, yes and no. Obviously, if you “sample” 100 percent of the population you get a 0 margin of error. But more importantly, for all this sample theory stuff to work, you have to be getting a truly random sample. If there is some process — like my sticky paint — that screws that up, then your tiny sample of the large whole stops being representative.
Charlie is right, sampling can only be statistically accurate if you can make a determination from each of the 1000 units sampled. I can’t pull out 1000 pieces of something at random and throw 200 pieces away and still claim my +/- 3% error margin passed on the remaining 800 pieces. Nor can I keep pulling pieces out until I get 1000 I can measure. Now the actual number of units I’ve looked at may be closer to 1200. I have now skewed the sample whether I like it or not.
I’m not saying that you can’t infer anything after that, it’s just that the confidence level drops dramatically, and probably nonlinearly, as the non-measurable units contacted grows as a percentage. I threw 20% out as a WAG. I really don’t know the correlation between non-participation and confidence level. And neither do the pollers exactly, that’s why they use comparisons to past exit polls, etc. But I think you can say, with confidence, that 90% non-participation renders the final result pretty much moot.
Quadratically, that is O(n^2).
LOL
OK Charlie,
I’m doing your experiment.
I’ve got a cement mixer with sticky blue and red things I’m sampling a thousand to get the proportion red because I want a 3% confidence interval.
Right. I’ve taken 10 handfuls of 100 each but because of the clumping I’ve got 8 handfuls which are all blue and two handfuls which are all red. So thats 800/1000 blue which is 80% blue which means 80%+-3% are blue in the mixer because I took 1000 individuals.
BUT
Now I’m looking at them and I’m thinking maybe I havent got 1000 samples of individuals. All the individuals in each handful are the same color. That means I’ve got just ten independent samples which means my 95% confidence interval isn’t 80+-3% but is actually has a bottom end of 50% and and a top end of 96% [1]. Maybe taking handfuls of individuals which are next to each other isn’t a good way to sample. I’ll use tweezers instead to sample individuals one at a time while the mixer is running.
Great. Now because each individual has equal chance of being picked by my tweezers they can be as clumped as they like in the mixer and I still get a sample with 1000 indepenent (of each other) individuals. Once I’ve done this 1000 times I’ll have my 1000 samples and +-3% confidence.
BUT
I keep dropping them. In fact I for every 100 I pick I only get 9 to my table to count. Well that just means I’ll have to do more work then I thought, I’ll have to sample about 11000 (crap!) to get the 1000 I can count, but once I’ve got those 1000 I’ll have my +-3% confidence interval, and I can by pretty sure (95% confidence) that the true proportion is somewhere inside that +-3%
SO
The cement mixer is the country;
the red and blue things are people;
the clumping represents localities in which people of a feather flock together;
sampling by taking handfuls would mean going to 10 neighborhoods and knocking on 100 doors in each one;
sampling by tweezers means dialing random phone numbers;
and dropping the objects means not getting an answer for every dial.
Now, Charlie, on what is your cliam that polling is useless based on? I mean what evidence? Based on our thought experiment it looks like current polling it should work fine doesn’t it?
Now go to your communities and go door to door. One person in eleven — 9 percent — answer the door and take your questions. The other ten out of eleven answer the door and say in response to the question “什麼？你是白痴嗎？” then shut the door in your face. Do you now infer that 100 percent of the population speaks English?
Depend. If the first question was “do you speak English” and also I was a flaming idiot in adition to being a flaming liberal (no tautology jokes unless they are actually funny) then maybe I would conclude that everyone speaks English.
What this comes down to is your, and others, assertion the democrats are more likely to answr polls then republicans. Usually on the vague basis that republicans are “more busy” (dressage practice? Whitening their spats?) with stuff that democrat voters don’t have to do.
But what’s your evidence for that assertion? Got any?
Evidence no. Just experience. Conservatives are much more likely to have a “screw the media” attitude.
Exactly…
Anyone I dont know in my neighborhood with a clipboard, IS UP TO NO GOOD WORKING FOR THE OTHER SIDE…
They are there to push an agenda, period.
Ditto for phone calls.
Basically, the world is made up of two kinds of people: “Pushers” and “Draggers”…..Pushers (Conservative, Republican, productive, working) are too busy with life, kids, school, work, running their business trying to get ahead in this world, to waste their time on “gripe talk” with some lefty-busybody who wants to “change the world”..Pushers are too busy navagiting it the way it is, and realize distractions like answering polls are of no benefit to them in that task. They dont anser polls. They hang up.
Draggers (liberals, teachers, media, Intern poll takers, etc) are generally unhappy with the the idea of being forced to “Push” in order to eat. They’re perfectly happy to Drag the system down to a screaching halt, and force people to STOP WHAT THEY ARE DOING and answer “why” the world must be so.
Pushers generally dont answer polls. Draggers generally do, delighted to gripe to a sympathetic ear about how THEY BOTH want the Pushers to stop making life so difficult for them.
Its the “systemic bias” he mentioned. Polsters and their typical “answeree” dont have anything better to do.
The rest of us do.
Root,
My brother used to refer to this as “boot straps” vs. “hand out” and he’d do the appropriate gestures. Boot straps…haaand out. Boot straps…haaand out. It was hilarious.
Reading comprehension is a wonderful thing, you should try it some time.
What I said was:
In other words, I expressly declined to conjecture what the uncounted 91 percent think.
Well they way I read it was.
1 it’s not possible to say if this has any effect.
2 but we can definitely say that it means that polls are useless.
And then I ask for evidence of a bias between responders and non responders and then we get “well republicans are busy ‘n’ stuff.”
Anyway, the main point ob which everyone should agree is get out and vote. And remeber that your vote is not a precious gift to be bestowed only on the deserving, but the best way you have of keeping/kicking the more mendacious liar out
Nemo, I think you have this confused idea that I’m writing all the comments — except, I suppose, yours.
Please, let me relate FYI a little story from the High School my two daughters are attending in East Central Florida (along the well known and discussed by political pundits “I-4 corridor”): they had a presidential vote in their HS, and the announced result was O – 50% and R – 48%.
Here’s the interesting part: my two daughters tell me that many kids in the School commented, after the announcement of the result, that the HS administration manipulated the outcome.
Get ready folks. If kids that know next to nothing about voter fraud and the stalinist “we count the votes” motto, already question the outcome of a mock election, imagine the reaction of conservatives on November 7th when the fraud will be awarded a second term.
Reading this, I thought of the method used by Johns Hopkins during the Iraq war, where some 900 Iraqi households were sampled & individuals gave the interviewer a verbal report of number of personal family members killed.
The “researchers” then extrapolated that number to the entire population of 25 million, coming up with a wildly inaccurate number in the hundreds of thousands of Iraqi civilians that had died in “Bush’s war”.
This “study” was further given legs in The Lancet by its ideologically left bent editor, Richard Horton.
Figures often beguile me, particularly when I have the arranging of them myself; in which case the remark attributed to Disraeli would often apply with justice and force:
“There are three kinds of lies: lies, damned lies and statistics.”
~Mark Twain
“There are lies, damned lies, and statistics.” – Benjamin Disraeli
“There are lies dammed lies and statistics”
Oscar Wilde
http://en.wikipedia.org/wiki/Lies,_damned_lies,_and_statistics
“Polling” is an evil variation on the sin of “transactional morality”, and it is a “non-science” where you can buy any result you want:
1) Polling is based on the concept of there is “no right or wrong”–there is only what is “popular” for the moment. “Circumstances” which decide what is good for us. The premise behind polling has been perverted from “let us view the pulse of our Republic” (to learn of trends or groups of shared opinions) to…”we will TELL YOU what you should think and do based on what others are thinking and doing”. WRONG. This is a wicked variation of the “joke” popular a long time back: “Twenty Million Frenchmen can’t be wrong.” WRONG. All Frenchmen can be wrong, and saying they are right because their idiocy or evil is shared by many or most does not change this.
2) Then, too, the fairness of the “pollsters” is “NEGOTIABLE”: How much money can a single pollster (ANYONE FROM ANYWHERE–lots to choose from) make on a single poll–$5,000…$50,000? How much money does BO & cronies have to BUY THE RESULTS, I.E. DETERMINE WHAT WILL BE REPORTED $5,000,000…$50,000,000? BO “borrowed” $5,000,000,000,000 (5 TRILL-Y-YUNS)and we have no idea where huge portions of that went. The Dem Rat “slush funds” available to buy polls, break laws, purchase perjury, reward co-conspirators, etc. IS SIMPLY INCALCULABLE. The BILLIONARY SOCIALISTS of the world, like George Soros, think nothing of spending millions to “rework democracy in their image”.
Baby Glock,
Dead on, I agree 100%…
A full size, old school, not very concealable “17″ (plus one in the pipe!) level of jam-free black polymer agreement!
On the basis of my own statistical study (the 2 individuals interviewed in Denver yesterday following Obama’s post debate speech), I have arrived at an absolute and inexorable conclusion, i.e., you can take this one to the bank.
Obama’s entire campaign rests on an appeal to the dumbest among us.
Communists manifacture data, events, news. They own the press.
Who buys the “unemployment” figure of 7.8% just released, let alone the “polls”?
It’s another sad day in (the former) America for true conservatives.
“manufacture”…sorry for the typo
Polls have a history of being “close” or “not close”, based upon their methodologies and lack of a “wishing bias”.
Methodologies, such as those illustrated here, have their own limitations. However, absent the “wishing bias” to the extent possible…the polls give a general impression or big picture snapshot.
Those polls that contain either a)a sample already tainted with a built-in bias; or, b)a “wishing bias” hidden within the question or the questioner that sways the responder in one direction or the other.
Retaining more of our animal origins than we might like to claim, we can sense through instinct “signals” that are being “sent” by strangers. Whether in a sales pitch, with children and stranger danger…or on the nightly news by listening to a “news” anchor with an agenda.
If the organization itself is not tremendously biased, but hires from a class of biased individuals, those who choose to answer WHO ALSO CAN BE SWAYED, will almost always sway toward the “wishing bias”. Coincidence? Of course not.
For those who will never be swayed, assuming no reason to lie, they will not react to the slant of the question. (Are you going to vote for President Obama or for Romney?). The bias in the question is subtle. Obama gets a title of honor and prestige…Romney doesn’t even get a polite title. Obama ALWAYS is listed first. If the wish bias has even the slightest inflection (Are you going to vote for [cheerful] President Obama, or [dry, cold] Romney?)
Again, you are not looking to change everyone’s voice vote. Just those who would be inclined to give you the answer you are looking for in the first place.
Again, assuming no bias in choosing your sample (for instance, if the polls are not giving you your “wishing bias”, you immediately adjust to calling specific neighborhoods that are nearly certain to hold nothing but blue BB’s, in order to “fix” the outcome the way you wish to present it), then the ONLY sample that is of interest would BB’s on the prediction date.
There are solid red BB’s and solid blue BB’s. They will not be swayed. Will they come out of the mixer on the date in question? What percentage of BB’s have “come out” strength?
The only remaining questions are about non-solid color BB’s or BB’s that are willing to change colors.
If we fix a baseline of the solid Red and Blue BB’s and come to the conclusion that they are equal as in this example…the ONLY questions we need to ask in POLLS are:
1)Are you willing to change from a Red BB to a Blue BB, or from a Blue BB to a Red BB?
If the response is 35 to 1 that “changeover BB’s” go from Blue to Red…that almost NOBODY is saying they would change from Red to Blue…AND…you have a baseline that is solidly at 50/50…then the BB’s most likely to impact the outcome are all in one direction.
2)Let’s take clear BB’s. As a clear BB, what color is most likely to rub off on you on this prediction date. If ALL the evidence suggests that the clear BB’s are HEAVILY leaning toward Red on this prediction date, AND your baseline is otherwise 50/50, then the BB’s most likely to impact the outcome are in that direction.
3)What is the likelihood as a Red BB or Blue BB that you will come out of the cement mixer on the prediction date? (the enthusiasm BB). If ALL the evidence is that enthusiasm and motivation are in one direction, that will impact the outcome.
IF, the baseline is 50/50, IF the clear BB’s all lean toward one color, IF the “changeover BB’s” are 35-1 in one direction, IF the clear BB’s heavily lean to the same color…THEN IT IS IMPOSSIBLE TO HAVE ANY…ANY…FAIR POLL THAT REFLECTS ANYTHING BUT A ROUT FOR THAT ONE COLOR.
The only explanation for polls giving an opposite result is either intentional bias or wishing bias.
And, that’s assuming a 50-50 baseline. If the baseline in reality tilts in favor of the Red BB, the polls are fatally flawed before the first phone rings.
Slightly better analogy:
The red BBs are copper and the blue BBs are steel. The copper BBs are denser than the steel ones, so no matter how much you mix, there will be more red at the bottom and more blue at the top. If you scoop the top, the blue will be oversampled. If you can core down to the bottom, you may get a representative sample.
They don’t know how to core to the bottom, because they’ve never seen the bottom, and aren’t even sure if it’s there. Because, you know, I don’t know anybody who voted for Nixon.
Looks like about 50% of the comments are above average today.
“Looks like about 50% of the comments are above average today.”
I disagree. At least half of the comments are below average today.
Oddly, I can’t find a single comment that’s actually exactly average.
No, but some are mean.
Snork:
“No, but some are mean”.
I was wondering who would finally “go there”!
Change your name to “snark” while your at it, you earned it today
Are they your standard deviations or typical deviants?
You do get a gold star for that line, by the way.
That comment was above par.
If we really want to gauge how my theory above works, let’s take an area that is known to be independent, but leans blue…but is willing to change color to red.
Let’s go not to Chicago, but to the 10th Congressional District …the northern suburbs of Chicago. Al Gore, Kerry and Obama all won there. Obama handily.
Glenn linked to what his happening there currently.
Blue is turning red…in a HUGE shift. Massive defection.
In Illinois.
If Illinois is going purple, the whole country is shifting. And, if the polls aren’t picking up on that, the polls are fatally flawed.
Zombie is a she. And thanks for this article.
Um… unless you’ve DNA sampled Zombie, I think Zombie is Zombie and her/his/its gender carefully guarded.
Or is this an invitation to a bet?
No but if that is Zombie as in Zombietime her sex was pretty well known in the past unless for some reason that was misdirection or she’s had some subsequent reason for anonymity. Or it’s somebody else.
I’m not sure why you’re being so confrontational on this since I was just trying to inform Charlie on something he didn’t seem to know. Or am I missing something?
There is one polling example that no one seems to be remembering: In 2004, on Election Day, the EXIT polls had Kerry winning the presidency.
Exit polls should be incredibly reliable: All you do is ask voters for whom they just voted.
If an exit poll is wrong, it is almost certainly caused by some sort of selection bias. It could be the pollsters on too concentrated in urban areas. It could be the pollsters were at the polling place for too limited a time. It could be a lack of geographic diversity. It could be the pollsters simply made too much of an effort to talk to young, good-looking members of the opposite sex.
As far as I know, the poor performance of 2004 exit polls has never be adequately explained.
My SWAG (semi-professional wild ass guess) is that the polling was falsified to sway the voters yet to cast their vote.
Actually, there’s an excellent explanation. The exit polltakers tend to be grad students, and when the pollsters are young female grad students, their sample leans heavily to Kerry.
If a young female grad student, who happened to frown when the people in front of me said they voted for Bush, asked me who I voted for, I might say Kerry even though I voted for Bush.
Actually that was largely explained. The exit polls were take late morning and early afternoon, so the responders were mostly stay at homes; mothers, students, elderly, etc. Also blue collar workers tend to get off earlier in the day as well. These demographics all tend to vote more Democrat.
When the white collar workers got off in the late afternoon, they were voting much more Republican but the exit polls were mostly completed by then. This was not factored in, and the polls skewed highly Democrat.
Regardless of any selection bias, all exit polls suffer from one fatal and obvious flaw — they totally ignore all absentee and early voters. That, IMO, renders them irrelevant.
The question I ask is the “who” responds to a telephone poll? What is the selection process or is it simply a random list of names from the respective party membership rolls or from registered voter lists? When I see the D+ 38% and R+ 32%, who makes up the 38% and 32%? Maybe my questions can be considered as based on ignorance but after reading the information provided by the polling companies, I still wonder about and question the methodology.
If a polling company were to call and I were to answer, if the first question would be if I planned to vote for President Obama or Mitt Romney, I would hang up. If the first question was posed as to which presidential candidate do you plan to vote for, then I may continue because the way the question was asked would be an indication of an unbiased question.
In the end, if we allow the results of a poll to influence how we vote or whether we vote or not, you can be sure of one thing. Ignorance reigns supreme in that household.
Phone polls are usually done with random number auto dialers. If you see more dems in the poll its because there were more dems in the population of people at the end of random phones. It’s not polling companies going out and looking for extra dems.
There are some ifs and buts but that’s pretty much that.
As I say, I really can’t get a clear concrete explanation of how the phone polls work exactly, because it appears to be considered a trade secret. But we’ve got a situation where ten of eleven numbers called are self-selecting out of the population. That, on its face, makes the sample suspicious.
I didn’t understand all of it Charlie, but I enjoyed it immensely!
Real politics is a little more like quantum mechanics: some BBs are yellow and some are green, but they turn red or blue upon observation. Or sometimes they just vanish.
And whether they turn red or blue, or vanish depends on how you observe them.
And blue BBs spontaneously turn red and vice versa, also.
This is a lot more difficult that you’re acknowledging, Charlie.
Of course, dead BBs always turn blue. And blue BBs can go into multistate superposition, and be counted in multiple states.
Quantum mechanics is a dirty business.
Bistro Math?
How much more difficult than “we don’t know anything because we can’t trust this sample” could it be?
Not a disciple of the Michael Mann school of statistics, are you? Didn’t you know that with enough garbage in, you get a perfect answer out?
Charlie, I assume that the Romney and Obama campaigns both want really accurate polling data, so how do they do it? I assume that they pay a lot more money for larger sample sizes than the MSM pays for entertainment polls. Do they also study the problem Michael Barone style with very accurate polls of small areas like swing precincts? Also, what steps do they take to test if the poll data makes sense?
Assuming that the Romney and Obama campaigns paid more and got accurate polls, how do outside observers figure out what they know? When the election isn’t close either way, they usually redeploy resources. When both sides have huge war chests, how much does each side bluff with long shot efforts?
George, if you notice, it’s very expensive to make the sample larger. Since the margin of error decreases as the square root of the sample size — or equivalently, the required sample grows as the square of the increased precision — simply making a bigger sample gets to be intractable. I don’t really know how the internal polls work, but I’ve read several people assert that the notion the internal polls are really better is a myth.
Good article. Thanks.
Lying to polsters. A time honored American tradition. I get a kick out of Frank Lunts’ ‘undecided’ dial groups. My patooty! They lie to get into the group. We know this because when Frank talks to them and asks questions most of em are obvious political junkies.
99% of political junkies knew who they were going to vote for at the end of the primaries and every conservative knew who he was going to vote against in January 2008.
From what little I know of statistics (not my best subject by a long shot) a statistical representation is a snapshot of what things are at that very moment. They do not predict the future by their very nature. How does it translate into election voting unless there is some built in “stickiness” to the numbers?
Obviously people are partisan and most likely have ideas or loyalties which will remain in play but what exactly are the underlying ideas of polling that it can be considered a worthwhile predictor of elections. If margin of error is only a number that translates into the margin of error that we really are producing what the electorate would vote today how is changing of minds, undecideds (true partisanship or merely preference) considered? What are these polls practical use?
Wonderful easy-to-understand explanation, Charlie. Good job!
Want an example of how to poll for a result? Just received it this morning.
My wife is a registered Democrat. Hasn’t voted for Democrat in the 25 years I’ve been married to her. I acted as her surrogate with the key pad.
Telephone polling (paraphrased) after a series of questions:
If you knew that the unemployment rate had dropped two full percentage points within the last two years, who would you likely vote for? Press 1 for Barack Obama; Press 2 for Mitt Romney; Press 3 for blah blah blah…
See a problem from the start? Pretty obvious, hey?
Barack Obama is going to be a big winner around my house.
I took seven graduate-level statistics courses and never had an instructor as good as Mr. Martin (I have a Ph.D. in rangeland ecology). Those freaks seemed to enjoy making it seem harder than it was. For example, in all those courses, I never encountered his sweet little formula “1/√n” for calculating the margin of error.
That said, what I did learn is that, based on Martin’s response-rate numbers, is that the samples pollsters get aren’t truly random and therefore are basically useless, in that they are self-selected (53% refuse to answer compared to 9% who do). An example from my own field would be say, a grazing-performance experiement in which the cattle used somehow volunteered by walking onto the scale of their own volition, rather than randomly picking the experiemental units out of a larger herd. No biological statistican, peer reviewer, or journal would accept such a thing. Apparently, “professional” polls are no more reliable than those you see on websites such as “Who won the debate?” These often bear the warning that they aren’t scientifically sound, for the same reason; self-selected samples.
I’ve always been dubious of polling/surveying as a method of collecting “scientific” data for these reasons, plus the fact that the answers contain the wildly unpredictable variable of different people’s honesty. If I weight a cow critter at 750 pounds, that’s a fact, within tolerance of the scale. A person’s “yes” or “no” might be fact or a lie. Also, it always seemed to me that surveys can be biased, perhaps unintentionally, by exactly how questions are phrased. For an extreme example: “Would you vote for Romney?” vs. “Would you vote for Romney given that he will push Grandma off a cliff?”
Keep up the good work, Mr. Martin. More sense, and more objective information, is what people need.
Thanks very much, John. I do recommend the Wikipedia article tro everyone, it’s quite good.
By the way, having grown up on a cattle ranch, I’d loe to learn more about rangeland ecology.
Where?
I’d like to get hold of you, but hesitate to post my actual email on the page. Can you access the one connected to the comments? If not we can dream something up.
San Luis Valley Colorado. South and west of Blanca. I’ll get your email.
Or maybe I won’t. You can email me at chasrmartin AT gmail you know where.
You know, I think that if that was the only practical way of getting the measurement (letting the beasts walk on) that you would get through peer reveiw just fine.
The reality is they usually require some encouragement. The problems with “volunteers,” be they beasts or humans, is that they may vary in some fundamental way from the entire population.
The significant factor is; Cattle can be depended on to behave in the same way as demonstrated, even in private. Humans CANNOT.
People will say something in person to someone asking a question in public where they can be overheard, but behave conversely in private. And as demonstrated, ad nauseam, few people answering their home phone are very reluctant to give DEPENDABLE answers.
Anyone that has had higher mathematics knows “statistics” is barely certifiable as “mathematics”. It’s no more dependable than using computers, loaded with every known weather pattern from the last 100 years, to predict what the weather will be on any given day. (But, actually, I do think these computers are more accurate than the most articulate, comprehensive polling.)
Polling, today, is used primarily for a predetermined result. And this has been demonstrated by some of the best in the business.
It does have an entertainment value to those that know better.
The significant factor is; Cattle can be depended on to behave in the same way as demonstrated, even in private.
Don’t know many cattle, eh?
No; Not that intimately.
My goodness, I didn’t think my rather far-fetched cow example would create such an on-going conversation. I do feel the need to respond to Geezer, re: the validity of statistics.
First, statistical procedures are in no way related to “models” designed to make predictions based on inputs for different variables. These can be biased by using unreasonable values for the variables. Statistics use real numbers (some measurement of samples collected) as inputs. Of course, statistics are subject to the “garbage in, garbage out” syndrome. If the data are sloppy, the results will be sloppy, but the analysis informs you of this by producing an unacceptably low confidence level. This is one of the main problems with polling data because we have no way of knowing if the answers are correct.
Statistical analysis of data is necessary because of the inherient variability of nature (including human nature). To cite another simple example from my field, suppose the task is to determine the total standing crop of vegetation of a pasture. If the pasture was absolutely 100% uniform across its entire area, you could take one sample (by clipping and weighing the veg from a small known area) and that would be the answer. But we know going in the pasture is never uniform. Production will vary tremendously at small to large scales by such factors as landscape position (top or bottom of a hill), soil differences, aspect (side of the hill), mix of plant species, past grazing patterns…I could go on and on. It’s been said the complexity of ecology makes quantum phyisics look like child’s play.
Therefore, we have to arrive at an average that has taken at least the major known variables into account. The most common type of statistical analyis, analyis of variance or ANOVA, does exactly that. ANOVA takes the total variation amoung the samples and assigns that part of the variation arising from a factor TO that factor. What’s left over is called “experimental error,” variation that can’t be accounted for by the known factors. Some of this is probably human error (one technician is clipping the veg at 1″ above the soil surface and the other one is clipping at 0.25″). The primary way to overcome this is to have a large enough sample size, arrived at by some function like Mr. Martin’s elegant 1/√n. On the other hand, for reasons of expense and time as related in the original article, we can’t over sample and so seek a sample size that’s merely good enough. Polling data would use different analysis methods because the data are categorical (yes or no) rather than a continum of weights of samples, but the end result still gives an answer with a measure of reliability.
My point is, statistics per se are absolutely valid, but only give an estimate of the truth, with a known level of reliability. And, like a meat cleaver, in the wrong hands they can be misused by building bias into the samples, among other ways (D + 9 polls for example).
Finally, I would like to apologize for any spelling errors. Before spell checkers came along I could compose, type and spell at 80 wpm. Nowadays I can’t spell cat without a spell checker. Computers certainly have made us dumber in some ways.
John O;
I appreciate your discourse very much. My mischievous comments are often taken seriously.
I scanned this article and comments for the term “probability” and it only came up 3 times. 2 times in the first comment, and once by Mr. Martin himself.
I think it’s called “Statistics and Probability” for a very valid reason. But, the “probability factor”, as in the polling being performed these days, has overwhelmed the statistical data; As you say garbage in/garbage out.
In science, engineering, and technology, it’s called a safety factor. And it usually is large enough to compensate for an undetermined error, e.g. the safety factors built into buildings and bridges.
It looks to me like Mr. Martin has displayed the actual value of the polling data being published today, by pollsters that have no real clue what taking a “mean statistical sample” really means, or cares. They are in the business of shaping public opinion, and are making a good living at it.
Well Charlie, I think the flaw in this is that as everyone knows, red BB’s are racist.
You know, I started to use black and white marbles and decided not to because I didn’t want to do the racist jokes. So much for that trick.
Quote:
“If we repeat this experiment 20 times — we should hire an intern for that — we should get somewhere between 500 and 560 red BBs about 19 of those 20 times.”
Wrong. About half the time you get fewer than 500 red BBs. What is probably true
is that you get between 440 and 560 red BBs 19 out of 20 times.
Nils Andersson
You didn’t read the whole thing. Try again.
What Snork said. The first example was 53 percent with a 3 percent MoE; so we should have between 50 and 56 percent, and since I conveniently chose a sample size of 1000, that is 500 to 560, 95 percent of the time.
The polling is useless, because of the lack of credibility. Human systems require faith in them for them to function for us. If banks lose the trust of the people, they fail. People pull out their money. If a President loses our trust, his Presidency fails. If the results of elections are not trusted, elections become meaningless. If the Press becomes suspect, it ceases to be news, but is instead propaganda, and people quit buying. If people get the view that polls have become corrupted, that they are mere propaganda tools, people quit participating.
The low participation is not a result of people not wanting to be disturbed. Folks like to have their voices heard. If you believe that the polling firm is utterly scrupulous and want to hear what you think, you’ll take that call. People are not taking the calls, because they refuse to enable the corruption.
Some people are taking the calls. Some still have faith in the polls and want to share their views, and some are happy to engage in the corruption, because they themselves are corrupt. Of course, some take the calls to goof with the corrupt, to mess up the polls even more.
Once the polls are perceived as mendacious, it’s over for them. Pollsters are killing their own livelihood, by being willing to skew results, so they can tell the client what the client wants to hear.
“Socioeconomic status” does not determine political belief.
There are lies, damn lies and liberals.
“There are lies, damn lies and liberals.”
I rephrase that to ‘Liars, damn liars, and liberals’.
1. In general, it does matter if a person does not respond to a poll. Remember, there are about 200,000,000 voters and 1,000 poll respondents. What is important, however, is that the “non-responders” be independent of the underlying demographics. If this can be assumed, then the repondents can be used to represent the non-respondents for that demographic group.
2. In general, political preference polling have been somewhat accurate in forecasting preferences. If you compare exit polling of real results to preference polling, they were pretty accurate in determining the preferences of major demographic groups. That is, if you look at a poll result and the exit polling, the “Republicans” showed preferences similar to the polls, the “Union HOuseholds” likewise.
3. Where the polls deviate from actual results the most is in sampling, particularly in the party affiliation, deviates from the actual election results. This is one of the hot issues right now, but it is not a new phenomena.
Edit: Number 1 should read “In general, it does NOT matter if a person does not respond to a poll”………….typo
So anyway I did an analysis of Gallup polls since (and including) Carter-Reagan. I looked at the difference between the predicted and actual share of the democrat vote.
Results were no siginificant bias (95% CI for average -0.9 to +3.8 pro dem)
Also the 95% tolerance interval for poll-actual was +-6%. So probably we can say that while additional sources of uncertainty inflate the size of the confidence interval for polls from around +-3% to +-6% there is no evidence of sytematic bias towards democrats.
If you look for them there are many more detailed analyses of many more polls which show a similar thing: uncertainty is larger than +-3% but no systematic bias and certainly polls arent useless.
Did those polls have a 91 percent failure rate? Look back at the original data I stole from Zombie: 53 percent of the potential respondents are self-selecting out.
I think the point is not that polls per se are useless — I didn’t sit through all those semesters or stats and probability because I’m a disbeliever — it’s that the peculiarities and paradoxes of the polls right now suggest something is going wrong right now.
Something going wrong right now that is different to the kinds of uncertainties that polls have in general.
Well the one thing I can think of is turnout. There’s nothing in this years polls that make them any worse than any other year at measuring intention, so if there is a special problem this year it must be the extent to which intention is converted into votes.
So the question is: is there something special about this election which means that the turnout will go towards reps and away from dems much more than a normal year. Lets think about that a little. In NEMO’S turnout model people get pulled to the voting both by their liking of ‘their guy’ or pushed to it by their loathing of the other guy.
One thing we could do is score the effect of ‘pull’ and ‘push’ on each candidate compared to a normal year (remembering that you cliam that this isn’t a normal year.) I’m goint to score them 1-5 with ’3′ being a normal year,
Democtrats-pulled to Obama:3
Democrats-pushed away from Romney:3
Republicans-Pulled to Romney: 2
Republicans-pushed away fromm Obama 4
So (a highly scientific) 6 points each I reckon (really this is just a concrete way of expressing my feeling in a way that can be argued about fruitfuly). This isn’t going to be a weird year where the supporters of one particular candidate just stay at home much more than the other. Taking ‘push and pull’ together I don’t think we’re going to see a relative turnout which moves a long way from peoples expressed intentions.
What those numbers express is:
Romney isn’t Reagan, or even Bush when he was most popular. Nor is he Bush when he was least popular.
Obama isn’t Obama when he was most popular, nor is he Carter.
It’s going to be a boring year where the polls get it about right.
Nemo, when you can’t trust your sample, the turnout model doesn’t matter.
Turn the question around: in order for the sample to be good, then declining to co-operate must be completely independent of party or Presidential preference. Can you make a case that it is? Otherwise, we have to question the sample because of self-selection.
Well we’ve already seen that polls in the recenet past are unbiased, so I’m just claiming that the non-response difference is no larger than say 2008, or 2004. Not that there is no difference.
Sure theres a lot of negative feeling about polling expressed mainly by conservatives. In fact about about 70% of republicans and only 14% of democrtats said that hey didin’t trust political polls, in a poll by PPP recently
But this kind of stuff about how polls ‘oversample dems’ has been knocking around for years now whithout showing up in real results.
So I’ll give you odds: 1) 4:1 that the popular vote is with +-3% of the final gallup poll or 2) 15:1 that it is within +-6%.
$1 too rich for you?
so I’m just claiming that the non-response difference is no larger than say 2008, or 2004.
The problem is that while you may claim that, it’s contrary to fact:
THE PRESS WILL TURN ON OBAMA: It hates LOSERS more than REPUBLICANS
The answer is “no doubt”. Next subject please. ABO2012
Polling is for stupid people. If you are totally stupid, you believe polls, or are influenced by them. WHO GIVES A CRAP what other people think, even if the polls were a direct query of 100% of the population. GROW A SPINE and vote on your own information and forget what the liars try to feed down your throat.. Jesus Christ.!
On one hand, you’re right, with such a low participation rate polls should be less accurate. On the other hand, while there was some chaos in the primaries, polling there usually wasn’t THAT far off. It may be that since pollsters were polling a very specific group it reduced the opportunity for bias, but it could also be that even 9% is a large enough sample to give you a general idea.
Course, I think a bigger problem is the intentional bias. I’m sorry, but I’ve seen so many polls making absurd assumptions about democratic turnout, that I simply cannot believe it is unintentional at this point.
Factors which pollsters and those who rely on them must make assumptions about:
1. Proportion of red BB’s to blue BB’s – reliance is placed on self-identification, corrected for by comparison to past election results, and screwed with by choosing WHICH election to use for correction (i.e. before the majority of the public came to the realization that mainstream media organizations – and presumably the polling organizations with which they associate – were putting their thumb on the scales or after… that is, before 2010 or after 2010.
2. Honesty of those who respond to polls and self-identify as either red or blue and who they intend to vote for.
3. This is the big one this time: Composition of the population of those who answer the phone and respond vs. those who either do not answer the phone or, once-answered, refuse to respond to the poll. If a significant population of those who deviate from the respondents is self-selecting OUT of the phone polling (say 75% of ALL those who plan to vote for Romney and who have-had-it-up-to-here-with-community-organizers-turned-president-who-urge-supporters-to-get-in-their-neighbors’-faces-and-the-media-and-polling-clowns-who-advocate-for-him), then the results are not reliable to any significant degree. Mock decent, reasonable, and fairly ordinary people with sexual insults like “tea-baggers” and politically self-serving claims that they are motivating murderous lunatics like Jared Loughner and James Holmes long enough and they will choose to marginalize you in return by refusing to talk with you when you want something from them.
Reports say that poll response rates are down dramatically. Pollsters try to figure out their own reasons for why this is so and to correct for it. Is there any doubt that their own biases and blindnesses will enter into their determination?
Watch the money contributions – Obama’s contributions from main street America are WAYYYYYY down. This election just doesn’t have the same historical motivation that the last one did – now it’s about the man and his true constituencies, rather than the symbol. This one’s for Americans now rather than the history books later.
The concrete mixer is looking like it will be delivering a load of blue concrete because the red BB’s are holding close to the bottom, eager to express themselves fully in the only poll that counts: when the load is emptied on election day and the power shifts AWAY from the aggressive community organizer and the entrenched media elite. Plan on a deep red BB landslide burying a thin layer of very surprised blue below.
Yes; I think the word “unexpectedly” is going to be used rather profusely in all the propaganda after November 6th.
Thanks, Charlie Martin, for your great sense of humor.
Charles,
The issue of skewed poll results have been out there for at least 10-15 years.
There are two very important developments in polling that break the present from the past.
1) The non-response rate, which is now on the order of only 1 usable response in 9 answered phone calls,
and
2) The rate at which people lie to the polls.
The mass turn over to cell phones and unlisted land line phone numbers is playing hob with whether political polls are in fact statistically representative of the actual voting public. The polling companies just don’t know and the only alternative is much more expensive face to face polling surveys.
Most media political polling companies cannot do them let alone do them well.
The second issue of “lying” is even more touchy with pollsters because telemarketers — in addition to making people less likely to answer without call screening — also makes people actively hostile and much more likely to lie than answer truthfully.
The extent of the second problem was explained to me in 2008 by a friend who moonlighted with political polling firms from his academic social researcher day job.
He explained to me what it would take to do accurate political polling election model, if you assumed a “hostile field protocol,” after a political blogger I read spotted the following in a TIPPs poll.
The TIPPs poll asked the following question:
“Do you display the American Flag”
…with the result being: “Obama 38/ McCain 52/ undecided 9.
and the political blogger stated something to the effect:
“People who wave a flag are undecided? Does anyone really believe this?”
For which my day-job academic said the following:
Your blogger seems to have hit on the idea of “hostile field polling” questions without being a pollster.
Yeah, that would be the start of it; if I were getting paid to do it and I thought of the flag question, I’d use it.
As he explained it to me, hostile field polling is what dictatorships use to find out what their people really think, because the people answering the questions are afraid to tell the truth. Pollsters in such situations have to ask very indirect questions, use huge sample sizes and take multiple polls. It is impossibly expensive for news organizations.
Americans are increasingly like folks in foreign dictatorships and will only describe reality to pollsters if they ask indirect proxy questions on polarized subjects like Gay marriage.
My pollster semi-pro then continued with this detail:
Typically for a sample size big enough to cover 67 stratifications (the current [2008] minimum number you need to adequately cover the American public — you’d need more to do it state by state, maybe as much as 3x as many) you need around 1200 respondents.
To do a good ‘hostiles protocol’ you need about 15 stand in questions for the one that you think they’re lying about (if you think they’re lying about more than one, you can overlap some questions, but obviously that produces a spurious correlation in the results. Or at least that’s obvious to me and the people who know what they’re doing; I’ve seen a lot of researchers screw that one up).
The 15 stand in questions all need to correlate with each other at about 60% or better.
And now you see why no network even tried it this time (2008).
To get those 15 questions you probably would need to test around 100 good ideas, in about 25 nationwide surveys (since they won’t all start out in the same survey and you have to see how they interact, so you’d test say 7 surveys in your first round … maybe 3/4 of questions would look promising in the second round, so you’d test 5 …. then 3 … then start doing mix-and-matches that would easily use up another 10 surveys).
You’d have to pay for 25 polls to make your polls be accurate again.
Forget all ideology; Fox wasn’t going to pay for anything like that any more than MSNBC.
By 2012, though, they may all have to.
I have not heard a whisper of any of the major political polling firms having done 24 national polls, of 1,200 people each, just testing out “how is the public lying to pollsters this election” questions.
At a guess it would cost $100,000 to $250,000 to set up the initial questions, with $50,000 of analysis per poll adjustment, and $500,000 to $1 million per poll using best face to face polling techniques to get that answer.
And the answer would only get you the _Right Polling Question Methodology_.
You would still have to do the polls using that methodology.
And to be really sure, you would have to take the same poll via face to face, and via phone only, to know the difference between the two polling methods so you could sell them via cost/quality/speed to your political consumers.
And that still leaves a “polling team people problem.”
According to my semi-pro acquaintance it takes a good, stable, team of experienced professionals to properly develop & execute a poll time after time.
It was his opinion that the reason Bill Clinton was so good, and one of the reasons Hillary got blind sided by Obama, was that Bill Clinton kept the same polling team from two years before he was president through his entire presidency.
Hillary’s Senate campaign was deeded that polling team by Bill.
She got rid of that team for money reasons before she started her Presidential run. That was why she was always a day late and a dollar short with Obama.
Her replacement polling team was no where near as good, even if it was cheaper.
Point in fact, the MSM’s dropping ad revenues mean they are going with increasingly cheap polls with skewed results they want, rather than good methodology.
Lying is cheaper.
Supreme Court to Hear Challenge to Restrictive State FOI Law The Supreme Court has agreed to hear a case, McBurney v. Young, challenging a Virginia state open government law that restricts access to residents and new media organizations operating within Virginia. Petitioners are out-of-state residents whose requests for state documents under the Virginia Freedom of Information Act were denied. The case presents the important issue of whether states can discriminate against non-residents by denying them access to state records. EPIC filed an amicus brief along with several open government organizations urging the Court to hear this case.
http://epic.org/2012/10/supreme-court-to-hear-challeng.html
Where this analogy breaks down is with the word ‘random’. Nothing random about human beings. They are NOT red and blue BB’s. Or BB’s of any color.
Polls can pick a winner, but so can my lucky silver dollar. After all the polling is done and the voting finished, either Obama will win or Romney will win. There is no choice ‘C’. Heads it’s Romney, tails it’s Obama. Mt silver dollar has a 50/50 chance of being right. So do the polls. The advantage of my silver dollar is that it’s worth at least a dollar.
A compromise between water and sewage is still sewage
Nemo
2 + 2 = 4, not 4 plus or minus 1. If polling was a science, the pollsters could tell us down to the last ballot what the outcome was. When statistical analysis is used to calculate air flow through a gas turbine or the temperature of OIL passing through a 3 meter stainless steel pipe, there is no plus and minus. There cannot be. People die if there is. That is frowned upon by engineers.
And using your definition, if physics were a science it could tell us exactly what the high temperature tomorrow will be.
Aside from all the other comments, I automatically disregard any poll that didn’t sample equal numbers of opposing groups.
When I worked at the State Department I found that nearly 90% of Foreign Service Officers were rated in the top 10%.
How do you like them BBs?
Using the figures in the article above:
If only 9 percent of those polled responded (say 500 folks, to get you your 3 percent margin of error), that would drop your actual number down to 45 respondents.
The margin of error for 10 respondents is 30 percent and the margin of error for 100 respondents is 10 percent. By doing a rough interpolation, you’d come up with a margin of error of about 20 percent for 45 respondents.
What this means is that the degree of certainty cannot be any more than 80 percent, or to put it another way, if you sampled 5 folks to pick either product A or B, you would expect 3 of them to pick B and 2 of them to pick A, but there would be a good chance that one of the folks who would be inclined to pick B, could go for A instead.