The Great Polling Hoax

Earlier this week, national media boldly reported that Jeb Bush, nearly doubling his polling numbers, had jumped ahead of the Republican pack. Further, Scott Walker had made big gains and now clearly was in second place, with Marco Rubio strongly in third. Six candidates (including Donald Trump) somehow each scored only 1 percent or less. Rick Santorum miraculously was at zero.

The resulting media spin can affect each candidate’s momentum and fundraising, as well as shape reporting over the next few weeks.

One plausible reason for the apparent Jeb Bush surge is that this widely heralded NBC News/Wall Street Journal survey was conducted June 14-18, a period dominated by coverage of Bush’s announcement on June 15.

But we need to take a closer look at this major national survey, a hybrid conducted jointly for the mainstream media by two veteran and competent private polling firms, Hart Research Associates, a Democratic firm, and Public Opinion Strategies. The methodology here raises new doubts about averaging five very different national polls to disqualify Republican presidential candidates from at least the first televised debate.

After all, we know that being in a debate gives a candidate the opportunity to pull ahead in the polls. So this is a circular argument. Exclusion from the debate could doom a candidate. That’s why I’ve opposed an opening debate limited to the top ten candidates. Instead, I have advocated that if polls are used, the first national debate run two successive nights, thus over two news cycles. Night One would have candidates polling 1, 3, 5, 7, 9, 11, 13 and 15, and Night Two would have candidates polling 2, 4, 6, 8, 10, 12, 14, and 16.

More relevant than these national polls would be polling only from the opening primary states. For example, a just-released Suffolk University poll of New Hampshire primary Republican voters shows Donald Trump in second place. I don’t know whether the survey is accurate or not. But I do know that when a candidate wins in a primary state next year, the resulting media coverage will quickly affect national polling numbers and provide momentum for the next primary.

But if early national polling is the criterion, let it at least be done plausibly and consistently. In reviewing the national surveys, it’s not only comparing apples and oranges, now there are vegetables, as we’ll see by examining this latest poll, the highly visible NBC/WSJ survey.

This particular survey consists of one thousand completed interviews with “adults.” Of those, 83 percent, a higher percentage than reality, say they are registered to vote, the socially acceptable response. My experience is that about 10-15 percent of these self-identified “registered voters” are not actually registered. Actual registration is not verified — another problem with samples not derived from a voter database, but that’s par for the course.

Yet, in some states, only registered Republicans can vote in a Republican presidential primary; in others, such as California’s open primary, independent voters and even Democrats can cross over. This survey dodges those variables as well as ignoring turnout (past voters, likely voters) and simply asks: “…Would you vote in the Democratic primary, the Republican primary, or would you wait to vote in the general election in November 2016?” This approach, perhaps not ideal, is acceptable.

But here’s the major problem. We end up with a national sample of only 236 self-described Republican primary voters, with a stated statistical margin of error of plus or minus 6.38 percent, a range that dramatically qualifies the results, especially at the clustered lower end. That sampling error assumes the universe of Republican primary voters is correctly described, but we just saw that the definition here may be problematic. Besides, that sampling error does not take into account non-sampling error, including this survey’s methodological challenges I will now discuss.

Before I do, consider that we are not asking about a zero-sum game of two candidates, where it’s either Smith or Jones, or undecided. For example, Smith goes down, so either Jones or undecided, or both, go up. With sixteen candidates, the possibilities and permutations are vast. This is another reason why a sample of 236 is woefully inadequate.

Near the outset, all respondents were asked to rank statements like “the decline in traditional moral values” or “too many candidates are wealthy and do not understand the economic problems of average Americans.” These statements might bias the ballot questions that generally should be asked earlier.

Although all respondents also are asked their positive and negative opinions on Republicans Jeb Bush, Carly Fiorina, Mike Huckabee, and Marco Rubio, many confused journalists simply and incorrectly assume the results published for those questions are only of Republicans.

More significantly, asking perception questions about only these four Republicans, before the Republican primary ballot question, could bias ballot results in their favor.

Before the primary ballot, respondents also are asked three general election ballots, with Hillary Clinton against Jeb Bush, against Mario Rubio, and against Scott Walker. Apparently, the other candidates don’t rise to the level of a general election ballot test, another possible bias.

Primary ballot questions, whether for Democrats or Republicans, should precede the general election ballot. After all, voters must select a party nominee first. But this survey is designed more to generate news and measure three high-profile general election ballot matchups. This bias against the missing thirteen Republicans thus could affect the primary ballot question and ultimately determine who is allowed in the first debate.

The good news is Republicans in the survey are asked whether they could support each of the possible sixteen Republican candidates. These sixteen mini ballot questions are asked in rotated order. Rotation is crucial, but with only 236 Republicans, that’s a maximum of only 15 per rotation pattern (236 divided by 15), adding more variability.

But, then, bias returns with a vengeance in the crucial ballot question:

“…Which one of the following candidates would you favor (RANDOMIZE) Jeb Bush, Scott Walker, Rand Paul, Marco Rubio, Chris Christie, Mike Huckabee, Ted Cruz, Rick Perry, or Ben Carson or would you vote for one of the other candidates that were mentioned in the previous question?” (Emphasis added.] The seven remaining candidates are mentioned only if the respondent answers “one of the other candidates.”

Can you see why this ballot question favoring nine candidates is a self-fulfilling prophecy?

Perhaps the rationale is that a cumbersome ballot question with sixteen candidates would not work in a telephone survey and could inflate the undecided. But excluding seven candidates from the pivotal ballot question unless the respondent effectively asks for their reading is highly prejudicial to them, especially for debate qualification.

It also distorts reality. Obviously, voters do not actually vote by telephone, they use a printed ballot with all the candidates. And in many states that ballot may further identify the candidate, such as U.S. senator or governor. This survey tests candidates only by their name, while other surveys test differently and can be weeks apart, but the Republican National Committee/CNN/Fox averaging approach for debate qualification would treat the most recent survey, or the largest sample, no differently than an older survey or a smaller sample.

Other national surveys for debate qualification are not consistent. Regardless, this focus on early polls, even if they were comparable, is misplaced. We know about volatility from so many past presidential primary elections. Using polls to exclude very closely clustered candidates seems arbitrary, and the exclusion of some candidates will appear inherently unfair

Instead, Republican leaders should encourage all the candidates, in return for participation, to debate issues and not attack each other. That’s the priority. Republicans leaders also have a legitimate interest in limiting the number of debates, finding a fair moderator, and recruiting panelists who ask substantive questions without a hidden agenda.