So What’s Wrong with Research 2000’s Polling?

After the elections in 2008, Daily Kos, the largest liberal community blog out there, contracted pollster Research 2000 to conduct a large number of race-specific and “State of the Nation” polls.  The results of Research 2000’s surveys have come under increasingly intense fire lately, and on July 1, Daily Kos founder Markos Moulitsas Zúniga filed suit against Research 2000 and its owner and CEO Del Ali. As a poll compilator who has used Research 2000’s polls in my projections, I am keenly interested in the beef that the uber-leftist Kos has with them.

Polling is an inexact science, and the numbers that polling firms produce are constantly scrutinized by political observers of all ideological stripes. Sometimes, when results seem out of line with others — or they don’t match one’s preferred outcomes — pollsters can be accused of driving a political agenda. But rarely, if ever, do you hear of a lawsuit brought against a pollster for outright fabrication. Simply oversampling one demographic or another, or asking respondents leading questions, is no grounds for legal action. Moulitsas must have seen something much more significant — and damaging — in Research 2000’s work than just methodological discrepancies

In fact, what he alleges is quite serious, indeed. The lawsuit states that Moulitsas “was approached by a number of independent statistical analysts with regards to Research 2000’s polling for Daily Kos. Their analysis of the published data revealed a number of statistical anomalies regarding the results which revealed that Research 2000 had almost certainly falsified the results in whole or in part.” As Greg Sargent of the Washington Post affirms, “this is a big, big deal. Research 2000 polls have been widely cited by many news organizations, and have helped shape the national political conversation.”

So what exactly left Kos feeling defrauded by Ali and Research 2000? Sifting through the lawsuit itself and comments made by others on both the right and the left of the political spectrum, I’ve identified three main factors which I believe drove him to sue.

Pollster-introduced error

Nate Silver, founder of fivethirtyeight.com, is partly to blame for the disillusion with Research 2000. He recently revised his extensive analysis of pollster performance and published the updated results. Using a massive database of thousands of polls from hundreds of polling firms and numerous weighted metrics, Silver constructed a quantitative method for ranking poll-taking skill. The end of all this number crunching is a value he dubs “pollster-introduced error.”  Essentially, the lower that number, the better the polling firm.

At the top of the list of 64 firms with at least 10 polls published during the sample period were such familiar names as The Field Poll and SurveyUSA. At the bottom? You guessed it. Research 2000’s pollster-introduced error was one of the five highest. In fact, no other polling firm releasing as many polls fared worse. Undoubtedly, when Kos saw the polls tied to his website falling so far down the accuracy scale, he must have felt he was not getting his money’s worth. Yet Silver’s calculations, while certainly disconcerting, were just the beginning of the rift between Daily Kos and Research 2000.

Astronomical coincidences

Pollster performance similar to Research 2000’s might typically result in lost business and damaged prestige, not litigation and accusations of fraud. But if there were deliberate fabrication of results, legal action could be justified.  That’s just what independent analysis of Research 2000’s work seems to uncover. In the lawsuit, Kos highlights a startling set of data points contained within Research 2000’s own poll reports. They involve the male and female sub-samples of one type of weekly polling question.

The … sub-samples either came out both with even numbers or both with odd numbers 776 of 778 times. … Since the odds of getting a match each time randomly is 50%, the odds of obtaining 776/778 matches is the odds of obtaining 776 heads on 778 tosses of a fair coin, an event which should occur one in every 10 ²²⁸ (ten followed by 228 zeroes) times.

In engineering circles, there is a term for those odds. The term is negligible, a practical impossibility. For a moment, though, I want to come to the defense of Research 2000. The accusations here suggest cooked numbers based on the parity of the gender sub-samples, not on the actual responses of those polled. And who’s to say whether Nate Silver’s complex and cumbersome calculations correctly measure polling accuracy? Del Ali is quoted in the lawsuit questioning that very thing. However, characterizing an established expert statistician like Silver as “nothing but a fringe blogger” doesn’t instill much confidence in the criticism. If only we could see the raw polling data Research 2000 accumulated while conducting all those polls, then we could judge the merit of their methodology.

Black box secrecy

Alas, Research 2000 seems reluctant to produce any raw data. According to Kos, many attempts have been made to coax this all-important information from Ali. All have been met with promises, obstructions, and extenuating circumstances, but no data.  Even if Silver’s analysis were all wet, and even if Research 2000 could reasonably explain the even/odd phenomenon of their sub-samples, their lack of transparency does nothing to ease suspicions of impropriety. If they have nothing to hide, why try so hard to hide it? Ali’s avoidance screams of foul play.  The most logical conclusion, then, is that Research 2000 did indeed falsify the data “in whole or in part.”

So what’s wrong with Research 2000’s polling? Dismal performance, statistical impossibilities, and no accountability. Their polling, at least over the last two years, illustrates perfectly an interesting and timely correlation Silver found in his study of polling performance.  Generally speaking, the less a pollster is committed to transparency and full disclosure, the more inaccurate are their polls. One thing is clear in this prognosticator’s mind now that the dubious details surrounding Research 2000’s polls have come to light.  You will see no more of them in the calculations at Election Projection.