Obama and Big Data: Follow-Up
In response to my post earlier about how the Obama campaign took its database into the post-campaign, a reader writes:
Not to be picking a nit here, but most of what's been described about OfA's database and social media savvy isn't all that new. As a matter of fact, some of it has been around for quite some time. Worst of all for the RNC is that many of the tools used can be had for a paltry $49 per month.
Check out Gnip.com, Hunch.com, Klout.com, optimize.ly, socialbro.com, and take a look at the developer APIs at Facebook and twitter. Hardcore data analysis is now in the hands of mere mortals thanks to Gnip. You can find out all kinds of things about people's preferences from Hunch. Klout let's you know which followers can best spread the message. Optimize.ly can help you optimize your website design to increase conversions (read: donations). Socialbro gives you an easy to read dashboard on how your social media campaigns are working. It also identifies power users and influencers who you might want to engage directly. You'd be shocked what the twitter and Facebook APIs give you access to, with the consent of the user, mind you.
Bottom line is that OfA wasn't using anything that hasn't been available for years. With the advent of NoSQL databases, Hadoop, and other tools for handling massive data sets it has become possible to provide actionable analysis in reasonable amounts of time.
The RNC really needs to get a lot more technologically savvy then they are. The tools and systems are only going to get better over the next four years.
The last thought is undoubtedly true, and I know some smart people who are making that happen.
It's also true that there are many off-the-shelf tools available and many campaigns already use them.
That said, I'm not sure that the Obama campaign/OfA used tools that had all been around "for years." I don't think that we who were never inside the campaign know all of the tools that they used or who used them. They haven't disclosed everything, never will, and would be fools if they do.
They have disclosed quite a bit, as I wrote about back in January in this post. The Obama campaign built the "Cave," which was a massive data operation run by hires from Silicon Valley. Many of those hires had never been in politics, but are experts in their given fields. The Obama campaign exploited their tools and talents. See page 14 of this PDF of the Cave's operations. They did use much off-the-shelf technology, such as R. They also built some of their own tech. According to page 53, they hired 40 software engineers for the 2012 campaign. Some of those hires came from Google, Facebook, Twitter and Quora.
Some of their success can be attributed to brute force. They put a lot of volunteers on phones to make a lot of calls and they A-B tested piles of emails until they figured out which messages would work most effectively for them. On page 37 of the PDF, they admit that they also used good old-fashioned fear. See page 44 to see how well fear and loathing worked for Obama.
But then, they didn't just trust their own campaign and its marvelous machines. The 2008 Obama campaign tried to get the IRS to go after Clinton and Edwards and then conservative groups. Starting in April 2010, the IRS was busy doing deep opposition research, pestering hundreds of groups and thousands of Americans in the year or so leading up to the election. The IRS pushed its internal investigation's findings on the abuse until after the election, misleading Congress and the American people in the process. Its tactics may or may not have been directly imported from the Obama 2008 campaign, but they certainly resembled them. Obama fixer Bob Bauer was on hand for the '08 campaign where he tried to get groups opposed to Obama suppressed, then went to work in the Obama White House, then went right back to his 2012 campaign. The unanswered question remains, did the NSA's data sweeping play any sort of role in all this? It's too much at this point to assume that the people who populate the top echelons of this government have the scruples to resist turning government into a political instrument.
Up against Obama's database Death Star, the Mitt Romney campaign was always playing catch-up and it was impossible for them to catch up. They didn't have the IRS chasing Progress This and Obama for That on their behalf. Their own dashboard management tech, ORCA, was a spectacular failure that was never even game-day tested. They didn't have droves of tech specialists begging to leave Google and Facebook to join their data-driven campaign.
One clue to why OfA morphed into OfA 2.0 and carried the database with it is on page 56 of the Cave PDF: "Lesson: Future campaigns may need LONGER than the 18 months Obama had to develop their technology stack."
How much longer? Permanent will suffice, thankyouverymuch. There's always a mid-term just a short ways away.