Get PJ Media on your Apple

PJM Lifestyle

Cocktail Napkin Website Planning and Obamacare

It was the horrible first day load that made the exchange websites fail, right?

by
Charlie Martin

Bio

October 3, 2013 - 2:00 pm
Page 1 of 2  Next ->   View as Single Page

Just for variety, today’s science column is about something I actually have some professional qualifications to write about.

Drinking-from-a-Hose

No, that really hasn’t ever slowed me down, I just wanted to note it.

In fact, my master’s thesis, “A Software Performance Engineering Environment,” was about tools to allow software engineers to develop performance models of software alongside the software. I spent some years in IBM and Sun’s consulting practices, usually dealing in one way or another with web-based businesses. I had a very popular talk, “Capacity Planning on a Cocktail Napkin,” which I later wrote as an article for SmartBear Software.

There really is only one explanation for the meltdown of the Obamacare exchanges since 1 October, and that’s utter incompetence.

Now, lemme ‘splain.

Back in the old days, at the Very Beginning Of The Web, all a web server could do was deliver a static piece of text. It was a brilliant hack by Tim Berners-Lee, who realized that he could build a little editor for a simple markup language and add one little change — a special tag that could address another page of text in the same markup language. To make it work, he needed a program that could return those files and a simple way the editor could ask for the files it wanted. The markup language was a subset of a commonly available commercial standard, SGML, called the Hyper Text Markup Language, HTML, the server program using a very simple text-based protocol called the Hyper Text Transfer Protocol, HTTP, and the rest, as they say was ….

Yes, class, that’s right, “history.”

From this simple hack the whole World Wide Web was made.

Now, with the same foresight that led me to buy Borland stock over Microsoft when they both went public, I thought at the time that it was a mildly amusing notion, but I didn’t see much future in it. I was head-down in category theory working on my dissertation.

Comments are closed.

Top Rated Comments   
"But you’re going to hear a lot of people in the next few days talking about how big the load was."
------------------------------------------------------

Oh - its a 'load' alright!

But a different 'load' than what you were referring to - and your editor won't let me spell it out.



42 weeks ago
42 weeks ago Link To Comment
They are so stupid, they don't even know the right questions to ask, let alone find the right answers.
When the kids who rioted in the sixties didn't have to take finals (because their schools were in ashes) my Mother predicted that we would have wished we shot them all, once they were in charge of the country, because not only would they be commies, they'd be STUPID commies.
And NOW, we have the generation that the STUPID commies TAUGHT!
Well, you deserve to reap what you've sown.
May God have mercy upon us.
42 weeks ago
42 weeks ago Link To Comment
"There really is only one explanation for the meltdown of the Obamacare exchanges since 1 October, and that’s utter incompetence."

I'd agree with that. Especially in an age where any old doofus with a credit card and paypal account can get a reliable eCommerce site up. However, I've seen a lot of crashing and burning in my time.

Having seen this close up, in that stench of failure I detect:

1. Doing it yourself, homegrown.
2. Using the well connected and politically reliable rather than skilled
3. Islands of information in fiefdoms. HHS, IRS, I'm sure there's more. That's my biggest problem today - departments that do not want others connecting to their stuff.
4. Bad architecture forced by the political realities of the bill. Every state doing their own?

This will be a hot mess for years.
42 weeks ago
42 weeks ago Link To Comment
All Comments   (70)
All Comments   (70)
Sort: Newest Oldest Top Rated
As a federal PM, I have seen presidential priorities proceed without due diligence of requirements mgmt., architecture reviews, testing, etc. Just get it done, with the assistance of the 'best value' contractor who is rarely equipped to make it happen.
41 weeks ago
41 weeks ago Link To Comment
Having been a part of designing and implementing a few large government systems, I can tell you with some authority what the fundamental problem with big systems is: almost nobody in government knows what the work of their agency is. There are lots of people in government who are really good at their jobs, but most of them are just super clerks; they've learned the rules and regs and SOPs so they can recite them like a Baptist minister can recite the New Testament. If they don't PO some political appointee, screw something up very obviously, and live long enough, they progress from Widgetmaker I, to II, to III, to IV, to Widgetmaking Supervisor, then Widgetmaking Manager. Above that in most agencies it takes some cocktail party skills or if female a horizontal career move. The right cocktail party or bed and the right person gets elected you're the politically appointed head of all or part of the Department of Widgetmaking, and you still don't have an effin' clue why widgets are made or what they're used for.

These people really can't describe to even the best program developers what a system is supposed to do. They only know what their little piece of it does, though they know that very well and guard that knowledge and the attendent status VERY jealously.

Trying to design big systems in government is like trying to write Civil War history from the memoirs of old soldiers; the only thing he knew in the first place was what was happening at most eight or ten feet away from him and he could only see that through thick clouds of powder smoke.
41 weeks ago
41 weeks ago Link To Comment
The actual data entry task of the problem is not particularly difficult. It's basically a shopping cart. I wouldn't mind betting the developers have elected to keep a lot of the workflow logic and input validation server-side rather than pushing it to the browser. It's easier to do during development but it scales poorly. What's the round-trip bandwidth consumed every time someone hits a Submit button? There's any number of high-efficiency client/server techniques that can mitigate this but they are quite a bit more complex to program. They can pay huge dividends, though. AJAXian interfaces can strip down the data-transfer costs by two orders of magnitude and leverage the power of the user's computer to do the grunt work of page rendering etc.. Now there's AJAX's big brother, WebSockets, making an appearance. I've got a WebSocket client/server implementation that can feed 500 separate client instantiations a second from a single server-side request broker process. A desktop-level PC running Linux doesn't even break a sweat. As for the database integration layer: I wonder how well designed the data model is, and what steps have been taken to support the sort of queries it is likely to be facing. Badly structured queries can alter the rate of transactions a given server can handle by a thousand times or more.

I think a lot of these issues are not really soluble without a tear-down and rebuild, which is clearly unacceptable. So this software will continue to suck for the foreseeable future.
41 weeks ago
41 weeks ago Link To Comment
Charlie:
1,200 visits an hour is 28,800 visits a day, not 288,000. I know 'cos my little usgovernmentspending.com is peaking at 1,200 visits an hour what with the shutdown and all.
41 weeks ago
41 weeks ago Link To Comment
I think my fingers are stuttering.
41 weeks ago
41 weeks ago Link To Comment
What happened to the hyper-sophisticated get-out-the-vote programs [http://www.technologyreview.com/featuredstory/508836/how-obama-used-big-data-to-rally-voters-part-1/] that the Obama re-election campaign used?

They couldn't ask the programmers who did those for help?

Question: Was the story of "our programmers can beat up you programmers" a steaming load in the first place? Another question: Was this initial failure due to deliberate sabotage?
41 weeks ago
41 weeks ago Link To Comment
I thought government was designed for inefficiency, or for doing things in the safest way. Perhaps they just don't really have smart people doing their work for them. I don't think the best programmers want to have anything to do with gov't work, unless they are paid and treated well. How can such work compare with the sexiness of the cutting-edge start-up, the chance to innovate and be stimulated in your work? Still, the degree of apparent failure is still impressive.
41 weeks ago
41 weeks ago Link To Comment
Governments with any sense don't want "really smart people" or "the best programmers" working for them. They want a few really smart people deciding what what should be done and predictable mediocrity below them; just like the military, you don't want heroes, you just want people who'll predictably do their assigned task.

If you need really smart people to do work for you or the best programmers, you contract for them, get the work done, and get them off your property and out of your budget as quickly as possible. This is where Democrats screw up: they don't know what really smart people or really good programmers are; they just know who thinks like they do. So, they hire somebody with reliable politics rather than reliable performance.
41 weeks ago
41 weeks ago Link To Comment
I'm not sure this can be blamed all on the programmers.

I have to wonder about who did what part of this work. A tenant in the same building I work in was deeply involved in the coding of [some part of] this project. It's a recognized name in the software biz. (Charlie, I'd be happy to give you the specifics offline.)

I know this because of the extra security we have in the building this week due to an unwanted media presence, including an incident where a few of the media breached this company's security and got inside their offices for a while. So, goons in the lobby asking to see badges all week, and strict instructions for us to NOT interact with the media.

But it's a small team. They only have one floor of a not particularly big building. I'd guess there's not more than a hundred or a hundred fifty people at the EXTREME high end. Might be more like 80 or so.

So, it's a pretty small team (not big enough to have done all of this) from a well-known company (and so presumably , reasonably competent), and they are (apparently) a key part of this fiasco. Hmmm.

I'd place my bets on the problems being due to an unusually high degree of bureaucratic micro-management of contractors, to include completely unrealistic goals, timelines, and budgets.

Consider: The people in charge of this are all certain to be dedicated leftists. What is one of the common denominators of dedicated leftists? They don't do reality well. They are prone to believing that things are the way they wish them to be, simply because they wish them to be! After all, they have all the good intentions and good ideas (unlike those terrible conservatives!), so what need is there to look at mere facts?

This isn't a character trait you want in a program director, and the higher up this kind of thing goes, the worse the results. In this case, it starts at the top and goes all the way down through the bureaucracy (you don't think they have anybody but their own on the team, do you?), and the private contractors get stuck holding the bag.
42 weeks ago
42 weeks ago Link To Comment
Mark, I'd love to hear the details. You can mail me at ask.charlie.martin@gmail.com. In the mean time, this reddit thread makes amusing reading:

http://www.reddit.com/r/webdev/comments/1nifc5/i_guess_a_couple_of_are_trying_to_sign_up_for/?sort=confidence
42 weeks ago
42 weeks ago Link To Comment
Oh, PL$ASE! I just saw this on the reddit thread:

"Ten thousand hits per second at times to the California exchange website. Could be the Tea Party trying to DDOS. But seriously people probably want healthcare, or are at least curious what Obamacare has to offer."

It's bizarre, but this really is how these Koolaid drinkers 'think'.

It's scary too, that there are so many who are so completely brainwashed and disconnected from reality.
41 weeks ago
41 weeks ago Link To Comment
Done, sir.

That reddit thread is hilarious. I'm no coder, but I know enough to follow the gist of it.

For the non-techie folks, basically, what we have there is professional programmers doing collective face-palms at the fundamental stupidity of whoever created the Obamacare software.
41 weeks ago
41 weeks ago Link To Comment
The administration could easily have borrowed the IT expertise of Ebay or Amazon, Google or Facebook, but instead they farmed the job out to a bunch of community organization groups with precious little in house web knowledge beyond how to post snarky tweets. One would almost think those many millions of dollars were more a reward for loyalty than well spent cash for a nation wide rollout.
42 weeks ago
42 weeks ago Link To Comment
What amused me the most were comments by Obama and Sibelius compare the Obamacare rollout to Apples release of the iOS 7 update. That's like me comparing myself to Derek Jeter. Yeah, I can catch a ball that my kid tosses to me.
42 weeks ago
42 weeks ago Link To Comment
At the beginnings of my "career" I used to work for a software company who used to do load testing for telecom equipment. So basically our tools were functioning as a black box simulating high stress traffic. We tested quite a few satellites and the company used to be hired as a 3-rd party in order to give an opinion about how the hardware and firmware behaved in high load conditions.

To the day I still think it is a neat idea to hire an independent contractor for testing your equipment or software.
42 weeks ago
42 weeks ago Link To Comment
1 2 3 4 Next View All