Get PJ Media on your Apple

PJM Lifestyle

Clown Car Web Design

What we can deduce from the Healthcare.gov collapse.

by
Charlie Martin

Bio

October 10, 2013 - 4:00 pm
Page 1 of 3  Next ->   View as Single Page

shutterstock_142170055

A Programming Sutra

This is the way I heard it. (That’s the way all sutras start.) Long long ago — about 1997 and I’m not naming names to protect the innocent and because I figure the statute of limitations is up for the guilty, and the company I’m going to talk about has been through bankruptcy and several acquisitions so it’s not the same company anyway — a major toy retailer ToysForKids (TFK) with stores in malls all over America heard about this nifty new thing called “the web.” As I heard the story, two programmers in IT had the idea that TFK should be selling toys on the internet. They got permission to do a sort of side project, semi-bootleg, to build a demonstration e-commerce web site, ToysForKids.com. (By the way, that domain name is now owned by a domain-squatter in Hong Kong called “iGenesis Limited”, but then ToysForKids never existed anyway.)

They built the web site on a desktop server using a scripting language called tcl, and demonstrated it. It looked so good they got permission to take it live, and they happily started making dozens of sales a day with it. It really was a lovely site, too, won lots of awards.

The CIO was so pleased that he arranged a demo for the CEO. The CEO was so pleased that he arranged a big advertising buy for Thanksgiving Day during the football game — as I recall, $50 million — so that everyone would know about the new ToysForKids.com.

Everyone did. And everyone’s mom, wife, and girlfriend that had a computer went and tried to start their Christmas Shopping sometime in the first quarter.

Now, remember this is 15 years ago. The desktop server they were using wouldn’t make a good iPad now, and the Internet connection, while good for the time, had less capacity than Comcast promises me.

And everyone who was bored with football and had computer access was trying to use it. The site pretty much melted down; it wasn’t long before the programmers had found different jobs, the CIO wanted to spend more time with his family, and the CEO, um, retired.

Comments are closed.

All Comments   (10)
All Comments   (10)
Sort: Newest Oldest Top Rated
The problem is much, much worse than that. There are hundreds of databases that have to communicate, and the software that acts on the data must know how to correctly query each and interpret the result. The general term for such a system is "multi-tier". One tier--the website--faces the user. Another carries out actions. It in turn may be divided into separate but linked parts that perform authentication, access different kinds of data from the government databases, access different kinds of data from the private company databases, enroll individuals ... etc., etc., etc.

EVERY communication must be designed, agreed upon by parties on both sides of the communication, designed, turned into programming, and tested -- SEPARATELY, so that it can be exercised in all possible ways, including error checking. It's not enough to test it as a whole system because (a) you don't know you are checking everything, (b) if something seems to work, you don't know if it's because the parts worked correctly or if it was a lucky accident that errors cancelled -this one time-, and (c) if a problem -is- evident at the user interface, you don't know where to look for it in a system composed of (tens of) thousands of databases, computers, and communications between them, with many of them owned by different agencies and companies, whom nobody has the authority to command to give the resources to fix the problem.

-Could- such a system be done in three years? Maybe, but not under government contracting rules, and not in the way it was meant to work, pushing the user through all the subsidy questions before showing them the plans.

If this is incompetence, it is wanton incompetence.
26 weeks ago
26 weeks ago Link To Comment
600 million? Back in '75 when there were still consequences for screwing up IBM employee Fred Brooks analyzed how he managed to get fired from Big Blue for failing to deliver an operating system on time. He kept going to the board for additional manpower and the project kept slowing down. As he learned "adding manpower to a late software project makes it later." The core of the problem is that the more people on the project, the more time everyone has to spend communicating about what they are doing through meetings, documentation etc, than actually doing coding. Commenter David H Dennis above is right on to it. So with $600 mill to thrown at the project it is both trivial and inevitable to create a prodigious virtual train wreck. Praise the Lord and pass the cash!
27 weeks ago
27 weeks ago Link To Comment
I would be a little dubious about load being the real problem here. After all, the site's pages always have loaded quickly for me. They have loaded with mistakes and error messages, but they have loaded quickly.

I agree, though, that the most important problem was that managment was constantly changing the specification and rewriting the rules, virtually up to the time of launch. No IT project can ever be successful if managed that way.
27 weeks ago
27 weeks ago Link To Comment
Charlie, I've been doing big servers with complex applications for decades. I am awestruck by the developers' ability to spend >$600,000,000 on this thing.

They may be lousy designers and coders, but wow, they're good at sucking in the money.

Seriously, I cannot imagine how that much money could be spent on an IT project like this. I could see ten or twenty million, but six hundred??!!??

Do you sources have comments on that?
27 weeks ago
27 weeks ago Link To Comment
Well, let's think. There appear to be dozens of government agencies involved. This means endless five-hour meetings where nothing is discussed and everything is put off until next time. I can imagine the mass of meetings and attempts at coordination giving developers little time to code. This can chomp up time and money like crazy, with no meaningful results produced.

That might be a start :).

D
27 weeks ago
27 weeks ago Link To Comment
How did these guys get the contract? What you describe are beginner mistakes (e.g. resource size and count), not something a qualified/experienced outfit would produce.

Edit: Wow, there's an edit function on this website now :-)
27 weeks ago
27 weeks ago Link To Comment
Hey boss, I got this idea, still back of the envelope but it might be worth pursuing...

Great! I'll bump it up the food chain, let's go live tomorrow!

Never again will I open my yap until it's in gold.
27 weeks ago
27 weeks ago Link To Comment
within a couple months of the rollout

Surely you jest, I'm sure it was within minutes of rollout, as well as during and after rollout.
28 weeks ago
28 weeks ago Link To Comment
Probably fair, but I'd bet from the state of the site that *someone* had a moment of clarity and said "no more changes". It was just too late.
27 weeks ago
27 weeks ago Link To Comment
Clown Car website matches the clown car presidency perfectly.
28 weeks ago
28 weeks ago Link To Comment
View All