I was blown away by Gawande’s article, which reminded me of a terrific book on the making of BONFIRE OF THE VANITIES called THE DEVIL’S CANDY: THE BONFIRE OF THE VANITIES GOES TO HOLLYWOOD by Julie Salamon. The book was researched and written while the film was being shot, which is to say long before anyone involved had the slightest idea the film would be not only bad but walk-out-in-the-middle bad.

Salamon’s book was almost a life-altering event for me, because it’s filled with enormously talented, creative, hard-working, experienced, dedicated individuals, all working reasonably well together, nothing noticeably dysfunctional . . . and all of that talent put together produces a catastrophic failure.

I remember finishing the book and thinking, ‘OK, if these folks can’t put together a decent piece of creative work, what hope is there for me?’

Gawande’s article is the same theme: talented, highly educated, experienced, professional medical personnel bringing all of their talents to bear on a problem . . . and still turning in a performance that’s mediocre compared to the folks out at the tip of the bell curve, no better.

Is it the case that all random populations conform to a bell curve?

If so, is it the case that the curve could be tightly distributed?

http://www.amazon.com/exec/obidos/tg/detail/-/0385308248/qid=1102539083/sr=8-2/ref=sr_8_xs_ap_i2_xgl14/103-2139800-6987832?v=glance&s=books&n=507846

In essence, Gawande constructs a crude, univariate analysis (skill is correlated with outcome) that fails to consider numerous confounders present in the patient population (as would be required for his theory were it to be published in a medical journal).

It’s basically a trumped-up hunch buttressed by some dubious statistics. I agree with Soxblog; it’s journalistic hackery of the worst kind.

Who is better skilled, a doctor who performs 10 easy operations with 100% success or a doctor who performs 10 challenging ones with 50% success? By publishing success rates you automatically encourage all doctors to become the former.

“that given a distribution with a mean μ and variance σ2, the sampling distribution of the mean approaches a normal distribution with a mean (μ) and a variance σ2/N as N, the sample size, increases.”

What I said. My gender, for instance, results from a single defective X chromosome that was unaccountably replaced by a Y. Gender doesn’t follow a normal distribution. However, the number of men and women in a large random sample will tend to fall into a normal distribution, as the count is the sum of many small defects in the X chromosome

Within many medical conditions, there are differing degrees of severity which we can anticipate by analysis of a sufferer’s particular cellular or genetic mutation. Sox blog says – and I don’t know if it’s true, but let’s assume it is- that there are 1,000 genetic differences across 30,000 CF patients. That means no 200-patient CF program (which soxblog says is a representative size) begins to have a representative sample of the whole. Gawande’s article cites four children from a single family in one program. In this case, a very rare and/or severe mutation may skew the reults from that program. Small sample sizes with large variation demand tremdous care in analysis. We need to know the range of disease severities, the frequency of various mutations, their related life expectancies, and relative geographical distribution. Mid-numbingly boring, I know, but otherwise we can’t really know whether a program is outperforming the prognosis.

Another thing missing in the New Yorker article is an analysis of the theraputic technologies available. They may be equally prevalent in all programs, although it seems doubtful. Still, if we’ve not considered the likelihood of a long-lived patient outcome by type of mutuation, and we haven’t made sure that all programs’ patients had equal access to the same tools for treatment, have we eliminated enough variables to begin judging the doctors’ “skills”, an even less objectively measurable quality?

I think soxblog’s questions about the findings here are well-founded. It may be that because Gawande’s main argument is about a bell curve among physican skill, he did all these preliminaries for us before he wrote the article, but I always prefer it when people “show their work.”

Benchmarking and modelling are good, universal methods for improvement, but it’s vital to be scrupulous with the baseline assessments, especially when the information is of such a consequential nature to people whose careers may not be in science or statistics.

The Central Limit Theorem states:

“that given a distribution with a mean μ and variance σ2, the sampling distribution of the mean approaches a normal distribution with a mean (μ) and a variance σ2/N as N, the sample size, increases.”

and for those who like mathematics:

http://mathworld.wolfram.com/CentralLimitTheorem.html

However, it is likely that although the skill level of Physicians who treat CF [or other conditions] is normally distributed, the effectiveness of the skill level is not. Many human processes with normally distributed skill levels do not result in normally distributed effectiveness. There is something called the 20% rule where the top 20% account for 90% of the results. You see this a lot in sports and warfare. I don’t see why it wouldn’t apply to medicine.

Random doesn’t imply a normal distribution. The normal distribution arises from the addition of small changes that are equally likely to be positive of negative. These small changes laid end to end define a scale, roughly speaking. There is a sort of Euclidean property. So normality depends on choosing the correct scale and on the fact that the measured effect is determined by the accumulation of many small defects.

I had exactly the same reaction to Soxblog’s writeup. He doesn’t seem to understand basic statistics. Of course physician’s skills are “random” and very likely normally distributed. One thing to keep in mind though is that the standard deviation of the distribution might be pretty tight. When people discuss statistical data they often argue over the mean when what they should really be discussing is the standard deviation (dispersion).