When Is a Climate Model 'Useful'?

Recently, a discussion on Twitter brought this aphorism from statistician George Box to my attention:

“All models are wrong but some are useful.”

Like all good aphorisms, it’s stating an essential truth that bears closer examination and deeper thought but also risks becoming a slogan that excuses deeper thought.

This misuse of this aphorism is particularly pernicious. On the one hand, people can focus on “always wrong” to dismiss all models. On the other hand, it’s sometimes used to excuse bad models as being useful.

What is a model?

So let’s think about this idea of models. First of all, of course, this isn’t talking about Cindy Crawford and Twiggy, although I’m sure pictures of either one would help my search results immensely. We’re talking about models as they’re used in a scientific or statistical sense.

Apple’s built-in dictionary gives as its third definition:

a simplified description, especially a mathematical one, of a system or process, to assist calculations and predictions: “a statistical model used for predicting the survival rates of endangered species.” [Emphasis my own, for clarity.]

The essential point here is that a model is a simplified (and for our purposes always mathematical) description of a system or process to assist calculations or predictions.

A model, in other words, is a formal mechanism to state a hypothesis about some collection of observations. It’s stated mathematically, or at least capable of being stated mathematically.

Science is always stated as a model, but then all knowledge is. You know that if a cat jumps from the bookshelf, the cat will fall; although you didn’t state it mathematically when you first observed this as a toddler, it’s certainly capable of being stated mathematically. Newton did, apocryphally when he saw an apple fall from a tree, but actually after trying to find an explanation of observations by Galileo, Tycho Brahe, and Kepler. And he had to invent differential calculus to do it.

The thing is, Newton’s model is wrong: if he’d had the right measuring instruments he could have seen that the apple didn’t fall exactly as fast as v=at would predict, because of air resistance that he didn’t (couldn’t) take into account. Three hundred some years later, Einstein’s General Theory of Relativity predicted that Newton was slightly wrong in a way that it took careful observations by Eddington to confirm.

Newton’s theory of gravitation was wrong, but that didn’t keep it from being useful for everything from computing the trajectory of cannon balls to the arc Luigi takes in Mario Brothers.

What makes Newton’s model useful is that it’s predictive: Newton had a hypothesis about the behavior of bodies under the influence of gravity, and even though Newton had no idea what gravity was (honestly we still don’t), his hypothesis was born out by innumerable successful predictions

Are models really always wrong?

Yes, at least in the sense we can never prove them to be entirely right. Newton’s model falls down in the face of better measurement, and science can’t explain Jesus with the loaves and fishes or the Buddha making the Lankavatara.

There’s another, mathematically deeper, sense in which some models are guaranteed to always be wrong. It was really first discovered by Poincaré but really not fleshed out until the 1970s by Lorentz and others. Some models are chaotic, which comes down to the discovery that some physical systems are not really, precisely, computable. These systems are said to be “sensitively dependent on initial conditions,” something that’s sometimes called “the butterfly effect” — the notion that a butterfly flapping its wings in Rio can cause a tornado in Kansas or a rainstorm in Rome.

When a system is sensitively dependent — no, I’m not going to write the entire phrase out every time — it means that tiny differences, even differences that are too tiny to measure, can accumulate over time to give big differences in the result.

So what makes a model “useful”?

In science, at least, a model is useful to the extent that it is predictive. Newton’s Laws do a really good job of predicting the motion of any freely falling object, whether it’s a line drive to left, a 16-inch shell, or the motion of New Horizons as it approached Ultima Thule. Even when they’re wrong, when they’re superseded by the better model of General Relativity, they’re really close to right in normal situations.

The original Twitter discussion was launched when Meet the Press announced that they were going to spend a whole hour on CLIMATE CHANGE OMG and that no “deniers” would be admitted. I discussed this a few days ago, so I won’t repeat it, but the underlying point is that all the climate change discussion is based on climate models.

Now, climate modeling is hard. I mean really hard. The sort of hard that consumes days of computation on the biggest supercomputers around.

There are actually lots of models in use, and each one represents a particular set of hypotheses about the way the climate and energy balance of Earth actually works.

A number of them, variations of the same basic hypotheses, all predict certain behavior of the climate that’s usually condensed down to the Global Average Surface Temperature (GAST) and climate sensitivity to CO2. Climate sensitivity is basically an estimate of how much GAST will increase for a certain amount of increase in the concentration of CO2. Those models predict certain things: the increase in temperature, and changes like increased melting of polar ice and increased storm intensity.

All of the standard models make similar predictions, and they all have similar problems, which come down to this: they don’t predict the actual observed changes in actual temperature.

Nor do some of the other predictions hold true. (For example, there’s no observable trend in actual storm damage once you take into account inflation and the trend to build bigger and more expensive buildings along the coasts. This is the observation that got Roger Pielke Jr. investigated by Congress.)

So, on the one hand, we see people who declaim “Oh, that’s just a model” and discount everything. We see a few people, like Richard Lindzen, Roy Spencer, and Nicholas Lewis and Judith Curry, who look at the data first and model later. Lewis and Curry, in particular, have a competing model which actually predicts observed temperatures pretty closely.

A third group instead are trying to save the existing models. In computers, we’d say they’re trying to “patch” the models. There are several different approaches to this: adjustments to the raw temperature data, a hypothetical sink of excess heat in the deep ocean, and so on.

Or, of course, the other favored tactic: force. Roger Pielke Sr. simply had his contrary results rewritten without his permission, which drove him to resign from the IPCC in protest in 1995. Judith Curry eventually left her tenured appointment, after being excommunicated by the climate clerisy. Pielke Jr. was driven from a column at 548.com, was investigated by Congress, and eventually was largely driven from the whole climate debate — and is still the target of a campaign to declare his heretical views dangerous scientific misconduct.

And of course, now we see the example of Meet the Press, where no dissent is even tolerated.

That’s the other pole of Box’s quote. “All models are wrong, but some are useful.” In science, a model is useful when it’s predictive. What makes a model useful in politics may be something entirely different.