Climategate Computer Codes Are the Real Story
So far, most of the Climategate attention has been on the emails in the data dump of November 19 (see here, here, and here), but the emails are only about 5 percent of the total. What does examining the other 95 percent tell us?
Here’s the short answer: it tells us that something went very wrong in the data management at the Climatic Research Unit.
We start with a file called “HARRY_READ_ME.txt.” This is a file containing notes of someone’s three-year effort to try to turn a pile of existing code and data into something useful. Who is Harry, you ask? Clearly, a skilled programmer with some expertise in data reduction, statistics, and climate science. Beyond that I won’t go. I’ve seen sites attributing this file to an identifiable person, but I don’t have any corroboration, and frankly the person who wrote these years of notes has suffered enough.
The story the file tells is of a programmer who started off with a collection of code and data — and the need to be able to replicate some results. The first entry:
1. Two main filesystems relevant to the work:
/cru/dpe1a/f014
/cru/tyn1/f014
Both systems copied in their entirety to /cru/cruts/
Nearly 11,000 files! And about a dozen assorted “read me” files addressing individual issues, the most useful being:
fromdpe1a/data/stnmon/doc/oldmethod/f90_READ_ME.txt
fromdpe1a/code/linux/cruts/_READ_ME.txt
fromdpe1a/code/idl/pro/README_GRIDDING.txt
(yes, they all have different name formats, and yes, one does begin ‘_’!)
Believe it or not, this tells us quite a bit. “Harry” is starting off with two large collections of data on a UNIX or UNIX-like system (forward slashes, the word “filesystem”) and only knows very generally what the data might be. He has copied it from where it was to a new location and started to work on it. Almost immediately, he notices a problem:
6. Temporarily abandoned 5., getting closer but there’s always another problem to be evaded. Instead, will try using rawtogrim.f90 to convert straight to GRIM. This will include non-land cells but for comparison purposes that shouldn’t be a big problem … [edit] noo, that’s not gonna work either, it asks for a “template grim filepath,” no idea what it wants (as usual) and a serach for files with “grim” or “template” in them does not bear useful fruit. As per usual. Giving up on this approach altogether.
Things aren’t going well. Harry is trying to reconstruct results that someone else obtained, using their files but without their help.
8. Had a hunt and found an identically-named temperature database file which did include normals lines at the start of every station. How handy — naming two different files with exactly the same name and relying on their location to differentiate! Aaarrgghh!! Re-ran anomdtb:
Okay, this isn’t so unusual, actually, but unless you document and describe your file structure, it’s pretty much opaque to a new reader. Still, Harry presses on:
11. Decided to concentrate on Norwich. Tim M uses Norwich as the example on the website, so we know it’s at (363,286). Wrote a prog to extract the relevant 1961-1970 series from the published output, the generated .glo files, and the published climatology. Prog is norwichtest.for. Prog also creates anomalies from the published data, and raw data from the generated .glo data. Then Matlab prog plotnorwich.m plots the data to allow comparisons. First result: works perfectly, except that the .glo data is all zeros. This means I still don’t understand the structure of the .glo files. Argh!
Comment submitted by Jim Anderson, anderson.james@att.net
While the geniuses were debating the data they derived from boring holes into trees in Russia and other places – yet another letter was exchanged.
For those who wonder whether our world is threatened by global warming – perhaps the answer turns on whether or not a reindeer relieved his bowels in the ‘wrong’ place.
The e-mail below – besides being from the collection posted on line, and being one showing the writers to be rather vindictive and mis-leading folks – also shows our fate may turn on reindeer poop.
E-mail follows – and I have put into capital letters the relevant portion of the e-mail:
**************************
Original Filename: 1136918726.txt | Return to the index page | Permalink |
Earlier Emails | Later Emails
From: Tom Wigley
To: Keith Briffa
Subject: Re: Nature: Review of manuscript 2xxx xxxx xxxx
Date: Tue, 10 Jan 2006 13:45:xxx xxxx xxxx
Keith,
Thanx for this. Interesting. However, I do not think your
response is very good. Further, there are grammatical and
text errors, and (shocking!!) you have spelled McKitrick
wrong. This is a sure way to piss them off.
They claim that three cores do not cross-date for TRW.
They also say (without results) that the same applies to MXD
(these results may be in their Supp. Mat. — I presume you
checked this).
So, all you need say is …
(1) TRW was not the only data used for cross-dating.
(2) When MXD is used there are clear t-value peaks,
contrary to their claim. You can show your Fig. 4 to prove
this.
(3) The 3-core-composite cross-dates with other (well-dated)
chronologies (Yamal and Polurula), confirming the MXD-based
dating. You can show your Fig. 5 to prove this.
You could say all this in very few words — not many more than
I have used above. As it is, your verbosity will leave any reader
lost.
THERE ARE SOME PROBLEMS STILL. I NOTE THAT 1032 IS NOT COLD IN YAMAL.
SEEMS ODD. IS IT COLD IN *ALL* OF THE THREE CHRONOLOGIES AT ISSUE?
OR DID A REINDEER CRAP NEXT TO ONE OF THE TREES?
Also, there seems to be a one-year offset in the 1020s in your
Fig. 6.
I hope this is useful. I really think you have to do (and can do) a
better job in combatting the two Ms. If this stuff gets into Nature,
you still have a chance to improve it. Personally, I think it would
be good for it to appear since, with an improved response, you can
make MM look like ignorant idiots.
Tom.
When one’s institute is maintaining the database and code used to inform political debate as well as be the basis for many, many other researchers this excuse just doesn’t cut it.
We have to use ISO 9003 standards for FDA work and I would argue that this work is every bit as important as a fragment of image processing software running on some CT workstation.
Every paper from CRU and every paper that depends upon CRU datasets/models is now invalid until the research can be replicated. This includes the IPCC reports btw. Furthermore, any paper that cites these papers should be reviewed by its authors to see if their conclusions were invalidated because of dependence upon this scientific misconduct.
NASA may well be as screwed up as CRU. They have resisted for years releasing their data and code. In the face of this corruption of science, there can be no reason that ANY publicly funded dataset or model is kept secret.
Furthermore, the datasets and models used for environmental science in particular should be put into the open source community for review and comment. If this piece of junk that that poor b@sta@rd had to deal with had been available to thousands of skilled engineers and scientists he might have not wasted 3 years of his life attempting to resurrect this travesty.
So, with millions of dollars of taxpayer money in their pockets – traveling to exotic locales for seminars and conferences – eating steak and lobster- and with trillions of dollars on the line globally and with the economy of every nation on earth on the line… they can’t make their own data work no matter how they go about it???
Why do those people still have jobs? Why are they still getting my money?
When one’s institute is maintaining the database and code used to inform political debate as well as be the basis for many, many other researchers this excuse just doesn’t cut it.
Don’t disagree with you for a second. Especially having spent time trying to straighten out data in much the way poor Harry did.
Warren, the bright point about this whole mess is I think it’s very probable they won’t continue having jobs. See what William Briggs predicts.
This sort of mess is apparently fairly common in graduate departments (I assume CRU is such), as students do what is necessary for their work, then move on.
That doesn’t excuse CRU, however, because of the use that is made of the data. The fact that nobody else has the data, and CRU doesn’t have it in a usable form (and may never be able to decipher it) means that, for scientific purposes, the data doesn’t exist.
Any papers based on this mess should be retracted.
— hell, practically water-boarding the data —
Great, Chaco. Thanks for digging into this story.
ACORN can fix the data and take care of those illegal breaches into the schools servers.
Breaking the Law And Lying to the World ?
Who you gonna call ?
ACORN / SEIU…..The Problem Solvers
Nor are they engineering project managers. A PM would have realized that this was a show stopper, and halted everything until this could have been straightened out.
But you don’t have to read far into the emails to realize that Jones frankly didn’t care whether his science was right. Even if this programming fubar hadn’t have happened due to personnel issues, he (and his American counterpart Mann) simply was a man of low ethics. When you have low ethical standards, sooner or later things spin out of control.
To paraphrase a post I made elsewhere, the code shows that their “models” are not models at all in the sense that physical scientists use that
word, i.e., simulations of physical processes. They are extrapolations of historical data taken from a variety of (in many cases) diverse sources that are then statistically manipulated and tweaked. That is why the programmer is having so much trouble, i.e., there are no physical laws
to refer back to, e.g., there is no way there could be an “energy budget balance” problem if they were doing physical modeling.
“Lies, damned lies, and statistics”
AGW and the Global Cooling,oops I mean,Warming, oops I mean,Cooling, etc. has the same problem most politically motivated people have. If you want your report or survey or study to come up with an answer before you even start, you sometimes have to just make sh*t up.
People have an effect on the planet, no doubt. Let’s don’t pollute if can can avoid it. Let’s find newer cleaner sources of energy….but LET’S NOT place our faith and futures in the hands of socialized and politicized scientists that have been less than honest about their theories than they would have us believe.
I hope these emails and data knock the AGW people off the rails. But, I hope we, as a group, continue to try and be as careful as we can with our world.
then statistically manipulated and tweaked
I have the impression that their statistical toolkit is rather limited, mostly PCR and PCA, i.e., fancy dress cousins of least squares polynomial fits. Then they throw out lots of ‘outliers’ and such. This makes extrapolation rather silly, because what they have isn’t a model in any normal sense of the word. And making an error estimate for the result can be rather tricky because of the ad hoc decisions about which outliers to eliminate. The divergence “problem” isn’t really a problem, it’s merely an indication that the idea didn’t pan out and it’s time to find another thesis topic. But it’s a problem for these guys because they can’t think of anything else to do.
That’s my impression, anyway. I am not, and never have been, a climate scientist.
The CRU emails show the activities of the small clique of researchers at the heart of AGW research. Here is an excerpt from Wegman’s 2006 social network analysis of temperature reconstruction, annotated with links to the searchable archive of CRU emails:
http://limitedmodifiedhangout.blogspot.com/2009/11/ad-hoc-committee-report-on-hockey-stick.html
One of the interesting questions associated with the ‘hockey stick controversy’ are the relationships among the authors and consequently how confident one can be in the peer review process. In particular, if there is a tight relationship among the authors and there are not a large number of individuals engaged in a particular topic area, then one may suspect that the peer review process does not fully vet papers before they are published. Indeed, a common practice among associate editors for scholarly journals is to look in the list of references for a submitted paper to see who else is writing in a given area and thus who might legitimately be called on to provide knowledgeable peer review. Of course, if a given discipline area is small and the authors in the area are tightly coupled, then this process is likely to turn up very sympathetic referees. These referees may have coauthored other papers with a given author. They may believe they know that author’s other writings well enough that errors can continue to propagate and indeed be reinforced….
The most damning line: “Each parameter has a tortuous history of manual and semi-automated interventions that I simply cannot just go back to early versions and run the update prog.”
Their data was run with manual interventions to provide the results they wanted! There can be no other interpretation of this sentence.
As a former analyst and programmer, if extracted data used does not produce the desired results, as in accounting, a 999,999,999 card can be used to balance a worksheet and account for missing or incomplete input.
The extent of the lies and deceit exposed by the AGW believers is astounding but not unexpected. We are on the brink of another major financial disaster and if our leaders in Washington have any sense of decency, this madness will stop now. However, after witnessing the psychosis permeating our elected representatives once they become entrenched, hope for sanity and common sense is fleeting.
We have a cluster of barn owls who truly believe they possess the intelligence to guide us into the future when all along, all they produce is owl crap and take flight when confronted with truth.
There is absolutely no reason for anyone in a position of authority to have a moment’s regret or delay in PRYING these money-sucking leeches out of whatever lab they’ve stashed themselves in, take their keys and credit cards, give them $100 pocket money and a $50 gift card for Walmart, change the locks and tell them to get lost. Permanently.
11. Desert Dog:
People have an effect on the planet, no doubt. Let’s don’t pollute if can can avoid it. Let’s find newer cleaner sources of energy….but LET’S NOT place our faith and futures in the hands of socialized and politicized scientists that have been less than honest about their theories than they would have us believe.
___________________________________________________
Sir you are exactly right!
I just wrote an article about this on Searchwarp…a writers blog:
http://searchwarp.com/swa557332-Meeting-Energy-Needs-For-The-Future.htm
So have I got this right?
1) They haven’t shown their data. ( at least pre compiled )
2) They can’t replicate their work to show how they compiled what they do have.
So where is the science here?
This whole thing provides me with scads of entertainment. If you don’t have the entire set of documents, get them. You don’t need to be a climate scientist to know the smell of poo. AGW jumped the shark a long time ago.
If these folks had any *real* intelligence, they would not record anything against self-interest in e-mails, they would not have open discussions about their lousy programming skills, and they would not have been so d@mn cocksure that such arrogant stridency wouldn’t be the shoelace that leads them to trip up (or rather have a 3rd party tie their laces tied together).
They became zealots in a religion where they were the prophets, where they were assured of a decent living through grant money and speaking engagements, and where their questioners were ruthlessly skewered to their great amusement.
Make ‘em walk the plank!
Also, I find it mildly amusing that the New York Times, publisher of the Pentagon Papers, and about two dozen other classified documents over the past three-plus decades, suddenly stands tall as the protector of the Climate-gate e-mails. To paraphrase, the e-mails were not intended for public eyes.
Time to shut down the Times. They aren’t too big to fail.
Now, I’m going to curl up and get warm. Where’s AGW when you need it?
Apropos #13, it well known among authors of scientific papers that many editors choose the reviewers from among the references. Thus, it is possible for the authors to, in some sense, pre-select the possible reviewers by careful tailoring of the references. This coupled with the lack of blind reviewing seriously calls into question the scientific peer review process in many cases.
amazing. as big a story as this is, the media puts their head in the sand and doesn’t report it. secondly, sites like Charles Johnson Little Green Footballs treat it as a non-story-since they believe the global warming narrative. LGF goes so far as to defend the hypocrisy of the scientists!
My favorite line from your article:
“They clearly have some history of massaging the data — hell, practically water-boarding the data — to get it to fit their other results.”
Bravo! And I think you’re exactly right about the data-corruption being the most damaging aspect of the ClimateGate scandal.
The entire “climate modeling” racket was built on corrupted data, corrupted programs, and corrupted researchers. And upon climate modeling stands the whole Climate Catastrophe Industry: a multi-billion-dollar, pseudo-scientific house of cards.
I wish Michael Crichton were still alive to witness and comment on this. Talk about a “See, I told you so.”
most people knuckle under from the peer pressure. that is how these global warming kooks threatened and bullied anyone who questioned the man made global warming religion.
just a bunch of whining bullies liars and cheats.
I would love to see them jailed and fined for the damage they have caused, but I don’t believe in magic and they may still get away with their crimes.
…and worst of all I don’t think this religion will be going away any time soon.
The previous revelations plus reading through “Harry’s” emails gives one the eerie feeling of being a front-bench witness to the greatest slow-motion train wreck in history.
Initially I was cautious. The original email files looked damning on the surface but as many noted, some of the phraseology could be taken several ways depending on whether one was supportive or pejorative. But the analysis now coming out of the data sets and program files seems to confirm the worst. Now the data/program files confirm the real intent of the emails and the emails confirm the fraudulent intent of the data manipulation; they cross-confirm each other.
Scientists assemble data, do their analysis, get the results and then go where the results take them. These crooks assembled the data, did the analysis, and then hacked it all to make the results go where they wanted them to. –Every last jot and titter of it by the way it’s beginning to look.
Wow. Just WOW !
15 Bart: Sorry buddy. Our leaders in Washington have no sense of decency. Stronger measures will be required.
This is absolutely a fact. Given when the code was developed and the timeframe, and my knowledge of how this community works, (let alone reading the travails in the email string)
If policy is going to be dicgtated from these’models’ then they should be revealed reviewed, and ultimately debugged
I went to check the growth rings in those crucial Siberian trees, but the trees were gone. Some high flautin Englishman had cut them into fire wood and set them alight, saying the atmosphere needed some warming.
The emails only showed intent, and they can be explained away by a clever lawyer. The models and the comments inside the model show the actual fraud, preserved for posterity.
More detail on this shameful fraud and two graphs that show reality vs. the “model” built by these “scientists”.
http://beforeitsnews.com/story/0000000000000590
In engineering school we were taught “models are always wrong, some are useful”. This model isn’t useful.
Bugs or no bugs, why does it matter? As long as the “scientists” could scam multimillions, and billions. A functional pgm would never produce the desired results. A programmer who wrote a correct program would have lost his job a long time ago.
Bush politicized science to deny the reality of man made global warming.
Al Gore, the soon to be billionaire, makes his billions on global warming. Should he give his Nobel back, or donate the prize money to charity, such as ACORN?
Confession: I have been breathing out C02 the entire time I was reading this.
Name withheld
Welcome to The Matrix.
They should have told Harry the programmer to turn the data upside down for better results. That’s what “Piltdown Mann” apparently did in a paper he published in 2008. I kid you not. See this link:
http://wattsupwiththat.com/2009/10/17/iq-test-which-of-these-is-not-upside-down/
So familiar. Like the $87 million website that tracks tens of thousands of jobs created in districts 00.
Check out the document RulesOfTheGame.pdf that’s in the root directory.
Despite all the revelations of the SCAM Al Bee Bee Ceera ( BBC) is STILL unashamedly punting Global Warming SCARE stories. This is from the BBC site today 25/11/09
“Warming globe
The past, present and possible future of climate change ”
Here is the link to the stories with all the Graphs based on what we now know is MANIPULATED information.
http://news.bbc.co.uk/2/hi/science/nature/8359629.stm
[why does the sum-of-squares parameter OpTotSq go negative?!!]
Because, obviously, the climate data are imaginary.
Neko-san, when I worked as an editor for reviewed journal, I looked for honest people outside the discipline for exactly that reason: bias. Didn’t want it, otherwise what was the use of publishing? Then I attended a technical journal conference and learned that the built-in biases and blinders do intentionally exist in some disciplines. Sad but true.
Yeah, but they are scientists, or at least they’re supposed to be. And scientists have to understand the data they’re using. They may not understand programming very well, they may not know how to use VCS or buddy testing or unit tests or code reviews or inline comments or any other common practices for computer engineering. But they ought to know enough to document their data – HARRY_READ_ME.txt shows that not only did nobody know how the software worked, they’d lost the metadata about a bunch of their data files. Poor Harry was reduced to guessing which columns might be longitude, and not able to find latitude at all.
(hmm, I’m no climatologist, but I have a sneaking suspicion that latitude might be kind of significant to temperature data sets. But then I haven’t had any papers published in the
clique-peer-reviewed press, so what do I know)My advice to the British Crown would be to shut down this hack of an institution, close the doors to CRU, fire the entire staff, delete all of the data, demolish the buildings, and forget it ever existed. Hire some real scientists to start a real research center to do real research with real data. If they start now, in ten years they might have something useful to report.
Charlie,
Another outstanding post. Thanks for the hard work.
Allow me to add what I think may be another piece to the puzzle.
Over at the BBC Science and Environment webpage, Roger Harrabin, the BBC’s Environment Analyst (whatever the heck that is) has had to address the CRU hacked documents. Harrabin does not allow readers to comment, and in the story he provides ample opportunity for the CRU proponents of AGW to state their case, and to essentialy claim this is much ado about nothing. Specifically, allow me to quote this bit which Harrabin tells us comes from an inside source of his at the CRU:
“My CRU source points out that its unpublished full data set is almost identical to the ones at the National Climatic Data Center and the Goddard Institute of Space Studies.
Both of these are in the US, where there are no restrictions on publication. The CRU view is that when the sceptics see the full data in due course they will be very disappointed.”
The BBC’s Harrabin concludes by mentioning that about AGW, “One leading figure told me unofficially that confidence was now at 99%.”
I think that’s the data you and “Harry” are agonizingly trying to analyze in this post Charlie.
Then I read this excellent breaking story by Clarice Feldman in The American Thinker, CRU E-Mails Spur Lawsuit.
Linking to Chris Horner writing for the American Spectator we get this from Clarice’s post:
“Today, on behalf of the Competitive Enterprise Institute, I filed three Notices of Intent to File Suit against NASA and its Goddard Institute for Space Studies (GISS), for those bodies’ refusal – for nearly three years – to provide documents requested under the Freedom of Information Act.”
Interesting.
On that side of the pond CRU is saying the data of the Goddard Institute of Space Studies, (GISS) is aparantly open and transparant for publication, while on this side of the pond that very same (GISS) is being sued for 3 years worth of hiding it’s data from FOI requests.
Now I do not know specifically what information is being demanded of NASA and the GISS in this Lawsuit, but wouldn’t it be interesting if it’s essentially the exact same data hacked from CRU.
If so, then the supposedly open to publication sources on our side of the pond are precisely what the CRU are using in Britain to attempt to quiet the firestorm and justify the legitimacy of their research over there. Would love it if it’s exactly the stuff that Clarice’s column here addresses as being the data that has been denied review by independent researchers in America.
If so, this would appear to me to be science scholarship done on the level of Ward Churchill.
Simply post bogus research, (hidden from outsiders) on both sides of the pond, then refer to the publication of that bogus research across the pond as validation for your own bogus research. Keep it all secret from independent researchers worldwide, and then have the BBC Environmental lapdog Reporters, who disallow comments BTW, pronounce that their ‘in the know, top level sources’ swear that “it’s 99% absolutely true,” so On to Copenhagen!
Keep up the outstanding work.
I’m glad to hear people saying, now that this code has leaked, that the conclusions drawn are not valid unless confirmed by outside work.
Why on earth was that not the case all along? What sort of “science” has been going on, and why the hell has my tax money been used to fund it?
Calvin Ball wrote: “When you have low ethical standards, sooner or later things spin out of control.” Ten for ten!! Or, as his baby sitter said, ” Calvin, I think you’re just making this up a you go along.” “Hobbes! She’s stumbled into the perimeter of wisdom! Run!!”
A common joke between myself and other engineers is ‘hammer to fit’. You can’t make up stuff this good.
ArtD0dger, you’re my hero. That was hilarious!
JMH wrote
My advice to the British Crown would be to shut down this hack of an institution, close the doors to CRU, fire the entire staff, delete all of the data, demolish the buildings, and forget it ever existed.
I agree except for the following minimal changes
My advice to the British Crown would be to shut down this hack of an institution, close the doors to CRU,
fire the entire staff, shoot the entire staff as a warning to traitors, delete all of the data,demolish the buildingsdemolish the buildings and seed the gound with salt, and forget it ever existed.The quality of their data and code might make more sense if one considers it in the context of a business instead of science. Need to get the product out, there are TTM pressures you know. Get that revenue cycle started, and we’ll fix it in production!
As an engineer it was obvious to me from the start that a system as complex as the climate of a planet was not something that could be modeled with any accuracy. Our experience is too limited, our time scale too short and knowledge too feeble. Indeed, one of the first things you are taught in engineering schools is that you cannot directly solve for temperatures in open or highly complex systems. Belief in AGW is almost non-existent amongst engineers for this very reason.
What I did not realize is that no attempt at modeling had even been attempted. Apparently all they did were statistical projections of trends in observed temperature proxies. There were no physical laws included whatsoever. This is in and of itself enough to invalidate their claims of “modeling” before we even get to the fact that their data is worthless and they can’t even reproduce their own results.
What they did was the equivalent of putting a wind gauge next to a highway as a means of guessing how fast the cars were traveling. Then taking that proxy data and extrapolating from there in whatever implausible way was necessary to generate what they wanted to see. But alas for them, one day the wind stopped blowing and their little cars stopped moving… thus proving their approach was worthless from the start.
We are lead by sub-moronic, scientifically illiterate people such as Nancy (natural gas isn’t a fossil fuel) Pelosi and Al (the center of the earth is millions of degrees) Gore and we act like they have some right to use force to restructure every aspect of our economy and our existence just by virtue of the positions they hold. Such a thing cannot help but end badly if they are allowed to have their way. The Constitution was written to limit the power of exactly this these sorts of people.
I don’t think it is much of an exaggeration to say that whoever released this information may have saved more blood and treasure than anyone else who has yet lived in this century or perhaps even the last century. Monuments need to be built to this man or woman.
If it doesn’t fit, aquit –
And if the facts don’t fit your theory, adjust them to fit -
In OJ Simpson case, in preparing the final “glove test” trick (OJ suffering of arthritis), the defense team stopped him for weeks to take his medicine -
The result – swollen hand(s) which simply the glove coudn’t accomodate – aquit!
Looks like the global warming scientists studied Cochran’s and Shapiro’s reality adjusting scheme -
#42: Someone got me one of those page a day calendars, where this was referred to as Nelson’s Law of Force: Don’t force it, get a bigger hammer.
Let’s not overlook the even larger problem at CRU. CRU is the repository for much of the world-wide historical temperature database. There are statements in these documents that indicate that these databases have been corrupted, with much of the metadata that describes location of the measurements missing, as though the original archived data files were used for calculations instead of copies of the files. This might be a far more important problem caused by mismanagement at CRU that affects a far wider range of potential scientific investigation.
Can anyone point me to a link to the actual datasets? Or better yet, the original .ZIP archive?
Ever since this story broke, the one thing I have most wanted to do has been to take a look at the data and the code for myself. So if somebody could point me to it, I’ll appreciate all over ya.
My doubts about man-made global warming have increased considerably in the last couple years and these hacked emails simply add to my skepticism. Now to identify the person or group who can speak clearly on the matter. 80% of the messages here slip into rant and bluster before the final period is placed.
Conservatives tend to take the approach that ANY regulation of the environment is bad because it gores the profits of their industrial pals and expands the role of government. Lefties go the other way, sometimes just to stick it to said industries.
In my view, said industries DO need a lot of regulation, but not based on tree rings from Siberia and unfathomable models, but rather clearly discernible effects on our land, water, and air. There will always be strong differences on the relative value of jobs vs spotted owls and snail-darters, and plenty of accompanying bluster, but I crave the clear-eyed centrist who can sort through this stuff.
“If you torture data sufficiently, it will confess to almost anything.”
-Fred Menger
Re #19 Mhu Cao: “If these folks had any *real* intelligence, they would not record anything against self-interest in e-mails, they would not have open discussions about their lousy programming skills, and they would not have been so d@mn cocksure that such arrogant stridency wouldn’t be the shoelace that leads them to trip up (or rather have a 3rd party tie their laces tied together).”
Good point that’s obviously lost on many. Considering the possibility that these guys aren’t complete morons, you wonder about all of the correspondence where they were smart enough not to put it in writing, or were smart enough to delete after the fact.
My advice to the British Crown would be to shut down this hack of an institution, close the doors to CRU, fire the entire staff, shoot the entire staff as a warning to traitors, delete all of the data, demolish the buildings demolish the buildings and seed the gound with salt, and forget it ever existed.
Strong letter follows.
#50, the zip file should be available via bittorrent. Search for Hadley CRU.
#46, It is not that making a physical model is impossible, but it is a very computationally intensive problem. For example, there are roughly
510M sq. km of surface area and the atmosphere is roughly 120 km high, so a model with 1km cubes (actually upside down truncated pyramids) would have more than 6G cells. And that is just the air!
Since heat transfer with the surface (oceans and land) would have to be also modeled, figure another 6G cells below the surface. Plus land and oceans need to be modeled differently, and the boundary between them is irregular. And, then, what time step does one use?
While that might sound within reach now, how much memory did the average computer have 10 years ago and how fast was the cpu? The problem would have been challenging even for supercomputers then.
To make it fit, one could increase the cell size. But as the cell size increases, that tends to obscure the potential effect of local features (like mountain ranges). The same with increasing the size of the time
step, e.g., on the scale of even one year, there are no hurricanes, tornados or monsoons.
Another way to simplify is to assume some sort of symmetry, but that
becomes increasingly unrealistic for simulating climate (though it works well for other kinds of problems).
Anyhow, not an easy problem, even with today’s computers.
The CRU scandal is the scientific version of the Madoff scandal. In both instances, billions of dollars are/were at risk. The political ramifications of the Madoff scandal favored rapid resolution and prosecution: Unfortunately, that is not the case where AGW is concerned. Furthermore, were see the ability of politics and its largesse to corrupt science.
To Dwight @ 57 Are you familiar with the file labeled “The Rules of the Game”; this looks to be a conference hand-out, pamphlet… It leads with the observation “Changing attitudes towards climate change is not like selling a particular brand of soap-it’s like convincing someone to use soap for the first time.” Is this how you feel?
“Dilbert” is a documentary;
Saw the same sort of thing
in the year before my company
went bankrupt.
Everyone has to realize, we know NOTHING about climate except by anecdote. What we thought we knew, now we can’t be sure.
The data is no good—much of it messy to begin with, and now hopelessly corrupted. Wherever there may be good, uncorrupted data, how can you tell the difference?
The models are no good. They can’t reproduce the past even with spoon-fed data and faked up parameters. One model even gives negative numbers for a sum-squares calculation.
We have to start over. Basic data in its original form, reviewed by honest scientists with forensic auditors by their side; and changes or adjustments discussed fully and transparently. All data subject to chain of custody just like evidence in a legal proceeding.
Models reviewed at the source code level, to ensure they are properly implementing good science. All parameters must be justified. Models, too, must be secured from tampering.
New programs to collect data going forward, including all handling of that data and any adjustments. Not just CO2 or GHGs, that limitation prejudges the result—must include EVERYTHING that could conceivably affect climate.
Eventually, bring the validated data into the validated models, and let’s see what we get and discuss where we have to go to improve the science and the modeling. And only at that point does any talk about public policy come back into play.
I guess 3-5 years if we start NOW. And anyone who objects we can’t waste that time, the threat is too grave and too imminent needs to be reminded that they have to unlearn what they have been told, because we actually now know almost nothing. Give ‘em some thorazine and send them to their rooms to grow up.
Meanwhile, Mann, Jones and most of this crew needs to spend their time defending themselves in court, before administrative panels, and academic discipline boards.
#58 Tom wrote: “Changing attitudes towards climate change is not like selling a particular brand of soap-it’s like convincing someone to use soap for the first time.” Is this how you feel?
————–
What? I don’t follow your context here.
As a software engineer, i am worried by the lack of rigour in the keeping of their software code. If they cannot process their data properly, let alone leave it undocumented and unmaintainable, how can the public accept that their results are valid??
Negative values for the sum of squares? Overflow.
Poor Harry. Feel for ya buddy. Been there, done that.
Just wrote my Congressional delegation about it.
HARRY_READ_ME.txt indicates very serious problems with that data set. If it were just one small study, one small study group, it would be one thing; but these CRU boys are big players.
Simply: Kyoto/Copenhagen and cap/trade are over-leveraged on a data set that appears to be crumbling. Danger ahead, for Democrat or Republican, mathematician or carpenter, doctor, lawyer or Indian chief.
As a trained engineer I can assure you that if you torture the data long enough…it will confess to anything. How can it be that despite the enormous significance of this scandal, so few in the MSM are covering it…at all.
Exactly. What the code comments tell us is that one of the major repositories for climate datasets in the world is hopelessly corrupted. Once those annotations are lost and the datasets become unknown then there is no reason to trust them any longer. And, given the fraudulent behavior of the warming advocates at CRU, no reason to believe the datasets haven’t been tampered with.
Hopefully the datasets can be recreated from raw data stored on tape/card/paper. If it is, the validation and verification process needs to be audited.
We still need to understand climate and climate research is the best way to do that. Next time maybe society can prevent this field of study from being suborned by a collection of unethical fanatics.
@45, for what it’s worth, there are plenty of business domains where you have to provide validated, controlled, audited data (pharmaceuticals or airplane repairs for example). A regulated business with this level of data quality would have been shut down long ago.
Let me repeat that. A business with a data shop like the CRU would not be allowed to sell cornflakes, much less reorganize the entire global economy.
Nothing to see here folks……Move along
You are all Racist, Climatephobes if you don’t accept the “Fake but True” Data
Dan Rather told me so
Someone has pooped on the world’s climate data, making it totally useless. When you think about it, it’s just temperature data files, not nuclear bomb design specs.
How could this happen? On purpose.
In a different thread, I sarcastically suggested that the climate “models” are like programs to predict a roulette wheel based on data about the history of the wheel, i.e., the result of past spins.
In retrospect and without the sarcasm, I think a more correct analogy is a program to predict stock market prices. Stock prices follow a random walk on a small scale but also can follow longer term trends. Right up until the point they don’t because of the influence of unpredictable outside forces (like a sudden liquidity crisis among financial institutions).
Similarly climate follows predictable trends right up until the point it doesn’t, e.g., Roman Warm Period, Dark Ages Cold Period, Medieval Warm Period, Little Ice Age, etc. The interesting question is why the transitions from warmer to colder or colder to warmer occur, but this
depends on little understood physical processes that climate “models” don’t address.
The problem with climate “models” isn’t just that the data has been lost or corrupted, the results not reconstructable, the programs broken, the raw data “adjusted” to fit the conclusion, etc., though those are all certainly problems; the real problem is that there is no scientific basis for the methodology at all.
Let me give you an example from finance to show the contrast. In securities analysis, the CSRP at the University of Chicago is considered the definitive source of data and is available to all researchers for a nominal fee. It is generated from published market prices and easy to verify, although what makes CSRP so good is that it has data on stocks that have been delisted due to mergers, bankruptcies, etc. So it’s really raw data.
Then, any findings generated using the data are easy to duplicate and verify, as papers are expected to provide enough information to make this possible. Also finance research is historical and makes no claims about the future – you as an investor might assume a 10-year, 20-year, or 40-year trend will continue, but that’s up to you. The research certainly doesn’t make those claims.
In contrast, this bogus climate research seems to use a proprietary closed data source, a proprietary closed model, and does make claims about the future. And although I don’t know much about climate science, it seems like reading a historical trend of station temps could be done on a home computer by the average college student – its not rocket science.
Predicting future financial performance involves decision making under uncertainty and predicting climate (like the 7- or 10- day forecasts) is probably similar. The way that’s done is you run a number of scenarios, called Monte Carlo. Its imperfect by definition and you’re wrong more often than you’re right – its really about determining what scenarios are theoretically possible rather than what’s going to happen. Those predictive models are heavily influenced by inputs of course. To really evaluate your model, you need to be able to evaluate and confirm past predictions over hundreds of trials.
Then there is Greenland where the ice cover is rapidly melting. But the model doesn’t give the expected results!!. Damned but what the f*ck and meh, lets hammer the precipitation grid in shape around the southern parts of Greenland. And Presto we have a new “scientific” reality.
From Harry’s README line 1496:
—- Firstly, wrote mmeangrid.for and cmpmgrids.m to get a visual comparison of old and new precip grids (old being CRU TS 2.10). This showed variations in ‘expected’ areas where changes had been made, it the Southern tip of Greenland. —
If you want the original file search on google for foi2009.zip.
Right now it seems the greater threat to mankind are these scientists. A time line of emails , data and code should be laid out and then an attempt should be made to find real input data and workable models that have statistical validity. Right now the input data sounds ridiculously fragile / tortured and thus I doubt the models and their predictive ability would be worth much in the light of day. What a weird group of so called scientists!
A double-blind test for all such computer modeling schemes whose cobblers, patchers, and tinkerers claim are able to make predictions is LONG OVERDUE.
Next in the docket: every publication that claims to have a peer review process yet ran with the guano produced by CRU affiliated ‘researchers.’
Here is my theory of what happened. Britain has a FOI law. Phil Jones was conspiring to violate the FOI law by deleting files and emails. Some poor IT shmuck was directed by Dr.Jones to do the deleting.
The poor IT shmuck was faced with the unenviable choice of deleting the files and email with the significant possibility that he would go to prison while Dr.Jones washed his hands and did nothing or the poor IT shmuck could refuse to delete the files and get fired into today’s very unfriendly job market.
The poor IT shmuck resolved his dilemma by leaking the emails, software and data which is going to get Dr.Jones fired instead.
If you are working in IT at some institution that is also faking climate data and is liable to FOI requests, you had better be maintaining your own zip file if you would like to stay out of prison.
Great article guys! As a PhD level Analytical Chemist, this utter lack of scientific integrity is sickening to me. Thanks again!
Harry leaked the emails.
The UEA should help to bring criminal charges against Phil Jones.
The is fraud on a massive scale. Pure and simple. Criminal fraud.
@Javelin
I was thinking the same thing. It is one thing to hack email servers but how did the ‘hacker’ get to the source and data as well?
Fix the programming/data problems and run the programs again.
It’ll be interesting to see how the results change.
If this was part of a Bones episode or something people would scratch their heads and say ah-ha…but in reality it’s been drummed into so many heads by the MSM it’s gonna take a’while to get through to them. More satire from Daily Show and SNL, as well as Jay Leno might help change their opinion…
As someone with a MS in meteorology and “training” in programming, I can tell you this is quite common in the meteorological community, esp. in academic and research circles. I have a lot of code that “works” but I wouldn’t bet multiple terabucks on the outputs. With all the money being flushed into climate research, it’s a scandal that software engineering is not demanded through the process.
Outside auditing of the entire process should be a requirement. “Peer review” is not an audit. If you want to make trillion-dollar changes in the economy, you should spend a few mil on proper software principles and be able to repeat your processes end-to-end. The fact that poor Harry couldn’t duplicate the results, or even understand how someone else got to the results he was handed, shows you can’t trust them.
this from the BBC..
http://news.bbc.co.uk/1/hi/sci/tech/8388485.stm
mmm…
to quote..
‘We argue that the evolving practice of science in the contemporary world must be different from the classic view of disinterested – almost robotic – humans establishing objective claims to universal truth.’
I try to keep an open mind on these issues, and am accordingly receptive to new information and ideas. The issue here is honesty. One of the reasons I became a global cooling skeptic was that for years I would read the global warming deniers’ writings on the web, and then refer to the scientific data they were commenting on. And they didn’t match.
For example the Heartland Institute once published a broadside on the melting of the Arctic icecap, quoting a press release of the US oceanographic institute, NOAH I think it’s called. I went to the NOAH web site and discovered to my dismay that the Heartland Institute was lying through its teeth. Its claims were not in the slightest borne out by the NOAH report.
I can search my files and give you the exact dates and cites, if you don’t believe me.
In any case, I would recommend that you devote at least a portion of your skepticism to those in your own ranks who produce nothing but lying propaganda. I think that would make global warming denial a considerably more respectable pursuit than is is at present.
HighPlanesDrifter12/8/2009
No matter what is discovered about the climate fraudsters, they will gleefully try to convince us that the darkness outside is really daylight…Really? Well this time fewer are buying it. .We “knowNothings Flatearthers” (REALISTS) have known for a long time that phony demagogues on the left will stop at nothing to lead the naive, the ill-informed, the un-informed and the mis-informed down this disastrous superhighway of deceit, whether it be single payer health care, cap/trade, or CAFE standards for cars. If they really cared about greenhouse gases, they would shut their traps and include nuclear power and NatGas on the menu. But they don’t so, we REALISTS must conclude it’s money and control they are after…With the latter being the main dish..HPD out….
The quality of their code & data might make more sense if one considers it in the context of a business rather than science. Need to get the product out, there are TTM pressures you know. Get that revenue cycle started, and fix it in production!
Outside auditing of the entire process should be a requirement. Peer review is surely not an audit.
I’m typically to blogging and i really recognize your content. The article has really peaks my interest. I’m going to bookmark your site and maintain checking for brand new information.
Harry is not a professional FORTRAN programmer. The forrtl: error (75): floating point exception usually division by zero or overflow. In this case, it is probably an overflow, hence the large negative number in OpTotSq. In the early days some computers had 16 or 32 bit registers only. The leftmost bit was reserved for sign bit to indicate positive or negative number. However, if a number was defined as an Integer and was too large , eg. for a 16 bit register larger than 65536, the leftmost bit of 0 (positive number) was changed to 1 ( negative number ) hence the large negative result.
This is FORTRAN 101. Harry must have been a scientist-turned-programmer without a proper training.
Of course, I would have to see the code to be 100%, but most probably that is what happened. The OpTotSq should have been defined as a floating point number and not as an Integer. That is why it worked before, the number was less than for example 65536 for 16 bit machine. Once over 65536, it became negative.