More than just Emails - Climate 'melt down' includes model code

[Phier]

One of the reasons I am a climate skeptic is that the models given are never 'predictive' to show how climate in the past worked. What I mean by that is that you can't plug in what we know about 1500 AD and reach what it was like in 1800 AD, or any other time period for that matter. Nor do they work backwards. Basically you have to take their results on part faith as its all highly speculative.

When I asked in a different thread in the science section to see such a program, I was told that of course they wouldn't release them. I'm not sure why, it 'is' about saving the planet after all, but apparently that excuse good enough for the believers.

Well it seems that the hacked or possibly leaked email achieve also contains some of the code used by CRU to make those scary all red graphs about how we are all going to cook in 50 years. The annotations alone seem to be quite ... enlightening.... as to their methods.

Spoiler Alert, click show to read: 

CRU code

• Francis at L'Ombre De L'Olivier says the coding language is inappropriate. Also inappropriate use of hard coding, incoherent file naming conventions, subroutines that fail without telling the user, etc etc.
• AJStrata discovered a file with two runs of CRU land temp data which show no global warming per the data laid out by country, and another CRU file showing their sampling error to be +/- 1°C or worse for most of the globe. Both CRU files show there has been no significant warming post 1960 era
• A commenter notes the following comment in some of the code:"***** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********"
• Good layman's summary of some of the coding issues with a file called "Harry". This appears to be the records of some poor soul trying to make sense of how the code for producing the CRU temperature records works. (rude words though, if you're a sensitive type)
• Some of annotations of the Harry code are priceless - "OH **** THIS. It's Sunday evening, I've worked all weekend, and just when I thought it was done I'm hitting yet another problem that's based on the hopeless state of our databases. There is no uniform data integrity, it's just a catalogue of issues that continues to grow as they're found."
• CRU's data collation methods also seem, ahem, amusing: "It's the same story for many other Russian stations, unfortunately - meaning that (probably) there was a full Russian update that did no data integrity checking at all. I just hope it's restricted to Russia!!"
• Borepatch discovers that CRU has lost its metadata. That's the bit that tells you where to put your temperature record on the map and so on.
• Mark in the comments notices a file called resid-fudge.dat, which he says contains, believe it or not, fudged residuals figures!
• Mark in the comments notes a program comment: "Apply a VERY ARTIFICAL correction for decline!! followed by the words `fudge factor' " See briffa_sep98_d.pro.
• From the programming file combined_wavelet.pro, another comment, presumably referring to the famous Briffa truncation: "Remove missing data from start & end (end in 1960 due to decline)".
• From the file pl_decline.pro": "Now apply a completely artificial adjustment for the decline only where coefficient is positive!)"
• From the file data4alps.pro: "IMPORTANT NOTE: The data after 1960 should not be used. The tree-ring density' records tend to show a decline after 1960 relative to the summer temperature in many high-latitude locations. In this data set this "decline" has been artificially removed in an ad-hoc way, and this means that data after 1960 no longer represent tree-ring density variations, but have been modified to look more like the observed temperatures."
• From the Harry readme:"What the hell is supposed to happen here? Oh yeah - there is no )'supposed', I can make it up. So I have :-)...So with a somewhat cynical shrug, I added the nuclear option - to match every WMO possible, and turn the rest into new stations (er, CLIMAT excepted). In other words, what CRU usually do. It will allow bad databases to pass unnoticed, and good databases to become bad, but I really don't think people care enough to fix 'em, and it's the main reason the project is nearly a year late. " (see Harry readme para 35.
• James in the comments says that in the file pl_decline.pro the code seems to be reducing temperatures in the 1930s and then adding a parabola to the 1990s. I don't think you need me to tell you what this means.

Here is an interesting analysis by some random internet programmer who looked at said code.

Spoiler Alert, click show to read: 

After a day or two looking through the code (I’m a programmer, not a climatologist) and getting some good responses from gavin at realClimate I am surprised at a few things.
1) I really thought code like this would be super complicated — obviously simulating weather long term is a very tough problem. However in the code it comes down to, get the measurements, apply arbitrary weighting/changes to the end set of data (blade portion), and then weight in hotter measured temps for the blade only (non linearly). One could argue that the numbers used to reduce proxy signals in the 40’s and amplify them in the blade portion were the result of hundreds of inputs and deep thought, but seeing -.03 and 2.6 there doesn’t inspire confidence. That’s like the ultimate answer being 42. The comments above the lines don’t help either, things like “****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE ******”.
2) Measured temperatures in recent times are way off temps predicted by proxies. Rather than question the proxies, the later measured data is adjusted. Then it is changed again by adding in the latest hotter measured temps — not by overlay, but mixing the signals together plus overlay. You just can’t do this and call it honest. Its made to look like the proxies were hard data this way, but the hard data ends at the foot of the blade. After seeing this in the code, I went to the graphs to check – and the graphs actually also say they are doing this in the notes! So two points for being upfront, but HOW does that get a pass? This isn’t a climate question, this is a statistics/graphing question, and I believe have plenty enough experience in that field to call foul. Then in the ‘IPCC comments file’ you see this brought up in a few places, all rejected (“its in the notes”). Wow. Noting a deception doesn’t make it ok. “Your pants are on fire*” ….. *actually they’re not.
3) In the famous harry file, the v3.0 is being fit to match the 2.1 results. That is fine, but it is by no means independent confirmation of previous data code. Also, the standards are shockingly low in every aspect of that process as well.
My impression is that in the initial stages people are trying hard to do good science, but then the results are filtered at the end to look however they need to look. This happens basically one step before the graph stage, and then again at the graph stage. You can tell a lot from how a graph is presented about bias of intent, but when the bias includes data adjustment as well you basically have to throw the whole thing out.
Anyway, I think I’ve wasted enough time on this. I’m almost glad the financial industry’s code wasn’t made public like this, it is probably similar and I’d have spent a week. 2009 is shaping up to be the year of “the death of the complex computer model”. Except under the covers it isn’t all that complex, just crudely manipulated : (.

Now I have a bit less faith in this than I do the emails in terms of being valid. Those are for all practical purposes confirmed, but is anyone honestly surprised by this sort of thing? I used to shout this from the mountain tops, but less people were listening and my life had to go on to more practical things like family. Oddly I'm feeling smugly vindicated in my skepticism.

Ah just saw this, apparently this has its own theme song now.

[Gaidin]

Holy crap one university has some shady business and you think it throws everything up in the air?! <facepalm>

Wait why is there a second thread...

[Phier]

Holy crap one university has some shady business and you think it throws everything up in the air?! <facepalm>

Wait why is there a second thread...

Oh I'm sorry, I'm surprised you could respond covering your eyes to everything and complaining about illegal hacking. BTW odds are more it was a leak.

Its another thread because its another issue, they FUDGED THE CLIMATE MODELS. I hope thats clear.

[Bovril]

Its another thread because its another issue, they FUDGED THE CLIMATE MODELS. I hope thats clear.

They wouldn't have bothered doing this if there wasn't an overwhelming concensus allready that they wanted to fit into. But maybe you think almost all the climatologists in the world are engaged in a shady conspiracy. Even the American Assosiation of Petroleum Geologists recognises that climate change is significantly influenced by human activity. They're not even a relevant body and still feel the need to assert this fact. All relevant scientific bodies agree.

[Jexiel]

They wouldn't have bothered doing this if there wasn't an overwhelming concensus allready that they wanted to fit into. But maybe you think almost all the climatologists in the world are engaged in a shady conspiracy. Even the American Assosiation of Petroleum Geologists recognises that climate change is significantly influenced by human activity. They're not even a relevant body and still feel the need to assert this fact. All relevant scientific bodies agree.

Good ol' appeal to authority. The fact that they agree does not make the evidence presented valid. Have you considered that perhaps these organizations agreed after studying manipulated data? After all, centuries ago a certain leading institution (Catholic Church) believed the Earth was the center of the Universe untile one man (N. Copernicus) presented evidence to the contrary; Likewise, that disease was caused by germs was not believed until Agostino Bassi suggested this in the 1800s and yet it took a while for it to be accepted in Medicine. Same goes for smoking, not considered harmful until evidence to the contrary was presented. More recently, the National Academy of Sciences supported the idea that trans fat were not harmful until evidence to the contrary was presented.

The good ol' appeal to authority defense is ineffective. Much less manipulated evidence to fit an ideology.

[Bovril]

Good ol' appeal to authority.

You have misunderstood this fallacy. It does not refer to the reliance on experts. We do this all the time in all fields of enquirey. It would be impossible to do research otherwise. Since you and I are not experts, it makes sense to defer to those who are. Putting your fingers in your ears is not a respectable position.

Oh and by the way, what is the "ideology" of climate change?

[Syron]

They wouldn't have bothered doing this if there wasn't an overwhelming concensus allready that they wanted to fit into.

If the "Overwhelming Consensus" (TM) was correct you would not need to fudge the data......

The question you should be asking is why they were doing this. If these guys had used their actual data showing this is a fraud they would be quite famous. However climate science has historically been poorly funded, nothing brings in research money like predictions global catastrophe and so it is invented.

Even the American Assosiation of Petroleum Geologists recognises that climate change is significantly influenced by human activity. They're not even a relevant body and still feel the need to assert this fact.

Because they are being told that's the case by people who are fudging data!!!!

As you say they're not the relevant body and so have not been conducting the research, so what is their word worth?

[Bovril]

If the "Overwhelming Consensus" (TM) was correct you would not need to fudge the data......

The question you should be asking is why they were doing this. If these guys had used their actual data showing this is a fraud they would be quite famous. However climate science has historically been poorly funded, nothing brings in research money like predictions global catastrophe and so it is invented.

When I advised you to read Kuhn I was not joking. Scientists doing 'normal science' necessarily question their paradigm last of all, whther this process is more or less conscious. Otherwise usefull research would simply not get done. Besides, you assesment of what brings in research money is clearly false. Much money has been pumped into climatology by corporations and governments wishing to minimize the idea that humans have an effect on climate. I can't for the life of me think of a major funding body that wishes from self interest to prove climate change is man made except those who wish to fit into a concensus established despite the overwhelming inconveniance to all powerful institutions that humans are causing potentially catastrophic climate change.

Because they are being told that's the case by people who are fudging data!!!!

Read again my comments on causal direction.

As you say they're not the relevant body and so have not been conducting the research, so what is their word worth?

The AAPG's case is simly illustrative of the overwhelming concensus. It's an aside to amuse those of us not wearing tin foil hats.

[Gaidin]

Oh I'm sorry, I'm surprised you could respond covering your eyes to everything and complaining about illegal hacking. BTW odds are more it was a leak.

Its another thread because its another issue, they FUDGED THE CLIMATE MODELS. I hope thats clear.

I'm not saying that its not a big deal. I'm saying that the hacked department isn't the only climate change department in the world. There are other departments that have verified research. ReadingComprehension++;

[CtrlAltDe1337]

Wow, more of this? And so quickly? I can't believe the man-made global warming scam is being destroyed so quickly. I suppose this may be fake, but I will wait and see what comes of it

[The spartan]

...So what...has GCC turned into a conspiracy now? Great...

[CtrlAltDe1337]

...So what...has GCC turned into a conspiracy now? Great...

Its not a conspiracy when the proponents state their positions openly, and it didn't "become" such, it always has been ever since Al Gore realized he could make money using Carbon credits...

[The spartan]

Its not a conspiracy when the proponents state their positions openly, and it didn't "become" such, it always has been ever since Al Gore realized he could make money using Carbon credits...

...what?

[Ummon]

The inability to produce a viable model is not an objection as much as a model reproducing a phenomenon is not proof that the phenomenon works that way.

No climate prediction can occur with present technology beyond a few weeks, and long term planetary climatology is surely beyond computation.

[Phier]

No climate prediction can occur with present technology beyond a few weeks, and long term planetary climatology is surely beyond computation.

A simple fudge factor is all you need for the IPCC, and if you can't trust them who can you trust? You must be mistaken sir, for who are you to question the consensus?

[Ummon]

I don't need to question anything, it is a well know fact that chaotic phenomena cannot be predicted beyond a certain limit, and that climate is a chaotic phenomenon.

In October 1987 a control data cyber 205 did predict good weather on the British Isles and the worst storms of the century ensued.

The rival Cray 1 predicted the storms correctly.

Chaos is not predictable with certainty.

We can though easily evaluate trends. Global warming is a very reasonable trend to foresee.

[Denny Crane!]

I am not in a position to judge the data, before they start taking my money though there needs to be more work done. In fact I don't trust government to handle any changes that need to be done so therefore they should just stop taking my money.

The science here is without value. By that I mean there is so much funding available from governments who want to see the proof for climate change out of concern that they will indeed find the results they want. Once the dust settles and there has been more research done I'll trust it more, but I still suspect technology will achieve change a million times faster than premature legislative action and spending now. It is like attacking a rugby player with a damp tissue using legislation against it.

[Bovril]

The science here is without value. By that I mean there is so much funding available from governments who want to see the proof for climate change out of concern that they will indeed find the results they want. Once the dust settles and there has been more research done I'll trust it more, but I still suspect technology will achieve change a million times faster than premature legislative action and spending now. It is like attacking a rugby player with a damp tissue using legislation against it.

You seem to think governments are using climate change as an excuse to raise taxes or something similar. Why then has it taken mass popular movements to get western governments to take climate change seriously, why are western governments making efforts that are laughable compared to the recomendations of climatologists and why do environmentalists uniformally campaign for actions that are in comparison to government actions, radical in the extreme. The US in particular is way, way behind its population on climate change. Most Bush voeters in 04 (about 60%) thought he supported the Kyoto protocol! This is hardly a government lead initiative.

[Denny Crane!]

You seem to think governments are using climate change as an excuse to raise taxes or something similar.

As an excuse to create more legislation and control perhaps.

Why then has it taken mass popular movements to get western governments to take climate change seriously, why are western governments making efforts that are laughable compared to the recomendations of climatologists and why do environmentalists uniformally campaign for actions that are in comparison to government actions, radical in the extreme. The US in particular is way, way behind its population on climate change. Most Bush voeters in 04 (about 60%) thought he supported the Kyoto protocol! This is hardly a government lead initiative.

This is my problem with government action at all. They talk a lot, perhaps introduce more tax or more legislation and damn all results are seem. They should just be offering tax breaks to energy research and renewables (though not subsidies that distort the market)

[Ænema]

People, can we please clear one thing up:

We are debating weather HUMANS ARE TO BLAME FOR OUR CHANGING CLIMATE.

There's no debate on weather climate change is real or not, because it is. Humans obviously didn't cause the Ice Age and didn't cause the Earth to warm up after it.