Who would have thought that the climate models used as the basis of IPCC greenhouse forecasts would violate 72 of 89 principles of forecasting. That’s the claim from forecastingprinciples.com a site run by J. Scott Armstrong,
Professor of Marketing at the Wharton Business School, University of Pennsylvania. He and Kesten C Green from Monash University have published an audit of the forecasts from the IPCC Fourth Assessment Report. (You can access the report from http://www.forecastingprinciples.com/Public_Policy/global_warming_audit.html). The auditors come to the view that while the scientists might know something about physics, they understand little about the science of forecasting.
This quote gives a flavour of the review:
9.3 Do not use fit to develop the model.
It is not clear to what extent the models produced by the IPCC are either based on, or have been tested against, sound empirical data. However, some statements were made about the ability of the models described in Chapter 8 to fit historical data, after tweaking of their parameters. Extensive research has shown that the ability of models to fit historical data has little relationship to forecast accuracy (See “Evaluating Methods†in Armstrong 2001.) It is well known that fit can be improved by making the model more complex. The consequence of increasing complexity to improve fit, however, is to decrease the accuracy of forecasts. The 12 authors of Chapter 8 appeared to be unaware of this principle.
In other words, by trying to make the models more accurate the scientists are doing the opposite.
It’s an entertaining read and reveals amongst other things that the opinions of experts are no better than the opinions of anyone else when it comes to forecasting. Worse, that the wrong opinions of experts can reinforce each other – an example of the law of group polarisation, and a fact which potentially undermines the validity of the refereed publishing process.
This report opens a new and important front in the CO2 wars. The lead author also runs an interesting site for those of us who are skeptical of all sorts of fortune telling.
but perhaps not even the IPCC believes they are into forcasting:
In a remarkable contribution to Nature magazine’s Climate Feedback blog, Trenberth concedes GCMs cannot predict future climate and claims the IPCC is not in the business of climate prediction. This might be news to some people.
Among other things, Trenberth asserts “. . . there are no (climate) predictions by IPCC at all. And there never have been”. Instead, there are only “what if” projections of future climate that correspond to certain emissions scenarios.
from http://www.news.com.au/sundaymail/story/0,,21977114-27197,00.html
Comment by Jennifer — July 8, 2007 @ 10:59 pm
Ah Jen, the authors of the audit got there first on this one on page 8 of their paper:
“In apparent contradiction to claims by some climate experts that the IPCC provides “projections†and not “forecasts, the word “forecast†and its derivatives occurred 37 times, and “predict†and its derivatives occur 90 times in the body of Chapter 8. Recall also that most of our respondents (29 of whom were IPCC authors or reviewers) nominated the IPCC report as the most credible source of forecasts (not projections) of global average temperature.”
Projections or forecasts? I’ll go with the plain meaning of the words they use. BTW, the paper also gives the IPCC Chapter 8 Flesch-Kinkaid and Gunning-Fog Scores as well. It does very poorly.
Comment by Graham Young — July 8, 2007 @ 11:19 pm
May be semantics but I see little real world difference between ‘projection’, ‘forecast’ and ‘prediction’ of an outcome based on a set of pre-defined criteria.
Comment by Stuart — July 10, 2007 @ 11:45 am
I’d like to point out that the “forecasting principles” folk publish the meticulously scientific terrorismforecasting.com and conflictforecasting.com.
What a joke.
The critics’ most interesting point is that their “scientific forecasting” catchphrase wasn’t mentioned. There is no protected monopoly on the technique of the educated guess!
The “audit” criticises the IPCC report both for taking a punt and for using the word “forecast” when it claims not to be able to make one. Sorry, but the word “forecast” wasn’t defined by a standards committee.
Everything else they have to say is standard fare for climate science, except their *absurd* conclusion which is to say that the (few and uncertain) predictions made in the IPCC report aren’t scientific.
A prediction is as scientific as the science which is used to make it, whether you use a buzzword or not.
The golden rule about making predictions from limited knowledge is to be clear about what you *don’t* know. The IPCC report (and all climate science literature) is very clear about what it doesn’t know, and therefore, quite reasonably, declines to make any specific predictions it can’t reasonably make.
The talk about “tweaking the model to fit the past” reducing predictive accuracy is just guff. Climate models are causal models, using physical principles to simulate physical processes. Real historical data about the physical processes is supplied as an input for time-periods when the historical data is available (eg. volcanic eruptions, industrial emissions). When the data isn’t available, the model is purely speculative.
The best that climate science can do is state — categorically — that perturbing the climate in a particular way will have a particular primary effect, and mention what feedbacks are expected to occur. Since some of the feedbacks involve unknown (but very large) quantities, quantitive predictions are obviously problematic.
*If* it is assumed that none of the unknown feedback mechanisms kicks in, reasonably good “scientific forecasts” might be made regarding the effect of greenhouse gas pollution. But some of the feedback mechanisms have not been sufficiently closely studied for climate science to predict their effect — thawing tundra and clathrates, for obvious examples of recent discoveries, but also cloud cover and storm behaviour which is notoriously unpredictable.
The best guess that the best-informed people can make is as scientific as it gets. If the science is uncertain, it’s uncertain. If the best guess is that a *minimum* change is certain, but it *might* be much higher, you’d better believe it.
That the “best guess” doesn’t (pedantically) follow the (exceeding vague) methodology on this website doesn’t make it “unscientific”.
Comment by xoddam — July 10, 2007 @ 4:13 pm
You can always tell the ones with the axe to grind – play the man and not the ball. And they are generally anonymous.
Xoddam, if you looked at the terrorist site you’d find that these guys specialise in complexity and games theory. You might sneer at attempts to get a handle on how terrorists will act, but it’s probably more useful than much of the climate modelling that’s been done to date.
Comment by Graham Young — July 11, 2007 @ 11:45 am
I think our gracious host might be indulging in a little “playing the man” himself there. My post was long, and except in the first three lines it most certainly did “play the ball”.
Allow me to introduce myself, Mr. Young. My name is Jonathan Maddox. I’m a professional engineer and a father. In the interests of full disclosure I’ll admit I have my hair in a ponytail and use public transport. I invariably sign myself online with my surname reversed, a conceit I started in primary school.
And I have an axe to grind — what of it?
Any attempt to reassure people that greenhouse pollution is nothing to worry about sets back our efforts to address the very real risk that its consequences will be utterly devastating to human populations and to the global economy.
Damage to the biosphere is, to me, a secondary concern — over geological timescales, life on earth has suffered worse and recovered. The modern industrial economy, however, has not existed over geological timescales, and won’t, if it doesn’t curb its excesses.
I’d rather my children and theirs inherit a world which looks more like Denmark than Afghanistan.
It is well-documented that powerful people with a lot to lose have invested huge resources in the climate-change-denial business, attempting to “junk” the science:
http://www.monbiot.com/archives/2006/09/26/whos-paying/
http://www.onlineopinion.com.au/view.asp?article=5992
Consider me a volunteer in the opposition to this industry of denial.
Whether Armstrong et al are connected with Exxon’s PR campaign or the coal-industry “greenhouse mafia”, or just well-meaning academic critics, I have no idea. Either way it is hardly relevant to the urgency of the issue.
Even if their criticism had substance (and any reader can see it is little more than nitpicking) they are doing the world a disservice by encouraging complacency.
I had no intention of dismissing Armstrong et al. or their work out of hand. But I do think they’re a pot calling a kettle black.
I looked over the conflict forecasting websites before mentioning them here. They do indeed discuss game theory, which is an entirely reasonable way to go about analysing unpredictable human behaviour.
But to describe this kind of study, closer to what goes on in a poker game than to physical modelling of physical processes, as “scientific forecasting” while criticising the IPCC report for its acknowledged uncertainties and use of words in their common English usage rather than the stricter usage of a particular academic field, labours a meaningless point.
There is a “science” of forecasting, being the study of people’s attempts to make forecasts, and the work done by Armstrong et al., including on the terrorismforecasting.com website, is mostly dedicated to showing how forecasts of all kinds are usually very poor.
The most accurate forecasts, including the ones Armstrong endorses, are ones where measurements are numerical and exhaustive, and unknowns are eliminated as common terms. Statistics reveal all: you don’t need to think about people’s complex motivations while compiling sales figures or election statistics, you just observe correlations.
With climate change, the unknowns are potentially bigger than known factors, and the data available simply don’t tell us enough to be able to make a statistical forecast relevant.
Armstrong’s own work, the “decision tree” at
http://www.forecastingprinciples.com/selection_tree.html
quite clearly directs would-be forecasters to use “causal models” for the kind of problem presented by climate change, but warns that their accuracy isn’t terribly good. That’s exactly what climate scientists do, and they don’t pretend that the resulting “forecasts” are accurate, merely indicative of possibilities. The idea that climate scientists, many of them expert meteorologists, are unaware of the principles and uncertainties of prediction is laughable.
Climate statistics tell us only that there is a small warming trend, possibly accelerating (or maybe that’s just natural variation). Armstrong and Green would have us believe that this is the best prediction science can make, in direct contradiction to their “decision tree” recommendation to use modelling. It is not.
Plain physics tells us — without room for doubt — that positive feedback mechanisms are amplifying the warming already seen, but these secondary effects are delayed and we have no idea how big the warming will eventually be, nor how effective the few known negative feedback mechanisms (like cloud formation) might be in keeping it back.
We also have no really good way of knowing how warming will change rainfall, which is the one thing about global warming (forget sea level rise) which risks the cost of millions of lives in coming decades as crops fail. With the stakes this high, we can’t afford not to be conservative in our response. Conservatism in our assesment of the risk is too risky!
There is also, of course, a huge uncertainty in whether human industrial behaviour will change as a response to the knowledge that there is a problem.
I’d like to help make our response a little more certain, and our industry part of a negative feedback mechanism, helping to stabilise the climate rather than upset it.
Comment by Jonathan Maddox aka xoddam — July 11, 2007 @ 1:41 pm
There you go again, playing the man, or the host. Yes, you did do a long post – long on words and short on analysis. You appear to have an a priori position that you’ll adhere to regardless.
And your opening words set the scene.
As have your later smears – all the doubters are funded by Exxon – as if!
We’ve already seen one of the climate icons – the hockey stick – knocked down by expert review. I’m not sure why you are so certain that climate modellers know what they are doing.
Comment by Graham Young — July 12, 2007 @ 5:28 pm
Mr Young, I did not allege that you, nor Armstrong & Green, are funded by Exxon. What I wrote is clear.
The “product of doubt” regarding climate change science is *known* to be peddled by vested interests. A priori, anything purporting to “open a new and important front in the CO2 wars” is suspect.
Exxon is neither the only nor the most influential fossil-fuel interest to “junk” science; it is just a big public company whose financial dealings are traceable.
Given the well-documented behaviour of Exxon and our local “greenhouse mafia”, surely other (less public) interests, not to mention those who misguidedly believe they also have something to lose from climate change mitigation, such as ideological fear of “creeping socialism” are equally keen to sell complacency by sowing doubt.
The facts speak loudly enough for themselves, but people have their own reasons for not listening.
> We’ve already seen one of the climate icons – the hockey stick – knocked down by expert review.
Oh My Goodness. I’m sure you already know that’s Classic Climate Myth Number Three:
http://environment.newscientist.com/channel/earth/climate-change/dn11646
http://environment.newscientist.com/channel/earth/mg18925431.400
There’s no doubt whatever that the present global average temperature is the highest in at least six thousand years, probably even 160,000 years.
Comment by xoddam aka Jonathan Maddox — July 13, 2007 @ 10:10 am
Jonathon, there you go again. Play the man. You are asserting that people who question the global warming alarmists are acting in bad faith. When people do that it is a sure sign that either they don’t understand the arguments, except as they are filtered through people’s reputations, or they do understand the arguments and don’t want to get into them because they know that they will lose.
You can throw as many links at me as you like, but the Wegman committee reviewed the Hockey Stick. They found that its algorithms made hockey sticks out of mere randomness (a point made by the auditors I reported on in this blog piece as a problem with some approaches to modelling) that it relied on insufficient data (I think there is only one bristle- cone representing the whole of the southern hemisphere) and that the peer review process was flawed because it tended to only involve colleagures of the modellers (another point covered by the auditors above).
We know that temperatures were warmer in Europe in the middle ages, because there were people there to observe them. We are less certain about the other parts of the world where populations were sparse and illiterate, but ice core samples also show it was warmer in the Antarctic than it is now quite recently – within the last 10,000 years.
So the hockey stick fails the basic test of meeting the observed facts. It is in fact an artefact – an interesting and flawed attempt at reconstruction.
Which is of course why you play the man, because there is so much that you can’t explain and don’t understand.
Comment by Graham Young — July 13, 2007 @ 10:51 am
This time I’ll be really careful not to question anyone’s motivations, OK?
The Wegman committee report was news to me. I read it while I should be working, but the boss is away so I can make up for it later. I can’t afford to have the last word every day 😉
http://energycommerce.house.gov/reparchives/108/Hearings/07192006hearing1987/Wegman.pdf
I have a reasonably good grasp of statistics and I believe I understand the report. It is a critique of two papers, published by Mann, Bradley & Hughes, in 1998 and 1999. These papers were the first credible attempt to reconstruct hemisphere-wide temperature data over the last 1,000 years. They compared instrumental temperature records (beginning in the late 1700s, but only really reliable since 1900 or so) with proxy data going back further, such as tree rings and isotopes in ice cores. Tree rings obviously provide an annual proxy, but are not a great direct indicator of average temperatures, rather of local growing conditions. Ice core proxies have a much poorer resolution (decades, not years) but are believed to reflect “rolling” average global temperatures quite well.
I understand that Mann et al’s motivation in centring the curve on the 20th-century data was to reflect the fact that the proxy data was calibrated to the 20th-century instrumental record, and was thus more prone to error prior to 1900.
The Wegman report points out that normalising the proxy-record data to the portion which is backed up by reliable instrumental data increases the estimated error and shrinks the variation in the earlier portion of the graph, while exaggerating definite trends in 20th-century data (Figure 4).
The technical point is entirely true as far as I can see, but Wegman’s re-centred graph, of tree-ring proxy data alone without reference to ice cores or actual temperature data, is not just re-centred but also uncalibrated: it looks like white noise, not only in the 20th century but it doesn’t clearly show the Medieval Warm Period or Little Ice Age either. If the re-centred graph used calibrated data instead, it would resemble Mann’s synthesis more closely. The off-centre graph shows all three, and obviously tracks the 20th century rise closely, but it’s still very noisy without calibration.
Tree rings *need* to be calibrated against known temperatures (preferably local to the tree, not merely a century’s hemispherical average) or the other factors influencing them, like local rainfall, overwhelm the temperature proxy signal. Older trees, that died before instrumental records, can only be calibrated against younger trees (again preferably nearby) whose lifespans overlapped with them, so the data is necessarily less reliable as you go back over centuries.
The Wegman report does *not* demonstrate that principal component analysis shows the ‘hockey stick’ when given completely random data, but rather that when 69 pieces of random data (in place of the proxy data) are combined with one piece of non-random data (the temperature reconstruction from the 1990 IPCC report) and correlated (“de-centred”) in the same way to reflect the 20th-century instrumental record, the analysis emphasises the actual signal and appropriately filters out an awful lot of noise (Figure 5). If his chosen “non-random” data point had matched the 20th century but shown a different shape for the non-calibrated period, his averages would have matched that instead.
This ability of calibrated CPA to pick a quiet signal out of a lot of noise should give *more* confidence in this methodology for analysing noisy proxies like tree rings, not less. Suppressing the error bars in the 20th century (where we *know* what the temperature really was) seems a small price to pay for getting a clear, if tentative, picture of times past. The error bars in the Mann graph do allow for the *possibility* that the MWP was actually warmer than now — it just wasn’t the best estimate they could make.
Wegman is endorsing valid criticisms made already in of Mann et al’s work in the refereed literature. He does not attempt a critique of subsequent papers (referenced alongside the original in the New Scientist link I gave you) which take this critique into account, amending the statistical methods accordingly. All the later papers — some using completely different methodologies — go on to provide new estimates which still indicate that the Medieval Warm Period was not as warm as the 1990s have been measured to be (note that none of the graphs in the Wegman report, except the original under critique, go beyond 1990). Wegman protests, validly, that professional statisticians weren’t consulted for the particular papers under scrutiny, but does not note that statisticians are increasingly involved in climate science and don’t challenge its general findings.
Wegman does mention some other studies, and criticises them for all using the same raw data. Nothing he has to say invalidates the raw data, which is the most comprehensive collection available. Of *course* authors all use the best data they can find, and researchers add to the same data pool when they make further field measurements.
Finally, Wegman also makes some gross insinuations regarding peer review and the authorship of the 2001 IPCC report. Before protesting that climate scientists know one another, collaborate, and review one another’s papers (anonymously), he (a statistician!) should pick a bigger sample and compare it with any other specialisation of academe. If anything, a wide-ranging interdisciplinary field like climate science is less “incestuous” than many of the arts or other “hard” sciences.
History, mathematics, and statistics are the exceptions, since everything has its own history, and every “hard” science uses mathematics and statistics.
Comment by xoddam aka Jonathan Maddox — July 13, 2007 @ 5:17 pm
Jonathon, I’ll take this as an acknowledgement that Wegman was correct. Something that no-one seems to have taken into account is the problems with calibrating tree rings – growth isn’t just a product of temperature. It is also produced by moisture, and CO2.
As CO2 has been increasing in the atmosphere plants have been growing faster. As far as I know this is not allowed for in the Manne et al reconstruction. It’s quite possible, without doing this that it could be as much a reconstruction of CO2 as it is of temperature.
There are good reasons for believing in the Medieval Warm period based on the historical record.
Comment by Graham Young — July 14, 2007 @ 9:04 am
The Medieval Warm Period is present on all reconstructions of northern-hemisphere temperatures; there is no doubt that it happened. The same goes for the Little Ice Age.
There is, however, a lot of uncertainty in the calibration of the proxies. The biggest argument over the MWP is whether or not temperatures were higher then than now. There’s enough uncertainty that we can’t say categorically that there was no one summer in medieval Italy which was the hottest in the last 1000 years (the error bars on even the original M,B&H paper do stretch above present levels) — but the best guesses all say the entire milennium preceding 1990 was cooler than the levels we have experienced in the last seventeen years.
Southern-hemisphere proxy data don’t indicate the MWP and LIA (if anything shey show a minor wiggle in the opposite direction), so they are thought to be northern-hemisphere phenomena only. The best guess is that they were the result of changes in thermohaline circulation of the North Atlantic.
I’ve written elsewhere that it was changes in the Gulf Stream, but have since learned that this was misleading — the name “Gulf Stream” refers only to a small fast-moving portion of the huge northward movement of warm surface waters in the Atlantic, and because it is a wind-driven current it is unlikely to be disturbed by changes in Arctic salinity. The larger but slower thermohaline circulation may well be affected by Greenland meltwater; if this were to stop it might easily cause temperatures in western Europe and eastern North America to drop by a degree.
CO2 levels have a crystal-clear proxy in the air pockets in ice cores; much better than the fickle proxies used to reconstruct the temperature record, and reconstructions show CO2 levels to be much less variable than temperatures. CO2 was not appreciably higher in the MWP nor lower in the LIA.
Wegman’s case boils down to “these guys should have centred their data points around the mean of the proxies, not the mean of the known part of the time series”. But they knew they weren’t certain about nine-tenths of it, centred their data on the portion they *were* certain of, and put in big fat error bars to make up for the fact.
Wegman would have them use smaller error bars, falsely indicating a higher level of confidence in earlier centuries, while failing to match the known-correct 20th century temperature curve!
Wegman is entirely correct when he says that it’s wrong to say “we are 100% certain that 1998 was the warmest year in the last milennium,” but M,B&H’s results don’t actually indicate that they were 100% certain.
Any statements by people like myself to that effect are made out of enthusiasm (and frustration with stubborn denialism), not pedantic rigour.
Wegman is, on the other hand, merely being rather rude when he casts aspersions on the validity of peer review in the climate science literature.
Comment by xoddam aka Jonathan Maddox — July 14, 2007 @ 3:21 pm
Jonathon, I think you’ve come in a little too late on this conversation, so to speak. The Manne et al hockey stick was controversial from the beginning. Not only did it eliminate the Medieval Warm Period, but it was also critiqued for it’s statistical methods by McKittrick and McIntyre who now collaborate on the climate audit site.
Without wanting to get into the fine detail, they discovered that the algorithms used by Manne tended to produce a hockey stick on random data. They also discovered that, contrary to proper scientific practice, Manne would provide raw data, but not the algorithms he used to interpret them, making it impossible to replicate his results, or even work out how he had got to them.
Worse, he established RealClimate to essentially slander McIntyre and McKittrick.
If Manne had been running a public company he would have been investigated and probably disqualified by the authorities from ever doing so again. His actions appear to me to be fraudulent.
McIntyre and McKittrick really only got the upper-hand when Wegman did his analysis which confirmed their complaints.
In this context it is entirely proper that the peer review process (as well as the scientific community) gets a whack from Wegman. The peer-reviewers were implicitly involved in the whole process of deception. He also takes a shot at the way that blogs were used.
BTW, the centring argument you are using appears to me to be a post hoc rationalisation. Why would centring it on more recent data which is an average be any more accurate than centring it on less accurate, but averaged data from the past. If you have faith in your reconstruction there should be little difference, especially as you are dealing in averages.
If you want to read up properly on this, devote the next month or so to shuttling between RealClimate and ClimateAudit.
Comment by Graham Young — July 14, 2007 @ 5:41 pm
In the main, the accuracy of that 1999 graph is irrelevant to the current problem.
We are very well aware of the kinds of effects that increased greenhouse gases have: warmer, warmer, warmer. The present warming trend since ca. 1970 has really only just begun — we could expect much more even if we were to halt emissions altogether, now.
Uncertainty in climate prediction is not in the primary effect of greenhouse gases, but (mostly) in the chaotic feedbacks from the warming we already anticipate.
The 1999 “hockey stick” did suppress the MWP and LIA a little. That was demonstrated. Subsequent temperature reconstructions show them more clearly. It’s still moot whether the MWP was warmer than the 1990s — but as I say, not relevant to the issue of greenhouse pollution.
Comment by xoddam aka Jonathan Maddox — July 17, 2007 @ 11:01 am