Prognostication

Mostly people have the perception of weather forecasters as horribly and habitually inaccurate, but this is mistaken. The simple fact is that predicting the weather is hard and there is bound to be degrees of uncertainty, but what weather forecasters have done that has made them vastly more accurate is they've tried to quantify the uncertainty. They don't just say it's going to rain tomorrow; they say that there's an 80% chance of rain or a 60% chance of snow. When we look at how accurate they are on these predictions by this standard of uncertainty, we find that weather forecasters are very accurate. In other words, if you look at all the times that a weather forecaster says "there's an 80% chance of rain tomorrow" you find that they're right about 80% of the time. Weather forecasters, of course, are constantly seeking to improve their predictions so that they predict things with higher degrees of certainty, but just being able to quantify the uncertainty is a big help, especially for people who vitally depend on good predictions, like farmers.

I mention this because I was just reading about a survey (ht) of the accuracy of the predictions of political commentators. Political pundits aren't that good at predicting the future. Part of the problem is that they're rewarded (in the form of attention and ratings) for being more partisan and simplistic, and also another part is that they don't really don't get accurate feedback (they don't check the accuracy of their own predictions). But perhaps they could also they could improve by couching their predictions in terms of degrees of uncertainty.

Arnold Kling, who blogs over at Econlog is a big fan of betting on future outcomes, instead of just predicting. The reasoning is that putting money on the line really incentivizes a person to contemplate and investigate their predictions more carefully, and it encourages people who don't really know what they're talking about to back away and make room for people who really know what they're talking about. When many of these bets are aggregated, predictions as accurate as a good weather predictions can emerge, for example at a site like intrade, an Irish site where people can wager money on various future events. The aggregation of many people's bets (which pay out based on whether an event happens as well as the odds when you place the bet) gives an overall bet on how likely an event is to happen. The more people tend to bet on a particular event, the more accurate the predictions tend to become. This can give a good picture of the certainty, at least so far as things look now (unsurprisingly the accuracy of the bets improves as the event approaches, but such is the case with all predictions). You can see how probable it is that various persons will get the Republican nomination for president in 2012 (Romney leads right now with 23.3%), who will win American Idol (James Durbin leads with 39.5%), whether the AV reform will pass in Britain (50% right now).

In fact, betting odds have been used in sports betting long before they were used by weather forecasters, leading to the surprising fact that we've long been much better at predicting the outcome of sporting events than we have at predicting the weather.

But the main point is that these predictions markets and this quantification of uncertainty provide much better models for predicting the future. The future will always be hazy and there will always be uncertainty, so whatever tools we can use to quantify that uncertainty will help us foresee future events with better clarity.

Mostly people have the perception of weather forecasters as horribly and habitually inaccurate, but this is mistaken. The simple fact is that predicting the weather is hard and there is bound to be degrees of uncertainty, but what weather forecasters have done that has made them vastly more accurate is they've tried to quantify the uncertainty. They don't just say it's going to rain tomorrow; they say that there's an 80% chance of rain or a 60% chance of snow. When we look at how accurate they are on these predictions by this standard of uncertainty, we find that weather forecasters are very accurate. In other words, if you look at all the times that a weather forecaster says "there's an 80% chance of rain tomorrow" you find that they're right about 80% of the time. Weather forecasters, of course, are constantly seeking to improve their predictions so that they predict things with higher degrees of certainty, but just being able to quantify the uncertainty is a big help, especially for people who vitally depend on good predictions, like farmers.

I mention this because I was just reading about a survey (ht) of the accuracy of the predictions of political commentators. Political pundits aren't that good at predicting the future. Part of the problem is that they're rewarded (in the form of attention and ratings) for being more partisan and simplistic, and also another part is that they don't really don't get accurate feedback (they don't check the accuracy of their own predictions). But perhaps they could also they could improve by couching their predictions in terms of degrees of uncertainty.

Arnold Kling, who blogs over at Econlog is a big fan of betting on future outcomes, instead of just predicting. The reasoning is that putting money on the line really incentivizes a person to contemplate and investigate their predictions more carefully, and it encourages people who don't really know what they're talking about to back away and make room for people who really know what they're talking about. When many of these bets are aggregated, predictions as accurate as a good weather predictions can emerge, for example at a site like intrade, an Irish site where people can wager money on various future events. The aggregation of many people's bets (which pay out based on whether an event happens as well as the odds when you place the bet) gives an overall bet on how likely an event is to happen. The more people tend to bet on a particular event, the more accurate the predictions tend to become. This can give a good picture of the certainty, at least so far as things look now (unsurprisingly the accuracy of the bets improves as the event approaches, but such is the case with all predictions). You can see how probable it is that various persons will get the Republican nomination for president in 2012 (Romney leads right now with 23.3%), who will win American Idol (James Durbin leads with 39.5%), whether the AV reform will pass in Britain (50% right now).

In fact, betting odds have been used in sports betting long before they were used by weather forecasters, leading to the surprising fact that we've long been much better at predicting the outcome of sporting events than we have at predicting the weather.

But the main point is that these predictions markets and this quantification of uncertainty provide much better models for predicting the future. The future will always be hazy and there will always be uncertainty, so whatever tools we can use to quantify that uncertainty will help us foresee future events with better clarity.

Prognostication