Superforecasting: The Art and Science of Prediction

This is actually a repeated problem that Philip Tetlock found when he was researching forecasting.

He's the guy that wrote Super Forecasting.

And what he found is that relatively educated amateurs who were not biased in a certain way based on their prior work were just as good as or better than these sorts of experts in predicting the future.

I think one problem that historians have in particular is because they usually think of things in a linear, causal way.

And so that's a great book on thinking about how a bunch of different really successful investors do that in different ways.

Super Forecasting, The Art and Science of Prediction by Phil Tetlock and Dan Gardner.

This is the Bible of forecast.

If you're going to be forecasting anything and you have not read this book, trust me, your forecasts are going to suck.

[instrumental music]Tarek Mansour, the CEO of Kalshi, calls Robin Hanson's work on prediction markets foundational, but he especially likes to cite the work of Philip Tetlock, a University of Pennsylvania psychologist who you may have heard on this show in the past talking about the folly of prediction.

Tetlock argued in his book Superforecasting that when it comes to predicting the future, even an expert can be beat by a thoughtful amateur.

Here's Mansour again.

He put a bunch of experts on a specific domain, on a specific topic like geopolitics, against a bunch of non-experts, kind of random people, where the condition was sort of intellectually curious, read the news.

So we would never, for example, want to induce an actor to adversely impact some outcome or some other person or to cause harm because we have a market that is live.

That's something actually we take very seriously.

We have specific rule book provisions that carve out what we do in cases of death.

We might have markets on what somebody might say in a speech, or whether somebody might attend an event, or what somebody might say at that event.

I'm going to go first with the recommendations because mine's related to the theme we're discussing today.

It's a book called Superforecasting, The Art and Science of Prediction.

It's by Philip Tetlock and Dan Gardner.

Now, Philip Tetlock's probably a familiar name.

In fact, it was not that much better than throwing a dart at a board.

Now that's not to say in present company that political expertise is pointless, right? It's just a humbling note about the limits of how far we can predict.

Now the follow-up book is the one I mentioned, Superforecasting the Art and Science of Prediction, in which Tetlock summarises a lot of the work he did subsequent to that first book, in which he tried to find methods for improving uh the the methodology of decision excuse me of of making forecasts and making predictions and he in fact set up separate teams uh using i believe funding from uh iapa which is the intelligence equivalent of darpa the defense advanced research projects agency so the united states actually invested money the united states intelligence agencies invested money in trying to refine the art of making political predictions and super forecasting is one of the results the book is one of the results of that long study that Tetlock did and the conclusion I came away from from this book is that at the margin if you apply the right methodology if you apply good systems and good techniques of thinking you can improve your ability to make political predictions by this much.

But even Tedlock and Gardner conclude in their book that once you get out to around five or 10 years, it's vanishingly difficult to make a difference with these specific techniques.

We have a wealth of human reasoning on Metaculus that, that bots can, can look at and other platforms.

So they're learning from how humans are doing this, and they're-- they've read Superforecasting, and they can forecast much faster than humans.

So they can forecast many, many questions for, whatever, like eight cents a question at worst, I think.

And yeah, so bots are catching up to the best humans, and they're certainly cheaper than the best humans.

This seems simple but can have serious consequences.

For example, in their book Superforecasting, Tetlock and Gardner recount a major miss by Bill Flack, one of their superforecasters, in a tournament to predict the odds of various geopolitical events.

The question at hand was whether Japanese Prime Minister Shinzo Abe would visit the controversial Yasukuni Shrine, a site revered by conservative nationalists like Abe because it reveres Japan's war heroes, but that also often brings international blowback due to the inclusion of accused war criminals among the honor's dead.

After strategically weighing the pros and cons for Abe, Flack decided that the bad outweighed the good and predicted that Abe would make the sensible choice not to go.