In this model the probabilities were measured by an ensemble of LLM's. We give them context and ask them to follow a specific methodology for making a forecast, and then aggregate their forecasts into a "crowd" forecast. We can get input from humans as well to compare/combine them further, but in this particular instance I just used our AI Forecaster.
You're right that a human analyst - especially a very experienced one - is going to be able to still do a better job of analysis most of the time, but the AI is useful for having loops to re-examine the question, consume and curate new information, etc. We've built the system with that assumption: let humans do what they're best at, let AI do what it's best at...
I think this comment encapsulates the fundamental state of AI at the moment.
The industry believes very strongly that this simulation of methodology is basically as good as the real thing (or at least "good enough" in most cases).
This belief is the fault line between the bullish and the bearish; it requires a leap of faith that not everyone is willing or capable of taking.
Every forecast question we ask has a verifiable outcome. We then score it based on what actually happens in reality and create a track record for all of it. In typical analysis, nothing is scored, and no one goes back and checks if it was right or wrong, but we do - whether we're scoring humans in human forecasting exercises, or seeing how AI does.
You're right that a human analyst - especially a very experienced one - is going to be able to still do a better job of analysis most of the time, but the AI is useful for having loops to re-examine the question, consume and curate new information, etc. We've built the system with that assumption: let humans do what they're best at, let AI do what it's best at...