AI Agents Still Lag Human Fantasy Soccer Managers, But Prediction Skills Are Rapidly Improving
A Wired experiment deploying multi-agent AI systems to manage a Premier League fantasy team reveals progress—and persistent gaps—in real-world forecasting.
Last verified:
Multi-Agent Prediction Performance Lags Human Judgment—For Now
A Wired experiment using Codex to orchestrate multiple AI agents for Premier League fantasy soccer team management reveals a nuanced picture: while agent-based systems now incorporate far richer real-world information than earlier attempts, they still make decisions a seasoned human would reject. According to Wired, the initial Codex-managed roster suggested benching Erling Haaland, one of the league’s most prolific scorers—a decision that illustrates how even sophisticated information pipelines can miss obvious context.
The experimental setup evolved from a simpler tool the author tested years prior. That earlier approach, sourced from the UK’s Turing Institute, had modeled historical fantasy league data but ignored external signals like player injuries and coaching changes. The new iteration, built with Codex, fundamentally changes the architecture: rather than a single prediction engine, the author deployed a swarm of specialized agents. Some crawled news and blogs; others analyzed recent match results; more ran predictive models; and a final cohort critiqued suggestions before synthesis. This multi-agent design proved substantially better than the baseline—but scaling beyond roughly a dozen agents yielded diminishing returns, with over 100 agents adding computational cost without meaningful gains.
Frontier Models Show Promise in Market Prediction—Outperforming Humans
The forecast for AI prediction capability improves sharply at the frontier. According to research by Haifeng Xu, a computer scientist at the University of Chicago studying AI in market forecasting, newer models are proving surprisingly adept at commercial prediction tasks. Xu and colleagues began testing frontier models on real-world forecasting questions—who would win the NBA Championship, which world leaders President Trump would meet—drawn from prediction markets like Kalshi and Polymarket.
The results surprised even the researchers. Wired reports that, according to Xu, models performed “really well” and sometimes surpassed average human forecasters without explicit guidance. In the comparative testing, GPT-5 ranked highest, followed by Grok 4, Claude Sonnet, Gemini, and Llama. This work ultimately led to the development of Prophet Arena, a benchmark for evaluating model forecasting skill.
Xu’s framing of the mechanism is instructive: “AI is just getting smarter in understanding the world—I feel like that’s the fundamental reason they’re improving.” The ability to consume and synthesize large volumes of structured and unstructured real-world data, combined with orchestration of complex multi-step reasoning, drives improvement.
Why This Matters
The gap between human and AI forecasting is narrowing in constrained, information-rich domains. Fantasy soccer—or prediction markets more broadly—sit at an inflection point: AI systems can now gather and analyze more relevant signals than individual humans, yet they remain vulnerable to high-level context and judgment calls that humans find trivial. For investors, product teams, and prediction-market participants, this suggests two near-term implications. First, hybrid systems (human judgment + AI signal aggregation) likely outperform either alone. Second, as frontier models improve at multi-step reasoning and real-world grounding, purely AI-driven forecasting systems may become competitive with—and eventually exceed—human performance in domains where prediction depends on information synthesis rather than embodied intuition.
The fantasy soccer experiment underscores that capability scaling does not guarantee usability; the author’s multi-agent system showed that more agents ≠ smarter decisions. This pattern will repeat in production forecasting systems until alignment between agent incentives and user objectives improves.
Frequently Asked Questions
How did the AI agent system perform compared to human managers?
The multi-agent setup showed marked improvement over earlier attempts but still made suboptimal decisions, such as benching high-performing players like Erling Haaland. The system excels at information aggregation but lags in contextual judgment.
Which AI models perform best at forecasting?
According to University of Chicago research, GPT-5 ranked first, followed by Grok 4, Claude Sonnet, Gemini, and Llama. Some models outperformed average humans without special instructions, though independent benchmarking is ongoing.
Can scaling the number of agents improve predictions?
Testing with over 100 agents did not meaningfully improve suggestions beyond a smaller multi-agent configuration, indicating diminishing returns after a certain agent count.