GPT-5.6 Sol's ARC-AGI-3 score jumps from 13.3% to 38.3% with retained reasoning and token compaction

OpenAI identified a critical source of noise in frontier model benchmarking: the gap between how models are evaluated under generic test harnesses versus optimized production configurations. According to the OpenAI Blog, enabling two settings—retained reasoning (which preserves chain-of-thought across actions) and compaction (which eliminates redundant token output)—increased GPT-5.6 Sol’s performance on the ARC-AGI-3 benchmark from 13.3% to 38.3% on the public task set, while simultaneously reducing output tokens by a factor of 6. This nearly three-fold improvement occurred without any model weight changes, suggesting the original low score reflected harness constraints rather than capability gaps.

The ARC-AGI-3 benchmark and its intentional minimalism

ARC-AGI-3 is designed to measure reasoning and generalization by presenting agents with unfamiliar 2D puzzle games and requiring them to infer gameplay rules without explicit instruction. According to the OpenAI Blog, the benchmark deliberately uses a sparse harness—no special tools, no model-specific optimizations—reasoning that this approach makes model shortcomings visible and enables fair cross-model comparison. The official leaderboard reported that no frontier model solved any puzzle beyond the first level; with retained reasoning enabled, GPT-5.6 Sol solved all six levels of the public set. Human testers, for comparison, scored approximately 48% on average using the same evaluation metric (Relative Human Action Efficiency).

How harness design masked model capability

OpenAI discovered two specific failure points in the standard harness. First, the harness discarded all private reasoning after each game action, forcing GPT-5.6 Sol to reconstruct the game state and strategy from scratch with each move, despite having access to past move history. Second, a rolling truncation window made older actions invisible as the history grew, further fragmenting the model’s world model. According to the OpenAI Blog, GPT-5.6 Sol could see a record of past moves and brief notes but could not access the plans, insights, or thoughts that produced them. Under these conditions, even though GPT-5.6 Sol has previously solved mathematics conjectures and beaten games like Pokémon FireRed, it appeared to struggle fundamentally with 2D puzzle inference.

Why This Matters

This finding reframes how the AI research community should interpret frontier model benchmarks. A score of 13.3% versus 38.3% represents not a model defect but a harness-model mismatch. For teams designing evaluations, the implication is clear: benchmark results reflect the joint system of model, configuration, and test harness design. The discovery also suggests that generic harnesses, while useful for exposing raw model limitations, may underestimate capability when a model’s reasoning process depends on persistent internal state. Organizations conducting comparative benchmarking against GPT-5.6 Sol should account for such configuration differences, or risk drawing incorrect conclusions about relative model strength. For the ARC-AGI competition itself, the result raises a question about whether the benchmark’s minimalism serves its goal of fair comparison, or whether it optimizes for measuring harness design rather than reasoning ability.