How enabling two settings tripled our scores on the ARC-AGI-3 benchmark
- ID
- 8924
- Status
- summarized
- Published
- 29 Jul 2026, 11:00 PM
- Fetched
- 30 Jul 2026, 7:17 AM
- Provider
- OpenAI News
- Category
- ai-labs
- Original URL
- https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores
- Source URL
- https://openai.com/news/rss.xml
Summary
- Score
- 7.0
- Created
- 31 Jul 2026, 4:35 PM
- Tags
- Audience
- ai_agent_usersai_ml_learnersdevelopers
What happened
OpenAI reports that GPT-5.6 Sol scored only 7.8% on the ARC-AGI-3 benchmark using the official harness, but solved all six puzzle levels when using their Responses API harness instead. The two settings that made the difference were retaining reasoning across steps and enabling compaction—essentially letting the agent remember what it has done and manage its context window more effectively.
Why it matters
If you are building AI agents that solve multi-step problems, this is concrete evidence that enabling reasoning retention and context compaction in your harness can be the difference between total failure and full success. Check whether your agent framework preserves intermediate reasoning state between steps or discards it, and whether compaction is available—these are not cosmetic settings.
Discussion angle
The official harness scored 7.8% while OpenAI's own harness solved all six levels—how much of this is a real architectural insight about agent memory versus a vendor gaming a benchmark on their own infrastructure?