Can Frontier Models See Past a Decoy? Elorian Tests Visual Reasoning
- ID
- 40575
- Status
- summarized
- Published
- 30 Oct 2026, 12:00 AM
- Fetched
- 29 Oct 2026, 6:20 AM
- Provider
- Latent Space
- Category
- developer-ai
- Original URL
- https://latent.space/p/elorian/
- Source URL
- https://www.latent.space/feed
Summary
- Score
- 6.8
- Created
- 29 Oct 2026, 6:20 AM
- Tags
- Audience
- developersai_ml_learnersai_agent_users
What happened
Elorian launched Glimpse, a visual reasoning model it describes as reasoning about the physical world from what it sees, and co-released a DecoyCount benchmark with Mercor to test whether models genuinely count objects or rely on visual shortcuts. In the accompanying paper, frontier models scored below 50% accuracy, with GPT-6 Astra highest at 43.17%; Elorian CEO Andrew Dai said Glimpse scored 50.48% in an internal test, though Glimpse is not mentioned in the paper. The paper concludes that stronger general reasoning does not by itself generalize to aligned visual reasoning.
Why it matters
If you are adding vision to agents or apps, do not treat strong general model reasoning as proof it can handle counting or visual-substitution traps: the best frontier score cited here was 43.17%, and Glimpse's 50.48% is an internal result not included in the paper. Run task-specific evals on your own images before switching models or trusting a vendor benchmark.
Discussion angle
Elorian co-released the benchmark and its model's 50.48% result is internal rather than in the paper — what eval hygiene should builders demand before acting on vendor-published benchmark scores?