I trained a small transformer in 1.5hrs and it beats many LLMs
- ID
- 20377
- Status
- summarized
- Published
- 01 Sep 2026, 5:52 PM
- Fetched
- 03 Sep 2026, 5:13 PM
- Provider
- Hacker News
- Category
- dev-community
- Original URL
- https://mvakde.github.io/blog/44-on-arc-1/
- Source URL
- https://hnrss.org/best
Summary
- Score
- 7.5
- Created
- 03 Sep 2026, 6:21 PM
- Tags
- Audience
- developersai_ml_learners
What happened
Mithil Vakde trained a small transformer from scratch in 1.5 hours on a single 5090 GPU for 67 cents, scoring 44% on ARC-AGI-1—matching TRM/HRM and beating many LLMs. Key upgrades from his previous model include SwiGlu instead of GELU, RMSnorm instead of layernorm, scaling to 8 layers, more data diversity, and better shuffling. The approach uses test-time training with 3D RoPE embeddings, color/dihedral permutations, and AAIVR augmentation, and the code is open source.
Why it matters
If you're an AI/ML learner or builder, this demonstrates that sample efficiency—not scale—is a tractable problem worth working on, and that meaningful ARC-AGI results are achievable on a single GPU for under a dollar. The specific architecture choices (SwiGlu, RMSnorm, 3D RoPE, test-time training per puzzle) are concrete techniques you can experiment with directly using the open-source code.
Discussion angle
What this means for the sample efficiency debate: a from-scratch small transformer with test-time training matches specialized approaches on ARC-AGI for 67 cents—does this suggest the field is over-indexing on scale and synthetic data when architecture and training-time choices matter more?