Import AI 481: AI sub-contractors; a wet lab for AI; and subterfuge during development
- ID
- 47050
- Status
- summarized
- Published
- 16 Nov 2026, 10:01 PM
- Fetched
- 16 Nov 2026, 11:37 PM
- Provider
- Import AI
- Category
- research-analysis
- Original URL
- https://importai.substack.com/p/import-ai-481-ai-sub-contractors
- Source URL
- https://importai.substack.com/feed
Summary
- Score
- 7.0
- Created
- 16 Nov 2026, 11:38 PM
- Tags
- Audience
- developersai_ml_learnersai_agent_users
What happened
Import AI 481 covers FrontierML, a benchmark of 21 long-horizon ML tasks (LLM training, efficiency, vision & audio, computer use, applied NLP, math & games) where agents get a 12-hour CPU sandbox, a wallet of $10/$50/$200, a few validation samples, and a task description, then submit an inference server scored on a held-out out-of-distribution test set on an air-gapped H100. Given $10, a human researcher baseline gains 56 points versus 45.4 for the best system, Opus 5.5; at $200 the human reaches 66.1 versus 58.8 for Opus 5.5, and the top open-weight model listed is Kimi K3.1 Pro (text cuts off there). The write-up also notes that in 28% of trajectories agents added newly trained components to pretrained models or trained a model from scratch.
Why it matters
The gap between a human researcher and an agent at $200 of compute is 66.1 vs 58.8 points — close enough that a $200 experiment is a reasonable way to test whether an agent can produce a usable fine-tune before you hire or buy tooling. Because 28% of trajectories involved training new models or bolting new trained parts onto pretrained ones, budget for GPU hours and data, not just API tokens, and note the eval design: limited validation samples plus a hidden out-of-distribution test set is the anti-overfitting pattern to copy in your own internal benchmarks.
Discussion angle
If an agent can convert $200 into roughly 89% of a human researcher's measured gain, does 'agent compute budget' become a line item next to salaries — and what would you need to see in the held-out eval before trusting the resulting model in production?