AI Weekly Malaysia

Back to items Summaries

Import AI 481: AI sub-contractors; a wet lab for AI; and subterfuge during development

ID
47050
Status
summarized
Published
16 Nov 2026, 10:01 PM
Fetched
16 Nov 2026, 11:37 PM
Provider
Import AI
Category
research-analysis
Original URL
https://importai.substack.com/p/import-ai-481-ai-sub-contractors
Source URL
https://importai.substack.com/feed

Summary

Score
7.0
Created
16 Nov 2026, 11:38 PM
Tags
Audience
developersai_ml_learnersai_agent_users

What happened

Import AI 481 covers FrontierML, a benchmark of 21 long-horizon ML tasks (LLM training, efficiency, vision & audio, computer use, applied NLP, math & games) where agents get a 12-hour CPU sandbox, a wallet of $10/$50/$200, a few validation samples, and a task description, then submit an inference server scored on a held-out out-of-distribution test set on an air-gapped H100. Given $10, a human researcher baseline gains 56 points versus 45.4 for the best system, Opus 5.5; at $200 the human reaches 66.1 versus 58.8 for Opus 5.5, and the top open-weight model listed is Kimi K3.1 Pro (text cuts off there). The write-up also notes that in 28% of trajectories agents added newly trained components to pretrained models or trained a model from scratch.

Why it matters

The gap between a human researcher and an agent at $200 of compute is 66.1 vs 58.8 points — close enough that a $200 experiment is a reasonable way to test whether an agent can produce a usable fine-tune before you hire or buy tooling. Because 28% of trajectories involved training new models or bolting new trained parts onto pretrained ones, budget for GPU hours and data, not just API tokens, and note the eval design: limited validation samples plus a hidden out-of-distribution test set is the anti-overfitting pattern to copy in your own internal benchmarks.

Discussion angle

If an agent can convert $200 into roughly 89% of a human researcher's measured gain, does 'agent compute budget' become a line item next to salaries — and what would you need to see in the held-out eval before trusting the resulting model in production?

Top