What is the best harness for each model?
- ID
- 47614
- Status
- summarized
- Published
- 17 Nov 2026, 10:53 PM
- Fetched
- 18 Nov 2026, 8:04 AM
- Provider
- Hacker News
- Category
- dev-community
- Original URL
- https://runta.com/blog/introducing-frontierharness-eval-v1-1/
- Source URL
- https://hnrss.org/best
Summary
- Score
- 7.0
- Created
- 18 Nov 2026, 8:04 AM
- Tags
- Audience
- developersai_ml_learnersai_agent_users
What happened
Runta (with Baseten serving models) published FrontierHarness Eval v1.1, a leaderboard of 22 harness configurations × 5 models × 62 terminal tasks × 4 attempts — 83B tokens and 27,280 agent runs, 76× the size of v1.0's single-model Kimi K3 run. The headline claim: the best-value harness changes with the model, so switching models should prompt reconsidering your harness. Best-value picks were Codex CLI with GLM-5.3, Pi with GLM-5.3 Flash, ZCode with DeepSeek V4.1 Flash 1017, Oh My Pi with Muse, and Alma with Kimi-K3; by cost per task the median harness shifted 11 of 22 places across models, and Codex CLI went from 1st with GLM-5.3 to 16th with Kimi-K3.
Why it matters
If you benchmark models while holding your harness fixed, this says the ranking you get may not transfer — the same harness swung from cheapest to 16th of 22 depending on the model behind it, and the spread between harnesses was up to 1.6× in cost per task (OpenCode v2 lowest at $0.561/task; ZCode top pass rate at 61.9%). Practically: when you A/B a new model, re-run it through the harness you'll actually ship, and treat the 'best-value' labels here as directional only — they come from Runta's own leaderboard and its own rule (lowest cost per pass, unless a rival costs ≤30% more per pass and passes ≥5 points more), which is a judgement call, not a law.
Discussion angle
The 30%-cost-for-5-points rule is arbitrary — what switching rule would you actually use for your workload, and does harness/model coupling mean eval suites should report harness×model pairs rather than single-axis rankings?