AI Weekly Malaysia

Back to items Summaries

What is the best harness for each model?

ID
47614
Status
summarized
Published
17 Nov 2026, 10:53 PM
Fetched
18 Nov 2026, 8:04 AM
Provider
Hacker News
Category
dev-community
Original URL
https://runta.com/blog/introducing-frontierharness-eval-v1-1/
Source URL
https://hnrss.org/best

Summary

Score
7.0
Created
18 Nov 2026, 8:04 AM
Tags
Audience
developersai_ml_learnersai_agent_users

What happened

Runta (with Baseten serving models) published FrontierHarness Eval v1.1, a leaderboard of 22 harness configurations × 5 models × 62 terminal tasks × 4 attempts — 83B tokens and 27,280 agent runs, 76× the size of v1.0's single-model Kimi K3 run. The headline claim: the best-value harness changes with the model, so switching models should prompt reconsidering your harness. Best-value picks were Codex CLI with GLM-5.3, Pi with GLM-5.3 Flash, ZCode with DeepSeek V4.1 Flash 1017, Oh My Pi with Muse, and Alma with Kimi-K3; by cost per task the median harness shifted 11 of 22 places across models, and Codex CLI went from 1st with GLM-5.3 to 16th with Kimi-K3.

Why it matters

If you benchmark models while holding your harness fixed, this says the ranking you get may not transfer — the same harness swung from cheapest to 16th of 22 depending on the model behind it, and the spread between harnesses was up to 1.6× in cost per task (OpenCode v2 lowest at $0.561/task; ZCode top pass rate at 61.9%). Practically: when you A/B a new model, re-run it through the harness you'll actually ship, and treat the 'best-value' labels here as directional only — they come from Runta's own leaderboard and its own rule (lowest cost per pass, unless a rival costs ≤30% more per pass and passes ≥5 points more), which is a judgement call, not a law.

Discussion angle

The 30%-cost-for-5-points rule is arbitrary — what switching rule would you actually use for your workload, and does harness/model coupling mean eval suites should report harness×model pairs rather than single-axis rankings?

Top