AI Weekly Malaysia

Dan Luu critiques a widely-cited claim that dynamic/concise languages like Clojure or J are 2-3x more token-efficient for LLM coding agents than static languages like Rust or Go. He argues the benchmarks rely on trivial Rosetta Code problems (70-109 token solutions) where performance doesn't generalize, and notes methodological flaws in supporting comparisons, including a symlink bug that corrupted test results.

Why: Don't choose your stack based on token-efficiency benchmarks from toy problems; if you're deciding between Python and Rust for an AI-assisted codebase, token cost on trivial tasks is not evidence of real-world agent performance. If you care about token efficiency, run your own eval on problems representative of your actual workload before committing.

JetBrains CTO Vladislav Tankov describes how his team evaluates frontier LLMs against private repositories, including their monorepo, rather than trusting public benchmark scores. Claude Fable 5 posted a 44.3% Python pass rate in JetBrains' suite versus 28.2% for Opus 4.8, solving 18 tasks Opus missed while losing only 2, and despite higher per-token cost, delivered lower cost per task on complex long-running work.

Why: If you're shipping AI-assisted coding features, JetBrains' approach is a concrete template: build eval sets on your own private codebase, track separate leaderboards for quality/cost-per-task/speed, and measure cost-per-task (not per-token) because a more expensive model can be cheaper on complex work. The 16-point pass-rate gap between Fable 5 and Opus 4.8 on real code is large enough to justify re-evaluating your current model choice.