AI Weekly Malaysia

livenerf is an append-only, pre-registered benchmark built to test whether a frontier model quietly degrades after launch, and it started the clock on Claude Opus 5.5, released 2026-09-22. Day 1 ran 2026-09-24 22:10 UTC, roughly 2.5 days after launch, and it now samples once a day for 30 days: days 1-10 form the baseline, then two 10-day windows, so the first possible drift call lands around 2026-10-24 and the first Results row after day 20. It runs through headless Claude Code (claude -p) on a Claude Max subscription with no API key, using frozen prompts, a pinned CLI version, exact graders and raw logs, built on the UK AI Security Institute's Inspect framework with error bars per Anthropic's 'Adding Error Bars to Evals'; the repo has 366 stars and the Hacker News thread has 343 points and 147 comments.

Why: If you ship anything on Claude models, this is the closest thing to a day-0 baseline anyone has published, and it says plainly that sampling parameters are gone and thinking can't be turned off, so reproducibility has to come from pinning the CLI version, freezing prompts and keeping raw logs. The concrete decision: pin your model and CLI version in a file the way this repo does, log raw outputs now, and treat any post-launch quality claim as unproven until there are thousands of samples with error bars - not vibes. Note the timeline: no drift verdict exists before roughly 2026-10-24, so anything claiming Opus 5.5 was 'nerfed' before then is speculation.