CWE-bench v2: More coverage, quality, and transparency
- ID
- 42580
- Status
- summarized
- Published
- 03 Nov 2026, 8:00 AM
- Fetched
- 04 Nov 2026, 1:31 AM
- Provider
- Hugging Face Blog
- Category
- developer-ai
- Original URL
- https://huggingface.co/blog/cwe-bench
- Source URL
- https://huggingface.co/blog/feed.xml
Summary
- Score
- 6.5
- Created
- 04 Nov 2026, 1:32 AM
- Tags
- Audience
- developersai_ml_learnersai_agent_users
What happened
Collinear AI published CWE-bench v2 on Hugging Face: 100 public tasks spanning 91 distinct CWE classes, where an agent gets a checkout of a real open-source repo plus a short instruction to find and fix a security weakness without breaking existing functionality. Scoring is fully programmatic and deterministic, every task was hand-reviewed, and verifiers are built to fail memorised solutions such as the known upstream patch. Across 11 evaluated models, Gemini 4 Argon leads at 67.8% pass@1 and 81.0% pass@4; a held-out set is available only by emailing cyber@collinear.ai.
Why it matters
The 67.8% top pass@1 is the number to plan around: if you wire a coding agent into security-fix work, roughly one in three realistic single-weakness fixes still fails on this set, so a human review step is not optional. The verifier design matters just as much — because checks reject the known upstream patch and grade behaviour rather than one specific diff, a model scoring well here is not just reciting fixes it saw in training, which is the failure mode you'd actually hit when pointing an agent at your own private repo. Note the asymmetry: you can run the 100 public tasks yourself for free, but the held-out set requires contacting Collinear.
Discussion angle
Ask what pass@1 on a 91-CWE benchmark you'd require before letting an agent open security-fix PRs in CI — and whether 'verifiers fail the upstream patch' is a strong enough guard against training-data memorisation, or just a guard against one specific leak.