Collinear AI opens CWE-bench v2 with 100 security tasks
🎧 Listen to this story
Collinear AI has opened CWE-bench v2 to the public, a benchmark with 100 tasks based on real open-source repositories. Agents must find and fix a security weakness without breaking existing functionality; evaluation is programmatic, deterministic and covers 91 categories of weaknesses (CWEs). The public set, grader and harness are open, although the dataset requires users to accept access conditions on Hugging Face.
In results reported by Collinear AI, Gemini 4 Argon leads 11 models with 67.8% pass@1 and 81.0% pass@4, ahead of Claude Opus 5.5. At least one model solved 95 tasks, but the leader fails 19% of the set on every attempt. DeepSeek V4.1 Flash (1017) reaches 57.0% pass@1 at $0.04 per rollout; Claude Opus 5.5 adds ten points at $1.83, while Gemini 4 Argon edges ahead at $3.92.
Why it matters · editorial interpretation
The public benchmark creates a reproducible reference for comparing agents that fix vulnerabilities without breaking existing code. Results reported by Collinear AI also highlight a trade-off between performance and cost: even the leader fails 19% of tasks on every attempt, while cheaper models trail behind.
Sources
ResearchSecurityCode