Research

CWE-bench v2 launches with 100 open-source vulnerability tasks, Gemini 4 Argon leads at 67.8% pass@1

Collinear AI releases CWE-bench v2, a public benchmark of 100 cybersecurity tasks spanning 91 vulnerability classes, with full reproducibility and programmatic scoring.

Last verified:

CWE-bench v2: A Reproducible Cybersecurity Benchmark

According to the Hugging Face Blog, Collinear AI has released CWE-bench v2, a public benchmark of 100 vulnerability-detection tasks designed to measure frontier AI capabilities in real-world cybersecurity scenarios. The benchmark spans 91 distinct Common Weakness Enumeration (CWE) classes, requiring models to identify and patch security flaws in authentic open-source repositories without introducing regressions or breaking existing functionality.

The release includes fully programmatic, deterministic scoring that enables reproducibility without marginal evaluation cost. Every task underwent manual review and curated verification checks, with Collinear AI specifically validating that instructions read like authentic security prompts (not contrived evaluation artifacts), that verifiers check for correct behavior rather than specific code patterns, and that solutions do not simply match known upstream patches.

Leaderboard Results and Performance Distribution

Google DeepMind’s Gemini 4 Argon leads the benchmark at 67.8% pass@1 accuracy and 81.0% pass@4, narrowly outpacing Anthropic’s Claude Opus 5.5 (67.3% pass@1, 78.0% pass@4). According to the benchmark results, 95 of the 100 tasks are solvable by at least one model, demonstrating the set’s realism; however, the leading model still fails 19% of tasks across all four generation attempts, signaling that the long-tail of vulnerability classes remains largely unsolved.

DeepSeek V4.1 Flash achieved competitive pass@1 scores while maintaining substantially lower per-token inference costs compared to closed-model competitors, according to the benchmark’s cost analysis. This efficiency-to-performance tradeoff highlights diverging optimization strategies across frontier model vendors: some favor absolute capability, others prioritize inference economics.

Transparency and Community Contribution

The public dataset, grading harness, and evaluation infrastructure are open-sourced and available on Hugging Face Hub. Open-weight models are tracked on the CWE-bench dataset page, where results auto-update via pull requests; closed-API models (Gemini, Claude, GPT) are tracked on a separate leaderboard Space. A held-out evaluation set remains withheld to allow Collinear AI to continue tracking frontier progress on unseen tasks; models can access this private evaluation by contacting cyber@collinear.ai.

Why This Matters

CWE-bench v2 addresses a critical gap in LLM evaluation: the absence of a reproducible, high-fidelity benchmark for autonomous vulnerability detection. As AI agents move from research artifacts to production security tooling, benchmarks that require real-world reasoning (multi-step vulnerability discovery, defensive thinking, regression prevention) become essential for enterprise adoption decisions. The 67–68% ceiling at pass@1 among frontier models suggests that autonomous patch generation still requires human review, shaping procurement strategies for teams evaluating AI-assisted security workflows. The public release also creates accountability: vendors can no longer claim superiority without submitting to a unified evaluation harness, accelerating the transition from anecdotal capability claims to measurable benchmarks in cybersecurity AI.

Frequently Asked Questions

What is CWE-bench v2 testing?

CWE-bench v2 evaluates an AI agent's ability to identify and fix security vulnerabilities in real open-source code repositories. The agent receives a repository checkout and a natural-language instruction, then must locate and remediate a weakness without breaking existing functionality.

How is scoring performed?

Scoring is fully programmatic and deterministic, allowing any model to be evaluated without additional cost. All checks were curated with LLM judge panels to balance hardness and fairness, and results are reproducible.

Why does the leaderboard show 95 solved tasks when there are 100 total?

95 tasks are solvable by at least one model in the evaluation set, indicating the benchmark's coverage is realistic. The remaining 5 appear to remain unsolved by current frontier models, representing the frontier's edge.

#cybersecurity #benchmarking #vulnerability detection #LLM evaluation #code agents