#benchmarks

A developer's DIY evaluation tool reveals gaps in off-the-shelf benchmarks for specialized use cases.

IBM Research and Hugging Face introduce ScarfBench, a benchmark evaluating AI agents on framework migration tasks that require working, deployable code—not just syntactic translation.

OpenAI released GeneBench-Pro, a benchmark of 10 case studies designed to evaluate AI models on practical genomics reasoning tasks.

OpenAI introduces GeneBench-Pro, a 129-problem benchmark measuring whether AI models can make higher-order scientific judgments in genomics and translational medicine.

Researchers use a classic mystery game to test how well AI agents reason through multi-step deduction and suspect elimination.

OpenAI released LifeSciBench, a benchmark designed to measure whether AI systems can handle real-world life science research workflows, not just answer biology questions.

A new benchmark reveals that even the most capable AI systems struggle with diagnosing complex infrastructure failures, scoring below 50% on Site Reliability Engineering scenarios.

A Reuters review of federal AI procurement records found Grok appeared in only 3 of 400+ documented government uses, compared to 230+ for OpenAI and dozens for Anthropic and Google.

Google unveiled Gemini 3.5 Flash at I/O 2026 as an agent-first model claiming frontier intelligence at sub-flagship latency, with Gemini Omni adding physics-aware video generation.

A new benchmarking framework evaluates complete AI agent systems—not just models—across six diverse tasks, reporting both quality and cost metrics for practical deployment decisions.

A GitHub repository curates datasets for LLM fine-tuning, instruction tuning, and benchmarking across medical, NLP, multimodal, and code domains.