Allen AI's new TutorMoments framework exposes how LLMs over-help students instead of fostering productive struggle—a critical flaw in AI tutoring systems.
A developer's DIY evaluation tool reveals gaps in off-the-shelf benchmarks for specialized use cases.
Intelligence, the company behind DesignArena, closed a seed round to expand its human-evaluation platform, now generating $60M ARR from frontier labs seeking design-space feedback.
Researchers identify how LLM-based evaluation can systematize AI model assessment while exposing vulnerabilities in Cohen's kappa and similar aggregate metrics.
OpenAI released GeneBench-Pro, a benchmark of 10 case studies designed to evaluate AI models on practical genomics reasoning tasks.
EEE and Community Evals now interoperate, unifying scattered evaluation results across 229K+ benchmark runs into a single standardized schema.
The stress-testing startup lands backing from Greenfield Partners to expand its simulated environments for autonomous AI systems.
The first open far-field ASR benchmark measures speech recognition performance across simulated rooms with realistic acoustics, revealing significant performance gaps in noisy conditions.
OpenAI released LifeSciBench, a benchmark designed to measure whether AI systems can handle real-world life science research workflows, not just answer biology questions.
Allen AI's olmo-eval extends the OLMES benchmark standard with flexible, composable evaluation infrastructure for model development iterations.
OpenAI shares lessons on designing trustworthy third-party evaluations for frontier AI models, emphasizing the role of task environments and validity checks.
A new benchmark reveals that even the most capable AI systems struggle with diagnosing complex infrastructure failures, scoring below 50% on Site Reliability Engineering scenarios.
A new benchmarking framework evaluates complete AI agent systems—not just models—across six diverse tasks, reporting both quality and cost metrics for practical deployment decisions.
Hugging Face introduces private ASR evaluation datasets from Appen Inc. and DataoceanAI to block benchmaxxing, with scores visible via an opt-in toggle.