AI Model Testing Goes Public as Benchmarks Multiply
Source: Every (jonathangreene@gmail.com)
Frontier labs have long guarded proprietary evaluations while publishing flashy benchmark wins. Now both Astra and Fable 5.1 are acing graduate science exams with no public methodology attached, forcing the field toward standardized, auditable testing that developers can't game. Evaluation opacity has become a competitive moat—claims of capability mean little without reproducible proof. As models proliferate into real applications in medicine, finance, and science, regulators and customers will demand access to how these systems were actually tested. Nobody can credibly compare systems or defend deployment decisions when evals remain proprietary black boxes.