A benchmark result is evidence about a test
A score is meaningful only with its model version, prompt template, tool access, dataset, sampling parameters, scorer and run date. Change any of these and the result may move. Public benchmarks are useful for comparison, but repeated optimization can contaminate them, and a high average can conceal severe failures in a minority subgroup or scenario. A laboratory should preserve the evaluation artifact, not only the headline: inputs, outputs, traces, code, environment and uncertainty.
The International AI Safety Report describes an evaluation gap: benchmark performance does not reliably predict real-world utility or risk. Deployment changes the distribution, incentives and consequences. Users may ask unexpected questions, tools expose new actions, and systems interact with organizational processes. Evaluation must therefore proceed from a use-case model: who uses the system, for what decision, with which data, under which constraints, and what happens when it is wrong.
Connect tests to a governance decision
NIST's AI Risk Management Framework separates four functions: Govern, Map, Measure and Manage. Governance establishes accountability, policies and risk tolerance. Mapping documents context and affected actors. Measurement analyzes capability, trustworthiness and impact. Management prioritizes treatment and decides whether development or deployment should proceed. The functions are continuous and interdependent; NIST explicitly warns that they are not an ordered checklist. A technically strong evaluation without an owner or action rule is unfinished governance.
Every critical evaluation should name its decision. Examples include release, expanded tool access, use with a new population, or continued operation after an incident. Before running the test, define the metric, uncertainty, threshold, escalation path and acceptable residual risk. This reduces the temptation to reinterpret a weak result after launch. When trade-offs exist, such as helpfulness versus refusal or accuracy versus privacy, the decision record should make them visible.
Build an evaluation portfolio, not a single gate
Capability tests establish what a system can do. Reliability tests examine consistency and calibration. Safety tests probe harmful outputs, misuse and loss of control. Security tests include prompt injection, data exfiltration and model or tool compromise. Fairness and accessibility tests evaluate performance across relevant groups and conditions. Human-factors tests examine reliance, comprehension and override. Operational tests cover latency, cost, observability and recovery. None can stand in for the others.
The UK AI Security Institute's Inspect framework supports datasets, solvers, scorers, tools, sandboxes and agents, and provides more than 200 implemented evaluations. The number demonstrates tooling breadth, not assurance by volume. A small set selected from a rigorous risk model can be more informative than hundreds of unrelated benchmarks. The portfolio should include hidden or refreshed cases, adversarial tests and realistic end-to-end workflows to reduce overfitting and capture interactions between the model and its environment.
Use incident evidence without overstating it
Stanford's 2026 AI Index reports that the AI Incident Database recorded 362 incidents in 2025, compared with 233 in 2024. The increase is an observation in a documented database, not a population incidence rate. Adoption, media attention, reporting practice and classification can all change the count. The OECD's monitor similarly discloses that public reports represent only a subset and are processed through a defined pipeline. Serious governance uses incident data to find patterns and test controls, while preserving these limits.
Organizations need their own reporting taxonomy: actual harm, near miss, policy violation, security event, quality escape and user complaint should not be collapsed into one number. Capture model and application versions, context, affected parties, detection route, severity, containment and recurrence. Link incidents back to the evaluation portfolio. If a deployment failure was not represented in pre-release tests, add a reproducible case and examine why monitoring or escalation failed.
Regulation sets floors, not a complete measurement system
The EU AI Act uses cumulative training computation above 10^25 floating-point operations as an initial presumption of high-impact capability for a general-purpose AI model, while allowing the threshold and complementary indicators to evolve. Compute is administrable and knowable before training finishes, but it is only a proxy. Algorithmic efficiency, modality, tool access, autonomy, reach and deployment conditions also influence risk. Compliance with a compute threshold does not establish that a model is safe, nor does falling below it establish low risk.
A research laboratory should maintain a crosswalk from legal duties and standards to technical evidence without confusing the two. Documentation, logging, impact assessment and human oversight can be regulatory requirements; the evaluation methods that demonstrate their effectiveness remain use-case specific. The NIST Generative AI Profile provides risk-management actions tailored to generative systems, while international standards and sector rules may add obligations. The evidence register should identify which claim each test supports.
Operate evaluation as a living control system
Pre-release testing is a baseline. After deployment, monitor input and output distributions, refusal and override patterns, error severity, tool actions, latency, cost and user-reported harm. Set triggers for investigation when drift or incidents exceed thresholds. Use canary releases, shadow evaluation and rollback where feasible. Preserve privacy by collecting only the evidence needed and controlling access to sensitive traces. Independent review is particularly important where developers have incentives to ship.
Report results with confidence intervals or repeated-run variation where applicable, subgroup breakdowns, known blind spots and failed tests. Distinguish measured results from projections and policy targets. A mature evaluation report should allow a reviewer to answer five questions: what was tested, under which conditions, against what threshold, with what uncertainty, and what decision followed. That is the point at which evaluation stops being a leaderboard and becomes operational research infrastructure.
Scope and limitations
Incident databases are affected by reporting and coverage bias, while benchmark results are sensitive to implementation. The EU compute threshold is a legal presumption rather than a scientific boundary. No general framework removes the need for domain-specific expertise, affected-party engagement, independent review and ongoing evidence from the deployed context.
References
Source review: 20 August 2026. Quantitative values retain their original definitions, periods, and boundaries.
- 01Artificial Intelligence Risk Management Framework 1.0
National Institute of Standards and Technology · 2023
doi.org ↗ - 02Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
National Institute of Standards and Technology · 2024
doi.org ↗ - 03Inspect: An open-source framework for large language model evaluations
UK AI Security Institute · 2026
www.aisi.gov.uk ↗ - 04AILuminate Safety Methodology
MLCommons · 2025
mlcommons.org ↗ - 05Regulation (EU) 2024/1689: Artificial Intelligence Act
European Union · 2024
eur-lex.europa.eu ↗ - 06Responsible AI: 2026 AI Index Report
Stanford Institute for Human-Centered AI · 2026
hai.stanford.edu ↗ - 07International AI Safety Report 2026
International AI Safety Report · 2026
internationalaisafetyreport.org ↗ - 08Overview and methodology of the AI Incidents and Hazards Monitor
OECD.AI · 2026
oecd.ai ↗