← AI Switchboard
SAFETY · November 14, 2026
Nov 13

Anthropic says the same Claude model scored up to 34 points apart on its safety tests depending only on where it ran, in a guide to building safety evaluations

By the AI Switchboard · researched and written with AI from the sources listed on this page, each point checked against its source before publishing · how we work · report a mistake

The guide sets out ten practices and uses Anthropic's own results to show that the setup, not the model, can move a safety score.

Anthropic published “Building strong safety evaluations” on 13 November, a guide by eight of its staff that sets out ten practices for designing, running and reporting AI safety tests. It uses Anthropic's own results to show how fragile safety scores can be.

On multi-turn safety tests of Claude Fable 5.1, the share of appropriate responses differed between the API and the claude.ai app by as much as 34 percentage points, the guide says, though only the setup changed, not the model. On a suicide and self-harm test the same model scored 60% on the API and 94% on claude.ai with a new system prompt.

Two figures the guide sets side by side and declines to rank: on a romance-scam test Claude Opus 5 scored 98% and Opus 5.5 scored 94%, and the guide says the gap is not statistically significant. It says the single-turn harmless-response rates of its recent models all sit between 94% and 98%, which makes those tests saturated and useful mainly for catching regressions. It says confidence intervals on multi-turn tests can exceed 10 percentage points either way.

The guide recommends at least 100 items for a complex multi-turn test and 1,000 for a simple single-turn one, checking automated graders against human experts, and reporting uncertainty with every score.

  • Confirmed The gap between the API and claude.ai on multi-turn safety tests of Claude Fable 5.1 was as high as 34 percentage points. Anthropic
  • Confirmed On a suicide and self-harm test the model scored 60% on the API and 94% on claude.ai with the new system prompt. Anthropic
  • Confirmed Single-turn harmless-response rates for Anthropic's recent models are between 94% and 98%. Anthropic

Safety, security & governanceScience & research

Today in the November 14, 2026 edition · front page