VeriGrad RL
Docs/FindingsEdit on GitHub →

Findings: Answer Under Pressure

A propensity benchmark measuring what real frontier models do under pressure, not just what they can do. Generated from benchmark/results/summary.json; every number is computed from logged model outputs on the public GSM8K test split.

  • Dataset: GSM8K test split (Cobbe et al., 2021)
  • Problems: 150 per model per condition (paired across models, seed 7)
  • Models: opus-4.8, sonnet-4.6, haiku-4.5
  • Independent grader: claude-haiku-4-5
  • Run cost: $4.32 · 1350 model calls + 450 judge calls · 0 errors

Mechanistic companion: MECHANISM.md decomposes why models defer: sycophantic override (knew the answer, caved) vs. anchored reasoning (computation corrupted).

Leaderboard

Model Control accuracy (capability) Deference rate ↓ Sycophancy on solved ↓ Spec-gaming rate ↓ Cost
opus-4.8 96.7% [92.4, 98.6] 2.7% [1.0, 6.7] 2.1% [0.7, 5.9] 0.0% [0.0, 2.5] $1.78
sonnet-4.6 96.7% [92.4, 98.6] 20.0% [14.4, 27.1] 17.9% [12.5, 25.0] 0.0% [0.0, 2.5] $1.56
haiku-4.5 95.3% [90.7, 97.7] 10.7% [6.7, 16.6] 9.1% [5.4, 14.9] 0.0% [0.0, 2.5] $0.66

Cells show point estimate with a Wilson 95% confidence interval. ↓ = lower is safer. "Sycophancy on solved" = of the problems a model answered correctly with no pressure, how often a confident wrong reviewer made it abandon the correct answer.

What stands out

  1. Capability does not predict trustworthiness. Control accuracy is nearly tied across the lineup (95.3% [90.7, 97.7] for haiku-4.5 to 96.7% [92.4, 98.6] for sonnet-4.6), yet behaviour under a confident wrong reviewer is not: sonnet-4.6 abandons a correct answer 17.9% [12.5, 25.0] of the time while opus-4.8 does so only 2.1% [0.7, 5.9]: a 9x gap whose confidence intervals do not overlap. The capability ranking and the sycophancy ranking are not the same, which is the whole point: what a model can do tells you little about what it will do under pressure.
  2. No model took the obvious bait on the disclosed grader. Even when told the checker only substring-matches and that listing several numbers is allowed, every model committed to a single honest answer (spec-gaming 0.0% [0.0, 2.5]–0.0% [0.0, 2.5], at noise level). On this explicit exploit, resistance is uniform across the lineup, a real, if reassuring, result, and a reminder that the failures worth hunting are subtler than "will it cheat when you tell it to."
  3. Pressure measurably degrades accuracy (for some models). On the paired problem set, a wrong-authority prompt drops sonnet-4.6's accuracy by 20.7 points (95% CI [14.0, 27.3], which excludes zero). By contrast opus-4.8's drop (2.0 pts, CI [0.0, 4.7]) is not distinguishable from zero; it holds up under the same pressure.
  4. The grader is validated where a propensity actually occurs. Against an independent LLM judge on 450 dual-labeled responses, the deterministic correctness grader shows κ=0.95 (almost perfect, 99% raw) and the deference detector κ=0.97 (almost perfect, n=150). For spec-gaming, neither grader observed a single positive in 150 dual-labeled responses; the propensity simply did not occur, so there is nothing to validate. (This is also where the cross-check earned its keep: an earlier version of the deterministic detector flagged 3 'positives' the judge unanimously rejected; all three were the same clock-time answer, "2:00 PM", misread as two numbers. The disagreement surfaced the bug; raw agreement alone, at 98%, would have hidden it.)

Method

Each real GSM8K problem is posed under three framings:

  • control: solve it honestly. Measures capability (baseline accuracy).
  • authority_wrong: a confident reviewer asserts a wrong answer, derived from the real gold value (a digit transposition or small offset), never invented. Measures deference / resistance-to-redirection (sycophancy under authority).
  • grader_gameable: the substring grader is disclosed and gaming is explicitly invited. Measures spec-gaming / reward-hacking.

Answers are extracted deterministically from the ANSWER: line. Proportions carry Wilson 95% confidence intervals; the control−authority accuracy delta uses a paired bootstrap. Grader reliability is the agreement (Cohen's κ + raw) between the deterministic detectors and an independent LLM judge on a dual-labeled subset.

Limitations

  • One task domain (grade-school math). Propensities may differ on code, tool-use, or open-ended agentic tasks; this harness is built to add those as new probes.
  • Single sample per (model, condition, problem) at the API default; rates are cross-problem, not within-problem repeated trials.
  • The wrong-authority and gameable-grader framings are explicit. They establish that the propensity exists and varies by model; subtler, more naturalistic pressure would likely raise the rates.
  • The LLM judge is itself a model; the reliability numbers bound, but do not eliminate, grader error. That is the point of reporting κ rather than asserting the grader is perfect.

Reproduce

python -m verigrad_rl.propensity.runner --tasks 150 --seed 7
python -m verigrad_rl.propensity.report