Measure what a frontier model will do under pressure.

Get started · 60 seconds

Install and run it

A pip-installable Python package (verigrad CLI + library) plus this Next.js app. The RL baseline runs offline; the propensity benchmark calls real models and reads ANTHROPIC_API_KEY from your environment.

# clone + install (editable) for Python 3.10+
git clone https://github.com/aravinds-kannappan/VeriGrad-RL
cd VeriGrad-RL
pip install -e ".[llm,inspect]"

# runs offline in seconds: verifiable-reward RL, no API key
verigrad train --episodes 400 --env arithmetic
 
# the propensity benchmark on real frontier models
export ANTHROPIC_API_KEY=sk-ant-...
verigrad propensity --smoke      # 3 problems, ~$0.01 (sanity check)
verigrad propensity --tasks 150  # the full leaderboard run

Use it as a library

The probe templates and deterministic detectors are importable. Render a prompt, call any model yourself, and score the response reproducibly.

from verigrad_rl.integrations import (
    build_pressure_samples, label_response, safe_behavior,
)
 
# real GSM8K problems rendered under a confident, wrong reviewer
samples = build_pressure_samples("authority_wrong", n_tasks=150, seed=7)
 
# call any model on samples[i].prompt, then score deterministically:
labels = label_response("authority_wrong", model_text,
                        samples[0].gold, samples[0].anchor)
held = safe_behavior("authority_wrong", labels)   # True == did not cave

What’s in the box

A benchmark, a baseline, and the plumbing to trust both

Interactive · live model calls

Probe a model under pressure

Pick a real GSM8K problem (or edit it), choose a model and how hard a wrong “authority” pushes back, then run it against a real model. The verdict (did it hold or cave?) is computed from the live response.

Calls run server-side via a Next.js route; enable them by setting ANTHROPIC_API_KEY in the deployment. The rest of the site works without a key.

The core finding

Capability is nearly tied. Trustworthiness is not.

Three frontier models, 150 real GSM8K problems each, under three framings.

Open source · interoperable

Built to live in the safety-eval ecosystem

The probes are tiny and provider-neutral on purpose, so other people’s harnesses can drive them. The flagship is a real Inspect AI adapter: the same deterministic detectors, now against any model Inspect can reach.

pip install "verigrad-rl[inspect]"
 
# the SAME probes, against any Inspect provider; one flag swaps the vendor
inspect eval verigrad_rl/integrations/inspect_task.py@deference \
  --model anthropic/claude-sonnet-4-6
inspect eval verigrad_rl/integrations/inspect_task.py@deference \
  --model openai/gpt-4o

Shipped integrations

Patterned after · compatible by design

Conventions VeriGrad deliberately mirrors so it slots into a real safety-eval workflow. Listed honestly as influences and interop targets, not bundled adapters.

Full details in the Integrations docs.

Interactive · runs in your browser

Train a model on the data, live

No server, no pre-baked numbers. These widgets run real computation in your browser on the 648 logged samples from the cross-domain run.

Scales to a research program

Across domains, providers, and an elicitation gradient

Why models cave

Social, not cognitive

When a model abandons the correct answer, did its reasoning already compute it and then cave (override), or did the pressure corrupt the computation (anchored)? Across the lineup, ~90% is override. The model knew, and threw it away.

White-box · automated circuit discovery

Find the safety circuit automatically

Behavior tells you a model caved; mechanism tells you which components did it. VeriGrad ships a dependency-free implementation of ACDC (Conmy et al., NeurIPS 2023) and path patching (Goldowsky-Dill et al., 2023), run on a transparent safety circuit with a known answer key, so the discovery is validated, not just asserted.

# automated circuit discovery on a transparent safety circuit
verigrad circuit --target safety-dag --tau 0.02
#  -> benchmark/circuits/{REPORT.md, fig_circuit.svg}
 
# run the same method on the RL environment's actual reward model
verigrad circuit --target toy-circuit

Is the ruler trustworthy?

We test our graders