Reference environment
A business workflow's purpose is to change database state. So the reward is: is the state correct afterwards?
Swivelbench is an open, synthetic environment for measuring whether an agent can complete a commercial-banking credit review that spans two systems which disagree with each other. Scoring is deterministic SQL run against the resulting databases. No LLM judge, no rubric, no expert consensus — assertions a compliance officer can read, evaluated in under a millisecond.
The problem
The hard part of enterprise work was never any single lookup.
It is that the same company exists in two systems under different keys, and correctness means both of them end consistent, with an audit trail. A credit analyst reads a number off the spreading tool, turns their chair, and retypes it into the core. That swivel is the workflow, and it is where the errors live.
In the seed task the borrower's tax ID is 94-3177119 in one system and its counterpart in the other has no TIN at all — it was lost in a core conversion. The record that does carry the matching TIN, and an exact match on the legal name, is an archived duplicate. A naive identifier join lands confidently on the wrong record. That is not a puzzle invented for a benchmark; it is what a legacy conversion leaves behind.
The agent gets no raw SQL. With raw SQL it writes the expected end state directly, scores 100%, and learns nothing. It works through a constrained typed surface — reads liberal, writes narrow — and every write is validated, typed, and traced. Illegal writes return a structured error rather than throwing, because the recovery behaviour is itself worth measuring.
The action layer validates types and existence, never policy. Writing to an archived obligor and testing a covenant on a matured facility are both permitted, and both caught by the verifier. If the API refused them, the negative assertions would be trivially satisfied and would measure nothing.
What is measured
Four kinds of assertion. Propagation is the one that carries the argument.
| Kind | Question | Share | Example |
|---|---|---|---|
| Positive | Did it do the thing? | 20% | The FY2025 ratio row exists on the audited statement, with the right value. |
| Propagation | Is it consistent across systems? | 30% | The leverage figure in the spreading tool and the tested value in the core are the same number. Tolerance 1e-9, not 5e-4. |
| Negative | Did it break anything else? | 35% | The archived duplicate is byte-identical. The matured facility was not tested. Zero deletes. |
| Trail | Is there evidence? | 15% | One audit entry per write, naming the same target key, in a controlled vocabulary. |
Each kind contributes a fixed share and assertions split their kind's share by their own weight. That is deliberate. With raw additive weights, adding a trap silently reweights the whole benchmark — ours drifted until the negative class was 46% of total weight simply because traps are easy to write. Normalising by kind keeps "add a task" and "change what the reward values" independent decisions.
Measured
Baselines, and what they cost us to believe.
| Model | Weights | Score | σ | Uncapped |
|---|---|---|---|---|
| meta/muse-spark-1.1 | closed | 100.0% | ±0.0 | 100% |
| openai/gpt-5.5 | closed | 100.0% | ±0.0 | 100% |
| anthropic/claude-opus-4.8 | closed | 100.0% | ±0.0 | 100% |
| qwen/qwen3-14b | open | 61.7% | ±36.7 | 88.7% |
| qwen/qwen3-8b | open | 30.0% | ±0.0 | 40.7% |
Frontier models saturate this instance. That is the honest headline. The environment was hardened four times — hiding record status from search, adding a superseded statement, a facility whose status flag and maturity date disagree, a prior-year figure the two systems record differently, two live duplicates requiring a tiebreak, and borrowers whose correct answer is to write almost nothing — and the frontier score went 100%, 100%, 97%, 100%, 100%. Adding another stated rule does not add difficulty; a careful reader clears any number of individually simple rules. The only lever that moved the number was horizon.
The environment discriminates sharply in the 8B–14B band, which is the band that matters for anything running on a customer's own hardware. qwen3-8b's failure is incompleteness rather than error: it resolves the obligor correctly, then attempts a fraction of the required writes. Notice its negative score stays at 90% — doing nothing is safe under negative assertions, which is why a completeness assertion had to be added and made disqualifying.
The gate
A reward you have not tried to fool is a reward you cannot trust.
Wrong end states are constructed by perturbing a correct rollout through the real action API, so each one is a state an agent could actually reach. Every one must score materially below a correct rollout, or the reward is blind.
| End state | Score | Uncapped |
|---|---|---|
| Correct rollout | 100.0% | 100.0% |
| Imputed a value policy declares uncomputable | 30.0% | 98.1% |
| Collateral damage to an untouched record | 30.0% | 97.3% |
| Skipped every escalation | 30.0% | 97.3% |
| Tested an ineligible facility | 30.0% | 96.9% |
| Used the superseded statement | 30.0% | 94.1% |
| Wrote to the archived record | 30.0% | 86.9% |
| No audit trail | 30.0% | 83.0% |
| Picked the wrong duplicate as master | 30.0% | 71.4% |
| Never propagated anything to system B | 30.0% | 61.8% |
Read the uncapped column. A rollout that imputes a value the policy declares uncomputable scores 98.1%. One that does the job perfectly and silently corrupts an adjacent record scores 97.3%. A weighted fraction cannot express "disqualifying." So a small set of assertions are marked critical and cap the reward when they fail — writing to a non-master record, testing an ineligible facility, mutating without logging, failing to escalate something beyond your authority.
That cap is right for a benchmark number and wrong for a training signal: a cliff edge means two rollouts differing by one row differ by sixty reward points, which is hard on advantage estimation. It is one flag, and both numbers are reported.
Honest limits
What is not built yet.
- The difficulty ladder
- One task family exists. The five-level L0–L4 parameterisation is not built, so there is no per-level breakdown yet.
- Snapshot and restore
- Each rollout rebuilds both databases from SQL. That is 3.97 ms, so it is not yet the bottleneck, but there is no snapshot API for GRPO sampling.
- Training and distillation
- The GRPO + LoRA script and the trajectory-distillation path are scaffolding targets, not written code. The on-prem claim depends on that chain existing; today it does not.
- A second domain
- One domain proves the idea. It does not prove the harness generalises. Until a second one exists, treat generality as a claim rather than a result.
- A model baseline at 14 borrowers
- The generated 14-borrower instance is verified by the scripted oracle and by the adversarial suite, but no frontier model has been scored against it yet.
The thing a credit analyst would object to
The fixed-charge coverage ratio is defined here as EBITDA over interest plus current maturities. No credit agreement uses that. A real FCCR nets cash taxes, unfinanced capex and distributions, and adding rent to the denominator; omitting them makes the covenant materially easier to pass. It is defined this way because the synthetic schema has no column for those inputs and the two coverage ratios needed to be distinguishable. Related: nobody downgrades a risk rating off a deterministic grid — that is committee judgment with qualitative overlays, and a breach-count-to-notches mapping is the most synthetic thing in the design. It has to be deterministic to be assertable. It is a benchmark convention, not credit policy, and we would rather say so here than have it found.
Provenance
Everything in it is invented.
Company names, customer identifiers, taxpayer numbers, financial statements, covenant thresholds, the rating grid and the ratio conventions were all constructed from generic commercial-lending domain knowledge. No customer data, no proprietary schema, no institution's credit policy language, and no real institution names. The databases are generated by a seeded script: --seed 1001 --borrowers 14 reproduces both databases and the assertion file byte-identically.