Skip to main content
A XORCISE run scores one agent on one mission. A playbook runs many models through many missions and puts the results side by side, so you can answer a question a single run cannot: of the models available to me, which is actually better at this kind of work? The harness is fixed — every contestant runs under the same sandboxed OpenHands CLI — so what you are comparing is models, not agent scaffolds. This is delivered by xorcise-playbooks, a skill in the xorcise-skills repository. It is not part of XORCISE itself: it drives your own install through the REST API and nothing more.
Experimental, and it spends real money. A benchmark is a matrix of full agent sessions, each billed by your model provider, plus a judge grading for every one of them. The skill shows you the matrix and waits for an explicit yes before anything runs — but the smallest preset is 9 runs per model and the largest is 150. It also sits outside XORCISE’s own release process, so it is not covered by the compatibility promises the rest of this documentation makes.
XORCISE eval card showing a summary panel, a leaderboard of four models ranked by mean overall score, and a model-by-mission heatmap

An eval card from xor-pb-001-broad_coverage — 4 models × 15 missions × 3 attempts, 180 graded runs, judged by claude-sonnet-4-5, run 29–30 July 2026. One dated result under one set of conditions, not a standing leaderboard.

​
What you actually do

Most of this page is detail you can ignore on a first read. The skill is conversational: your agent reads it and does the work, asking you only what it cannot determine itself.
1

Install the skill

Clone the repository and point your agent at SKILL.md.
2

Say you're ready

The skill opens with a welcome and stops. It touches nothing on your machine until you say go.
3

Answer a handful of questions

Which playbook, which models, and how many runs in parallel. It fills in the rest from your install.
4

Approve the spend

You get the run matrix and a ballpark cost, and nothing starts until you say yes.
5

Read the eval card

Hours later for a large playbook, minutes for the smoke test. The report is a single local HTML file.
Everything below exists so you can check the skill’s work, tune a long run, or drive the scripts yourself. You do not need it to get a first result.

​
How a benchmark works

Each cell of the matrix is an ordinary XORCISE run. The skill creates it over the REST API, launches one disposable Docker sandbox that joins the mission network and drives the model, waits for the grade, and records the row. Nothing about grading changes: every run is still scored 0.5 × deterministic + 0.5 × judge, exactly as How grading works describes. Two roles matter, and they are configured in different places:
  • Contestants are the models under test. You give the skill a short name and a litellm model string for each, for this run only.
  • The judge is configured on your XORCISE install, not in the skill — it is the same judge that grades any other run. The skill only verifies it answers. Self-grading is allowed: a contestant may also be the judge.
With no judge configured on XORCISE the judge half scores 0, so every contestant is capped at 50% and the benchmark ranks them on the deterministic half alone. Configure a judge before you spend anything on a matrix.

​
What it costs

The matrix is the whole cost model:
Each run is a full agent session that may use up to its budget_seconds of model time, and agents usually use most of it. Each run then costs one grading, which is itself one judge call per rubric criterion — what grading costs sets out how that scales with transcript size. There is no cost estimator — XORCISE reports token counts but leaves the cost field unpopulated, as Roadmap notes — so treat every figure here as a shape, not a price. The four presets, sized per model: Multiply by your model count. Two models on broad coverage is 300 runs and 300 gradings; the eval card above took 44 hours of wall-clock for 180 runs.
Run xor-pb-000-smoke first, always. Nine runs per model is cheap, it proves the whole pipeline end to end, and it gives you a measured per-run cost to extrapolate from instead of a guess.

​
What you need

XORCISE running

Reachable over REST, with a judge model configured. The skill never installs, starts, or reconfigures it.

Docker

Permission to build images and run containers, plus /dev/net/tun — sandboxes join the mission network in TUN mode.

Python 3

With PyYAML and litellm for the runner and the connectivity check.

A long-lived API key

One per provider you are benchmarking. Short-lived session tokens expire mid-matrix.
One read-only script checks all of it and prints a fix for anything missing:
It verifies XORCISE answers, Docker is up, the sandbox image exists — building it is a later step, so missing is fine — the Python packages are importable, a judge is configured, and /dev/net/tun is present. Warnings are safe to proceed past. The skill assumes http://127.0.0.1:3001 and only asks where XORCISE is if that check fails. If yours is elsewhere, set this once in the shell you run everything from, before the preflight:
That one value flows everywhere — mission browsing and each sandbox’s route back to run control both derive from it. Setting it late is the cause of the most confusing failure mode there is: runs get created correctly, then every one of them fails because the sandbox is still dialling the old default.

​
Install

xorcise-playbooks/SKILL.md is a plain Markdown skill file. Point whichever skill-aware agent you use at it — however that agent loads skills — and run it from inside the xorcise-playbooks directory, since every path in it is relative to the skill root.

​
Your first benchmark

This is the smoke run as the skill performs it. You do not type these — they are here so you can follow along, and so you can reproduce a run without the skill later. The model map names each contestant. Copy the template somewhere temporary, because nothing about a run is meant to persist:
Each entry maps a short name — the name the playbook and --models use — to a full litellm string:
Provider keys never go on a command line. Either export them in your shell before the run, or have a helper scaffold a credentials file — it reads your model map, writes exactly the variable names those contestants need, chmod 600s it, and opens it in your editor to fill in:
Either way the runner loads the keys mechanically and forwards them to sandboxes by name — it never prints a key value or puts one on a command line. Nothing carries between runs, and the file is yours to keep, rotate, or delete. Then a dress rehearsal — one tiny call per contestant, XORCISE live-testing its own judge, and a reachability check on the OTLP endpoint:
Fix every FAIL before spending. A model whose provider blocks offensive-security prompts shows up here as content_filter, which is much cheaper to discover now than 80 runs in. Build the sandbox image if the preflight said it was missing, then rehearse the orchestration with no keys and no spend:
And launch:
--otel is not optional. XORCISE reconstructs and grades the agent transcript from its telemetry, so a run with telemetry off produces nothing to grade and scores near 0. There is no way to tell those apart from genuinely bad runs after the fact. See Send traces and logs for what the stream carries.
Delete $RUN_DIR when you are done with it.

​
Reading the eval card

The card is one self-contained HTML file, in nine sections: a Summary of the matrix and its totals; a Leaderboard ranked by mean overall, with pass@1, pass@k, standard deviation, flags and tokens; a Model × Mission heatmap; per-model cards splitting the deterministic and judge halves; a Missions table showing which missions were hard for everyone; Consistency, which reports how far apart a cell’s attempts landed; Methodology and conditions; Run notes, which is where excluded runs and small-n caveats are stated; and Reproducibility. It scopes itself to whatever your run actually covered, so a custom playbook renders its own grid — the same code that draws a fifteen-mission generalist board draws an eight-mission red-team one:
XORCISE eval card for the offensive playbook, showing five models ranked by mean overall and an eight-column model-by-mission heatmap

The same card from xor-pb-002-offensive_core — 5 models × 8 missions × 3 attempts, 120 graded runs, judged by gpt-5.4, run 31 July 2026. Same layout as the card above, scoped to a different playbook.

Labels come from manifest.json, written alongside the results; without it the card falls back to labels derived from the run data rather than showing stale values. The numbers behind it live in the output directory, and these are the source of truth:
Keep the card local. It can contain mission flags and other spoilers, so publishing it externally both leaks answers and degrades the missions for everyone else. The screenshot at the top of this page is a deliberate exception that was reviewed line by line before publication — treat your own cards as private by default.
Before you treat a card as final, re-run the casualties. A run that made zero tool calls never really started, and is excluded from scores rather than counted as a zero:
This touches only the failed and ungraded cells and gives up after --max-attempts, so a persistently dead model is left out instead of retried forever — which is the correct outcome, not a bug.

​
Going bigger

​
Build your own playbook

A custom playbook composes missions already installed or available on your platform. The skill cannot author missions; for that, see Author a mission.
--as-playbook prints a ready-to-paste missions: block. Filters are --specialty, --proficiency, --type (lab or static), and --installed-only. Ask your own library what it holds rather than working from a list here — the taxonomy has changed before, and --list-specialties prints the current values with a mission count for each. Paste the block into a copy of a preset and set the rest:
Leave intel_policy: none for a clean capability comparison — disclosing authored intel helps the agent and makes your numbers incomparable with anyone else’s. See Runs for what intel is.

​
Choose a concurrency

--concurrency is how many attempts run at once. The intuitive limit is memory — each sandbox is about 400 MB — but that is rarely what breaks first. The real ceiling is mission provisioning. Every parallel run asks the core to bring up a mission container, and too many starting at once can exceed its readiness window and wedge the worker pool: runs pile up and the core stops answering, well before you run out of RAM.
  • Start at 3, which is the default, and raise it only if the core stays healthy. On a small host use 1–2; past about 4–6 you need both spare memory and core headroom.
  • For bursty starts, give the core more room before launching: export XORCISE_READINESS_TIMEOUT_SECONDS=240 (the default is 90).
  • The runner already staggers sandbox starts about 8 seconds apart so their telemetry inits do not collide. You do not need to do anything about that.

​
Survive a long run

A large matrix runs for hours, and the tooling assumes something will go wrong. Check progress any time, safely, against a run in flight — it reads only that run’s own files:
Resume after a crash, an OOM, or a reboot by re-running the identical command with --resume. Every completed run is checkpointed to progress.jsonl, so it skips what is already graded and finishes the rest. Nothing is lost and nothing is repeated. Detach — run under tmux or a systemd unit so a dropped SSH session cannot kill the matrix. On very long runs you can also let a core wedge self-heal:

​
Clean up

The runner tears down each sandbox as its run finishes, so there is usually nothing left. To sweep stragglers, scope the filter to this run’s session tag:
Never use a bare name=xorcise-run- filter. Without the session tag it matches every benchmark sandbox on the host — other playbooks, other people’s live runs — and deletes them. The session tag is what makes a playbook safe to run alongside anything else.
Leave mission containers alone. The core owns them and reaps them once your runs go terminal; removing them by hand while other runs are live breaks those runs.

​
Running the scripts yourself

Nine scripts, all driveable directly if you want a benchmark in CI or on a schedule rather than in a conversation. run_playbook.py is the only one with a real surface: A legacy --no-sandbox flag also exists, which runs agents directly on your host with no isolation. Don’t use it.

​
Limits and gotchas

Content filtering is normal. Some providers block offensive-security prompts for some models. The run returns no output and makes zero tool calls, shows as tool_calls=0, and is excluded from scores rather than counted as a failure. If a model is consistently filtered on the offensive playbooks it cannot compete there — drop it. Grades arrive late. Grading can land seconds to minutes after a run goes terminal, and the runner waits up to about five minutes for a real grade. Rows with overall=None usually mean the grade never landed or the transcript was too large for the judge; raising the judge’s token ceiling or the per-run budget is the fix. Small n is small. Three runs per cell is a pilot. Read a cell as mean ± standard deviation, not a point estimate, and be careful comparing two models three points apart. One judge carries its own biases. Half of every score comes from a single model reading a rubric. That is the same trade-off any XORCISE run makes, and How grading works sets out its limits — a leaderboard just multiplies it. Use long-lived keys. A session token that expires mid-matrix can wedge grading. On Bedrock prefer a long-term key in AWS_BEARER_TOKEN_BEDROCK over STS credentials. The skill ships reference/TROUBLESHOOTING.md with the full list, including the symptom-to-cause table worth skimming before a large run.