> ## Documentation Index
> Fetch the complete documentation index at: https://docs.xorcise.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluate Agents with Playbooks
> Run several models through the same missions under one fixed harness, and get a single eval card ranking them — using the xorcise-playbooks skill.
A XORCISE run scores one agent on one mission. A **playbook** runs many models through many missions and puts the results side by side, so you can answer a question a single run cannot: of the models available to me, which is actually better at this kind of work?
The harness is fixed — every contestant runs under the same sandboxed OpenHands CLI — so what you are comparing is *models*, not agent scaffolds. This is delivered by `xorcise-playbooks`, a skill in the [xorcise-skills](https://github.com/xorcise-ai/xorcise-skills) repository. It is not part of XORCISE itself: it drives your own install through the [REST API](/reference/api) and nothing more.
**Experimental, and it spends real money.** A benchmark is a matrix of full agent sessions, each billed by your model provider, plus a judge grading for every one of them. The skill shows you the matrix and waits for an explicit yes before anything runs — but the smallest preset is 9 runs per model and the largest is 150. It also sits outside XORCISE's own release process, so it is not covered by the compatibility promises the rest of this documentation makes.
## What you actually do
Most of this page is detail you can ignore on a first read. The skill is conversational: your agent reads it and does the work, asking you only what it cannot determine itself.
Clone the repository and point your agent at `SKILL.md`.
The skill opens with a welcome and stops. It touches nothing on your machine until you say go.
Which playbook, which models, and how many runs in parallel. It fills in the rest from your install.
You get the run matrix and a ballpark cost, and nothing starts until you say yes.
Hours later for a large playbook, minutes for the smoke test. The report is a single local HTML file.
Everything below exists so you can check the skill's work, tune a long run, or drive the scripts yourself. You do not need it to get a first result.
## How a benchmark works
```mermaid theme={"theme":"css-variables"}
%%{init: {'theme':'base','themeVariables':{'fontFamily':'JetBrains Mono, ui-monospace, SFMono-Regular, Consolas, monospace','fontSize':'12px','primaryColor':'#232323','primaryTextColor':'#f2ead6','primaryBorderColor':'#3a3a3a','secondaryColor':'#1a1a1a','tertiaryColor':'#141414','background':'#1a1a1a','mainBkg':'#232323','nodeBorder':'#3a3a3a','lineColor':'#6e6144','textColor':'#c7bb9f','clusterBkg':'transparent','clusterBorder':'#2f2f2f','edgeLabelBackground':'#1a1a1a','actorBkg':'#232323','actorBorder':'#3a3a3a','actorTextColor':'#f2ead6','actorLineColor':'#6e6144','signalColor':'#6e6144','signalTextColor':'#c7bb9f','labelBoxBkgColor':'#232323','labelBoxBorderColor':'#3a3a3a','labelTextColor':'#f2ead6','noteBkgColor':'#141414','noteTextColor':'#c7bb9f','noteBorderColor':'#3a3a3a','sequenceNumberColor':'#0a0805'},'flowchart':{'padding':26,'nodeSpacing':40,'rankSpacing':46},'sequence':{'useMaxWidth':true}}}%%
flowchart TD
P["Playbook
missions × runs-each"] --> M["One cell per
model × mission × run"]
M --> S["A throwaway sandbox
per cell"]
S --> R["A normal XORCISE run,
graded as usual"]
R --> C["Eval card
+ summary.csv"]
```
Each cell of the matrix is an ordinary XORCISE run. The skill creates it over the REST API, launches one disposable Docker sandbox that joins the mission network and drives the model, waits for the grade, and records the row. Nothing about grading changes: every run is still scored `0.5 × deterministic + 0.5 × judge`, exactly as [How grading works](/concepts/grading) describes.
Two roles matter, and they are configured in different places:
* **Contestants** are the models under test. You give the skill a short name and a litellm model string for each, for this run only.
* **The judge** is configured on your XORCISE install, not in the skill — it is the same judge that grades any other run. The skill only verifies it answers. Self-grading is allowed: a contestant may also be the judge.
With no judge configured on XORCISE the judge half scores 0, so every contestant is capped at 50% and the benchmark ranks them on the deterministic half alone. Configure a judge before you spend anything on a matrix.
## What it costs
The matrix is the whole cost model:
```text theme={"theme":"css-variables"}
total runs = models × missions × runs-per-mission
```
Each run is a full agent session that may use up to its `budget_seconds` of model time, and agents usually use most of it. Each run then costs one grading, which is itself one judge call per rubric criterion — [what grading costs](/concepts/grading#what-grading-costs) sets out how that scales with transcript size. There is no cost estimator — XORCISE reports token counts but leaves the cost field unpopulated, as [Roadmap](/roadmap) notes — so treat every figure here as a shape, not a price.
The four presets, sized **per model**:
| Preset | What it measures | Scale per model |
| --------------------------- | ---------------------------------------------------------------------- | ----------------------- |
| `xor-pb-000-smoke` | Pipeline shakeout across a three-rung difficulty spread. Start here. | 3 missions × 3 runs = 9 |
| `xor-pb-001-broad_coverage` | Fifteen missions spanning the library — the generalist leaderboard. | 15 × 10 = 150 |
| `xor-pb-002-offensive_core` | Red-team focus: exploitation, web, crypto, reversing. | 8 × 10 = 80 |
| `xor-pb-003-defensive_blue` | Blue-team focus: forensics, detection, network defense, secure coding. | 9 × 10 = 90 |
Multiply by your model count. Two models on broad coverage is 300 runs and 300 gradings; the eval card above took 44 hours of wall-clock for 180 runs.
Run `xor-pb-000-smoke` first, always. Nine runs per model is cheap, it proves the whole pipeline end to end, and it gives you a measured per-run cost to extrapolate from instead of a guess.
## What you need
Reachable over REST, with a judge model configured. The skill never installs, starts, or reconfigures it.
Permission to build images and run containers, plus `/dev/net/tun` — sandboxes join the mission network in TUN mode.
With PyYAML and litellm for the runner and the connectivity check.
One per provider you are benchmarking. Short-lived session tokens expire mid-matrix.
One read-only script checks all of it and prints a fix for anything missing:
```bash theme={"theme":"css-variables"}
bash scripts/preflight.sh
```
It verifies XORCISE answers, Docker is up, the sandbox image exists — building it is a later step, so missing is fine — the Python packages are importable, a judge is configured, and `/dev/net/tun` is present. Warnings are safe to proceed past.
The skill assumes `http://127.0.0.1:3001` and only asks where XORCISE is if that check fails. If yours is elsewhere, set this once in the shell you run everything from, before the preflight:
```bash theme={"theme":"css-variables"}
export XORCISE_BASE=http://:
export XORCISE_OTLP_PORT= # only if OTLP is not on 4318
```
That one value flows everywhere — mission browsing and each sandbox's route back to run control both derive from it. Setting it late is the cause of the most confusing failure mode there is: runs get created correctly, then every one of them fails because the sandbox is still dialling the old default.
## Install
```bash theme={"theme":"css-variables"}
git clone https://github.com/xorcise-ai/xorcise-skills.git
```
`xorcise-playbooks/SKILL.md` is a plain Markdown skill file. Point whichever skill-aware agent you use at it — however that agent loads skills — and run it from inside the `xorcise-playbooks` directory, since every path in it is relative to the skill root.
## Your first benchmark
This is the smoke run as the skill performs it. You do not type these — they are here so you can follow along, and so you can reproduce a run without the skill later.
The model map names each contestant. Copy the template somewhere temporary, because nothing about a run is meant to persist:
```bash theme={"theme":"css-variables"}
RUN_DIR="$(mktemp -d)"
MODELS_YAML="$RUN_DIR/agent-models.yaml"
cp templates/agent-models.example.yaml "$MODELS_YAML"
```
Each entry maps a short name — the name the playbook and `--models` use — to a full litellm string:
```yaml theme={"theme":"css-variables"}
model_list:
- model_name: gpt5
litellm_params: {model: openai/gpt-5}
- model_name: sonnet
litellm_params: {model: anthropic/claude-sonnet-4-6}
```
| Provider | Example litellm string |
| ------------- | ----------------------------------------------- |
| OpenAI | `openai/gpt-5` |
| Anthropic | `anthropic/claude-sonnet-4-6` |
| AWS Bedrock | `bedrock/us.anthropic.claude-sonnet-4-6` |
| Google Gemini | `gemini/gemini-2.5-pro` |
| OpenRouter | `openrouter/meta-llama/llama-3.1-405b-instruct` |
Provider keys never go on a command line. Either export them in your shell before the run, or have a helper scaffold a credentials file — it reads your model map, writes exactly the variable names those contestants need, `chmod 600`s it, and opens it in your editor to fill in:
```bash theme={"theme":"css-variables"}
RUN_CREDS="$RUN_DIR/creds.env"
bash scripts/setup-creds.sh "$RUN_CREDS" --models "$MODELS_YAML"
```
Either way the runner loads the keys mechanically and forwards them to sandboxes **by name** — it never prints a key value or puts one on a command line. Nothing carries between runs, and the file is yours to keep, rotate, or delete.
Then a dress rehearsal — one tiny call per contestant, XORCISE live-testing its own judge, and a reachability check on the OTLP endpoint:
```bash theme={"theme":"css-variables"}
python3 scripts/check-connectivity.py --llm-config "$MODELS_YAML" --creds "$RUN_CREDS"
```
Fix every `FAIL` before spending. A model whose provider blocks offensive-security prompts shows up here as `content_filter`, which is much cheaper to discover now than 80 runs in.
Build the sandbox image if the preflight said it was missing, then rehearse the orchestration with no keys and no spend:
```bash theme={"theme":"css-variables"}
docker build -t xorcise-agent-sandbox:latest sandbox/
python3 scripts/run_playbook.py playbooks/xor-pb-000-smoke.yaml --dry-run
```
And launch:
```bash theme={"theme":"css-variables"}
python3 scripts/run_playbook.py playbooks/xor-pb-000-smoke.yaml \
--otel \
--llm-config "$MODELS_YAML" \
--creds "$RUN_CREDS" \
--models gpt5,sonnet \
--outdir ./xorcise-results/xor-pb-000-smoke \
--concurrency 3
```
**`--otel` is not optional.** XORCISE reconstructs and grades the agent transcript from its telemetry, so a run with telemetry off produces nothing to grade and scores near 0. There is no way to tell those apart from genuinely bad runs after the fact. See [Send traces and logs](/guides/telemetry) for what the stream carries.
Delete `$RUN_DIR` when you are done with it.
## Reading the eval card
```bash theme={"theme":"css-variables"}
python3 scripts/gen_report.py \
--data ./xorcise-results/xor-pb-000-smoke/data \
--out ./xorcise-results/xor-pb-000-smoke/report.html
```
The card is one self-contained HTML file, in nine sections: a **Summary** of the matrix and its totals; a **Leaderboard** ranked by mean overall, with pass\@1, pass\@k, standard deviation, flags and tokens; a **Model × Mission heatmap**; per-model cards splitting the deterministic and judge halves; a **Missions** table showing which missions were hard for everyone; **Consistency**, which reports how far apart a cell's attempts landed; **Methodology and conditions**; **Run notes**, which is where excluded runs and small-*n* caveats are stated; and **Reproducibility**.
It scopes itself to whatever your run actually covered, so a custom playbook renders its own grid — the same code that draws a fifteen-mission generalist board draws an eight-mission red-team one:
Labels come from `manifest.json`, written alongside the results; without it the card falls back to labels derived from the run data rather than showing stale values.
The numbers behind it live in the output directory, and these are the source of truth:
| File | What it holds |
| ---------------- | ---------------------------------------------------------------- |
| `summary.csv` | One row per run — the file to load into anything else |
| `rows.json` | The same data with full structure |
| `manifest.json` | Models, providers, mission specialties, judge, run configuration |
| `progress.jsonl` | The checkpoint `--resume` reads |
| `session.txt` | This run's session tag, used to scope cleanup |
**Keep the card local.** It can contain mission flags and other spoilers, so publishing it externally both leaks answers and degrades the missions for everyone else. The screenshot at the top of this page is a deliberate exception that was reviewed line by line before publication — treat your own cards as private by default.
Before you treat a card as final, re-run the casualties. A run that made zero tool calls never really started, and is excluded from scores rather than counted as a zero:
```bash theme={"theme":"css-variables"}
python3 scripts/run_playbook.py --rerun-failed --max-attempts 2
```
This touches only the failed and ungraded cells and gives up after `--max-attempts`, so a persistently dead model is left out instead of retried forever — which is the correct outcome, not a bug.
## Going bigger
### Build your own playbook
A custom playbook composes missions **already installed or available on your platform**. The skill cannot author missions; for that, see [Author a mission](/guides/author-a-mission).
```bash theme={"theme":"css-variables"}
python3 scripts/list-missions.py --list-specialties
python3 scripts/list-missions.py --specialty Penetration --proficiency Expert --as-playbook
```
`--as-playbook` prints a ready-to-paste `missions:` block. Filters are `--specialty`, `--proficiency`, `--type` (`lab` or `static`), and `--installed-only`.
Ask your own library what it holds rather than working from a list here — the taxonomy has changed before, and `--list-specialties` prints the current values with a mission count for each.
Paste the block into a copy of a preset and set the rest:
```yaml theme={"theme":"css-variables"}
name: my-playbook
harness: openhands
budget_seconds: 1200
runs_per_mission: 10
intel_policy: none
missions:
- sparse-signal
- definer-cascade
```
Leave `intel_policy: none` for a clean capability comparison — disclosing authored intel helps the agent and makes your numbers incomparable with anyone else's. See [Runs](/concepts/runs) for what intel is.
### Choose a concurrency
`--concurrency` is how many attempts run at once. The intuitive limit is memory — each sandbox is about 400 MB — but that is rarely what breaks first.
The real ceiling is **mission provisioning**. Every parallel run asks the core to bring up a mission container, and too many starting at once can exceed its readiness window and wedge the worker pool: runs pile up and the core stops answering, well before you run out of RAM.
* Start at **3**, which is the default, and raise it only if the core stays healthy. On a small host use 1–2; past about 4–6 you need both spare memory and core headroom.
* For bursty starts, give the core more room before launching: `export XORCISE_READINESS_TIMEOUT_SECONDS=240` (the default is 90).
* The runner already staggers sandbox starts about 8 seconds apart so their telemetry inits do not collide. You do not need to do anything about that.
### Survive a long run
A large matrix runs for hours, and the tooling assumes something will go wrong.
**Check progress** any time, safely, against a run in flight — it reads only that run's own files:
```bash theme={"theme":"css-variables"}
python3 scripts/status.py ./xorcise-results/xor-pb-000-smoke --total 18 --health
```
**Resume** after a crash, an OOM, or a reboot by re-running the identical command with `--resume`. Every completed run is checkpointed to `progress.jsonl`, so it skips what is already graded and finishes the rest. Nothing is lost and nothing is repeated.
**Detach** — run under tmux or a systemd unit so a dropped SSH session cannot kill the matrix. On very long runs you can also let a core wedge self-heal:
```bash theme={"theme":"css-variables"}
WATCHDOG_RESTART_CMD='xorcise restart' scripts/watchdog.sh &
```
### Clean up
The runner tears down each sandbox as its run finishes, so there is usually nothing left. To sweep stragglers, scope the filter to this run's session tag:
```bash theme={"theme":"css-variables"}
SESSION=$(cat ./xorcise-results/xor-pb-000-smoke/session.txt)
docker ps -a --filter "name=xorcise-run-${SESSION}-" -q | xargs -r docker rm -f
```
**Never use a bare `name=xorcise-run-` filter.** Without the session tag it matches every benchmark sandbox on the host — other playbooks, other people's live runs — and deletes them. The session tag is what makes a playbook safe to run alongside anything else.
Leave mission containers alone. The core owns them and reaps them once your runs go terminal; removing them by hand while other runs are live breaks those runs.
## Running the scripts yourself
Nine scripts, all driveable directly if you want a benchmark in CI or on a schedule rather than in a conversation.
| Script | What it does | Reads or writes |
| ----------------------- | ----------------------------------------------------- | -------------------------- |
| `preflight.sh` | Check every prerequisite | Read-only |
| `list-missions.py` | Browse and filter missions, emit a playbook block | Read-only |
| `harness-info.sh` | Print the OpenHands and litellm versions in the image | Read-only |
| `setup-creds.sh` | Scaffold this run's credentials file | Writes `creds.env` |
| `check-connectivity.py` | Test each contestant, the judge, and OTLP | Makes one call per model |
| `run_playbook.py` | Create, launch, and record every run | Creates runs, spends money |
| `status.py` | Report progress on a run in flight | Read-only |
| `gen_report.py` | Render the eval card | Writes HTML |
| `watchdog.sh` | Restart a wedged core during a long run | Restarts XORCISE |
`run_playbook.py` is the only one with a real surface:
| Option | Default | Meaning |
| ---------------- | ------------------------------ | ------------------------------------------------------------------------------ |
| `--otel` | off | Enable telemetry export. Required for any score to be meaningful. |
| `--models` | playbook's list | Comma-separated short names, overriding the playbook |
| `--llm-config` | — | Path to the model map |
| `--creds` | — | Path to this run's credentials file |
| `--outdir` | — | Where results, checkpoints, and the session tag land |
| `--concurrency` | `3` | Attempts in parallel. Raise only if the core stays healthy. |
| `--runs` | playbook's value | Override runs per mission |
| `--budget` | playbook's value | Override per-run `budget_seconds` |
| `--intel-policy` | `none` | `none`, `all`, or a list such as `i1,i3` |
| `--dry-run` | off | Orchestrate and print launch commands without spawning agents. No keys needed. |
| `--resume` | off | Skip cells already recorded in `progress.jsonl` |
| `--rerun-failed` | off | Re-run only failed or ungraded cells |
| `--max-attempts` | `1` | Bound on `--rerun-failed` |
| `--image` | `xorcise-agent-sandbox:latest` | Sandbox image |
A legacy `--no-sandbox` flag also exists, which runs agents directly on your host with no isolation. Don't use it.
## Limits and gotchas
**Content filtering is normal.** Some providers block offensive-security prompts for some models. The run returns no output and makes zero tool calls, shows as `tool_calls=0`, and is excluded from scores rather than counted as a failure. If a model is consistently filtered on the offensive playbooks it cannot compete there — drop it.
**Grades arrive late.** Grading can land seconds to minutes after a run goes terminal, and the runner waits up to about five minutes for a real grade. Rows with `overall=None` usually mean the grade never landed or the transcript was too large for the judge; raising the judge's token ceiling or the per-run budget is the fix.
**Small *n* is small.** Three runs per cell is a pilot. Read a cell as mean ± standard deviation, not a point estimate, and be careful comparing two models three points apart.
**One judge carries its own biases.** Half of every score comes from a single model reading a rubric. That is the same trade-off any XORCISE run makes, and [How grading works](/concepts/grading) sets out its limits — a leaderboard just multiplies it.
**Use long-lived keys.** A session token that expires mid-matrix can wedge grading. On Bedrock prefer a long-term key in `AWS_BEARER_TOKEN_BEDROCK` over STS credentials.
The skill ships `reference/TROUBLESHOOTING.md` with the full list, including the symptom-to-cause table worth skimming before a large run.