How runs are counted
Every run that has reachedterminal is summarised per agent, per mission version, and per agent on each mission version. The headline figure is the median overall score, and it is decided once, on the server: the web UI, the dashboard and xorcise leaderboard all read the same summary, so no two of them can count the same runs differently.
The median is over the runs graded with a judge. Every finished run falls into exactly one of these kinds:
Every run left out is still counted and named beside the median, so the number is never read as covering runs it does not. The one exception is a run that ended before its agent started —
cancelled, because you stopped it, or never_started, because its budget ran out first. The agent never did anything, so it is not a run of the agent’s at all: it is left out of every performance view, including the run counts and the dashboard’s Evaluated count. A median, rather than an average, means a few unusually good or bad runs move it less.
A mission’s checks and rubric belong to its version, and a new version can change both. So runs on an older mission version (one that is no longer the installed version) are summarised separately and marked as older, never mixed with runs on the current one.
1. Rank your agents
- CLI
- Web UI
Output
no finished runs yet — nothing to rank, and --json returns [] rather than prose, so a script can branch on an empty list.
In the web UI each agent’s card draws its run strip: one column per run, oldest to newest, as many recent runs as fit the card. A column’s height is the run’s overall score. Its lower part is what the deterministic checks earned and its upper part what the judge earned, each carrying half of the score. The dashed line across the strip is the median. Every kind of run is drawn by its column:
- a red cap: the run timed out;
- grey: an operator stopped it;
- a dashed outline for the judge’s half: no judge;
- hatching above the judge’s half: a partial judge, running from its lower to its upper bound;
- an empty grey striped slot: an environment failure;
- an empty dashed slot: grading failed, or the run is still grading.
Results › Agents. Each card's strip draws every run, and the line under it says how many are in the median.
2. Read one agent’s runs
Ranking tells you where an agent sits. Its runs tell you which way it is moving, and on which missions.- CLI
- Web UI
3. Compare by mission instead of by agent
The same results grouped the other way answer a different question: not “which agent is best” but “which mission is beating everyone”, and whether the two halves of its grading agree. The Missions tab on the Results page shows one card per mission version, each with a grader agreement plot. Every run is a point: across is what the deterministic checks gave it, up is what the judge gave it. The diagonals join runs with the same overall score, at 25, 50 and 75. A point in the upper left is a run the judge rated above its checks; one in the lower right passed checks the judge marked down. A run with no judge has no place on the judge axis, so it sits on a strip under the plot at its checks score. Under each plot, a chip per agent gives that agent’s median on the version. Point at a chip, or press it, to light up that agent’s runs. The card’s Judge vs checks figure is how far apart the two halves sit on average, over the runs the judge graded in full. When a version’s runs were judged by more than one model, the card warns that their scores may not compare. From the CLI or a script,GET /api/results/summary?by=mission returns the same per-version summaries.
4. Export the ledger
The All Runs tab is a dense table of every run: agent, mission, status, overall, deterministic, judge, and date. Ungraded and still-running runs show— rather than a zero, so an unfinished run never reads as a failure, and a run whose judge never ran reads No judge in the judge column rather than a 0% the judge never gave.
For the summaries as JSON:
avg_overall (the mean), best_overall, completion_rate, partial_rate and judge_degraded (the No judge count). They are computed by the rule above, so avg_overall includes timed-out runs and leaves out runs with no judge. xorcise leaderboard lists every field.
The summaries hold the scores. For the evidence behind them, export the runs themselves: one directory per finished run, holding its report, result, raw traces and events.
--mission and --since narrow it the same way. --genuine-only leaves out the runs the agent did not finish itself: operator kills, budget timeouts, deploy failures and crashes. A run still grading is left out and named on stderr. xorcise run export has the layout and the exit codes.
Comparing scores without fooling yourself
Four things make two scores incomparable, and all four are easy to miss. A run with no judge configured is capped at 50%. The judge half scores0.0 and is still part of that run’s overall score. The median leaves such runs out and counts them as No judge, but a series where you configured a judge partway through still has fewer scored runs early on. Re-evaluate the older runs against the sealed evidence rather than re-running them, and they join the median. xorcise run list marks each such score with *, so you can find them before you compare.
The model can change while the agent stays the same. The leaderboard and xorcise agent history are grouped by agent, and neither shows the model the agent’s harness reported; history’s model column is only the one you declared. xorcise run status, each run’s report and the web UI’s result page show the reported model. If your harness can run a different model from one run to the next, check it on the runs you compare before you read a change in score as the agent improving.
Deterministic and judge halves move for different reasons. A rising overall score built entirely on the deterministic half means the agent is passing more checks; one built on the judge half means a model’s assessment of its method changed. The run strip and the run ledger draw both halves of every run, and the agreement plot puts them on separate axes, for exactly this reason. Read them, not just the median.
The median counts timeouts, not stops or failures. An agent that often runs out of budget has those runs in its median at what they earned, which is usually low. That is deliberate. Runs you stopped, runs whose grading failed and runs an environment failure cut short are left out and counted beside the median, so a high Infra failed count points at your environment, not your agent.
Related pages
How grading works
Where each half of a score comes from, and how to re-evaluate a sealed run.
CLI reference
Every flag on
leaderboard, agent history, and the JSON output they emit.Evaluate Agents with Playbooks
Comparing models rather than tracking one agent — many models over the same missions, under a fixed harness. Experimental.