Skip to main content
This document is for people changing XORCISE. It records the constraints the codebase depends on and the reasoning behind them — the things you would otherwise have to rediscover by breaking something. It is deliberately not a user guide. How to install XORCISE, run an agent against a mission, read a trace or interpret a score is documented for users across the rest of this site. Nothing here repeats that; everything here is a rule, an invariant, or the reason an obvious-looking simplification is wrong. Rules are stated in the imperative and are testable. Where a rule is machine-enforced, the enforcing file is named — if you disagree with a rule, change the enforcement and this document together, in the same change.

​
1. Shape of the product

XORCISE is one pip distribution. There is no server tier, no second package, and no fork per deployment: xorcise up boots one long-lived local process that serves the REST API, the web UI and the OTLP receiver, and drives Docker over the local socket.
  • All internal code lives under xorcise.core.*. The bare xorcise.* namespace is reserved for future products, and scripts/check_single_distribution.py fails CI if a second distribution appears. Do not add top-level modules.
  • Imports must have no side effects. Importing any package must not read config, open a database, touch the network, or start a thread. Settings load only when get_settings() is called.
  • Bring Your Own Model. XORCISE bundles no LLM and no key. Every feature that needs a model — the judge, terrain attribution — uses the operator’s own OpenAI-compatible endpoint. Never add a bundled or default model, and never ship a credential.
  • Vendor-neutral about the agent under test. XORCISE never launches the agent and ships no agent templates. The agent is given a connect prompt and reaches the run over the network on its own terms.
The data model is agent-centric. The agent is the first-class record: it is registered before anything else can happen, and a run belongs to exactly one agent, one mission and one budget. Run creation resolves the named agent and refuses with a 409 if none is registered — there is no anonymous run. Results accumulate into that agent’s track record, which is what makes scores comparable across runs. XORCISE judges an agent by what it did — its recorded trace and what it submitted — never by inspecting its source. Do not add a code-analysis path to the grading flow. xorcise.core.eval and xorcise.core.code are swap seams for a future premium tier. core.code is an intentionally empty stub. Do not delete it and do not fill it in the open-source tree; only core.seams may import a premium implementation.

​
2. Layering and imports

The dependency rule is one-directional and machine-enforced by .importlinter (uv run lint-imports, wired into CI and pre-commit):
Everything may depend inward; nothing may depend outward. contracts is the innermost layer and imports nothing of ours.
  • Delivery surfaces sit above domain modules. rest and frontend may import a domain module; a domain module must never import a delivery surface.
  • Part-islands are mutually non-importing. The runner, headscale and otel parts must not import each other. They communicate through typed clients in orchestration/clients/. When a part needs a fact owned by another module, it receives it as an injected callback, not an import — the headscale part learns which runs are active this way, and stays pure.
  • The kernel splits: config sits below db and observability. config is shared-kernel and must not import an application-layer module — this is why it re-derives the home path locally instead of importing xorcise.core.home.
  • Only the runner part touches Docker. Ingestion drives image builds through a builder port that it stubs in tests.

​
3. Persistence

The kernel owns SQLAlchemy 2.0 and Alembic; models are owned by the module they belong to.
  • Migrations are hand-written. Autogenerate stays off. The Alembic environment must never import an application-layer model, so there is no target_metadata to diff against. Write the migration by hand.
  • Never add a model-level ForeignKey across modules. The foreign key lives in the migration DDL only. SQLite does not enforce foreign keys by default, and a model-level FK across modules would break part-island independence.
  • Cascades are application-layer and orchestrated by the delivery surface. Deleting an agent deletes its runs and results. Do not “simplify” this to ON DELETE CASCADE — it would not fire on SQLite and would reintroduce the cross-module FK. This is the canonical case of an obvious simplification being wrong.
  • Never migrate populated data on boot. A database is classified fresh (scaffold it to head), ready (boot), or stale (refuse loudly and defer to xorcise db upgrade). Bootstrap happens on up; migration is always an explicit operator action.

​
4. Run lifecycle and durable state

A run’s state is authoritative in the database, not in a process-local map.
  • Persist within the operation that changes the state. In-memory maps are caches and must always have a durable fallback. A fact that outlives a request, or crosses a process or crash boundary, is written through the runs domain layer at the moment it changes.
  • Address external resources by run-derived identity — the router node is {run_id}-router, the container is named for the run — so a reconciler that never saw the creating process can still find and reap them.
  • Make create, teardown, terminal transition and result-recording idempotent and replayable. Reconcile-on-startup re-runs them to convergence after a crash; anything that is not replayable will corrupt state instead of healing it.
  • The server never calls its own REST API to persist its own state. Go through the domain layer.
  • Terminal is first-wins and grading is asynchronous. Sealing happens synchronously so the agent gets an immediate answer; grading runs on a background task. The safety comes from a pair of guards — result-recording is idempotent first-write-wins on run_id, and the grade-and-record path is a guarded no-op if it has already run. Do not remove either guard, and do not move grading back onto the request thread. The budget watchdog runs on its own thread.
  • Versions bump only on an explicit update. Registering a duplicate name is a 409, never a silent overwrite. A mission version is stamped only through the shared atomic-install helper; a new install path that bypasses it silently breaks cross-run comparability.

​
5. Network isolation

Each run gets its own fenced network. The guarantee is that the mission is unreachable from the host and from other runs — it is not an egress firewall, and mission containers still reach the internet through ordinary NAT.
  • Mission containers run on a Docker network nested inside a fused container, so no route exists from the host to a mission address.
  • A private tailnet carries the agent in. The access policy is fail-closed: exactly one rule per run, no default-allow. Refuse to apply a policy that does not match that shape.
  • Render subnet allocation and the access policy from persisted non-terminal runs, unioned with the in-process in-flight set, and reconcile the policy after the row persists. Deriving them from in-process state alone lets concurrent runs collide and lets one process clobber another’s policy.
  • Delete the run’s router node by name on the terminal transition. A completed run that leaves an online node behind leaks into the next run’s allocation.
  • Agents reach targets by IP; there is no DNS inside the fence. The tailnet login server is never host.docker.internal.
  • The per-run join key is read through its narrow getter only and is never surfaced on the general run entity.

​
6. Missions

A mission is mission.json plus a pullable OCI image — never an archive format. The server-to-runner deploy verb carries a thin reference only; mission bytes never ride the wire between them.
  • The mission and its runner are fused into one self-contained image per mission, with the inner stack baked in and loaded on boot so a deploy needs no inner pull. The cost is that a change to runner control logic requires rebuilding every mission image. That is an accepted trade, not a bug.
  • The two acquisition paths never cross. A library mission is pulled as a prebuilt fused image; a local mission is ingested and built locally. Both write their installed record through the same shared helper.
  • Attachment bytes never ride run-control. The agent asks for a companion file by name and receives a short-lived HMAC-signed URL it fetches separately. Signing secrets are generated per process unless explicitly pinned.

​
7. Telemetry and the evidence pipeline

Raw OTLP is canonical and immutable. Everything else is derived.
  • The live view and the grader read natively off the persisted raw store. The OTLP receiver is ingress-only — it never becomes a routing, correlation or persistence layer.
  • The agent-event projection is a rebuildable cache, never a source of truth. It is versioned by adapter name and version so a changed adapter invalidates and re-derives it.
  • The grader must never read the agent-event projection. It consumes the sealed raw record only. This one is enforced by name in .importlinter.
  • Adapters must be total. An unrecognised span becomes a generic event, an adapter exception becomes exactly one error event, and neither ever produces a 5xx. Supporting a new agent must require no frontend change and no core change.
  • Adapter selection is authoritative from the run’s recorded agent, not from agent-emitted resource attributes, which are forgeable.
  • A run is sealed on terminal; spans arriving afterwards are rejected. Replay never affects a score.
  • Telemetry is optional. A run with no traces is valid: deterministic checks read artifacts and observed facts, and criteria the judge cannot evidence score unknown and are renormalised out rather than scored zero. Never make a trace a precondition for grading.

​
8. Grading

A score is half deterministic checks, half model judgement, over the sealed evidence only.
  • Deterministic checks assert a named operation over a value resolved from a named source. Fetching is decoupled from asserting so a new check op needs no new resolver.
  • Agent-controlled evidence rides the user role, fenced as untrusted and stripped of fence markers; instructions ride the system role. Never interpolate agent output into the system prompt.
  • The transcript budget is counted with an explicitly configured tokenizer. Tokenizers are not interchangeable, so the choice is recorded rather than inferred. Going over budget must fail loudly; never silently truncate evidence.
  • A degraded judge produces a recorded, re-gradeable failure — never a silent zero.

​
9. Agent control surface

REST is the only agent control surface. There is no MCP plane and no reserved port for one. If an MCP surface is ever wanted, it is a fresh decision starting from the REST contract, not a resumed stub.
  • The agent’s verbs are /mission, /intel, /artifacts, /complete, plus the connect and join helpers. Authentication is a per-run bearer token.
  • There is one submission verb. The flag is not special: it is the artifact named flag, credited by a deterministic check. Never reintroduce a dedicated flag endpoint.
  • Post-terminal run-control is uniformly 409 — no per-verb special cases, no “late submission” path.
  • Note that the agent-event vocabulary retains MCP call and result kinds. Those describe the agent under test invoking its own MCP tools, which is a different thing that happens to share a name.

​
10. Configuration

Local configuration, including the BYOM judge key, is settable through the REST API so the UI and CLI share one path.
  • The config API is a fixed, judge-shaped contract, never a generic settings passthrough — a generic one could brick a running server.
  • Secrets are write-only. A stored key reads back only as its last few characters.
  • All writers go through the shared upsert helper and then clear the settings cache. Writing the environment file by any other path leaves the running server serving stale configuration.
Security note. In the default local topology the REST and OTLP planes bind loopback plus, on native Linux, the Docker bridge gateway — the narrowest set that still lets an agent in a container reach them via host.docker.internal. They widen to 0.0.0.0 only on an explicit XORCISE_HOST=0.0.0.0, or when that gateway cannot be determined at boot. Neither plane authenticates the operator surface. A loopback bind is therefore not a security boundary here: every local process and every container on that bridge can read runs, create runs, and write configuration. Do not add secret-bearing endpoints on the assumption that the surface is loopback-only, and do not assume the wildcard case never happens — it is a live fallback path.

​
11. Deferred, and deliberately absent

Stated so nobody re-derives them as gaps:
  • Remote and cloud-hosted mission planes are not implemented. XORCISE runs on one machine. No Terraform ships in this repo, and the remote and cloud auth commands are placeholders.
  • There is no streaming trace endpoint. The live view polls with an incremental cursor. This is a deliberate simplification.
  • Telemetry mirroring to an external service is a reserved, default-off seam. Enabling it fails fast rather than silently exporting: nothing leaves the operator’s machine by default.

​
12. Contributor ground rules

Two short rule sets travel with the codebase. They are stated here so a contributor finds every load-bearing rule on one page.

​
Coding standards

  • ruff (lint+format), mypy --strict. No # type: ignore without a reason comment.
  • Small, focused files (one responsibility). Typed DTOs (xorcise.core.contracts) over loose dicts at boundaries.
  • TDD: failing test → minimal impl → green → commit.
  • CI is GitHub Actions (.github/workflows/ci.yml); any CI lane, dependency setup, guard, or test-marker change must be reflected there.

​
Security

  • The application layer never holds the Docker socket (only runner). Nothing leaves by default (egress is opt-in).
  • Secrets live in ~/.xorcise/.env (0600), never in the repo. BYOM keys are the user’s.