Checks and ops

​

Sources and ops at a glance

​

How a check is evaluated

​

Sources

​

artifacts

• What it does — reads the artifacts the agent submitted over run-control with POST /artifacts. The flag is the artifact named flag; there is no separate flag field.
• Arguments — ref is the artifact name as the agent submitted it. Every value is a string.
• Passes when — the agent submitted an artifact under exactly that name and the op returns true for its string value.
• Fails when — no submission carries that name, so the value resolves to null and every op fails on it; or the op returns false. A name mismatch records no error on the verdict.

{
  "id": "flag-correct",
  "source": "artifacts",
  "ref": "flag",
  "op": "equals",
  "args": { "expected": "XORCISE{sql_injection_auth_bypass}" }
}

​

otel-stats

• What it does — reads a projection over the run’s sealed raw trace. The values are derived from what the agent exported, so they are forgeable: treat otel-stats as a convenience, not an anchor.
• Arguments — ref is a stat key. Exactly two exist, and both equal the number of stored raw trace records: turn-count and span-payload-count.
• Passes when — ref names one of those two keys and the op returns true for the count.
• Fails when — ref names anything else, so the value resolves to null; or the count fails the op.

{
  "id": "efficient-solve",
  "source": "otel-stats",
  "ref": "turn-count",
  "op": "lesser_than",
  "args": { "value": 40 }
}

​

observed-facts

• What it does — reads the facts XORCISE recorded about the run itself. The agent cannot write these, which makes this the anti-forgery source.
• Arguments — ref is a fact name from the closed list below. Two facts recorded under the same name are last-writer-wins.
• Passes when — XORCISE recorded that fact for this run and the op returns true for its value.
• Fails when — ref is not a name XORCISE records, or this run never produced that fact. A static mission has no network or targets, so it records only prompt, attachment and collector.

{
  "id": "telemetry-arrived",
  "source": "observed-facts",
  "ref": "collector",
  "op": "observed"
}

​

Ops

​

equals

• What it does — compares the resolved value with a declared literal for exact equality.
• Arguments — expected, the literal to compare against. No other key is accepted.
• Passes when — the resolved value equals expected, type included.
• Fails when — the values differ in content or in type. Artifacts are always strings, so {"expected": 3} never matches a submitted "3".

{
  "id": "flag-correct",
  "source": "artifacts",
  "ref": "flag",
  "op": "equals",
  "args": { "expected": "XORCISE{sql_injection_auth_bypass}" }
}

​

matches_format

• What it does — matches the resolved value against a regular expression.
• Arguments — pattern, the regex. Escape backslashes for JSON: XORCISE\\{.+\\}.
• Passes when — the value is a string and the pattern matches the whole of it.
• Fails when — the value is not a string, including null; or the pattern matches only part of the value. This is a full match, not a search, so an anchor-free pattern must still span the entire value.

{
  "id": "flag-shaped",
  "source": "artifacts",
  "ref": "flag",
  "op": "matches_format",
  "args": { "pattern": "XORCISE\\{.+\\}" }
}

​

observed

• What it does — asserts that the value is present and non-empty.
• Arguments — none. Omit args or declare it as {}.
• Passes when — the value exists and is truthy.
• Fails when — the ref resolved to null, or the value is "", 0, false, or an empty list.

{
  "id": "writeup-submitted",
  "source": "artifacts",
  "ref": "writeup",
  "op": "observed"
}

​

lesser_than

• What it does — compares the resolved value numerically against a threshold.
• Arguments — value, the threshold. The argument really is named value; the resolved value reaches the op positionally, so the two do not collide.
• Passes when — the resolved value and the threshold are both numeric and the resolved value is strictly smaller.
• Fails when — either side is not numeric, or the value is greater than or equal to the threshold. An artifact is always a string and is therefore never numeric.

{
  "id": "efficient-solve",
  "source": "otel-stats",
  "ref": "turn-count",
  "op": "lesser_than",
  "args": { "value": 40 }
}

​

Pairing a source with an op

​

Weighting

• All checks declare weight — the declared numbers are used as-is. Each must be greater than 0 and at most 1, and they must sum to 1.0. A sum that misses is rejected with check weights must sum to 1.0, got 0.9000.
• No check declares weight — every check gets 1/n. The resolved weight is written back onto each verdict, so the result breakdown always shows real numbers.

"checks": [
  {
    "id": "flag-correct",
    "source": "artifacts",
    "ref": "flag",
    "op": "matches_format",
    "args": { "pattern": "XORCISE\\{.+\\}" },
    "weight": 0.7
  },
  {
    "id": "efficient-solve",
    "source": "otel-stats",
    "ref": "turn-count",
    "op": "lesser_than",
    "args": { "value": 40 },
    "weight": 0.3
  }
]

​

Traps that fail silently

​

Observed-fact names that are never recorded

​

Artifact names the agent did not use

​

required: true does not gate anything

​

Related pages

• How grading works — how passing weights become half the score.
• Mission manifest — where checks sits in mission.json.