An experiment in inspectable evaluation

Rubric Rumble.

Give candidates the same task.
Then open the decision.

A runnable experimental model workbench. Formerly Mortal Kombat. Choose a way in, or inspect the offline replay below.

Below: the offline replay. Its exact-field rules judge is separate from the GPT-5.4 judge in the historical battles.

2 source records3 candidates1 recorded battle0 provider callsJudge: exact-field rules

The replay needs JavaScript, but the evidence does not. Open the recorded result, the task, or the observed ladder trace. The reproduction commands and source links below remain available.

01 / Inspect the evidence

Same task. Different answers.

One point for each exact field match. Two fields on each of two records: four possible points.

The instruction

Extract the owner and next action. Return a JSON object with string fields owner and action, preserving the exact wording.

Choose a source record

The meaningful difference

Plain-text fixture Disqualified / not a third-place finish

Captured response

The response arrived successfully. Validation rejected its shape. This is different from a failed network call.

See the exact disqualification record

02 / Follow the engine

One battle. Four observable phases.

These ladder states were observed during an independent replay of the unchanged engine. The phases are not extra battles.

Actual seed queue

No prior ranking or rates are supplied. The engine sorts these candidates by provider and candidate ID; it does not use their order in the task file.

Before this phase

After this phase

Excluded from ranking: malformed-fixture

5 captured responses, 4 accepted

The malformed response remains in the evidence but does not increment the engine's successful-evaluation count. Its second artifact is skipped.

Evaluate first, compare second

All candidates pass through output checking before the ladder starts. A valid output is reused during comparisons. This run has zero cache hits.

The judge defines the result

Exact fields decide this fixture. A tie would use candidate ID order, explicitly as a tie-break. A different rubric can reward different behavior.

03 / Make it reviewable

Take the run with you.

The task, captured result, observed trace, and source hashes travel with this page. No account is needed for the fixture.

git clone https://github.com/CinvanaAI/rubric-rumble.git
cd rubric-rumble
python -m pip install -e .
rubric-rumble demo --out runs/first-evaluation

Requires Git and Python 3.11 or newer. If you already have the repository, start inside it with the install command. Open runs/first-evaluation/report.html; result.json is beside it. Choose a new directory for each run.

Download recorded JSON ↓Download task ↓

Effective task / SHA-25628129bbae2a0f3413b26f819d4cc32b6c66fb90c84ad3ad6ee37363e498ea276

This fingerprints the validated task as sorted-key, compact JSON. It is not the raw task-file hash. Match this value and the outcomes when reproducing; timestamps will differ.

Print the fingerprint from your new result

python -c "import json; print(json.load(open('runs/first-evaluation/result.json'))['task_sha256'])"

Observed state trace · Source and result hashes

Start with the installed command ↗main(argv) → load the task and reserve a new result directory Validate and fingerprint the task ↗validate_task(raw) / fingerprint(task) → explicit task contract Score the exact fields ↗assess_output(text, expected) → field matches or invalid JSON Capture outputs and supply the judge ↗run_task(task) → evaluate(model, artifact) and judge(left, right) Compare and insert into the ladder ↗run_tournament(...) → model A is candidate; model B is incumbent

This page replays saved evidence; it does not call a provider or run Python in your browser. The source links show the snapshot behind this recorded replay. The operating atlas explains the current application.

To supply your own task: rubric-rumble example my-task.json, then rubric-rumble run my-task.json to inspect the plan. Execution is explicit through --execute.