CinvanaAI
← All projects

Experiment · Models & evaluation

Transcript Model Evaluator

Study a desktop experiment that kept tasks, model attempts, judgments and pricing views together.

Repository: CinvanaAI/transcript-model-evaluator

Explore the repository ↗Try or inspect it ↓

Recorded synthetic example. An executable experiment using supplied synthetic inputs.

See the idea in action.

The original judge contract assembles the request and parses the decision. A separate check reports that the shipped sample task lacks its rubric, artifacts and registered pairwise judge. No models are called.

Example input
One source artifact, a rubric, two synthetic responses and a fixture judge reply.
Captured result
{
  "mode": "Workbench contract study; synthetic judge response",
  "snapshot_battle_readiness": {
    "prompt_id": "sample_review",
    "ready": false,
    "checks": [
      {
        "name": "rubric",
        "present": false
      },
      {
        "name": "artifacts",
        "present": false
      },
      {
        "name": "pairwise_judge_registration",
        "present": false
      }
    ],
    "scope": "Files and registration only; no provider availability or GUI execution test."
  },
  "request": "Judge only against the source and rubric.\n\nPrompt Metadata:\n- prompt_id: synthetic-owner\n- name: Extract an owner\n\nSource Prompt Text:\nReturn the named owner.\n\nModel A: Complete fixture\nModel B: Missing fixture\n\nRubric ID: exact-owner\nRubric Total Points: 1\n\nRubric Categories:\n- Owner (1 pts): Preserve the exact named owner.\n\nArtifact Results:\n\nArtifact ID: owner\nArtifact Filename: owner.txt\nSource Artifact:\nMorgan owns this item.\n\nModel A (Complete fixture):\nRun Outcome: success\nRaw Output:\nMorgan\n\nModel B (Missing fixture):\nRun Outcome: success\nRaw Output:\nUnknown\n",
  "synthetic_judge_text": "{\"winner\":\"model_a_better\",\"short_reason\":\"The complete fixture preserves Morgan.\",\"confidence\":1.0}",
  "parsed_decision": {
    "winner": "model_a_better",
    "short_reason": "The complete fixture preserves Morgan.",
    "confidence": 1.0
  },
  "network_calls": 0
}

Try the example.

From the repository root, follow the dependency requirements in the README. This example uses supplied synthetic material.

python -m pip install -r requirements.txt
python -m examples.offline_case

The original judge contract assembles the request and parses the decision. A separate check reports that the shipped sample task lacks its rubric, artifacts and registered pairwise judge. No models are called.

Complete setup and instructions ↗

The interesting part.

Prompt definitions, current model records, raw provider evidence and derived views represent different kinds of information; separating them avoids treating a cache as truth.

Preserved research case

Where it came from.

Archived integrated transcript-evaluation operator console.

March 2026 — Its change_workbench subsystem is dated March 2026; not a date for every evaluator generation. Source ↗

Preserves the application as a study with synthetic data; the independent MK workflow develops reusable pieces.

The shipped desktop sample does not complete a tournament. Its platform ancestry does not identify a branch of current Skeleton.

Source ↗

Follow the family: Transcript evaluation and Rubric Rumble