CinvanaAI
← All projects

Library · Models & evaluation

Model Provider Compatibility Lab

Record what one model/provider route did on one task, separating support from configuration problems.

Repository: CinvanaAI/model-provider-compatibility-lab

Explore the repository ↗Try or inspect it ↓

Recorded synthetic example. Source with a complete offline example.

See the idea in action.

The actual adapter/ledger records supported, fixable and unknown states; returned token usage is retained, while unobserved inventory is null.

Example input
Three fake HTTP outcomes for the same owner-extraction task: usable output, unknown model and a timeout.
Captured result
{
  "mode": "Synthetic HTTP; actual adapter normalization and attempt ledger",
  "request": "Morgan owns this item.",
  "outputs": [
    {
      "model": "working-fixture",
      "text": "{\"owner\":\"Morgan\"}",
      "usage": {
        "input_tokens": 12,
        "output_tokens": 7,
        "input_tokens_details": {
          "cached_tokens": null
        }
      }
    }
  ],
  "records": [
    {
      "provider_key": "demo",
      "model_id": "missing-fixture",
      "task_name": "extract_owner",
      "status": "fixable",
      "can_list_models": null,
      "can_execute_task": false,
      "blocked_reason": "The selected model or endpoint was not found.",
      "fix_hint": "Refresh inventory or correct the model/endpoint, then retry.",
      "error_type": "provider_error",
      "error_message": "unknown model"
    },
    {
      "provider_key": "demo",
      "model_id": "timeout-fixture",
      "task_name": "extract_owner",
      "status": "unknown",
      "can_list_models": null,
      "can_execute_task": false,
      "blocked_reason": "The attempt did not establish task compatibility.",
      "fix_hint": "Retry when the provider is reachable.",
      "error_type": "connection_error",
      "error_message": "connection timed out"
    },
    {
      "provider_key": "demo",
      "model_id": "working-fixture",
      "task_name": "extract_owner",
      "status": "supported",
      "can_list_models": null,
      "can_execute_task": true,
      "blocked_reason": null,
      "fix_hint": null,
      "error_type": null,
      "error_message": null
    }
  ],
  "transport_calls": 3,
  "network_calls": 0
}

Try the example.

From the repository root, follow the dependency requirements in the README. This example uses supplied synthetic material.

python -m pip install -e .
python -m examples.offline_demo

The actual adapter/ledger records supported, fixable and unknown states; returned token usage is retained, while unobserved inventory is null.

Complete setup and instructions ↗

The interesting part.

A failed call is evidence with a scope, not proof that the model lacks an entire capability.

Focused extraction

Where it came from.

Provider/model/task evidence rules from two generations of transcript evaluation tools.

Creates focused public boundaries with synthetic evidence; retains provider normalization, compatibility and pricing separation.

Its documented integrated parent is Transcript Model Evaluator; a successful call is not a quality grade.

Source ↗

Follow the family: Transcript evaluation and Rubric Rumble