briefcase.integrations.evals

pip install briefcase-ai[evals]

Records evaluation results as decision records, and parses inspect-ai and lm-eval-harness logs. The parsers are stdlib only and never import either framework. The extra installs zstandard on Python < 3.14, which stdlib zipfile needs to read inspect-ai .eval archives.

EvalRun

from briefcase.integrations.evals import EvalRun

with EvalRun("gsm8k", model="claude-opus-4-5") as run:
    run.log_case(
        "q1",
        inputs="What is 2+2?",
        outputs="4",
        target="4",
        passed=True,
        scores={"exact_match": 1.0},
        input_tokens=8,
        output_tokens=1,
    )

print(run.summary()["pass_rate"])

EvalRun(name, *, exporter=None, async_capture=False, run_id=None, model=None,
        metadata=None, cost_calculator=None, drift_calculator=None)
  .log_case(case_id, *, inputs=None, outputs=None, target=None, scores=None,
            passed=None, input_tokens=None, output_tokens=None, tags=None,
            metadata=None)      -> dict          # emits one "eval.case"
  .ingest(cases)                -> int           # normalized dicts from a parser
  .summary(*, include_drift=False) -> dict
  .finish(*, include_drift=False)  -> dict       # emits one "eval.run"; idempotent

async_capture defaults to False, unlike @capture: a batch job exits the moment it finishes, and a background thread would drop the tail of the run. log_case after finish() raises RuntimeError.

summary() returns run_id, name, model, total_cases, passed, failed, pass_rate, per-score mean/min/max/count, input_tokens, output_tokens, cost, drift, and metadata. Cost needs a model and token counts; drift needs include_drift=True and at least two string outputs. Both yield None rather than raising when the calculator is unavailable.

from_inspect_log, from_lm_eval_results, replay

from briefcase.integrations.evals import (
    from_inspect_log, from_lm_eval_results, replay,
)

parsed = from_inspect_log("logs/2026-08-12_gsm8k.eval")
print(parsed.name, parsed.model, parsed.metrics)

replay(parsed)
replay(from_lm_eval_results("results.json", "samples_gsm8k.jsonl"))

from_inspect_log(path) -> ParsedEvalLog                 # .json log or .eval archive
from_lm_eval_results(results_path, samples_path=None, *, task=None) -> ParsedEvalLog
replay(parsed, *, exporter=None, name=None, async_capture=False) -> EvalRun

ParsedEvalLog carries source, name, model, cases, and metrics. A file that does not match the expected shape raises ValueError naming the format; a missing file raises FileNotFoundError. Reading a .eval archive without a zstd backend raises ValueError with install instructions.

Record types

Type	Emitted	Carries
eval.case	per case	case id, inputs, outputs, target, scores, pass flag, tokens
eval.run	per run	pass rate, score statistics, token totals, cost, optional drift

Both export through the exporter configured by briefcase.observe(...), or one passed as exporter=. See Evaluation Runs for the narrative version.