Deterministic Replay

Outcome. Re-run a stored decision and compare what comes back against a policy you set.

Prerequisites. The replay extra, a persisted DecisionSnapshot, and a deterministic executor you supply.

Packages. briefcase-ai

Languages. Python

Load a recorded decision out of storage and run it back through a ReplayEngine, which reconstructs the snapshot, applies a ReplayPolicy, and reports a ReplayResult you can act on.

replay For: regression & verification

Set an executor, or nothing re-executes

ReplayEngine needs to be told how to re-run your decision. Pass a callable to with_executor() and a replay calls it with the recorded inputs, compares its answer to the recorded one, and sets outputs_match accordingly. Without one, replay() loads the snapshot and reports status "pending" with outputs_match False, which means “not checked” rather than “changed”.

When you’d reach for this

You ship a new version of the classify_ticket model and want to know, before it reaches users, whether it still routes the same tickets the same way. Replay the decisions you already recorded against the new build: outputs_match turning False on a ticket that used to be stable is a regression you caught instead of a customer noticing it.

How it works

1. Persist a DecisionSnapshot to a storage backend (this is the original you’ll compare against).

2. Replay it through a ReplayEngine carrying an executor, which re-runs the decision against your current build and compares the answers.

3. Interpret the ReplayResult — outputs_match, status, and policy_violations tell you whether the decision held.

flowchart LR
    A[Recorded decision<br/>in storage] --> B[ReplayEngine.replay]
    X[Your executor<br/>current build] --> B
    B --> C{outputs_match?}
    C -->|True| D[Reproducible]
    C -->|False| E[Regression]

Diagram description

A recorded decision in storage and your executor, which runs the current build, both flow into ReplayEngine.replay. The engine compares the fresh answer against the recorded one. A match means the decision is reproducible; a mismatch is a regression to investigate.

Install

pip install briefcase-ai[replay]

from briefcase.replay import ReplayEngine, ReplayPolicy, ReplayStats

Persist, then replay

import briefcase
from briefcase import DecisionSnapshot, Input, Output
from briefcase.storage import SqliteBackend
from briefcase.replay import ReplayEngine

briefcase.init()

decision = DecisionSnapshot("classify_ticket")
decision.add_input(Input("text", "Reset my password", "string"))
decision.add_output(Output("category", "account_access", "string"))
decision.with_execution_time(12.5)

backend = SqliteBackend.in_memory()
decision_id = backend.save_decision(decision)

def classify_ticket(inputs):
    """Your current build. Briefcase calls it with the recorded inputs."""
    return {"category": "account_access"}


engine = ReplayEngine(backend)
engine.with_executor(classify_ticket)

result = engine.replay(decision_id, "strict")

print(result.status)          # success
print(result.outputs_match)   # True — the build still agrees
print(result.replay_output)
print(result.policy_violations)

ReplayEngine(backend) takes a storage backend and replay(decision_id, mode) takes the mode explicitly. with_executor(fn) is what makes the replay re-run anything: fn receives a dict of the recorded input names and values, and returns either a dict of output names to values or a single value recorded as result.

Strict vs. tolerant

The mode decides how exactly the replayed output must match the original. Pick it from how deterministic the decision is supposed to be.

Mode	Matches when	Reach for it when
"strict"	The replayed output is identical to the original	The decision is meant to be deterministic — a fixed classifier, temperature=0, a routing rule. Any difference is a regression.
"tolerant" (default)	Minor differences are allowed	The output is free-form or sampled (a generated reply, a summary) where wording can vary but meaning should not.

Note

"tolerant" is the engine default. For per-field control — exact match on category, a similarity threshold on a free-text summary — use a ReplayPolicy instead of choosing one mode for the whole decision.

Interpreting a ReplayResult

ReplayResult is what you act on. The table maps each field to the decision it should drive.

Field	What it tells you	What to do
outputs_match	True when the fresh answer matched the recorded one	False on a decision that used to be stable is a regression. Investigate the change you just made.
status	"success", "failed", or "pending"	"pending" means no executor was set, so nothing was checked. Treat it as a harness problem, not a pass.
replay_output	The outputs your executor produced this run	Diff against original_snapshot.outputs to see exactly what moved.
policy_violations	The rules that failed, with expected and actual	Non-empty names the field and why. See Replay with a policy.
execution_time_ms	How long this replay took	A large swing flags a performance regression even when the output still matches.
original_snapshot	The recorded decision being replayed	The baseline for the comparison and your audit reference.

Replay with a policy

A ReplayPolicy declares how each output field must match. Combine exact-match fields with similarity-threshold fields when one decision has both a structured label and free text.

from briefcase.replay import ReplayPolicy

policy = ReplayPolicy("output-consistency")
policy.with_exact_match("category")
policy.with_similarity_threshold("summary", 0.95)

result = engine.replay_with_policy(decision_id, policy, "strict")
print(result.status)
print(result.policy_violations)

Here category must match exactly (a misroute is unacceptable) while summary only has to stay 95% similar (wording may vary). Each violation names the field, what was recorded, and what came back.

Replay in batches

Verify a whole regression set at once instead of one decision at a time.

results = engine.replay_batch([decision_id], "strict", 4)
for result in results:
    print(result.status, result.outputs_match)

replay_batch(decision_ids, mode, max_concurrent) replays many decisions concurrently, bounded by max_concurrent.

Aggregate replay statistics

from briefcase.replay import ReplayStats

stats = engine.get_replay_stats([decision_id])
print(stats.total_replays)
print(stats.successful_replays)
print(stats.exact_matches)
print(stats.success_rate)
print(stats.average_execution_time_ms)

success_rate across your regression set is the one number to watch release over release: a drop means more decisions changed than you expected.

Key classes

Class	Why it matters
ReplayEngine	Loads a persisted decision from a backend and re-executes it — the entry point for every replay.
ReplayResult	Outcome of a single replay; the fields you act on to catch a regression.
ReplayPolicy	Per-field match rules for replay_with_policy so structured and free-text fields can be judged differently.
ReplayStats	Aggregate counts and rates across many replays — your release-over-release health signal.

Limits

Without an executor nothing is checked. replay() on an engine that was never given one returns status "pending" and outputs_match False, and every policy rule reports actual: "not replayed". That is deliberate: an unchecked replay must never read as a pass. Set one with with_executor().

The executor is yours to make deterministic. Briefcase AI calls it with the recorded inputs and compares what comes back. If it hits a live model at temperature 1, a mismatch tells you about sampling rather than about your build. Pin the model and seed, or use "tolerant" mode and a similarity policy.

Only inputs are restored, not the world. The executor receives the recorded input names and values. Databases, feature stores, clocks, and network state are whatever they are now, so a decision that depended on them can differ for reasons that are not a regression.

A raising executor fails the replay. The exception propagates out of replay() rather than being recorded as a mismatch, so a broken harness cannot masquerade as a regression. Catch it if you are running a batch.

Similarity is edit distance. with_similarity_threshold compares text by normalized Levenshtein distance, so it tolerates wording changes rather than meaning changes. Drift Detection explains the same tradeoff in more detail.

replay_batch runs the executor once per decision, concurrently. An executor that calls a rate-limited model needs its own throttling; max_concurrent bounds the replays, not what your executor does inside them.

API reference

briefcase.replay has the full signatures, including the "validation_only" mode, which skips comparison entirely and returns as soon as the snapshot loads.

Where this fits

Replay is the start of the Replay & Verify act: re-run a stored decision, then measure how far it moved over time and prove the record is intact.

Storage Adapters Previous: persist the decisions that replay re-executes.

Drift Detection Next: measure how consistent outputs stay across many runs.

Audit Bundles Then: bundle a decision into a tamper-evident, verifiable record.

Decision Recording Build the DecisionSnapshot objects replay compares against.