Drift Detection

Outcome. Turn a vague sense that the model is flakier lately into a consistency score you can alert on.

Prerequisites. The drift extra and several outputs sampled from the same prompt.

Packages. briefcase-ai

Languages. Python

Measure how consistent a model’s outputs are across repeated runs, so you can tell the difference between normal variation and a model that is quietly drifting.

drift For: monitoring & accountability

When you’d reach for this

Two weeks ago your classify_ticket agent labeled the same five “reset my password” tickets account_access every time. Today, on the same tickets, it returns account_access three times and billing twice. Nothing in your code changed — but a provider-side model update, a prompt tweak, or a creeping context shift made the agent less consistent. Drift detection turns “it feels flakier lately” into a number you can alert on.

How it works

A DriftCalculator takes a list of outputs sampled from the same prompt and returns DriftMetrics: a consistency score, an agreement rate, the consensus output, and the indices of any outliers. You feed it the outputs; it tells you how much they disagree.

flowchart LR
    A[Same prompt,<br/>many runs] --> B[Collect outputs]
    B --> C[DriftCalculator.calculate_drift]
    C --> D{consistency_score<br/>below threshold?}
    D -->|No| E[Stable — keep watching]
    D -->|Yes| F[Drifting — emit an event]

Diagram description

The same prompt is run many times and the outputs are collected into a list. DriftCalculator.calculate_drift measures them and produces a consistency score (1.0 = identical outputs, 0.0 = all different). If the score is at or above your threshold the outputs are stable and you keep watching; if it falls below, the model is drifting and you emit an event so something downstream can react.

Install

pip install briefcase-ai[drift]

from briefcase.drift import DriftCalculator, DriftMetrics

Calculate drift

from briefcase.drift import DriftCalculator

calculator = DriftCalculator()

outputs = ["account_access", "account_access", "billing", "account_access", "account_access"]
metrics = calculator.calculate_drift(outputs)

print(metrics.consistency_score)
print(metrics.agreement_rate)
print(metrics.drift_score)
print(metrics.consensus_output)
print(metrics.outliers)
print(metrics.get_status(calculator))

calculate_drift(outputs) accepts a list of outputs sampled from the same prompt and returns DriftMetrics. get_status(calculator) classifies the result by consistency_score against fixed bands: "stable" at 0.85 and above, "drifting" at 0.5 to 0.85, "critical" below that.

A monitoring workflow

In production you don’t measure once — you measure repeatedly and react when consistency slips. The recorded decisions you already store are the source of the outputs.

1. Record multiple runs of the same decision through Decision Recording over a sampling window (a day, a week).

2. Extract the outputs for the prompt you’re watching into a plain list.

3. Measure them with calculate_drift and read consistency_score / get_status.

4. If it crosses your threshold, emit an event so something downstream — an on-call alert, a routing change — can respond. See Multi-Agent & Events.

import asyncio

from briefcase.drift import DriftCalculator
from briefcase.events import emit_drift_detected

calculator = DriftCalculator()


async def monitor(decision, outputs):
    metrics = calculator.calculate_drift(outputs)
    status = metrics.get_status(calculator)

    if status != "stable":
        # emit_drift_detected is a coroutine; await it in an async context
        await emit_drift_detected(decision, {"drift_score": metrics.drift_score})

    return status


# `decision` is the recorded classify_ticket decision; `outputs` are this window's labels
asyncio.run(monitor({"id": "dec-1"}, outputs))

Interpreting DriftMetrics

These scores are only useful if you know what action each implies.

Field	What it tells you	What to do
consistency_score	Overall consistency of the sampled outputs	Trend it window over window. A steady decline is the early warning, even before any single window looks bad.
agreement_rate	Fraction of outputs that match the consensus	A falling agreement rate means more runs are disagreeing — tighten the sampling and look at the outliers.
drift_score	How far the outputs diverge from one another	The value to put in an alert threshold and pass to emit_drift_detected.
consensus_output	The most common output across samples	What the model “usually” decides — your baseline for what changed.
outliers	Indices of outputs that disagree with the consensus	Index back into your list to read the exact runs that broke ranks.

Call metrics.get_status(calculator) to turn the scores into a status label. The bands are fixed at 0.85 and 0.5 on consistency_score; alert on drift_score directly when you want a threshold of your own.

Tune the similarity threshold

A stricter threshold makes near-matches count as disagreement, so small wording differences register in agreement_rate. It does not move consistency_score or the get_status bands.

from briefcase.drift import DriftCalculator

calculator = DriftCalculator()
calculator.with_similarity_threshold(0.95)

metrics = calculator.calculate_drift(["approve", "approve", "aprove"])
print(metrics.agreement_rate)
print(metrics.get_status(calculator))

Compare outputs over time

Run the same calculator across successive sampling windows to watch consistency change — this is the two-week-drift scenario made concrete.

from briefcase.drift import DriftCalculator

calculator = DriftCalculator()

windows = [
    ("week 1", ["account_access", "account_access", "billing", "account_access", "account_access"]),
    ("week 2", ["account_access", "billing", "billing", "account_access", "billing"]),
    ("week 3", ["billing", "billing", "account_access", "billing", "billing"]),
]

for label, outputs in windows:
    metrics = calculator.calculate_drift(outputs)
    print(label, metrics.consistency_score, metrics.get_status(calculator))

A consistency score that falls across the windows is exactly the signal to alert on.

Key classes

Class	Why it matters
DriftCalculator	Computes drift over a list of outputs; with_similarity_threshold(threshold) tunes how strict matching is.
DriftMetrics	Consistency score, agreement rate, drift score, consensus output, and outlier indices — the numbers you alert and act on.

Limits

Similarity is edit distance, not meaning. Text pairs are scored by normalized Levenshtein distance; two strings that both parse as numbers are compared by relative difference. "approve the loan" and "loan approved" score 0.125, because nothing here understands that they say the same thing. Drift detection watches for a model whose wording destabilizes, and it will call a rephrasing drift.

Fewer than two samples always reads as stable. An empty list and a single output both return consistency_score 1.0 and status "stable". A window that collected nothing looks identical to a window that agreed perfectly, so check total_samples before you trust a green result.

Outputs must be strings. Passing a dict raises TypeError. Serialize structured outputs yourself, and use a stable key order, or key shuffling alone will register as drift.

outliers uses a stricter cutoff than agreement_rate. An output counts as an outlier only below 0.7 times the similarity threshold, so a run can drag the agreement rate down while outliers stays empty. Read both.

Comparison is pairwise, so cost is quadratic in the sample count. 5,000 outputs is 12.5M comparisons, about 2 seconds on this machine. Sample a window rather than feeding a day of traffic.

API reference

briefcase.drift has the full DriftCalculator and DriftMetrics surface.

Where this fits

Drift detection sits in the Replay & Verify act: replay proves a single decision is reproducible, drift detection proves the model stays consistent across many — and when it doesn’t, an event hands control back to your governance layer.

Deterministic Replay Previous: re-run one stored decision and prove it still matches.

Decision Recording Records the runs whose outputs you sample for drift.

Multi-Agent & Events Emit a drift event so something downstream can react.

Guardrails Turn a drift signal into a control that gates the next action.