Validation Engine

Outcome. Reject a prompt whose references no longer resolve, before a model ever sees it.

Prerequisites. The validate extra plus the extractor and resolver objects you supply; the engine ships neither.

Packages. briefcase-ai

Languages. Python

The validation engine checks that the references in a prompt (document paths, section numbers, identifiers) actually resolve against a versioned knowledge base before the prompt reaches a model.

validate For: governance & control

When you’d reach for this

Your triage agent builds a prompt that cites handbook/onboarding.md and Section 4.2.3. Last week someone moved the handbook and the section number changed. Without a check, the model answers confidently from a reference that no longer resolves. The validation engine catches that before the prompt runs and hands back structured remediation instead of a silent, wrong answer.

It is a pure framework: you supply an extractor that finds references and a resolver that checks them, and the engine orchestrates the layers and records the commit it validated against.

How it works

1. Extract — your extractor finds candidate references in the prompt. No references means the prompt passes immediately.
2. Resolve — your resolver checks each reference against the versioned knowledge base and returns a ValidationError for anything that fails.
3. Semantic (optional) — if there are no errors, an optional semantic_validator can add warnings based on the prompt’s meaning.
4. Stamp — the engine records the knowledge-base commit it validated against, so the result is reproducible.

Install

pip install briefcase-ai[validate]

from briefcase.validation import PromptValidationEngine
from briefcase.validation import ValidationError, ValidationErrorCode

Validate a Prompt

import re

from briefcase.validation import PromptValidationEngine
from briefcase.validation import ValidationError, ValidationErrorCode


class HandbookExtractor:
    """Finds ``handbook/*.md`` paths and ``Section X.Y`` references in a prompt."""

    _REF = re.compile(r"handbook/[\w/]+\.md|Section\s+[\d.]+")

    def extract(self, prompt: str) -> list:
        return self._REF.findall(prompt)


class KnowledgeBaseResolver:
    """Resolves references against an allowlist of known documents."""

    def __init__(self, known_references: set):
        self._known = known_references

    def resolve_all(self, references: list) -> list:
        errors = []
        for ref in references:
            if ref not in self._known:
                errors.append(
                    ValidationError(
                        code=ValidationErrorCode.REFERENCE_NOT_FOUND,
                        message=f"Reference not found in knowledge base: {ref}",
                        reference=ref,
                        severity="error",
                        layer="resolution",
                        remediation="Add the document to the knowledge base or fix the reference.",
                    )
                )
        return errors


class DemoKnowledgeBase:
    """Stand-in for VersionedClient.get_commit() so the example runs offline."""

    def get_commit(self, repository: str, branch: str) -> str:
        return "demo0000000000000000000000000000000000000"


engine = PromptValidationEngine(
    extractor=HandbookExtractor(),
    resolver=KnowledgeBaseResolver(
        known_references={"handbook/onboarding.md", "Section 4.2.3"},
    ),
    lakefs_client=DemoKnowledgeBase(),
    repository="knowledge-base",
    branch="main",
    mode="strict",  # fail on errors
)

prompt = """
Follow the onboarding policy in handbook/onboarding.md and reference
Section 4.2.3 for account-setup steps. Also see handbook/missing.md.
"""

report = engine.validate(prompt)

print(report.status)              # "failed" — handbook/missing.md is unknown
print(report.references_checked)  # 3
print(report.lakefs_commit[:8])   # "demo0000"

if report.has_errors:
    for error in report.errors:
        print(error.reference, "->", error.message)
        print("  fix:", error.remediation)

In production, pass any versioned-data client (the bundled briefcase.integrations.lakefs.VersionedClient, or your own via the vcs protocol) as lakefs_client, to resolve references against a live, version-controlled knowledge base. The engine calls lakefs_client.get_commit(repository, branch) to stamp every report with the commit it validated against.

Pluggable Protocols

You provide objects that satisfy two protocols.

Note

The engine ships no built-in extractors or resolvers and installs no third-party dependencies for them. You supply the objects that find and check references.

# briefcase.validation exports these as runtime-checkable protocols; any object
# with the right method signature satisfies them (no base class required).
from typing import Protocol


# Extractor: find references in a prompt.
class Extractor(Protocol):
    def extract(self, prompt: str) -> list: ...


# Resolver: check each reference, return a list of ValidationError.
class Resolver(Protocol):
    def resolve_all(self, references: list) -> list: ...

A resolver returns ValidationError objects. Errors with severity="error" become report.errors; any other severity becomes report.warnings.

An optional third layer runs only when there are no errors: pass a semantic_validator with a validate_semantic(prompt, references) -> list method to attach warnings based on the meaning of the prompt.

class KeywordSemanticValidator:
    def validate_semantic(self, prompt: str, references: list) -> list:
        return []  # return ValidationError warnings based on the prompt's meaning


engine = PromptValidationEngine(
    extractor=HandbookExtractor(),
    resolver=KnowledgeBaseResolver(known_references=set()),
    lakefs_client=DemoKnowledgeBase(),
    repository="knowledge-base",
    semantic_validator=KeywordSemanticValidator(),  # optional third layer
)

Validation Layers

flowchart LR
    A["Prompt"] --> B["extractor.extract()"]
    B -- "no refs" --> P["status: passed"]
    B -- "refs found" --> C["resolver.resolve_all()"]
    C --> D{"errors?"}
    D -- "yes" --> F["status: failed"]
    D -- "no" --> E["semantic_validator (optional)"]
    E --> G["status: passed / warning"]

Diagram description

A prompt runs through extractor.extract(). If no references are found, the status is passed. If references are found, resolver.resolve_all() runs and the engine checks for errors. Any error yields status failed. With no errors, the optional semantic_validator runs and the status becomes passed or warning.

ValidationReport

validate() returns a ValidationReport.

Field	Description
status	"passed", "warning", or "failed"
errors	List of ValidationError with severity="error"
warnings	List of ValidationError with other severities
references_checked	Number of references the extractor found
validation_time_ms	Wall-clock validation time
lakefs_commit	Commit SHA the validation ran against
has_errors	True when errors is non-empty
has_warnings	True when warnings is non-empty

ValidationError

Each error carries structured remediation context.

ValidationError(
    code=ValidationErrorCode.REFERENCE_NOT_FOUND,
    message="Reference not found in knowledge base: handbook/missing.md",
    reference="handbook/missing.md",
    severity="error",
    layer="resolution",
    remediation="Add the document to the knowledge base or fix the reference.",
)

ValidationErrorCode is the shared vocabulary your resolver reports with. The engine constructs no errors of its own; it collects what your resolver and semantic validator return, so these codes mean whatever your resolver uses them to mean:

Code	Meaning
INVALID_SYNTAX	Reference is malformed
REFERENCE_NOT_FOUND	Reference does not exist in the knowledge base
REFERENCE_AMBIGUOUS	Reference matches more than one document
REFERENCE_GONE	Reference existed but was removed
VERSION_MISMATCH	Reference resolves to an unexpected version
SCHEMA_INVALID	Resolved document fails schema checks
LAKEFS_UNAVAILABLE	The versioned knowledge base could not be reached

Validation Modes

The mode argument controls how status is derived.

Mode	Errors	Warnings only	Clean
"strict"	failed	warning	passed
"tolerant"	failed	passed	passed
"warn_only"	passed	passed	passed

Limits

The engine is a pipeline, not a validator. It finds nothing and checks nothing on its own: briefcase/validation/engine.py never constructs a ValidationError. Every error and warning in a report came from the extractor, resolver, or semantic validator you supplied. What the engine contributes is the ordering, the mode logic, the timing, and the commit stamp.

Exceptions from your components propagate. An extractor or resolver that raises comes straight out of validate(). Nothing is caught and converted to a failed report, so wrap the call if a validator crash must not take down the request.

An unreachable knowledge base does not fail validation. If lakefs_client.get_commit() raises, validate() still returns a report and stamps lakefs_commit as "unknown". lakefs_client=None behaves the same way. Check that field before treating a report as reproducible; a report stamped "unknown" records no version at all.

In warn_only, status and has_errors disagree by design. A report with a real error returns status="passed" and has_errors=True. Pick one field and gate on it consistently, and know that warn_only never blocks.

Validation reads the prompt, not the retrieved content. It checks that references resolve, not that the resolved documents say what the prompt assumes. The optional semantic layer is where that would live, and you write it.

API reference

briefcase.validation has the engine, protocols, report, and error types.

Where this fits

Validation is a Control act: it stops a prompt with stale references before it runs, and stamps the commit it checked against so the result is reproducible. It pairs naturally with versioned retrieval.

Guardrails Previous: enforce deny-by-default controls before an agent action runs.

RAG Versioning Pin retrieval to a versioned knowledge base so references resolve to a known commit.

Reproducible RAG (guide) End-to-end: validate references and pin retrieval so a run reproduces exactly.