briefcase.validation

pip install briefcase-ai[validate]

The validation engine is pluggable: supply an extractor (finds references in a prompt), a resolver (checks each reference), and a versioned client (records the commit the validation ran against).

PromptValidationEngine

import re

from briefcase.validation import PromptValidationEngine
from briefcase.validation.errors import ValidationError, ValidationErrorCode


class RegexExtractor:
    _REF = re.compile(r"[\w/]+\.md")

    def extract(self, prompt: str) -> list:
        return self._REF.findall(prompt)


class AllowlistResolver:
    def __init__(self, known: set):
        self._known = known

    def resolve_all(self, references: list) -> list:
        errors = []
        for ref in references:
            if ref not in self._known:
                errors.append(
                    ValidationError(
                        code=ValidationErrorCode.REFERENCE_NOT_FOUND,
                        message=f"Reference not found: {ref}",
                        reference=ref,
                        severity="error",
                        layer="resolution",
                        remediation="Add the document to the knowledge base.",
                    )
                )
        return errors


class DemoLakeFS:
    def get_commit(self, repository: str, branch: str) -> str:
        return "demo0000000000000000000000000000000000000"


engine = PromptValidationEngine(
    extractor=RegexExtractor(),
    resolver=AllowlistResolver({"kb/faq.md"}),
    lakefs_client=DemoLakeFS(),
    repository="knowledge-base",
    branch="main",
    mode="strict",
)

report = engine.validate("See kb/faq.md and kb/missing.md")
print(report.status, report.references_checked, report.has_errors)

PromptValidationEngine(
    extractor,
    resolver,
    lakefs_client,
    repository,
    branch="main",
    mode="strict",
    semantic_validator=None,
)
  .validate(prompt) -> ValidationReport

ValidationReport

Attributes: status, errors, warnings, references_checked, validation_time_ms, lakefs_commit, has_errors, has_warnings, plus to_dict().

ValidationError

ValidationError(
    code,           # ValidationErrorCode
    message,
    reference,
    severity,
    layer,
    remediation=None,
    metadata=None,
)

ValidationErrorCode

Enum: INVALID_SYNTAX, REFERENCE_AMBIGUOUS, REFERENCE_NOT_FOUND, REFERENCE_GONE, VERSION_MISMATCH, SCHEMA_INVALID, LAKEFS_UNAVAILABLE.

Pluggable protocols

Extractor.extract(prompt) -> list, Resolver.resolve_all(references) -> list, and SemanticValidatorProtocol.validate_semantic(prompt, references) -> list.