A Pydantic AI application is ready for a limited production release when its team can demonstrate what it may do, which data it may access, how failures reach users and how an operator can stop it. A successful demonstration establishes only a small part of that evidence. Real requests introduce concurrent users, stale records, interrupted operations, private information and costs that continue after a page stops loading.
This checklist follows a fictional support assistant through eight reviews. The assistant reads an authenticated customer’s ticket, reports its recorded status and may prepare a status change for approval. Start with the read-only task. Enable writes only when the separate approval and recovery checks pass. The examples use synthetic records, so you can practise the decisions without connecting customer systems.
For every review, retain a requirement, a repeatable check, an observed result, an evidence date and an actual owner. Use passed, failed and not run as distinct states. A missing result cannot become a pass because the demonstration looked convincing. Record the exact application version and enabled tools alongside the decision, especially when approving a smaller pilot after finding a defect.
The five complete Python examples below were checked with Python 3.12.14, Pydantic AI 2.54.0 and Pydantic 2.13.5. Run each separately in an environment containing those packages. They disable real Pydantic AI model requests and use TestModel or local application logic. No provider key, external service or telemetry account is required. These exercises verify specific controls; they do not measure a live model’s answer quality or establish production capacity.
1. Define the allowed task and failure outcomes
Begin with a task contract that a product owner and developer can both explain. For the support assistant, a useful contract is: report the current status of a ticket the signed-in customer may read, using the ticket service as the authority. The assistant cannot promise resolution dates, inspect other customers’ records or change a status merely because a message requests it. A proposed change is a separate operation with a separate decision.
Make the task small enough to test. “Help with support” leaves almost every boundary open. “Summarise the status and last customer-visible update for one selected ticket” tells you which fields to fetch, what evidence supports the response and which unavailable-service cases need handling. The Pydantic AI agents guide provides context for choosing a narrow responsibility before adding more tools.
What should happen when the assistant cannot complete the task?
Define outcomes that the interface and downstream systems recognise. An answer, a clarification request, an access denial, a review request and a temporary service failure have different consequences. Give each outcome an honest next step. A review queue needs someone who checks it; an unavailable response needs a retry route that does not duplicate work. Avoid displaying a confident answer when the system actually exhausted its checks.
| Situation | Allowed outcome | Evidence to inspect |
|---|---|---|
| Accessible ticket exists | Report its recorded status and update time | Answer agrees with the selected record |
| Ticket selection is missing | Request a ticket selection | No arbitrary record is fetched |
| Record is inaccessible | Give the agreed unavailable-record response | No private fields enter model messages |
| Ticket service fails | Explain that current status cannot be checked | No invented status is substituted |
| Customer requests a change | Prepare a proposal if this capability is enabled | No update occurs before valid approval |
Agree on what counts as a useful answer before tuning instructions. “The ticket is open; the record does not give a completion date” can be a successful answer. “It will be resolved today” fails when the record contains no such commitment. Include an observation time when freshness matters, and label any permitted cached information. A correct statement about yesterday’s record may be misleading when presented as a fresh lookup.
Next, map the contract into API behavior. Use stable application outcome codes and keep diagnostic detail out of public error messages. FastAPI supports HTTP exceptions and custom exception handlers; your application still decides which failures become which responses. A provider exception containing request details should not become the response body by accident. Test the response payload as well as the status code.
Review exercise: write one synthetic request for every table row. Have a reviewer predict the outcome before running it. Compare the actual API response, model-visible information and any effects with that prediction. If the team disagrees about the expected behavior, resolve the contract first. Prompt changes cannot settle an undefined product rule.
Keep as evidence: the task contract, five observed outcomes, prohibited effects and the destination for unresolved work. Mark the review failed if the application needs an invented answer or an unowned queue to appear successful.
2. Protect identity, permissions and secrets
Authenticate through the application and derive the current user’s identity on the server. A user can select a ticket identifier, but cannot establish ownership by typing a customer name into the prompt. Pass trusted identity and scoped service clients through dependencies. The Pydantic AI dependency guide documents deps_type, per-run deps and access through RunContext. Those mechanisms carry context; the ticket service must enforce access.
OWASP recommends least privilege, default denial and permission checks on every request. Apply that principle where protected records are fetched and where effects occur. A model instruction saying “only access your account” cannot replace a database filter. Check conversation ownership, stored attachments and cached results too. Otherwise a protected tool can coexist with a history endpoint that exposes its previous responses.
| Boundary | Synthetic challenge | Required observation |
|---|---|---|
| Ticket lookup | Customer A selects customer B’s identifier | Only A’s permitted data is eligible for return |
| Conversation storage | Reuse another customer’s conversation ID | History is withheld before agent execution |
| Retrieval cache | Repeat an identical question under two accounts | Cached evidence respects the second account’s scope |
| Provider payload | Inspect a synthetic request containing extra fields | Unnecessary private fields are omitted |
| Service credentials | Attempt a write with the pilot’s read credential | The destination denies the write |
Example 1: verify a scoped lookup through an agent
This tool accepts no model-generated record selector. The application supplies a selected identifier and an authenticated tenant in a fresh dependency object. The local repository indexes both together. Its private sentinel must never appear in the denied run’s messages. The test model invokes the tool, so the exercise covers dependency wiring as well as the repository lookup.
import json
from dataclasses import dataclass
from pydantic_ai import Agent, RunContext, models
from pydantic_ai.models.test import TestModel
models.ALLOW_MODEL_REQUESTS = False
records = {
("tenant-a", "T-1"): {"status": "open", "note": "Awaiting review"},
("tenant-b", "T-2"): {"status": "closed", "note": "PRIVATE-B-741"},
}
@dataclass(frozen=True)
class Deps:
tenant_id: str
selected_ticket: str
agent = Agent(TestModel(), deps_type=Deps)
@agent.tool
def read_selected_ticket(ctx: RunContext[Deps]) -> dict[str, str]:
key = (ctx.deps.tenant_id, ctx.deps.selected_ticket)
record = records.get(key)
if record is None:
return {"outcome": "unavailable"}
return {"outcome": "found", "status": record["status"]}
allowed = agent.run_sync("Read selected ticket", deps=Deps("tenant-a", "T-1"))
denied = agent.run_sync("Read selected ticket", deps=Deps("tenant-a", "T-2"))
missing = agent.run_sync("Read selected ticket", deps=Deps("tenant-a", "T-9"))
assert json.loads(allowed.output)["read_selected_ticket"] == {
"outcome": "found", "status": "open",
}
assert json.loads(denied.output) == json.loads(missing.output)
assert json.loads(denied.output)["read_selected_ticket"] == {"outcome": "unavailable"}
assert b"PRIVATE-B-741" not in denied.all_messages_json()
assert b'"closed"' not in denied.all_messages_json()
assert allowed.usage.tool_calls == 1
assert denied.usage.tool_calls == 1
print("PASS: allowed lookup, cross-tenant denial, missing record, message isolation")The unavailable response deliberately treats missing and inaccessible records alike. Choose that disclosure policy explicitly for your API. This example does not implement authentication, database row policies or cache isolation. Replace the fixture with a trusted session and an authorised repository, then repeat the same observable checks at the HTTP boundary. See the tools and dependencies article for the surrounding tool design.
Retrieved documents remain untrusted even when they come from an internal search service. A ticket comment might tell the assistant to reveal another account or send data to a new address. OWASP’s prompt injection guidance supports enforcing permissions outside the model and restricting available tools. Rehearse a document with malicious instructions; inspect attempted tools and their permitted effects, not just whether the final wording sounds cautious.
Keep as evidence: the access matrix, denied-request messages, cache and history checks, and a field-level map of external destinations. Rehearse credential rotation separately. A replacement API key should work without exposing the old key in logs; rotation cannot repair missing ownership checks.
3. Validate results beyond their shape
A typed result gives the application a predictable structure. It does not establish that a record exists, a citation supports a claim or a proposed change is allowed. Treat those as separate checks with separate failure reasons. This makes diagnosis easier: a malformed field needs a different repair from correct JSON that contains an invented status.
The official output guide describes structured output and agent.output_validator. A validator can inspect trusted application context and request another attempt with ModelRetry. Set a bounded output retry budget. Repeating the same unsupported claim should end in a deliberate failure outcome, not an indefinite attempt to persuade the model. The structured output article develops the distinction between a valid schema and a valid application decision.
| Layer | Example failure | Check before acceptance |
|---|---|---|
| Structure | Status is an unsupported value | Schema validation rejects the field |
| Identity | Output references a different ticket | Identifier matches the authorised selection |
| Evidence | Citation points to an unknown source | Source belongs to the authorised evidence set |
| Fact | Open ticket is described as closed | Claim agrees with authoritative recorded data |
| Business rule | Approved change targets an outdated version | Current state still permits execution |
Example 2: reject plausible but unsupported output
The fixture uses one authorised source whose entire claim is the ticket’s status. That permits an exact evidence check. The bad cases exercise schema rejection, status disagreement, a fabricated source and the wrong record. Output retries are zero so every invalid fixture produces an immediate, observable failure. Production may allow a small repair budget when a second attempt can reasonably help.
from dataclasses import dataclass
from typing import Literal
from pydantic import BaseModel
from pydantic_ai import Agent, ModelRetry, RunContext, models
from pydantic_ai.exceptions import UnexpectedModelBehavior
from pydantic_ai.models.test import TestModel
models.ALLOW_MODEL_REQUESTS = False
class Answer(BaseModel):
ticket_id: str
status: Literal["open", "closed"]
source_id: str
@dataclass(frozen=True)
class Evidence:
ticket_id: str = "T-1"
status: str = "open"
source_id: str = "ticket:T-1:v3"
agent = Agent(
TestModel(), output_type=Answer, deps_type=Evidence,
retries={"output": 0},
)
@agent.output_validator
def verify(ctx: RunContext[Evidence], value: Answer) -> Answer:
expected = ctx.deps
if value.ticket_id != expected.ticket_id:
raise ModelRetry("Ticket does not match selected evidence")
if value.source_id != expected.source_id:
raise ModelRetry("Source is outside the authorised evidence set")
if value.status != expected.status:
raise ModelRetry("Status disagrees with the source")
return value
good = {"ticket_id": "T-1", "status": "open", "source_id": "ticket:T-1:v3"}
with agent.override(model=TestModel(custom_output_args=good)):
result = agent.run_sync("Return the recorded status", deps=Evidence())
assert result.output.status == "open"
bad_cases = [
{**good, "status": "pending"},
{**good, "status": "closed"},
{**good, "source_id": "invented-source"},
{**good, "ticket_id": "T-2"},
]
for payload in bad_cases:
with agent.override(model=TestModel(custom_output_args=payload)):
try:
agent.run_sync("Return the recorded status", deps=Evidence())
except UnexpectedModelBehavior:
pass
else:
raise AssertionError(f"Accepted unsupported output: {payload}")
print("PASS: valid evidence accepted; four invalid outputs rejected")An evidence identifier alone is insufficient for longer answers. A document can exist without supporting the sentence attached to it. For a policy answer, retain the permitted passage and review the relationship between the passage and the claim. For a numeric answer, independently recompute important totals when feasible. Document where deterministic checks end and a person or separately evaluated scoring method must assess meaning.
Also review presentation. If a client displays streamed text before validation finishes, a later rejection cannot erase what the user already read. Decide which outputs are provisional and which are withheld until final checks pass. Never use partial JSON or an unfinished proposal to trigger an effect. Test a stream interrupted just before its last field and verify that the interface does not mark the operation completed.
Keep as evidence: accepted and rejected fixtures, their reasons, the evidence snapshot and a check of the actual client response. Fail this review when unsupported claims receive the same verified presentation as checked facts.
Put this into practice with guided training
Explore the Pydantic AI course for the syllabus, guided projects and training options.
4. Bound calls, retries, time and concurrency
Give the complete user operation a resource budget. It includes incoming content, accumulated history, retrieved passages, model attempts, tool calls and queue waiting. A small question can still produce a large context after retrieval. Bound attachments and retrieved material before constructing a provider request, and set limits appropriate to the task rather than copying unexplained numbers from another application.
Pydantic AI’s usage limits include model request and successful tool invocation limits. They operate on the usage tracked for the run. Catch UsageLimitExceeded at the application boundary and return the agreed outcome. A run limit is not a customer-wide quota or a complete invoice cap: multiple runs, external APIs, storage and infrastructure need their own accounting.
| Control | Main purpose | Separate check |
|---|---|---|
| Input and history bounds | Limit accepted and assembled content | Retrieved text and attachment expansion |
| Run usage limits | Bound tracked model requests and tool work | Usage across users, runs and services |
| Attempt timeout | Stop waiting for one provider or service attempt | Provider-specific setting support and retries |
| Operation deadline | Bound the user’s complete wait | Whether remote effects continue |
| Concurrency and admission | Limit active work and waiting requests | Worker count, fairness and overload responses |
Example 3: exercise request and tool budgets
A zero allowance is useful for testing the denial path. The positive controls establish that work succeeds with a sufficient allowance. The tool counter matters: observing an exception alone would not prove the tool stayed unexecuted. TestModel’s reported usage is synthetic and should not be interpreted as a billing estimate.
from pydantic_ai import Agent, models
from pydantic_ai.exceptions import UsageLimitExceeded
from pydantic_ai.models.test import TestModel
from pydantic_ai.usage import UsageLimits
models.ALLOW_MODEL_REQUESTS = False
plain = Agent(TestModel(custom_output_text="ready"))
try:
plain.run_sync("Check", usage_limits=UsageLimits(request_limit=0))
except UsageLimitExceeded:
pass
else:
raise AssertionError("Zero request allowance was ignored")
result = plain.run_sync("Check", usage_limits=UsageLimits(request_limit=1))
assert result.output == "ready"
assert result.usage.requests == 1
calls = []
with_tool = Agent(TestModel(call_tools=["read_status"]))
@with_tool.tool_plain
def read_status() -> str:
calls.append("read")
return "open"
try:
with_tool.run_sync("Read", usage_limits=UsageLimits(tool_calls_limit=0))
except UsageLimitExceeded:
pass
else:
raise AssertionError("Zero tool allowance was ignored")
assert calls == []
result = with_tool.run_sync(
"Read", usage_limits=UsageLimits(request_limit=2, tool_calls_limit=1),
)
assert calls == ["read"]
assert result.usage.tool_calls == 1
assert result.usage.requests == 2
print("PASS: request denial, request allowance, tool denial, tool allowance")Count retries across layers. If an application retries an entire run three times and a client makes three attempts inside each run, one user operation can produce nine client attempts. Validation retries, transport retries and queue redelivery are different mechanisms. Decide which layer owns recovery, retain the original operation identifier and refuse a retry when the remaining deadline cannot support another useful attempt.
The Pydantic AI timeout guide distinguishes model attempts, tool timeouts and whole-run duration. Provider support for model timeout settings varies. A synchronous tool may continue in its worker thread after the caller stops waiting. For the overall asynchronous operation, Python’s timeout context manager supports a surrounding deadline. Cancellation still does not prove a remote update was undone; reconcile effects separately.
Load-test admission as well as execution. Record active requests, queue length, rejected work and waiting time with the intended worker count. A process-local semaphore is not a fleet-wide quota. For FastAPI services, shared clients can be created and closed through the documented lifespan mechanism. Check connection pool exhaustion and shutdown behavior with those actual clients. A fast single request provides little evidence about contention.
Keep as evidence: chosen budgets, limit failures, observed attempt counts, deadline behavior and a workload report. State arrival rate, request mix, worker count and percentile latency. Keep synthetic offline checks separate from provider throttling and capacity results.
5. Make actions recoverable and deliberate
A write capability needs an operation design. Store the proposed target, intended change, relevant record version and approval state before execution. A chat message saying “yes” is meaningful only when the application can identify what was approved. If the proposal changes, its previous approval must not authorise the replacement. Recheck access at execution because permissions may change while a proposal waits.
OWASP’s transaction authorisation guidance supports showing significant transaction data and enforcing authorised state transitions on the server. For this assistant, the approval view should identify the ticket, old status, proposed status and any important consequences. Rendering a generic “Approve action” button without the actual target makes it difficult for a reviewer to spot a mistaken proposal.
| State or event | Required response | Evidence to retain |
|---|---|---|
| Proposal awaits approval | Prevent execution | Proposal identity and pending state |
| Approved content changes | Reject the changed operation | Mismatch and requirement for fresh approval |
| Record changes meanwhile | Revalidate against current state | Version conflict and review outcome |
| Acknowledgement is lost | Query or reconcile the existing operation | Original identifier and destination receipt |
| Same operation is repeated | Return its result without another effect | One business effect for that operation |
Example 4: bind approval and reject duplicate effects
This local rehearsal models immutable proposals, current permissions and receipts. Approval fixtures represent records written by a trusted approval service, not fields supplied by the model. It checks an approved action, a repeated request, a changed proposal, missing approval, stale state and revoked access. Its dictionaries are intentionally process-local; the exercise demonstrates rules, not durable or concurrent transaction handling.
from dataclasses import dataclass, replace
from pydantic_ai import models
models.ALLOW_MODEL_REQUESTS = False
@dataclass(frozen=True)
class Proposal:
tenant: str
ticket: str
expected_version: int
new_status: str
record = {"tenant": "tenant-a", "ticket": "T-1", "version": 3, "status": "open"}
proposal = Proposal("tenant-a", "T-1", 3, "closed")
approved = {("tenant-a", "op-1"): proposal}
completed = {}
permissions = {("reviewer-a", "tenant-a", "T-1")}
effects = []
def execute(actor: str, operation_id: str, value: Proposal) -> str:
key = (value.tenant, operation_id)
if (actor, value.tenant, value.ticket) not in permissions:
raise PermissionError("Current access denied")
if approved.get(key) != value:
raise ValueError("Approval does not match this proposal")
if key in completed:
previous, receipt = completed[key]
if previous != value:
raise ValueError("Operation identifier reused with changed content")
return receipt
if (value.tenant, value.ticket) != (record["tenant"], record["ticket"]):
raise ValueError("Unknown target")
if value.expected_version != record["version"]:
raise ValueError("Record changed after proposal")
if value.new_status not in {"open", "closed"}:
raise ValueError("Unsupported status")
record.update(status=value.new_status, version=record["version"] + 1)
effects.append(key)
receipt = "receipt-" + operation_id
completed[key] = (value, receipt)
return receipt
assert execute("reviewer-a", "op-1", proposal) == "receipt-op-1"
assert execute("reviewer-a", "op-1", proposal) == "receipt-op-1"
assert len(effects) == 1
assert record["version"] == 4
def must_reject(operation_id: str, value: Proposal, error: type[Exception]) -> None:
try:
execute("reviewer-a", operation_id, value)
except error:
return
raise AssertionError("Invalid operation was accepted")
must_reject("op-1", replace(proposal, new_status="open"), ValueError)
must_reject("unapproved", proposal, ValueError)
approved[("tenant-a", "op-stale")] = proposal
must_reject("op-stale", proposal, ValueError)
permissions.clear()
must_reject("op-1", proposal, PermissionError)
assert len(effects) == 1
print("PASS: approval, replay, changed payload, missing approval, stale state, revoked access")Production needs atomic persistence around the decision and effect, or a destination that accepts and enforces an idempotency key. A dictionary check followed by a write can race across concurrent requests and disappears after restart. Even a local database transaction cannot atomically include an unrelated remote API. Use the destination’s operation lookup or a reconciliation design for that gap. Do not claim exactly-once execution merely because your caller remembers an identifier.
Rehearse the difficult interruption: the destination applies the update, then the acknowledgement is lost. Restart the worker and recover using the original operation identifier. Check the destination’s record and receipt before another attempt. Pydantic AI offers durable execution integrations for workflows that must survive interruption; the selected backend and action semantics still need their own recovery test.
Keep as evidence: one approved proposal, rejected modifications, a lost-acknowledgement rehearsal and before-and-after effect counts. Leave writes disabled when duplicate or uncertain effects cannot be reconciled. A separately reviewed read-only capability can have a different release decision.
6. Add useful observability without excessive data capture
An operator should be able to explain a failed request without first opening its entire conversation. Start with a server-generated request identifier, release identifier, outcome, component, elapsed time and attempt counts. Connect the API request to agent and tool activity. Use a separate operation identifier for effects so a retried request can still be associated with the same proposed update.
The Logfire integration guide describes tracing Pydantic AI activity. Before exporting traces, decide which destinations receive which fields and who may read them. Prompts, tool arguments, source passages and exception messages can contain confidential material. A useful trace records that a permission check failed without requiring the protected ticket’s contents.
| Operational question | Useful fields | Collection check |
|---|---|---|
| Which release regressed? | Release ID and outcome category | Every compared request has a known release |
| Where did time go? | Queue, model and tool durations | Waiting time is distinguishable from execution |
| Why did work repeat? | Operation ID and attempt number | Redeliveries preserve the original operation link |
| Was access denied? | Policy decision and component | No protected record is embedded in the event |
| Is cost changing? | Model identity and available usage totals | Provider billing is reconciled separately |
Pydantic AI’s instrumentation settings include include_content=False for omitting prompts, completions and tool arguments and responses from the relevant events. This is not a universal redaction switch. Your application logger, HTTP middleware, exception tracker and custom span attributes can independently collect content. Inspect exported data from each destination, including failed requests, before relying on a configuration setting.
Example 5: construct a small operational event
This example builds an event from explicitly selected fields and a controlled outcome vocabulary. It checks that a synthetic secret cannot enter through a discarded message field or through an unsupported outcome. The function is application logic; it does not configure Logfire, inspect an exporter or implement a general-purpose secret detector.
import json
import re
from pydantic_ai import models
models.ALLOW_MODEL_REQUESTS = False
OUTCOMES = {"answered", "denied", "unavailable", "needs_review"}
def event_for(request_id: str, release_id: str, result: dict) -> dict:
for identifier in (request_id, release_id):
if not re.fullmatch(r"[a-z0-9-]{1,48}", identifier):
raise ValueError("Invalid operational identifier")
outcome = result.get("outcome")
if outcome not in OUTCOMES:
raise ValueError("Unrecognised outcome")
duration = result.get("duration_ms")
if type(duration) is not int or not 0 <= duration <= 3_600_000:
raise ValueError("Invalid duration")
return {
"request_id": request_id,
"release_id": release_id,
"outcome": outcome,
"duration_ms": duration,
}
fake_secret = "SYNTHETIC-SECRET-962"
raw = {
"outcome": "denied", "duration_ms": 12,
"message": fake_secret, "tool_arguments": {"token": fake_secret},
}
event = event_for("req-17", "release-4", raw)
assert set(event) == {"request_id", "release_id", "outcome", "duration_ms"}
assert fake_secret not in json.dumps(event)
assert event["outcome"] == "denied"
for payload in ({**raw, "outcome": fake_secret}, {**raw, "duration_ms": -1}):
try:
event_for("req-17", "release-4", payload)
except ValueError:
pass
else:
raise AssertionError("Invalid telemetry field was accepted")
print("PASS: selected fields only, secret excluded, invalid outcome and duration rejected")The identifiers must come from trusted application code. A character pattern cannot prove that a permitted-looking identifier is free of private information. Do not put customer email addresses in request IDs, and do not use request IDs as labels on aggregate metrics. High-cardinality labels make metrics difficult to operate and may expose identifiers to more people than a restricted trace would.
OWASP’s logging guidance covers excluding sensitive data, restricting access and verifying logging behavior. Apply the chosen retention period to each destination, including copies and exports. Where detailed content is justified for a particular investigation, make the scope and retention explicit. Routine troubleshooting should not quietly become indefinite storage of all conversations.
Review exercise: inject a distinctive fake secret into an input, a tool failure and an exception. Follow each request through the actual export pipeline and inspect stored events. Then trigger one actionable alert in a test environment. Confirm that the responsible operator can find the request and decide what to disable without needing a developer to interpret every field.
Keep as evidence: the collected-field list, export inspection, access and retention settings, and an acknowledged test alert. Also rehearse an exporter outage: ordinary diagnostic failures should not silently remove required action auditing or make the request path wait without a bound.
Put this into practice with guided training
Explore the Pydantic AI course for the syllabus, guided projects and training options.
7. Version and evaluate changes
Define a release identity that links application code, the dependency lock, prompt instructions, tool definitions, output schema, model configuration and retrieval snapshot. Include enabled capabilities and operational limits. When an answer changes, you need enough information to determine whether the cause was a model setting, a new source document, a package update or an altered permission rule.
Keep the exact dependencies used for the verified release. The versions stated in this article describe its test environment, not a claim that later versions behave identically. Record the resolved provider model identifier when available, as an alias alone may not identify a permanently fixed implementation. Repeat relevant checks when changing a provider or output mode; a schema that works with TestModel may still encounter provider-specific restrictions.
| Method | What it establishes | What remains unverified |
|---|---|---|
| Local deterministic examples | Specific control paths behave as asserted | Language understanding and live model quality |
| API integration tests | Identity, status codes and payload handling work together | Production dependencies unless included |
| Representative model evaluation | Observed quality on a defined dataset | All future inputs and untested populations |
| Load and fault rehearsal | Behavior under the tested demand and faults | Higher demand or different infrastructure |
| Limited monitored pilot | Behavior for the participating audience | Unrestricted deployment readiness |
The official unit-testing guide documents TestModel and disabling real model requests. TestModel generates controlled responses; it does not reason about the prompt like a production model. Consequently, a sentence asking the model to respect privacy is not tested merely because a canned answer passes. Assert the application boundary, then evaluate language behavior separately. The Pydantic AI testing guide explains how to organise these layers.
Build the evaluation set around the task contract. Include routine tickets, missing selections, contradictory evidence, stale records, requests outside scope, ambiguous approval language and malicious source instructions. Label the required behavior before comparing releases. Keep development examples separate from a held-back set so improvements are not judged only on the cases repeatedly used while editing prompts.
Pydantic Evals provides datasets, cases and evaluators for systematic comparisons. Choose a rubric that represents the actual job: correct status, supported claims, appropriate escalation and useful completion. Report denominators and failure categories. For example, 92 correct answers out of 100 answerable cases means something different from 92 correct answers among only the cases the new version chose to answer.
Never hide an access leak or unintended write inside a high average quality score. Track these as release-blocking outcomes with individual case review. At the same time, measure unnecessary refusal and escalation. A release that sends every question to a person can have few incorrect answers while creating an unusable service and an overflowing queue.
Compare the candidate with the currently approved release on the same evidence snapshot. Record model runs and settings, including any repeated evaluations used to understand variability. Investigate important disagreements manually. When an automated judge is involved, retain its rubric and version and check a sample of its decisions against human review. A judge’s score is an assessment method with its own limits.
Finally, test saved state across the upgrade. A new required output field can break pending proposals created by the old release. Decide whether those operations are migrated, completed under their original version or cancelled with a visible outcome. Exercise the same choice during rollback. A restored code package is insufficient when its database records no longer match the schema it expects.
Keep as evidence: the release manifest, deterministic results, evaluation dataset version, before-and-after outcomes and the saved-state transition test. Mark provider evaluations not run when they were not run; the offline exercises in this article cannot substitute for them.
8. Plan the pilot, ownership and rollback
Choose an audience, allowed task, observation period and supported operating hours before inviting users. Assign real people or maintained team roles to service operation, access policy, data handling and review queues. Describe who can stop model traffic, disable a tool and restore the previous configuration. Ownership must include cover for absence; a name attached to an unread inbox is not an operational plan.
Set expansion and stop conditions using the consequences of failure. For the support assistant, cross-account disclosure or an unapproved write should stop the affected capability immediately. An increase in low-impact clarification requests may justify investigation without the same response. Do not invent universal latency or accuracy targets. Agree on thresholds using user expectations, workload measurements and the capacity of the human fallback.
| Stage | Work to complete | Decision evidence |
|---|---|---|
| Freeze the candidate | Identify code, configuration, data and enabled tools | Reproducible release manifest |
| Run local controls | Check access, validation, budgets and action rules | Observed passes, failures and tests not run |
| Rehearse the service | Test API behavior, load, outages and recovery | Results tied to the candidate environment |
| Approve the pilot scope | Resolve blockers and assign operating ownership | Written decision for the exact capability set |
| Observe and decide | Review user outcomes, queues and resource use | Expand, hold, narrow or stop with a recorded reason |
Walk one candidate through the launch decision
First freeze the candidate and collect the eight review records in one place. Each record should identify the check, input fixture, expected observation, actual result and owner. Link to retained evidence that another developer can access. A screenshot of a green dashboard is weaker than a repeatable check with a known release and failing case.
Second, run the offline controls and API tests using synthetic accounts. Follow with the necessary provider evaluation and infrastructure rehearsals in the approved test environment. These are separate activities with separate permissions and costs. Do not mark them passed because the local examples passed. Check external credentials, actual provider schema support and destination permissions before moving on to a limited audience.
Third, review unresolved findings against the proposed scope. Suppose access isolation, status evidence and timeout handling pass, but replaying a completed update produces another effect. Keep writes disabled. A read-only pilot may still be considered after removing write credentials and tools, updating instructions and responses, and rerunning the relevant checks on that narrower configuration. Record the duplicate-effect defect as unresolved, with an owner.
Fourth, rehearse stopping the candidate. Disable new write execution at the service that performs writes, not only at the chat interface. Inspect already queued and in-progress operations. A worker holding old configuration must not continue accepting newly forbidden effects unnoticed. Decide how it learns the stop decision and verify that behavior with an actual queued item.
Fifth, restore the previous known configuration and exercise a representative request. Check saved-state compatibility, routing and client behavior. Tell users what happened to interrupted work through the existing application channel. Rolling back cannot unsend a notification or automatically reverse every remote status change. Operations with uncertain outcomes still need reconciliation after the software is restored.
During the pilot, review outcomes by category and audience, not only overall request volume. Look at unsupported answers, unnecessary escalation, unresolved review age, denied access, exhausted budgets and uncertain actions. Include requests abandoned by users, since excluding them can make latency and completion look healthier than the experience actually was. Agree who decides whether the evidence supports expanding the audience.
Diagnose common failures before changing limits
Preserve one failing request and its release identity before changing several settings. Correlate the API event, agent run, tool attempt and destination receipt where applicable. A longer timeout can conceal a queue problem, and extra retries can intensify an outage. Repair the component responsible for the observation, then repeat the original failing case.
| Observed symptom | Possible explanation | Next diagnostic check |
|---|---|---|
| User times out but work continues | Only the HTTP wait ended | Trace worker completion and query destination effects |
| Provider attempts multiply | Several layers retry independently | Count attempts under one operation and deadline |
| Typed answers contain wrong facts | Only structural validation runs | Compare claims with authorised evidence |
| Accounts see inconsistent data | Cache or history scope differs from tool scope | Repeat the request under isolated synthetic identities |
| Only concurrent requests fail | Shared state, pool exhaustion or overloaded queues | Inspect dependency lifetimes and waiting time |
| Quality drops after document refresh | Evidence changed or retrieval omitted relevant text | Compare source snapshots and retrieved passages |
| Rollback leaves pending work broken | Stored schema or operation state is incompatible | Replay a saved item against the restored version |
The Pydantic AI FastAPI guide helps connect these application outcomes to endpoint behavior. Retain a support route outside the assistant itself, so users can report problems during an outage. If the fallback queue cannot handle pilot demand, reduce the scope or pause invitations instead of silently dropping requests.
Keep as evidence: the signed-off scope, real operating owners, stop conditions, rollback rehearsal and treatment of interrupted operations. The final decision should say exactly what is approved and what remains disabled. Revisit it after material changes to tools, permissions, model configuration or source data.
Frequently asked questions
Does using Pydantic AI make an application production-ready?
No. Pydantic AI supplies agent execution, typed outputs and supporting integrations. Production readiness concerns the complete application, including identity, services, stored state, user experience and operations. A framework cannot decide your business permissions or prove your fallback works. Use these eight reviews to assemble evidence for a specific release and task. Treat approval as limited to that scope, rather than a permanent label attached to the project.
What does passing all five offline examples actually prove?
It shows that the stated dependency lookup, output checks, usage limits, local approval rules and event projection behave as asserted in the tested environment. It does not prove that a real model understands ambiguous requests, that authentication is implemented correctly or that external writes survive a restart. The official testing approach deliberately separates deterministic control checks from model evaluation. Keep those evidence categories separate in your release record.
Can I launch read-only access while write recovery is unfinished?
Possibly, after evaluating the narrower release. Remove write tools and credentials, update the interface and rerun checks against the resulting configuration. Read-only access still needs data isolation, evidence validation, limits, monitoring and operational ownership. Record the unfinished write capability as disabled and unresolved. A prompt asking the agent not to write is insufficient when its tools or service credentials can still perform the action.
How many retries and how much timeout should I allow?
Choose these from the task’s deadline, observed latency and effect semantics. Measure normal and degraded conditions, decide which failures merit another attempt and budget all layers together. Stop retrying when another attempt cannot fit the remaining useful wait. For writes, reconcile uncertain effects before repeating them. The zero limits in the example are deliberate test settings; they are not recommended production defaults or a substitute for workload measurements.
Does a request limit cap my complete operating bill?
No. A request limit bounds tracked requests within its configured run scope. Request sizes can differ, many runs can execute concurrently, and tools or infrastructure can have separate costs. Review available token and cost controls for the deployed version and provider, including when those checks take effect. Reconcile application usage with provider records, and set account or customer budgets separately. Never infer a complete spend guarantee from one passing limit test.
Should I store every prompt to make incidents easier to investigate?
Start with the operational questions and collect fields that answer them. Request identifiers, release identity, outcomes, durations and attempt counts often locate the failure without full content. Where content capture is justified, define access, scope and retention and verify actual exported records. Turning off framework message capture does not change unrelated application logs or exception trackers. Test each destination with synthetic sensitive values before trusting the combined setup.
What should block release even when average quality is good?
Unresolved access leaks, unintended consequential actions, repeatable duplicate effects and an inability to stop the service should block the affected release scope. Missing evidence for an essential control should also remain visible as not run. Other quality defects may permit a monitored pilot when users have an effective fallback and the remaining task is useful. Record the rationale, owner and conditions for reconsideration instead of averaging every failure into one score.
For structured practice with the underlying concepts, explore the Pydantic AI course and compare its published learning outcomes with the areas you need to strengthen. A useful project submission includes the task contract, runnable checks and an honest release record. Training and a working demonstration support learning; deployment approval still depends on evidence from the application and environment being released.
Download the Pydantic AI practice pack
Get nine offline Python examples, a project-review checklist and an evaluation case sheet. The examples use simulated model responses and do not need an API key. Submit this form to open the ZIP download.
Brolly Academy will store your request and use your details to handle it. Course follow-up is optional; the checkbox is not selected automatically.
Read our Privacy Policy for information about handling your details.
Put this into practice with guided training
Explore the Pydantic AI course for the syllabus, guided projects and training options.
