Pydantic AI Interview Questions
120 Practical Answers, Coding Examples and Interview Scenarios
These Pydantic AI interview questions take you from Python basics to real agent-application decisions. Work through 120 detailed answers with coding examples, comparison tables and practical failure scenarios.
Learn how to explain typed outputs, tools, RAG, MCP, FastAPI, testing and deployment in your own words. These are practice questions, not a claim about any employer’s interview.
Share
Table of Contents
Choose your interview preparation path
Start with the section matching the job description. Explain an answer aloud before reading it again. For implementation questions, describe the input, the decision, a failure and how you would check the result. Be clear about work you have actually done.
| Focus | Questions | What to demonstrate |
|---|---|---|
| Python and agent foundations | 1-30 | Explain a run and configure a small agent. |
| Typed outputs and tools | 31-60 | Validate a result and call a controlled service. |
| Knowledge and integrations | 61-80 | Retrieve evidence and connect an MCP tool. |
| Backend and workflow design | 81-100 | Handle requests, streaming and multi-step work. |
| Quality and production readiness | 101-120 | Test failures, defend boundaries and investigate incidents. |
New to the framework? Start with the Pydantic AI beginner tutorial. The examples below use fixed test responses or ordinary Python so you can practise without model charges. A passing fixture test does not measure a live model’s accuracy.
Fundamentals and Python prerequisites
Understand where Pydantic AI fits in a Python application, which Python concepts its APIs rely on, and how to explain its benefits without overstating its guarantees.
1. What problem does Pydantic AI solve for a Python application developer?
Pydantic AI provides a Python framework for building applications around language models, including typed outputs and an agent interface for managing model interactions. Its official overview describes these building blocks. The useful interview point is what your application gains: a defined boundary between model-generated content and the code that consumes it.
Consider a support inbox that needs a category, a summary, and an urgency level for every message. An agent can produce that defined result instead of leaving each caller to extract fields from prose. A backend developer still needs rules for failures, ambiguous requests, and downstream actions. Successful validation confirms the declared data requirements; it does not establish that the model understood the customer correctly.
2. How is Pydantic AI different from the Pydantic validation library?
Pydantic validates and represents data; Pydantic AI manages interactions with language models and uses Pydantic within that process. A Pydantic model does not independently ask an LLM to fill its fields. An agent supplies the model interaction around the data contract.
| Component | Responsibility | Example |
|---|---|---|
| Pydantic | Validate a declared shape | Check that quantity is a positive integer |
| Pydantic AI | Request and process model output | Ask for order details from an email |
| Application | Apply business decisions | Confirm stock before accepting an order |
For a data engineer, this distinction prevents a common design mistake: treating extraction, validation, and acceptance as the same operation. A well-formed order can still name an unavailable product.
3. When would you choose an agent, a direct model request, or ordinary Python code?
Choose an agent when you need its managed output handling and interaction flow. Choose a direct model request when you need closer control over an individual exchange. Use ordinary Python when the task already has a reliable deterministic solution. Pydantic AI documents the distinction between its direct API and Agent API.
For example, interpreting a loosely written shipping complaint may justify an agent. A platform engineer implementing a custom request pipeline may prefer the direct API. Calculating a delivery surcharge from a published weight table should usually remain a normal function. Compare the amount of behavior you must build yourself, the variability you can tolerate, and the ease of testing. Adding an LLM to a fixed calculation introduces failure modes without adding useful interpretation.
4. What happens during a basic Pydantic AI agent run?
A run prepares the request, obtains a model response, processes it, and finishes when an acceptable final output is available. Validation failures can lead to another attempt within the configured limits. The run API exposes the execution state and final result.
Imagine extracting an appointment from a message. The first response might contain an invalid date; a later response may satisfy the output contract. Those exchanges belong to one run, so one application call need not equal one model request. For a backend interview, explain where you would report success: after the final result is accepted, not immediately after receiving response bytes. Authentication failures, exhausted retries, and invalid output need distinguishable handling because each points to a different remedy.
5. What does the word model mean in Pydantic AI, and why can it be confusing?
A language model generates a response, while a Pydantic data model defines fields and validation rules. Pydantic AI also has model adapter classes that translate between its interfaces and a model service. The models and providers guide describes that integration layer.
In an invoice extractor, the remote language model reads the supplied text; an adapter handles the request format; an Invoice class describes the result your Python code expects. Changing the invoice fields does not train the language model. Changing the remote model does not automatically update your downstream database contract. During a design discussion, say ‘language model,’ ‘model adapter,’ or ‘output schema’ explicitly. That small vocabulary choice makes configuration bugs and schema changes much easier to discuss.
6. Do Python type hints automatically validate values returned by an LLM?
No. Python annotations describe intended types, but Python does not generally enforce them when a function runs. Pydantic supplies runtime validation where it is explicitly used. The Python typing documentation and Pydantic AI’s typed agent design address these different responsibilities.
Writing def classify(text: str) -> int does not stop an ordinary implementation from returning a string. A static checker may flag that mistake before execution; validating external data catches unsuitable values during execution. For an application engineer, both matter: type checking helps catch incorrect use of the result, while runtime checks guard the model boundary. Neither test answers whether a valid integer is the correct category for the original text.
7. Why are Literal and Enum useful for classification outputs?
They express a finite set of permitted choices instead of allowing any string. Literal['billing', 'technical', 'other'] suits a small label set; an enum is useful when the same named choices appear throughout an application. Pydantic documents validation of both in its standard library types reference.
A support-routing classifier should not casually invent labels such as ‘money issue’ when the queue system accepts ‘billing.’ A finite type makes that mismatch detectable. Include a deliberate uncertain or other route when the business process needs one, and define what happens there. For an ML engineer, measure classification accuracy separately: an allowed label can still be the wrong label. Also treat adding a new choice as a contract change for every consumer.
8. What is the difference between a nullable field and a field with a default?
A nullable field accepts None; a default lets validation fill in a value when the field is omitted. In Pydantic, str | None alone does not supply a default. The fields guide explains this distinction.
For a profile extractor, requiring middle_name while allowing null asks for an explicit answer about that field. Adding = None also permits the key to be absent. Choose deliberately: silently defaulting a missing quantity to zero can turn incomplete extraction into an apparently complete record. The example checks requiredness without calling a language model. A missing key, an explicit null, and an empty string may deserve different application behavior even when users describe all three as ‘blank.’
from pydantic import BaseModel
class RequiredNullable(BaseModel):
middle_name: str | None
class DefaultedNullable(BaseModel):
middle_name: str | None = None
assert RequiredNullable.model_fields['middle_name'].is_required()
assert not DefaultedNullable.model_fields['middle_name'].is_required()
assert RequiredNullable(middle_name=None).middle_name is None
assert DefaultedNullable().middle_name is None
9. When should you use await agent.run instead of agent.run_sync?
Use await agent.run(...) inside asynchronous application code. Use agent.run_sync(...) from a synchronous entry point without an already running event loop. The Agent API documents the synchronous wrapper and its event-loop restriction.
An asynchronous web handler should await the agent so other work can proceed while it waits. A small command-line script can use the synchronous wrapper for simplicity. In notebooks that already run an event loop, top-level await is often the appropriate entry point. The offline example demonstrates the asynchronous call from a standalone script. Its fixed response checks the calling pattern; it does not measure a model’s reasoning quality or simulate real service latency.
import asyncio
from pydantic_ai import Agent
from pydantic_ai.models.test import TestModel
async def main():
agent = Agent(TestModel(custom_output_text='Ready'))
result = await agent.run('Reply with the status.')
assert result.output == 'Ready'
asyncio.run(main())
10. What should a Python developer understand about decorators before using Pydantic AI?
A decorator receives a function when that function is defined; the registered function can then be called later for a particular purpose. For example, @agent.instructions registers a function that contributes instructions during a run. Pydantic AI explains this under dynamic instructions.
Suppose the instructions need today’s date. Returning the date from the registered function lets it be calculated when the run needs it. Computing the date once in a module-level string can leave a long-running service using yesterday’s value. An interview answer should distinguish registration time from execution time and explain ordinary Python function arguments and return values. The decorator does not make arbitrary Python statements part of a prompt, and simply defining a function does not make the agent use it.
11. How do a JSON string, a Python dictionary, and a validated output object differ?
JSON is a text representation, a dictionary is an in-memory Python mapping, and a validated output object has passed the checks associated with its declared type. Parsing text into a dictionary does not by itself validate your business fields. Pydantic’s JSON documentation covers validation directly from JSON.
For example, {"quantity": "many"} is valid JSON and becomes a valid dictionary, but it is unsuitable for a schema that requires an integer quantity. In a data-ingestion interview, explain which boundary has failed: text syntax, field shape, or domain rules. When Pydantic AI already returns a typed result.output, repeatedly converting it to text and reparsing it adds complexity and can obscure the error that actually needs attention.
12. How should you explain validation errors instead of hiding them behind a generic failure?
A useful explanation identifies the failing field and the rule it violated. Pydantic’s error handling reference describes structured validation details, including the location, error type, and message. These details help separate bad input from a programming mistake.
For an extracted order, ‘items, index 2, quantity must be positive’ gives an engineer much more direction than ‘AI failed.’ Catch errors at a boundary where you can take a meaningful action, such as requesting correction or returning an invalid-input response. Do not turn every exception into an empty successful record: callers may then process missing data as real data. A backend developer should also avoid exposing raw error inputs indiscriminately when they contain customer text or other sensitive values.
13. What do the generic parameters in Agent[object, Invoice] communicate?
The first parameter describes the dependency type and the second describes the output type. In Agent[object, Invoice], the output contract is Invoice; these parameters let type-aware tools follow that information into the run result. Pydantic AI explains this in its type-safe agent design.
For an application developer, the practical benefit appears when using result.output: the editor can offer invoice fields, and a checker can flag a nonexistent attribute before deployment. The generic annotation alone is not a substitute for configuring output_type=Invoice. Keep the runtime configuration and annotation consistent. This matters especially in helper functions that return agents, where an overly broad return annotation can hide useful information from the rest of the codebase.
14. How would you make a Pydantic AI development environment reproducible?
Use an isolated environment, record the Python version, and lock the package versions and provider dependencies your application needs. The installation guide specifies Python 3.10 or later and documents both the standard and slim distributions. The slim option lets you select the integrations you need.
For a platform engineering interview, describe reproducing a failure with the same dependency lock before upgrading anything. Record Pydantic AI and Pydantic separately because both affect the code you run. Check imports and execute a small offline example after installation. A version pin alone does not freeze a hosted model’s behavior, credentials, or endpoint configuration. Those belong in the deployment and evaluation record as well, with secrets kept outside the committed project files.
15. How can you demonstrate a first Pydantic AI agent without an API key?
Use TestModel to exercise the agent interface locally. It can provide a configured response without calling a remote language model. Pydantic AI’s testing guide explains its purpose and limitations.
The following example creates an agent, runs it synchronously, and reads the final output. A junior developer can inspect the application flow before setting up a provider account. However, the returned greeting is supplied by the test setup; it is not evidence that a real model followed the prompt. Treat this as a wiring check. Later checks against the intended service must cover actual model behavior, supported output modes, and failures that a fixed local response cannot reproduce.
from pydantic_ai import Agent
from pydantic_ai.models.test import TestModel
agent = Agent(TestModel(custom_output_text='Hello, learner!'))
result = agent.run_sync('Give a short greeting.')
assert result.output == 'Hello, learner!'
print(result.output)
Go deeper with our beginner tutorial.
Agents, models and configuration
Explore how agent configuration, model adapters, request settings, and conversation state work together in a Python service.
16. Should a service create a new Agent for every incoming request?
Usually, an application can reuse an agent whose configuration stays the same. Pydantic AI describes agents as reusable objects in its agent guide. Creating one for every request is unnecessary when only the prompt and conversation history change.
A support service might define one summarization agent during application setup, then pass each ticket’s text to a separate run. Keep request-specific data out of mutable global variables and avoid changing shared configuration while other requests use it. Reusing the agent does not mean reusing every customer’s messages. A backend engineer should distinguish the reusable component from each execution and from application-owned conversation storage. Separate agents make sense when responsibilities or output contracts are genuinely different, such as summarization versus invoice extraction.
17. When should you use a model name string instead of an explicit model object?
Use a provider-prefixed model string for straightforward configuration; construct a model object when you need explicit integration settings or a customized provider. The model configuration guide supports both approaches, including selecting a model when a run starts.
For example, a small service can read its approved model identifier from deployment configuration. A service that talks through a custom endpoint may need an explicit provider and adapter instead. Centralize this decision so the same model selection logic is not copied across handlers. Also validate what configuration is allowed: accepting an arbitrary user-supplied model identifier could bypass your intended cost or capability choices. Swapping a string is easy; demonstrating equivalent application behavior still requires representative checks against the replacement.
18. How do model adapters, providers, and model profiles differ?
An adapter handles a model API’s request and response format, a provider supplies the connection to a service, and a profile describes model capabilities and relevant behavior. The official provider and profile references describe these separate responsibilities.
| Layer | Typical concern | Useful question |
|---|---|---|
| Adapter | API protocol | Is this the correct request format? |
| Provider | Client, endpoint, credentials | Are requests reaching the intended service? |
| Profile | Supported output features | Does the model support this output mode? |
A platform engineer investigating rejected structured output should not immediately rotate credentials. First determine whether authentication failed, the endpoint expects a different protocol, or the chosen feature is unsupported. Updating a profile describes capabilities; it cannot add capabilities to the remote model.
19. What belongs in agent instructions, and what belongs in the user prompt?
Put the application’s behavioral guidance in instructions and the current request or source material in the user prompt. Pydantic AI supports configured instructions and a prompt supplied to its run methods.
For an email summarizer, instructions might require a short factual summary and prohibit inventing missing dates. The prompt contains the actual email being summarized. Keeping those responsibilities clear helps a product engineer change the summarization policy without rewriting every input. Delimit quoted material so it is clear what should be analyzed, but do not mistake prompt formatting for a security guarantee. A customer email that says ‘ignore previous instructions’ is still customer content. Validate the output and enforce any consequential application decisions in ordinary code.
20. How do instructions differ from system_prompt when continuing a conversation?
With explicit message history, Pydantic AI uses the current agent’s instructions rather than carrying forward instructions from earlier messages. System prompts in history can remain part of subsequent requests. The instructions documentation recommends instructions unless retained system prompts serve a specific purpose.
Consider handing a draft from a creative writing agent to a strict fact-checking agent. You generally want the second agent’s current rules to govern its task, without accumulating every earlier instruction. By contrast, an intentionally persistent system prompt may belong to a conversation design that requires continuity. A developer should inspect the actual request context when behavior is surprising. Passing history changes more than the user’s visible dialogue, so copying messages blindly can preserve guidance the new stage was never meant to use.
21. Which model settings win when defaults and overrides disagree?
For ordinary configuration, run settings override agent settings, which override model defaults; unspecified keys remain available from earlier layers. The current settings guide also places capability settings between agent and run settings when capabilities are involved.
A summarizer might default to a moderate token allowance, while one request asks for a shorter response. Override the allowance for that run without mutating shared settings. The offline example observes the resolved values through FunctionModel, whose API exposes request settings. It verifies both an overridden key and a retained key. This catches a subtle configuration bug: assuming that a run-level dictionary replaces every default instead of contributing overrides to the existing configuration.
from pydantic_ai import Agent
from pydantic_ai.messages import ModelResponse, TextPart
from pydantic_ai.models.function import FunctionModel
def respond(messages, info):
assert info.model_settings['temperature'] == 0.1
assert info.model_settings['max_tokens'] == 120
return ModelResponse(parts=[TextPart('Configured')])
model = FunctionModel(respond, settings={'temperature': 0.7, 'max_tokens': 120})
agent = Agent(model, model_settings={'temperature': 0.4})
result = agent.run_sync('Summarize.', model_settings={'temperature': 0.1})
assert result.output == 'Configured'
22. Does setting temperature to zero make an agent deterministic?
No. Temperature is a sampling control, not an application-level guarantee that every run produces identical results. Pydantic AI’s settings reference describes sampling parameters and their provider support. Check the actual model’s supported settings before configuring them.
For an extraction service, lowering randomness may be a useful experiment, but exact equality should not be assumed across service changes or different request contexts. An evaluation should compare the fields that matter, such as extracted dates and categories, rather than demand identical punctuation in explanatory text. Keep unit tests deterministic with controlled responses, then use separate model evaluations for real behavior. Also avoid treating a seed as a universal promise: its availability and effect depend on the model service, not merely on the presence of a Python setting.
23. How does max_tokens differ from UsageLimits?
max_tokens controls a model response’s output allowance, while UsageLimits can bound accumulated work across a run, including model requests and tokens. See the settings and usage references.
A response allowance of 300 tokens does not stop an agent from making several requests. For a service with a limited budget, consider both the size of each response and the number of exchanges permitted. Token limits checked after a response can detect an overrun without preventing that response’s cost. Do not describe them as exact prepaid billing caps. Equally, setting the response allowance too low can truncate structured data and create avoidable validation failures. Measure realistic output sizes before tightening the limit.
24. Does a model request timeout limit the duration of the entire agent run?
No. A request timeout and a deadline for the whole run have different scopes. Pydantic AI’s timeout guide recommends wrapping an asynchronous run with a mechanism such as asyncio.timeout when a total wall-clock deadline is needed.
A run can contain multiple model requests, so several individually acceptable waits may exceed a web endpoint’s response deadline. A backend engineer should decide what the caller receives when that total deadline expires and ensure cancellation is handled properly. The example applies a five-second boundary around an offline run; its fast completion demonstrates placement rather than testing slow-network behavior. In a live integration, also consider the provider client’s retry policy because retries can consume additional time within the application’s overall deadline.
import asyncio
from pydantic_ai import Agent
from pydantic_ai.models.test import TestModel
async def main():
agent = Agent(TestModel(custom_output_text='Complete'))
async with asyncio.timeout(5):
result = await agent.run('Check the request.')
assert result.output == 'Complete'
asyncio.run(main())
25. What should you inspect in an AgentRunResult besides its output?
Inspect usage and message history when you need to understand how the result was produced. The current AgentRunResult reference exposes result.usage, all_messages(), and new_messages(), alongside the typed output.
For a classification endpoint, an apparently simple label might have required multiple requests after validation problems. Recording the request count and suitable timing information helps identify that regression. Use messages selectively for diagnosis, with controls appropriate to their contents. Do not assume a missing cost estimate means the run was free: pricing and usage availability can vary. Also distinguish measured provider usage from the synthetic usage of a test model. A performance report based only on local stub responses cannot establish production token consumption or latency.
26. Does an Agent automatically remember previous conversations?
In ordinary agent runs, conversation continuity requires supplying the relevant message history. Reusing an agent object alone does not append every past exchange to the next request. The message history guide shows how to pass earlier messages to a later run.
After a user asks about an order, a follow-up such as ‘When will it arrive?’ needs the correct earlier context. Store and retrieve messages under the intended conversation and user boundaries. A shared global history list could mix unrelated conversations even though the agent configuration is valid. Preserve structured message records when needed rather than flattening everything into a transcript string. For a chat application developer, the key design responsibility is choosing which conversation state belongs in each request and when old context should be excluded.
27. When does FallbackModel switch to another model?
FallbackModel switches according to its configured fallback conditions. The current API reference documents ModelAPIError as the default trigger and allows custom conditions. A successful but mediocre answer is not automatically recognized as a reason to switch.
For an availability-sensitive summarizer, a second provider may help when the primary service cannot answer. Both candidates still need to support the output contract and satisfy your data-handling requirements. A platform engineer should test the actual trigger, record which model served the result, and consider how much additional latency a failed first attempt creates. Treat quality-based routing as an explicit policy with measurable criteria. Otherwise, a fallback list can give the appearance of resilience while leaving the failure you care about untouched.
28. How should provider credentials and custom endpoints be configured?
Configure authentication and endpoint details through the provider integration, using environment variables or an explicitly configured provider as appropriate. The OpenAI integration guide illustrates both default credentials and custom client configuration.
A deployment might inject credentials from a secret store while application configuration selects an approved endpoint. Keep those values out of prompts, example source code, and ordinary diagnostic output. If requests fail, inspect the selected provider, endpoint, and credential availability without printing the credential itself. A compatible endpoint still needs to implement the API features your application uses; a familiar request format does not guarantee every output mode works. For a platform engineer, make endpoint changes reviewable because they determine where application data is sent.
29. What should you verify before changing the language model behind an existing agent?
Verify the features your application actually relies on, then evaluate the replacement on representative inputs. Pydantic AI’s model profiles describe capabilities such as native schema output, but supported integration does not imply equivalent task performance.
For a document classifier, compare label accuracy, handling of missing information, schema failures, response time, and cost across the same examples. Include ambiguous documents and long inputs, not only easy demonstrations. Check whether your chosen settings and output mode are accepted by the new model’s endpoint. An ML engineer should define acceptance thresholds before switching traffic. A cheaper model that repeatedly needs correction may not improve the complete workflow, while a more capable model may add unnecessary latency to a simple task.
30. Why can constructing an Agent fail before the first model request?
A named model is normally resolved during agent construction, which can check required environment variables before a run begins. The Agent constructor documentation offers defer_model_check=True to postpone that resolution until the first run.
This matters when a module defines a production agent but a test imports the module without production credentials. Deferred resolution can make an override practical, or the test can use an agent configured directly with a local test model. Deferral does not repair a missing credential or prove that the service is reachable. Choose the initialization behavior deliberately: early configuration failures are useful at application startup, while unnecessary production initialization makes isolated tests harder to run. Model construction and successful remote inference are separate checks.
Go deeper with our agent configuration guide.
Structured outputs and validation
Design output contracts that applications can consume, distinguish different kinds of validation, and handle incomplete or unacceptable model responses deliberately.
31. How should you choose an output_type for a new agent?
Choose the smallest type that expresses what the caller needs: text for prose, a scalar for one value, a list for a collection, or a named record for related fields. Pydantic AI accepts these and other supported Python types through output_type.
A topic generator may need only a list of strings. An appointment extractor benefits from a named object containing a date, location, and missing-information status. A Pydantic model is useful when you need field constraints or custom validation; a simpler supported type may be enough for a small result. Avoid returning a large generic dictionary just to accommodate hypothetical future requirements. A data engineer should design the contract around actual consumers, including how they will distinguish an empty collection from an unsuccessful extraction.
32. How do ToolOutput, NativeOutput, and PromptedOutput compare?
They differ in how the requested structure reaches the model and how the response is delivered. Pydantic AI exposes these choices through its output marker classes.
| Mode | Mechanism | Selection check |
|---|---|---|
| ToolOutput | Output arrives as tool-call arguments | Model supports suitable tool calling |
| NativeOutput | Provider receives a structured response schema | Endpoint supports native schema output |
| PromptedOutput | Instructions describe the schema for text output | Model can follow the format reliably |
Unmarked structured types ordinarily use tool output. For a form-filling application, select a supported mode and evaluate invalid responses on realistic inputs. Native schema support does not establish whether the entered facts are correct, and prompted JSON still needs validation. The output mechanism is distinct from deciding whether the resulting record should be accepted by the business workflow.
33. What does a minimal typed extraction agent look like?
Define a Pydantic model, pass it as output_type, and read the validated instance from result.output. The offline example supplies a controlled response using TestModel.custom_output_args.
Here, the application expects a course name and a positive seat count. Accessing result.output.seats makes the contract visible to the consuming Python code. The test response is deliberately supplied by the developer: TestModel does not actually extract the seat count from the prompt. A real extraction check must compare model output with known source text. This example verifies the boundary that remains useful in either case: the result has the expected type and its declared field constraint is applied before the application uses it.
from pydantic import BaseModel, Field
from pydantic_ai import Agent
from pydantic_ai.models.test import TestModel
class Enrollment(BaseModel):
course: str
seats: int = Field(gt=0)
model = TestModel(custom_output_args={'course': 'Python', 'seats': 3})
agent = Agent(model, output_type=Enrollment)
result = agent.run_sync('Three seats for the Python course.')
assert isinstance(result.output, Enrollment)
assert result.output.seats == 3
34. How do field descriptions differ from validation constraints?
A description explains meaning; a constraint restricts acceptable values. Pydantic’s field documentation and JSON Schema guide show how these are represented. Writing ‘positive number’ in a description alone does not create the same runtime check as Field(gt=0).
For a parcel extractor, describe whether weight is in grams or kilograms and enforce that the amount is positive. The numeric constraint can reject a negative weight, but it cannot detect every unit misunderstanding: 500 kilograms is still positive. A data engineer should use explicit units and test examples that expose such confusion. Inspect the generated schema to confirm that the information you expect is present. Clear descriptions guide generation, while executable checks enforce only the rules you have actually encoded.
35. Why use nested models for outputs such as invoices or purchase orders?
Nested models express repeated structured components without flattening away their relationships. Pydantic supports nested data models, allowing an invoice to contain a list of line items with their own fields and checks.
Each line item might require a description, quantity, and unit price. Separate arrays of descriptions and quantities are harder to reason about because their positions can become misaligned. With nested records, each quantity stays attached to its item. A validation error can also point to a particular item and field, which helps a data engineer diagnose extraction failures. Keep the structure proportionate to the source: deeply nested optional objects can make a simple document unnecessarily difficult to extract. Internal consistency checks, such as reconciling the total, still require explicit rules.
36. When is strict validation preferable to accepting type coercion?
Strict validation is useful when converting an input could hide a contract violation. Pydantic’s default validation can accept conversions such as the string '7' to an integer; strict mode narrows accepted inputs. Its exact behavior depends on the type and whether validation receives Python data or JSON.
For a normalized machine-to-machine record, a string quantity may indicate that an upstream contract has changed. For text extraction, accepting a harmless numeric conversion may be desirable. Choose based on the boundary, not a blanket preference for stricter settings. The example demonstrates Python-input behavior only. Also distinguish Pydantic’s validation strictness from provider-level strict schema generation: similar terminology does not mean those settings control the same stage of processing.
from pydantic import BaseModel, ValidationError
class Count(BaseModel):
value: int
assert Count.model_validate({'value': '7'}).value == 7
try:
Count.model_validate({'value': '7'}, strict=True)
except ValidationError as exc:
assert exc.errors()[0]['type'] == 'int_type'
else:
raise AssertionError('Strict validation should reject a string integer.')
37. How should an output schema handle unexpected fields?
Choose an explicit extra-field policy. Pydantic’s configuration reference documents ignoring, allowing, or forbidding extras; ConfigDict(extra='forbid') rejects undeclared keys. By default, extra data is ignored.
If a model produces discount_override in a quote extraction result, silently dropping it might conceal a recurring misunderstanding. Forbidding extras makes that mismatch visible. In a different ingestion workflow, accepting unknown metadata may be intentional. Decide per contract and test the policy rather than relying on the model to omit unexpected keys. For nested output, apply the appropriate policy to the nested model types too. A backend engineer must still enforce who can approve discounts or other actions; an output schema’s field policy is not an authorization system.
38. When should you add a field_validator to an output model?
Add a field validator when one field needs a rule that ordinary type annotations and built-in constraints do not express clearly. Pydantic’s validator guide explains before and after validation; an after validator receives the value after standard field validation.
For a packaging request, a positive integer may still be invalid if the factory accepts only batches divisible by twelve. A field validator can enforce that rule, return the accepted value, and raise ValueError for an unsuitable batch size. Keep reusable data rules here so they also apply when records enter outside the agent. Do not assume custom Python validation code becomes an equivalent JSON Schema rule visible to the model. Describe relevant requirements as well and test both accepted and rejected values.
39. How can you validate a relationship between two output fields?
Use a model validator when the rule depends on the complete record. An after model validator runs after field validation and should return the validated instance.
For a booking request, two individually valid dates can still form an invalid interval if the end precedes the start. A field-level check alone does not naturally express that relationship. The example validates the interval locally and demonstrates rejection, without involving an LLM. A booking-service developer must also distinguish internal consistency from availability: a correctly ordered interval does not prove that a room is free. Keep that distinction visible in the workflow so a validated request is not accidentally treated as a confirmed booking.
from datetime import date
from typing import Self
from pydantic import BaseModel, ValidationError, model_validator
class BookingWindow(BaseModel):
start: date
end: date
@model_validator(mode='after')
def check_order(self) -> Self:
if self.end < self.start:
raise ValueError('end must be on or after start')
return self
valid = BookingWindow(start='2026-10-10', end='2026-10-12')
assert valid.end >= valid.start
try:
BookingWindow(start='2026-10-12', end='2026-10-10')
except ValidationError:
pass
else:
raise AssertionError('A reversed interval must be rejected.')
40. When is an agent output validator more appropriate than a Pydantic validator?
Use an agent output validator for checks tied to the run, especially when validation needs asynchronous work. Pydantic AI’s output_validator API accepts synchronous or asynchronous functions. Raising ModelRetry asks for a corrected response.
For example, checking whether an extracted identifier exists in a live catalog belongs beyond simple field-shape validation. Keep such checks free of irreversible actions because correction can involve another attempt. The offline example uses a small text rule to demonstrate the retry mechanism: the first response is rejected, and a controlled second response succeeds. Its two-request assertion verifies that correction actually occurred. It does not imply that a real model will always repair the problem or that an unavailable catalog can be fixed by rewriting output.
from pydantic_ai import Agent, ModelRetry
from pydantic_ai.messages import ModelResponse, TextPart
from pydantic_ai.models.function import FunctionModel
responses = iter(['OK', 'Received your application.'])
def respond(messages, info):
return ModelResponse(parts=[TextPart(next(responses))])
agent = Agent(FunctionModel(respond), retries={'output': 1})
@agent.output_validator
def require_detail(output: str) -> str:
if len(output) < 10:
raise ModelRetry('Provide at least ten characters.')
return output
result = agent.run_sync('Acknowledge the application.')
assert result.output == 'Received your application.'
assert result.usage.requests == 2
41. How should output validation retries be bounded and handled?
Set an explicit correction budget and handle unsuccessful completion as a failure path. In the current retry enforcement documentation, text-output validation uses a shared run budget, while structured tool output uses a per-output-tool limit. ToolOutput.max_retries can override that output tool’s limit.
A service extracting mandatory delivery details should not keep asking indefinitely when the source contains no address. Provide a missing-information outcome where appropriate and cap correction attempts. Repeated identical validation errors may indicate an impossible schema or unclear instruction, not a need for a larger budget. Treat transport retries separately: asking a model to correct data is different from resending a failed HTTP request. A request-count limit can provide an additional bound on the overall interaction.
42. How can an agent return either a successful result or an explicit inability to answer?
Define distinct output types, such as ExtractedAddress and MissingAddress, and allow either through the agent’s output specification. The caller can then handle a successful extraction and a stated inability as separate cases.
This is useful when a shipping email simply omits the destination. Forcing every response into a complete address encourages an unsuitable success-shaped answer. A missing-information type can carry a reason and the specific fields needed next. Avoid adding plain str casually if every outcome must remain structured, because allowing strings also permits free-text output. A backend developer should test both branches and refuse to treat the inability branch as a completed record. The model’s choice of branch still needs evaluation against the original input.
43. How does a discriminated union make mixed structured records easier to validate?
A discriminated union uses a designated field to select the matching variant. Pydantic’s union guide shows how literal tags and Field(discriminator=...) identify that choice.
For a notification record, an email needs a subject while an SMS needs message text. A tagged union avoids a single object filled with optional fields whose combinations are difficult to interpret. The example places the union inside an output envelope, making the variant explicit for downstream consumers. A missing or unknown tag is then a validation problem rather than an invitation to guess the record type. An integration engineer should also test the resulting schema with the selected model provider, since local validation support does not by itself prove remote schema compatibility.
from typing import Literal
from pydantic import BaseModel, Field
class EmailNotice(BaseModel):
kind: Literal['email']
subject: str
class SmsNotice(BaseModel):
kind: Literal['sms']
text: str
class NoticeEnvelope(BaseModel):
notice: EmailNotice | SmsNotice = Field(discriminator='kind')
value = NoticeEnvelope.model_validate(
{'notice': {'kind': 'sms', 'text': 'Your order is ready.'}}
)
assert isinstance(value.notice, SmsNotice)
assert value.notice.text == 'Your order is ready.'
44. Why should streamed structured output be treated differently from the final result?
Streamed structured values can be partial, so receiving an intermediate value is not the same as accepting the complete result. The streamed result API documents partial validation in stream_output() and final validation through get_output().
A form preview may show fields as they arrive, but a partially received quantity or date should not trigger a submission. Keep previews distinguishable from committed state and handle failure after partial content has already appeared. Output validators can also run during streaming; RunContext.partial_output distinguishes partial processing from final processing. For a frontend and backend team, agree on the completion signal before persisting the record. A stream that looked plausible halfway through can still finish with an invalid result or be interrupted.
45. How should a validated output be serialized for an API response or stored record?
Use Pydantic’s serialization methods deliberately: model_dump() produces Python data, model_dump(mode='json') produces JSON-compatible data, and model_dump_json() produces JSON text. The serialization guide explains these boundaries.
An API framework expecting a mapping should receive the appropriate mapping, not a JSON string that it may encode again. Dates and other Python-specific values make this distinction visible. The example checks a JSON-compatible date and a serialization round trip. A data engineer should separately decide whether to include defaults, nulls, or selected fields, because those choices affect downstream interpretation. Serialization does not add factual verification or permission to publish a record; it converts an already accepted object into the representation required by its next consumer.
from datetime import date
from pydantic import BaseModel
class Shipment(BaseModel):
reference: str
ship_date: date
shipment = Shipment(reference='S-104', ship_date=date(2026, 10, 7))
payload = shipment.model_dump(mode='json')
assert payload['ship_date'] == '2026-10-07'
encoded = shipment.model_dump_json()
assert Shipment.model_validate_json(encoded) == shipment
Go deeper with our structured-output guide.
Put this into practice with guided training
Explore the Pydantic AI course for the syllabus, guided projects and training options.
Tools and dependency injection
Tools connect model requests to application functions. Dependency injection supplies the services and trusted context those functions need, while validation, execution rules, and approval checks control how calls proceed.
46. When should you use @agent.tool instead of @agent.tool_plain?
Use @agent.tool when a function needs the current run’s context, and @agent.tool_plain when its arguments are enough. With context-aware tools, the first parameter is RunContext; the model supplies the remaining arguments, not that context object.
| Decorator | Good example | Data source |
|---|---|---|
| @agent.tool | Look up my enrolled courses | Authenticated learner in dependencies |
| @agent.tool_plain | Convert minutes to hours | Explicit function arguments |
A course lookup should receive a course identifier from the model but obtain the learner identity from trusted application code. Do not add context to a simple conversion just for consistency; keep its interface small and easy to test.
47. How do type hints and docstrings affect a tool’s schema?
Pydantic AI derives a function tool’s argument schema from its signature, while its docstring explains the operation and parameters to the model. In the tool-schema documentation, supported docstring styles supply parameter descriptions; require_parameter_descriptions=True can catch missing descriptions during registration.
For a delivery quote, describe whether weight_kg includes packaging and whether destination means a country code or street address. A float annotation alone cannot communicate those meanings. Put measurable limits into types or field constraints and keep business rules in executable checks. Inspect the generated schema during review: an unclear parameter can produce valid arguments that request the wrong operation, even when every value passes type validation.
48. What happens when the model supplies invalid tool arguments?
For ordinary function tools, Pydantic AI validates the arguments before calling the function. Missing fields or incompatible values generate validation feedback for the model, subject to the tool’s retry limit. The tool-execution documentation describes this correction loop.
Suppose a seat-reservation tool requires a positive integer quantity and receives a negative number. With the corresponding constraint declared, validation rejects the call before the reservation code runs. Test this by checking that the booking service was never invoked. Do not confuse validation with strict rejection of every different Python type: Pydantic may coerce values under its normal rules. Also set a finite retry allowance so repeated malformed requests eventually fail visibly.
49. How do ModelRetry, ToolFailed, and ordinary exceptions differ?
ModelRetry asks the model to correct its approach. ToolFailed reports a failed tool result without spending the tool’s correction budget. An ordinary unhandled exception propagates out of the run. These are distinct outcomes in the exception API.
For example, an ambiguous customer name can justify ModelRetry with guidance to use an identifier. A temporarily unavailable catalogue can produce ToolFailed when the assistant should explain the limitation or choose another source. A programming bug should usually reach application error handling. Network retry policies are a separate layer; asking the model to rephrase an argument will not repair an expired credential or a broken connection.
50. What is the difference between deps_type and deps?
deps_type declares the expected dependency type when constructing an agent; deps supplies the actual object for a particular run. The dependency documentation explains that deps_type supports type checking rather than runtime validation.
A training assistant might declare a dataclass containing a student identifier and a course repository. Each request passes an instance for the authenticated student. Declaring that dataclass does not build the repository, fetch credentials, or validate an incoming request body. Construct and check the instance in application code before calling the agent. This distinction matters when moving from a demonstration with a single string dependency to an application with several services and user-specific values.
51. Are dependencies automatically visible to the language model?
No. Dependencies are application objects accessed through ctx.deps; they are not automatically added to the model’s argument schema or prompt. However, code can expose their contents through instructions, tool responses, or other model-visible messages. The dependency examples show how tools and instruction functions read them.
Keep a billing client’s token in the client or dependency object, then return only the invoice fields needed to answer the question. Returning the entire dependency object for debugging defeats that separation. Review error messages as well: a failed HTTP request should not reveal authorization headers. Dependency injection makes the data flow explicit, but your return values and logging choices still determine where sensitive information travels.
52. When is ToolReturn more useful than returning a string?
Use ToolReturn when a tool needs to distinguish its result, additional model-visible content, and application metadata. The advanced-return documentation identifies return_value as the tool result, content as additional user-message content, and metadata as information not sent to the model.
A document lookup could return selected passages as its result and store retrieval timing or internal trace identifiers in metadata. A plain string is sufficient when there is only one short answer. Do not place required citation identifiers exclusively in metadata if the model must cite them, because the model will not see them there. Likewise, metadata is not a secret store: application code can still inspect and record it.
53. Should Pydantic AI tools be synchronous or asynchronous?
Match the tool to the libraries it uses. An asynchronous database or HTTP client belongs in an async def tool; a blocking library can be called from a synchronous def tool. Pydantic AI runs synchronous callbacks in worker threads, as described in its dependency execution guidance.
For example, awaiting an async stock lookup lets other work proceed while the service responds. Merely changing the function header to async def while keeping a blocking HTTP call can stall the event loop. Using run_sync for the agent does not change this rule. Check the dependency’s own concurrency restrictions too, especially when a database connection or a legacy client is shared by several tools.
54. How should tools handle parallel execution and shared state?
Assume several tool calls may overlap. Current Pydantic AI supports sequential=True for a tool that must run alone within the tool-call batch; it acts as a barrier between earlier and later calls. The parallel-execution guidance explains that behavior.
Independent product lookups can benefit from concurrency. Updating and then reading a shared shopping basket requires more care: use a transaction or a service operation that expresses the required sequence. Marking one tool sequential does not lock the same basket against other users, processes, or agent runs. Test the business operation under concurrent requests, and use database constraints or version checks for shared records. Tool scheduling is only one part of coordinating state changes.
55. How can you expose a tool only when it is relevant?
A tool’s prepare function can inspect RunContext and return its definition, a modified definition, or None to omit it for that step. The Tool API documents this preparation hook.
For example, expose a shipping-estimate tool only after the application knows the destination country. This keeps unavailable actions out of the model’s choices. When changing a definition, preserve its existing fields; copying it with dataclasses.replace avoids accidentally discarding approval or scheduling settings. Availability is not authorization: the service must still check whether the caller can perform the action. Test both the visible and hidden cases, including a later step where the required information becomes available.
56. Why group functions into a FunctionToolset?
A FunctionToolset packages related functions so multiple agents can reuse the same tool collection. Toolsets also support composition, filtering, and name prefixes, making them useful when an assistant grows beyond a few local functions.
A learning platform could maintain one catalogue toolset and one attendance toolset. A public course adviser receives catalogue access, while an authenticated support assistant receives both. This avoids copying decorators and descriptions across agents. Keep each collection focused on a coherent service, with a clear dependency contract. A single collection containing every administrative action makes accidental exposure harder to spot. Review the final tool list for each agent, since composition determines what the model can discover and request.
57. How can you replace dependencies while testing a tool?
Use agent.override(deps=...) to provide controlled dependencies for a block of calls. Pair it with TestModel to exercise tool wiring without a provider request, following the testing guidance. This complete example checks that the override wins even when the run supplies another value.
import json
from dataclasses import dataclass
from pydantic_ai import Agent, RunContext
from pydantic_ai.models.test import TestModel
@dataclass
class Deps:
seats: int
agent = Agent(TestModel(), deps_type=Deps)
@agent.tool
def available_seats(ctx: RunContext[Deps]) -> int:
return ctx.deps.seats
with agent.override(deps=Deps(seats=7)):
result = agent.run_sync("Check seats", deps=Deps(seats=0))
assert json.loads(result.output)["available_seats"] == 7
print("Dependency override passed")The assertion proves which dependency reached the tool. It does not prove that a real model will choose that tool appropriately. For a repository dependency, substitute a fake with known records and check requested identifiers, including a missing-record case.
58. Who should create and close clients passed through dependencies?
The application should own the lifetime of resources it passes as dependencies. Supplying an HTTP client or database connection through deps does not instruct the agent to create or close that resource. The dependency examples place client use inside an explicit context manager.
Create a reusable connection pool at the appropriate application boundary, then provide each run with the services and identity it needs. A short script can keep creation, the agent call, and cleanup in one context. Avoid returning an agent result stream or deferred operation that still needs a client after that context has closed. Also avoid sharing a mutable per-user dependency instance: changing its student identifier during another run can make otherwise correct tools read the wrong record.
59. Where should authorization and business rules live in a tool?
Put them in trusted application or service code before performing the operation. A valid tool schema establishes an acceptable argument shape; it does not establish who owns a record or whether an action is permitted. RunContext provides access to dependencies that can carry the authenticated identity.
Consider changing an exam booking. Let the model propose a booking identifier and date, then have the service verify ownership, the rescheduling window, and seat availability before committing. Never accept a model-supplied student identifier as proof of identity. A useful failure test supplies another student’s valid booking identifier and confirms that no update occurs. Keep this service check effective even when the operation is called outside the agent.
60. How do deferred tools support approval before an action?
A tool marked requires_approval=True waits for an approval decision before its body runs. For conditional approval, a tool can raise ApprovalRequired. The deferred-tools documentation describes both inline handlers and a separate follow-up run using DeferredToolRequests and DeferredToolResults.
For example, present the exact recipient and content of a certificate email before approving its send operation. Match the decision to the pending tool-call identifier, and preserve the original history when resuming through the separate-run flow. Approval should cover the actual arguments, not a vague earlier request. Recheck time-sensitive conditions when execution resumes: a course cancellation approved yesterday may no longer be valid after attendance has been recorded.
Go deeper with our tools and dependencies guide.
RAG, retrieval and citations
Retrieval-augmented generation answers questions with evidence from selected sources. Good results depend on finding the right passages, preserving their origin, respecting access rules, and checking how the answer uses them.
61. What does Pydantic AI contribute to a RAG application?
Pydantic AI coordinates the agent, retrieval tools, dependencies, and answer handling. Your application still chooses how to load documents, split them, store searchable representations, and retrieve useful passages. The official RAG example connects an agent to a PostgreSQL database with pgvector through a search tool.
For a course-policy assistant, ingestion could prepare searchable policy sections before any learner asks a question. During a run, the retrieval tool searches those sections and returns evidence for the answer. This separation makes it possible to replace the search backend without rewriting the conversation logic. A tool-based design lets the model request further evidence; a fixed retrieval step is easier to control when every answer must start from a predetermined search.
62. How should you split documents and preserve source information for RAG?
Split documents into passages that retain enough context to answer a question, and keep stable source information alongside every passage. The Pydantic AI RAG example stores documentation sections with titles, content, and URLs; the exact chunking strategy remains an application decision.
For a fee policy, keep the refund condition with its exceptions. Separating the sentence describing eligibility from the sentence giving the deadline can produce a misleading answer. Add a document identifier, section location, and revision date to your records so a citation can point to the correct version. Compare several chunk sizes on real questions. Larger passages preserve context but spend more input tokens; smaller passages need careful boundaries and sometimes neighboring text.
63. Why must document and query embeddings be compatible?
Vector comparison is meaningful only when document and query embeddings belong to a compatible representation space. Keep the embedding model and dimensional settings consistent, and use the appropriate document and query modes. Pydantic AI’s Embedder API provides embed_documents() and embed_query() because some providers treat those inputs differently.
If an index contains vectors from one model and queries suddenly use another, matching dimensions alone does not make the results reliable. Plan an index rebuild when changing incompatible embeddings. Record the model and settings with the index, and run known-question checks after an update. A deterministic test embedder can check integration wiring, but its generated vectors do not establish real semantic retrieval quality.
64. When should a retrieval tool use keyword, vector, or hybrid search?
Choose the search method according to the questions and documents, then measure it. Pydantic AI can call your retrieval function regardless of the search backend. Its RAG example demonstrates vector retrieval, while pgvector documents hybrid search with PostgreSQL full-text search.
| Method | Useful case | Check |
|---|---|---|
| Keyword | Named error or course code | Exact identifiers survive indexing |
| Vector | A policy question phrased differently | Meaning matches the intended topic |
| Hybrid | Questions mixing names and descriptions | Combined ranking improves relevance |
For example, test both a specific course code and a paraphrase of its prerequisites. Do not select a more complex hybrid design unless it improves those results enough to justify the extra tuning.
65. How do you decide how many retrieved passages to send to the model?
Set a retrieval and context budget based on measured answer quality, not a universal passage count. The official RAG example limits its database results, but its chosen limit is an example configuration rather than a recommended value for every application.
A practical experiment is to retrieve a larger candidate set, remove duplicates, rank the remaining passages, and send a smaller set that fits the context budget. pgvector’s guidance includes reranking patterns. For a prerequisites question, several near-identical introductions may be less useful than one prerequisite section and one exception. Track missing evidence and irrelevant context separately. More passages increase cost and can bury the useful evidence; too few can omit the condition that changes the answer.
66. How can you reject citations to documents that were not retrieved?
Require source identifiers in the output and check them against the evidence available to that run. An output validator can reject unknown identifiers with ModelRetry. This offline example checks both acceptance and rejection using a controlled model response.
from pydantic import BaseModel
from pydantic_ai import Agent, ModelRetry, RunContext
from pydantic_ai.exceptions import UnexpectedModelBehavior
from pydantic_ai.models.test import TestModel
class Answer(BaseModel):
text: str
source_ids: list[str]
def response(source_id: str) -> TestModel:
return TestModel(custom_output_args={
"text": "The course includes a project.",
"source_ids": [source_id],
})
agent = Agent(response("course-guide"), deps_type=set[str],
output_type=Answer, retries=0)
@agent.output_validator
def check_sources(ctx: RunContext[set[str]], answer: Answer) -> Answer:
if not answer.source_ids or not set(answer.source_ids) <= ctx.deps:
raise ModelRetry("Cite identifiers from the supplied evidence.")
return answer
result = agent.run_sync("Check citations", deps={"course-guide"})
assert result.output.source_ids == ["course-guide"]
try:
agent.run_sync("Check citations", deps={"course-guide"},
model=response("unknown-guide"))
except UnexpectedModelBehavior:
print("Unknown citation rejected")
else:
raise AssertionError("Unknown citation was accepted")The allowed identifiers should come from the actual retrieval results, and application code can map them to stored URLs. This check prevents invented source identifiers, but it does not prove that a cited passage supports the sentence. Verify claim support separately, and represent an unsupported answer explicitly instead of demanding a citation when no evidence exists.
67. What should a RAG agent do when it finds no useful evidence?
Return an explicit insufficient-evidence outcome instead of turning a weak match into a confident answer. Define an application output with a status, answer text, and evidence identifiers; Pydantic AI’s structured outputs and validators can enforce the relationships you specify between those fields.
For example, an assistant asked about next year’s exam schedule should say that the available documents do not establish the dates if it only retrieves last year’s calendar. It may ask for a specific batch or an updated document. Distinguish an empty search from a failed search service: both limit the answer, but they need different explanations and follow-up actions. A valid output structure still requires a well-designed evidence decision.
68. How should a RAG agent handle instructions found inside retrieved documents?
Treat retrieved text as evidence to examine, not as authority to change the agent’s task or permissions. A course document containing a request to reveal credentials is still untrusted content. Limit the tools available to the answering agent and keep consequential checks in application code.
Pydantic’s optional PromptInjectionDefender capability can inspect supported local tool results, but its documented default is report-only; blocking requires configuration. It also has scope limits, so it should not be described as complete protection. Include a deliberately hostile passage in retrieval tests and check whether the answer stays on task and whether any unexpected tool is called. Formatting a passage as JSON or validating its fields does not remove embedded instructions.
69. How do you keep one tenant’s documents out of another tenant’s answers?
Apply access restrictions during retrieval, using an identity established by the application. Never retrieve every tenant’s passages and ask the model to hide the unauthorized ones. Pydantic AI’s RunContext can supply trusted dependencies to the retrieval function; the database or search service must enforce the access rules.
For example, include the permitted organization in a parameterized search and in any cache key. Where appropriate, PostgreSQL row-security policies can provide another enforcement layer, provided the querying role is subject to those policies. Test with identical document titles in two organizations and inspect the returned passages, citations, and cache hits. Even a correct final answer can conceal an earlier unauthorized retrieval.
70. How would you evaluate retrieval quality separately from answer quality?
Evaluate whether retrieval found the needed evidence before judging how the model used it. Then assess whether the answer addresses the question and whether each factual claim is supported. Pydantic Evals documents RAG metric rubrics for context precision, context recall, faithfulness, and answer relevance.
Build cases with a question, reference evidence, and an expected answer. A fee question that retrieves the right policy but invents a discount has an answer problem; a careful answer based on the wrong policy version has a retrieval problem. The documented rubrics evaluate supplied context, so capture your actual retrieved passages if you want to assess a complete run. Combine identifier checks and human-reviewed cases with model judgments rather than treating one judge score as proof.
Go deeper with our RAG tutorial.
MCP and integrations
MCP and tool adapters connect agents to existing services. The key decisions concern interface choice, session ownership, identity, error handling, and which parts of the integration a test actually exercises.
71. What problem does MCP solve for a Pydantic AI agent?
MCP provides a common interface for connecting an AI application to tools and services. Pydantic AI can consume tools from an MCP server, and an agent can also be used inside a server. The MCP overview describes these two directions.
For example, a catalogue service could expose the same search tool to a course assistant and another compatible client. A direct Python function is simpler when both the agent and service already live in one application. MCP becomes useful when the integration must cross application or language boundaries, or serve multiple clients. It standardizes the connection, but you still need meaningful tool descriptions, compatible implementations, and a clear contract for returned data.
72. How do stdio, Streamable HTTP, and legacy SSE connections differ?
Use stdio for a local subprocess and Streamable HTTP for a separately running server. Legacy HTTP+SSE remains relevant for older integrations. The MCP transport specification explains the process and communication boundaries; Pydantic AI’s client supports these connection types.
| Connection | Server location | Practical check |
|---|---|---|
| stdio | Local subprocess | Executable and arguments are correct |
| Streamable HTTP | Independent service | MCP endpoint is reachable |
| Legacy SSE | Older HTTP service | Client matches the legacy endpoint |
Streamable HTTP can itself use SSE responses, so observing an event stream does not prove the server uses the old transport. For stdio, keep ordinary diagnostic output off standard output because that channel carries protocol messages.
73. When should you use the MCP capability instead of MCPToolset?
The MCP capability is the current high-level entry point. Use MCPToolset directly when you need detailed client configuration, explicit connection management, or sharing across agents. The capability defaults to client-side MCP execution and can prefer provider-native execution with native=True, where supported.
For a simple remote knowledge service, the capability keeps configuration compact. For a connection requiring a custom transport or client, the lower-level toolset offers more control. Here, client-side execution does not mean a remote server moves onto your machine; it means your application handles the MCP calls. Choose deliberately because provider-native execution changes which party contacts the service and which local hooks can observe the operation.
74. How should an application manage an MCP connection’s lifetime?
Use async with agent to keep registered MCP connections open across the enclosed runs, or manage an individual toolset with async with toolset. Without an explicit context, Pydantic AI opens and closes toolsets as needed. The client lifecycle guidance explains this behavior.
A script asking three related questions can reuse one session inside a context instead of repeatedly reconnecting. For stdio, that lifetime also controls the launched subprocess. For HTTP, the remote server must already be running; closing the client does not shut down the service. Keep all calls that depend on the session inside its lifetime, and test cleanup after an exception as well as after a successful answer.
75. How do you handle tool-name collisions across MCP servers?
Assign clear prefixes and expose only the tools needed for the task. Pydantic AI’s toolset transformations include .prefixed() for naming and .filtered() for availability, and they can wrap MCP toolsets.
If two servers both offer search, prefixes can distinguish courses_search from tickets_search. Include descriptions that explain what each search covers; a unique name alone may still be unclear. An admissions assistant probably needs course search but not ticket deletion. Check the final names after composing wrappers, because a filter applied after prefixing sees the changed names. Explicitly inspect the exposed list when a server adds tools, rather than assuming the original list will remain fixed.
76. How should MCP authentication work when every user has different credentials?
Create a separate authenticated MCP toolset for each concurrent user run. The per-user authentication guidance warns that a shared MCPToolset has a shared session identity; changing task-local credentials does not isolate overlapping runs on that instance.
A dynamic toolset factory using @agent.toolset(per_run_step=False) can read the user’s token from dependencies and construct one toolset for that run. Authentication options are documented in the MCP API. For example, two instructors querying private attendance records should establish independent authenticated sessions. Test both requests concurrently, since a sequential test can miss identity reuse. Keep tokens out of tool arguments and verify permissions again at the MCP service.
77. How should you handle an error returned by an MCP tool?
Choose tool_error_behavior according to what should happen next. In the MCPToolset API, 'retry' asks the model to correct the call, 'failed' presents a failed tool result, and 'error' propagates the exception to the application.
An invalid catalogue filter may be correctable; a service refusing an operation may be better reported as a failure. A broken integration can require application handling. Do not assume every connection problem is a completed tool failure: protocol and transport errors have different paths from server-reported tool results. Test each category with controlled failures, and check the message given to the model. It should explain the actionable problem without exposing service credentials or internal diagnostics.
78. What is the difference between MCP sampling and elicitation?
Sampling lets an MCP server request model generation through the client; elicitation lets it request additional structured input. The sampling and elicitation documentation describes separate configuration for these interactions.
A report server might request sampling to summarize its findings, while a scheduling server might request a missing date through elicitation. They require different decisions: generation has model usage and data-sharing implications, whereas elicitation needs a way to collect or decline the requested input. Check compatibility before designing around either feature. Current Pydantic AI documentation notes that FastMCP 4’s modern protocol mode does not support server-initiated sampling or elicitation; its legacy mode retains that behavior. A working ordinary tool call does not establish support for these additional exchanges.
79. Can Pydantic AI reuse LangChain tools without adopting a LangChain agent?
Yes. Pydantic AI provides tool_from_langchain() for an individual tool and LangChainToolset for a collection. Its third-party tool documentation explains how to attach those tools while retaining a Pydantic AI agent.
This is useful when a team already maintains a specialist search connector as a LangChain tool. The adapter avoids rewriting that connector, but it does not remove its dependencies or configuration requirements. There is also an important validation difference: Pydantic AI does not validate the adapted tool’s arguments itself; the LangChain tool is responsible for rejecting invalid inputs. Test that behavior directly. Compare an in-process adapter with MCP when the same connector must also serve clients outside the Python application.
80. How can you test an MCP integration without a remote server or paid model?
Use an in-process FastMCP server with MCPToolset and a controlled test model. The MCPToolset API accepts a FastMCP instance, and TestModel can exercise tools without a model-provider call. This complete example requires the MCP dependencies and checks discovery, execution, and the returned value.
import asyncio
import json
from fastmcp import FastMCP
from pydantic_ai import Agent
from pydantic_ai.mcp import MCPToolset
from pydantic_ai.models.test import TestModel
server = FastMCP("Interview test")
@server.tool()
def service_status() -> dict[str, str]:
return {"status": "ready"}
async def main():
toolset = MCPToolset(server)
agent = Agent(TestModel(call_tools=["service_status"]),
toolsets=[toolset])
async with agent:
result = await agent.run("Check service status")
assert json.loads(result.output)["service_status"]["status"] == "ready"
print("In-process MCP check passed")
asyncio.run(main())This test isolates the Python integration. It does not cover HTTP authentication, network interruptions, subprocess startup, or whether a real model selects the right tool. Add focused connection tests for the transport you actually use, while retaining this fast check for changes to tool registration and result handling.
Go deeper with our MCP integration guide.
Put this into practice with guided training
Explore the Pydantic AI course for the syllabus, guided projects and training options.
FastAPI, async and streaming
Serving an agent introduces request lifetimes, concurrent users, partial responses and failures after a response has started. These questions focus on how to keep a Pydantic AI service responsive and predictable.
81. How should an async FastAPI endpoint call a Pydantic AI agent?
Call await agent.run(...) inside the async endpoint and return the result’s output. The endpoint already runs on an event loop; a synchronous runner is not the appropriate entry point there. Pydantic AI’s FastAPI example follows the same async integration pattern.
This complete offline example checks the HTTP response using a local test model. The request and response types describe the HTTP contract, while the agent remains a reusable application object. A real deployment would add authentication, request limits and an external model configuration. Passing this test confirms endpoint wiring and serialization; it does not measure answer quality.
from fastapi import FastAPI
from fastapi.testclient import TestClient
from pydantic import BaseModel
from pydantic_ai import Agent, models
from pydantic_ai.models.test import TestModel
models.ALLOW_MODEL_REQUESTS = False
agent = Agent(TestModel(custom_output_text="Ticket received"))
app = FastAPI()
class TicketRequest(BaseModel):
message: str
class TicketReply(BaseModel):
reply: str
@app.post("/tickets", response_model=TicketReply)
async def ticket(body: TicketRequest) -> TicketReply:
result = await agent.run(body.message)
return TicketReply(reply=result.output)
with TestClient(app) as client:
response = client.post("/tickets", json={"message": "Login failed"})
assert response.status_code == 200
assert response.json() == {"reply": "Ticket received"}82. What should be shared across FastAPI requests, and what belongs in per-request agent dependencies?
Share the agent configuration and reusable connection pools; pass user identity, tenant context and request-specific resources through a fresh dependency object. FastAPI’s lifespan mechanism can manage startup and shutdown of shared clients. Pydantic AI accepts application resources through its dependency system.
For example, two support requests can use the same HTTP client but must carry different authenticated customer IDs. A global variable such as current_customer can be overwritten while the first request is awaiting a model response. Database pools can be shared when their library supports it; a mutable transaction or session should follow its own concurrency rules. Test overlapping requests with different identities to catch accidental state sharing.
83. An async agent endpoint freezes other requests during a database lookup. What would you investigate?
Look for blocking work running directly inside an async def function. Declaring a tool async does not make a synchronous database driver or HTTP library nonblocking. Pydantic AI runs async tools on the event loop and offloads registered synchronous tools to threads, as explained in advanced tool execution.
A practical repair is to use an async database driver, register genuinely blocking I/O as a synchronous tool, or offload the specific blocking call. Check connection thread-safety before moving it. Heavy Python computation may need a separate process or worker. Measure event-loop delay and tool duration under concurrent traffic: a fast model cannot compensate for one blocking lookup that stalls every request sharing the loop.
84. How do you choose between run_stream, run_stream_events and a UI adapter?
Choose according to the information the client needs. The agent documentation distinguishes output streaming from observing the full execution. UI adapters additionally translate requests and events into an established frontend protocol.
| Interface | Useful when | Application responsibility |
|---|---|---|
| run_stream | Displaying final output progressively | Render output snapshots or text deltas |
| run_stream_events | Showing tool activity and all run events | Interpret events and assemble displayed content |
| UI adapter | Using a supported chat frontend protocol | Authenticate requests and manage application state |
A travel assistant displaying search progress needs more than final-answer tokens. The event interface continues through the run, while run_stream can treat suitable early text as the final output. Test a response that mixes text and tool calls before choosing the interface.
85. Why does a streaming chat interface display repeated words even though the model response is correct?
The frontend may be appending cumulative snapshots as though they were new text fragments. With stream_text(delta=False), each value represents the text accumulated so far. Replace the displayed message with that value. With delta=True, append the new fragment instead. The streaming output documentation describes these modes.
For example, appending both Hello and Hello there produces a duplicate greeting. This offline example verifies both rendering strategies against a controlled stream. Also assign each response a stable message ID: rendering every snapshot as a separate chat message causes a similar symptom even when the text assembly itself is correct.
import asyncio
from pydantic_ai import Agent, models
from pydantic_ai.models.function import FunctionModel
models.ALLOW_MODEL_REQUESTS = False
async def emit_text(messages, info):
yield "Hello"
yield " there"
agent = Agent(FunctionModel(stream_function=emit_text))
async def main():
snapshots = []
async with agent.run_stream("Greet me") as result:
async for text in result.stream_text(delta=False, debounce_by=None):
snapshots.append(text)
assert snapshots[-1] == "Hello there"
fragments = []
async with agent.run_stream("Greet me") as result:
async for text in result.stream_text(delta=True, debounce_by=None):
fragments.append(text)
assert "".join(fragments) == "Hello there"
asyncio.run(main())86. When should a Pydantic AI application deliver updates over SSE rather than WebSockets?
Choose server-sent events (SSE) when the browser mainly receives progress and answer updates; choose WebSockets when both sides need frequent messages over the same connection. SSE is one-way, while FastAPI WebSockets support sending and receiving. The transport is separate from Pydantic AI’s choice of output or event stream.
For a report generator, submit the request over HTTP and subscribe to progress through SSE; a cancel button can make a separate HTTP request. An interactive session exchanging frequent live commands may justify WebSockets. EventSource reconnection does not recreate missed application events: resumable delivery still needs event IDs and server-side replay. Test proxy buffering, disconnects and slow consumers, and match any UI adapter to its documented frontend protocol.
87. What should happen when a user closes the browser while an agent is streaming?
For a request-bound chat, stop consuming events and release the run’s resources through cancellation and context-manager cleanup. Current stream cancellation guidance explains that leaving the run_stream_events context cancels its background run task. Ensure the response generator owns that context for the entire stream.
Test an actual disconnect after several chunks and check that local tasks and connections finish. A provider may not guarantee that stopping the local iterator immediately stops remote generation or billing. If the work must survive disconnects, give it a durable job identity and a separate status endpoint. Do not record a disconnected preview as a successfully completed answer, or assume cancellation undoes a tool action already committed.
88. Why can an async Pydantic AI service still fail under a sudden traffic spike?
Async execution allows many requests to wait efficiently, but it does not create unlimited provider capacity, database connections or memory. Put explicit limits around scarce resources and bound the waiting queue. Pydantic AI provides concurrency limiters, including model wrappers and configurable running and queued operation limits.
If 500 chats arrive together, allowing all of them to reach the provider may produce throttling and retries that worsen congestion. Decide how much work can run, how long requests may wait and when to reject overload. A limiter object shared within one process does not automatically coordinate multiple server processes. Load-test the deployed worker count and inspect queue time separately from model response time.
89. How should a FastAPI service report an agent failure before versus after streaming starts?
Before response headers are sent, translate the failure into an appropriate HTTP response. Once streaming has started, report failure through the stream protocol and mark the response incomplete; a later exception cannot replace the status already sent. Use the documented Pydantic AI exception types to distinguish usage limits, provider failures and unexpected model behavior.
For instance, return a temporary-service error for an unavailable upstream service, but treat a malformed client request separately. Keep raw provider bodies and tool arguments out of public error messages. A client receiving half an answer should show an interrupted state and a request reference. Retry decisions must consider whether earlier tools changed external state.
90. How would you design an API for a long-running agent job with status and cancellation?
Persist a job record, enqueue the work and return 202 Accepted with a server-issued job ID and status URL. Attach a request correlation ID to submission logs and the job record. A worker executes the agent independently of the original HTTP connection; Pydantic AI’s durable execution integrations are relevant when work must survive restarts.
For a ten-minute research report, expose queued, running, succeeded, failed and cancelled states, plus a result reference when ready. A cancel endpoint records the request and signals the worker using the runtime’s cancellation mechanism. Report cancellation as requested until the worker confirms it; a completed action is not undone. Test duplicate submissions, worker restarts and cancellation racing with completion. Closing a status page should not silently delete the job.
Go deeper with our FastAPI tutorial.
Multi-agent workflows and graphs
Multi-agent systems need explicit ownership of decisions, state and failure handling. These questions examine when another agent helps, how graphs coordinate work and where the application must supply guarantees.
91. When would you choose ordinary Python orchestration, agent delegation or a graph?
Choose the structure that makes the workflow’s decisions easiest to inspect. Use ordinary Python for a known sequence, delegation when a parent needs specialist assistance, and a graph when branching and merging become central to the application. The official multi-agent patterns describe these options.
| Requirement | Suitable approach | Main tradeoff |
|---|---|---|
| Classify, then draft | Python orchestration | Simple, but growing branches need care |
| Consult a product specialist | Delegation | Extra model calls and coordination |
| Investigate, merge findings, escalate | Graph | Explicit paths with more setup |
Do not split one reliable prompt into five agents merely to increase agent count. First identify a measurable benefit, such as separate access rights, independent evidence gathering or a reusable specialist.
92. How much context and access should a parent agent give a specialist agent?
Give the specialist a focused task, the evidence it needs and the smallest useful dependency set. Pydantic AI’s delegation guidance allows the child’s dependencies to be a subset of the parent’s. Constructing that boundary is an application design decision.
A shipping specialist may need an order reference and a read-only tracking client, but not the customer’s complete support history or a refund client. Pass conversation history deliberately when it is necessary; naming another agent does not automatically transfer the right context. Return a defined result containing findings and evidence references. This makes failures easier to trace and prevents a child from silently depending on unrelated details in the parent’s conversation.
93. How do you keep delegated agent calls within one overall usage budget?
In an ordinary in-process delegation, pass usage=ctx.usage to the child so its consumption contributes to the shared counters. Also apply the intended usage_limits to child runs; shared accounting and limit enforcement are separate concerns. See delegation usage accounting and the usage reference.
For example, a parent allowed six model requests should not launch three specialists with unrestricted retry loops. Check the combined requests after a controlled test and leave capacity for the parent’s final response. Token limits depend on reported consumption and can be exceeded by an individual response. Durable integrations may copy execution context, so verify their accounting rules instead of assuming an in-memory usage object crosses process boundaries.
94. How does delegating a subtask differ from handing control to another agent?
Delegation returns a specialist’s result to the parent, which continues deciding what to do. A handoff moves responsibility for the next stage elsewhere. Pydantic AI supports programmatic handoff through application code; an output function can also transfer control without another parent-model turn.
For a support request, the parent might consult a billing specialist and then compose one combined answer. Alternatively, application code can route the conversation into a dedicated billing flow with its own state and permissions. Decide which component owns the user-facing response and subsequent messages. Otherwise, both agents may answer, or the original agent may overwrite a specialist’s decision. Test the expected sequence of calls as well as the final text.
95. What belongs in graph state, node input and graph dependencies?
Put evolving workflow facts in state, the current operation’s payload in node input, and external services in dependencies. The graph context and builder interfaces distinguish these roles, although your application chooses the actual types.
In a document review flow, state might track document ID, revision count and unresolved issues. A review step receives the current draft. Dependencies supply the document store and specialist agents. Avoid storing an open database connection alongside business facts that may later need serialization. Also avoid treating shared state as a scratch dictionary for parallel branches. Let branches return explicit results, then combine them in one place so that execution order cannot silently change the outcome.
96. Does using Pydantic Graph automatically make a workflow resumable after a restart?
No. Current Pydantic Graph does not automatically snapshot graph state. The builder documentation explicitly says that the old graph persistence module was removed in V2. A workflow requiring crash recovery needs an appropriate persistence design or a durable execution engine.
For example, keeping a document review’s state in a dataclass does not preserve it when a worker process dies. Saving chat messages is also different from resuming a partially completed workflow. Decide what must survive: conversation context, completed step results, pending approvals or the execution itself. Then test recovery by stopping the worker between steps and verifying which work resumes and which work is repeated.
97. How would you combine independent parallel graph results without relying on completion order?
Have each branch return an identified result and collect those results through a join. Apply an explicit ordering rule afterward when presentation order matters. Pydantic Graph provides joins and reducers for this aggregation, alongside parallel execution.
The offline example collects estimates from independent checks. Sorting by name makes the result stable regardless of task scheduling. In a real specialist workflow, return the source, finding and status rather than an unlabeled string. Decide whether one failed branch should fail the whole workflow or produce a partial report. Bound access to external services separately; graph parallelism does not increase a provider’s quota.
import asyncio
from pydantic_graph import GraphBuilder, StepContext, reduce_list_append
builder = GraphBuilder(output_type=list[tuple[str, int]])
@builder.step
async def checks(ctx: StepContext[None, None, None]) -> list[str]:
return ["delivery", "stock"]
@builder.step
async def estimate(ctx: StepContext[None, None, str]) -> tuple[str, int]:
days = {"delivery": 3, "stock": 1}
return ctx.inputs, days[ctx.inputs]
collect = builder.join(reduce_list_append, initial_factory=list)
builder.add(
builder.edge_from(builder.start_node).to(checks),
builder.edge_from(checks).map().to(estimate),
builder.edge_from(estimate).to(collect),
builder.edge_from(collect).to(builder.end_node),
)
graph = builder.build()
async def main():
results = await graph.run()
assert sorted(results) == [("delivery", 3), ("stock", 1)]
asyncio.run(main())98. A writer and reviewer agent keep sending work back to each other. How would you stop the loop?
Make termination an explicit application rule: limit review rounds, define acceptance criteria and provide an escalation outcome. A reviewer saying that a draft needs improvement should not be enough to create unlimited additional runs. Use graph state and transitions to track progress, with usage limits as an additional bound.
For example, allow two revisions and require the reviewer to identify concrete unsupported claims. If the same issue remains, return the draft with unresolved findings for human review. Compare issue IDs across rounds to detect stalled progress. Record why the workflow ended: acceptance, exhausted rounds, budget exhaustion and cancellation are different outcomes and should not all appear as successful completion.
99. A graph reaches the wrong specialist even though every step completes successfully. How would you debug it?
Inspect the actual routing value and selected transition before changing the specialist’s prompt. A successful step only means its execution completed; it does not prove the route was appropriate. The graph builder supports step-by-step execution and diagrams that help compare runtime behavior with intended paths.
Suppose an account-access ticket reaches billing. First check whether the classifier produced the wrong category, the application mapped the right category to the wrong branch, or an unknown value fell into an unsuitable default. These require different repairs. Reproduce the transition with a fixed classifier output, then evaluate classification separately. Add an explicit unsupported-category outcome so new labels cannot quietly enter a convenient but incorrect branch.
100. How would you pause a workflow for an approval that may arrive hours later?
Represent the pending action explicitly and persist the information required for a later run. In the stop-and-resume deferred-tool flow, the agent returns DeferredToolRequests. A later invocation receives the original message history plus DeferredToolResults associated with the pending tool-call IDs.
For a publishing workflow, store the draft revision, requested action, pending calls and decision status in application storage. Close the original HTTP request instead of holding a worker open for hours. When approval arrives, verify the pending record and resume with its saved context. The follow-up is a new run; use a conversation or workflow identifier for correlation. Handle denial and expired requests as normal outcomes, and test restoration after a server restart.
Go deeper with our framework comparison.
Testing, evaluation and debugging
Reliable agent applications need both predictable software checks and evidence about model behavior. These questions cover offline tests, evaluation design, useful traces and diagnosis of failures that ordinary assertions can miss.
101. What should unit tests prove, and what should an agent evaluation measure?
Unit tests should prove controlled application behavior; evaluations should measure how well the system performs representative tasks. Pydantic AI’s testing guide supports model replacements, while Pydantic Evals runs cases against evaluators.
| Check | Useful evidence | Cannot establish alone |
|---|---|---|
| Offline unit test | Retry and routing behavior | Real model quality |
| Provider integration test | Live request compatibility | Broad task reliability |
| Task evaluation | Accuracy across selected cases | Correctness on every future input |
For a ticket router, test the category-to-queue mapping deterministically and evaluate classification on realistic tickets. A perfect mapping test is compatible with poor classification. Conversely, a strong evaluation score can hide a broken error handler that its dataset never exercised.
102. Why might TestModel call a tool that the user prompt never requested?
TestModel is a procedural test double, not a language model interpreting the prompt. By default, it calls the agent’s tools with generated arguments. Its documented controls include call_tools, custom_output_text and custom_output_args.
If a test prompt says to only summarize, a registered deletion tool is not protected by that wording. Replace external dependencies with harmless fakes, or select the exact tools appropriate to the test. Use FunctionModel when the call sequence and arguments matter. A passing TestModel run demonstrates that selected plumbing works with generated values; it does not demonstrate that a production model will select the right tool or infer a customer’s actual intent.
103. How can you test recovery from an invalid model output without calling a provider?
Use FunctionModel to script an invalid first response and a valid response after the agent sends correction feedback. Assert both the accepted result and the presence of a retry message. This exercises the failure path under controlled conditions rather than hoping a live model produces a suitable mistake.
The complete offline example rejects a negative item count, then accepts two. It reads the generated output-tool name instead of assuming a fixed name. A separate exhaustion case should keep returning invalid data and assert failure when the retry allowance is spent. These tests verify application behavior around validation; they do not predict how frequently a provider will need correction.
from pydantic import BaseModel, Field
from pydantic_ai import (
Agent, ModelRequest, ModelResponse, RetryPromptPart, ToolCallPart, models,
)
from pydantic_ai.models.function import FunctionModel
models.ALLOW_MODEL_REQUESTS = False
class ItemCount(BaseModel):
count: int = Field(ge=0)
def respond(messages, info):
corrected = any(
isinstance(part, RetryPromptPart)
for message in messages
if isinstance(message, ModelRequest)
for part in message.parts
)
return ModelResponse(parts=[
ToolCallPart(
info.output_tools[0].name,
{"count": 2 if corrected else -1},
)
])
agent = Agent(FunctionModel(respond), output_type=ItemCount, retries=1)
result = agent.run_sync("Count the items")
assert result.output.count == 2
assert result.usage.requests == 2
assert any(
isinstance(part, RetryPromptPart)
for message in result.all_messages()
if isinstance(message, ModelRequest)
for part in message.parts
)104. Does setting ALLOW_MODEL_REQUESTS to False make an entire test suite offline?
No. It blocks normal non-test model requests through Pydantic AI’s supported checks, but it is not a general network firewall. The model request guard is useful for detecting forgotten model overrides. A tool, database client or custom callback may still perform independent I/O.
For example, replacing the model does not replace a shipping tool that calls a real tracking service. Set the guard in test configuration, provide fake dependencies, and use transport mocks or network restrictions where necessary. Keep explicitly authorized provider integration tests separate. Also remove production credentials from ordinary test jobs: a model replacement should be one layer of isolation, rather than the only reason a test cannot reach a live service.
105. How do you test application code that uses a globally configured agent without rewriting that code?
Use agent.override(...) as a scoped context manager to replace the model, dependencies or toolsets while executing the existing application path. The official testing examples use this approach with fixtures. It lets the test exercise real orchestration while controlling external behavior.
For a refund-preview endpoint, override the model with a scripted response and the account service with an in-memory fake, then call the endpoint normally. Assert the response and which account lookups occurred. Let the context manager or fixture clean up the override even if an assertion fails. Mutating the agent’s global configuration directly can leave later tests using the wrong model or dependency, producing order-dependent failures.
106. How would you build an evaluation dataset that exposes realistic support-agent failures?
Include representative successful tasks and distinct failure conditions, with expected behavior defined for each. In Pydantic Evals, a case can contain inputs, an expected output, metadata and case-specific evaluators. Use those fields to describe the task and assess the outcome without leaking the answer into the task input.
A support dataset should include ambiguous requests, missing order IDs, outdated policy evidence, unavailable tools and requests the user cannot authorize. Tag cases by category and risk so that a large group of easy questions cannot hide weak escalation behavior. Keep a held-out set for release decisions. When a production incident becomes a regression case, remove unnecessary personal information while preserving the feature that caused the failure.
107. When is an LLM judge useful, and how would you check whether its scores are trustworthy?
An LLM judge helps assess qualities that are difficult to express as exact comparisons, such as whether an explanation addresses a question using supplied evidence. Give it a specific rubric and the context needed to apply it. The LLMJudge guidance also documents nondeterminism, bias and evaluation cost.
For a policy assistant, check required fields and citation IDs with code, then judge whether cited passages support the claims. Calibrate the judge against examples labeled by people, including fluent but incorrect answers. Review its reasons and disagreements instead of accepting one average score. A low temperature can improve consistency, but it does not make a judge an infallible or fully deterministic source of truth.
108. How can an evaluation detect a forbidden tool call when the final answer looks harmless?
Evaluate the execution trace as well as the final output. Span-based evaluation can inspect tool activity and relationships between operations. This matters when an agent returns an innocent summary after performing an action it should never have attempted.
Suppose a read-only assistant calls an account-update tool and then says it only checked the account. The text alone misses the violation. Match the relevant tool span or inspect captured calls in a controlled test. Be careful with negative queries: finding one span that is not an update does not prove that no update occurred anywhere. Likewise, two separate presence checks do not establish execution order. Missing telemetry should produce an inconclusive or failed check, not automatic success.
109. A new prompt has a better average evaluation score but users report worse performance. What would you compare?
Compare results by task category, repeated runs, latency and resource use rather than relying on one overall score. Pydantic Evals supports repeated evaluations and custom metrics and attributes. Keep the dataset, evaluator configuration and model settings comparable between experiments.
The new prompt may improve easy summaries while doubling tool calls on complex tickets or weakening refusal behavior. Inspect failures and slow cases, not only successful averages. Control evaluation concurrency so provider throttling does not masquerade as a prompt regression. Report quality alongside time to first useful output, total completion time and cost per completed task, with enough repeated observations to expose instability.
110. How would you investigate an unexpected model-behavior error when no successful result object is returned?
Capture the messages around the failing run and correlate them with tool and model traces. The capture_run_messages context manager can preserve the exchange for inspection even when the run raises. Use the exception reference to distinguish exhausted correction attempts from unrelated application errors.
Look for the first divergence: an unexpected tool name, malformed arguments, repeated output rejection or a tool result the model misunderstood. Record model, prompt and schema versions so the failure is reproducible. Reduce the exchange to a redacted FunctionModel test before changing prompts. Increasing retries without understanding the repeated failure can simply turn a fast error into a slower, more expensive error.
Go deeper with our testing and evaluation guide.
Put this into practice with guided training
Explore the Pydantic AI course for the syllabus, guided projects and training options.
Security, deployment and interview scenarios
Production decisions require controls outside the model conversation. These scenarios examine untrusted content, permissions, irreversible actions, operating limits and release practices for Pydantic AI applications.
111. How should a frontend safely render HTML or Markdown generated by a Pydantic AI agent?
Treat generated HTML and Markdown as untrusted browser input. Display plain text through framework escaping or textContent on an ordinary element. If formatting is required, disable raw HTML in the Markdown renderer and sanitize the resulting HTML with a maintained sanitizer. Follow OWASP’s XSS prevention guidance, including URL-scheme allowlists for links and context-appropriate output handling.
A generated support answer could contain an image event handler or a javascript: link, even inside otherwise useful prose. Test those cases through the actual renderer, including streamed previews. Never insert unsanitized chunks into innerHTML, and do not let later transformations reintroduce unsafe markup after sanitization. Content Security Policy adds protection but does not replace safe rendering. Pydantic validation of a string field does not make its contents safe HTML.
112. How do you prevent SSRF when a Pydantic AI tool fetches a model-selected URL?
Enforce destination restrictions inside the fetching service, regardless of how the model obtained the URL. Server-side request forgery (SSRF) occurs when an attacker induces the server to contact unintended destinations. Prefer predefined source identifiers over arbitrary URLs. Where URLs are necessary, follow OWASP’s SSRF prevention guidance: restrict schemes and destinations, disable automatic redirects and validate resolved IPv4 and IPv6 addresses.
For a public-web research tool, block loopback, private, link-local and cloud metadata destinations. Bind connections to validated addresses while preserving hostname-based TLS checks; a separate unchecked DNS lookup can enable rebinding. Apply network egress restrictions as another layer. Test a public URL redirecting to an internal service and a hostname whose address changes. A valid URL schema or a model’s explanation does not establish that fetching it is safe.
113. A user approved a tool call yesterday, but the target record changed overnight. Is the approval still sufficient?
Approval should authorize a specific action on a specific version of the relevant data, and the application should recheck permissions and preconditions before execution. Pydantic AI’s approval workflow transports the decision; it does not decide whether an old approval still matches current business state.
Suppose a manager approved publishing revision seven, but revision eight now contains different claims. Store the approved revision, arguments, approver and expiry, then compare them at execution. A changed payload should require a fresh decision instead of silently reusing the earlier approval. Also reject replayed or already-consumed approvals. Test changed ownership, revoked permissions and edited arguments, since the person reviewing the request may have seen something different from what would now execute.
114. A refund tool times out, the request is retried, and two refunds appear. How would you prevent that incident?
Make the refund operation idempotent using an application-controlled operation ID that survives retries. Persist the operation’s status and use the payment service’s idempotency support where available. A timeout means the caller did not receive a timely result; it does not establish whether the payment was processed.
The Pydantic AI timeout guide also notes that a synchronous tool may keep running after its deadline. After an uncertain result, reconcile the existing operation before attempting another refund. Do not derive the idempotency key from newly generated model text or create a fresh key for every retry. A useful regression test simulates successful external execution followed by a lost response and verifies that the second attempt returns the original outcome.
115. How would you investigate agent incidents without collecting every customer’s conversation in telemetry?
Record operational facts by default and collect sensitive content only under a deliberate access and retention policy. Pydantic AI supports InstrumentationSettings(include_content=False) to omit message content from its instrumentation, as described in the observability documentation.
Useful incident data can include request references, model identifiers, tool names, durations, usage and failure categories. A support dashboard can reveal a slow account lookup without storing the customer’s full message or invoice. Review custom spans, application logs, exception handlers and external SDK logging separately; one instrumentation setting does not sanitize every logging path. Test with recognizable synthetic secrets and inspect the exported telemetry. Where content is necessary for diagnosis, redact it and restrict who can retrieve it.
116. How would you control one expensive run, a traffic burst and a customer’s daily spending budget?
Use separate controls because these problems have different scopes. Pydantic AI provides run usage limits and concurrency controls; account-wide quotas require shared application accounting.
| Risk | Control | Important boundary |
|---|---|---|
| Runaway agent loop | Request, tool and token limits | One run and explicitly shared usage |
| Traffic burst | Concurrency limit and bounded queue | Coordinate across deployed workers |
| Daily tenant overspend | Shared quota ledger and reservations | All requests in the billing period |
A token limit can be exceeded by the response that reveals the usage, so do not promise an exact financial ceiling from it. Reserve capacity before admitting costly work, then reconcile measured usage. Separate tenant identity from model-generated arguments and account for simultaneous requests when checking the remaining quota.
117. How should provider fallback behave when a model fails halfway through a streamed answer?
Do not assume fallback can transparently replace a response whose content has already reached the user. The FallbackModel reference distinguishes eligible failures while opening a stream from failures during streaming, which propagate. Define an explicit interrupted-response policy.
For example, if a summary stops halfway through, mark that attempt incomplete and offer or initiate a separately identified retry according to the product’s rules. Do not append a different model’s full answer to the partial text. Verify that fallback candidates support the required tools and output mode. Before restarting a whole agent run, inspect whether previous tools changed external state; model fallback does not undo those actions or make replay harmless.
118. What checks would you require before deploying a Pydantic AI dependency upgrade?
Pin and test a complete dependency set, read migration guidance, and compare the new build against the current production behavior. The version policy explains that minor releases may add message parts or stream events and change telemetry attributes, so event consumers and dashboards need attention as well as agent calls.
Run offline behavior tests, a small provider compatibility suite and representative evaluations. Exercise stored conversation loading, cancellation, approvals and the frontend stream protocol. Treat unexpected events deliberately instead of assuming an exhaustive list can never grow. Release to a limited share of traffic with observable failure thresholds and a rollback route. A dependency lock improves reproducibility, but it does not freeze external model behavior or provider responses.
119. An interviewer asks you to build an agent that answers database questions by generating SQL. What safeguards would you propose?
Separate query generation from execution, and constrain execution with database permissions and resource limits. The official SQL generation example checks generated SQL using PostgreSQL EXPLAIN; that is useful feedback for query correction, not a complete authorization system.
For a reporting assistant, use a read-only database role, approved views, tenant filtering, a statement timeout and a row limit. A string beginning with SELECT is not sufficient evidence that the query is acceptable or inexpensive. Decide how sensitive columns and expensive functions are restricted, and inspect queries using a suitable parser where needed. Test unauthorized tables, cross-tenant requests and queries that scan excessive data. Return a controlled failure when the question cannot be answered within those boundaries.
120. A support assistant returns a perfectly structured answer citing the wrong policy. How would you diagnose and fix it?
Trace the evidence from retrieval to the final claim and check whether the policy was relevant, current and available to that customer. A well-formed answer can still be factually wrong. The official RAG example illustrates retrieval feeding an agent; correctness still depends on the retrieved material and how the answer uses it.
For example, an assistant may quote a cancelled subscription policy for an active annual plan. Inspect document version, eligibility filters, retrieved passages and citation IDs before adjusting the prompt. Return uncertainty or escalate when evidence is missing or contradictory. Add cases for outdated and conflicting policies, and evaluate claim support with evidence-aware judging plus deterministic citation checks. Merely retrying schema validation would not address this incident.
Go deeper with our 45-project practice library.
Turn the answers into a practical interview demonstration
Pick one small application rather than claiming expertise in every topic. A support-ticket assistant is enough to demonstrate a typed output, a permitted lookup, an uncertain result and a failing test. Use fictional records and keep credentials out of the demonstration.
- Explain the contract: show the request fields and the outcomes the caller accepts.
- Run one successful case: identify which parts use fixtures and which services are real.
- Show a controlled failure: use an unknown citation, denied lookup or invalid output.
- Explain your evidence: state what the test proves and what still needs evaluation.
- Discuss a tradeoff: compare cost, latency, complexity and the consequences of a wrong action.
The complete code examples in this guide were checked offline with Python 3.12, Pydantic AI 2.54.0 and Pydantic 2.13.5. Check your installed versions when following newer documentation. No live-model quality scores are claimed. Explore the Pydantic AI project tutorials when you are ready for a larger example.
Download the Pydantic AI practice pack
Get nine offline Python examples, a project-review checklist and an evaluation case sheet. The examples use simulated model responses and do not need an API key. Submit this form to open the ZIP download.
Brolly Academy will store your request and use your details to handle it. Course follow-up is optional; the checkbox is not selected automatically.
Read our Privacy Policy for information about handling your details.
Put this into practice with guided training
Explore the Pydantic AI course for the syllabus, guided projects and training options.
