The runnable retrieval example is a mock retrieval baseline, not production vector RAG. It filters a tiny in-memory list by owner and topic, then uses a fixed test response. There are no embeddings, vector indexes or live generation calls. That makes the flow inspectable while leaving search quality and real answer quality as separate learning tasks.
Basic Python dictionaries, functions and Pydantic models are enough to follow the exercises. Start with the Pydantic AI beginner tutorial if you have not used an agent before. By the end of this guide, you should be able to trace a claim back to a permitted passage and identify which layer failed when that trace breaks.
What does Pydantic AI do in a RAG application?
Pydantic AI can organise the agent, retrieval tools and typed answer contract. Your retrieval system still needs to prepare documents, find relevant passages and enforce permissions. The framework does not automatically make a poor document collection useful.
The official RAG example demonstrates a search-backed agent. Its setup includes external services and embedding work, so review the requirements and potential costs before running it. The smaller exercise here focuses on the decisions that make an answer trustworthy.
Think of two workflows. Document preparation turns approved source material into searchable passages. Answering starts with a user’s question and selects passages for that particular request. An application can have excellent preparation and still answer badly, or a capable model and still retrieve the wrong text. Keep those workflows visible in your project instead of hiding everything behind one chat function.
| Component | Concrete job | Useful evidence of success |
|---|---|---|
| Document preparation | Extract text and retain headings, dates and ownership | A passage still makes sense outside its original page |
| Retriever | Find relevant passages within permitted scope | Known questions return their reviewed supporting passages |
| Agent | Produce a bounded answer using supplied evidence | Claims match the selected source material |
| Application validator | Check states, citations and business rules | Invalid or unsupported results take a defined failure path |
| Reader interface | Show the answer and useful source references | A reader can locate the relevant version and passage |
Choose the right starting point for your project
Use this tutorial when you want a Python assistant to answer from documents you control. It suits a developer building a policy helper, a learner preparing a knowledge-base project, or a tester checking whether answers have evidence. It is not a replacement for a search engine when people simply need a list of matching documents. Sometimes a well-labelled search result is more useful than a generated paragraph.
The first two exercises below isolate citation checks and retrieval. The third connects an actual search tool to an agent using a scripted model. All three run offline. They let you inspect your application before connecting a paid provider. A later section explains how the same responsibilities extend to embeddings and PostgreSQL. No live-model accuracy or vector-search benchmark is claimed for these offline exercises.
| Approach | Good starting use | What you still need to check |
|---|---|---|
| Document search | The reader wants to find the original policy | Relevance, permissions and source freshness |
| Retrieve once, then answer | A narrow question has a small supporting evidence set | Evidence coverage and supported claims |
| Agent chooses a search tool | A question may require a follow-up lookup | Tool choices, search limits and whether another search helps |
| Long context without a search index | A small approved document set fits the model’s usable context | Input cost, attention to exceptions and access scope |
| Fine-tuning | The project needs a measured change in model behaviour | Training data quality and evaluation; this is not a document-update mechanism |
Do not build all five approaches at once. Pick one user question, one approved source collection and one measurable outcome. For example, the lab helper should show the applicable weekday closing time with its source, and admit when a shipping policy is absent. That is specific enough to test. A requirement such as “answer anything accurately” is not a usable acceptance test.
Prepare a reproducible Python environment
The examples were checked with Python 3.12 and pydantic-ai-slim==2.54.0. Create a separate virtual environment for the exercises so they do not change another project’s installed packages. Install the pinned slim package with your environment’s Python executable. The slim package is sufficient for the offline model fixtures used here; a real model provider may need an additional integration package.
python -m venv .venv
# Activate the environment using the command for your operating system.
python -m pip install "pydantic-ai-slim==2.54.0"
python -m pip show pydantic-ai-slim pydanticKeep each complete Python example in a separate file when practising. Run it with the same Python executable that installed the package. If an import fails, first confirm which environment your terminal or notebook is using. Repeatedly installing packages into a different interpreter will not repair the interpreter that runs your code.
There is no API key to enter for these tests. The agent examples explicitly disable external model requests. This is intentional: passing a plumbing test must not quietly send documents to a provider. The official unit-testing guide describes the distinction between test models and real model calls. Retain the offline checks after adding a provider so you can diagnose application regressions independently of network behaviour.
Keep a short environment record with the Python version, installed dependency versions, document snapshot and command used to run each test. A screenshot of a successful answer is not enough to reproduce an issue later. Do not publish a credentials file or real learner documents alongside that record. Synthetic examples are sufficient for a public learning portfolio.
Step 1: Define the questions and source boundaries
Start with questions the application should answer. For the fictional lab assistant, these might concern opening hours, equipment booking and support contacts. List questions it should not answer, such as personal account details or policies missing from its source collection.
Use a small set of documents you are authorised to process. Give each one an identifier, title and version. A file called final-policy-new-final.pdf is not a reliable versioning scheme. Record which source is current and how older versions should be handled.
Do not upload real learner records as convenient test data. Synthetic policies and invented accounts are enough to learn the retrieval workflow.
Write down an expected evidence source for each answerable question before implementing retrieval. For example, “When does the weekday lab close?” should point to a weekday-hours policy. “Can I enter after closing with an instructor?” needs an exception or access policy. They share vocabulary, but the first document may not answer the second question.
Separate source identity from passage identity. A policy can contain several passages, and a revised policy can contain passages with similar wording. A citation should let you recover the exact material used for an answer, even after the current document changes. Keeping a version identifier also makes failures reproducible when someone reports an old answer.
| Field | Fictional example | Why it matters |
|---|---|---|
| Source and passage IDs | LAB-POLICY, weekday-hours-01 | Locate the precise supporting text |
| Title and heading | Lab access, Weekday hours | Preserve meaning when a passage is retrieved alone |
| Version and effective date | Revision 3, effective 1 October | Resolve changes without guessing from filename order |
| Access scope | Public practice policy or a named tenant | Filter before returning text to the agent |
| Source location | Approved document page or internal record | Build a reference the reader is allowed to open |
The example’s short dictionaries omit most of this metadata for readability. Treat that omission as a deliberate limit. When you extend the collection, introduce fields because they support an actual retrieval, access or citation decision, rather than collecting arbitrary metadata that nobody checks.
Step 2: Keep useful context when splitting documents
A passage needs enough surrounding information to make sense. Splitting “Access ends at 6 pm” away from the heading “Weekend labs” could lead an assistant to apply the rule to every day.
Preserve headings, source identifiers and relevant scope. Tables need their column meanings. Exceptions need to stay connected to the rule they modify. Choose a chunking strategy based on the document structure, then inspect the retrieved passages with real questions.
There is no universal chunk size that guarantees better answers. A useful experiment compares two reasonable strategies on the same question set and records which evidence each one retrieves.
Try a fictional source with this meaning: weekday access ends at 6 pm; weekend access ends at 2 pm; instructor-led events can have separately published hours. A chunk containing only “ends at 2 pm” loses the weekend qualification. A huge chunk containing every building policy can bury the same qualification among unrelated rules. Both failures can occur before a model sees anything.
How can you test chunking without calling a model?
Print the passage returned for each practice question and read it without the original document. Ask whether the passage identifies the location, day and relevant exception. Then check whether a reviewer could answer correctly from that passage alone. This simple inspection often reveals missing table headers, broken lists and extraction mistakes more quickly than changing an agent prompt.
Overlap can keep a sentence near a boundary, but too much overlap produces nearly identical results. Deduplicate repeated passages before building the final context so the same rule does not occupy most of the evidence budget. Preserve an exception’s relationship to its rule; repeated fragments are not a substitute for coherent source structure.
Step 3: Retrieve only permitted evidence
Retrieval can use keywords, embeddings or a combination. The method matters less than whether the retrieved passages answer the question and belong to the user’s allowed scope.
Apply permission filtering before sending content to the model. Filtering citations after generation is too late if protected text has already entered the model context. A multi-tenant application must keep one customer’s documents from another customer’s answer path.
For learning, start with a simple retrieval baseline you can inspect. Improve it only after you understand its failures. A complicated vector setup is not useful if you cannot explain why a passage was returned.
| Method | Useful behaviour to test | Characteristic limitation |
|---|---|---|
| Exact topic matching in this lab | A question containing hours selects the hours passage | Closing time does not contain the required topic string |
| Keyword or lexical search | Exact policy names, product IDs and unusual terms | Different wording can hide relevant material |
| Embedding-based search | Paraphrases retrieve semantically related passages | A related passage can still have the wrong scope or date |
| Hybrid retrieval | Combine exact-term and semantic candidates | Ranking and duplicate handling need evaluation |
| Reranking selected candidates | Put the most useful evidence earlier | Cannot recover a passage missing from the candidate set |
Start with a small, explicit evidence budget. Returning more passages can improve coverage, but it also adds cost and competing information. Test a few candidate counts against the same reviewed questions. Measure whether the required passage is present and whether irrelevant or outdated passages make the answer harder to produce.
If you later use embeddings, record which embedding model and settings produced the stored vectors. Document and query vectors must use compatible representations. A migration needs a tested reindexing path, an index version and a rollback decision. Do not assume an index remains usable merely because a replacement model happens to produce vectors with the same length.
The official example uses PostgreSQL with pgvector and generates embeddings during its build stage. That is a different exercise from the in-memory example below. Follow its documented setup in an isolated environment when you are ready to evaluate vector retrieval; there is no need to run its embedding build to complete this article.
Put this into practice with guided training
Explore the Pydantic AI course for the syllabus, guided projects and training options.
Step 4: Define an answer with evidence identifiers
A typed answer can contain an answer state, answer text and source IDs. An unanswered state is important: it gives the application a legitimate route when evidence is missing.
The following complete offline exercise checks the citation contract. It does not retrieve documents or call a language model. The source text is fictional.
from typing import Literal
from pydantic import BaseModel
class GroundedAnswer(BaseModel):
state: Literal["answered", "not_found"]
text: str
source_ids: list[str]
retrieved = {"POLICY-01": "The fictional practice lab closes at 6 pm."}
def check_citations(answer: GroundedAnswer, evidence: dict[str, str]) -> bool:
if answer.state == "not_found":
return not answer.source_ids
return bool(answer.source_ids) and all(
source_id in evidence for source_id in answer.source_ids
)
good = GroundedAnswer(state="answered", text="The lab closes at 6 pm.",
source_ids=["POLICY-01"])
bad = GroundedAnswer(state="answered", text="The lab closes at 6 pm.",
source_ids=["INVENTED-99"])
assert check_citations(good, retrieved)
assert not check_citations(bad, retrieved)
print("Known citation accepted; invented citation rejected.")
This check only proves that cited IDs appear in the retrieved set. It does not prove that the text supports the answer. If the answer said “The lab closes at 10 pm” with POLICY-01 as its citation, the identifier check would still pass. A separate evidence-support check must catch that mistake.
Run the example as a script and expect Known citation accepted; invented citation rejected. Then inspect the not_found branch carefully: it checks only that source IDs are absent. It does not inspect the text. An object claiming not_found while confidently inventing a closing time would still pass that particular check. This is why a citation helper is only one part of the answer policy.
| Check | Example that should fail | What the sample establishes |
|---|---|---|
| Shape and allowed state | An unsupported state label | The Pydantic model restricts the state values |
| Citation membership | An answered result citing INVENTED-99 | The helper rejects IDs outside retrieved evidence |
| Claim support | A 10 pm claim citing a 6 pm passage | Not checked by the membership helper |
| State consistency | not_found with a confident unsupported answer | Text meaning is not checked by the helper |
For a larger answer, consider which source supports each material claim. One source at the bottom of a long paragraph may support the opening sentence but not the later explanation. Keep the answer as short as the task permits, and attach evidence at a useful level of detail. The structured output guide explains how a typed contract and application checks work together.
Connect retrieval to a typed agent: an offline exercise
This second, complete example connects a small keyword retriever to an agent and validates the resulting source identifiers. It needs pydantic-ai-slim==2.54.0. Retrieval really runs, but generation is a fixed TestModel response. There is no vector database, embedding service or paid model call. Use this to inspect the data flow, not to measure semantic search or model accuracy.
from typing import Literal
from pydantic import BaseModel
from pydantic_ai import Agent, models
from pydantic_ai.models.test import TestModel
models.ALLOW_MODEL_REQUESTS = False
documents = [
{"id": "LAB-1", "owner": "learner-a", "topic": "hours",
"text": "The fictional lab closes at 6 pm."},
{"id": "LAB-2", "owner": "learner-b", "topic": "hours",
"text": "A private lab closes at 9 pm."},
]
class Answer(BaseModel):
state: Literal["answered", "not_found"]
text: str
source_ids: list[str]
def retrieve(user: str, question: str) -> dict[str, str]:
return {d["id"]: d["text"] for d in documents
if d["owner"] == user and d["topic"] in question.lower()}
evidence = retrieve("learner-a", "What are the lab hours?")
assert set(evidence) == {"LAB-1"}
agent = Agent(TestModel(custom_output_args={
"state": "answered", "text": "The lab closes at 6 pm.",
"source_ids": ["LAB-1"]}), output_type=Answer,
instructions="Use only supplied evidence; say not_found when absent.")
answer = agent.run_sync(
f"Question: What are the lab hours? Evidence: {evidence}"
).output
assert answer.source_ids and set(answer.source_ids) <= set(evidence)
assert answer.text == "The lab closes at 6 pm."
assert retrieve("learner-a", "What is the shipping fee?") == {}
print("Allowed retrieval, typed response and missing-topic checks passed.")
The exact sentence assertion is appropriate only because this fixture deliberately supplies that sentence. In a live version, compare claims with evidence instead of requiring one wording. Add an explicit application branch for empty retrieval, then evaluate a real model separately using an approved question set. The naive topic match here will miss synonyms; that is a visible baseline limitation to improve, not a hidden production feature.
The expected terminal message is Allowed retrieval, typed response and missing-topic checks passed. Read the order of operations: Python retrieves evidence before the agent runs. Retrieval is not registered as a model-selected tool in this example. Calling it agentic search would imply a decision loop that this code does not implement.
What do the three assertions actually demonstrate?
The first establishes that learner-a receives LAB-1 rather than learner-b’s LAB-2. The answer assertion establishes that the fixed result uses the expected text and allowed citation. The last establishes that a question about shipping returns an empty dictionary from this retriever. It does not show the agent generating a not_found response for that question, because the agent is never run on the empty retrieval case.
The owner string is hard-coded practice context, not authenticated identity. In a service, derive it from the authenticated request and enforce the same boundary in your data query. Do not let a model choose the owner value. The tools and dependencies guide develops that distinction, while the FastAPI integration article explains the request boundary.
For your next exercise, intercept an empty evidence set before generation and return a deliberate unsupported-answer state. Separately, simulate a retrieval exception and return a service failure. “Nothing found” and “search did not run” need different outcomes; otherwise an outage can quietly look like a gap in your knowledge collection.
Work through three questions before changing the architecture
First ask, “What are the lab hours?” The fixture should retrieve LAB-1 for learner-a and produce the configured 6 pm reply. Keep the question, user scope, retrieved IDs and final answer together when reviewing this result. Looking only at the final sentence would hide whether the right passage was ever selected.
Next ask, “When does the lab shut?” The meaning is similar, but this simple retriever requires the literal topic hours. Its empty result is a retrieval limitation, not proof that the source collection lacks the answer. A synonym-aware baseline or a measured semantic search experiment could improve this case.
Finally ask, “What time does learner-b’s private lab close?” The correct access boundary still uses learner-a’s trusted identity. Depending on retrieval wording, the system may find nothing or find an irrelevant public passage; neither outcome authorises returning LAB-2. Add relevance checks and a refusal or clarification path without weakening the owner filter.
Turn retrieval into an agent tool with trusted context
This third complete example registers search as an agent tool. Unlike the earlier retrieve-once example, the agent run executes the tool call and receives its result. A FunctionModel scripts that decision for repeatable testing. It is not a live language model deciding how to search, and the retriever remains a tiny exact-topic baseline.
The important boundary is the tool signature. The model supplies a search query, but it does not supply the owner whose documents to read. That owner comes from RequestContext. This example hard-codes a fictional user at the call site; a real service must obtain the identity from authenticated application context. The function-tools documentation explains how tools receive the run context.
from dataclasses import dataclass, field
from typing import Literal
from pydantic import BaseModel, Field
from pydantic_ai import Agent, RunContext, models
from pydantic_ai.messages import ModelResponse, ToolCallPart, ToolReturnPart
from pydantic_ai.models.function import FunctionModel
from pydantic_ai.usage import UsageLimits
models.ALLOW_MODEL_REQUESTS = False
DOCUMENTS = (
{"id": "LAB-A-v1", "owner": "learner-a", "topic": "hours",
"text": "The fictional weekday lab closes at 6 pm."},
{"id": "LAB-B-v1", "owner": "learner-b", "topic": "hours",
"text": "The other private lab closes at 9 pm."},
)
class Reply(BaseModel):
state: Literal["answered", "not_found"]
text: str = Field(min_length=1)
source_ids: list[str]
@dataclass
class RequestContext:
user_id: str
retrieved: dict[str, str] = field(default_factory=dict)
def simulated_model(query: str):
# Script the model's tool decision so the application test is repeatable.
def respond(messages, info):
returns = [part for message in messages for part in message.parts
if isinstance(part, ToolReturnPart) and part.tool_name == "search"]
if not returns:
return ModelResponse([ToolCallPart("search", {"query": query})])
evidence = returns[-1].content
if evidence:
source_id = next(iter(evidence))
output = {"state": "answered", "text": evidence[source_id],
"source_ids": [source_id]}
else:
output = {"state": "not_found", "text": "No matching source was found.",
"source_ids": []}
return ModelResponse([ToolCallPart(info.output_tools[0].name, output)])
return FunctionModel(respond)
def build_agent(query: str):
agent = Agent(simulated_model(query), deps_type=RequestContext, output_type=Reply,
instructions="Search first. Use returned evidence only. Do not invent sources.")
@agent.tool
def search(ctx: RunContext[RequestContext], query: str) -> dict[str, str]:
"""Find permitted fictional lab policies for a query."""
evidence = {d["id"]: d["text"] for d in DOCUMENTS
if d["owner"] == ctx.deps.user_id and d["topic"] in query.lower()}
ctx.deps.retrieved.update(evidence)
return evidence
return agent
def verify_citations(reply: Reply, context: RequestContext):
if reply.state == "answered":
if not reply.source_ids or not set(reply.source_ids) <= context.retrieved.keys():
raise ValueError("Answer cites unavailable evidence")
elif reply.source_ids:
raise ValueError("Missing-answer response must not contain citations")
def ask(user_id: str, question: str):
context = RequestContext(user_id=user_id)
result = build_agent(question).run_sync(question, deps=context,
usage_limits=UsageLimits(request_limit=3))
verify_citations(result.output, context)
return result.output, context
if __name__ == "__main__":
answer, context = ask("learner-a", "What are the weekday hours?")
assert answer.source_ids == ["LAB-A-v1"]
assert set(context.retrieved) == {"LAB-A-v1"}
assert "9 pm" not in answer.text
missing, _ = ask("learner-a", "What is the shipping fee?")
assert missing.state == "not_found" and not missing.source_ids
synonym, _ = ask("learner-a", "When does the lab shut?")
assert synonym.state == "not_found"
try:
verify_citations(Reply(state="answered", text="Invented answer",
source_ids=["LAB-B-v1"]), context)
except ValueError:
pass
else:
raise AssertionError("Unavailable citation was accepted")
print("Tool retrieval passed: permitted source, missing topic, synonym limit and rejected citation.")
The success message is Tool retrieval passed: permitted source, missing topic, synonym limit and rejected citation. Read that claim narrowly. The tests show that the tool runs, receives the permitted owner, returns the expected source, handles an empty match, and rejects an unavailable citation. They do not establish whether a real model will choose a useful query, resist a malicious document or write a faithful summary.
The reply deliberately repeats the selected source sentence in this fixture. That makes the application test deterministic. A live model may paraphrase it, so evaluate whether the closing time, weekday scope and exceptions remain correct instead of requiring identical words. The final citation check establishes membership only; even this extended example does not prove semantic support for arbitrary generated claims.
Notice that retrieved evidence belongs to one request context, not a global dictionary. Otherwise one request could accidentally validate its citation against evidence loaded by a previous user. In a multi-turn conversation, decide explicitly which earlier passages remain permitted and current. Reusing conversation text does not remove the need to recheck access.
The request limit is a small safety bound for this exercise. It prevents an unbounded sequence of model turns, but it is not a full cost budget or a replacement for timeouts. A deployed service should also bound query length, returned context, concurrency and provider usage. Keep unexpected tool exceptions distinguishable from a successful search with no matching evidence.
Step 5: Handle missing and conflicting information
Ask a question whose answer is absent, such as the fictional lab’s international shipping fee. The correct behaviour is not to invent a plausible number. The assistant should state that its sources do not provide the answer and offer an appropriate next step.
Now provide two policy versions with different hours. The application needs a version rule or an explicit conflict response. The model should not silently choose whichever passage appears first.
Retrieved documents can also contain instructions directed at an assistant. Treat source text as evidence to analyse, not as permission to change tools, disclose secrets or ignore the user’s request. Test that separation in your authorised practice environment.
Distinguish at least three kinds of uncertainty. Missing information means no suitable source covers the question. Ambiguity means the question leaves out something necessary, such as weekday versus weekend. Conflict means relevant sources disagree under the same scope. A useful interface says which condition occurred and either asks a targeted clarification or directs the reader to the policy owner.
For conflicting versions, define authority before asking the model to summarise. An effective-date rule might select the current policy for today’s access question, while a historical question needs the policy effective on its requested date. “Use the latest document” is not sufficient for both tasks. If the collection lacks reliable dates, surface that limitation instead of inventing a resolution.
Test a passage that combines a legitimate closing-hours sentence with an unrelated instruction to reveal another tenant’s records. The permitted answer may use the hours statement as evidence, but no tool or permission should change. Keep retrieval credentials outside prompt text and enforce tool access in ordinary application code.
Move from the offline baseline to vector RAG
The offline topic matcher misses “When does the lab shut?” because the required topic word is absent. That is a concrete reason to test a better retriever. Embeddings represent text numerically so a search system can compare a query with stored passages. They can help with paraphrases, but similarity alone cannot determine whether a passage is current, permitted or sufficient to answer.
A practical upgrade has two separate jobs. First, an ingestion process extracts approved documents, preserves useful scope, creates passage IDs, computes document embeddings and stores the searchable records. Second, a request embeds the question, searches within the caller’s allowed collection, selects useful passages and passes them to the agent. Do not recompute every document embedding for each question.
- Freeze a reviewed document snapshot. Record source versions and access scopes before comparing retrieval strategies.
- Choose and record an embedding configuration. Query and document representations must be compatible, including any task-specific settings required by the provider.
- Build a separate practice index. Store text, source references and permissions beside the vector so ranking does not discard their meaning.
- Apply access restrictions within retrieval. Use trusted identity to restrict the candidate set before text reaches the agent.
- Measure the candidate set. Compare known supporting passages across keyword and vector retrieval on the same questions.
- Evaluate complete answers. Test citations, unsupported questions and source conflicts with the actual model configuration you plan to use.
The official Pydantic AI RAG example offers a PostgreSQL and pgvector implementation to study. Its embedding build calls an external API and may incur charges. We did not run that service-backed build for this article. Use its documented requirements in an isolated development environment, and review provider data handling before submitting your own documents.
The Pydantic AI embeddings guide is useful when selecting a supported embedding integration. Avoid combining vectors from unrelated embedding configurations in one comparison merely because their dimensions match. Keep an index version and a measured migration plan. Build a new practice index when experimenting instead of deleting a working production collection.
| Decision | What to inspect | Why it affects the application |
|---|---|---|
| Permissions | How metadata filters or access policies limit results | The right answer must also be authorised |
| Update workflow | Replace, delete and version behaviour | Corrected policies must stop serving old evidence |
| Search behaviour | Exact versus approximate retrieval and supported distance measures | Configuration can change which passages enter the candidate set |
| Operational fit | Backups, monitoring, support and team familiarity | A service must be maintainable after the demo works |
| Measured cost | Storage, ingestion, queries and optional reranking | A small demo’s bill does not establish production cost |
Use pgvector’s own documentation when evaluating its operators, index options and filtering behaviour. A database choice should follow your access, latency and maintenance requirements. Do not add several stores to a beginner project just to list more technology names. One retriever with an explained test report is a stronger learning result.
Step 6: Evaluate retrieval and answers separately
| Layer | Question to ask | Example failure |
|---|---|---|
| Retrieval | Did the right evidence reach the agent? | Weekend question retrieves weekday rules |
| Grounding | Does each important claim follow the evidence? | Correct passage, invented closing time |
| Citations | Can the reader locate the supporting passage? | Nonexistent or irrelevant source ID |
| Abstention | Does the assistant admit missing information? | Invented answer to an uncovered question |
| Permissions | Was only authorised content used? | Another user’s document appears in context |
Create a reviewed question set with answerable, unanswerable, ambiguous and conflicting cases. Keep a small holdout set you do not repeatedly tune against. Record the document snapshot and model configuration for each evaluation.
Report sample size and failure categories alongside any score. A single overall number can hide a serious access-control problem. The official testing guide explains how to separate deterministic checks from variable model behaviour.
For a simple retrieval measure, take ten reviewed answerable questions and ask whether the required passage appears among the returned candidates. If it appears for eight, report eight of ten for that exact collection and candidate count. Do not relabel that number answer accuracy: the generator can still misread the eight successful retrievals.
Then review the final answers for supported claims and useful abstentions. Include negative controls with similar wording but a different day, location or policy version. Keep permission failures as explicit failures regardless of the overall average. An improvement in helpful answers does not compensate for exposing one tenant’s private evidence to another.
Put this into practice with guided training
Explore the Pydantic AI course for the syllabus, guided projects and training options.
Build a small evaluation sheet before adding more documents
Start with a written case sheet that another learner can inspect. It should contain the question, permitted identity, expected evidence IDs, expected answer state and the reason the case matters. Keep answers short enough to review. For a multi-part question, record all required pieces of evidence instead of accepting any one vaguely related passage as success.
| Case | Question or change | Expected check |
|---|---|---|
| Direct answer | What are the weekday lab hours? | The current weekday passage is retrieved and the time is supported |
| Paraphrase | When does the lab shut? | Record the baseline miss; compare an improved retriever |
| Missing topic | What is the shipping fee? | No invented amount or fabricated source |
| Different user | Use the same question as another authenticated user | Only that user’s permitted evidence can enter context |
| Wrong scope | Ask about weekend access | Weekday evidence is not presented as a weekend rule |
| Conflicting sources | Provide two applicable policies that disagree | Apply an explicit authority rule or surface the conflict |
| Removed policy | Withdraw a previously indexed passage | The old passage is absent from search and relevant caches |
| Broken retrieval | Make the search dependency unavailable | Return a service failure, not a misleading no-answer result |
| Instruction in source | Insert an unrelated command into a practice passage | No extra access, changed tool permissions or secret disclosure |
| Invented citation | Return an ID outside the evidence set | Application validation rejects the result |
These are proposed evaluation cases, not a claim that all ten are automatically tested in the short scripts. The scripts cover only their listed assertions. Run source-conflict, stale-index, provider-error and prompt-injection cases against the extended system that actually implements those features. Mark unrun cases as unrun rather than treating a planned test as a passing result.
Compare one change at a time. For example, keep the model and question set fixed while changing the retriever. Then keep retrieval fixed while adjusting answer instructions. If you change the documents, embedding model, ranking and generator together, a better answer does not tell you which change helped. Keep the earlier configuration available so improvements can be checked rather than remembered.
Manage freshness, caching and retrieval costs
RAG freshness depends on document ingestion and retrieval, not just on when a model was released. Record when a document was accepted, indexed, superseded or removed. A policy correction should reach the searchable collection and invalidate affected cached answers. Test deletion with a previously answerable question so you know an old index or cache is not still supplying the removed passage.
A cache keyed only by question text can cross permission boundaries. The same words from different users may have different permitted evidence. Any reuse policy must account for access scope, collection version and relevant configuration. Avoid storing raw private questions indefinitely merely because caching is convenient.
Measure ingestion and serving costs separately. Ingestion can include extraction, embedding and storage; answering can include query embedding, search, reranking, generation and retries. A new document snapshot may change both quality and cost. Record the number of passages supplied to the model and their total size so you can explain expensive outliers.
Use the offline collection to validate data flow before paying for larger experiments. Once live evaluation is authorised, choose a bounded question set and budget, retain the model and corpus versions, and stop on repeated infrastructure failures. A bigger model will not restore a source that never reached the prompt.
Troubleshoot the evidence path
| Observed problem | Inspect first | Useful corrective experiment |
|---|---|---|
| No result for a familiar question | Query terms, filters and indexed text | Compare the original wording with a known matching query |
| Wrong day’s policy | Headings and chunk scope | Preserve the day label beside the closing time |
| Correct citation, wrong answer | Claim against the cited sentence | Evaluate evidence support separately from ID membership |
| Old rule keeps appearing | Document versions and answer cache | Reindex the changed source and invalidate affected entries |
| Private text reaches context | Retrieval permissions and cache partitioning | Run two-user isolation tests before resuming service |
| Empty answers during outages | Search exception handling | Separate unavailable retrieval from a successful empty result |
Present sources and uncertainty clearly to the reader
A useful answer interface displays the answer, the source title and enough information to locate its supporting passage. For the lab example, show the policy heading and version beside the closing time. Do not label a search score as “answer confidence” unless you have a separate, justified way to calibrate that claim. Search similarity and answer correctness are different measurements.
Build source links from approved application records rather than trusting a model to invent complete URLs. If a citation is internal, check that the requesting user may open it. If a source is unavailable, state that limitation and offer a suitable route to the document owner. A broken link with a confident answer is a poor experience even when the underlying passage once existed.
Use plain messages for failures. “I found no weekday access policy in the permitted documents” is more informative than “No data.” A service outage should say the evidence service is temporarily unavailable. A scope ambiguity should ask a short question such as “Do you mean weekday or weekend access?” Each message gives the reader a next step without pretending the system knows more than it does.
What should the finished portfolio project contain?
Include a permitted source collection, a document preparation explanation, retrieval examples, an answer contract and a test report. Demonstrate at least one missing-answer case and one source conflict. Make it clear whether the system uses a live model, a local model or fixed test responses.
Show a failure you found and how you diagnosed it. For example, a policy exception lost during chunking is a more meaningful engineering lesson than a polished screenshot with no supporting evidence.
Frequently asked questions
Does RAG remove hallucinations?
No. Retrieval can provide useful evidence, but the system can retrieve the wrong passage or generate an unsupported interpretation. Test both layers.
Do I always need a vector database?
No. The retrieval approach depends on the collection and task. Start with a suitable baseline and measure what a more complex approach improves.
Is a citation enough to prove an answer?
No. The source must exist, be accessible to the reader and actually support the claim. A valid-looking reference is not evidence by itself.
Does this example test embedding quality?
No. Its retrieval is a mock baseline using a literal topic match. Comparing embedding models requires a real indexed collection and a reviewed retrieval question set; the fixed agent response cannot supply that evidence.
Should every question trigger model-controlled search?
No. Retrieving once in application code is easier to inspect for a narrow task. Tool-driven retrieval can help when a task needs follow-up searches, but it adds decisions, latency and stopping rules that you must evaluate.
Connect retrieval to broader agent skills
Review structured output checks before designing the answer contract. For guided learning across Python agents, tools and testing, explore the Pydantic AI course.
Download the Pydantic AI practice pack
Get nine offline Python examples, a project-review checklist and an evaluation case sheet. The examples use simulated model responses and do not need an API key. Submit this form to open the ZIP download.
Brolly Academy will store your request and use your details to handle it. Course follow-up is optional; the checkbox is not selected automatically.
Read our Privacy Policy for information about handling your details.
Put this into practice with guided training
Explore the Pydantic AI course for the syllabus, guided projects and training options.
