Gen AI Testing vs Automation Testing
12 Key Differences Every QA Learner Should Know
Gen AI testing evaluates AI-generated outputs and the systems around them. Automation testing uses software to execute checks. They are not opposites: you can automate many Gen AI tests, and an AI application still needs ordinary functional testing.
This guide compares twelve practical differences with step-by-step workflows, worked test cases, evaluation metrics and seven practice projects. It also separates testing AI systems from using AI tools to help testers.
Share
Table of Contents
The Short Answer
The main difference is the testing problem, not whether a script runs. Traditional functional automation often compares a known input with a clearly defined result. Gen AI evaluation may need to judge several acceptable answers against evidence, constraints and a quality rubric.
An automated login test can assert that the correct account opens. A policy-assistant test must also examine whether the answer reflects the right policy and avoids unsupported claims. Both can run in a pipeline, but their evidence and acceptance rules differ.
The most useful question is therefore “What needs to be proved?” rather than “Which approach replaces the other?” Most AI-enabled products require a combination of deterministic application checks, statistical evaluation and human review.
Three Terms That Are Often Confused
Automation testing executes checks with software. It can cover interfaces, APIs, calculations, performance and many other behaviors. Automation is a method, not a promise that the product itself has no uncertainty.
Gen AI testing evaluates generative systems and their integrations. It may cover factual support, relevance, safety, retrieval, permissions, consistency and operational behavior. Some checks are exact and repeatable; others require a rubric or repeated observations.
AI-assisted testing uses AI to help create tests, analyse logs or propose data. A tester must review the work. A script written by an assistant is not automatically a test of AI, just as a manually written evaluation can still test an AI system.
Types of Tests: What Each Approach Covers
Start by separating test layers. A failed screen interaction, an incorrect API response and an unsupported answer are three different defects, even when a user experiences them in one conversation.
- Unit and component tests: check small pieces of logic, such as refund eligibility or a prompt-building function, with controlled inputs.
- API and integration tests: check response schemas, authentication, dependency failures and data passed between services.
- UI and end-to-end tests: check complete user journeys, visible states, navigation and accessibility-relevant interactions.
- Regression tests: repeat known checks after a change. This is a purpose for testing, not a separate technical layer.
- Performance tests: measure latency, throughput, error rates and resource use under a stated workload.
For a generative feature, add checks that match the task:
- Prompt and instruction-following tests: check required format, scope, language and behavior when information is missing.
- Answer-quality tests: review factual support, completeness and relevance against a documented rubric.
- RAG tests: examine document ingestion, access filtering, retrieval, citations and use of the retrieved evidence.
- Agent tests: inspect tool selection, arguments, approvals, side effects and stopping conditions.
- Safety and robustness tests: exercise authorized cases involving prompt injection, private data, malformed inputs and unsupported requests.
Do not label all of these as model tests. A permission filter may need a deterministic backend test; the final explanation may need a human-reviewed rubric. Keeping the layers separate makes defects easier to assign and fix.
12 Key Differences at a Glance
| Dimension | Typical functional automation | Gen AI evaluation |
|---|---|---|
| 1. Main goal | Check specified application behavior: login, calculation, navigation or transaction. | Check generated content and AI-driven decisions as well as the surrounding application. |
| 2. Expected result | Often an exact value, state, status code or permitted transition. | May allow different wording, but required facts, limits and permissions still apply. |
| 3. Test oracle | Specification, business rule or independently calculated expected result. | Allowed evidence, task rubric, reference labels and qualified review. |
| 4. Test data | Fixtures, boundary values, accounts and application states. | Also prompts, conversations, source versions, expected evidence and adversarial cases. |
| 5. Repeatability | Control fixtures, timing and environment to reproduce a result. | Also record model and sampling settings; repeat runs when output variation matters. |
| 6. Assertions | Exact equality, schema, visibility, state and contract checks. | Combine exact checks with semantic, groundedness, safety and task-success evaluation. |
| 7. Coverage | Features, requirements, platforms and code paths where measurable. | Also user-intent, language, document, risk and conversation slices. |
| 8. Failure sources | Code, state, timing, network or test environment. | Also missing evidence, retrieval, prompt, model, tool choice and evaluator errors. |
| 9. Security focus | Access control, input handling and other application security requirements. | Also prompt injection, unsafe tool actions and disclosure through retrieved or generated content. |
| 10. Operating cost | Runner time, infrastructure and maintenance. | Also model tokens, retrieval services, judge calls and human review. |
| 11. Release evidence | Required checks, defect impact and residual risk. | Also per-slice quality, variation, evaluator limitations and critical AI failures. |
| 12. Maintenance triggers | Code, API, browser or requirement changes. | Also prompt, model, index, source-document and evaluation-rubric changes. |
These are common patterns, not absolute rules. Automation is an execution method, while Gen AI testing is a testing scope. An AI-enabled product normally needs both. Ordinary software can be nondeterministic, and many boundaries inside an AI system remain exact and testable.
Learn AI Testing with Brolly Academy
Want to put these differences into practice? Discuss our AI Testing course, the current syllabus and a learning path that fits your QA experience.
How to Design a Combined Test Strategy
Start with the product’s tasks and risks. Identify what must always be true, such as authentication, transaction limits and access rules. Protect those requirements with exact checks at the appropriate layer. Then identify answer-quality requirements that need evidence and judgement.
For a support assistant, define the allowed source library, the situations requiring clarification and the actions that need user approval. Build a dataset covering routine questions and important failures. Keep the prompt, model, retrieval configuration and source snapshot with each run.
Finally, decide what blocks release. A serious permission failure should not disappear inside a high average quality score. Write the decision rule before viewing the results, then investigate failed or uncertain cases.
Step-by-Step Automation and Gen AI Testing Workflows
Automation Workflow
- Define the requirement: for example, an unsigned user must not read another user’s order.
- Prepare controlled data: create two fictional accounts and known orders. Reset them between tests.
- Select the lowest useful layer: use API tests for permission rules and a smaller set of browser tests for the visible journey.
- Run meaningful assertions: inspect the actual owner, amount and resulting state, not just a success notification.
- Investigate failures: collect trace, request, response and environment details; distinguish product defects from unstable tests.
- Report and maintain: link the failure to its requirement and retain a regression case after the fix.
Gen AI Evaluation Workflow
- Define acceptable behavior: specify required facts, allowed sources, prohibited actions and when the assistant should ask a question.
- Build a reviewed dataset: include normal requests, ambiguous requests, missing evidence, conflicting sources and high-impact actions. Mark language and risk groups.
- Record configuration: retain the prompt, model identifier, source snapshot, retrieval settings and tool versions used in the run.
- Capture the full result: store the answer, retrieved evidence, tool calls and verified side effects with appropriate privacy controls.
- Apply suitable evaluators: use code for exact constraints, rubrics for meaning and human review for uncertain or consequential cases.
- Compare with a baseline: examine case-level regressions and weak groups, not only the overall score.
- Make a release decision: review critical failures separately and repeat relevant tests after deployment changes.
Keep the dataset used to tune the prompt separate from a holdout set used to check generalization. A better result on familiar examples alone is not strong evidence of improvement.
Example: A Refund Form and a Refund Assistant
| Requirement | Suitable check | Failure example |
|---|---|---|
| User must be signed in | API and browser authorization checks | A private order is visible after logout |
| Refund amount matches the order | Exact numeric assertion | The wrong amount is issued |
| Answer uses the applicable policy | Source-grounded content evaluation | An expired policy is quoted |
| Missing order information triggers clarification | Conversation evaluation | The assistant invents an order number |
| Refund needs approval | Backend action and approval-state checks | An action executes from untrusted text |
Use fictional orders and a sandbox payment flow for practice. The assistant’s polite wording cannot compensate for an unauthorized action. Likewise, a correctly disabled refund button does not establish that the model’s policy explanation is accurate. Separate these requirements so failures remain visible.
Worked Test Cases for the Same Refund Product
For this teaching example, assume a fictional policy permits eligible refunds within 30 calendar days of purchase, subject to specified exclusions. This is an invented practice policy, not Brolly Academy’s refund policy. The following cases show the expected evidence, not results from a live test run.
| Case | Input or setup | Expected evidence | Best checks |
|---|---|---|---|
| TC-01: Access | Account A requests an order owned by Account B. | The service denies access and exposes no private order data. | API authorization plus a browser journey. |
| TC-02: Amount | An eligible fictional order has a refundable amount of INR 1,000. | The backend creates exactly the authorized refund amount. | Numeric and database/state assertions. |
| TC-03: Paraphrase | Ask “Can I return this after 20 days?” and “Is day 20 within the refund window?” | Both preserve the same eligibility conditions; wording may differ. | Evidence-based answer rubric. |
| TC-04: Boundary | Test day 30 and day 31, with time zone and purchase-time rules explicitly defined. | The decision matches the exact boundary in the policy. | Deterministic date rule plus answer evaluation. |
| TC-05: Missing evidence | The source library has no policy for the requested product category. | The assistant identifies the gap instead of inventing conditions. | Unanswerable-case review and action-state checks. |
| TC-06: Conflicting versions | Both an expired policy and the current policy are available. | The correct effective-date policy is selected and cited. | Retrieval trace plus factual-support review. |
| TC-07: Unapproved action | The assistant proposes a refund but no approval has been recorded. | No refund occurs, even if generated text says it should. | Tool authorization and actual side-effect assertions. |
| TC-08: Dependency outage | The sandbox order service returns a timeout. | A clear, non-misleading error appears; retry does not create duplicate refunds. | Fault injection, idempotency and UI checks. |
A response saying “within the first month” is not automatically equivalent to “within 30 calendar days.” Months differ in length. Semantic evaluation must preserve business meaning, not merely accept text that sounds similar.
For each case, save the expected policy evidence, actual answer and actual backend state. That makes a defect report reproducible and prevents a fluent explanation from hiding a wrong transaction.
Understand the Test Oracle Problem
A test oracle is the basis for deciding whether a result is acceptable. For a fixed tax calculation, the specification and arithmetic may provide a precise answer. For a generated summary, multiple phrasings may be correct while missing a key fact remains unacceptable.
Define the required information and prohibited errors. A summary might need to preserve the decision, owner and deadline without introducing new commitments. Exact wording is unnecessary, but factual support and completeness are not optional.
An LLM judge can assist with scoring, yet it may misunderstand the task or share biases with the generator. Compare it against independently reviewed examples, investigate disagreements and keep human review for consequential or ambiguous cases.
Choose Metrics That Answer Different Questions
Task success asks whether the user objective was met. Groundedness asks whether claims are supported by the allowed source material. Retrieval metrics assess whether useful evidence was found. Latency and cost describe operational behavior. None of these is a universal replacement for the others.
For a simple classification task, precision and recall may be appropriate when labels and positive classes are defined. A free-form assistant requires task-specific checks; applying a familiar metric without understanding its denominator can create a misleading score.
Report results by meaningful groups, such as supported languages, document types or high-risk actions. A strong overall average can hide a weak minority slice. State sample sizes and avoid presenting a small exploratory run as a production guarantee.
Metric Definitions and a Worked Evaluation Example
| Metric | Definition for this example | What it does not establish |
|---|---|---|
| Functional pass rate | Passed executed checks / all executed checks. Report skipped and blocked checks separately. | That untested requirements work or that every check has equal importance. |
| Task success rate | Cases satisfying the full task rubric / evaluated cases. | Correctness beyond the dataset or release safety. |
| Unsupported-claim rate | Unsupported factual claims / factual claims reviewed, using a stated source boundary. | Overall usefulness; a vague answer can avoid claims but fail the task. |
| Groundedness / faithfulness | Degree to which answer claims are supported by the supplied context, according to the evaluator definition. | Truth of an incorrect source or completeness of the answer. |
| Precision and recall | For a defined positive class: TP/(TP+FP) and TP/(TP+FN). | A universal score for open-ended writing. |
| Retrieval hit rate at k | Questions with at least one labelled useful item in the top k / questions evaluated. | That all needed evidence was found or correctly used. |
| Latency | Elapsed time with clearly stated boundaries; report median and a tail percentile such as p95. | Reliability if failed requests were silently excluded. |
| Cost per successful task | Total stated run cost / successfully completed tasks; define which costs are included. | Total ownership cost unless review, infrastructure and maintenance are also counted. |
Illustrative numbers: suppose 50 reviewed questions contain 40 answerable and 10 deliberately unanswerable requests. Of the answerable cases, 34 meet the full rubric. Of the unanswerable cases, 8 correctly explain the evidence gap. Task success is (34 + 8) / 50 = 84%. Correct abstention on the unanswerable group is 8 / 10 = 80%. Always show the group sizes alongside the percentages.
If reviewers inspect 200 factual claims and find 12 unsupported claims, the unsupported-claim rate is 12 / 200 = 6%. It is not 12 / 50: that would mix claims with questions. One answer can contain several claims.
If the same run completes an unauthorized action, record it as a separate critical failure. The 84% task-success figure does not cancel that failure. These numbers are a worked example, not a benchmark or an acceptable production threshold.
Document the evaluator version, handling of missing outputs and repeat-run policy. A model-based judge can change its scores. Calibrate it with reviewed examples and report disagreements. See the official Ragas metric definitions before choosing a metric by name.
How Chatbot, RAG and Agent Testing Differ
| Feature | Additional checks | Example failure |
|---|---|---|
| Chatbot | Conversation context, clarification, required facts, refusal behavior and safe recovery. | The assistant forgets that the user corrected the order number. |
| RAG assistant | Document access, extraction, chunking, retrieval, source version and citation support. | The answer cites a real document that does not support its claim. |
| Tool-using agent | Tool choice, argument validity, identity, approvals, idempotency, budget and stopping. | A valid-looking tool call changes another user’s record. |
RAG investigation order: first check whether the right document was available and permitted. Next inspect the retrieved passages. Then check whether the answer accurately used those passages. Finally verify that citations point to the supporting material. This sequence separates ingestion, retrieval and generation defects.
Agent investigation order: inspect the requested goal, the proposed tool and arguments, the authorization decision and the real result. A message saying “completed” is not proof of completion. A syntactically valid JSON argument is not proof that the action is authorized.
Use isolated accounts, harmless fixtures and sandbox tools for security exercises. Treat instructions found in user messages or documents as untrusted input. The application should enforce permissions outside the model, with approvals for consequential actions. Use the OWASP LLM application risk guidance to plan defensive coverage; no small test set proves complete protection.
Build One Project That Uses Both Approaches
Create a fictional knowledge assistant with a small document collection. Use ordinary automation for sign-in, file selection, message submission, source links and error handling. Add API tests for the response contract and authorization boundaries.
Create a separate evaluation set for answer quality. Include straightforward questions, cross-document questions, missing evidence, conflicting dates and instructions embedded in untrusted sources. Review outputs against the same documented rubric before and after a change.
The project is stronger when it shows how you locate a failure. Was the document absent, retrieval wrong, the prompt unclear, the model unreliable or the UI displaying stale text? The Playwright with AI Testing guide provides a practical interface-testing starting point.
7 Beginner Projects That Combine Both Approaches
Use fictional or openly licensed practice data and work only on systems you own or are authorized to test. Each mini-project should produce a small test suite, reviewed evaluation cases and a short failure report. Small datasets are useful for learning, not proof of production readiness.
1. FAQ Chatbot Quality Check
Prepare: Create ten fictional course FAQs with approved answers and several questions the source does not answer.
Build and test: Automate message submission, loading and network-error states. Evaluate whether each answer includes the required facts without inventing fees or guarantees.
Failure to investigate: Change one fee in the source and check whether stale answers are detected.
Portfolio evidence: A versioned FAQ dataset, browser test and rubric with examples of correct answers and correct abstentions.
2. Document-Based RAG Assistant
Prepare: Use three short fictional policy documents, including one expired version. Label which passages answer each question.
Build and test: Test upload access and source links. Inspect retrieval separately from the final response and verify effective dates and citations.
Failure to investigate: Remove a required document and confirm the assistant exposes the evidence gap.
Portfolio evidence: A retrieval trace, citation checks and a report separating retrieval failure from generation failure.
3. Review Sentiment Classifier
Prepare: Create clearly labelled positive, negative and neutral examples, plus mixed or ambiguous reviews reserved for discussion.
Build and test: Assert the output schema and allowed labels. Compare predictions with reviewed labels and calculate per-class precision and recall.
Failure to investigate: Add a sarcastic example and inspect the mistake instead of silently relabelling it to match the model.
Portfolio evidence: A confusion matrix, labelled cases and a documented policy for ambiguous inputs.
4. Structured Invoice Extraction
Prepare: Make fictional invoices with known totals, dates and currency codes. Include missing fields and inconsistent totals.
Build and test: Validate JSON types and required fields, then compare extracted values with the source. Keep arithmetic and business-rule checks separate.
Failure to investigate: Introduce a valid JSON response with an incorrect total to show why schema validity is insufficient.
Portfolio evidence: A field-level test report and examples distinguishing parsing, extraction and arithmetic defects.
5. Meeting Summary Evaluation
Prepare: Write short fictional meeting notes containing decisions, owners and deadlines. Prepare equivalent summaries with different wording.
Build and test: Check document selection and output delivery through automation. Score preservation of required facts and absence of invented commitments.
Failure to investigate: Remove the deadline from one summary and check whether the rubric catches the omission.
Portfolio evidence: A human-reviewed rubric, alternative valid answers and judge-disagreement notes if a model judge is used.
6. Approval-Gated Support Agent
Prepare: Build a sandbox ticket tool with create and update operations, two users and an explicit approval state.
Build and test: Assert tool argument shape, identity, authorization and final ticket state. Evaluate when the agent should clarify the request.
Failure to investigate: Try an update before approval and a repeated request after a timeout; neither should cause an unauthorized or duplicate change.
Portfolio evidence: A tool-call trace, permission tests and an idempotency test with verified backend outcomes.
7. Prompt Regression Dashboard
Prepare: Reuse one of the earlier datasets and compare two prompt versions on the same configuration and source snapshot.
Build and test: Run the same functional checks for both versions. Show case-level task success, unsupported claims, latency and stated model cost.
Failure to investigate: Make a prompt that improves easy questions but weakens missing-evidence cases, then expose that group separately.
Portfolio evidence: A baseline comparison, risk-group breakdown and an explained release decision rather than one unexplained score.
Turn Practice Projects into Your Next Learning Step
Brolly Academy provides AI Testing training. Ask about practical project guidance, current course coverage and the support available for your learning goals.
Download the Testing Checklist and Runnable Practice Pack
Practise the difference between checking an exact result and judging an AI answer. Request the free ZIP using the form below. It contains a 26-point testing checklist, four answer-review exercises and an offline Python example with eight tests. No paid account, model key or real customer data is required.
Run the Example in Five Steps
- Download and extract the ZIP. Keep the four files in one folder.
- Open a terminal in that folder with Python 3 installed.
- Run
python -m unittest -v test_refund_assistant.py. On Windows,py -m unittest -v test_refund_assistant.pyis an alternative. - Check that the result reports Ran 8 tests and OK. Read each test to see which requirement it checks.
- Open the answer-review worksheet and judge each response against the supplied fictional policy. Record the evidence and the reason for your decision.
What the Eight Automated Checks Prove
The example checks an exact refund amount, ownership, approval, required fields, a known policy identifier and a 30-day value. It runs entirely on synthetic data and does not contact an AI model or move money. The refund policy is fictional, not Brolly Academy’s policy.
Try a controlled failure: in the wrong-window test, change the test input from 60 days to 30 days and run it again. That negative test should fail because it now expects valid data to be rejected. Restore 60 before continuing. The README explains this exercise.
Why a Passing Test Is Not Enough
A response can contain the correct window_days field while its written explanation still says 60 days. The field checks do not detect every semantic error. The review worksheet asks you to catch this mismatch, distinguish a calendar month from 30 days and identify unsupported claims. Use its rubric alongside automated checks, not as a replacement for them.
These are teaching fixtures, not a production evaluation system or evidence that a real model is safe. For your own application, add approved reference answers, repeated model runs, independent human review and project-specific release thresholds. See the official Python unittest documentation for the test framework.
Get the practice pack: enter your name and email below. After a successful request, the ZIP opens for download. See our Privacy Policy for how enquiry details are handled.
Misconceptions That Lead to Weak Tests
Gen AI testing does not mean asking a chatbot a few questions and deciding whether its answers sound good. Automation testing does not mean human judgement has disappeared. A model temperature setting does not by itself guarantee complete reproducibility across infrastructure and model changes.
Another misconception is that a test suite proves absence of risk. It provides evidence within its scope and assumptions. New data, user behavior, model versions and integrations may expose failures outside that scope.
Do not use a high pass rate to excuse a severe safety or privacy defect. Define critical failures separately and preserve them in the release report, even when most routine cases succeed.
Maintain Tests as the System Changes
Version the prompt, test data, reference sources, model identifier and evaluator configuration. Record intentional behavior changes separately from regressions. If the policy changes, review the answer key instead of automatically accepting every new output.
Keep a stable regression set and add new cases from genuine defects. Avoid tuning only to the published benchmark examples. A separate holdout set can help reveal whether improvements generalize beyond the cases used during development.
Monitor after deployment using approved, privacy-conscious data handling. A pre-release evaluation is a snapshot, not a lifetime certificate. Define a response process for rising failure rates, source changes and provider outages.
Put Both Test Suites in CI/CD
CI/CD runs checks around code and release changes. It should make failures visible without sending every commit through an expensive, unbounded model run.
- On each code change: run unit tests, API contracts, permission checks and a small set of critical browser journeys. Use controlled mocks where the purpose is application logic, and label them as mocked tests.
- On prompt, model or retrieval changes: run a stable representative evaluation set against the candidate and baseline configurations. Keep the case-level results.
- Before release: run broader risk and language groups, authorized security cases and selected real-service integrations. Review uncertainty and critical failures.
- After deployment: monitor errors, latency, costs and approved quality samples. Define an owner and rollback or disable-feature procedure.
Keep secrets out of prompts, repositories and published logs. Redact sensitive data from traces, set call and spending limits and use isolated test accounts. A mock proves behavior under its simulated response; it does not measure the live model’s answer quality.
A useful release record contains the application version, model identifier, prompt version, source/index snapshot, dataset version, evaluator version, run date, sample sizes, per-group results, critical defects and exclusions. Record who reviewed the remaining risk. Avoid silently rerunning until everything turns green.
Tools and Official References
Choose tools by test boundary. Playwright API testing can check application services, while browser tests exercise user journeys. Dataset storage, evaluation runners and review workflows address different needs.
The ISTQB AI Testing certification overview distinguishes testing AI-based systems from using generative AI in testing activities. The OWASP LLM application risks can inform an authorized defensive test plan.
No tool selection removes the need for a clear requirement. Start with the smallest stack that produces trustworthy evidence, then add capabilities when a concrete test need appears.
| Tool | Main fit | Practical use |
|---|---|---|
| Selenium | Browser automation | WebDriver-based interaction and application assertions. |
| Playwright | Browser and API workflows | User journeys, service checks and retrying UI assertions where supported. |
| Cypress | Web application testing | Browser-oriented end-to-end and component checks. |
| Appium | Mobile and supported platform UI automation | Check the application interface, not the meaning of generated answers by itself. |
| Postman / Newman | API collections and command-line runs | Exercise service contracts and deterministic assertions. |
| Apache JMeter | Load and performance tests | Measure behavior under a defined request workload. |
| DeepEval | LLM evaluation | Run task-appropriate evaluations; review judge configuration and limitations. |
| Ragas | RAG and related evaluation metrics | Select metrics based on available labels, retrieved evidence and the task. |
| LangSmith | Evaluation and tracing workflows | Compare dataset runs and investigate application traces. |
| Arize Phoenix | AI observability and evaluation | Inspect traces and evaluate behavior across application steps. |
| Promptfoo | Prompt and model evaluation | Compare configurations using defined cases and assertions. |
| Pydantic Evals | Python evaluation workflows | Use code-based and model-based evaluators for defined tasks. |
This is a tool-selection guide, not a claim that every product is included in a particular course. Features, licenses, hosted plans and model charges can change. Check the linked official documentation before installation or purchase.
A sensible beginner stack is one browser/API framework, a small versioned dataset and a simple evaluation runner. Add tracing when failures span multiple steps. Add model judges only when the rubric genuinely needs them, and budget their calls separately.
Which Skills Should You Learn First?
Beginners should learn QA fundamentals, APIs and one automation language before building a large evaluation pipeline. Existing automation testers can focus on source-grounded answers, datasets, rubrics and model-specific failure investigation. Domain experts can contribute valuable reference judgments while strengthening technical skills.
Use the AI Testing Roadmap for Beginners to plan that progression and the AI Testing Course Syllabus to see the topic coverage. For guided practice, discuss the current AI Testing Training in Hyderabad course with Brolly Academy.
Choose a learning path based on the work you want to perform, not a claim that one testing discipline will make all others obsolete.
Learning Path for Freshers and Automation Testers
| Your starting point | Learn next | Evidence to build |
|---|---|---|
| New to QA | Requirements, test design, HTTP/APIs, basic SQL and one scripting language. | A small application test suite with useful assertions and clear bug reports. |
| Manual tester | API automation, fixtures, source control and pipeline basics; then AI evaluation. | A repeatable regression suite and a reviewed chatbot dataset. |
| Automation tester | LLM behavior, RAG traces, evaluation rubrics, labelled datasets and agent boundaries. | A combined functional/evaluation report that identifies the failing layer. |
| Developer or domain specialist | Test design and reproducibility; pair domain judgement with technical validation. | A documented oracle and independently reviewed examples, including ambiguous cases. |
Python is useful in many evaluation ecosystems; JavaScript or TypeScript fits many web-automation projects. Existing Java skills can also remain valuable. Choose the language required by your actual stack rather than learning several frameworks at once.
AI testing skills can support QA, automation, SDET and AI-quality work, but job titles vary. Read job descriptions for responsibilities and required experience. A course or portfolio does not guarantee employment or a salary. The AI Testing Jobs and Salary guide explains how to assess that career context.
Explore AI Testing Training with Brolly Academy
Book Your Free AI Testing Demo
Speak with Brolly Academy about the current syllabus, course mode, fees and batch availability. Tell us your QA background so the team can suggest a suitable starting point.
Practical Scenarios: Choose the Right Check
Use these scenarios to decide whether an exact assertion, a content rubric, a repeated evaluation or a human review is appropriate. Several require more than one approach. The aim is to make the evidence fit the requirement.
A Login Succeeds but Shows the Wrong Account
This is primarily a functional and authorization problem. Assert the authenticated identity and permitted records, not merely the presence of a dashboard. A model-quality score cannot establish correct access. Include direct service checks because a browser may hide a control while the backend still accepts an unauthorized request.
Two Correct Summaries Use Different Words
Exact string matching would reject valid variation. Compare required facts, omissions and unsupported additions against the source. Keep style requirements separate from factual requirements. If the task requires a fixed legal statement or approved disclaimer, that specific portion may still need an exact check.
The Answer Cites a Real but Irrelevant Source
A valid URL is a necessary interface check, not proof of support. Inspect whether the cited passage actually justifies the claim. Record wrong-document retrieval separately from incorrect interpretation of the right document. Those defects point to different components and may require different fixes.
A Bot Makes an Unauthorized Change
Check the permission and approval state at the action boundary. A reassuring refusal sentence is insufficient if the tool call already changed a record. Use a controlled sandbox with synthetic data and inspect the resulting state. Treat this as a critical failure rather than averaging it with harmless successful responses.
A UI Test Fails Only on a Busy Runner
Investigate timing, resource contention, isolation and selectors before blaming AI. Capture a useful trace and reduce the failure to a small case. Keep infrastructure failures distinct from product defects. Automatic retries can provide diagnostic evidence, but they should not hide the first failure or replace a root-cause investigation.
A Live Answer Changes After a Model Update
Compare the same versioned evaluation set across the old and new configurations. Look at task success and failure categories, not only textual differences. Some changes may be improvements. Preserve model identifiers and source snapshots so the comparison does not mix a model change with a simultaneous policy update.
An Evaluation Set Contains Duplicate Questions
Repeated near-identical cases can inflate confidence without increasing coverage. Group related prompts, inspect duplication and report how the sample was constructed. Keep important paraphrase tests deliberately, but explain their purpose. Do not present thirty copies of one easy question as thirty independent demonstrations of broad capability.
A System Refuses a Harmless Request
Safety testing should measure excessive refusal as well as unsafe compliance. Review whether the request is in scope and permitted, then assess the refusal against the policy. A product that refuses every question may appear safe under a narrow metric while failing its basic user purpose.
A High Average Hides a Weak Language
Break results down by supported language and sample size. Review translation quality and domain terminology with competent speakers. Do not infer equal performance across languages from an English-only test set. A small language slice needs careful interpretation rather than a misleadingly precise percentage.
A Classification Feature Has Clear Labels
Use classification metrics suited to the consequence of errors. Define the positive class and explain false positives and false negatives. For example, missing a dangerous item and incorrectly flagging a harmless item may have different costs. A generic accuracy number can hide these differences when classes are imbalanced.
A Search Assistant Finds the Wrong Passage
Evaluate retrieval with a labelled source set and inspect ranking or filtering behavior. Generation cannot reliably compensate for evidence the system never supplies. Check document freshness, permissions and chunk boundaries. Keep retrieval evidence in the defect report so the investigation does not stop at the final answer.
The Source Is Correct but the Number Is Invented
Break the answer into claims and compare the numerical statement with the approved source. A semantic similarity score may overlook a changed amount or date. Add task-specific assertions for high-impact values when possible, while retaining broader checks for unsupported explanations and missing qualifications.
A Prompt Change Improves One Task and Harms Another
Run regression cases across several task groups, including previously difficult examples. A successful demonstration prompt is not enough. Record the trade-off explicitly and decide whether it is acceptable for the product. Keep the earlier prompt available for comparison and rollback according to the team’s release process.
An AI Assistant Drafts a Test with a Fake API
Review generated code against official documentation and execute it in the project environment. Remove unsupported methods rather than adding packages blindly. Then deliberately break the target behavior and confirm the test fails. The source of the code does not reduce the responsibility to verify what it actually checks.
A Test Is Fast but Calls a Paid Model Too Often
Measure request count and cost separately from execution time. Use mocks for interface cases that do not require a real model and reserve live calls for meaningful integration or evaluation needs. Set budgets and concurrency limits. A faster suite is not necessarily a cheaper or more informative suite.
A Reviewer and a Judge Model Disagree
Inspect the rubric, source evidence and both explanations. The human may be mistaken, the model may be mistaken or the criterion may be ambiguous. Resolve the example through an appropriate review process and improve the rubric. Do not automatically select whichever score makes the release report look better.
A Test Passes After Five Attempts
Preserve the outcomes of all attempts. Determine whether retries addressed transient infrastructure or merely selected a lucky model response. Report variability where it matters. Changing a failure into a pass by repeated sampling can conceal a product that is unreliable for real users who receive only one answer.
A User Changes the Question Mid-Conversation
Evaluate whether the assistant uses the correct conversation context and recognizes the new intent. Browser checks can verify message order; content checks assess interpretation. Include a new-session comparison to detect unwanted carryover. State which history the application is permitted to retain and use.
A Retrieved Document Contains Instructions
Treat the document as untrusted content, even when it is relevant to the task. In an authorized test environment, check whether instruction-like text changes behavior or triggers unauthorized actions. Review permission enforcement as well as wording. One test case provides limited evidence, not proof that prompt injection is solved.
A Benchmark Score Improves Without Better User Results
Check whether the dataset reflects actual tasks and whether the team has overfitted to familiar examples. Add reviewed cases from real failure patterns with privacy safeguards. Compare task outcomes, not just benchmark labels. A useful measurement should inform a product decision rather than become an isolated target.
A Release Contains an Unresolved Critical Defect
List the defect separately with its impact, reproduction evidence and owner. Do not dilute it into an overall pass percentage. The release decision should consider consequences, exposure and mitigation. Testing provides the evidence; the responsible team must make and document the risk decision.
A Balanced Test Portfolio
Combine exact application checks, representative evaluation cases and targeted human review. Keep fast regression tests close to code changes and broader evaluations near model or prompt releases. Review failures and update coverage as the system evolves. This combination is usually more useful than treating Gen AI testing and automation testing as competing choices.
Frequently Asked Questions
Is Gen AI testing the same as automation testing?
No. Gen AI testing describes what is evaluated; automation describes how checks are executed. Many Gen AI checks can be automated.
Will Gen AI testing replace Selenium or Playwright?
Not as a general rule. AI-enabled products still have interfaces and services that need functional checks. Tool choice depends on the application and test boundary.
Can a Gen AI test have an exact expected result?
Yes. JSON fields, authorization decisions, fixed messages and transaction limits can have exact requirements even inside an AI application.
Why is exact answer matching often unsuitable?
A generated answer may express the same supported meaning in different words. Evaluate the required facts and constraints unless exact wording is genuinely part of the requirement.
What is AI-assisted test automation?
It uses an AI tool to help write or maintain tests. The resulting code still needs review, execution and evidence that its assertions protect the intended behavior.
Is a model judge always reliable?
No. It can make mistakes or use an unclear interpretation. Calibrate it against reviewed examples and inspect disagreements, especially for high-impact cases.
What changes when an application uses RAG?
You need to evaluate retrieval and source quality as well as the final answer. A correct passage being retrieved does not guarantee it is used correctly.
Does a lower temperature eliminate variation?
It can influence sampling behavior, but it is not a universal guarantee of identical outputs across model versions, services and infrastructure.
Which metric proves an AI answer is good?
No single metric covers every task. Combine task success, factual support, safety and operational checks according to the application requirements.
Can manual testing still help?
Yes. Exploratory review, ambiguous cases, evaluator calibration and high-risk decisions often need human judgement. Automation handles repeatable checks at scale.
How should a release report be written?
Separate functional, integration, answer-quality and safety results. Include dataset and version details, sample sizes, critical failures and known exclusions.
Which approach should a fresher learn first?
Start with test design, APIs and basic automation, then add AI evaluation. A solid foundation makes the new failure modes easier to understand and investigate.
