AI Testing Interview Questions
170 Best Answers for Freshers & QA Professionals
Prepare for AI Testing Interview Questions with 170 answers for freshers, manual testers and QA professionals. The guide covers testing AI applications and using AI to support software testing, with practical scenarios for prompts, large language models (LLMs), retrieval-augmented generation (RAG), automation and evaluation.
Start with the level you need, then practise explaining the risk, test data, expected result and evidence. Use the worked example and practice sheets to build your own honest project answers.
Share
Table of Contents
Quick Answer
AI testing interviews assess two related skills: checking AI-powered systems and using AI responsibly in the testing process. Strong answers explain what could fail, how you would test it and what evidence supports your decision. Knowing tool names alone is not enough.
New to the topic? Follow the AI Testing Course Syllabus and AI Testing Roadmap for Beginners.
Jump to: Basics · Experienced QA · Tools · Applied questions · Practice sheets
| Interview area | What to prepare | Why it matters |
|---|---|---|
| Basics | AI Testing meaning, QA fundamentals, test cases and defects. | These questions check your foundation. |
| LLM testing | Prompts, hallucination, relevance, tone and safety. | These questions check AI output quality thinking. |
| RAG testing | Retrieval, grounding, citations and source validation. | These questions check document-based AI testing. |
| Automation | Playwright, Selenium, API testing and regression checks. | These questions check practical execution. |
| Projects | Chatbot testing report, golden questions and defect evidence. | These questions check proof of skill. |
How to Answer AI Testing Interview Questions
Use a simple answer format: define the concept, explain why it matters, give one practical example, mention how you would test it and explain how you would report a defect. This structure helps you avoid one-line answers and makes you sound practical.
- Do not only memorize definitions.
- Use examples from chatbots, document assistants, AI search, AI summarizers or AI customer support.
- Mention evidence such as screenshots, source documents, expected answers and actual answers.
- For tool-based practice, read Playwright with AI Testing.
- For job-focused preparation, read AI Testing Jobs and Salary in Hyderabad.
Basic AI Testing Interview Questions
1. What is AI Testing?
AI Testing means testing software features that use artificial intelligence, machine learning or large language models. The tester checks whether the AI output is accurate, useful, safe, relevant and supported by the right data. It includes normal QA checks plus special checks for hallucination, bias, privacy, prompts, context and unpredictable answers.
2. How is AI Testing different from normal software testing?
Normal software testing checks fixed behavior, such as whether a button, form, API or payment flow works. AI Testing checks those items plus the quality of the AI response. The same prompt may not always produce the exact same words, so testers judge meaning, correctness, safety and business usefulness instead of only exact output.
3. What skills are required for AI Testing?
Start with quality assurance (QA), test design, clear bug reports, basic application programming interfaces (APIs) and careful reasoning. Then learn how prompts, retrieved documents and model outputs can fail. Automation helps with repeated checks, but a tester must still understand the expected result. You do not need to master every AI framework before your first project. Use the AI Testing Course Syllabus to identify topics you have not practised, and bring evidence of what you can actually test.
4. Can freshers learn AI Testing?
Yes. A fresher can start with software testing basics and a small, clearly scoped chatbot project. Learn to write a test case, choose an expected result, record the actual answer and explain a defect. Add API and browser automation after these basics. The AI Testing Roadmap for Beginners gives a learning sequence. A practice project helps demonstrate your skills, but it does not guarantee a job or replace the requirements of a particular role.
5. What is LLM testing?
LLM testing is the process of testing applications that use large language models to generate text, summaries, answers, code or recommendations. The tester checks if the response is correct, relevant, safe, complete and aligned with the user’s request. LLM testing also includes prompts, context, grounding and refusal behavior.
6. What is prompt testing?
Prompt testing means testing how an AI system responds to different user instructions. A tester gives clear prompts, unclear prompts, long prompts, risky prompts and edge-case prompts. The goal is to see whether the AI follows the request correctly without leaking data, inventing facts or giving unsafe output.
7. What is hallucination in AI Testing?
An AI hallucination is an invented, incorrect or unsupported statement presented as an answer. Examples include a made-up policy, a non-existent source or a false calculation. A confident tone is not proof of correctness, and a cautious tone does not make a false statement correct. Compare each important claim with a suitable reference. The guide to hallucination in generative AI explains the concept with more examples.
8. How do you test hallucination?
Prepare a labelled set of answerable and unanswerable questions with approved, versioned sources. For each response, identify the factual claims and check whether the evidence supports them. Record unsupported claims separately from omissions and broken citations. Have reviewers resolve ambiguous cases using the same rules. Repeat high-risk prompts because one successful response does not establish reliability. Save the prompt, source version, model settings and actual output so another tester can investigate the same conditions.
9. A chatbot gives two different answers for the same question. Is it a bug?
Different wording is not always a bug in AI systems. It becomes a bug when the meaning changes, the answer contradicts earlier output, the facts are different or the user gets confused. Testers should define acceptable variation and unacceptable inconsistency before deciding whether it is a defect.
10. How will you test an AI customer support chatbot?
For an AI customer support chatbot, I would test greetings, common queries, refunds, account issues, escalation, unsupported questions, privacy, tone and fallback responses. I would also check whether the bot uses company policy correctly and transfers the user to a human when the query is sensitive or unresolved.
11. How will you test an AI resume screening tool?
For an AI resume screening tool, I would test accuracy, fairness, role matching, skill extraction, privacy and bias. I would use resumes with different names, formats, gaps, experience levels and skill wording. The tool should not reject good candidates because of formatting, gender, location or irrelevant personal details.
12. How will you report an AI bug?
An AI bug report should include the prompt, actual response, expected response, test data, source proof, screenshot, defect category and risk. For example, if a chatbot invents a refund policy, the report should include the prompt, generated answer and the correct policy document section.
13. Can Playwright be used in AI Testing?
Yes. Playwright can test the browser workflow around an AI feature: entering a prompt, uploading a file, waiting for the result and checking error states. For example, a test can verify that the response panel is visible and an expected citation link exists. That alone does not prove the answer is correct. Add source checks or a reviewed evaluation rubric for meaning and factual support. Practise these separate layers in Playwright with AI Testing.
14. What AI Testing scenarios can be automated?
Good automation candidates include login, prompt submission, file upload, API status, response rendering, timeout messages, export buttons and basic keyword checks. Repeated regression prompts can also be automated. Deep factual correctness, bias review and complex safety judgment usually need human evaluation or a strong scoring rubric.
15. Should AI answers be matched exactly in automation?
Not every answer should be matched word for word. Use exact assertions when the requirement is exact, such as an allowed status value, a fixed calculation, a required disclaimer or a tool parameter. For open-ended explanations, check required facts, source support, format and safety instead of one exact sentence. A suite can combine both approaches. The official Playwright assertion documentation explains browser checks; the business requirements determine which answer-quality checks you need.
16. What is RAG testing?
RAG stands for retrieval-augmented generation. A RAG application retrieves documents or records and supplies them to a model when producing an answer. Testing checks document ingestion, retrieval, permissions, source versions, generated claims and citations. Retrieved content is not automatically trustworthy: it may be stale, incorrect or contain malicious instructions. Include questions the sources cannot answer, and verify the product’s required no-answer or clarification behaviour.
17. How do you test source-grounded answers?
To test source-grounded answers, I would ask questions whose answers are clearly present in the source documents. Then I would check whether the AI uses the right section, gives the right meaning and avoids adding unsupported facts. Citations should point to the same source that supports the answer.
18. What is a golden question set?
A golden question set is a reviewed benchmark of prompts, expected facts, source references and scoring rules. Include normal requests, edge cases and high-risk failures rather than only easy examples. Version the set and keep some evaluation cases separate from prompts used for tuning, otherwise the system may improve only on familiar tests. Run the same benchmark after a change and inspect individual failures, not just the average. LangSmith evaluation describes dataset-based comparisons and different evaluation methods.
19. Explain one AI Testing project.
One useful AI Testing project is a document assistant for company policies. Upload policy PDFs, ask employee questions, verify answers against the source and report hallucinations or missing citations. The project can include prompt test cases, expected answers, screenshots, defect reports and a small test summary.
20. What should freshers add to resume for AI Testing?
List skills you can demonstrate, not every tool name in a job description. A useful fresher project entry states the application tested, your role, the test cases you designed, tools you used and evidence you produced. Keep a redacted test sheet and defect report ready. Label a self-directed project honestly rather than presenting it as company experience. For role expectations and career context, read AI Testing Jobs and Salary in Hyderabad.
Fresher-Level AI Testing Interview Questions
21. What is the role of a tester in an AI project?
A tester in an AI project protects the user from wrong, unsafe or confusing AI behavior. The tester validates the application flow, prompts, outputs, sources, privacy rules and business logic. They also explain AI defects clearly to developers, product owners and business teams.
22. What is the difference between AI Testing and AI-assisted testing?
Testing AI means checking the behaviour of an AI-powered product, such as a model, chatbot or document assistant. AI-assisted testing means using AI to help test software, for example by suggesting test cases or reviewing a failure log. The two can overlap, but they are different responsibilities. See Gen AI Testing vs Automation Testing. ISTQB also distinguishes AI Testing from Testing with Generative AI; these links are references, not a claim that this article grants certification.
23. What is model output validation?
Model output validation means checking whether the AI response meets expected quality rules. The tester reviews accuracy, relevance, completeness, safety, tone, format and source support. It is not only about whether the model replied, but whether the reply is useful and correct.
24. What is non-deterministic output?
Non-deterministic output means the AI may give different wording or structure for the same input. This is common in LLM systems. Testers handle it by validating meaning, facts and required criteria instead of expecting one fixed sentence every time.
25. What is an AI test oracle?
An AI test oracle is the method used to decide whether an AI answer is acceptable. It may be a source document, expert-approved answer, checklist, rubric, database value or human reviewer. AI needs clear oracles because output can vary in wording.
26. What is a prompt test case?
A prompt test case contains the prompt, test objective, input data, expected behavior, actual response, evaluation criteria and result. For risky cases, it should also include what the AI must refuse or avoid. This makes prompt testing repeatable and easy to review.
27. What is AI response evaluation?
AI response evaluation is the process of scoring an AI answer against quality rules. The tester may check correctness, relevance, completeness, clarity, tone, safety, citation and format. The result can be pass, fail, partial pass or needs human review.
28. What is an AI evaluation rubric?
A rubric describes observable criteria for judging an answer. For a policy assistant, score factual support, required details, citation support and usefulness separately. Write examples of a full pass, partial pass and failure before the test starts. Reviewers should be able to apply the same rules to the same response. Keep critical failures, such as exposing another user’s record, outside an average score so good wording cannot cancel out a serious problem.
29. What is bias testing in AI?
Bias testing in AI checks whether the system treats users unfairly because of names, gender, age, region, language, background or other sensitive signals. Testers use varied profiles and compare outputs. If similar users get different treatment without a valid reason, it is a serious defect.
30. What is privacy testing in AI applications?
Privacy testing in AI applications checks whether the system protects personal, company and sensitive data. Testers verify that prompts, uploaded documents, logs, responses and integrations do not expose confidential information. Privacy testing is important because AI tools often process user text and files.
Intermediate AI Testing Interview Questions
31. How do you test prompt injection?
In an authorized test environment, check whether instructions in an untrusted prompt or document can redirect the application away from its approved task. Verify both the visible answer and any tool calls: a polite refusal does not prove that no unauthorized action occurred. Use synthetic records, limited test accounts and clear expected outcomes. OWASP’s prompt injection guidance explains the risk. Do not run disruptive tests against systems you do not have permission to test.
32. How do you test jailbreak attempts?
Jailbreak testing checks whether users can trick the AI into unsafe behavior. I would try roleplay, emotional pressure, encoded text, indirect wording and step-by-step bypass prompts. The expected result is a safe refusal or a helpful alternative that does not break policy.
33. How do you test AI answer relevance?
To test answer relevance, compare the response with the user’s exact question. The answer should stay on topic, address the main intent and avoid unnecessary unrelated information. A relevant answer can be short or long, but it must solve the user’s actual request.
34. How do you test completeness of an AI response?
Completeness testing checks whether the AI answer includes all important points needed by the user. I would create expected key points before testing, then mark which points are present, missing or wrong. A partial answer may be useful, but it should not be marked fully passed.
35. What is source attribution testing?
Source attribution testing checks whether an answer identifies its evidence correctly. Map each important claim to the cited document and passage, then confirm that the passage actually supports the claim. A real document title is not enough. Record source version, page or section and user access level. Distinguish three failures: no citation, a citation that cannot be opened, and a working citation that does not support the statement.
36. What is context window testing?
Context window testing checks how well the AI handles long input or long conversation history. I would test long PDFs, long chats and important details placed at the beginning, middle and end. The AI should not forget key facts or mix unrelated details.
37. How do you test multi-turn conversations?
To test multi-turn conversations, I would ask follow-up questions that depend on earlier messages. The AI should remember the correct context, handle corrections and avoid contradicting previous answers. I would also test topic switching to confirm it does not carry wrong context forward.
38. How do you test AI tone and communication quality?
Tone testing checks whether the AI communicates in the right style for the user and business. A banking assistant should be careful and formal, while a learning assistant can be friendly. The tester checks politeness, clarity, empathy and whether the tone matches the situation.
39. What is fallback testing in AI chatbots?
Fallback testing checks what the AI does when it cannot answer. A good fallback admits the limitation, asks for missing details or routes the user to the right next step. A bad fallback invents information or gives a confident answer without support.
40. How do you test escalation flow?
Escalation flow testing verifies whether the AI sends the user to a human, ticket, call or support team at the correct time. I would test complaints, legal issues, payment failures, medical risks and repeated failures. The escalation should be clear and easy to follow.
Advanced AI Testing Interview Questions
41. How do you test a RAG pipeline?
Test the RAG pipeline in stages. First check that files were parsed and indexed correctly. Then inspect which passages were retrieved and their ranking. Finally compare the generated answer and citations with those passages. Use a fixed set of labelled queries and repeat it after changes. A wrong answer caused by an obsolete source needs a different fix from a wrong answer generated from the correct source. Keep the retrieved context in the test evidence so these failures are distinguishable.
42. What is retrieval quality in RAG testing?
Retrieval quality describes whether the right evidence is found before the model answers. Label the passages needed for a test question, then examine the retrieved results. Useful checks include whether the necessary evidence was found, how many returned passages are relevant, where they rank and whether permissions were respected. The Ragas metrics guide lists evaluation options. Choose a metric that matches your labels and task, and inspect examples rather than treating a tool score as proof of truth.
43. What is chunking issue in RAG systems?
A chunking issue happens when documents are split badly before indexing. Important sentences may be separated from headings, tables or context. Testers find chunking issues by asking questions that need connected information and checking whether the retrieved chunk has enough surrounding meaning.
44. How do you test conflicting source documents?
For conflicting source documents, I would test whether the AI identifies the latest or highest-priority source. The answer should not merge contradictions blindly. It should mention conflict, use approved rules and show which source was followed.
45. What is model drift?
Drift describes changes in the conditions a model faces over time. Data drift means the input distribution changes, such as more requests in a new language. Concept drift means the relationship between inputs and the outcome changes. A regression after editing a prompt or deploying a model is a change-related failure, not automatically drift. Compare recent data and outcomes with a baseline, investigate affected groups and verify whether the monitoring signal reflects a real quality problem.
46. What is AI regression testing?
AI regression testing reruns approved checks after changes to the application, model, prompt, tools or knowledge base. Keep the dataset, evaluator and configuration versioned so comparisons are meaningful. Include both fixed software assertions and answer-quality criteria. Review new failures, critical risk categories, latency and cost rather than accepting a higher average score alone. When outputs vary, use repeated runs and record that variation instead of claiming a single run proves stability.
47. How do you test AI safety?
AI safety testing checks whether the system avoids harmful, illegal, private, biased or misleading output. Testers use risky prompts, sensitive scenarios and misuse cases. The system should refuse unsafe requests while still helping the user with safe alternatives.
48. What is red teaming in AI Testing?
Red teaming in AI Testing means testing the system like an attacker or difficult user. The tester deliberately tries to break safety rules, leak data, create harmful output or bypass restrictions. It helps teams find serious weaknesses before real users do.
49. What is human-in-the-loop testing?
Human-in-the-loop testing means humans review AI outputs, labels, risk cases or uncertain answers. It is useful when full automation is not reliable. The human reviewer confirms correctness, handles exceptions and improves the evaluation rules over time.
50. What is AI observability?
AI observability means collecting enough information to understand the system’s behaviour after deployment. Useful evidence includes request IDs, tool calls, retrieved document IDs, model versions, latency and evaluation outcomes. For a wrong policy answer, a trace can reveal whether retrieval used an old document. Protect logs with access controls, redaction and retention rules. A dashboard is useful only when alerts lead to investigation and a responsible owner can reproduce the problem.
Worked example: investigate a wrong policy answer. This is a hypothetical exercise, not a measured student or production result.
| Check | Teaching example |
|---|---|
| Approved source | Returns policy v3, section 2: unopened items can be returned within 14 days. |
| User request | Can I return an unopened item after 20 days? |
| Expected result | Explain that this falls outside the stated policy; cite section 2 without inventing an extension. |
| Faulty result | Yes, returns are accepted for 30 days. |
| First check | Inspect the retrieved document version and passage. |
| If the source was correct | Check prompt assembly, generated claims and citation mapping. |
| Evidence to keep | Prompt, source version, retrieved passages, response, model configuration and test ID. |
| Interview follow-up | Explain how your investigation changes if the retriever supplied an obsolete policy. |
Ready to practise these answers?
Explore AI Testing Training in Hyderabad at Brolly Academy. Ask which practical exercises, prerequisites and current batches match your experience.
AI Testing Tools and Evaluation Workflows
51. Which checks belong in AI API tests rather than browser tests?
Test authentication, permissions, request schemas, response schemas, status codes, timeouts and rate-limit handling at the API layer. Then test browser-specific behaviour such as loading states and how errors are displayed. An API may return HTTP 200 with an unacceptable answer, so also validate the application result and relevant content criteria. Keep secrets out of saved test evidence and use synthetic data when reproducing failures.
52. How do you compare two prompts without choosing the winner by intuition?
Run both prompt versions against the same held-out questions, sources and model configuration. Decide the evaluation criteria before seeing results. Compare correctness, required details, risk failures, latency and cost, then inspect examples where scores disagree. Repeat variable cases and have reviewers assess outputs without knowing the prompt version where practical. Report a trade-off honestly: one prompt may improve concise answers while making missing-information handling worse.
53. When would you use Ragas, LangSmith or ordinary test assertions?
Use deterministic assertions for fixed rules such as JSON fields and authorization results. A RAG evaluator can help compare retrieval and answer-quality measures across examples. Tracing and dataset-evaluation tools can help investigate where a pipeline failed. These approaches complement one another; no single tool certifies that every answer is correct. Choose based on the task, available reference data, deployment constraints and how the team will review uncertain results.
54. How would you test AI-generated tests before adding them to a suite?
Treat generated tests as suggestions that require review. Map them to requirements, check the expected result and run them against controlled examples. Include negative and boundary cases, remove duplicates and verify that assertions would catch a real defect. A test that only opens a page or repeats the implementation’s own assumptions can pass without protecting the user. Review synthetic test data for privacy and licensing constraints before committing it.
55. How do you choose test data for an AI evaluation?
Cover real user tasks, different input formats, missing information, permission boundaries and known failure modes. Use consented or synthetic data and remove personal information that is not needed. Keep a development set for improving the system and a separate held-out set for assessment. Document source versions and labels. If the test set contains only simple questions, a high score cannot support claims about complex or high-risk use cases.
| Task | Suitable approach | What it does not prove |
|---|---|---|
| Check login, upload and response controls | Browser tests with Playwright or an established UI framework | That the generated facts are correct |
| Check API contracts and permissions | API tests and schema validation | That a valid response is useful or source-supported |
| Compare retrieval and generated answers | Labelled datasets, RAG metrics and human review | That one benchmark represents every real user |
| Investigate a pipeline failure | Traces, versioned inputs and application logs | That the root cause is fixed without a retest |
AI Feature Testing Scenarios
56. How do you test multilingual AI responses?
To test multilingual responses, I would ask the same intent in different languages and mixed-language sentences. The AI should understand the meaning, preserve facts and respond in the expected language. I would also check whether translations change numbers, names or policy details.
57. How do you test personalization in AI systems?
Personalization testing checks whether the AI uses user profile data correctly and safely. It should personalize based on allowed data, not private or guessed information. I would test new users, returning users, changed preferences and users with restricted permissions.
58. How do you test AI recommendations?
AI recommendation testing checks whether suggestions are relevant, fair and explainable. I would test different user histories, empty history, repeated clicks, unusual preferences and business rules. Recommendations should not expose private data or push harmful options.
59. How do you test AI summarization?
AI summarization testing checks whether the summary keeps the main meaning while removing unnecessary detail. I would compare the summary with the original document for missing key points, wrong facts, changed tone and invented conclusions.
60. How do you verify that AI-generated code meets a requested function?
Turn the request into executable requirements: inputs, outputs, edge cases and error handling. Run the generated code in an isolated test environment with normal, boundary and invalid inputs. Include a test the first draft fails, so you can show that the checks detect incorrect logic. Review assumptions about dates, units and null values. Passing a few happy-path examples is evidence about those examples, not proof that the entire solution is correct.
Behavioral, HR and Project Interview Questions
61. Why do you want to work in AI Testing?
A good interview answer is: I want to work in AI Testing because companies are adding AI features and those features need careful validation. My QA skills help me check accuracy, safety, usability and risk, not just whether the screen works.
62. How should you explain your AI Testing project in an interview?
Use a real project and separate your work from the team’s work. Explain the feature, the main risk, the test data, one important failure and the evidence you collected. Then describe the fix or decision and what remained uncertain. A fresher can use an honestly labelled practice project. Do not invent a client, a pass-rate improvement or production experience. Bring redacted artefacts that demonstrate your process rather than memorizing a success story.
63. How would you handle a developer who says an incorrect answer is just normal AI variation?
First separate wording variation from a requirement failure. Two sentences can mean the same thing, but two different refund limits cannot both match the same policy. Show the prompt, source version and actual response, and ask the developer to review the agreed criteria. Reproduce the issue under recorded conditions and explain user impact. The goal is a shared decision based on evidence, not winning an argument about whether AI is unpredictable.
64. How would you prioritize AI defects when release time is limited?
Group failures by harm, reach and the core user task. Exposing private data or performing an unauthorized action needs different treatment from a minor tone problem. Present the evidence, affected cases and possible mitigations to the product owner. Agree which risks block release and who owns any accepted risk. Explain your reasoning openly, and do not hide a serious failure inside an overall pass percentage.
65. What should you say when you have not used an AI tool mentioned by the interviewer?
Be honest about the gap. Explain the related testing problem you understand, such as dataset evaluation or API validation, and describe how you would learn the tool using its documentation and a small test project. Give an example from a tool you have actually used. Interviewers can ask detailed follow-ups, so inventing hands-on experience is less useful than showing a clear learning method and sound testing judgement.
More Scenario-Based AI Testing Interview Questions
66. An answer passes a keyword check but gives the wrong policy. What would you change?
A keyword check can confirm that a word appears, not that the meaning is correct. Replace it with explicit requirements: the correct limit, required exception, supporting source and prohibited claim. Keep exact checks for fixed facts and a rubric for flexible explanation. Add the failing example to regression tests. In the defect report, show why the old assertion passed and which new assertion exposes the actual error.
67. How do you test an AI chatbot when the user asks an unclear question?
When the user asks an unclear question, the chatbot should not guess important details. It should ask a follow-up question or give a general answer with clear assumptions. I would test vague, incomplete and mixed-topic prompts.
68. How many times should you repeat a non-deterministic test?
There is no universal repeat count. Choose the number based on output variability, risk, cost and the confidence needed for the decision. Fix the configuration and log each result. For a teaching exercise, several repeats can reveal variation, but they do not establish a production reliability guarantee. Investigate inconsistent critical outcomes even when most runs pass, and explain the limits of your sample in the report.
69. How do you test AI response consistency?
AI response consistency means similar inputs should produce compatible answers. I would test repeated prompts, paraphrased prompts and multi-turn follow-ups. Inconsistency is a defect when the AI changes facts, contradicts itself or gives different instructions for the same case.
70. How do you test AI response accuracy?
Accuracy testing checks whether the AI answer is factually correct. I would verify claims against source documents, databases, calculations or official references. If the answer contains one important wrong fact, the response should not be treated as fully passed.
71. How would you score an answer that is correct but does not answer the user’s question?
Score correctness and relevance separately. If the user asks for a refund deadline and receives a correct explanation of shipping times, the facts may be accurate but the task has failed. Write the user’s required outcome before testing, then compare the answer with that outcome. Record an off-topic failure rather than rewarding it for containing true information. This helps the team fix intent handling without confusing it with factual accuracy.
72. What would you do when an answer misses an important exception?
List the mandatory points and exceptions in the expected result. If a policy allows a return only for unopened products, an answer that leaves out unopened may mislead the user. Mark the missing condition and assess its impact, even if the rest of the answer is correct. Use a rubric that makes critical omissions a failure. Add a follow-up question that checks whether the system understands the exception rather than merely repeating the main rule.
73. How do you test AI response clarity?
Clarity testing checks whether the answer is easy to understand. I would look for confusing wording, long sentences, missing steps and technical terms without explanation. The answer should help the target user take the next action confidently.
74. How do you test a chatbot that must change its tone without changing facts?
Use paired prompts asking for the same supported information in different styles, such as a short customer reply and a formal internal explanation. The wording may change, but numbers, conditions, caveats and sources should remain consistent. Compare those fixed facts separately from the tone rubric. This catches a common problem: simplifying the answer so much that an important restriction disappears.
75. How do you measure over-refusal as well as unsafe answers?
Include both disallowed requests and harmless requests that use similar words. Record unsafe compliance and unnecessary refusal as separate outcomes. For example, a defensive security explanation may be allowed while an unauthorized action is not. Judge against the product’s documented policy and intended users. A system that refuses everything can look safe on a narrow test but still fail its main purpose.
AI Testing Questions for Manual Testers
76. Can manual testers move into AI Testing?
Yes, manual testers can move into AI Testing because many AI defects need human judgment. Manual testing skills like scenario design, exploratory testing and defect reporting are very useful. The tester must add prompt testing, AI output evaluation and basic automation awareness.
77. What should manual testers learn first for AI Testing?
Manual testers should first learn how AI features work at a user level. Then they should practice prompt test cases, expected answer design, hallucination checking, API basics and bug reporting for AI outputs. They do not need advanced data science to begin.
78. Is manual testing still useful in AI Testing?
Manual testing is still useful because AI output quality cannot always be verified by scripts. A human tester can judge meaning, tone, safety, source support and business impact. Automation helps with repetition, but manual review is still important.
79. How can a manual tester run a useful exploratory AI testing session?
Start with a focused mission, such as testing a support bot’s handling of missing order details. Set a time limit, use synthetic accounts and vary one condition at a time. Save unexpected responses with the conversation context. At the end, group observations into defects, questions about requirements and useful future regression tests. Exploratory testing complements a fixed prompt set by finding behaviours that the prepared cases did not anticipate.
80. What evidence should a fresher include in a chatbot testing portfolio?
Include a short project scope, a test-case sheet, approved source material, examples of actual responses and a defect report. Show at least one failed case and explain how you decided it failed. Redact personal data and label simulated examples clearly. A smaller project with traceable evidence is more convincing than a long list of tools. The practice sheets below provide a starting format, not a claim of work already completed.
AI Testing Questions for Automation Testers
81. How can automation testers move into AI Testing?
Automation testers can move into AI Testing by adding AI output validation to their existing skills. They already understand scripts, APIs and regression. They should learn prompt testing, response scoring, RAG basics and how to handle non-deterministic outputs.
82. What should automation testers learn for AI Testing?
Automation testers should learn API testing for model calls, Playwright for UI flows, JSON response checks, prompt datasets, evaluation rubrics and test reporting. They should also understand where automation should stop and human review should begin.
83. Can we automate AI Testing fully?
No, AI Testing cannot be fully automated in most real projects. Automation can repeat prompts, check flows and detect obvious failures. Human or rubric-based review is still needed for meaning, factual correctness, tone, bias and complex safety decisions.
84. How do you keep a Playwright test from passing before the AI response finishes?
Use a stable completion signal defined by the application, such as a finished-status indicator or an enabled follow-up control. A visible response container may contain only a partial stream. Avoid relying on an arbitrary sleep that can fail under slow conditions. After completion, check the required UI state and collect the answer for evaluation. Also test cancellation and timeout states so a hung request cannot be mistaken for a successful result.
85. How would you test a structured JSON response from an AI endpoint?
Parse the response and validate its schema, required fields, data types and allowed values. Test malformed JSON, missing fields, unexpected fields and conflicting values. Then validate business rules: syntactically valid JSON can still contain a wrong amount or an unauthorized action. Keep schema validation separate from factual evaluation and check that invalid output is handled safely by downstream code rather than silently accepted.
AI Testing Questions About Test Cases
86. How do you write test cases for AI applications?
AI test cases should include prompt, user role, test data, expected behavior, evaluation criteria, actual response and result. For RAG systems, include source documents and expected citations. For safety cases, include what the AI must refuse or avoid.
87. What are positive test cases in AI Testing?
Positive AI test cases check expected successful behavior. Examples include asking a supported question, uploading a valid document or requesting a normal summary. The AI should answer clearly, use correct sources and follow the expected format.
88. What are negative test cases in AI Testing?
Negative AI test cases check how the system behaves with invalid, unsupported or risky input. Examples include missing files, unsupported formats, private data requests or prompt injection attempts. The system should fail safely instead of guessing or exposing data.
89. What are edge cases in AI Testing?
Edge cases are unusual but possible situations that may break AI behavior. Examples include very long prompts, mixed languages, contradictory documents, empty files, noisy OCR text and ambiguous questions. Edge cases help find defects before real users face them.
90. What are risky prompts?
Risky prompts are inputs that may lead to unsafe, private, biased, illegal or misleading output. Testers use risky prompts to verify guardrails. The AI should refuse harmful requests, protect sensitive data and give safe alternatives where possible.
AI Testing Questions About Defects
91. What are common AI defects?
Common AI defects include hallucination, irrelevant answers, missing context, wrong citations, biased responses, unsafe advice, private data exposure, poor tone, repeated answers, broken fallback and slow response. AI defects can affect trust even when the page itself works.
92. How do you classify AI defects?
AI defects can be classified by type: accuracy, relevance, completeness, safety, privacy, bias, citation, performance, usability and integration. Classification helps teams assign the issue to the right owner and understand whether it is a model, prompt, data or UI problem.
93. What is severity in AI Testing?
Severity in AI Testing means how much damage the defect can cause. A wrong medical, finance or legal answer may be high severity. A small tone issue may be lower severity. Severity depends on user impact, risk and business context.
94. What is priority in AI Testing?
Priority means how soon the defect should be fixed. A defect can be high priority if it affects many users, blocks release or damages trust. Priority is decided with product and business teams, not only by technical seriousness.
95. How do you write a good AI defect report?
A good AI defect report is clear and reproducible. It should include prompt, steps, actual response, expected behavior, source proof, screenshots, environment, severity and suggested category. For AI issues, evidence is very important because output can change later.
RAG Testing: Grounding, Citations and Source Versions
96. What is grounding in AI Testing?
Grounding means the AI answer is based on trusted source data instead of only model memory. In testing, grounding is checked by comparing the response with the retrieved document, database or approved knowledge source. Ungrounded answers create hallucination risk.
97. A citation opens correctly but supports a different claim. Does the answer pass?
No. A working link does not establish that the cited source supports the answer. Read the relevant passage and compare it with the specific claim, including dates, units and exceptions. Record the mismatch with the source location and the generated sentence. Check whether the wrong citation came from retrieval, passage-to-answer mapping or the model. Do not merge this defect with a simple broken-link report.
98. How do you test missing information in RAG?
To test missing information in RAG, ask questions that the source documents do not answer. The AI should say the information is not available or ask for more data. It should not invent an answer just to satisfy the user.
99. How do you test outdated documents in RAG?
Create clearly labelled old and current versions of a test policy, with at least one changed fact. Index them under the product’s intended versioning rules and ask a question that depends on the change. Check the retrieved version as well as the final answer. Then update or remove the source and test cache invalidation. If freshness rules are unclear, resolve that requirement first rather than assuming that the newest upload must always win.
100. How do you test document-based AI assistants?
To test document-based AI assistants, use different document formats, long files, scanned files, tables, headings and conflicting sections. The AI should answer from the uploaded document, cite relevant parts and admit when the answer is not present.
Turn RAG knowledge into interview evidence
Brolly Academy provides AI Testing training. Ask about document-assistant testing, practical projects and the support included in the current course.
AI Testing Questions About Metrics
101. What metrics can be used in AI Testing?
Choose metrics for the task and its risk. A classifier may need precision, recall and F1; a RAG assistant may need retrieval relevance, evidence support and task completion. Also track latency, cost and important safety failures. Avoid combining unlike dimensions into one unexplained score. State the dataset, units, scoring rule and limits of the result. A high average can hide serious failures in a small but important user group.
102. What is accuracy score in AI Testing?
For a fixed test set, strict accuracy is fully correct answers divided by all tested answers. If 16 of 20 answers are fully correct, strict accuracy is 80%. If partial credit is useful, define a separate rubric, such as 1 for correct, 0.5 for partly correct and 0 for wrong. Call that result a rubric score. Keep critical safety and privacy failures separate so a high average cannot hide them.
103. What is hallucination rate?
Define the unit before calculating the rate. A response-level rate is responses with at least one annotated hallucination divided by responses assessed. A claim-level rate uses unsupported or false claims divided by assessed factual claims. These measures are not interchangeable. Document the reference sources, reviewer rules and dataset. A lower rate on one benchmark does not prove better reliability on every task, especially if the system avoids answering difficult questions.
104. What is response time in AI Testing?
Response time in AI Testing means how long the AI feature takes to answer after the user sends a request. It includes model processing, retrieval, network and UI rendering. Slow response time can fail the user experience even if the answer is correct.
105. What is user feedback testing?
User feedback testing means using real or sample user reactions to improve AI quality. Feedback can show whether answers are helpful, confusing, too long, unsafe or incomplete. It should be combined with tester review because users may not catch every factual error.
AI Testing Questions About Real-Time Applications
106. How do you test a voice AI assistant?
Voice AI assistant testing checks speech recognition, pronunciation, background noise, interruptions, language switching, response delay and safe behavior. The tester should verify both what the assistant heard and what it answered, because mistakes can happen in either step.
107. How do you test an AI image generator?
Test whether the image follows the requested subject, format, size and important details. Include prompts that challenge text rendering, counts, spatial relationships and the product’s content rules. Review unwanted stereotypes and consistency across repeated attempts. Check content-rights and brand-policy requirements with the appropriate owner rather than assuming a style resemblance alone settles a legal question. Keep visual quality and policy compliance as separate evaluation categories.
108. How do you test whether a summary preserves numbers, conditions and warnings?
Mark the critical facts in the original document before reading the summary. Check numbers with their units, the people or items they refer to, policy conditions and warnings. A short summary may reasonably omit minor detail, but not a condition that changes the reader’s decision. Include tables and conflicting sections in the test set. Record each important omission or distortion with the supporting passage.
109. How do you check dependencies suggested by an AI coding assistant?
Confirm that the named package exists, is the intended package and supports the project’s language and version. Review maintenance status, known vulnerabilities, licensing requirements and installation steps through approved tooling. Test with pinned dependencies in an isolated environment. Do not install an unfamiliar package simply because the generated code imports it. Record unsupported or invented dependency suggestions separately from errors in the code’s business logic.
110. How would you test recommendations for a user with no history?
Create a new test account with no clicks or purchases and inspect the initial recommendations. Check the agreed fallback, catalog restrictions, variety and privacy rules. Compare this with an account that has explicit preferences, then clear those preferences and test again. The system should not claim to know personal interests it has never received. Judge results against the product’s defined cold-start behaviour, not your own shopping preference.
More Advanced AI Testing Interview Questions
111. How do you test AI memory in a chatbot?
AI memory testing checks whether the chatbot remembers allowed information and forgets what it should not keep. I would test saved preferences, deleted preferences, session boundaries and private details. Good memory improves experience but must respect privacy rules.
112. How do you test session handling in AI applications?
Session handling testing checks whether user context stays inside the correct session. I would test logout, refresh, multiple tabs, expired sessions and two users on the same device. The AI must not mix one user’s chat or files with another user’s session.
113. What is prompt leakage?
Prompt leakage means hidden system instructions, developer prompts or internal rules appear in the AI response. It is risky because attackers can learn how to bypass protections. Testers check whether the AI refuses requests to reveal hidden instructions.
114. How do you distinguish prompt leakage from sensitive-data leakage?
Prompt leakage exposes internal instructions; sensitive-data leakage exposes protected records or secrets. They may occur together, but the evidence and impact differ. Use synthetic markers and test records to identify which boundary failed. Check visible responses, citations, logs and tool output. Keep real credentials out of prompts, and do not assume that hiding the system prompt is a substitute for enforcing authorization in application code.
115. What is data leakage in AI Testing?
Data leakage in AI Testing means private, confidential or unauthorized data is exposed through prompts, responses, logs, citations or integrations. Testers check access control, masking, uploaded files, conversation history and whether one user can see another user’s information.
116. How do you test role-based access in AI systems?
Role-based access testing checks whether the AI respects each user’s permissions. I would test admin, employee, customer and guest roles with the same question. The AI should answer only with information that the current role is allowed to access.
117. How do you test document permissions after a user’s access is revoked?
Give a test user access to a synthetic document, confirm the permitted answer, then revoke that access through the normal workflow. Repeat the query in an existing conversation, a new session and any cached retrieval path. Check that the application does not expose protected snippets or citations after revocation according to its documented policy. An answer generated before access changed and a fresh retrieval need clear, separately tested handling.
118. How do you test scanned PDF documents in AI assistants?
Scanned PDF testing checks OCR quality. I would upload clear scans, blurry scans, rotated pages, handwritten text and mixed-language scans. The AI should tell the user when text extraction is poor instead of inventing missing content.
119. How do you test tables inside documents?
Tables inside documents need special testing because AI may mix rows and columns. I would ask questions about totals, dates, comparisons and specific row values. The answer should preserve table relationships and not join values from different rows.
120. How do you test whether important evidence is lost in a long document?
Place labelled facts at the beginning, middle and end of a representative document. Ask questions that need one location and questions that combine locations. Include tables and repeated headings. Inspect extracted text and retrieved passages before blaming the model. If the application truncates input, test whether it warns the user or retrieves the needed evidence through its intended workflow. Do not assume that accepting an upload means every page was processed.
AI Testing Questions on Test Strategy
121. How do you create an AI Testing strategy?
An AI Testing strategy should define scope, risks, test data, prompt sets, evaluation rubrics, automation plan, human review process, release criteria and monitoring. It should explain how the team will test both software behavior and AI response quality.
122. What should be included in an AI test plan?
An AI test plan should include feature scope, user roles, prompt categories, source documents, expected behavior, tools, environments, metrics, safety checks, regression set and reporting process. It should also define who reviews uncertain AI responses.
123. How do you decide test coverage for AI applications?
AI test coverage is decided by user intent, business risk, data sources, prompt types, roles, languages, integrations and failure impact. High-risk areas like privacy, payments, healthcare or HR need deeper coverage than low-risk informational features.
124. How do you prioritize AI test cases?
Prioritize AI test cases by risk, frequency and user impact. Start with core user journeys, sensitive data, unsafe outputs, wrong-answer risk and release-blocking flows. Lower-risk formatting or wording checks can come after critical behavior is stable.
125. How do you test AI features before release?
Before release, test the main prompts, edge cases, safety cases, RAG sources, role permissions, latency, fallback behavior and regression set. Also review logs and user-facing messages. Release should happen only when major quality and safety risks are acceptable.
126. What is acceptance criteria in AI Testing?
Acceptance criteria in AI Testing define what a good AI response must satisfy. Criteria may include correct facts, relevant answer, required format, safe refusal, source citation, maximum response time and no private data exposure.
127. How do you handle AI answers that are partially correct?
Apply the agreed rubric and the consequence of the missing or wrong part. A response can earn partial credit in a learning exercise, but an omitted condition may make a business decision unsafe. Record which criteria passed and failed. Have a reviewer adjudicate unclear cases and update the rubric when needed. Avoid changing the scoring rules after seeing a disappointing result, because that makes comparisons between versions unreliable.
128. How do you test AI output for business rules?
To test business rules, prepare scenarios based on company policy, pricing rules, eligibility rules or process rules. The AI should follow the approved business rule even if the user asks for an exception. Wrong business-rule answers can create real operational problems.
129. How do you test AI systems after model updates?
Pin or record the old and new model versions and keep other settings stable for comparison. Rerun the benchmark, critical safety cases, tool workflows and fixed output checks. Compare individual failures, not just averages, and record latency and cost changes. Repeat variable cases. A model upgrade can improve general answers while breaking a particular format or workflow, so decide rollout and rollback criteria before deployment.
130. How do you test after prompt changes?
After prompt changes, test the exact areas the prompt controls plus regression cases. Prompt edits can improve one scenario and break another. I would check format, tone, refusal behavior, source use and examples that previously passed.
AI Testing Questions on Performance and Reliability
131. How do you test AI application performance?
AI application performance testing checks latency, throughput, timeout behavior, response streaming and user experience under load. I would test short prompts, long prompts, document questions and repeated requests. Performance matters because users lose trust when AI feels slow.
132. What is timeout testing in AI applications?
Timeout testing checks what happens when the AI or retrieval service takes too long. The system should show a clear message, avoid duplicate submissions and allow the user to retry safely. It should not hang forever or lose the user’s input.
133. How do you test AI error handling?
AI error handling testing checks whether the system responds clearly when model calls, retrieval, file parsing or APIs fail. The user should see a helpful message, not raw technical errors. Logs should capture enough detail for the team to debug.
134. How do you test retry behavior?
Retry behavior testing verifies what happens after temporary failures. The system should retry only when safe, avoid duplicate charges or duplicate tickets and show clear status. Retrying blindly can create cost, data or user-experience problems.
135. How do you test AI rate limiting?
Rate limiting testing checks whether the application handles too many requests safely. Users should see a clear limit message and retry guidance. The system should protect backend services from abuse without exposing internal limits or technical details unnecessarily.
136. How do you test AI application logs?
AI application logs should be tested for usefulness and privacy. Logs should capture request IDs, errors, latency and evaluation details, but they should not store sensitive user data in an unsafe way. Good logs help investigate AI defects later.
137. How do you test AI monitoring?
AI monitoring testing checks whether dashboards and alerts capture important failures after release. I would verify alerts for high latency, API failures, rising hallucination reports, retrieval errors and safety events. Monitoring should help the team react quickly.
138. How should a RAG assistant behave when its retrieval service is unavailable?
Test the product’s documented failure mode by simulating the dependency error in a controlled environment. If answers must be source-grounded, the assistant should not silently answer from model memory as if retrieval succeeded. It may explain the limitation, retry safely or hand off. Verify the user message, error classification and recovery when the service returns. Keep this separate from a valid search that simply finds no answer.
139. How do you test AI availability?
AI availability testing checks whether the AI feature works when users need it. I would test model downtime, network issues, retrieval service failure and high traffic. The application should degrade gracefully and explain the situation to users.
140. How do you test AI cost-related behavior?
Cost-related testing checks whether the AI system avoids unnecessary expensive calls. I would test long prompts, repeated clicks, retries, file uploads and duplicate submissions. The application should control token usage, prevent loops and show limits where needed.
AI Testing Questions on Security
141. What is prompt injection?
Prompt injection is an attack where a user tries to make the AI ignore its rules or reveal protected information. It can be direct in the chat or hidden inside documents and web pages. Testing prompt injection is part of AI security testing.
142. How do you test indirect prompt injection?
Indirect prompt injection happens when malicious instructions are hidden inside a document, email, web page or retrieved content. The user may not type the attack directly. Testers check whether the AI follows the trusted system rules instead of hidden external instructions.
143. What is jailbreaking in AI?
Jailbreaking in AI means tricking the model into ignoring safety rules. Attackers may use roleplay, emotional pressure, coded language or step-by-step bypasses. The tester’s job is to check whether guardrails still work under these attempts.
144. How do you test AI privacy risk?
AI privacy risk testing checks whether the system exposes personal, company or customer data. I would test prompts asking for private records, uploaded documents, logs and cross-user data. The AI should follow privacy rules and refuse unauthorized requests.
145. How do you test AI authentication issues?
Authentication testing checks whether only valid users can access the AI feature. I would test expired sessions, missing tokens, invalid tokens and logged-out requests. The AI should not answer protected questions unless the user is properly signed in.
146. How do you test authorization in AI chatbots?
Authorization testing checks whether signed-in users can access only what they are allowed to see. I would test different roles asking the same question. The chatbot should not reveal admin-only, HR-only or customer-specific data to the wrong role.
147. What is sensitive data masking?
Sensitive data masking means hiding or replacing private information such as phone numbers, emails, IDs, account numbers or health details. Testers verify that AI responses, logs and screenshots do not expose sensitive data unnecessarily.
148. How do you test AI-generated links?
AI-generated link testing checks whether links are real, safe and relevant. The AI should not invent URLs or send users to harmful pages. I would click or validate generated links and check whether they match the user’s request.
149. Can an AI citation leak information even when the answer hides it?
Yes. A title, snippet, filename or URL can reveal information the user is not authorized to see. Use synthetic restricted documents and check both the answer and its citation metadata for each test role. Directly opening a citation must also enforce permissions. Redacting the response text alone is not enough when a search result or generated link exposes the protected document’s existence or contents.
150. How do you test AI compliance requirements?
Compliance testing checks whether the AI follows rules required by the industry or company. Examples include privacy, audit logs, consent, disclaimers, record keeping and content restrictions. Testers should work with business and legal teams for exact compliance criteria.
Plan your next learning step
Discuss your QA experience and learning goals with Brolly Academy. Review the AI Testing syllabus and confirm course mode, fees and batch availability before enrolling.
AI Testing Questions on Domain Use Cases
151. How do you test AI in healthcare applications?
Healthcare AI testing must be strict because wrong answers can affect patient safety. I would test accuracy, disclaimers, emergency escalation, privacy and unsupported medical claims. The AI should not replace a doctor or give dangerous instructions.
152. How do you test AI in finance applications?
Finance AI testing checks calculations, risk warnings, eligibility, privacy and compliance. I would test interest, fees, recommendations, fraud cases and sensitive account questions. The AI should avoid unsupported financial advice and protect customer data.
153. How do you test AI in education applications?
Education AI testing checks whether the AI explains concepts correctly, avoids wrong answers and supports learning. I would test grade-level language, plagiarism risk, harmful shortcuts, feedback quality and whether answers match the syllabus or source material.
154. How do you test AI in HR applications?
HR AI testing checks fairness, privacy and policy accuracy. I would test resume screening, leave policy, interview questions and employee support. The AI must not discriminate or reveal confidential employee information.
155. How do you test AI in customer support?
Customer support AI testing checks common questions, refund policy, complaint handling, escalation and tone. The AI should solve simple issues, follow company policy and transfer sensitive or unresolved cases to the right support channel.
156. How do you test AI in ecommerce?
Ecommerce AI testing checks product recommendations, search, order queries, refunds, delivery status and pricing information. The AI should not show wrong prices, invent stock availability or recommend products that break user filters.
157. How do you test AI in cybersecurity tools?
AI in cybersecurity tools must be tested for alert accuracy, false positives, false negatives and safe explanations. The tool should help analysts without leaking sensitive incident data or giving attackers useful instructions.
158. How do you test AI in legal applications?
Legal AI testing checks source accuracy, jurisdiction, disclaimers and citation quality. The AI should not invent laws or present legal advice as final truth. It should refer users to qualified professionals when the situation is serious.
159. How do you test an AI coding assistant’s change across an existing repository?
Review the proposed change against the requested scope and the project’s existing conventions. Run relevant unit, integration and regression tests, inspect changed interfaces and check error paths. Confirm that unrelated files were not modified and required tests were not weakened merely to make the suite pass. Use a disposable branch or isolated workspace. Report what was tested and any remaining gaps instead of equating a successful build with a correct change.
160. How do you test AI in marketing tools?
Marketing AI testing checks brand voice, factual claims, compliance, originality and audience fit. The AI should not create misleading offers, unsupported claims or off-brand content. Testers should also check whether generated copy matches campaign rules.
Applied AI Testing: Metrics, Tools and Release Decisions
161. When is precision more useful than recall, and when does recall matter more?
Precision asks how many positive predictions were correct; recall asks how many actual positives were found. Suppose a test has 8 true positives, 2 false positives and 4 false negatives: precision is 8/10 or 80%, while recall is 8/12 or about 66.7%. The cost of false alarms versus missed cases determines the trade-off. Do not choose a threshold from accuracy alone. See the official model evaluation metrics reference.
162. How would you detect leakage between training and test datasets?
Check for duplicate records, overlapping users or time periods, and fields that reveal the target. Split the data before learning preprocessing steps; fit those steps using training data only. Keep the final test set out of model and prompt tuning. Leakage can make offline results look excellent while the system fails on new cases. The scikit-learn data leakage guidance explains this distinction. Dataset leakage is different from exposing private user information.
163. How would you validate AI-based visual regression testing?
Use controlled screenshots containing both genuine visual defects and harmless changes. Check whether the system detects missing controls, clipped labels and wrong states without treating every small rendering difference as a bug. Review thresholds with representative devices and accessible states. Keep known defects as regression cases, and inspect false negatives manually. A visual similarity score does not prove that the page is functional or that keyboard users can complete the task.
164. A self-healing locator selects the wrong button. How do you detect it?
Assert the intended business result as well as the click. If a locator silently changes from Save to Delete, finding an element is not a successful repair. Record the original and replacement targets, accessible names and the resulting application state. Require review for sensitive workflows and test deliberate layout changes. Self-healing should reduce maintenance without hiding defects or changing what the test was meant to verify.
165. How do you check whether an LLM judge agrees with human reviewers?
Prepare representative responses with human-reviewed labels and a clear rubric. Compare the judge’s decisions with those labels, inspect disagreements and test whether changing answer order or length changes the score unfairly. Version the judge model and instructions. Use more than one reviewer for ambiguous high-risk cases where practical. A judge is an evaluation aid, not an unquestionable source of truth, and its own failures need monitoring.
166. Retrieval scores improve, but the final answer gets worse. How do you investigate?
Inspect the passages supplied to the generator, their ordering and the amount of extra context. Better retrieval scores may still introduce conflicting sources, irrelevant detail or a context-length problem. Compare answers using a fixed set of approved passages to isolate generation behaviour. Check prompt assembly and citation mapping, then evaluate the end-to-end task. Optimizing one component metric is useful only when it supports the user’s actual outcome.
167. How would you test an AI agent’s tool permissions and approval boundaries?
Use a sandbox with test-only records and separate roles. Ask for tasks that need permitted tools, forbidden tools and explicit approval. Inspect the tool calls and resulting state, not just the assistant’s message. A draft-email task should not send an email without the required authorization. Test failed actions, retries and loop limits too. Permissions should be enforced outside the model so a persuasive prompt cannot grant extra access.
168. How can metamorphic testing help when you do not know every correct output?
Define a relationship that should remain true when the input changes in a controlled way. For example, reordering independent entries in a list should not change their total, and an equivalent paraphrase should preserve a policy deadline. Run both inputs and check the relationship. A violation can reveal a defect even without a complete answer oracle. The relationship must be valid for the task; not every wording change is meaning-preserving.
169. How do you measure time to first token separately from total response time?
Record when the request starts, when the first displayed token arrives and when the response completes. The first interval describes perceived responsiveness; the complete interval describes total waiting time. Test representative prompt lengths and concurrency, and report units and percentiles such as p50 and p95. A fast first token can hide a very slow full answer. Also check that streaming failures are clearly distinguished from completed responses.
170. What evidence would make you approve or block an AI release?
Use agreed acceptance criteria covering the main user tasks, critical safety and privacy checks, regression results and operational limits. For a teaching example, a team might require all mandatory permission tests to pass and task-quality results to meet a pre-agreed threshold. These are project decisions, not universal numbers. Review failures by risk, document remaining limitations and identify the decision owner. An average score alone should not override a release-blocking defect.
AI Testing Interview Practice Sheets
Use these free practice sheets to record your own testing work. The filled rows are hypothetical teaching examples, not student results or a claim about a production system. Replace them with your observations and use synthetic or properly authorized data.
The interview rubric below is a practice aid, not an employer’s official scorecard or a certification pass mark. Score each criterion from 0 to 2, then explain one improvement you would make to your answer.
| Criterion | 0 points | 1 point | 2 points |
|---|---|---|---|
| Risk | No risk identified | General risk named | Specific user impact explained |
| Test data | No example | One happy path | Normal and meaningful failure cases |
| Expected result | Only says it should work | Partly observable | Clear, checkable criteria |
| Evidence | No evidence plan | A screenshot only | Inputs, sources, configuration and results |
| Judgement | Unsupported certainty | Mentions a limitation | Explains trade-offs and review ownership |
Official References for Interview Preparation
Use these official resources to check terminology and explore the topics in more depth. Certification eligibility, exams and tool features can change; consult the issuing organization for current details.
Final Interview Tip
A useful interview answer connects the concept to a testable requirement. Explain the risk, choose representative inputs, state the expected result and show how you would investigate a failure. Be clear about what you have actually practised and what still needs review.
Need structured preparation? Explore Brolly Academy’s AI Testing Training in Hyderabad or ask about course details on WhatsApp. Confirm current prerequisites, projects, schedule and fees with the team.
