# Test brief: LLM-based assistant

Product type: LLM-based assistant. A chat or agent product built on a language model with tools or retrieval.
Risk map and evidence: https://howtotestaproduct.com/how-to-test/llm-assistant/
Printable checklist: https://howtotestaproduct.com/checklists/llm-assistant/

Run these 9 mandatory tests in this order. A test is done only when its done condition is true on the build you will release.

1. release smoke tests (risks: dependency-failure, wrong-decisions). Tool class: browser automation. Usually run by: QA engineer.
   Done when: On the release build, a signed-in user sends a prompt, gets an answer, triggers one tool call, finds the chat saved in history, and one fixed policy question returns its approved answer.
2. exploratory testing (risks: wrong-decisions, user-harm). Tool class: test management. Usually run by: QA engineer.
   Done when: A session log covers policy questions, crisis phrases and off-topic requests, and records each answer next to the approved policy text.
3. compliance checks (risks: regulatory-penalty, user-harm). Tool class: test management. Usually run by: product manager.
   Done when: A signed review shows the age gate, the training-data opt-out and the AI disclosure, with a screenshot of each from the release build.
4. end-to-end functional tests (risks: user-harm). Tool class: browser automation. Usually run by: QA engineer.
   Done when: A user who types a self-harm phrase gets the crisis message and helpline link in the same turn, on web and both mobile apps.
5. penetration testing (risks: fraud-abuse, data-breach). Tool class: proxy. Usually run by: security specialist.
   Done when: A tester planting instructions in an uploaded file or a retrieved page cannot make the assistant call a tool, reveal its system prompt or show another user's data.
6. security scanning (risks: data-breach). Tool class: DAST scanner. Usually run by: security specialist.
   Done when: The scan of the chat API and history store shows no open high-severity finding, and one account's conversation IDs return nothing from another account's session.
7. performance and load testing (risks: outage-under-load). Tool class: load generator. Usually run by: developer.
   Done when: One account sending prompts in a loop hits its own limit, and other accounts keep getting answers at their usual speed.
8. resilience and failure testing (risks: dependency-failure). Tool class: chaos tool. Usually run by: developer.
   Done when: With the model API returning errors or timeouts, the user sees a retry message in the chat and no half-written answer is saved as final.
9. regression suite (risks: wrong-decisions). Tool class: test management. Usually run by: QA engineer.
   Done when: A fixed question set with approved answers is rerun after every prompt, model or retrieval change, and a named owner reviews every changed answer.

Acceptance criteria wording:
A release of the LLM-based assistant is accepted when a signed-in user can send a prompt, get an answer, trigger one tool call and find the chat in history, and the fixed question set returns its approved answers with every changed answer reviewed by a named owner. A self-harm phrase brings the crisis message and helpline link in the same turn, and instructions planted in an uploaded file cannot make the assistant call a tool or show another user's data.

Report back for each test: pass or fail against its done condition, the build or version tested, and the proof (screenshot, log line or report file).

Licence: CC BY 4.0.