> For the complete documentation index, see [llms.txt](https://docs.usepylon.com/pylon-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.usepylon.com/pylon-docs/agents/support-agent/monitor-and-iterate/quality-assurance.md).
# Quality Assurance
Quality assurance helps you trust your agent's replies before customers see them, and keep checking them once it's live. Pylon gives you three ways to check quality:
* **Before launch**: Test and backtest your agent on sample and past issues. See Test and Backtest.
* **During rollout**: Run the agent in **QA mode** so your team reviews its replies before customers see them.
* **After launch**: Use **Agent QA** to score every reply on live issues automatically.
#### Review replies in QA mode
In QA mode, the agent posts its first reply to each new issue as an internal note instead of sending it to the customer. Your team can read what the agent would have said and decide when it's ready to reply on its own.
To turn on QA mode, open your agent, go to **General**, and set **Agent mode** to **QA mode**. Then choose which issues get a QA note:
* **All new issues**
* **New issues that meet certain conditions**
* **Issues that would have been assigned to the agent**
When you're happy with the replies, switch **Agent mode** to **Full support**.
{% hint style="warning" %}
Tools you've enabled for the agent still run in QA mode, including MCP tools. They can make changes in Pylon and connected systems.
{% endhint %}
#### Score replies with Agent QA
Agent QA grades your agent's replies on resolved issues against a rubric you define. Go to **Settings > AI controls > Agent QA** to set it up.
**Metrics**
A metric checks one behavior and is graded **Pass**, **Fail**, or **N/A**. Pylon includes **Answer accuracy** by default, and you can add your own, such as relevance, empathy, or policy compliance.
To create a metric, open the **Metrics** tab and click **Create Metric**. Give it a name and **pass conditions**: statements that must be true for the reply to pass. Add examples if they help.
Mark a metric as **Critical** if failing it should always count as a serious problem, such as sharing wrong pricing or breaking a policy. Critical fails are tracked separately so they stand out.
{% hint style="success" %}
**Tip:** Good metrics can be answered with true or false. Focus each one on a single clear behavior, and define exactly what counts as a pass.
{% endhint %}
**Rubrics**
A rubric is a set of metrics that grades an agent. To create one, open the **Rubrics** tab and click **Create Rubric**, then:
1. Name the rubric and describe what it check
2. Under **Apply to agents**, select the agents that use it.
3. Under **Metrics**, add the metrics it checks.
4. Under **Score distribution**, set how much each metric counts toward the score. Weights must total 100%.
An agent's score is its weighted pass rate across the rubric's metrics. N/A results don't count toward the score.
**Review results**
* **Issues** shows each graded issue with its pass/fail result. Open an issue to see each metric's grade and the reasoning behind it. From the same view, teammates who can edit issues can rate the agent's replies with **Good response** or **Bad response** and leave a note. These ratings are the same per-message feedback as on the issue page, so they show in both places.
* **Agents** shows each agent's pass rate and critical fails. These also appear on the Support agents list.
* **Metrics** shows how often each metric passes
If you disagree with a grade, change it on the issue. Your change is saved as an **override**. Pylon uses overrides to suggest better pass conditions for your metrics: click **Review Improvements** on a metric to accept or discard the suggestions.
**Grade past issues**
Agent QA grades issues as they're resolved. Before you set it up, or to regrade after changing a rubric, click **Backfill QA** and choose how many past days to include:
* **Fill missing QA**: Grades only issues that haven't been graded yet. Existing grades are kept.
* **Redo QA**: Grades every issue again. Previous grades and overrides stay in history.
#### Use QA to improve your agent
Low scores show you where to focus. Open the failed issues, find the pattern, and update the agent's guidance, skills, or resources. Then run a backtest with **Grade with Agent QA** turned on to confirm the fix before you roll it out.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://docs.usepylon.com/pylon-docs/agents/support-agent/monitor-and-iterate/quality-assurance.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
