Test
Testing makes sure you can be confident your AI Agent will behave as you want it to.
Custom Chat
Simulate an interaction with the AI and see how it handles different questions. Either choose from one of the sample questions or ask a question directly.

To test agent behavior for a specific customer or channel, you can select your desired values for Channel, Account, and Contact.
Backtest
Backtesting shows how your agent would have handled real past issues. Use it to check a new agent before you deploy it, or to confirm that a change to guidance, skills, or resources actually helped.
To start, go to Settings > AI controls > Agents and click Backtest, or click Backtest on an agent's Variations page.
Run a backtest
Click New Backtest and give it a name.
Select test issues. Choose one:
Custom selection: Filter your past issues and pick the ones to test.
Issues with bad AI responses: Start with issues where an agent's reply was rated poorly. These are the issues most worth fixing.
Existing dataset: Reuse a saved set of issues (see Datasets).
Select which agent to use. Choose up to 3 agents or variations to run against the same issues, so you can compare them side by side.
Optionally, turn on Grade with Agent QA to score every reply with the same rubric Agent QA uses on live issues.
Click Start Backtest.
Backtests never message customers or change the original issues.
Review results
For each issue, you can see what the agent replied and what it did. That includes the actions it took, whether it escalated, and its reasoning. Each result shows how the agent handled the issue:
Replied only: The agent answered without escalating.
Replied + escalated: The agent answered, then handed off to your team.
Escalated: The agent handed off to your team.
Silent escalation: The agent handed off without replying to the customer.
If you turned on Grade with Agent QA, each reply also gets a score, with pass rates and critical fails for each agent. Click Review Issues to go through results one by one. Confirm or change each grade with Pass, Fail, or N/A, and add a note explaining why.
Quality AssuranceTurn feedback into fixes
When a reply isn't right, tell Pylon what went wrong, such as incorrect information, too wordy, or wrong action. Describe what the agent should have said or done instead, and Pylon suggests updates to the agent's guidance, skills, or knowledge base articles.
After you make a change, click Re-run Backtest to check that it fixed the problem without breaking anything else.
Datasets
A dataset is a saved set of past issues you can reuse across backtests. Reusing the same set makes it easy to compare results over time. Each dataset holds up to 1 To create one, open the Datasets tab and n select issues to add. To build a dataset from issues an agent has already worked on, filter by Support agent history. That includes issues the agent handled in QA mode.
Variations
A variation is an alternate version of an agerent guidance, skills, or tools. Create avariation from the agent's Variations page, backtest it against the current version, and switch to whichever one performs better.
Last updated
Was this helpful?

