A user reports a bad answer and the operator has to find it
A bad-answer report is used to locate the specific complained-about exchange for inspection.
What this scenario means
This scenario tests whether the system can take a complaint about a wrong answer and find the exact exchange that caused it. The hard part is that the report usually gives only partial context, so the system must identify one specific conversation rather than just summarising failures in bulk. A good system makes the exchange available for inspection so a reviewer can see what was asked and how it was answered.
What we evaluate
- Whether a specific complained-about exchange can be located from a bad-answer report.
- Whether the system uses the report's limited context to find one exact exchange rather than only a summary of failures.
- Whether the located exchange is available for inspection.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Analytics & Observability
Lets whoever runs the agent see what people actually asked and what actually happened — and reports numbers that match reality. ON TRIAL: aggregate analytics are S57 and S24; individual inspection is S58 and S59. Demotes if the scenarios cannot discriminate. Deliberately NOT the same capability as the Customer Support benchmark's 'Analytics and observability' (C8) — there the operator is a different person from the user; here the buyer IS the user and sees every answer (rule C-10).
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
AI Database Agents
Which AI database agent answers business questions about a live relational database correctly — in the company's own terms, and honestly when the data cannot answer?