The agent used the wrong definition and the user corrects it
A user corrects an earlier definition, and the agent must apply that correction consistently in later questions and rephrasings.
What this scenario means
This scenario tests whether the agent can update its behaviour after being given the correct company rule, then keep using that rule on later turns. It is hard because the agent must do more than repeat a one-off fix; it has to retain the correction and apply it when the same idea is asked again in different words.
What we evaluate
- Whether the agent uses the corrected definition on the next related question.
- Whether the corrected definition still holds when the later question is phrased differently.
- Whether the agent keeps the correction in force instead of reverting to the earlier wrong rule.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Training
Whether supplying company-specific information measurably improves future behaviour and generalises beyond the example given.
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
AI Database Agents
Which AI database agent answers business questions about a live relational database correctly — in the company's own terms, and honestly when the data cannot answer?