Scenario in benchmark · Version 1
Correction changes future behavior where the product claims learning
A prior correction is expected to change how the agent answers later when the product claims to learn from feedback.
How the tools performed
Every in-scope tool is visible. Outcomes come from current published Results for this scenario and benchmark version.
No published results for this scenario yet.
No published result does not tell you whether a tool has been tested. Test coverage is shown only for comparable published Results.
| Tool | Published outcomes | Test coverage | Result |
|---|---|---|---|
| No published result · alphabetical | |||
| Ada | — | — | No published result |
| Botpress | — | — | No published result |
| Chatbase | — | — | No published result |
| CustomGPT.ai | — | — | No published result |
| Decagon | — | — | No published result |
| Freshdesk Freddy | — | — | No published result |
| FS Agent (DIY control) | — | — | No published result |
| Gorgias | — | — | No published result |
| Help Scout | — | — | No published result |
| Intercom Fin | — | — | No published result |
| Kommunicate | — | — | No published result |
| Tidio Lyro | — | — | No published result |
| Zendesk AI | — | — | No published result |
Assessed = Pass + Fail + Not gradable. Gradable = Pass + Fail. Both use the published Result’s pinned-test denominator. — means not publicly available.
Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Feedback and learning
What this scenario evaluates
- Whether the system accepts a correction after an earlier wrong answer.
- Whether a later interaction reflects the correction rather than the earlier error.
- Whether the earlier wrong behaviour is not simply repeated once feedback has been given.
- Whether the behaviour change appears in a later interaction when the product claims to learn from feedback.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the results are graded
- Pass: the test-case expectations hold.
- Fail: an expectation demonstrably does not hold.
- Not gradable: the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned.
Benchmark methodology →