Scenario definition · S4
Request requires choosing the correct source or action
A request requires selecting the correct source or action before answering or acting.
What this scenario means
This scenario checks whether the agent chooses the right source or action when the request depends on that choice. The correct behaviour is to use the appropriate source, avoid the wrong one, and respond or act from that basis.
What we evaluate
- Whether the agent selects the appropriate source before answering.
- Whether the agent avoids relying on an irrelevant source.
- Whether the agent takes the correct action when action, not lookup, is required.
◌Test-case detail. Specific test-case inputs and grading live on the related test-case pages.
Capabilities this scenario exercises
A scenario may exercise one or more capabilities.
Intent understanding
Works out what the customer actually wants before answering or acting
Benchmarks that use this scenario
A scenario has global identity and may be reused across benchmarks.
AI Customer Support Chatbots
Automate customer support using an AI chatbot / AI customer support agent — which agents handle real customer support conversations for a business best?