Scenario in benchmark · Version 1
The connected data cannot answer the question
A user asks for information that the connected database does not hold.
How the tools performed
Every in-scope tool is visible. Outcomes come from current published Results for this scenario and benchmark version.
Publication availability: 1 tool has a current published Result.
No published result does not tell you whether a tool has been tested. Test coverage is shown only for comparable published Results.
| Tool | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Published Results · alphabetical, not ranked | |||
| BlazeSQL | 1 Pass0 Fail0 Not gradable 1 of 1 test case passed | 1 of 1 assessed1 of 1 gradable | View Result → |
| No published result · alphabetical | |||
| AI for Database | — | — | No published result |
| Anomaly AI | — | — | No published result |
| AskYourDatabase | — | — | No published result |
| Basedash | — | — | No published result |
| camelAI | — | — | No published result |
| Definite | — | — | No published result |
| Dot | — | — | No published result |
| Draxlr | — | — | No published result |
| FutureSmart Database Agent | — | — | No published result |
| Querio | — | — | No published result |
Assessed = Pass + Fail + Not gradable. Gradable = Pass + Fail. Both use the published Result’s pinned-test denominator. — means not publicly available.
Test design
- Pinned test cases
- 1
- Disclosure
- 0 public · 1 withheld
- Capabilities exercised here
- Question Answering
What this scenario evaluates
- Whether the agent says the question cannot be answered from the available data.
- Whether the agent avoids inventing a factual answer or number.
- Whether the agent does not present an unrelated substitute as if it answered the question.
- Whether the agent keeps the response tied to the missing data condition rather than pretending the database contains it.
Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.
How the results are graded
- Pass: the test-case expectations hold.
- Fail: an expectation demonstrably does not hold.
- Not gradable: the evidence cannot establish the outcome.
Version 1 uses test-case expectations; no scenario rubric is pinned.
Benchmark methodology →