Can BlazeSQL use user-written descriptions to reach the right data when database names are cryptic?
BlazeSQL used the user’s documentation to answer from the cryptic table and column. It reported 6 returns currently awaiting manual fraud review, split across 5 approved and 1 received. The stored note was applied to returns.flg_x2, and the follow-up answer came from that documented column.
1 of 1 test case passed
Every test case in this scenario has a result.
The test case
Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above counts this row.
Documented cryptic table and columnPassEvidence
Input 1: “flg_x2 on the returns table marks whether a return was flagged for manual fraud review - true means it needs manual review before refunding.”
Input 2: “How many returns are currently flagged for manual fraud review?”
There are 6 flagged returns currently awaiting manual fraud review: • ✅ Approved (5) — flagged but already approved, refund held pending review • 📦 Received (1) — item received back, refund held pending review Would you like to dig deeper into any of these, such as the customer details or refund amounts involved?
Transcribed from tc47-blazesql-followup-result
1 SELECT
2 status,
3 COUNT(*) AS flagged_returns
4 FROM public.returns
5 WHERE
6 flg_x2 = true
7 AND status NOT IN ('refunded', 'rejected')
8 GROUP BY status
9 ORDER BY flagged_returns DESCTranscribed from tc47-blazesql-followup-result
These are scenario-level criteria. Each test case's Expected and Found are listed separately.
- Whether the agent uses the supplied documentation when resolving a later question about the cryptic names.
- Whether the answer is grounded in the documented table or column rather than in a guess from the raw name.
- Whether the same documentation still works when the later question is phrased differently.
| ✓ | Found: The follow-up used returns.flg_x2 and returned approved 5 plus received 1. |
The product accepted the note for returns.flg_x2 and used that documented column in the follow-up query. The visible query logic and counts match the required result.
Rejected rows excluded. The visible query excludes rejected rows and returns 6 flagged returns, split into approved 5 and received 1.
Configuration and setup
How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.
How this scenario is graded
How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.
Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.
- Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
- Fail: at least one expected behaviour clearly didn't happen; the row says which and quotes the tool.
- Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the row says what's missing.
Where this sits in the benchmark
This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.
| Level | Name | Scope |
|---|---|---|
| Benchmark | AI Database Agents → | 28 scenarios · 4 tools |
| Capability | Training → | |
| Scenario | The database's names are cryptic and the user documents them → | |
| Tool | BlazeSQL → |
Global scenario definition → · Global capability definition → · BlazeSQL product page →
History of this result
What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.
Act on this result
Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.
You run the same kind of test against your own setup and get the same behaviour.
Agree →Yours behaves differently. Tell us what you got, with a screenshot if you have one.
Disagree →Something here is wrong — a reference value, a transcription, a grade.
Report an issue →Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.
Vendor notice →The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.
Verify the proof files
These files support this result. Open a file to inspect the original evidence.
File fingerprints (SHA-256)
A fingerprint identifies the exact file used for this result.