Misplaces a bank-statement transaction across dates: a row that belongs to 18 Jun is attached to the 19 Jun record after merging, so date association is unreliable.
What was measured
Extraction Accuracy
Are field values correct, complete, and free of OCR or parsing errors, including numerical precision on financial fields?
decisive for this rankingtransformation
Correct field values are the core of the job; wrong or incomplete extraction means the tool failed to retrieve the structured data from the document. (3 of 3 judges)
What was given, what came back
Test input: Bank Statement PDF · pdf · group: business-document-extraction
Input — what we sent
A four-page bank statement PDF with dense transaction tables, balance-forward bridges, account metadata, rewards data, and disclaimer text. It was used to stress schema-driven extraction, multi-page continuity, row completeness, and financial numerical accuracy.
Why this input is hard
- · Table extraction across 50+ transaction rows
- · Multi-page continuity with balance-forward bridges
- · Parsing structured account metadata alongside unstructured transaction descriptions
- · Numerical accuracy for deposits, withdrawals, running balances, and summaries
- · Extraction of nested rewards and disclaimer sections
Output — unretouched
Also checked on this input — same tool, 5 other criteria
Schema Adherence✓ WorkedCan emit the bank-statement extraction as nested schema-shaped JSON, with separate metadata, account_holder/account, branch, transactions, summary, rewards, and disclaimers objects instead of flat OCR text.Semantic Field Enrichment◐ MixedClassifies bank-statement transaction_type inconsistently on merged rows: the 28 Jun record is labeled Deposit even though the visible row shows only a 399 withdrawal amount, and another merged row is labeled Deposit/Withdrawal.Semantic Field Enrichment✗ FailedLeaves the derived bank-statement transaction_id field null, including on the 18 Jun ATM withdrawal that visibly contains the embedded ID 916910098754.Table & Record Completeness⚠ StruggledMerges adjacent bank-statement rows into one record; the 21 Jun crop shows two separate transactions, but the extracted description collapses them into one long UPI/merchant string.Table & Record Completeness✗ FailedOver-segments the bank-statement table: the report says 54 transaction records were extracted where 51 were expected, a +3 overcount caused by continuation rows and split records.
Provenance
- Observation
- 3f57949b-b930-488c-9c76-b9ebc3b73d51
- Evidence run
- a061b9e7-a9c5-443d-a171-b296aaf51b8c
- Study
- Extract and query structured data from documents using natural language
- Research task
- 86b9y25e5
- Tested at
- not recorded
- Source
- first-party
- Evidence state
- verified
- Proof shown
- input + output shown
- Cost / latency
- not captured
- Repeat run
- not captured
- Tester
- not captured
The last three rows are honest blanks, not placeholders — our capture has no field for them yet.
Query this
get_evidence({
tool: "datalab",
scenario: "business-document-extraction"
})MCP · mcp.aidemos.com/api/mcp
Free with attribution.
Same input, same check — 8 other tools
measured on Extraction Accuracy
Docsumo◐ MixedIn the rewards scheme table, zero-value cells render as blank rather than as 0, so the extractor preserves the row structure but loses explicit zero values.Extend AI✗ FailedThe summary aggregation is incorrect: `total_transactions` is 49 in the output, while the report says the expected count after exclusions is 40.Landing AI✓ WorkedPreserves statement metadata and balances with exact values, including bank_name "Standard Chartered", statement_date "16 Jul 2019", currency "INR", opening_balance 114453.65, and closing_balance 116149.46.LlamaParse⚠ StruggledThe tool leaves value_date blank on some bank transactions even though the source statement contains value dates, so transaction metadata is only partially accurate.Nanonets✓ WorkedCorrectly extracts high-level statement values, including account number 42710540422 and total deposits 70986.83, rather than corrupting the header fields.Reducto✓ WorkedCorrectly extracts the bank statement currency as INR and attaches high-confidence citation metadata, with extract confidence shown as 0.984.Retab✓ WorkedDocsumo extracts the bank statement into structured customer, branch, and summary fields, including the account holder name, address, totals, and closing balance, and also supports QA over the document.Unstract✗ FailedUnderstates the derived bank-summary count: summary.total_transactions is reported as 43 even though the PDF contains 51 transactions, a 16% undercount.
This evidence is published in
Real inputs and real outputs, no retouching · every cell queryable via API & MCP · aidemos.com