Graded 6 October 2026
Can Dify read a cleanly scanned document accurately?
Dify did not surface the scanned page as usable text in this scenario. The checked document stayed available with one stored segment that was only an image link, and retrieval missed the target document. The scanned PDF itself had no extractable text.
1 of 1 test case failed
Every test case in this scenario has a result.
Pass rate0%0 of 1 with a result
Coverage1 of 1test cases with a result
0PassNo test case passed.
1FailMissed at least one thing it was expected to do; the reason and proof are on the row.
0Not gradableEvery result here could be graded.
0UntestedEvery test case in this scenario has a result.
The pass rate is a summary. The evidence is the test case below: what we sent, what we checked, what the tool returned, and the proof.
The test case
Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above counts this row.
Fact that exists only as pixelsFailEvidence
What we sent
“What as-found bias is recorded on calibration certificate C-77413?”
InputDocument (PDF)
What the tool returned
Text99 lines
=== POST /datasets/{dataset_id}/retrieve === Request body: { "query": "What as-found bias is recorded on calibration certificate C-77413?" } Status: 200 Response body: { "query": { "content": "What as-found bias is recorded on calibration certificate C-77413?" }, "records": [ { "segment": { "id": "bf7724c1-2b89-4fe9-9cb5-09c6c46ad5b4", "position": 6, "document_id": "9ba90c1b-007a-4b5b-91fd-030f45e73bcb", "content": "Calibration\n\nEvery instrument leaves the laboratory with a signed certificate recording the as-found and as-left bias at four reference points across the working range. The recommended recalibration interval for the Tolvan A3 is 22 months from the date of the last certificate. An instrument returned early because of a suspected drift is checked as found before anything is adjusted, so that the drift can be quantified rather than merely corrected.\n\nFirmware", "sign_content": null, "answer": null, "word_count": 460, "tokens": 85, "keywords": [], "index_node_id": "8efd2764-befa-4d64-af98-b766aa1cb3a4", "index_node_hash": "0580a3b7e28d801a55bc838ea0bc0e98e06d80f3fd10dad6733ccdec698d92fc", "hit_count": 0, "enabled": true, "disabled_at": null, "disabled_by": null, "status": "completed", "created_by": "f1034bbd-04df-4592-ba22-56dac49ae8ad", "created_at": 1789754309, "indexing_at": 1789754309, "completed_at": 1789754312, "error": null, "stopped_at": null, "document": { "id": "9ba90c1b-007a-4b5b-91fd-030f45e73bcb", "data_source_type": "upload_file", "name": "A10_tolvan_a3_product_overview.md", "doc_type": null, "doc_metadata": { "region": "group" } } }, "child_chunks": [], "score": 0.54985493, "tsne_position": null, "files": [], "summary": null }, { "segment": { "id": "be655143-3a49-4b5f-a130-1ea6abfab1fb", "position": 5, "document_id": "24d8e7b7-2b95-412f-8492-390a47422461", "content": "Drift is a calibration matter, not a defect.\r\n\u2022 Any instrument whose serial number has been removed or altered.\r\n4. Making a claim\r\nA claim is made through the service desk, quoting the serial number and the delivery note. The service\r\ndesk raises a ticket, arranges return carriage and tells the customer within five working days whether the\r\nfailure is covered.", "sign_content": null, "answer": null, "word_count": 363, "tokens": 74, "keywords": [], "index_node_id": "13804d8f-76a5-4980-8544-289d3c32295b", "index_node_hash": "5e26687d25a112c0b0b1552639d9fd7a0ea98678901d405bbef4344c5f4de5a1", "hit_count": 0, "enabled": true, "disabled_at": null, "disabled_by": null, "status": "completed", "created_by": "f1034bbd-04df-4592-ba22-56dac49ae8ad", "created_at": 1789754298, "indexing_at": 1789754298, "completed_at": 1789754300, "error": null, "stopped_at": null, "document": { "id": "24d8e7b7-2b95-412f-8492-390a47422461", "data_source_type": "upload_file", "name": "A08_warranty_terms_sales_pack.pdf", "doc_type": null, "doc_metadata": { "region": "group" } } }, "child_chunks": [], "score": 0.46607465, "tsne_position": null, "files": [], "summary": null }, { "segment": { "id": "321f07e2-5765-416b-865b-b9170e8286b1", "position": 5, "document_id": "cd32e6aa-7c16-4d60-bff2-bd7250451958",
Copied from Proof 1 · Output file (text), lines 1–99 · an excerpt; the link below opens the whole file
Open the text file ↗Proof 1Output file (text)
Part of this file is printed above, under What the tool returned. Show the whole file
Text302 lines
=== POST /datasets/{dataset_id}/retrieve === Request body: { "query": "What as-found bias is recorded on calibration certificate C-77413?" } Status: 200 Response body: { "query": { "content": "What as-found bias is recorded on calibration certificate C-77413?" }, "records": [ { "segment": { "id": "bf7724c1-2b89-4fe9-9cb5-09c6c46ad5b4", "position": 6, "document_id": "9ba90c1b-007a-4b5b-91fd-030f45e73bcb", "content": "Calibration\n\nEvery instrument leaves the laboratory with a signed certificate recording the as-found and as-left bias at four reference points across the working range. The recommended recalibration interval for the Tolvan A3 is 22 months from the date of the last certificate. An instrument returned early because of a suspected drift is checked as found before anything is adjusted, so that the drift can be quantified rather than merely corrected.\n\nFirmware", "sign_content": null, "answer": null, "word_count": 460, "tokens": 85, "keywords": [], "index_node_id": "8efd2764-befa-4d64-af98-b766aa1cb3a4", "index_node_hash": "0580a3b7e28d801a55bc838ea0bc0e98e06d80f3fd10dad6733ccdec698d92fc", "hit_count": 0, "enabled": true, "disabled_at": null, "disabled_by": null, "status": "completed", "created_by": "f1034bbd-04df-4592-ba22-56dac49ae8ad", "created_at": 1789754309, "indexing_at": 1789754309, "completed_at": 1789754312, "error": null, "stopped_at": null, "document": { "id": "9ba90c1b-007a-4b5b-91fd-030f45e73bcb", "data_source_type": "upload_file", "name": "A10_tolvan_a3_product_overview.md", "doc_type": null, "doc_metadata": { "region": "group" } } }, "child_chunks": [], "score": 0.54985493, "tsne_position": null, "files": [], "summary": null }, { "segment": { "id": "be655143-3a49-4b5f-a130-1ea6abfab1fb", "position": 5, "document_id": "24d8e7b7-2b95-412f-8492-390a47422461", "content": "Drift is a calibration matter, not a defect.\r\n\u2022 Any instrument whose serial number has been removed or altered.\r\n4. Making a claim\r\nA claim is made through the service desk, quoting the serial number and the delivery note. The service\r\ndesk raises a ticket, arranges return carriage and tells the customer within five working days whether the\r\nfailure is covered.", "sign_content": null, "answer": null, "word_count": 363, "tokens": 74, "keywords": [], "index_node_id": "13804d8f-76a5-4980-8544-289d3c32295b", "index_node_hash": "5e26687d25a112c0b0b1552639d9fd7a0ea98678901d405bbef4344c5f4de5a1", "hit_count": 0, "enabled": true, "disabled_at": null, "disabled_by": null, "status": "completed", "created_by": "f1034bbd-04df-4592-ba22-56dac49ae8ad", "created_at": 1789754298, "indexing_at": 1789754298, "completed_at": 1789754300, "error": null, "stopped_at": null, "document": { "id": "24d8e7b7-2b95-412f-8492-390a47422461", "data_source_type": "upload_file", "name": "A08_warranty_terms_sales_pack.pdf", "doc_type": null, "doc_metadata": { "region": "group" } } }, "child_chunks": [], "score": 0.46607465, "tsne_position": null, "files": [], "summary": null }, { "segment": { "id": "321f07e2-5765-416b-865b-b9170e8286b1", "position": 5, "document_id": "cd32e6aa-7c16-4d60-bff2-bd7250451958", "content": "Drift is a calibration matter, not a defect.\r\n\u2022 Any instrument whose serial number has been removed or altered.\r\n4. Making a claim\r\nA claim is made through the service desk, quoting the serial number and the delivery note. The service\r\ndesk raises a ticket, arranges return carriage and tells the customer within five working days whether the\r\nfailure is covered.", "sign_content": null, "answer": null, "word_count": 363, "tokens": 74, "keywords": [], "index_node_id": "d15c7f3d-1bea-4f13-863c-b458e9f22f74", "index_node_hash": "5e26687d25a112c0b0b1552639d9fd7a0ea98678901d405bbef4344c5f4de5a1", "hit_count": 0, "enabled": true, "disabled_at": null, "disabled_by": null, "status": "completed", "created_by": "f1034bbd-04df-4592-ba22-56dac49ae8ad", "created_at": 1789754303, "indexing_at": 1789754306, "completed_at": 1789754306, "error": null, "stopped_at": null, "document": { "id": "cd32e6aa-7c16-4d60-bff2-bd7250451958", "data_source_type": "upload_file", "name": "A09_warranty_statement_service_pack.pdf", "doc_type": null, "doc_metadata": { "region": "group" } } }, "child_chunks": [], "score": 0.46607465, "tsne_position": null, "files": [], "summary": null }, { "segment": { "id": "38403981-4749-4253-8ae1-30a5b5eaf666", "position": 5, "document_id": "e04044ec-990a-4bf1-9908-34be00b3fa17", "content": "What the depot does not do\n\nThe depot does not calibrate. An instrument that needs a certificate goes to the laboratory, is booked in under its own job number, and comes back to the depot only once the certificate has been signed.\n\nCommissioning: before travelling", "sign_content": null, "answer": null, "word_count": 264, "tokens": 53, "keywords": [], "index_node_id": "137b3abe-fae4-4e77-a17c-33e93fa7db36", "index_node_hash": "13cdf7935207577a55f0bd797e858af534098afc6f667a3c4f3df2b4ca31ef04", "hit_count": 0, "enabled": true, "disabled_at": null, "disabled_by": null, "status": "completed", "created_by": "f1034bbd-04df-4592-ba22-56dac49ae8ad", "created_at": 1789752857, "indexing_at": 1789752858, "completed_at": 1789752862, "error": null, "stopped_at": null, "document": { "id": "e04044ec-990a-4bf1-9908-34be00b3fa17", "data_source_type": "upload_file", "name": "A05_depot_and_commissioning_note.md", "doc_type": null, "doc_metadata": { "region": "uk" } } }, "child_chunks": [], "score": 0.4369651, "tsne_position": null, "files": [], "summary": null } ] } === GET /datasets/{dataset_id}/documents (A03 entry) === { "id": "2539b1fb-9263-49b0-9a7b-318f850587d1", "position": 2, "data_source_type": "upload_file", "data_source_info": { "upload_file_id": "f6a118e9-8328-4089-a41a-a23ef4f735a1" }, "data_source_detail_dict": { "upload_file": { "id": "f6a118e9-8328-4089-a41a-a23ef4f735a1", "name": "A03_calibration_certificate_scan.pdf", "size": 208037, "extension": "pdf", "mime_type": "application/pdf", "created_by": "f1034bbd-04df-4592-ba22-56dac49ae8ad", "created_at": 1789752844.612517 } }, "dataset_process_rule_id": "e419c967-56c4-43cf-8bcc-f9d2cb37f58f", "name": "A03_calibration_certificate_scan.pdf", "created_from": "api", "created_by": "f1034bbd-04df-4592-ba22-56dac49ae8ad", "created_at": 1789752845, "tokens": 34, "indexing_status": "completed", "error": null, "enabled": true, "disabled_at": null, "disabled_by": null, "archived": false, "display_status": "available", "word_count": 87, "hit_count": 0, "doc_form": "text_model", "doc_metadata": [ { "id": "ea28fe4d-2949-4d40-a58e-c98b251a0ab5", "name": "region", "type": "string", "value": "uk" }, { "id": "built-in", "name": "document_name", "type": "string", "value": "A03_calibration_certificate_scan.pdf" }, { "id": "built-in", "name": "uploader", "type": "string", "value": "Dify" }, { "id": "built-in", "name": "upload_date", "type": "time", "value": "1789752845.0" }, { "id": "built-in", "name": "last_update_date", "type": "time", "value": "1789754366.953213" }, { "id": "built-in", "name": "source", "type": "string", "value": "file_upload" } ], "summary_index_status": null, "need_summary": false } === GET /datasets/{dataset_id}/documents/{document_id}/segments (A03) === { "data": [ { "id": "642ba8da-0dea-4072-99ce-0dc54174dc9f", "position": 1, "document_id": "2539b1fb-9263-49b0-9a7b-318f850587d1", "content": "", "sign_content": "", "answer": null, "word_count": 87, "tokens": 34, "keywords": null, "index_node_id": "5ead3070-24a4-4c4f-9687-3a10544acc4d", "index_node_hash": "ecfdac89ee89ba2b4a16925862d8a0604972eda10c50c0b7b5b5ce3490089a41", "hit_count": 0, "enabled": true, "disabled_at": null, "disabled_by": null, "status": "completed", "created_by": "f1034bbd-04df-4592-ba22-56dac49ae8ad", "created_at": 1789752853, "updated_at": 1789752853, "updated_by": null, "indexing_at": 1789752856, "completed_at": 1789752856, "error": null, "stopped_at": null, "child_chunks": [], "attachments": [ { "id": "516a105f-bb73-4406-9ada-ae843a284ad8", "name": "image_files/a49c83a3-8574-4949-992c-2f2e1fd66de8/3a3e9c30-3ff1-4f5e-95f1-6a03a1ce33f8.jpg", "size": 204657, "extension": "jpg", "mime_type": "image/jpeg", "source_url": "https://upload.dify.ai/files/516a105f-bb73-4406-9ada-ae843a284ad8/image-preview?timestamp=1789931632&nonce=095f19ab9062726771a44049f9c0677f&sign=V9qbeEDEf9j-VDPgQjC6Bt1BQ_qsABjkCQMYbrorKRY=" } ], "summary": null } ], "doc_form": "text_model", "total": 1, "has_more": false, "limit": 20, "page": 1 }
Expected vs. Found
| ✗ | Expected: The fact should return, or the tool should say the document yielded no text instead of presenting it as complete.Found: The fact did not return; the target document was absent from retrieval, and the document still showed as available with one image-link segment. |
Supporting proof
Proof 2Screenshot
Proof 3Screenshot
Why this result
The retrieval call missed the target document, and the stored segment contained only an image link while the document stayed marked available. The scanned PDF itself yielded no extractable text, so the miss is clear.
Also observed on this row
Image stored as attachment. The page image is stored as a JPEG attachment on the segment, so the content stays in storage instead of entering the text index.
Word count includes placeholder. The document's word count includes the image placeholder and its link, which makes the available status look more informative than the text actually is.
Tested by Vivek Tomar · evidence dated 20 September 2026
Configuration and setup
How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.
Software that produced the output
Dify
Build or version
Dify Cloud
Surface
API
Set up before the run
A scan that held the fact only in its image layer was loaded. The question asked for the wording visible on the page.
Call count
Three calls against the SS-v1 dataset
Tested
20 September 2026 · Vivek Tomar
How this scenario is graded
How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.
How results are decided
Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.
The rules
- Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
- Fail: at least one expected behaviour clearly didn't happen; the row says which and quotes the tool.
- Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the row says what's missing.
Where this sits in the benchmark
This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.
| Level | Name | Scope |
|---|---|---|
| Benchmark | Managed RAG → | 15 scenarios · 7 tools |
| Capability | Data Ingestion → | |
| Scenario | A cleanly scanned document → | |
| Tool | Dify → |
Global scenario definition → · Global capability definition → · Dify product page →
History of this result
What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.
9 October 2026First publishedManaged RAG v1
Act on this result
Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.
This matches what I see
You run the same kind of test against your own setup and get the same behaviour.
Agree →This does not match
Yours behaves differently. Tell us what you got, with a screenshot if you have one.
Disagree →Point out an issue
Something here is wrong — a reference value, a transcription, a grade.
Report an issue →We have fixed this
Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.
Vendor notice →Filed against this evidenceNothing yet. Challenges, counter-evidence and fix notices appear here with their outcome, and stay on the page after they are resolved.
Cite this result
aidemos.com/benchmarks/managed-rag/results/dify/a-cleanly-scanned-document · 0 pass · 1 fail · coverage 1/1 · graded 2026-10-06
The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.
Verify the proof files
These files support this result. Open a file to inspect the original evidence.
Input · Document (PDF) ↗application/pdf · 203 KB
Proof 2 · Screenshot ↗image/png · 279 KB
Proof 1 · Output file (text) ↗text/plain; charset=utf-8 · 10 KB
Proof 3 · Screenshot ↗image/png · 381 KB
File fingerprints (SHA-256)
A fingerprint identifies the exact file used for this result.
Input · Document (PDF)89d80458dbda895f11ee6d6c8a2d0cb6ed8d7b79bb3bf9304feecfbd20547c1b
Proof 2 · Screenshot90c6ade682d8a166172fdfa8d9a3e9624b584918441bd3a86a3caa42876b9086
Proof 1 · Output file (text)dd38bf048e4bd747eec06acc3e6b4d92f292c0bd46b9e5ed7006a585ffe85cf6
Proof 3 · Screenshot8655c18fad70ad9f91d1506f39354889738e2107ee6be8a61fc4045654c7f5c4