Benchmark methodology

Converting a complex PDF into clean Markdown with an open-source library

The public evaluation method approved for this benchmark definition.

How the benchmark works

A public summary of the evaluation method. The same defined scope and evidence standard apply to every tool assessed under this version.

Evaluation unit

Each verdict applies to one test case run against one tool.

Outcomes

A result is pass, fail, or not gradable; not gradable is not a failure.

Consistency

Use the registered scenario versions and the registered resource versions for every run.

Evidence standard

Keep the recorded stimulus, the complete output, and the relevant registered references with the verdict.

Active test cases are partly withheld to reduce benchmark gaming; capabilities, scenarios, method, and resource types stay public.