Use a fixed document set
Each run uses a fixed document set before scoring starts. Documents are tagged by file type, language, length, and layout challenge.
Methodology
How Yardstick Lab scores tool categories
Each category needs a fixed task set, the same requested outputs, and scoring against hand-checked labels. The current protocol is applied first to document AI.
Protocol
Ranking protocol
The protocol is designed so a ranking can be inspected rather than accepted on trust.
Ask for the same output
Every system receives the same target fields, nested structures, array requirements, and extraction instructions for a task.
Preserve raw output
Each tool output is stored before normalization so errors can be inspected later.
Date the ranking
Published rankings are dated and versioned. Later system changes produce a new run, not a silent overwrite.
Separate task slices
Overall scores are broken into slices so buyers can compare the documents closest to their workflow.
Keep baselines separate
Direct model baselines are identified as baselines, not product platforms.
Score weights
What goes into the score
The score emphasizes extraction correctness first. Runtime and cost matter only after the output quality clears the bar.
40%
Field accuracyCorrectness of extracted values after documented normalization.
20%
Array and item matchingRecall and placement for line items, transactions, tables, and repeated sections.
15%
Schema validityWhether output conforms to the requested structure and expected field types.
15%
Hallucination penaltyPenalty for values that are not supported by the source document.
10%
Runtime and costSecondary score for latency and price once accuracy clears the bar.
Verification
What readers can verify
Each public score eventually traces to the run date, source document, expected output, raw tool output, and scoring decision.
The same task
Each tool is judged against the same documents, schema, field definitions, and expected answer set.
The source of the score
Published rows point back to per-document output, expected values, and the scoring decision.
The run date
Scores are attached to a dated run so later product changes can be compared without rewriting history.
The failure mode
A useful review shows whether an error came from a missing value, wrong row, invalid schema, or unsupported extra value.
Normalization
Normalization rules
Normalization is limited to transformations that do not change field meaning.
Deterministic formats
Dates, currencies, percentages, and numeric separators are normalized only when the transformation is deterministic.
Meaning-preserving text
Whitespace and punctuation differences are ignored only when they do not change the field meaning.
Error categories
Missing values, unsupported extra values, and values from the wrong row or page are counted separately.