Use a fixed document set
Each run uses a fixed document set before scoring starts. Documents are tagged by file type, language, length, and layout challenge.
Methodology
How Yardstick Lab scores tool categories
Each category needs a fixed task set, the same requested outputs, and scoring against hand-checked labels. The current protocol is applied first to document AI.
Protocol
Ranking protocol
The protocol is designed so a ranking can be inspected rather than accepted on trust.
Ask for the same output
Every system receives the same target fields, nested structures, array requirements, and extraction instructions for a task.
Preserve raw output
Each tool output is stored before normalization so errors can be inspected later.
Date the ranking
Published rankings are dated and versioned. Later system changes produce a new run, not a silent overwrite.
Separate task slices
Overall scores are broken into slices so buyers can compare the documents closest to their workflow.
Require complete coverage
A configuration enters the current ranking only after it has a score for every document in the run.
Scoring rule
What goes into the score
The leaderboard is accuracy only. Cost and latency are useful buying inputs, but they do not move a vendor's rank here.
01
Normalize valuesCase, whitespace, punctuation, and harmless numeric formatting differences are normalized deterministically.
02
Score leaf fieldsNon-array leaves receive a binary exact-match score after normalization.
03
Match array itemsRepeated records are paired greedily by best match, without penalizing a correct row order difference.
04
Average by documentEach document gets one field-weighted score, then all document scores receive equal weight in the headline result.
Verification
What readers can verify
Each public score traces to the source document, schema, expected output, raw tool output, and open scoring code. Open any document alongside its answer key and every system's extraction in the data explorer, or download the replication kit and re-run the scorer to reproduce the numbers yourself.
The same task
Each tool is judged against the same documents, schema, field definitions, and expected answer set.
The source of the score
Published rows point back to per-document output, expected values, and the scoring decision.
The run date
Scores are attached to a dated run so later product changes can be compared without rewriting history.
The failure mode
A useful review shows whether an error came from a missing value, wrong row, invalid schema, or unsupported extra value.
Normalization
Normalization rules
Normalization is limited to transformations that do not change field meaning.
Deterministic formats
Dates, currencies, percentages, and numeric separators are normalized only when the transformation is deterministic.
Meaning-preserving text
Whitespace and punctuation differences are ignored only when they do not change the field meaning.
Schema scope
Extra fields outside the schema are ignored in ordinary objects. Extra or missing array content affects item matching and recall.