LLM Evaluation Checklist for Document Analysis Teams in 2026

Measure character error rate (CER) across a representative sample of at least 200 documents. CER should be below 1% for printed text and below 5% for handwritten text. Track CER separately by font type, document quality, and language.

Calculate word error rate (WER) as a more practical accuracy metric than CER. A single character error that changes a word's meaning (e.g., '1000' to '100C') has outsized downstream impact. Track WER by document section type.

Measure precision and recall for key entity types: person names, company names, dates, amounts, addresses, and reference numbers. Entity extraction errors propagate directly into databases and workflows. Evaluate per entity type with at least 50 samples each.

Test extraction accuracy on documents containing multiple languages, including language switching within the same paragraph. Multi-language documents are common in international business. Benchmark each language pair relevant to your use case.

Evaluate extraction accuracy on degraded documents: faded text, coffee stains, creases, low-resolution scans, and photocopied copies. Real-world documents are rarely clean. Build a test set graded by document quality (good/fair/poor) and track accuracy per grade.

Test extraction of currency symbols, mathematical notation, legal symbols (section signs, paragraph marks), and accented characters. These are frequently misrecognized. Build a targeted test set of 100+ symbol-heavy snippets.

Verify that the system correctly identifies and handles headers, footers, page numbers, and watermarks. These elements should be excluded from body text extraction unless specifically requested. Test across varied document layouts.

Evaluate accuracy of paragraph boundaries, section breaks, and logical text flow reconstruction across columns and page breaks. Incorrect flow reconstruction garbles the reading order. Test with multi-column layouts and flowing text.

Verify that extraction confidence scores are well-calibrated: high confidence should correlate with high accuracy. Miscalibrated confidence scores undermine review prioritization. Plot calibration curves and measure expected calibration error.

Separately benchmark handwritten text recognition accuracy, as it typically requires different models and has much higher error rates. Test across handwriting styles and document contexts. Set explicit quality thresholds for handwritten content.

Measure precision and recall for detecting tables in documents, including tables without visible borders. Missed tables mean lost data; false positive table detections corrupt adjacent text extraction. Test on 100+ documents with varied table styles.

Evaluate accuracy of extracting individual cell values and correctly mapping them to row-column positions. A single row misalignment corrupts an entire table. Test with tables of varying complexity: merged cells, multi-line cells, and nested tables.

Verify correct identification of header rows and columns that provide context for cell values. Without correct headers, extracted data is just disconnected numbers. Test with single-row headers, multi-row headers, and rotated column headers.

Test extraction of tables with merged cells spanning multiple rows or columns. Merged cells are common in financial reports and invoices. Verify that the system correctly associates merged cell values with all relevant rows and columns.

Evaluate handling of tables that span multiple pages. The system should correctly stitch table parts across page breaks, maintaining row and column alignment. Test with tables spanning 2, 3, and 5+ pages.

Verify that extracted numbers maintain their original format and precision: decimal separators, thousands separators, currency formatting, and percentage notation. Format conversion errors (European vs. US decimal notation) cause data errors. Test across locales.

Test extraction of key-value pairs from forms: check boxes, radio buttons, fill-in fields, and dropdown selections. Form understanding requires spatial reasoning beyond simple text extraction. Build a test set of 50+ form types.

Evaluate the accuracy of converting extracted tables into structured formats (JSON, CSV, database records). The output schema should be consistent and complete. Verify end-to-end from table image to structured data.

If extracting data from charts and graphs, measure accuracy of value extraction, axis label identification, and legend interpretation. Chart data extraction is significantly harder than table extraction. Benchmark per chart type.

Evaluate whether the system identifies relationships between multiple tables in the same document, such as summary tables referencing detail tables. Cross-table relationships provide essential context for downstream processing.

Measure classification accuracy across all document types your system handles: invoices, contracts, reports, correspondence, forms, and IDs. Classification errors route documents to wrong processing pipelines. Target 98%+ accuracy on a balanced test set.

Evaluate whether classification confidence scores support reliable automated routing. Documents with low classification confidence should be flagged for manual review. Set confidence thresholds that maximize automation while minimizing misrouting.

Beyond broad categories, test fine-grained subtype classification: purchase order vs. sales order, lease agreement vs. service agreement. Subtype classification determines which extraction schema to apply. Evaluate per subtype.

Test automatic language detection for incoming documents, especially for multilingual documents or documents with non-standard fonts. Incorrect language detection cascades into extraction failures. Benchmark across all supported languages.

Evaluate the system's ability to identify duplicate submissions, near-duplicates (same document with minor changes), and versioned documents. Duplicate processing wastes resources and creates data conflicts. Test with intentional duplicate sets.

Test whether the system identifies urgent documents (time-sensitive contracts, expiring certificates, overdue invoices) for expedited processing. Missed urgency detection has direct business consequences. Create test cases with varied urgency indicators.

Verify that classification remains consistent when documents are processed individually versus in batches. Batch processing can introduce inconsistencies due to context bleed or resource constraints. Compare individual vs. batch results on 100+ documents.

Test system behavior when it encounters document types not in its classification taxonomy. It should flag unknown documents for human review rather than forcing them into incorrect categories. Measure unknown type detection rate.

Measure classification latency per document to ensure it does not bottleneck the processing pipeline. Classification should complete within 1-2 seconds for real-time workflows. Profile under load to identify scaling limits.

Test the system's resistance to adversarial documents designed to trick classification: altered logos, modified headers, or misleading content. Adversarial inputs should not bypass classification guardrails. Run targeted red-team exercises.

Verify that all extracted data conforms to the expected output schema: required fields are present, data types are correct, and enum values are valid. Schema violations break downstream integrations. Implement automated schema validation on 100% of outputs.

Apply domain-specific business rules to validate extracted data: invoice totals matching line items, date ranges being logical, reference numbers following expected patterns. Business rule violations flag likely extraction errors. Implement rule checks per document type.

Validate that related fields within a document are consistent: billing address matches shipping address format, currency symbols match amounts, and document dates follow chronological logic. Cross-field inconsistencies indicate extraction errors.

Test that output formats (JSON, XML, CSV, API payloads) are correctly structured for downstream system consumption. Format errors cause integration failures that may not surface immediately. Validate against downstream system schemas.

Verify that the system correctly distinguishes between fields that are empty in the source document versus fields that failed to extract. These require different downstream handling. Test with documents that have intentionally blank fields.

Evaluate normalization of dates (various formats to ISO 8601), currencies (symbol to code), and addresses (free text to structured) for correctness and consistency. Normalization errors are subtle but create matching failures downstream.

For batch processing, verify that all input documents produce corresponding outputs and that no documents are silently dropped. Silent data loss is extremely difficult to detect after the fact. Implement input-output count reconciliation.

Verify that reprocessing the same document produces identical results. Non-deterministic extraction creates reconciliation nightmares. Test idempotency across 50+ documents processed at different times.

Evaluate the quality and actionability of error reports when extraction fails or produces low-confidence results. Error reports should identify the specific field, document location, and likely cause. Poor error reports make manual review inefficient.

Benchmark the document processing API under realistic load for response time, throughput, and error rates. Set SLAs per document type and complexity. Identify throughput ceilings and plan scaling accordingly.

Verify that personally identifiable information (names, SSNs, account numbers) is detected and handled according to your data protection policies. Test PII detection across all document types. Ensure PII is redacted from logs and intermediate storage.

Verify that document data is processed and stored in compliance with data residency requirements (GDPR, data sovereignty laws). Documents may need to be processed in specific geographic regions. Audit data flow across all processing stages.

Test that document access controls prevent unauthorized users from viewing or processing restricted documents. Financial, legal, and HR documents often have strict access requirements. Verify controls at every processing stage.

Verify that a complete audit trail records who submitted each document, when it was processed, what was extracted, and who accessed the results. Audit trails are required for regulated industries. Test audit completeness for complex processing workflows.

Test that documents and extracted data are retained and deleted according to your retention policies. Verify that deletion is complete across all storage tiers (primary, cache, backups). Test the end-to-end deletion workflow.

Verify that documents are encrypted at every storage and transmission point: upload, processing, storage, and output delivery. Any gap in encryption creates a data exposure risk. Audit encryption coverage across the full pipeline.

If using third-party OCR or AI APIs, verify their data handling practices: do they store documents, use them for training, or share data with subprocessors? Review DPAs and terms of service. Conduct vendor security assessments.

Verify that output documents and extracted data do not inadvertently include metadata, hidden text, or tracked changes from the source document. Document metadata can contain sensitive information. Sanitize all outputs.

Evaluate the ability to generate compliance reports on document processing: volumes processed, PII detected, access logs, and deletion records. Automated reporting reduces compliance overhead. Test report accuracy against manual audits.

Test the incident response process for scenarios where a data breach is detected in the document processing pipeline. Response should include containment, notification, and forensics. Conduct tabletop exercises quarterly.