Many operations teams still run on documents. Invoices arrive as PDFs and phone photos. Claims come with scanned forms, handwritten notes and damage images. KYC packs mix identity documents, utility bills and bank statements in whatever format the customer had.
Traditional OCR and template rules handled the clean cases and left the rest for manual keying. Multimodal models change that. They read layout, tables, stamps, handwriting and images together. They are not magic, though. Production value comes from the pipeline around the model.
Classify before you extract
A single upload often contains several documents. The first step is splitting and classifying: which pages are the invoice, which is the delivery note, which is a cover letter nobody needs.
Classification errors cascade. Extracting invoice fields from a purchase order produces confident nonsense. Build classification as its own step, with its own evaluation set and its own confidence threshold.
Extract to a schema, not to prose
Ask the model for structured output against a defined schema: field names, types, allowed values, required versus optional. Free-text answers are hard to validate and harder to integrate.
For each field, capture:
- The extracted value
- Where it came from — page and region where possible
- A confidence signal from the model or from validation
Source location matters. Reviewers verify faster when they can see the highlighted region, and auditors can trace any value back to the original.
Validate with business rules
Models misread digits, swap dates and invent plausible values for blank fields. Deterministic checks catch much of this:
- Line items sum to the subtotal; subtotal plus tax equals total
- Tax identifiers and bank details match expected formats
- Dates fall in sensible ranges
- Vendor or customer matches a master record
- Policy numbers exist and were active on the incident date
Cross-document checks are where multimodal pipelines pay off. The name on the identity document should match the application. The invoice should reconcile with the purchase order and goods receipt.
Route by confidence and risk
Not every document needs a human, and not every document should skip one.
- Straight-through when classification, extraction and validation all pass and the value is below a risk threshold
- Review queue when a check fails or confidence is low, with the failing field highlighted
- Mandatory review for high-value payments, suspected fraud signals or any regulated decision your policy reserves for people
Thresholds are product decisions agreed with operations and risk, not defaults left by engineering.
Sensitive data needs deliberate handling
KYC and claims documents contain identity numbers, health information and financial details. Decide early:
- Where documents and extracted data are processed and stored
- Whether a hosted model is acceptable for each document type, or a private deployment is needed
- What is masked in logs and review screens
- How long images and extracted fields are retained
Requirements differ by sector and jurisdiction. Confirm specifics with counsel and your compliance team.
Evaluate on your documents
Vendor demos use clean samples. Your evaluation set should include the faxed, skewed, stamped, multilingual and handwritten documents your team actually receives. Label a representative set, measure field-level accuracy, and track the review rate as well as errors that slipped through.
Re-run evaluation whenever the model, prompt or schema changes. A new model version can be better overall and worse on one field you care about.
Integration closes the loop
Extraction that ends in a spreadsheet still leaves keying work. Post validated data into the ERP, claims or onboarding system through its APIs, and feed reviewer corrections back as labelled examples. Corrections are the cheapest source of new test cases you will have.
Where to start
- Pick one document type with high volume and clear downstream use
- Collect and label a realistic sample, including the messy ones
- Define the schema and validation rules with the people who key it today
- Launch with conservative straight-through thresholds and widen them as evidence builds
Incubics designs document pipelines with validation, review queues and audit trails from the first increment.