A response can pass a JSON schema and still put the wrong amount into a business system. We tested that distinction in a small local-model exercise on 21 November 2026. The useful result was a comparison of completed tasks and review decisions, including the failures.
A fixed task with explicit exceptions
We authored 16 synthetic invoice-like documents and an answer key for invoice identifier, amount due, currency and the need for review. The cases include a partial payment, decimal comma, an unreadable amount, conflicting totals, a credit note and an instruction embedded in document text.
Two already-installed local models received the same JSON schema and the same two frozen instruction variants. The short prompt asks for extraction. The bounded prompt adds rules for amount selection, missing values, ambiguity and treating the document as data rather than instructions.
There were no client records, PDF parsing or OCR in this test. Full fixtures, raw outputs and the reproduction script are available in the evaluation resource.
Schema validity did not establish correctness
All 16 outputs in each of the four model/prompt combinations passed the required shape and types. This was expected to help because the local generator received the schema. It did not ensure the extracted facts or review decision were correct.
With qwen2:0.5b, the short instructions produced 6 task-correct outputs out of 16. The bounded instructions produced 2. The longer instructions reduced incorrect records marked ready from 10 to 3, but flagged 13 outputs for review. More caution did not mean more correct completed tasks.
With llama3:latest, the short instructions produced 7 task-correct outputs and the bounded instructions produced 11. The bounded variant still marked 5 incorrect records as ready. Improvement on this exercise was not a production acceptance decision.
The second model was evaluated after inspecting the first run. Both prompts and the answer key stayed unchanged, but this remains an exploratory comparison. One generation per case does not establish repeatability, statistical confidence or a generally best model.
Correct review and automatic completion are different
Our task-correct count includes the expected review decision. A document with missing currency can be correctly handled by preserving known facts and asking for review. That is useful behaviour, but it is not an automatically completed record.
A useful evaluation therefore separates correct automatic results, appropriate review, unnecessary review and incorrect records marked ready. A system that flags everything may reduce false acceptance while returning most of the work to a person. A system that confidently fills every field can look efficient while creating correction work downstream.
Compare the complete workflow
The recorded local median and p95 latency appear alongside quality in the resource. They are wall-time measurements on one machine after warm-up. Caching and runtime conditions can affect them. They are not cloud latency guarantees or a priced operating-cost comparison.
For a real system, include parsing, retrieval, retries, exceptions, review and recovery. Measure cost per completed task using actual usage and approved pricing. Do not infer savings from token count alone, and do not attach a hosted tariff to a local run that did not use it.
Use the failures to choose the next investigation
Before changing a model, identify whether the error began in the source document, parsing, retrieval, business rule or output policy. Compare a bounded change against the same cases and retain the ones it still fails. Agree what must happen when evidence is missing or contradictory.
The Meddox delivery map describes a separate document-processing product with a human checkpoint. It is not the source of these synthetic results or evidence of their clinical relevance.
If you already run an AI workflow, an existing-AI assessment can define the permitted examples, acceptance criteria and improvement boundary. Explore AI implementation to discuss that scope.


