Valid JSON Is Not a Correct Extraction: A Measured Exercise
Inspect two local models, two frozen prompts and 16 synthetic documents. See task correctness, review failures, local latency and the reproducible raw outputs.
What we actually ran
On 21 November 2026 we ran two already-installed local models, qwen2:0.5b and llama3:latest, against 16 authored synthetic invoice-like documents. Each model received the same two prompts and the same JSON output schema. One prompt gives short extraction instructions; the other defines amount selection, ambiguity, missing fields, review and the boundary between document data and instructions.
The expected fields are invoice identifier, amount due, currency and whether review is required. Cases include partial payment, decimal comma, missing values, conflicting totals, a credit note and an instruction embedded in document text. None is a client record or medical document.
How to read the results
Schema-valid means the response has the required keys and types. The schema is supplied to the local generator, so success here is not evidence of factual accuracy.
Task-correct means all four expected fields, including the review decision, match this exercise’s answer key. Amounts use a half-cent tolerance. A correct review/abstention result is included, so task-correct is not the number of records processed automatically.
Incorrect records marked ready counts valid-shaped outputs with at least one wrong expected field and no review flag. This makes a superficially convenient response visible as a workflow risk.
Review flagged is the number of responses asking for review, whether that decision was correct or unnecessary. A higher review count can reduce false acceptance while increasing work for people.
Latency is local wall time after warm-up, one generation per prompt/case. The table uses the median and nearest-rank p95 across 16 observations. These are not cloud-service latency guarantees. Inference usage is recorded in the raw results, but hardware, electricity and hosted-model tariffs were not costed.
What this exercise supports
A longer instruction can change review behaviour and still reduce task correctness. A different model can change quality and local latency, but this small set cannot identify a universally best model. Any candidate should meet the workflow’s acceptance criteria before it can act on records.
For an existing AI assessment, start with a known baseline and representative failures. Compare one bounded change, retain the failed cases and include the human effort that remains. A lower token count or valid JSON alone does not establish a better completed task.
Limits and reproducibility
The prompts and answer key were authored before inference. The second model was evaluated after inspecting the first run, making this an exploratory comparison, not a blind pre-registered experiment. No prompt was tuned between the two model runs.
There is one generation per prompt and case. Order alternates between variants, but caching, hardware state and model loading still matter. This is a small synthetic text exercise, not a PDF/OCR test, adversarial security certification, client benchmark or clinical validation. No result here establishes readiness for your production documents.
Download the full results and reproduction package below. They include all raw outputs and failures, the expected answers, prompts, model digests, software version, settings and fixture checksum. Rerunning on another machine or version may produce different results. The local runner does not download a model or send documents to a cloud inference service.
The request format follows the official Ollama generate API. For the related product-delivery boundary, see the Meddox pipeline. That client project is distinct from this synthetic exercise.