Six months of GPT-class models in production: an honest report card

Between December 2025 and May 2026 we had 43 GPT-class systems running in client production — extraction pipelines, drafting assistants, triage and routing, retrieval-backed Q&A. Together they made about 2.1 million model calls. This is the report card we wish someone had published before we started: what the models reliably did well, the three ways they failed, how we catch those failures, and what one mid-size client actually pays per month.

Where the models earned their keep

Extraction is the workhorse. Eleven systems pull structured fields out of invoices, purchase orders and delivery notes. With schema validation and cross-checks against the client’s own master data, field-level accuracy sits between 96.8% and 99.2% depending on document quality. Be clear about the credit split, though: the validators do a third of the work. A model that reads “total: 4,120” correctly is good; a validator that knows this customer never orders above 2,000 and routes the document to a human is what makes the system deployable.

Drafting earns its keep when a human presses send. In the two customer-service deployments, median time to edit a drafted reply is 22 seconds against roughly four minutes to write one. Nobody has asked us to remove the human, and we would argue against it — the editing step is the quality system.

Triage is quietly the best value per token. Routing tickets and classifying intake agrees with senior staff 91% of the time, and the cost of a misroute is a short delay, not a wrong answer to a customer. Cheap errors are the friendliest place to put a model.

A failure taxonomy

Six months of logs produce a short, repetitive list. Every incident we handled fits one of three bins.

1. Hallucinated fields

An extraction model asked for a PO number will sometimes produce one when the document has none — plausible format, correct prefix, entirely invented. The fix is structural: every field in the schema is nullable, “not present” is a first-class answer we log and reward in evals, and any populated identifier is cross-checked against the customer master before it touches the ERP. After that change, invented identifiers stopped reaching downstream systems entirely.

2. Silent format drift

The provider updates a model, and JSON that parsed cleanly for five months starts arriving with a sentence of preamble in 0.3% of calls. Nothing crashes — the retry loop quietly eats the failures, latency and cost tick up, and nobody sees anything for weeks. The countermeasure is embarrassingly simple: a parse-failure counter with an alert threshold, and pinned model versions wherever the provider offers them. We now consider that counter part of the minimum viable deployment.

3. Cost creep

Prompts only ever grow. Every incident review adds a defensive paragraph; every edge case adds an example. One system’s cost per task doubled in nine weeks with zero traffic growth — pure prompt inflation plus retries. We now track cost per completed task, not per token, and prompt changes go through the same review as code.

The eval harness

Every system ships with a frozen golden set of 150–400 examples drawn from real, redacted traffic — never synthetic. A nightly job runs the production prompt and model against the set. Extraction and triage are scored with exact-match and schema checks; drafting is the only place we use a model as judge, scored against a written rubric and re-calibrated monthly against human ratings, because judges drift too. A two-point drop against the golden set pages a human. That harness is how “silent” drift stopped being silent.

What it costs a mid-size client

A distributor running three of these systems (extraction on ~38,000 documents a month, drafting, triage) pays approximately:

Monthly running costs, March 2026 invoice, rounded

Line item	Monthly
Model calls — extraction	$310
Model calls — drafting and triage	$95
Embeddings and retrieval infrastructure	$45
Nightly evals and monitoring	$60
Human review, ~9 hours at loaded cost	$270
Total	$780

Against roughly 310 staff-hours the same work took before, the arithmetic is not close. But notice that the largest single line is human time, and that is typical — if a vendor quotes you model costs only, they have quoted you less than half the bill.

The three we switched off

Honesty section. A meeting summariser averaged four opens a week — the summaries were fine and nobody wanted them. A support chatbot answered questions the existing search box already answered, slower. And a demand-forecasting experiment lost to a seasonal moving average on the client’s own backtest, so it never left the pilot. All three failed for the same reason: we (and the client) had guessed at demand for the output instead of measuring it.

What we'd tell you to do

• Make “not present” a first-class answer in any extraction schema, and validate identifiers against data you already trust.
• Put a parse-failure alarm in before launch. It is an afternoon of work and it catches the failure mode you cannot see.
• Meter cost per completed task from day one, and review prompt changes like code changes.
• Build the golden set from real traffic before you build the feature.
• Switch off systems nobody uses — quickly and without ceremony. The models are rarely the problem; unread output is.