The line we draw around a language model

The interesting engineering question about language models is no longer what can they do. It is which decisions they should be allowed to own. Those are different questions, and conflating them is how a promising prototype becomes a product that quietly does the wrong thing at scale.

A model that is right ninety-five per cent of the time is extraordinary as a research result and unacceptable as an irreversible action taken a thousand times a day. The five per cent does not distribute itself politely. So before we hand any decision to a model, we ask three questions.

1. Is the action reversible?

Drafting an email is reversible. Sending it is not. Suggesting a schedule is reversible. Cancelling the appointments to apply it is not. This is the cheapest and most reliable filter we have, and it costs nothing to apply: separate the generation of a proposal from its execution, and keep a human on the second step whenever the first cannot be undone.

The failure mode here is not the model. It is the product decision to collapse two steps into one because the demo felt smoother.

2. Can the answer be checked more cheaply than it was produced?

This is the asymmetry that makes some tasks safe to automate. If a model proposes a route, a solver can verify the constraints in microseconds. If it extracts a figure from a document, a regular expression can confirm the figure appears in the source. If it writes code, tests run.

Where a cheap verifier exists, the model is doing search and the verifier is doing the deciding — and the composite system is far more reliable than the model alone. Where no verifier exists, the model's output is the decision, and confidence in it has to come from somewhere else entirely.

Generation is not the hard part. Verification is where the reliability actually comes from.

3. Would the user notice if it were wrong?

This is the one that gets underweighted. A wrong answer a user can immediately recognise as wrong is an annoyance. A wrong answer that is plausible, well-formatted, and about a domain the user consulted the product on precisely because they do not know the answer is something else. The interface's fluency becomes the evidence for the content's correctness.

In that regime we do not ship the raw model output. We either ground it in a retrievable source the user can open, constrain it to a structured space where invalid answers cannot be represented, or attach a deterministic computation the model is not permitted to override.

What this looks like in practice

• Structured output over free text whenever the result feeds another system. A schema the parser rejects is a failure you can see; a paragraph that is subtly wrong is not.
• Retrieval with visible citations, so the user's trust attaches to the source rather than to the fluency of the prose.
• Deterministic computation for anything numeric. The model may choose the formula; it does not get to evaluate it.
• An evaluation suite that runs on every change, with cases collected from real failures rather than imagined ones.
• A stated refusal boundary — the set of questions the product answers with "this needs a person", written down before launch and not negotiated afterwards under pressure from a roadmap.

None of this is a limitation on what language models can do. It is the ordinary discipline of building on top of a component with a known error rate, which is something engineers have done with sensors, networks and floating-point arithmetic for decades. The novelty of the component is not a reason to abandon the practice.