Where AI Reliably Fails

Language models do not err at random. They err in patterns, and knowing the patterns is what lets you spot the cases where an answer sounds convincing and still is not true. Five of them turn up in daily work constantly.

1. The invented citation

The model names a study, a ruling, a statute, a page number – in the right format, with a plausible title, and without it existing. This is the most expensive error because it looks like the opposite of an error: like evidence. Rule: every source you pass on is one you opened yourself.

2. Yesterday’s state of affairs

Models have a knowledge cutoff, and it lies in the past. Prices, legal positions, product versions, people in office – precisely the facts that change are the ones a model will state unflinchingly in their old form. For anything dated: check yourself, and check the primary source.

3. Arithmetic

A language model does not calculate, it continues sequences of numbers that look plausible. With three line items that works; with percentages across several steps it often does not. If you need numbers, have the model output the working and check it – or calculate yourself and use the model for the explanation.

4. Agreeableness

Disagreement costs a model nothing, and neither does agreement – but agreement comes more often. Build an assumption into your question and you will usually get it confirmed. The counter-check is simple: ask the same case once with the opposite assumption. If the answers move in the same direction as your questions, you do not have a judgement, you have an echo.

5. The missing context it guesses at

When a detail is missing, a model rarely asks. It fills the gap with the average – the usual case, the standard size, the common practice. In niches the average is almost always wrong. Hence the boundary from the prompt station: gaps should show up as gaps, not as assumptions.

What that means in practice

Sort tasks by how expensive an undetected error would be. Rewriting an email: cheap. A legal reference, a calculation, a public statement: expensive. For the expensive side the rule is that the model may draft and you must check. That is not scepticism about the technology but the same division of labour you would have with a very fast, very well-read and occasionally overconfident intern.