Doing the memory maths: does the model fit?
Whether a model runs on your machine is decided by memory – and can be worked out before the download. The rule of thumb from parameters and quantisation, and what context and speed change about it.
Boaz Lichtenstein
The most common disappointment with local models: twenty gigabytes downloaded, and then the model answers at one word per second – or not at all. That can be worked out beforehand.
What a model consists of
A language model is at its core a very long list of numbers, the parameters. Their count is in the name: “8B” is eight billion, “27B” twenty-seven. More parameters generally mean more ability – and more memory.
In the original, every parameter takes 16 bits. A model with eight billion parameters would thus be 16 gigabytes in size.
Quantisation: the same model, smaller
Quantisation stores every parameter with fewer bits – coarser, but almost as good.
The 4-bit version is what tools such as Ollama load by default. That turns the 16 gigabytes into just under five.
The rule of thumb
For the usual 4-bit version:
memory in GB ≈ billions of parameters × 0.6That is the model alone. Then comes the context.
Context costs extra
Everything the model currently has “in mind” – your brief, the document, the conversation so far – takes up additional memory. For a short chat that is a few hundred megabytes, for a long document or an agent session several gigabytes.
So allow a quarter of headroom on top of the figure from the table. And remember what else is running: browser, spreadsheets, video call.
Where it has to fit
PC with a graphics card: a model is only fast if it fits entirely into graphics memory. If it does not, part of it moves into ordinary RAM – it still runs then, but many times more slowly.
Machines with unified memory – Macs with Apple chips, for instance: here system and model share the same memory. A machine with 16 GB does not have 16 GB free for the model, but perhaps ten.
As guide values for total memory, with headroom included:
The table says what runs – not whether it is enough for your task. Only a trial with real documents shows that. And whether the “AI PC” label changes anything about this calculation is settled in the next station.
Big and coarse beats small and fine
If you are choosing between a 14B model at 4 bits and an 8B model at 8 bits – both need a similar amount of memory – take the larger one. The extra parameters bring more than the more precise storage. Below 4 bits that reverses.
Speed
Speed is measured in tokens per second; a token is about three quarters of a word. From roughly ten tokens per second a chat feels fluent. For agents that read and write long texts, every bit more counts – what is tolerable in chat turns an agent session into a test of patience.
The tools display the figure after every answer. Measure it once with a long document, not with “hello”.