One rule explains most of this: the model must fit in memory. Everything else — speed, model choice, whether you need a GPU — follows from that constraint. Get memory right and the rest is optimisation.
A quantised model needs approximately its parameter count in gigabytes, plus overhead for context. An 8-billion-parameter model at common quantisation wants around 8–10GB. A 70B model wants 40GB or more.
| Memory available | Comfortable model size | What that gets you |
|---|---|---|
| 8GB | 3B, small 7B quantised | Basic summarising and classifying. Usable, limited. |
| 16GB | 7B – 8B comfortably | The practical entry point. Good for most business tasks. |
| 32GB | 13B – 14B, 30B quantised | Noticeably better quality. A good default target. |
| 64GB+ | 70B quantised | Closest to cloud quality available locally. |
Apple Silicon Macs share memory between processor and graphics, so a machine with 64GB of unified memory can load a model that would otherwise require an expensive dedicated GPU. For a small business wanting one capable machine without building a workstation, a Mac with generous memory is often the simplest good answer — and it is quiet enough to sit in an office.
On the PC side, video memory on the graphics card is the binding constraint rather than system memory. A card with 24GB of VRAM handles considerably more than one with 8GB, regardless of how much system RAM the machine has.
For usable speed, effectively yes — or unified memory doing the same job. Models run on CPU alone, but generation slows to the point where people stop using the system, which is the real failure mode. A tool that takes forty seconds to answer gets abandoned no matter how private it is.
The laptop you already own, if it has 16GB. Costs nothing, and tells you within a week which workflows local AI can actually handle.
A Mac with 32–64GB unified memory, or a PC with a 16–24GB graphics card. This is the sweet spot for most small businesses.
A dedicated machine with a strong GPU on the network, serving several people, with authentication in front of it.
A rented GPU instance. Monthly cost, but a private instance rather than a shared service — a reasonable middle ground for regulated data.
With 16GB of memory, yes — an 8B-class model runs comfortably and handles summarising, drafting and classification well. With 8GB you are limited to small models and will feel it. Below that, local inference is not a realistic proposition.
You need either a capable GPU or Apple Silicon unified memory. CPU-only inference works but is slow enough that people abandon the tool, which defeats the purpose. The cheapest viable path is usually a Mac with generous unified memory rather than building a PC around a high-VRAM card.
It depends entirely on volume. For occasional use, no — hardware costs far more than a few dollars of API calls. For continuous high-volume work, hardware pays back within months and then costs nothing per use. Estimate your monthly API spend honestly before buying anything.
A good middle path: a private GPU instance gives you a dedicated environment without capital expenditure, and for regulated data a single-tenant instance is often acceptable where a shared consumer service is not. You pay monthly, but the instance is yours rather than a shared service.
Private AI systems in environments you control — your data never trains public models, and engagements are available under NDA.