Running capable language models on ordinary business hardware became realistic surprisingly recently. Not for everything — the largest cloud models remain meaningfully better at the hardest reasoning — but for a wide band of practical work, a model running on a machine in your office is now good enough, and the data never leaves the building.
What works well locally
- Summarising and extracting. Pulling structured information out of documents, emails, transcripts and reports. Local models are strong here and the documents are often the most sensitive thing you have.
- Drafting and rewriting. First drafts, tone adjustment, translation, standard correspondence.
- Classification and routing. Sorting incoming enquiries, tagging tickets, triaging documents.
- Search over your own material. Answering questions from your own files, contracts and history — the single most valuable local use case for most businesses, and the one that would be most uncomfortable to send elsewhere.
- Anything high-volume and repetitive. Where per-call cloud costs accumulate, local inference is free after hardware.
What still favours the cloud
Being straight about this matters more than being enthusiastic:
- Hard multi-step reasoning. Complex analysis, difficult code, long chains of inference. The frontier models are better and the gap is real.
- Very long documents. Large context windows are memory-hungry; local models handle shorter contexts comfortably.
- Anything needing the newest capabilities. Cloud providers ship improvements continuously; local models move in releases.
- Occasional use. If you run a handful of requests a week, hardware is a poor investment against a few dollars of API cost.
What you need
| Setup | Realistic use | Rough cost |
|---|
| A modern laptop with 16GB | Small models, light daily work | Hardware you likely own |
| 32GB desktop or Mac | Mid-size models, comfortable daily use | Mid four figures |
| Dedicated GPU workstation | Larger models, multiple users, speed | Upper four to five figures |
| Rented private server | Team access without buying hardware | Monthly, but your instance |
Detail on sizing: local LLM hardware requirements.
How to start without committing
1
Install Ollama on a machine you already have. Free, and it runs on Mac, Windows and Linux.
2
Pull a small model and use it for a week on real work — summarising, drafting, sorting. You will learn quickly where it is good enough.
3
Pick one workflow that is genuinely sensitive and try moving just that one locally. Client documents, HR material, financial records.
4
Measure honestly. Compare output against what you get from a cloud model on the same task. Sometimes the gap does not matter; sometimes it does. Both answers are useful.
5
Only then decide about hardware. Buying a workstation before knowing which workflows benefit is how expensive machines end up idle.
The most common mistake is treating this as all-or-nothing. Most businesses that do this well end up with a split: sensitive and repetitive work runs locally, hard reasoning goes to a cloud model, and the boundary is a deliberate decision rather than a default.
Common questions
Is local AI as good as ChatGPT or Claude?
Not for the hardest tasks — the frontier cloud models remain better at complex reasoning, long documents and code. For summarising, drafting, classifying and answering questions over your own material, good open-weight models are close enough that the difference rarely changes the outcome. Test on your actual work rather than on benchmarks.
What does it cost to run AI locally?
After hardware, essentially nothing per use — which is what makes it attractive for high-volume repetitive work. The cost is upfront: a machine with enough memory, and the time to set it up. Against a few dollars a month of API usage, hardware makes no sense; against continuous processing of thousands of documents, it pays back quickly.
Do I need a technical person to set this up?
For a single machine running Ollama, no — installation is a download and a command. For serving a team, integrating with existing systems, or handling regulated data properly, yes. The gap between "it runs on my laptop" and "the business depends on it" is where most of the real work sits.
Can local AI access my company files safely?
That is precisely its advantage: a local model reading your contracts, invoices and correspondence sends nothing anywhere. The security question shifts from "what does the provider do with this" to "who in my business can query this system", which is a question you control and can audit.