How to Run LLMs Locally (2026): Hardware, Models & Setup

This is the hands-on companion to our overview, Local LLMs in 2026: What They Are and Why They Matter. If you have already decided that reducing your dependence on cloud frontier models is worth doing — a decision that got a lot easier on 13 June 2026, when the US government forced Anthropic to pull Fable 5 and Mythos 5 for all foreign nationals — this is the how. (For the business and regulatory case — especially for Australian firms with privacy obligations — see Private AI for Australian Business.)

We will cover the stack, how to choose a model, the quantization maths that determines whether a model fits on your hardware, real VRAM and RAM requirements by model size, the tools to use, and a step-by-step first run. Written for technical readers, but the hardware tables are useful to anyone scoping a budget.

The local LLM stack: three layers

Running a model locally means assembling three things:

1. The model (weights) — the trained file you download, e.g. an 8B Llama or a 32B Qwen. Usually distributed via Hugging Face.
2. The inference engine — the software that loads the weights and runs the maths (token generation). Examples: llama.cpp, vLLM, MLX, Ollama's bundled engine.
3. The interface — how you talk to it: a desktop chat app, a terminal, or an OpenAI-compatible HTTP API your own code calls.

Most beginner-friendly tools (Ollama, LM Studio) bundle all three. Production setups (vLLM) separate them so you can serve many users efficiently.

Step 1: Choosing a model

The open-weight landscape moves fast, so think in families and sizes rather than memorising version numbers. As of 2026 the families worth knowing:

Model size (parameters) is the first lever. More parameters generally means better quality and more hardware:

• 1B–3B — fast, runs almost anywhere (even CPU). Good for simple classification, autocomplete, basic drafting.
• 7B–9B — the sweet spot for a capable single-GPU or laptop assistant. Solid general use, summarising, RAG.
• 13B–14B — noticeably better reasoning; needs a mid-range GPU.
• 30B–34B — strong, professional-grade output; needs a high-end GPU.
• 70B+ and large MoE (100B+) — approaches frontier quality on many tasks; needs serious VRAM or a high-memory Mac.

For coding specifically, the long-tail searches bear out what we see in practice: dedicated coding-tuned models (Qwen-Coder-style, DeepSeek-Coder-style) at 7B–32B are the practical picks for a local coding assistant, and many developers now wire them into editors — yes, you can even point tools like Claude Code or Cursor at a local OpenAI-compatible endpoint.

To compare current models objectively, check the Hugging Face Open LLM Leaderboard and LMArena rather than trusting any single blog's "best of."

Step 2: Understand quantization (this is what makes it fit)

A model's parameters are numbers. Quantization shrinks how many bits each number uses, trading a little quality for a lot less memory. This is the single most important concept for running models on real hardware.

• FP16 (16-bit / "full precision") — ~2 bytes per parameter. A 7B model ≈ 14 GB.
• 8-bit (Q8) — ~1 byte per parameter. 7B ≈ 7 GB. Near-lossless.
• 4-bit (Q4) — ~0.5 byte per parameter. 7B ≈ 3.5–4 GB. The most popular default — big memory savings, small quality hit.

You will see formats like GGUF (used by llama.cpp, Ollama, LM Studio — great for CPU/Mac/mixed) and AWQ/GPTQ (used by GPU-serving engines like vLLM).

Rule of thumb for Q4: memory needed ≈ parameters (in billions) × ~0.6 GB, plus headroom for the context window (the KV cache), which grows with how much text you feed in. Budget an extra 1–4 GB for context on top of the weights.

Step 3: Hardware requirements by model size

Here is the practical table we use when scoping. Assumes Q4 quantization (the common case) and includes rough context headroom. "VRAM" = dedicated GPU memory; on Apple Silicon, unified memory plays both roles.

Three notes that save people money:

• Apple Silicon is the dark-horse option. Because memory is unified, a Mac with 64 GB or 128 GB can run large models that would otherwise need multiple datacentre GPUs — slower than an A100, but astonishingly accessible. A Mac mini or Mac Studio is one of the best price-to-capability local boxes in 2026.
• VRAM is the binding constraint, not raw speed. A model that doesn't fit in VRAM either spills to system RAM (much slower) or won't load. Buy memory first.
• Context length costs memory too. Long documents and big RAG prompts inflate the KV cache. If you need 128k-token context, add meaningful headroom.

When you outgrow a laptop: dedicated local-AI hardware

Past roughly the 30B mark, a laptop or a single consumer GPU stops being enough — but you do not need a server rack either. A whole category of desk-side AI machines now fills that middle ground:

• Multi-GPU workstation — two RTX 4090/5090-class cards (or one RTX 6000-class card with 48–96 GB) give you 48 GB+ of fast VRAM and the best token throughput per dollar. The trade-off is power draw, heat, and noise.
• NVIDIA DGX Spark — the GB10 "desktop AI supercomputer" (originally announced as Project DIGITS). Roughly 128 GB of unified memory and around 1 petaFLOP of FP4 compute in a Mac-mini-sized box for about US$3–4k. It is purpose-built to run models up to ~200B parameters locally, and you can link two units over its built-in high-speed networking to handle a ~405B model. This is the cleanest answer to "I have outgrown a laptop and want a quiet box on my desk."
• NVIDIA DGX Station — the bigger sibling (GB300-class, hundreds of GB of coherent memory) for a serious power user or a small team's shared local model.
• Apple Mac Studio — configurable to roughly half a terabyte of unified memory on the top spec, it runs very large models at remarkable accessibility — slower than datacentre GPUs, but the price-to-capability story is hard to beat.
• AMD Strix Halo mini-PCs (Ryzen AI Max+, e.g. the Framework Desktop) — up to 128 GB of unified memory at a lower price than DGX Spark; a strong budget route to big-model inference.
• Datacentre GPUs (A100 / H100 / H200) — when you need to serve many concurrent users, rent these in the cloud or colocate them rather than buying a desk-side box (see production serving below).

Rule of thumb: buy memory capacity for the model size you want, and raw GPU speed for how many people you need to serve. A DGX Spark or a maxed-out Mac Studio gets a big model running; a multi-GPU server gets it running fast, for a crowd.

Step 4: Pick your tool

If you are evaluating: start with Ollama or LM Studio to prove it works, then move to vLLM when you need to serve more than one person reliably.

Step 5: Run your first local model (5 minutes)

Using Ollama (macOS, Windows, Linux):

# 1. Install from https://ollama.com, then pull and run a model:
ollama run llama3.1:8b

# 2. That drops you into a chat. Type a prompt and you're running a local LLM.

Call it from code via its OpenAI-compatible endpoint — note this request never leaves your machine:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "Summarise this contract clause..."}]
  }'

Prefer a GUI? Install LM Studio, search for a model (it tells you whether it fits your RAM/VRAM before you download), click download, and chat. Toggle on its local server to expose the same OpenAI-compatible API.

Step 6: Serve it to your team or app

For more than one user, switch to vLLM, which is built for throughput and concurrency:

pip install vllm
vllm serve Qwen/Qwen2.5-32B-Instruct-AWQ --port 8000
# Now any OpenAI-compatible client can hit http://your-server:8000/v1

Because the endpoint is OpenAI-compatible, most existing code, SDKs, and agent frameworks work by changing only the base URL and model name. That is the whole trick behind a clean hybrid architecture: your application talks to one interface, and a router decides whether a given request goes to a local model or a cloud one. Swapping or failing over becomes a config change, not a rewrite.

Production considerations

Once it leaves your laptop, the engineering starts:

• Throughput and concurrency — vLLM's batching is what makes one GPU serve many users. Size the GPU to your peak concurrent token load, not your average.
• Cost break-even vs API — local wins on high, steady volume; cloud wins on spiky or low volume. Do the maths: GPU amortised over its life + electricity vs. your monthly token spend. (Our Claude API pricing breakdown has the cloud side of that equation.)
• Security — a local model removes data-exfiltration risk to a provider, but you now own network isolation, access control, and patching.
• Monitoring and evals — log quality, latency, and failures. A local model that quietly degrades on an edge case is your problem to catch.
• Updates — open-weight models improve constantly; budget time to re-test and roll forward.

When you should still use a cloud frontier model

Being honest, as we always try to be: do not go fully local if your workload is dominated by the hardest reasoning, you have low or spiky volume, or you have no appetite to own GPUs and ops. In those cases a cloud frontier model — used with eyes open about the dependency risks in our overview piece — is still the right tool. The goal is not purity. It is removing the single point of failure while keeping the frontier's brains on tap.

Build it with us

Specifying the right model, sizing hardware, setting up vLLM, and wiring a clean cloud/local routing layer into an existing product is exactly the kind of work we do. We are a Claude-first consultancy that builds model-agnostic, resilient architectures when privacy, data residency, or continuity demand it.

Get in touch — tell us your workload and constraints, and we will tell you honestly whether to run it local, in the cloud, or hybrid, and what the build looks like.

Frequently Asked Questions

How do I run an LLM locally?

The fastest way is to install Ollama (ollama.com) or LM Studio, download an open-weight model such as an 8B Llama or Qwen, and run it. With Ollama it is one command: ollama run llama3.1:8b. Both tools also expose an OpenAI-compatible API on localhost so your own code can call the model — and nothing leaves your machine.

What hardware do I need to run a local LLM?

It depends on model size at 4-bit quantization: 1B–3B models run on almost any laptop (2–4 GB); 7B–9B need ~6–8 GB of VRAM or a 16 GB Mac; 13B–14B need ~10–12 GB; 30B–34B need ~20–24 GB (an RTX 3090/4090); and 70B models need ~40–48 GB (dual GPUs or a 64–128 GB Apple Silicon Mac). VRAM or unified memory is the binding constraint — buy memory first.

What is quantization and why does it matter?

Quantization reduces the bits used per model parameter, shrinking memory needs for a small quality trade-off. Full precision (FP16) is ~2 bytes/parameter, so a 7B model is ~14 GB; 8-bit halves that to ~7 GB; and 4-bit (the popular default) is ~3.5–4 GB. As a rule of thumb for 4-bit, budget roughly parameters-in-billions × 0.6 GB, plus 1–4 GB headroom for the context window.

What is the best local LLM in 2026?

There is no single winner — it depends on your hardware and task. For general use, current Llama and Qwen models at 7B–14B are excellent on a single GPU; for coding, dedicated coder-tuned Qwen or DeepSeek models are strong; for near-frontier quality, 70B or large mixture-of-experts models on high-memory hardware. Check the Hugging Face Open LLM Leaderboard and LMArena for the current state rather than any fixed list.

Can I use a local LLM with Claude Code or Cursor?

Yes. Local runtimes like Ollama and vLLM expose an OpenAI-compatible endpoint, so coding tools that let you set a custom base URL and model can point at a local model. Quality on hard tasks still trails the top cloud models, so many developers use local models for fast, private, or offline work and keep a frontier model for the heaviest reasoning.