Guide · Layer 01 · Hardware
Local AI hardware: memory, bandwidth, and what actually fits
The buying decision for local inference is a memory decision first and a bandwidth decision second. This guide gives you the arithmetic behind both, the hardware tiers they map to, and the trade-offs between a discrete GPU and unified memory, without a price list that would be wrong by next quarter.
- Series
- 01 of 09
- Sections
- 07
- Updated
- Sep 14, 2026
- Reading time
- 11 min
- Written for
- Developers + operators
The two numbers that decide everything
Every other spec on a GPU box is secondary to two: how much memory sits next to the compute, and how fast that memory can be read. A model has to fit in the first, and each token it generates is bounded by the second. Get those two right and the rest of the build is preference.
Capacity decides which models you can run at all. The weights of the model, the working memory for its context (the KV cache), and the runtime's own buffers all have to be resident at once. If they do not fit, the runtime either refuses to load or spills layers to system RAM over a bus that is an order of magnitude slower, and generation speed collapses.
Bandwidth decides how fast a fitted model runs. Generating one token means reading every active weight once. A dense 8B model quantized to about 4.85 bits per weight is roughly 4.85 GB of weights, so a card that can read 1,000 GB/s has a theoretical ceiling near 200 tokens per second on that model, and a laptop bus that reads 90 GB/s tops out near 18. Real throughput lands below the ceiling, but the ceiling is what sorts hardware into classes.
How much memory a model needs
The weights are the bulk of it, and the size is a straight multiplication: parameters × bits per weight ÷ 8. The quantization format sets the bits. Full precision is 16 bits; the common 4-bit GGUF format (Q4_K_M) lands around 4.85 bits per weight once its scale factors are counted; 8-bit (Q8_0) is about 8.5.
| Parameters | Q4_K_M (~4.85 bpw) | Q8_0 (~8.5 bpw) | FP16 (16 bpw) |
|---|---|---|---|
| 1B | 0.6 GB | 1.1 GB | 2.0 GB |
| 3B | 1.8 GB | 3.2 GB | 6.0 GB |
| 8B | 4.9 GB | 8.5 GB | 16 GB |
| 14B | 8.5 GB | 14.9 GB | 28 GB |
| 24B | 14.6 GB | 25.5 GB | 48 GB |
| 32B | 19.4 GB | 34 GB | 64 GB |
| 70B | 42.4 GB | 74.4 GB | 140 GB |
Two more lines go on top of the weights. The runtime itself needs working memory for compute buffers, the CUDA or Metal context, and activations, which is usually somewhere between half a gigabyte and a gigabyte and a half depending on the backend. And the context needs a cache, which is the line most people forget.
The KV cache: the budget people forget
Attention keeps a key and a value vector for every token in the context, at every layer. That cache grows linearly with context length, and for long-context work it can be larger than the weights. The size per token is fixed by the model's architecture:
bytes per token = 2 (K and V)
× layers
× KV heads
× head dimension
× bytes per element (2 for FP16, 1 for 8-bit, 0.5 for 4-bit)
cache size = bytes per token × context lengthModels that use grouped-query attention have far fewer KV heads than attention heads, which is what keeps this manageable. Take three published architectures as worked examples:
| Model (published config) | Per token, FP16 | 8k context | 32k context | 128k context |
|---|---|---|---|---|
| Llama 3.1 8B — 32 layers, 8 KV heads, 128 dim | 128 KiB | 1.1 GB | 4.3 GB | 17.2 GB |
| Qwen2.5 7B — 28 layers, 4 KV heads, 128 dim | 56 KiB | 0.5 GB | 1.9 GB | 7.5 GB |
| Llama 3.3 70B — 80 layers, 8 KV heads, 128 dim | 320 KiB | 2.7 GB | 10.7 GB | 42.9 GB |
The practical consequence: an 8B model that comfortably fits a 12 GB card at 8k context does not fit the same card at 128k. If your workload is long documents or agent loops that accumulate context, size for the cache, not just the weights. Most runtimes can quantize the cache to 8-bit with little measurable loss, which is the first lever to pull when context is the constraint.
Discrete GPU or unified memory
There are two credible architectures for a local box, and they trade the two numbers against each other.
Discrete GPUs
A discrete card has the highest bandwidth available to an individual — high-end consumer cards read their memory at roughly 1–1.8 TB/s — but capacity is capped by what fits on the card. Consumer cards top out at 24–32 GB; workstation cards reach 48–96 GB at several times the price. NVIDIA has the deepest software support (CUDA is what every inference server targets first); AMD's ROCm support in llama.cpp and vLLM has matured to the point of being a real option, with a smaller ecosystem.
The result is a box that runs anything that fits very fast, and runs nothing that does not fit. A 24 GB card is the sweet spot for dense models up to about 24B at 4-bit with room for context, or a 32B at a tighter quantization with a short context.
Unified memory
Apple's M-series chips and the newer large-memory x86 APUs share one memory pool between CPU and GPU. Capacity is the headline: Max-class Apple chips are configurable to 128 GB, and Ultra-class chips go higher still, which is enough to hold a 70B model at 8-bit or a large mixture-of-experts model that no consumer GPU can load. Bandwidth is the trade: a Max-class chip reads memory at roughly a third to a half of a top discrete card.
The result is the inverse box: it runs models a GPU cannot hold, at a fraction of the speed a GPU would run a smaller one. For a single developer whose work is large models, long context, or fine-tuning experiments that need memory more than throughput, it is often the better machine. For serving concurrent users, it is not.
| Class | Memory | Bandwidth (published) | What it means |
|---|---|---|---|
| Dual-channel DDR5-5600, CPU only | System RAM | ~90 GB/s | Small models only; fine for 1–4B at Q4 |
| Mid consumer GPU | 12–16 GB | ~300–700 GB/s | 8B–14B at Q4 with moderate context |
| High-end consumer GPU (4090 / 7900 XTX class) | 24 GB | ~960–1,000 GB/s | Up to ~24B at Q4; 32B tight |
| Top consumer GPU (5090 class) | 32 GB | ~1,800 GB/s | 32B at Q4 with room for context |
| Apple Max-class unified | 36–128 GB | ~300–550 GB/s | 70B at Q4/Q8, large MoE; slower per token |
| Apple Ultra-class unified | 96 GB+ | ~800 GB/s | 70B+ at Q8; the largest open models |
| Workstation GPU (RTX 6000 class) | 48–96 GB | ~960–1,800 GB/s | 70B at Q4 on one card, fast |
| Datacenter GPU (A100 / H100 class) | 80 GB | ~2,000–3,350 GB/s | Production serving; multi-user throughput |
Reading a tier, not a part number
Rather than chasing a specific card, decide which tier your work needs and buy the best value in it. The tiers are set by capacity, because that is the hard constraint:
- 8–12 GB. 7B–8B models at 4-bit with 8k–16k context. Enough for a coding assistant on a single repository, classification, extraction, and most RAG over a modest corpus.
- 16–24 GB. 14B–24B at 4-bit, or 8B at 8-bit with long context. The tier where local models stop feeling like a compromise for everyday work.
- 32 GB. 32B at 4-bit with working context. The current ceiling of a single consumer GPU.
- 48–64 GB. 32B at 8-bit, 70B at a tight 4-bit, or mid-size MoE models. Reached with a workstation card, two consumer cards, or unified memory.
- 96–128 GB and up. 70B at 8-bit, the largest open MoE models, and fine-tuning experiments that need headroom. Unified memory or multiple cards.
Run the numbers for your own model and context before committing to a tier:
- Weights
- 4.87 GB
- KV cache · 16k
- 2.15 GB
- Runtime overhead
- 1.00 GB
- Total resident
- 8.02 GB
Smallest tier with 15% headroom: 12 GB · mid GPU9.22 GB needed · 131,072 bytes of cache per token
Multiple GPUs
Two 24 GB cards are not one 48 GB card, but they are close enough for inference. llama.cpp splits layers across cards with --tensor-split; vLLM shards with tensor parallelism. The cost is the interconnect: consumer cards talk over PCIe, and a split model moves activations across it on every token. For a layer split the overhead is small; for tensor parallelism it is noticeable without a fast interconnect. Plan for the second card to add capacity, not to double speed.
Power, heat, and noise
A high-end card draws 350–575 W under load and the rest of the box adds another 100–200 W. That is a space heater in a home office and a real line on the electricity bill at sustained utilization, which is why the cost model carries power as its own term. Inference is bursty, so average draw is far below peak, but plan the power supply and the cooling for the peak. If the box shares a room with a person, consider a case and fan curve that stay quiet at idle, because a local model is idle most of the time.
Decision rules
- Size for the model you want to run in a year, not the one you are testing today. Model sizes at a given quality have been trending down, but context lengths and agent workloads have been trending up, and the KV cache is where that lands.
- If your work is interactive and single-user, buy bandwidth. If your work is large models and long context, buy capacity. If you are serving other people, buy a datacenter-class card or rent one.
- Prefer one large card to two small ones for the same total memory. The interconnect is never free.
- Leave 10–20% headroom above the calculator's number. Runtimes allocate more than the arithmetic suggests, and a model that barely fits is a model that fails on the first long prompt.
- Quantize the KV cache before quantizing the weights further. It is the cheaper loss.
Once the box is chosen, the next decision is what runs on it. The inference server guide covers that, and the quantization guide explains what the bits-per-weight number costs you in practice.