Guide · Layer 01 · Hardware

Local AI hardware: memory, bandwidth, and what actually fits

The buying decision for local inference is a memory decision first and a bandwidth decision second. This guide gives you the arithmetic behind both, the hardware tiers they map to, and the trade-offs between a discrete GPU and unified memory, without a price list that would be wrong by next quarter.

The package
Memory stacked beside the compute. Capacity decides what fits; bandwidth decides how fast it runs.
Series
01 of 09
Sections
07
Updated
Sep 14, 2026
Reading time
11 min
Written for
Developers + operators

The two numbers that decide everything

Every other spec on a GPU box is secondary to two: how much memory sits next to the compute, and how fast that memory can be read. A model has to fit in the first, and each token it generates is bounded by the second. Get those two right and the rest of the build is preference.

Capacity decides which models you can run at all. The weights of the model, the working memory for its context (the KV cache), and the runtime's own buffers all have to be resident at once. If they do not fit, the runtime either refuses to load or spills layers to system RAM over a bus that is an order of magnitude slower, and generation speed collapses.

Bandwidth decides how fast a fitted model runs. Generating one token means reading every active weight once. A dense 8B model quantized to about 4.85 bits per weight is roughly 4.85 GB of weights, so a card that can read 1,000 GB/s has a theoretical ceiling near 200 tokens per second on that model, and a laptop bus that reads 90 GB/s tops out near 18. Real throughput lands below the ceiling, but the ceiling is what sorts hardware into classes.

How much memory a model needs

The weights are the bulk of it, and the size is a straight multiplication: parameters × bits per weight ÷ 8. The quantization format sets the bits. Full precision is 16 bits; the common 4-bit GGUF format (Q4_K_M) lands around 4.85 bits per weight once its scale factors are counted; 8-bit (Q8_0) is about 8.5.

ParametersQ4_K_M (~4.85 bpw)Q8_0 (~8.5 bpw)FP16 (16 bpw)
1B0.6 GB1.1 GB2.0 GB
3B1.8 GB3.2 GB6.0 GB
8B4.9 GB8.5 GB16 GB
14B8.5 GB14.9 GB28 GB
24B14.6 GB25.5 GB48 GB
32B19.4 GB34 GB64 GB
70B42.4 GB74.4 GB140 GB
Table 01 Weights only, computed from parameters × bits ÷ 8 in decimal gigabytes. Bits-per-weight values are the approximate effective sizes of llama.cpp's quantization types; a specific file may differ by a few percent. Add the KV cache and runtime overhead below before comparing to a card.

Two more lines go on top of the weights. The runtime itself needs working memory for compute buffers, the CUDA or Metal context, and activations, which is usually somewhere between half a gigabyte and a gigabyte and a half depending on the backend. And the context needs a cache, which is the line most people forget.

The memory budget
Weights, KV cache and runtime buffers stacked against the memory tier, with headroom on top. Proportions are a sketch; the calculator further down does the arithmetic for a real model.

The KV cache: the budget people forget

Attention keeps a key and a value vector for every token in the context, at every layer. That cache grows linearly with context length, and for long-context work it can be larger than the weights. The size per token is fixed by the model's architecture:

kv-cachetext
bytes per token = 2 (K and V)
                × layers
                × KV heads
                × head dimension
                × bytes per element   (2 for FP16, 1 for 8-bit, 0.5 for 4-bit)

cache size = bytes per token × context length

Models that use grouped-query attention have far fewer KV heads than attention heads, which is what keeps this manageable. Take three published architectures as worked examples:

Model (published config)Per token, FP168k context32k context128k context
Llama 3.1 8B — 32 layers, 8 KV heads, 128 dim128 KiB1.1 GB4.3 GB17.2 GB
Qwen2.5 7B — 28 layers, 4 KV heads, 128 dim56 KiB0.5 GB1.9 GB7.5 GB
Llama 3.3 70B — 80 layers, 8 KV heads, 128 dim320 KiB2.7 GB10.7 GB42.9 GB
Table 02 Arithmetic from the formula above using each model's published configuration. Sizes are decimal gigabytes at FP16 cache precision. Runtimes that quantize the cache to 8-bit halve these; models using sliding-window attention on some layers need less than the formula suggests.
Cache against context
The cache grows linearly with context length while the weights stay fixed. Bar lengths are proportional to the context, for any one model.

The practical consequence: an 8B model that comfortably fits a 12 GB card at 8k context does not fit the same card at 128k. If your workload is long documents or agent loops that accumulate context, size for the cache, not just the weights. Most runtimes can quantize the cache to 8-bit with little measurable loss, which is the first lever to pull when context is the constraint.

Discrete GPU or unified memory

There are two credible architectures for a local box, and they trade the two numbers against each other.

Discrete GPUs

A discrete card has the highest bandwidth available to an individual — high-end consumer cards read their memory at roughly 1–1.8 TB/s — but capacity is capped by what fits on the card. Consumer cards top out at 24–32 GB; workstation cards reach 48–96 GB at several times the price. NVIDIA has the deepest software support (CUDA is what every inference server targets first); AMD's ROCm support in llama.cpp and vLLM has matured to the point of being a real option, with a smaller ecosystem.

The result is a box that runs anything that fits very fast, and runs nothing that does not fit. A 24 GB card is the sweet spot for dense models up to about 24B at 4-bit with room for context, or a 32B at a tighter quantization with a short context.

Unified memory

Apple's M-series chips and the newer large-memory x86 APUs share one memory pool between CPU and GPU. Capacity is the headline: Max-class Apple chips are configurable to 128 GB, and Ultra-class chips go higher still, which is enough to hold a 70B model at 8-bit or a large mixture-of-experts model that no consumer GPU can load. Bandwidth is the trade: a Max-class chip reads memory at roughly a third to a half of a top discrete card.

The result is the inverse box: it runs models a GPU cannot hold, at a fraction of the speed a GPU would run a smaller one. For a single developer whose work is large models, long context, or fine-tuning experiments that need memory more than throughput, it is often the better machine. For serving concurrent users, it is not.

ClassMemoryBandwidth (published)What it means
Dual-channel DDR5-5600, CPU onlySystem RAM~90 GB/sSmall models only; fine for 1–4B at Q4
Mid consumer GPU12–16 GB~300–700 GB/s8B–14B at Q4 with moderate context
High-end consumer GPU (4090 / 7900 XTX class)24 GB~960–1,000 GB/sUp to ~24B at Q4; 32B tight
Top consumer GPU (5090 class)32 GB~1,800 GB/s32B at Q4 with room for context
Apple Max-class unified36–128 GB~300–550 GB/s70B at Q4/Q8, large MoE; slower per token
Apple Ultra-class unified96 GB+~800 GB/s70B+ at Q8; the largest open models
Workstation GPU (RTX 6000 class)48–96 GB~960–1,800 GB/s70B at Q4 on one card, fast
Datacenter GPU (A100 / H100 class)80 GB~2,000–3,350 GB/sProduction serving; multi-user throughput
Table 03 Memory and bandwidth are the vendors' published specifications for representative parts in each class, rounded. Check the current spec sheet for the exact part you are considering; generations change these numbers.

Reading a tier, not a part number

Rather than chasing a specific card, decide which tier your work needs and buy the best value in it. The tiers are set by capacity, because that is the hard constraint:

  1. 8–12 GB. 7B–8B models at 4-bit with 8k–16k context. Enough for a coding assistant on a single repository, classification, extraction, and most RAG over a modest corpus.
  2. 16–24 GB. 14B–24B at 4-bit, or 8B at 8-bit with long context. The tier where local models stop feeling like a compromise for everyday work.
  3. 32 GB. 32B at 4-bit with working context. The current ceiling of a single consumer GPU.
  4. 48–64 GB. 32B at 8-bit, 70B at a tight 4-bit, or mid-size MoE models. Reached with a workstation card, two consumer cards, or unified memory.
  5. 96–128 GB and up. 70B at 8-bit, the largest open MoE models, and fine-tuning experiments that need headroom. Unified memory or multiple cards.

Run the numbers for your own model and context before committing to a tier:

Model fitweights + cache + runtime
Context length
KV cache precision
Weights
4.87 GB
KV cache · 16k
2.15 GB
Runtime overhead
1.00 GB
Total resident
8.02 GB

Smallest tier with 15% headroom: 12 GB · mid GPU9.22 GB needed · 131,072 bytes of cache per token

Arithmetic only · presets are published configsNo benchmark is implied

Multiple GPUs

Two 24 GB cards are not one 48 GB card, but they are close enough for inference. llama.cpp splits layers across cards with --tensor-split; vLLM shards with tensor parallelism. The cost is the interconnect: consumer cards talk over PCIe, and a split model moves activations across it on every token. For a layer split the overhead is small; for tensor parallelism it is noticeable without a fast interconnect. Plan for the second card to add capacity, not to double speed.

Power, heat, and noise

A high-end card draws 350–575 W under load and the rest of the box adds another 100–200 W. That is a space heater in a home office and a real line on the electricity bill at sustained utilization, which is why the cost model carries power as its own term. Inference is bursty, so average draw is far below peak, but plan the power supply and the cooling for the peak. If the box shares a room with a person, consider a case and fan curve that stay quiet at idle, because a local model is idle most of the time.

Decision rules

  • Size for the model you want to run in a year, not the one you are testing today. Model sizes at a given quality have been trending down, but context lengths and agent workloads have been trending up, and the KV cache is where that lands.
  • If your work is interactive and single-user, buy bandwidth. If your work is large models and long context, buy capacity. If you are serving other people, buy a datacenter-class card or rent one.
  • Prefer one large card to two small ones for the same total memory. The interconnect is never free.
  • Leave 10–20% headroom above the calculator's number. Runtimes allocate more than the arithmetic suggests, and a model that barely fits is a model that fails on the first long prompt.
  • Quantize the KV cache before quantizing the weights further. It is the cheaper loss.

Once the box is chosen, the next decision is what runs on it. The inference server guide covers that, and the quantization guide explains what the bits-per-weight number costs you in practice.

Questions people ask about this.

Answered in full, in plain text — the same words the page's structured data carries.

  1. Is a Mac with unified memory good for running local LLMs?

    For a single developer who needs to run large models or long contexts, yes: a Max-class machine holds models that no consumer GPU can fit, and the Metal backends in llama.cpp and MLX are mature. The trade is speed. Memory bandwidth on unified-memory chips is a fraction of a top discrete GPU's, so tokens per second on any model that fits both machines will be lower on the Mac. For serving concurrent users, a discrete GPU or a rented datacenter card is the better tool.

  2. How much VRAM does a 7B or 8B model need?

    At 4-bit quantization, about 5 GB for the weights, plus the KV cache for your context and roughly a gigabyte of runtime overhead. On an 8 GB card that leaves room for a short context; on 12 GB it runs comfortably at 16k context; at 8-bit weights you want 12–16 GB. The fit calculator on this page does the exact arithmetic for a given context length and cache precision.

  3. Does a faster CPU or more system RAM help GPU inference?

    Only at the margins, as long as the model fits in GPU memory. Prompt processing and generation both happen on the card. System RAM matters if the model does not fit and layers spill over, but at that point the fix is a different model or a different card, not more RAM. CPU-only inference is a different regime: there, memory bandwidth of the system RAM is the ceiling, and it is low.

Next step

Send the stack you have.

Describe the model, the data and the hardware in a few lines. Strygon replies with what the architecture likely needs, before any talk of price.