Guide · Layer 02 · Runtime
Ollama vs vLLM vs llama.cpp: choosing a local inference server
Three runtimes account for most local inference, and they are not competitors so much as answers to different questions. One is built for a single developer to get a model running in a minute, one is built to serve many users from one GPU, and one is the engine both of the others learned from. This guide sets out which question each one answers, and why most teams end up running two.
- Series
- 02 of 09
- Sections
- 07
- Updated
- Sep 14, 2026
- Reading time
- 10 min
- Written for
- developers
What each one is
llama.cpp
A C/C++ inference engine that runs models in the GGUF format on almost anything: CPUs, NVIDIA and AMD GPUs, Apple Silicon through Metal, and Vulkan for the rest. It is the reference implementation for quantized inference on consumer hardware, and it exposes every knob: how many layers go to the GPU, the context length, the KV cache precision, the batch size, speculative decoding with a draft model. Its llama-server binary serves an OpenAI-compatible HTTP API.
Ollama
A packaging and management layer around a llama.cpp-derived engine. It adds a model registry (ollama pull), a Modelfile format for baking in a system prompt and parameters, automatic GPU offload, a desktop app, and an OpenAI-compatible API on port 11434. It is the fastest route from nothing to a running model, at the cost of hiding most of the knobs and adding some overhead above the raw engine.
vLLM
A Python serving engine built for throughput. Its two defining techniques are PagedAttention, which manages KV cache memory in blocks so many sequences can share a GPU without fragmentation, and continuous batching, which keeps the GPU busy by admitting new requests mid-batch instead of waiting for the slowest one to finish. It loads standard safetensors weights at FP16, BF16, FP8, AWQ, or GPTQ, shards across GPUs with tensor parallelism, and serves an OpenAI-compatible API with Prometheus metrics.
Adjacent projects worth knowing: SGLang covers the same ground as vLLM with a strong prefix-caching design that helps agent workloads; MLX and mlx-lm are Apple's native stack for M-series chips; LM Studio is a desktop application that wraps both llama.cpp and MLX engines behind a GUI and a local server; TabbyAPI and ExLlama serve the EXL2 format on NVIDIA cards.
The question that sorts them: how many users
A single person chatting with a model sends one request at a time. The GPU is idle between keystrokes and the only number that matters is tokens per second on that one stream. Here, llama.cpp and Ollama are the right shape, and the difference between them is convenience against control. vLLM works for one user too, but its setup floor, a Linux machine with a supported GPU and a Python environment, is paid for nothing.
Ten users, or one application making parallel calls, changes the problem entirely. A single-stream engine serves them one after another and total throughput barely rises. A batching engine serves them together, and total throughput rises with concurrency until the GPU's compute saturates. This is the regime vLLM and SGLang were built for, and in it they are not a little faster than a single-stream engine; they are a different category.
Model formats decide which engine you can use
| Format | Used by | Notes |
|---|---|---|
| GGUF (Q4_K_M, Q8_0, IQ4_XS…) | llama.cpp, Ollama, LM Studio | Quantized for consumer hardware; CPU + GPU + Metal; one file per model |
| safetensors FP16 / BF16 | vLLM, SGLang, transformers | Full precision; needs the most memory; the source most quantizations start from |
| AWQ, GPTQ (4-bit) | vLLM, SGLang, transformers | GPU-only quantizations with fast kernels; good throughput at 4-bit |
| FP8 | vLLM, SGLang (recent NVIDIA / AMD) | Near-lossless on hardware with FP8 units; halves memory against FP16 |
| EXL2 | ExLlama, TabbyAPI | Flexible bits-per-weight on NVIDIA; single-user oriented |
| MLX | mlx-lm, LM Studio | Apple Silicon native; 4-bit and 8-bit variants |
The practical consequence is that the choice of engine and the choice of quantization are the same choice. A GGUF file does not load in vLLM without conversion, and an AWQ model does not run on a CPU. Pick the engine for the workload, then find or make the model in the format it wants. The quantization guide covers what each format costs in quality.
The same API on every one of them
All three serve the OpenAI chat-completions shape, which is why an application can be written once against http://localhost:PORT/v1 and moved between engines by changing a base URL. The details differ (streaming is universal; tool calling, JSON mode, and structured outputs depend on the engine and on the model's chat template) but the surface is shared, and every editor integration and agent framework that speaks OpenAI speaks all three.
# Ollama — one command from nothing to a running model
ollama pull llama3.1:8b
ollama run llama3.1:8b
# API on http://localhost:11434/v1
# llama.cpp — the same model, every knob exposed
llama-server -m Llama-3.1-8B-Instruct-Q4_K_M.gguf \
--ctx-size 16384 --n-gpu-layers 99 \
--cache-type-k q8_0 --cache-type-v q8_0 \
--port 8080
# API on http://localhost:8080/v1
# vLLM — a batching server for an application
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
--max-model-len 16384 --gpu-memory-utilization 0.90
# API on http://localhost:8000/v1, metrics on /metricscurl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "local",
"messages": [{"role": "user", "content": "Summarize this ticket in one line."}],
"stream": true
}'Where each one runs
| llama.cpp | Ollama | vLLM | |
|---|---|---|---|
| CPU only | Yes, and well | Yes | Experimental at best |
| NVIDIA (CUDA) | Yes | Yes | Yes — primary target |
| AMD (ROCm) | Yes | Yes | Yes |
| Apple Silicon (Metal) | Yes | Yes | No |
| Windows | Yes | Yes | Via WSL |
| Multi-GPU | Layer split | Layer split | Tensor parallel |
| Concurrency model | Parallel slots | Parallel slots | Continuous batching |
| Metrics endpoint | Basic | Basic | Prometheus |
The two-server pattern
Most teams settle on two runtimes rather than one, and it is the right outcome rather than a failure to decide. On a developer's machine, the job is iteration: pull a model, try a prompt, swap the model, try again. Ollama or LM Studio is the tool. Behind an application, the job is throughput and observability at a predictable memory footprint. vLLM or SGLang is the tool. Because the API surface is shared, the code does not change between them; only the base URL and the model name do.
The one thing to keep constant between the two is the model. Evaluate your prompts against the same weights at the same quantization you will serve, or the local-machine results will not predict the production ones. Ollama's registry and vLLM's Hugging Face loading will happily give you two different quantizations of the "same" model, and the difference is measurable on hard tasks.
Knobs that matter on day one
- Context length. Every engine defaults it, and the default is often shorter than the model supports. Set it explicitly to what the workload needs and no more; the KV cache is charged per token of context whether or not you use it.
- GPU offload. In llama.cpp and Ollama, make sure every layer is on the GPU. A single layer left on the CPU can halve throughput.
- KV cache precision. 8-bit cache is the first thing to enable when memory is tight; it is a small, well-understood loss.
- Flash attention. On for any GPU that supports it; it reduces memory traffic during attention and is usually a clear win.
- Speculative decoding. A small draft model proposes tokens the large model verifies in one pass. On workloads with predictable text (code, structured output) it can substantially raise single-stream speed at no quality cost.
- Memory utilization (vLLM). Set
--gpu-memory-utilizationdeliberately. It pre-allocates the KV cache pool, and the default leaves less room for the model than a tight quantization might need.
A local coding assistant is the workload where these settings show up most, because it sends long contexts many times a minute. The coding agent guide walks through a configuration for it.