Guide · Layer 02 · Runtime

Ollama vs vLLM vs llama.cpp: choosing a local inference server

Three runtimes account for most local inference, and they are not competitors so much as answers to different questions. One is built for a single developer to get a model running in a minute, one is built to serve many users from one GPU, and one is the engine both of the others learned from. This guide sets out which question each one answers, and why most teams end up running two.

One runtime, many clients
A server loads the weights once and answers one OpenAI-shaped endpoint for every client.
Series
02 of 09
Sections
07
Updated
Sep 14, 2026
Reading time
10 min
Written for
developers

What each one is

llama.cpp

A C/C++ inference engine that runs models in the GGUF format on almost anything: CPUs, NVIDIA and AMD GPUs, Apple Silicon through Metal, and Vulkan for the rest. It is the reference implementation for quantized inference on consumer hardware, and it exposes every knob: how many layers go to the GPU, the context length, the KV cache precision, the batch size, speculative decoding with a draft model. Its llama-server binary serves an OpenAI-compatible HTTP API.

Ollama

A packaging and management layer around a llama.cpp-derived engine. It adds a model registry (ollama pull), a Modelfile format for baking in a system prompt and parameters, automatic GPU offload, a desktop app, and an OpenAI-compatible API on port 11434. It is the fastest route from nothing to a running model, at the cost of hiding most of the knobs and adding some overhead above the raw engine.

vLLM

A Python serving engine built for throughput. Its two defining techniques are PagedAttention, which manages KV cache memory in blocks so many sequences can share a GPU without fragmentation, and continuous batching, which keeps the GPU busy by admitting new requests mid-batch instead of waiting for the slowest one to finish. It loads standard safetensors weights at FP16, BF16, FP8, AWQ, or GPTQ, shards across GPUs with tensor parallelism, and serves an OpenAI-compatible API with Prometheus metrics.

Adjacent projects worth knowing: SGLang covers the same ground as vLLM with a strong prefix-caching design that helps agent workloads; MLX and mlx-lm are Apple's native stack for M-series chips; LM Studio is a desktop application that wraps both llama.cpp and MLX engines behind a GUI and a local server; TabbyAPI and ExLlama serve the EXL2 format on NVIDIA cards.

The question that sorts them: how many users

A single person chatting with a model sends one request at a time. The GPU is idle between keystrokes and the only number that matters is tokens per second on that one stream. Here, llama.cpp and Ollama are the right shape, and the difference between them is convenience against control. vLLM works for one user too, but its setup floor, a Linux machine with a supported GPU and a Python environment, is paid for nothing.

Ten users, or one application making parallel calls, changes the problem entirely. A single-stream engine serves them one after another and total throughput barely rises. A batching engine serves them together, and total throughput rises with concurrency until the GPU's compute saturates. This is the regime vLLM and SGLang were built for, and in it they are not a little faster than a single-stream engine; they are a different category.

Model formats decide which engine you can use

FormatUsed byNotes
GGUF (Q4_K_M, Q8_0, IQ4_XS…)llama.cpp, Ollama, LM StudioQuantized for consumer hardware; CPU + GPU + Metal; one file per model
safetensors FP16 / BF16vLLM, SGLang, transformersFull precision; needs the most memory; the source most quantizations start from
AWQ, GPTQ (4-bit)vLLM, SGLang, transformersGPU-only quantizations with fast kernels; good throughput at 4-bit
FP8vLLM, SGLang (recent NVIDIA / AMD)Near-lossless on hardware with FP8 units; halves memory against FP16
EXL2ExLlama, TabbyAPIFlexible bits-per-weight on NVIDIA; single-user oriented
MLXmlx-lm, LM StudioApple Silicon native; 4-bit and 8-bit variants
Table 01 Format support as documented by each project. Check the project's current documentation for the exact list; support expands with releases.

The practical consequence is that the choice of engine and the choice of quantization are the same choice. A GGUF file does not load in vLLM without conversion, and an AWQ model does not run on a CPU. Pick the engine for the workload, then find or make the model in the format it wants. The quantization guide covers what each format costs in quality.

The same API on every one of them

All three serve the OpenAI chat-completions shape, which is why an application can be written once against http://localhost:PORT/v1 and moved between engines by changing a base URL. The details differ (streaming is universal; tool calling, JSON mode, and structured outputs depend on the engine and on the model's chat template) but the surface is shared, and every editor integration and agent framework that speaks OpenAI speaks all three.

three servers, one APIbash
# Ollama — one command from nothing to a running model
ollama pull llama3.1:8b
ollama run llama3.1:8b
# API on http://localhost:11434/v1

# llama.cpp — the same model, every knob exposed
llama-server -m Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  --ctx-size 16384 --n-gpu-layers 99 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --port 8080
# API on http://localhost:8080/v1

# vLLM — a batching server for an application
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
  --max-model-len 16384 --gpu-memory-utilization 0.90
# API on http://localhost:8000/v1, metrics on /metrics
the call is identicalbash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Summarize this ticket in one line."}],
    "stream": true
  }'

Where each one runs

llama.cppOllamavLLM
CPU onlyYes, and wellYesExperimental at best
NVIDIA (CUDA)YesYesYes — primary target
AMD (ROCm)YesYesYes
Apple Silicon (Metal)YesYesNo
WindowsYesYesVia WSL
Multi-GPULayer splitLayer splitTensor parallel
Concurrency modelParallel slotsParallel slotsContinuous batching
Metrics endpointBasicBasicPrometheus
Table 02 Platform support as documented by each project at time of writing.

The two-server pattern

Most teams settle on two runtimes rather than one, and it is the right outcome rather than a failure to decide. On a developer's machine, the job is iteration: pull a model, try a prompt, swap the model, try again. Ollama or LM Studio is the tool. Behind an application, the job is throughput and observability at a predictable memory footprint. vLLM or SGLang is the tool. Because the API surface is shared, the code does not change between them; only the base URL and the model name do.

The one thing to keep constant between the two is the model. Evaluate your prompts against the same weights at the same quantization you will serve, or the local-machine results will not predict the production ones. Ollama's registry and vLLM's Hugging Face loading will happily give you two different quantizations of the "same" model, and the difference is measurable on hard tasks.

Knobs that matter on day one

  • Context length. Every engine defaults it, and the default is often shorter than the model supports. Set it explicitly to what the workload needs and no more; the KV cache is charged per token of context whether or not you use it.
  • GPU offload. In llama.cpp and Ollama, make sure every layer is on the GPU. A single layer left on the CPU can halve throughput.
  • KV cache precision. 8-bit cache is the first thing to enable when memory is tight; it is a small, well-understood loss.
  • Flash attention. On for any GPU that supports it; it reduces memory traffic during attention and is usually a clear win.
  • Speculative decoding. A small draft model proposes tokens the large model verifies in one pass. On workloads with predictable text (code, structured output) it can substantially raise single-stream speed at no quality cost.
  • Memory utilization (vLLM). Set --gpu-memory-utilization deliberately. It pre-allocates the KV cache pool, and the default leaves less room for the model than a tight quantization might need.

A local coding assistant is the workload where these settings show up most, because it sends long contexts many times a minute. The coding agent guide walks through a configuration for it.

Questions people ask about this.

Answered in full, in plain text — the same words the page's structured data carries.

  1. Is Ollama slower than llama.cpp?

    Slightly, on the same hardware and the same model file, because Ollama is a management layer over a llama.cpp-derived engine and adds its own overhead. For a single user the difference is rarely noticeable in practice. Where the gap opens is configuration: llama.cpp exposes context, cache precision, offload, and speculative decoding directly, and a tuned llama-server can outrun a default Ollama setup by a meaningful margin on the same card.

  2. Can vLLM run GGUF models?

    vLLM has experimental GGUF loading, but it is not the format it is designed around and performance and compatibility lag well behind safetensors, AWQ, GPTQ, and FP8. If the model you want exists only as GGUF, run it in llama.cpp or Ollama; if you need vLLM's batching, find or produce an AWQ or FP8 version of the same weights.

  3. Which runtime should a single developer start with?

    Ollama, or LM Studio if a GUI is preferred. Either gets a model running in minutes with an API an editor can talk to. Move to llama-server when you need a specific setting (long context, cache quantization, a draft model), and to vLLM or SGLang only when an application, not a person, is the client.

Next step

Send the stack you have.

Describe the model, the data and the hardware in a few lines. Strygon replies with what the architecture likely needs, before any talk of price.