Guide · Layer 06 · Application
Running a local coding agent: models, context, and guardrails
A coding agent is the local workload that stresses every part of the stack at once: long contexts sent many times a minute, tool calls that have to be parsed exactly, and outputs that must compile. It is also the one where keeping the model local has the clearest justification, because the input is your source code. This guide covers the model, the serving configuration, the editor and tool integration, and the guardrails that stop an autonomous agent from doing something you did not ask for.
- Series
- 08 of 09
- Sections
- 07
- Updated
- Sep 14, 2026
- Reading time
- 10 min
- Written for
- developers
What a coding workload needs
Three things, in order. Context: the agent reads files, diffs, test output, and its own prior turns, and a session accumulates tens of thousands of tokens quickly. Tool discipline: an agent works by calling tools (read a file, run a command, apply an edit) and the model has to produce those calls in the exact structure the harness expects, every time. Speed: an agent loop makes many calls per task, so tokens per second is multiplied by the number of turns.
Those map directly onto the hardware guide: context is KV cache memory, tool discipline is model quality at the chosen quantization, and speed is bandwidth. A coding agent is the workload where a 24 GB card earns its cost over a 12 GB one.
Choosing the model
Coding models are a distinct class, trained on far more code than a general chat model of the same size, and the difference is large. Every major open-weight family ships a code variant, and the state of the art moves every few months, so the right move is to choose by tier and evaluate the current release in that tier rather than name a winner here:
| Memory tier | Model class | What it handles |
|---|---|---|
| 8–12 GB | 7B–8B coder at Q4, ~16k context | Completion, single-file edits, explaining code, tests for small functions |
| 16–24 GB | 14B–32B coder at Q4, 32k context | Multi-file changes, refactors within a module, agent loops on a small repository |
| 32–48 GB | 32B at Q6/Q8 or a mid MoE, 64k+ context | Whole-repository reasoning, longer agent runs, reliable tool calling |
| 96 GB+ | 70B+ or large MoE at Q4, long context | Approaching hosted-model quality on most everyday tasks |
Two properties to check before anything else. The model must have been trained for tool calling in the format your harness uses, or the agent will fail on formatting rather than on reasoning. And its context window must genuinely hold the sessions you run; a model that advertises 128k and degrades sharply past 32k is a 32k model for this purpose. The quantization guide's evaluation method applies: a small set of real tasks from your codebase, scored by whether the tests pass.
Budgeting the context
Every token in the context costs KV cache memory and is re-read on every turn. The agent harness decides what goes in, and the defaults are often greedy. Three habits keep it in budget:
- Send a map, not the repository. A file tree with signatures lets the model ask for the files it needs instead of receiving all of them.
- Truncate tool output. Test logs and build output are long and mostly noise; a harness that trims them to the failing lines saves thousands of tokens per turn.
- Summarize old turns. Once a task has run for twenty turns, the early turns are history; a rolling summary replaces them at a fraction of the size.
Serve with the context the workload actually needs. A 32k context on an 8B model at FP16 cache is a few gigabytes; the same on a 32B model is far more. Quantize the cache to 8-bit before shrinking the model.
Serving it
Every coding tool that supports local models talks to an OpenAI-compatible endpoint, so the runtime choice is about the machine, not the editor. On a developer workstation, llama-server with the settings below is the tuned option and Ollama the convenient one; for a team sharing one GPU, vLLM or SGLang, whose prefix caching pays off on agent loops that resend the same system prompt and file context every turn.
llama-server \
-m Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf \
--ctx-size 32768 --n-gpu-layers 99 \
--cache-type-k q8_0 --cache-type-v q8_0 \
--flash-attn on \
--model-draft Qwen2.5-Coder-1.5B-Instruct-Q8_0.gguf \
--port 8080Point the editor at the endpoint. Continue, Cline, Aider, and Zed all accept a base URL and model name; most others do too. Keep the model name stable across tools so an evaluation on one predicts the others.
Tools through MCP
The Model Context Protocol has become the common way to give a model tools: a small server exposes functions (read a file, query a database, run a linter) with typed schemas, and any MCP-aware harness can call them. For a local agent this is the seam where capability is added without touching the model. Run MCP servers locally too; a tool that phones home defeats the point of a local model.
{
"mcpServers": {
"repo": { "command": "mcp-server-git", "args": ["--repository", "."] },
"db": { "command": "mcp-server-sqlite", "args": ["--db", "./dev.sqlite"] }
}
}Guardrails for an agent that acts
An assistant suggests; an agent runs commands. The difference is what makes an agent useful and what makes it dangerous, and the guardrails are not optional:
- A sandbox. The agent runs in a container or VM with only the repository mounted and no credentials it does not need. A local model does not exfiltrate data, but a shell command it writes can.
- A separate worktree or branch. Every task runs on its own branch. Nothing the agent does touches main until a person has read the diff.
- Command allowlists. The harness executes only commands on a list (test runners, linters, the package manager) and asks before anything else.
rm, network calls, and anything touching secrets stay on the ask list. - Secrets scanning on the way out. Before a diff is presented, scan it for keys, tokens, and personal data. Agents copy what they read.
- Budgets. A cap on turns, on tokens, and on wall-clock time per task. A loop that has not converged in forty turns is not going to.
- A review gate. The output is a pull request, and a person merges it. The agent's job ends at the diff.
Measuring whether it is useful
Leaderboards measure models on public benchmarks; the question is whether the agent completes tasks in your repository. Build a set of ten to thirty real tasks with a checkable outcome (a failing test that should pass, a lint error that should clear, a feature with an acceptance test), run the agent against each from a clean branch, and record the pass rate, the number of turns, and the tokens used. Rerun on every model or configuration change. A model that scores higher on a public benchmark and lower on this set is the wrong model.
The strategy for independent developers covers where a local coding agent fits in a solo workflow and what it is worth against a hosted one. For teams that need a private coding stack engineered and operated, that is a scoped AI engineering engagement.