Guide · Layer 06 · Application

Running a local coding agent: models, context, and guardrails

A coding agent is the local workload that stresses every part of the stack at once: long contexts sent many times a minute, tool calls that have to be parsed exactly, and outputs that must compile. It is also the one where keeping the model local has the clearest justification, because the input is your source code. This guide covers the model, the serving configuration, the editor and tool integration, and the guardrails that stop an autonomous agent from doing something you did not ask for.

The agent loop
An editor, a terminal and a loop — edit, run, test — against a model on a local endpoint.
Series
08 of 09
Sections
07
Updated
Sep 14, 2026
Reading time
10 min
Written for
developers

What a coding workload needs

Three things, in order. Context: the agent reads files, diffs, test output, and its own prior turns, and a session accumulates tens of thousands of tokens quickly. Tool discipline: an agent works by calling tools (read a file, run a command, apply an edit) and the model has to produce those calls in the exact structure the harness expects, every time. Speed: an agent loop makes many calls per task, so tokens per second is multiplied by the number of turns.

Those map directly onto the hardware guide: context is KV cache memory, tool discipline is model quality at the chosen quantization, and speed is bandwidth. A coding agent is the workload where a 24 GB card earns its cost over a 12 GB one.

Choosing the model

Coding models are a distinct class, trained on far more code than a general chat model of the same size, and the difference is large. Every major open-weight family ships a code variant, and the state of the art moves every few months, so the right move is to choose by tier and evaluate the current release in that tier rather than name a winner here:

Memory tierModel classWhat it handles
8–12 GB7B–8B coder at Q4, ~16k contextCompletion, single-file edits, explaining code, tests for small functions
16–24 GB14B–32B coder at Q4, 32k contextMulti-file changes, refactors within a module, agent loops on a small repository
32–48 GB32B at Q6/Q8 or a mid MoE, 64k+ contextWhole-repository reasoning, longer agent runs, reliable tool calling
96 GB+70B+ or large MoE at Q4, long contextApproaching hosted-model quality on most everyday tasks
Table 01 Tiers follow the memory arithmetic in the hardware guide; which specific model is best in a tier changes with each release cycle. Evaluate on your own repository.

Two properties to check before anything else. The model must have been trained for tool calling in the format your harness uses, or the agent will fail on formatting rather than on reasoning. And its context window must genuinely hold the sessions you run; a model that advertises 128k and degrades sharply past 32k is a 32k model for this purpose. The quantization guide's evaluation method applies: a small set of real tasks from your codebase, scored by whether the tests pass.

Budgeting the context

Every token in the context costs KV cache memory and is re-read on every turn. The agent harness decides what goes in, and the defaults are often greedy. Three habits keep it in budget:

  • Send a map, not the repository. A file tree with signatures lets the model ask for the files it needs instead of receiving all of them.
  • Truncate tool output. Test logs and build output are long and mostly noise; a harness that trims them to the failing lines saves thousands of tokens per turn.
  • Summarize old turns. Once a task has run for twenty turns, the early turns are history; a rolling summary replaces them at a fraction of the size.

Serve with the context the workload actually needs. A 32k context on an 8B model at FP16 cache is a few gigabytes; the same on a 32B model is far more. Quantize the cache to 8-bit before shrinking the model.

Serving it

Every coding tool that supports local models talks to an OpenAI-compatible endpoint, so the runtime choice is about the machine, not the editor. On a developer workstation, llama-server with the settings below is the tuned option and Ollama the convenient one; for a team sharing one GPU, vLLM or SGLang, whose prefix caching pays off on agent loops that resend the same system prompt and file context every turn.

a workstation configurationbash
llama-server \
  -m Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf \
  --ctx-size 32768 --n-gpu-layers 99 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --flash-attn on \
  --model-draft Qwen2.5-Coder-1.5B-Instruct-Q8_0.gguf \
  --port 8080

Point the editor at the endpoint. Continue, Cline, Aider, and Zed all accept a base URL and model name; most others do too. Keep the model name stable across tools so an evaluation on one predicts the others.

Tools through MCP

The Model Context Protocol has become the common way to give a model tools: a small server exposes functions (read a file, query a database, run a linter) with typed schemas, and any MCP-aware harness can call them. For a local agent this is the seam where capability is added without touching the model. Run MCP servers locally too; a tool that phones home defeats the point of a local model.

a harness's MCP config — two local toolsjson
{
  "mcpServers": {
    "repo": { "command": "mcp-server-git", "args": ["--repository", "."] },
    "db":   { "command": "mcp-server-sqlite", "args": ["--db", "./dev.sqlite"] }
  }
}

Guardrails for an agent that acts

An assistant suggests; an agent runs commands. The difference is what makes an agent useful and what makes it dangerous, and the guardrails are not optional:

  1. A sandbox. The agent runs in a container or VM with only the repository mounted and no credentials it does not need. A local model does not exfiltrate data, but a shell command it writes can.
  2. A separate worktree or branch. Every task runs on its own branch. Nothing the agent does touches main until a person has read the diff.
  3. Command allowlists. The harness executes only commands on a list (test runners, linters, the package manager) and asks before anything else. rm, network calls, and anything touching secrets stay on the ask list.
  4. Secrets scanning on the way out. Before a diff is presented, scan it for keys, tokens, and personal data. Agents copy what they read.
  5. Budgets. A cap on turns, on tokens, and on wall-clock time per task. A loop that has not converged in forty turns is not going to.
  6. A review gate. The output is a pull request, and a person merges it. The agent's job ends at the diff.

Measuring whether it is useful

Leaderboards measure models on public benchmarks; the question is whether the agent completes tasks in your repository. Build a set of ten to thirty real tasks with a checkable outcome (a failing test that should pass, a lint error that should clear, a feature with an acceptance test), run the agent against each from a clean branch, and record the pass rate, the number of turns, and the tokens used. Rerun on every model or configuration change. A model that scores higher on a public benchmark and lower on this set is the wrong model.

The strategy for independent developers covers where a local coding agent fits in a solo workflow and what it is worth against a hosted one. For teams that need a private coding stack engineered and operated, that is a scoped AI engineering engagement.

Questions people ask about this.

Answered in full, in plain text — the same words the page's structured data carries.

  1. Can a local model replace a hosted coding assistant?

    For completion, explanation, single-file edits, and tests, a 14B–32B coding model on a 24 GB card does the everyday work well. For long autonomous tasks across a large repository, hosted frontier models still hold an advantage that the memory tier largely determines. Many developers run both: local for the high-frequency, code-sensitive work, hosted for the occasional hard task, with the hybrid decided by the same cost and privacy logic as any other workload.

  2. How much VRAM do you need for a local coding assistant?

    For a good experience, 16–24 GB: a 14B coder at 4-bit with a 32k context and an 8-bit KV cache fits comfortably in 24 GB and tightly in 16. An 8B coder at 4-bit with 16k context runs on 12 GB, and 8 GB works for completion with a short context. Unified-memory laptops with 32 GB or more run the same models at lower speed.

  3. Is it safe to let a local agent run commands?

    Only inside the guardrails above: a sandbox with no unnecessary credentials, a separate branch, an allowlist of commands, a scan of every diff before it is shown, and a person merging. The model being local removes the data-exfiltration risk of sending code to a server; it does nothing about the risk of a badly written shell command, which is a property of agents, not of where the model runs.

Next step

Send the stack you have.

Describe the model, the data and the hardware in a few lines. Strygon replies with what the architecture likely needs, before any talk of price.