Definition · the stack · the instruments
Local AI compute
lo·cal AI com·pute/ˈloʊ.kəl ˌeɪ.ˈaɪ kəm.ˈpjuːt/noun
Running language, vision, and embedding models on infrastructure the organization or the individual owns and controls, rather than calling a model hosted by a third party.
Compute you own.Models that never leave the building.
What local AI compute is
Local AI compute is the practice of running language, vision, and embedding models on infrastructure that the organization or the individual owns and controls, rather than calling a model hosted by a third party.
Local AI is inference and training that runs on hardware the business or the developer controls: a workstation, a server on premises, a device in a customer's hand. The data stays inside a boundary that can be drawn on a diagram. Strygon architects, builds, and operates that layer for regulated operators, and publishes the engineering behind it here for independent developers.
The defining property is where the data goes. On a local stack, prompts, documents, and outputs stay on the machine or the network they started on; there is no external call to log, audit, or contract for. The economics follow from that: cost is a fixed hardware line plus power and operations rather than a per-token meter, which is cheaper at high sustained volume and more expensive at low volume, as the cost model sets out.
The stack is made of open-weight models, a runtime that serves them, and the hardware that holds them. Each of those is a decision with a small number of right answers, which is what the guides on this page cover, layer by layer.
- Runs on
- A workstation GPU, an on-premises server, unified-memory hardware, or the customer's own device
- Data boundary
- Nothing leaves the infrastructure the owner controls; no third-party call exists to audit
- Cost shape
- Fixed hardware and operations; near-zero marginal cost per token
- Model class
- Open-weight models, sized to the memory available, quantized to fit
- Fits
- Regulated work, high-volume pipelines, offline and latency-sensitive products, and anything the owner cannot send elsewhere
Six layers, each with one decision in it.
Local AI is not one choice but a short column of them. Hardware carries every layer above it; each row links to the guide that makes its decision.
- HardwareThe memory that holds the model and the bandwidth that reads it.DecisionDiscrete GPU for speed, unified memory for capacity; size to the model you will run in a year.
- RuntimeThe server that loads the weights and answers an OpenAI-shaped API.Decisionllama.cpp or Ollama for one user; vLLM or SGLang for an application. Most stacks run two.
- ModelsOpen weights, quantized to fit, occasionally fine-tuned to behave.DecisionQ4 by default, bigger model over higher precision, and an evaluation set before either.
- RetrievalThe corpus the model answers from, indexed on the same box.DecisionHybrid search plus a reranker, in the database you already run, with citations checked.
- GuardrailsThe layer that finds and replaces identifiers before any model sees them.DecisionRules plus a local NER model, pseudonymize inside the boundary, audit every document.
- ApplicationThe agent, pipeline, or on-device feature the stack exists to run.DecisionNarrow and reliable over broad and impressive; measured on your own tasks, not a leaderboard.
Three readers, one stack.
The same architecture serves a regulated operator, an independent developer, and a product team. What changes is who owns the box and what they need from this page.
Weights, cache, and the box they have to fit in.
A model runs locally only if it fits in memory: the weights, plus working memory for the conversation, plus the runtime. When it does, answers are produced on the machine and nothing leaves it. Every number in the drawing is the fit formula run on a published configuration.
- FP168.03B × 16 bits ÷ 816.1 GB
- Weights8.03B × 4.85 bits ÷ 84.87 GB
- KV cache131,072 B/token × 16,3842.15 GB
- Runtimeoverhead1.00 GB
- Totalweights + cache + runtime8.02 GB
- Fits× 1.15 ≤ 12 GB9.22 GB
Two calculators that do the arithmetic, on your numbers.
Both compute exactly the formulas the guides print and nothing else. Presets are published configurations; defaults are labelled placeholders. Neither knows a benchmark or a price, and neither pretends to.
Will it fit
- Weights
- 4.87 GB
- KV cache · 16k
- 2.15 GB
- Runtime overhead
- 1.00 GB
- Total resident
- 8.02 GB
Smallest tier with 15% headroom: 12 GB · mid GPU9.22 GB needed · 131,072 bytes of cache per token
The formula
Three lines, added.
weights = params × bits ÷ 8
cache = 2 × layers × KV heads
× head dim × bytes × context
total = weights + cache + runtime
fits = total × 1.15 ≤ tierThe same formula the hardware guide prints, with the 15% headroom it recommends above the arithmetic.
The hardware guideWhat neither will tell you
- How fast. No measured speed: throughput depends on the box, the runtime and the load.
- What it costs to buy. No price list; prices move faster than a page can.
- Which model wins. Families and tiers only; your own evaluation set decides.
Local vs API break-even
- API per month
- $90
- Local per month
- $404
- hardware · power · ops
- $83 · $21 · $300
- Crossover volume
- 9.0M / day
- Payback on hardware
- never
At these inputs, the API is cheaper per month.Local wins above 9.0M tokens a day. Not modelled: quality gap, privacy, and the price of your attention.
The honest table
When local wins, and when it does not.
A page that only argued for local would be selling. The cost guide sets out the arithmetic; this is the summary.
Local wins whenOwn the box
- The data cannot leave: regulated records, source code under NDA, customer content under contract
- Sustained high volume, where a fixed cost beats a per-token meter
- Latency or offline operation the product depends on
- A cost ceiling the business needs to be able to promise
- A long-lived workload that will run the same model for a year
The API wins whenRent the call
- Low or spiky volume, where a fixed cost is a fixed cost against nothing
- The task genuinely needs a frontier model an open-weight one cannot match at your memory tier
- A prototype that changes weekly and has not been evaluated yet
- A solo builder whose hours are the scarce input and belong on the product
- No one to own the box: drivers, updates, and a wedged server at 2 a.m.
Nine guides, in reading order.
Written for the engineer making the decision. Each one states its formulas, names its sources, and stops where measurement would be required.
The strategy for developersPart 1The machine
Part 2The knowledge
Questions, answered in full.
What is the difference between local AI and private AI?
Local AI describes where the model runs: on hardware the owner controls, from a laptop to an on-premises server. Private AI describes the guarantee: that data is not exposed to a party outside a defined boundary. Local is the most direct way to get private, because there is no external call at all, but a private deployment also needs the guardrails around the model (de-identification, access control, audit) that a location alone does not provide. Strygon builds both halves.
Is local AI cheaper than using an API?
At sustained high volume, usually. At low volume, usually not, once hardware amortization, power, and the hours spent operating the box are counted. The crossover depends on four numbers the reader has to supply, which is what the break-even calculator on this page is for. For regulated work the question is often moot, because the data cannot go to an API at any price.
What hardware do you need to run AI models locally?
Enough memory to hold the model, its context cache, and the runtime, with headroom. A 7B–8B model at 4-bit runs on an 8–12 GB GPU; 14B–24B wants 16–24 GB; 32B wants 32 GB; 70B needs 48 GB and up, which today means a workstation card, two consumer cards, or unified memory. The fit calculator on this page does the arithmetic for a specific model and context length.
Can a small business run AI on its own hardware?
Yes, and for a business whose data is regulated it is often the only responsible way. A single workstation-class GPU runs the models that handle document extraction, classification, summarization, and retrieval over a private corpus. The work is in the architecture around the model: the intake, the de-identification layer, the evaluation, and the operations. Strygon scopes that as a written plan before anything is built.
Which open-weight models are best for local use?
It changes every release cycle, which is why these pages name model families and memory tiers rather than a winner. The durable advice: pick the largest model that fits the tier at 4-bit, prefer a coder or domain variant for a narrow task, check the licence before shipping anything, and evaluate on a held-out set of your own inputs rather than on a leaderboard.
Does Strygon offer local AI to individual developers?
The guides, the instruments, and the reference architectures on this site are published for independent developers to use directly. Where a developer or a small product team needs the architecture engineered rather than described, Strygon scopes a written local-AI architecture review as a bounded engagement, and larger builds run as the same service regulated operators buy.
Next step
Start with the data that cannot leave.
Describe the workload in a few lines. Strygon replies with what the architecture likely needs, before any talk of price.