Definition · the stack · the instruments

Local AI compute

lo·cal AI com·pute/ˈloʊ.kəl ˌeɪ.ˈaɪ kəm.ˈpjuːt/noun

Running language, vision, and embedding models on infrastructure the organization or the individual owns and controls, rather than calling a model hosted by a third party.

Compute you own.Models that never leave the building.

What local AI compute is

Local AI compute is the practice of running language, vision, and embedding models on infrastructure that the organization or the individual owns and controls, rather than calling a model hosted by a third party.

Local AI is inference and training that runs on hardware the business or the developer controls: a workstation, a server on premises, a device in a customer's hand. The data stays inside a boundary that can be drawn on a diagram. Strygon architects, builds, and operates that layer for regulated operators, and publishes the engineering behind it here for independent developers.

The defining property is where the data goes. On a local stack, prompts, documents, and outputs stay on the machine or the network they started on; there is no external call to log, audit, or contract for. The economics follow from that: cost is a fixed hardware line plus power and operations rather than a per-token meter, which is cheaper at high sustained volume and more expensive at low volume, as the cost model sets out.

The stack is made of open-weight models, a runtime that serves them, and the hardware that holds them. Each of those is a decision with a small number of right answers, which is what the guides on this page cover, layer by layer.

Runs on
A workstation GPU, an on-premises server, unified-memory hardware, or the customer's own device
Data boundary
Nothing leaves the infrastructure the owner controls; no third-party call exists to audit
Cost shape
Fixed hardware and operations; near-zero marginal cost per token
Model class
Open-weight models, sized to the memory available, quantized to fit
Fits
Regulated work, high-volume pipelines, offline and latency-sensitive products, and anything the owner cannot send elsewhere

Weights, cache, and the box they have to fit in.

A model runs locally only if it fits in memory: the weights, plus working memory for the conversation, plus the runtime. When it does, answers are produced on the machine and nothing leaves it. Every number in the drawing is the fit formula run on a published configuration.

Two calculators that do the arithmetic, on your numbers.

Both compute exactly the formulas the guides print and nothing else. Presets are published configurations; defaults are labelled placeholders. Neither knows a benchmark or a price, and neither pretends to.

Will it fit

Model fitweights + cache + runtime
Context length
KV cache precision
Weights
4.87 GB
KV cache · 16k
2.15 GB
Runtime overhead
1.00 GB
Total resident
8.02 GB

Smallest tier with 15% headroom: 12 GB · mid GPU9.22 GB needed · 131,072 bytes of cache per token

Arithmetic only · presets are published configsNo benchmark is implied

The formula

Three lines, added.

weights = params × bits ÷ 8
cache   = 2 × layers × KV heads
          × head dim × bytes × context
total   = weights + cache + runtime
fits    = total × 1.15 ≤ tier

The same formula the hardware guide prints, with the 15% headroom it recommends above the arithmetic.

The hardware guide

What neither will tell you

  • How fast. No measured speed: throughput depends on the box, the runtime and the load.
  • What it costs to buy. No price list; prices move faster than a page can.
  • Which model wins. Families and tiers only; your own evaluation set decides.

Local vs API break-even

Local vs API · break-evenplaceholder inputs
Workload
The box
Operations
API per month
$90
Local per month
$404
hardware · power · ops
$83 · $21 · $300
Crossover volume
9.0M / day
Payback on hardware
never
$0.00$7,273$14,5450m12m24m36m
API, cumulativeLocal, cumulative

At these inputs, the API is cheaper per month.Local wins above 9.0M tokens a day. Not modelled: quality gap, privacy, and the price of your attention.

Formula only · the cost guide prints itPlaceholders — replace every value

The honest table

When local wins, and when it does not.

A page that only argued for local would be selling. The cost guide sets out the arithmetic; this is the summary.

Local wins whenOwn the box

  1. The data cannot leave: regulated records, source code under NDA, customer content under contract
  2. Sustained high volume, where a fixed cost beats a per-token meter
  3. Latency or offline operation the product depends on
  4. A cost ceiling the business needs to be able to promise
  5. A long-lived workload that will run the same model for a year

The API wins whenRent the call

  1. Low or spiky volume, where a fixed cost is a fixed cost against nothing
  2. The task genuinely needs a frontier model an open-weight one cannot match at your memory tier
  3. A prototype that changes weekly and has not been evaluated yet
  4. A solo builder whose hours are the scarce input and belong on the product
  5. No one to own the box: drivers, updates, and a wedged server at 2 a.m.

Nine guides, in reading order.

Written for the engineer making the decision. Each one states its formulas, names its sources, and stops where measurement would be required.

The strategy for developers

Questions, answered in full.

  1. What is the difference between local AI and private AI?

    Local AI describes where the model runs: on hardware the owner controls, from a laptop to an on-premises server. Private AI describes the guarantee: that data is not exposed to a party outside a defined boundary. Local is the most direct way to get private, because there is no external call at all, but a private deployment also needs the guardrails around the model (de-identification, access control, audit) that a location alone does not provide. Strygon builds both halves.

  2. Is local AI cheaper than using an API?

    At sustained high volume, usually. At low volume, usually not, once hardware amortization, power, and the hours spent operating the box are counted. The crossover depends on four numbers the reader has to supply, which is what the break-even calculator on this page is for. For regulated work the question is often moot, because the data cannot go to an API at any price.

  3. What hardware do you need to run AI models locally?

    Enough memory to hold the model, its context cache, and the runtime, with headroom. A 7B–8B model at 4-bit runs on an 8–12 GB GPU; 14B–24B wants 16–24 GB; 32B wants 32 GB; 70B needs 48 GB and up, which today means a workstation card, two consumer cards, or unified memory. The fit calculator on this page does the arithmetic for a specific model and context length.

  4. Can a small business run AI on its own hardware?

    Yes, and for a business whose data is regulated it is often the only responsible way. A single workstation-class GPU runs the models that handle document extraction, classification, summarization, and retrieval over a private corpus. The work is in the architecture around the model: the intake, the de-identification layer, the evaluation, and the operations. Strygon scopes that as a written plan before anything is built.

  5. Which open-weight models are best for local use?

    It changes every release cycle, which is why these pages name model families and memory tiers rather than a winner. The durable advice: pick the largest model that fits the tier at 4-bit, prefer a coder or domain variant for a narrow task, check the licence before shipping anything, and evaluate on a held-out set of your own inputs rather than on a leaderboard.

  6. Does Strygon offer local AI to individual developers?

    The guides, the instruments, and the reference architectures on this site are published for independent developers to use directly. Where a developer or a small product team needs the architecture engineered rather than described, Strygon scopes a written local-AI architecture review as a bounded engagement, and larger builds run as the same service regulated operators buy.

Next step

Start with the data that cannot leave.

Describe the workload in a few lines. Strygon replies with what the architecture likely needs, before any talk of price.