Five moves · one machine · two instruments
Ship privacy‑first products on compute you control.
An independent developer has one advantage over every well-funded competitor: nothing prevents them from building the version of a product that keeps the customer's data on the customer's machine. Local inference is what makes that version possible.
- Weights
- 4.87 GB
- KV cache · 16k
- 2.15 GB
- Runtime overhead
- 1.00 GB
- Total resident
- 8.02 GB
Smallest tier with 15% headroom: 12 GB · mid GPU9.22 GB needed · 131,072 bytes of cache per token
Will it fit · edit any fieldArithmetic, not a benchmark · the formula
Five moves, each leaving an artifact.
This page sets out the strategy, a reference stack for one machine, the instruments that do the arithmetic, and where Strygon fits if the architecture needs engineering rather than a guide.
The order matters. Position before scope, scope before stack, stack before economics, and shipping last. Each move ends in something written down that the next one builds on.
strygon position --market client-data01 · PositionMake privacy the product, not a feature
- A cloud feature can be copied by pointing at the same API.
- A feature that runs on the device and never phones home has to be engineered, and it is the one thing a large competitor is structurally reluctant to build.
- Choose the market where that matters: professionals with client data, regulated work, people who have been burned.
Leaves: a positioning statement one sentence long
strygon scope --eval-set ./inputs02 · ScopeDesign around what a small model does reliably
- Classification, extraction, summarization, rewriting, search over the user's own documents.
- A 1B–8B model does these well; it does not do open-ended reasoning well.
- The product that wins is narrow and dependable, measured on a held-out set of the real task before a line of UI is written.
Leaves: an evaluation set of 100–300 real inputs
strygon stack --runtime llama.cpp --quant q4_k_m03 · StackOne machine, one runtime, one model, versioned
- Develop on the hardware your customers have, not the best box you can afford. llama.cpp or Ollama behind an OpenAI-shaped endpoint, a model at Q4 chosen by the evaluation set, retrieval in the database you already use.
- Every model swap runs the same evaluation.
Leaves: a reference stack with pinned versions
strygon cost --break-even04 · EconomicsLet the customer's hardware carry the marginal cost
- On-device inference costs nothing per call, which is what makes a free tier or a one-time price viable.
- Where a server is needed, run the numbers with the break-even instrument before buying anything, and price your own hours honestly.
- A hybrid, local by default and an API for the rare hard call, is usually the answer.
Leaves: a cost model with your numbers in it
strygon ship --verify-hash --fallback local05 · ShipPackage, verify, stream, and fail toward local
- Download the model after install and verify its hash; stream every token; warm up before the user needs it; degrade to the non-AI path rather than a silent cloud call.
- Read the model licence before the model is chosen and keep it with the build.
Leaves: a product a privacy-conscious customer can trust
One machine, one runtime, one model, versioned.
What a single developer’s local stack looks like when it is built to ship rather than to demo. Every layer links to the guide that decides it.
llama-server -m model-Q4_K_M.gguf \
--ctx-size 16384 --n-gpu-layers 99 --port 8080- OpenAI-shaped endpoint on localhost:8080/v1
- Same model file the product will ship
- Evaluation set runs against it on every change
- Nothing leaves the machine
The engineering behind each move.
Formulas stated, sources named, and a stop where a measurement would be needed. Read in order, or open the one your next decision needs.
The break-even calculatorPart 1The machine
Part 2The knowledge
Where Strygon fits
A local-AI architecture review, scoped in writing.
For a developer or a small product team whose local stack needs to be engineered rather than described. Bounded, written, and yours to keep whether or not anything gets built afterwards.
Price
Scoped after the situation is read, not before.
What the review covers
- Hardware sizing against the models and context the product actually needs
- Model and runtime selection, with the quantization decided by an evaluation set built during the review
- Retrieval architecture over the product's own data, on one machine
- The privacy boundary: what stays local, what may leave with consent, and the de-identification layer if the data is regulated
- A packaging and update plan for on-device deployment
- A written architecture document with sequence and scope for a build, priced separately if wanted
- Shape
- A bounded review that ends in a document
- What comes back
- A written architecture: hardware, runtime, model, retrieval, boundary, and sequence
- Ownership
- The document is the developer's to keep and to build from, with or without Strygon
Send the product, the data, and the hardware you have or plan to have. Strygon comes back with a read on what the architecture likely needs before anyone talks about scope.
Questions developers ask first.
Answered in full, from the strategy and the guides above.
Do you need a GPU to build a local AI product?
Not necessarily. Develop on the hardware your customers have, not the best box you can afford: a 16–24 GB GPU workstation or a unified-memory laptop with 32 GB or more covers the reference stack on this page. On the customer's side, llama.cpp bindings, MLX, or WebGPU run the model on the device itself, which is what takes the cost per call to nothing.
Which inference runtime should you start with?
llama.cpp (llama-server) when you want control, Ollama when you want convenience. Both expose the same OpenAI-shaped endpoint, so the application code does not change between them. vLLM or SGLang are for serving an application to many users at once, and most stacks end up running two. The inference servers guide sets out the trade-offs.
How do you choose a model when the leaderboards keep changing?
With an evaluation set, not a leaderboard. Collect 100–300 real inputs for the task with a scriptable score, then pick the largest model that fits your memory tier at Q4_K_M and passes it. Every model swap runs the same evaluation. Read the model licence before the model is chosen, and keep it with the build.
Is a small local model good enough for a real product?
For narrow jobs, yes. Classification, extraction, summarization, rewriting, and search over the user's own documents are what a 1B–8B model does reliably; open-ended reasoning is not. Design the product around the reliable jobs. Where a rare call genuinely needs more, a hybrid that is local by default and calls an API only with consent is usually the answer.
What does the architecture review cost?
It is scoped after the situation is read, not before. Send the product, the data, and the hardware you have or plan to have, and Strygon comes back with a read on what the architecture likely needs before anyone talks about scope. The review ends in a written architecture document that is yours to keep, whether or not anything gets built afterwards.
Can you use the guides without hiring Strygon?
Yes. The guides, the two calculators, and the reference architectures are published for independent developers to use directly. The review exists for the case where the architecture needs to be engineered rather than described.
Next step
Send the stack you have.
Describe the product, the data and the hardware in a few lines. Strygon replies with what the architecture likely needs, before any talk of price.