Five moves · one machine · two instruments

Ship privacy‑first products on compute you control.

An independent developer has one advantage over every well-funded competitor: nothing prevents them from building the version of a product that keeps the customer's data on the customer's machine. Local inference is what makes that version possible.

Model fitweights + cache + runtime
Context length
KV cache precision
Weights
4.87 GB
KV cache · 16k
2.15 GB
Runtime overhead
1.00 GB
Total resident
8.02 GB

Smallest tier with 15% headroom: 12 GB · mid GPU9.22 GB needed · 131,072 bytes of cache per token

Arithmetic only · presets are published configsNo benchmark is implied

Will it fit · edit any fieldArithmetic, not a benchmark · the formula

Five moves, each leaving an artifact.

This page sets out the strategy, a reference stack for one machine, the instruments that do the arithmetic, and where Strygon fits if the architecture needs engineering rather than a guide.

The order matters. Position before scope, scope before stack, stack before economics, and shipping last. Each move ends in something written down that the next one builds on.

  1. 01Position
  2. 02Scope
  3. 03Stack
  4. 04Economics
  5. 05Ship
  1. strygon position --market client-data

    01 · PositionMake privacy the product, not a feature

    • A cloud feature can be copied by pointing at the same API.
    • A feature that runs on the device and never phones home has to be engineered, and it is the one thing a large competitor is structurally reluctant to build.
    • Choose the market where that matters: professionals with client data, regulated work, people who have been burned.

    Leaves: a positioning statement one sentence long

  2. strygon scope --eval-set ./inputs

    02 · ScopeDesign around what a small model does reliably

    • Classification, extraction, summarization, rewriting, search over the user's own documents.
    • A 1B–8B model does these well; it does not do open-ended reasoning well.
    • The product that wins is narrow and dependable, measured on a held-out set of the real task before a line of UI is written.

    Leaves: an evaluation set of 100–300 real inputs

  3. strygon stack --runtime llama.cpp --quant q4_k_m

    03 · StackOne machine, one runtime, one model, versioned

    • Develop on the hardware your customers have, not the best box you can afford. llama.cpp or Ollama behind an OpenAI-shaped endpoint, a model at Q4 chosen by the evaluation set, retrieval in the database you already use.
    • Every model swap runs the same evaluation.

    Leaves: a reference stack with pinned versions

  4. strygon cost --break-even

    04 · EconomicsLet the customer's hardware carry the marginal cost

    • On-device inference costs nothing per call, which is what makes a free tier or a one-time price viable.
    • Where a server is needed, run the numbers with the break-even instrument before buying anything, and price your own hours honestly.
    • A hybrid, local by default and an API for the rare hard call, is usually the answer.

    Leaves: a cost model with your numbers in it

  5. strygon ship --verify-hash --fallback local

    05 · ShipPackage, verify, stream, and fail toward local

    • Download the model after install and verify its hash; stream every token; warm up before the user needs it; degrade to the non-AI path rather than a silent cloud call.
    • Read the model licence before the model is chosen and keep it with the build.

    Leaves: a product a privacy-conscious customer can trust

Where Strygon fits

A local-AI architecture review, scoped in writing.

For a developer or a small product team whose local stack needs to be engineered rather than described. Bounded, written, and yours to keep whether or not anything gets built afterwards.

Price

Scoped after the situation is read, not before.

What the review covers

  • Hardware sizing against the models and context the product actually needs
  • Model and runtime selection, with the quantization decided by an evaluation set built during the review
  • Retrieval architecture over the product's own data, on one machine
  • The privacy boundary: what stays local, what may leave with consent, and the de-identification layer if the data is regulated
  • A packaging and update plan for on-device deployment
  • A written architecture document with sequence and scope for a build, priced separately if wanted
Shape
A bounded review that ends in a document
What comes back
A written architecture: hardware, runtime, model, retrieval, boundary, and sequence
Ownership
The document is the developer's to keep and to build from, with or without Strygon

Send the product, the data, and the hardware you have or plan to have. Strygon comes back with a read on what the architecture likely needs before anyone talks about scope.

Questions developers ask first.

Answered in full, from the strategy and the guides above.

  1. Do you need a GPU to build a local AI product?

    Not necessarily. Develop on the hardware your customers have, not the best box you can afford: a 16–24 GB GPU workstation or a unified-memory laptop with 32 GB or more covers the reference stack on this page. On the customer's side, llama.cpp bindings, MLX, or WebGPU run the model on the device itself, which is what takes the cost per call to nothing.

  2. Which inference runtime should you start with?

    llama.cpp (llama-server) when you want control, Ollama when you want convenience. Both expose the same OpenAI-shaped endpoint, so the application code does not change between them. vLLM or SGLang are for serving an application to many users at once, and most stacks end up running two. The inference servers guide sets out the trade-offs.

  3. How do you choose a model when the leaderboards keep changing?

    With an evaluation set, not a leaderboard. Collect 100–300 real inputs for the task with a scriptable score, then pick the largest model that fits your memory tier at Q4_K_M and passes it. Every model swap runs the same evaluation. Read the model licence before the model is chosen, and keep it with the build.

  4. Is a small local model good enough for a real product?

    For narrow jobs, yes. Classification, extraction, summarization, rewriting, and search over the user's own documents are what a 1B–8B model does reliably; open-ended reasoning is not. Design the product around the reliable jobs. Where a rare call genuinely needs more, a hybrid that is local by default and calls an API only with consent is usually the answer.

  5. What does the architecture review cost?

    It is scoped after the situation is read, not before. Send the product, the data, and the hardware you have or plan to have, and Strygon comes back with a read on what the architecture likely needs before anyone talks about scope. The review ends in a written architecture document that is yours to keep, whether or not anything gets built afterwards.

  6. Can you use the guides without hiring Strygon?

    Yes. The guides, the two calculators, and the reference architectures are published for independent developers to use directly. The review exists for the case where the architecture needs to be engineered rather than described.

Next step

Send the stack you have.

Describe the product, the data and the hardware in a few lines. Strygon replies with what the architecture likely needs, before any talk of price.