Guide · Layer 06 · Application
Shipping on-device inference in a product
On-device inference is the version of local AI that a customer never has to set up: the model ships inside the product and runs on their phone, laptop, or browser. It is the strongest privacy story available, it works offline, and its marginal cost is zero. It is also a different engineering problem from running a model on a workstation. This guide covers the runtimes per platform, how to size a model for a device, and the packaging and product decisions that decide whether it works in the field.
- Series
- 07 of 09
- Sections
- 07
- Updated
- Sep 14, 2026
- Reading time
- 11 min
- Written for
- developers
Why put the model on the device
- Privacy by construction. The input never leaves the device, so there is no server to secure, no data-processing agreement to sign, and nothing to disclose in a privacy policy beyond the fact of local processing.
- Offline and low latency. No network round trip, no rate limit, no outage. A model that answers in a hundred milliseconds from local memory changes what the interface can do.
- Zero marginal cost. Every inference is paid for by the user's own hardware. For a product with a free tier or unpredictable usage, that is the difference between a viable business and an API bill.
- A durable differentiator. A cloud feature can be copied by pointing at the same API. A local one has to be engineered.
The trade is capability. Device models are small (one to a few billion parameters) and the tasks they do well are narrower: classification, extraction, summarization, rewriting, short-form generation, on-device search. Designing the feature around what a small model does reliably is most of the work.
Runtimes by platform
| Platform | Runtime | Model format | Notes |
|---|---|---|---|
| macOS / iOS | llama.cpp (Metal), MLX, Core ML | GGUF, MLX, .mlpackage | Apple also exposes its own on-device foundation model to apps through a system framework on recent OS versions |
| Android | llama.cpp (JNI), LiteRT / MediaPipe LLM Inference, ONNX Runtime | GGUF, .task / .tflite, ONNX | Vendor NPUs are reachable through delegates; coverage varies by chip |
| Windows / Linux desktop | llama.cpp (CUDA, Vulkan, CPU), ONNX Runtime GenAI | GGUF, ONNX | Vulkan reaches most consumer GPUs without a vendor SDK |
| Browser | WebLLM (WebGPU), transformers.js (ONNX Runtime Web) | MLC, ONNX | WebGPU is required for useful speed; WASM fallback is CPU-only |
| Cross-platform app frameworks | Native bindings to the above | As above | Flutter, React Native, Tauri and Electron all reach llama.cpp through a native module |
llama.cpp is the common denominator: one engine, one model format, bindings for every language that matters, and a Metal, CUDA, Vulkan, or CPU backend depending on where it lands. Platform-native stacks (MLX and Core ML on Apple, LiteRT on Android) can be faster or more power-efficient on their own hardware and are worth the extra build target once the feature is proven.
Sizing a model for a device
The hardware guide arithmetic applies unchanged, with a smaller budget. A phone gives an app a fraction of its RAM before the OS kills it; a browser tab gets less; a laptop gets a lot, but the user is running other things. The working ranges:
| Device class | Memory an app can count on | Model size that fits | Typical file at Q4 |
|---|---|---|---|
| Phone, current generation | ~2–4 GB for the model | 1B–3B, occasionally 4B | 0.7–2.5 GB |
| Tablet / thin laptop | ~4–8 GB | 3B–8B | 2–5 GB |
| Desktop with a GPU or unified memory | 8 GB+ | 8B–14B and up | 5–9 GB |
| Browser tab (WebGPU) | ~1–4 GB, varies by browser | 1B–3B | 0.7–2 GB |
Quantize to 4-bit for anything on a phone or in a browser; the memory and bandwidth savings are what make it fit, and small models at 4-bit are usable for the narrow tasks that suit them. Keep context short: the KV cache is charged per token, and a device model with a 32k context is a device model that gets killed by the OS.
Packaging and updates
- Do not bundle a multi-gigabyte model in the installer. App store size limits, install abandonment, and every update re-shipping the weights all argue for downloading on first use. Both mobile platforms have a mechanism for large assets delivered after install; on desktop and web, a download with a progress bar and a resumable transfer is the standard.
- Verify what you downloaded. Ship the model's hash with the app and check it before loading. A model file is executable in effect; treat its integrity like code.
- Version the model separately from the app. The app declares which model versions it can run; the model updates when it improves, and old apps keep working against the version they have.
- Cache once, per user. Two apps from the same developer should not download the same model twice; on desktop, a shared cache directory does this.
- Plan the delete. The user should be able to remove the model and reclaim the space, and the app should degrade gracefully when it is gone.
Product behaviour that survives contact with users
- Stream every output. A small model on a phone produces tokens at a rate that is fine when streamed and unbearable when buffered.
- Warm up before the user needs it. Loading the model takes seconds; do it on launch in the background, or on the screen before the feature, so the first request does not pay the cost.
- Respect thermals and battery. Sustained generation heats a phone and the OS will throttle. Keep sessions short, back off when the device reports thermal pressure, and never run inference in the background without a clear reason and a clear disclosure.
- Fail toward the local answer. If the model is unavailable, say so and offer the non-AI path. If a cloud fallback exists, it is opt-in, with the data implication stated in the moment, not in a settings page.
- Be honest in the interface about what is local. Users who chose the product for privacy will look; a feature that quietly calls a server is the trust failure the whole approach exists to avoid.
The licence check
Every open-weight model ships under a licence, and they are not the same. Some are permissive open-source licences (most Qwen releases are Apache-2.0; Microsoft's Phi models are MIT). Others carry a custom community licence with usage restrictions, attribution requirements, or acceptable-use terms (Meta's Llama and Google's Gemma models each have their own). For a model that ships inside a product you sell, the licence is a contract term you are agreeing to on behalf of the business. Read it before the model is chosen, not before launch, and keep a copy with the build.
Licences also govern derivatives: a fine-tuned version inherits the base's terms. Distillation from a model with restrictive output terms into one you ship is a question for the licence text, not for a forum.
A minimal integration
// The shape of a native call: load once, stream tokens, keep the
// session short. Bindings differ per platform; the lifecycle does not.
let model = try LlamaModel(path: modelURL.path, contextTokens: 2048)
try await model.warmUp()
for try await token in model.generate(prompt: prompt, maxTokens: 256) {
await MainActor.run { output += token } // stream to the UI
}The strategy piece for independent developers treats on-device inference as the defensible core of a privacy-first product and sets out how to build a business around it. Strygon builds this shape for clients whose product is the device, and for operators whose data cannot leave the building.