Guide · Layer 06 · Application

Shipping on-device inference in a product

On-device inference is the version of local AI that a customer never has to set up: the model ships inside the product and runs on their phone, laptop, or browser. It is the strongest privacy story available, it works offline, and its marginal cost is zero. It is also a different engineering problem from running a model on a workstation. This guide covers the runtimes per platform, how to size a model for a device, and the packaging and product decisions that decide whether it works in the field.

On the device
The model ships inside the product and runs on the customer's hardware. No network call is made.
Series
07 of 09
Sections
07
Updated
Sep 14, 2026
Reading time
11 min
Written for
developers

Why put the model on the device

  • Privacy by construction. The input never leaves the device, so there is no server to secure, no data-processing agreement to sign, and nothing to disclose in a privacy policy beyond the fact of local processing.
  • Offline and low latency. No network round trip, no rate limit, no outage. A model that answers in a hundred milliseconds from local memory changes what the interface can do.
  • Zero marginal cost. Every inference is paid for by the user's own hardware. For a product with a free tier or unpredictable usage, that is the difference between a viable business and an API bill.
  • A durable differentiator. A cloud feature can be copied by pointing at the same API. A local one has to be engineered.

The trade is capability. Device models are small (one to a few billion parameters) and the tasks they do well are narrower: classification, extraction, summarization, rewriting, short-form generation, on-device search. Designing the feature around what a small model does reliably is most of the work.

Runtimes by platform

PlatformRuntimeModel formatNotes
macOS / iOSllama.cpp (Metal), MLX, Core MLGGUF, MLX, .mlpackageApple also exposes its own on-device foundation model to apps through a system framework on recent OS versions
Androidllama.cpp (JNI), LiteRT / MediaPipe LLM Inference, ONNX RuntimeGGUF, .task / .tflite, ONNXVendor NPUs are reachable through delegates; coverage varies by chip
Windows / Linux desktopllama.cpp (CUDA, Vulkan, CPU), ONNX Runtime GenAIGGUF, ONNXVulkan reaches most consumer GPUs without a vendor SDK
BrowserWebLLM (WebGPU), transformers.js (ONNX Runtime Web)MLC, ONNXWebGPU is required for useful speed; WASM fallback is CPU-only
Cross-platform app frameworksNative bindings to the aboveAs aboveFlutter, React Native, Tauri and Electron all reach llama.cpp through a native module
Table 01 Runtimes and formats as documented by each project. Platform support moves quickly; check the current documentation for the versions you target.

llama.cpp is the common denominator: one engine, one model format, bindings for every language that matters, and a Metal, CUDA, Vulkan, or CPU backend depending on where it lands. Platform-native stacks (MLX and Core ML on Apple, LiteRT on Android) can be faster or more power-efficient on their own hardware and are worth the extra build target once the feature is proven.

Sizing a model for a device

The hardware guide arithmetic applies unchanged, with a smaller budget. A phone gives an app a fraction of its RAM before the OS kills it; a browser tab gets less; a laptop gets a lot, but the user is running other things. The working ranges:

Device classMemory an app can count onModel size that fitsTypical file at Q4
Phone, current generation~2–4 GB for the model1B–3B, occasionally 4B0.7–2.5 GB
Tablet / thin laptop~4–8 GB3B–8B2–5 GB
Desktop with a GPU or unified memory8 GB+8B–14B and up5–9 GB
Browser tab (WebGPU)~1–4 GB, varies by browser1B–3B0.7–2 GB
Table 02 Ranges, not specifications: what an app can allocate depends on the OS, the device generation, and what else is running. Test on the oldest device you support, under memory pressure, with the model already warm.

Quantize to 4-bit for anything on a phone or in a browser; the memory and bandwidth savings are what make it fit, and small models at 4-bit are usable for the narrow tasks that suit them. Keep context short: the KV cache is charged per token, and a device model with a 32k context is a device model that gets killed by the OS.

Packaging and updates

  1. Do not bundle a multi-gigabyte model in the installer. App store size limits, install abandonment, and every update re-shipping the weights all argue for downloading on first use. Both mobile platforms have a mechanism for large assets delivered after install; on desktop and web, a download with a progress bar and a resumable transfer is the standard.
  2. Verify what you downloaded. Ship the model's hash with the app and check it before loading. A model file is executable in effect; treat its integrity like code.
  3. Version the model separately from the app. The app declares which model versions it can run; the model updates when it improves, and old apps keep working against the version they have.
  4. Cache once, per user. Two apps from the same developer should not download the same model twice; on desktop, a shared cache directory does this.
  5. Plan the delete. The user should be able to remove the model and reclaim the space, and the app should degrade gracefully when it is gone.

Product behaviour that survives contact with users

  • Stream every output. A small model on a phone produces tokens at a rate that is fine when streamed and unbearable when buffered.
  • Warm up before the user needs it. Loading the model takes seconds; do it on launch in the background, or on the screen before the feature, so the first request does not pay the cost.
  • Respect thermals and battery. Sustained generation heats a phone and the OS will throttle. Keep sessions short, back off when the device reports thermal pressure, and never run inference in the background without a clear reason and a clear disclosure.
  • Fail toward the local answer. If the model is unavailable, say so and offer the non-AI path. If a cloud fallback exists, it is opt-in, with the data implication stated in the moment, not in a settings page.
  • Be honest in the interface about what is local. Users who chose the product for privacy will look; a feature that quietly calls a server is the trust failure the whole approach exists to avoid.

The licence check

Every open-weight model ships under a licence, and they are not the same. Some are permissive open-source licences (most Qwen releases are Apache-2.0; Microsoft's Phi models are MIT). Others carry a custom community licence with usage restrictions, attribution requirements, or acceptable-use terms (Meta's Llama and Google's Gemma models each have their own). For a model that ships inside a product you sell, the licence is a contract term you are agreeing to on behalf of the business. Read it before the model is chosen, not before launch, and keep a copy with the build.

Licences also govern derivatives: a fine-tuned version inherits the base's terms. Distillation from a model with restrictive output terms into one you ship is a question for the licence text, not for a forum.

A minimal integration

sketch, not a libraryswift
// The shape of a native call: load once, stream tokens, keep the
// session short. Bindings differ per platform; the lifecycle does not.
let model = try LlamaModel(path: modelURL.path, contextTokens: 2048)
try await model.warmUp()

for try await token in model.generate(prompt: prompt, maxTokens: 256) {
    await MainActor.run { output += token }     // stream to the UI
}

The strategy piece for independent developers treats on-device inference as the defensible core of a privacy-first product and sets out how to build a business around it. Strygon builds this shape for clients whose product is the device, and for operators whose data cannot leave the building.

Questions people ask about this.

Answered in full, in plain text — the same words the page's structured data carries.

  1. Can you run an LLM inside an iOS or Android app?

    Yes. llama.cpp has bindings for Swift and for Android through JNI, Apple's MLX and Core ML run models natively on its chips, and Google's LiteRT and MediaPipe expose an on-device LLM API on Android. Models in the 1B–3B range at 4-bit fit in the memory an app can allocate on a current phone. The engineering is in sizing the model to the task, downloading and verifying the weights after install, and managing memory and thermals in use.

  2. Can a language model run in the browser?

    With WebGPU, yes: WebLLM and transformers.js run quantized models of one to a few billion parameters entirely in the page, with the weights cached by the browser after first download. Without WebGPU the fallback is WebAssembly on the CPU, which is too slow for interactive generation beyond the smallest models. Feature-detect and degrade accordingly.

  3. Do you need to pay to use open-weight models in a commercial product?

    Usually not, but you do need to comply with the licence. Apache-2.0 and MIT models can be used commercially with attribution. Models under custom community licences generally allow commercial use with conditions, which can include acceptable-use policies, attribution, and, for some, thresholds above which different terms apply. The licence text for the specific model and version is the only authority; read it and keep it with the build.

Next step

Send the stack you have.

Describe the model, the data and the hardware in a few lines. Strygon replies with what the architecture likely needs, before any talk of price.