Field note · Local AI

Local AI for regulated work: what on-device inference changes

Self-hosted models move the compliance question from policy promise to architecture. What stays inside the building, and why that makes AI usable in places a public API cannot go.

The boundary
Model, prompts, and documents stay inside; the outside link is cut.

The blocker on AI in healthcare, finance, and other regulated work is rarely the model’s capability. It is where the data goes. A public API means prompts and documents leave the building, and a policy document ends up doing the work a boundary should be doing.

On-device changes the boundary

Self-hosted and on-device inference keeps the model, the prompts, the documents, and the results on infrastructure the business controls. There is no third-party call to audit, because none gets made. Compliance becomes a property of the architecture rather than a promise in a contract.

A de-identification and policy layer sits in front of every model, so regulated data can be used without being exposed. That is the same layer that stands between an in-house product like ClaimFlow and a HIPAA-eligible deployment.

The trade nobody mentions

Local models trade a little convenience for predictable cost and control. There is no per-token meter, the models can be fine-tuned on the business’s own language, and the data never becomes someone else’s training set. For regulated work, that trade is usually the whole point.

Next step

Start with the problem.

Describe the situation in a few lines. The reply covers what’s likely wrong and what it would take to fix, before any talk of price.