Guide · Layer 04 · Retrieval
Private RAG on one machine: a reference architecture
Retrieval-augmented generation is how a local model answers questions about documents it was never trained on, and it is the workload where keeping everything on one machine pays off most, because the documents are usually the sensitive part. This is a reference architecture for a private RAG stack: every component runs locally, every choice is justified, and the evaluation is built in from the first day rather than bolted on.
- Series
- 05 of 09
- Sections
- 10
- Updated
- Sep 14, 2026
- Reading time
- 12 min
- Written for
- developers
What retrieval is for, and what it is not
A language model's knowledge is frozen at training and bounded by its context window. Retrieval fixes both at once: for each question, find the handful of passages that are relevant, put them in the prompt, and ask the model to answer from them. The model supplies language and reasoning; the corpus supplies facts. Done well, the answer is grounded, current, and citable.
What retrieval does not do is teach the model anything. It does not change its style, its format discipline, or its grasp of domain vocabulary; those are what fine-tuning is for. And it is only as good as the retrieval step. When a RAG system gives a wrong answer, the cause is usually that the right passage was never in the prompt, not that the model misread it. Which is why this architecture spends most of its effort before the model is ever called.
The pipeline
documents ──► ingest ──► chunk ──► embed ──► store
│
question ──► embed ──► retrieve (vector + keyword) ──► rerank ──► prompt ──► generate ──► cite
│
evaluateEvery arrow above runs on one machine. The models involved are small: an embedding model of a few hundred million parameters, a reranker of similar size, and a generation model sized to the hardware. On a 24 GB card all three fit alongside an 8B–14B generator with room for context; on a unified-memory laptop they fit alongside something larger and slower.
Ingest and chunk
Ingest is parsing: PDFs, office documents, HTML, tickets, email, whatever the corpus is. The parser's job is to preserve structure, because structure is what makes chunks coherent. Headings, tables, and list boundaries should survive into the text; a PDF flattened to a stream of lines loses the sections that would have made good chunks.
Chunking splits documents into passages small enough to embed as one meaning and large enough to answer a question on their own. There is no universally right size; a few hundred tokens with an overlap of ten to twenty percent is a defensible default, and splitting on structure (a section, a paragraph, a table row group) beats splitting on a fixed count. Attach metadata to every chunk: source document, section path, page, date, access group. Retrieval filters on metadata are what keep an answer from citing a document the asker should not see.
Embed
An embedding model turns a chunk into a vector such that similar meanings land near each other. The open models in this class are small, fast on any GPU and tolerable on CPU, and run under the same runtimes as everything else: llama.cpp serves GGUF embedding models, and the sentence-transformers library runs the safetensors ones. Families to evaluate include the bge, e5, nomic-embed, and gte lines, and newer releases appear regularly; pick by testing on your corpus, and keep the dimension and the model fixed once you have indexed, because a change to either means re-embedding everything.
Two rules of thumb. Embed the question with the same model and, if the model distinguishes them, the same query-versus-passage prefix it was trained with. And store the embedding model's name and version beside every vector, so an index can never silently mix two models.
Store
For one machine, the vector store should be boring. The candidates that earn a place on a single box are the ones that need no separate cluster to run:
| Store | Shape | Fits when |
|---|---|---|
| pgvector | Extension to Postgres | The corpus already lives in Postgres, or metadata filtering and joins matter |
| sqlite-vec | Extension to SQLite | An embedded application or a desktop product; one file, no server |
| LanceDB | Embedded, columnar, on disk | Large corpora on one machine; versioned data; fast scans |
| Qdrant / Chroma | Local server or in-process | You want a purpose-built vector API and filtering without Postgres |
Index type matters less than it seems below a few million vectors. An exact search is often fast enough; an HNSW index buys speed at large scale at the cost of a small recall loss and more memory. Start exact, add the index when latency says so.
Retrieve: vectors and keywords together
Vector search finds passages that mean the same thing as the question. It is weak at exact strings: a part number, a name, an error code, a legal citation. Keyword search (BM25 or a full-text index) is the opposite. The standard fix is hybrid retrieval: run both, merge the ranked lists with reciprocal rank fusion, and pass the union to the reranker. Postgres and SQLite both have full-text search built in, so on a single box this costs one extra query.
# Hybrid retrieval, merged by reciprocal rank fusion.
def rrf(rankings, k=60):
scores = {}
for ranked in rankings: # each is a list of chunk ids, best first
for rank, cid in enumerate(ranked, start=1):
scores[cid] = scores.get(cid, 0) + 1 / (k + rank)
return sorted(scores, key=scores.get, reverse=True)
vec_hits = store.vector_search(embed(question), top=40, filter=acl_filter)
kw_hits = store.keyword_search(question, top=40, filter=acl_filter)
candidates = rrf([vec_hits, kw_hits])[:40]Rerank
The first-stage retrievers are fast and approximate. A cross-encoder reranker reads the question and each candidate passage together and scores the pair directly, which is slower and much more accurate. Rerank the top forty candidates down to the five or eight that go in the prompt. This single step is usually the largest quality gain available in a RAG system, and it costs one small model call per candidate on the local GPU.
Generate and cite
The prompt carries the selected passages, each with an identifier, and an instruction to answer only from them and to reference the identifiers used. The model's job narrows to reading and composing, which is what small models are good at. Two disciplines make this work:
- Refusal is a valid answer. If no passage supports an answer, the instruction says to say so. An ungrounded guess is the failure mode retrieval exists to prevent; the prompt has to make the honest answer the easy one.
- Citations are checked, not trusted. After generation, verify that every cited identifier was in the prompt and that the cited passage plausibly supports the sentence beside it (a small model can score that). Drop or flag anything that fails.
Answer the question using only the passages below. Cite the passage id
in square brackets after each claim, like [p3]. If the passages do not
contain the answer, say "Not found in the provided documents."
[p1] (Handbook › Leave › Parental, p.12) ...
[p2] (Handbook › Leave › Sick, p.9) ...
...
Question: {question}Evaluate from day one
A RAG system without an evaluation set is a demo. Build the set before the pipeline, and grow it as real questions arrive. Three measurements cover most of what goes wrong:
- Retrieval hit rate. For each question with a known source passage, is that passage in the top-k after reranking? This isolates the retriever from the model, and it is where most failures live.
- Faithfulness. Does every claim in the answer trace to a cited passage? Score with a rubric or with a second local model acting as a judge, and spot-check the judge.
- Answer quality. Against a reference answer, does the response cover what it should? Exact match for factual lookups; a rubric for open questions.
Run the set on every change: a new embedding model, a different chunk size, a new generator, a quantization change. The numbers will tell you which stage moved, and the set becomes the regression suite that lets the stack evolve without regressing.
The reference stack, in one table
| Stage | Runs on | Notes |
|---|---|---|
| Ingest | Parser of choice (per format) | Preserve structure; attach metadata and access groups |
| De-identify | Rules + local NER model | Before embedding, if the corpus is regulated |
| Embed | llama.cpp or sentence-transformers | One model, versioned, stored beside the vectors |
| Store | Postgres + pgvector, or SQLite + sqlite-vec | Vectors, full-text index, and metadata in one place |
| Retrieve | Vector + keyword, RRF merge | Filter by access group at query time, never after |
| Rerank | Local cross-encoder | Top 40 → top 6; the biggest quality lever |
| Generate | llama-server / Ollama / vLLM | 8B–14B at Q4 on 24 GB; larger on unified memory |
| Cite & check | Post-processing + small model | Every citation verified against the prompt |
| Evaluate | Held-out set, run on every change | Hit rate, faithfulness, quality |
Strygon builds this shape for operators whose documents cannot leave the building, and it is the retrieval layer underneath its own in-house products. The Local & private AI service describes how that is scoped as an engagement.