Guide · Layer 04 · Retrieval

Private RAG on one machine: a reference architecture

Retrieval-augmented generation is how a local model answers questions about documents it was never trained on, and it is the workload where keeping everything on one machine pays off most, because the documents are usually the sensitive part. This is a reference architecture for a private RAG stack: every component runs locally, every choice is justified, and the evaluation is built in from the first day rather than bolted on.

One machine
Documents, index and answer inside one boundary. Nothing crosses the dashed line.
Series
05 of 09
Sections
10
Updated
Sep 14, 2026
Reading time
12 min
Written for
developers

What retrieval is for, and what it is not

A language model's knowledge is frozen at training and bounded by its context window. Retrieval fixes both at once: for each question, find the handful of passages that are relevant, put them in the prompt, and ask the model to answer from them. The model supplies language and reasoning; the corpus supplies facts. Done well, the answer is grounded, current, and citable.

What retrieval does not do is teach the model anything. It does not change its style, its format discipline, or its grasp of domain vocabulary; those are what fine-tuning is for. And it is only as good as the retrieval step. When a RAG system gives a wrong answer, the cause is usually that the right passage was never in the prompt, not that the model misread it. Which is why this architecture spends most of its effort before the model is ever called.

The pipeline

the shapetext
documents ──► ingest ──► chunk ──► embed ──► store
                                               │
question ──► embed ──► retrieve (vector + keyword) ──► rerank ──► prompt ──► generate ──► cite
                                                                                   │
                                                                              evaluate

Every arrow above runs on one machine. The models involved are small: an embedding model of a few hundred million parameters, a reranker of similar size, and a generation model sized to the hardware. On a 24 GB card all three fit alongside an 8B–14B generator with room for context; on a unified-memory laptop they fit alongside something larger and slower.

Ingest and chunk

Ingest is parsing: PDFs, office documents, HTML, tickets, email, whatever the corpus is. The parser's job is to preserve structure, because structure is what makes chunks coherent. Headings, tables, and list boundaries should survive into the text; a PDF flattened to a stream of lines loses the sections that would have made good chunks.

Chunking splits documents into passages small enough to embed as one meaning and large enough to answer a question on their own. There is no universally right size; a few hundred tokens with an overlap of ten to twenty percent is a defensible default, and splitting on structure (a section, a paragraph, a table row group) beats splitting on a fixed count. Attach metadata to every chunk: source document, section path, page, date, access group. Retrieval filters on metadata are what keep an answer from citing a document the asker should not see.

Embed

An embedding model turns a chunk into a vector such that similar meanings land near each other. The open models in this class are small, fast on any GPU and tolerable on CPU, and run under the same runtimes as everything else: llama.cpp serves GGUF embedding models, and the sentence-transformers library runs the safetensors ones. Families to evaluate include the bge, e5, nomic-embed, and gte lines, and newer releases appear regularly; pick by testing on your corpus, and keep the dimension and the model fixed once you have indexed, because a change to either means re-embedding everything.

Two rules of thumb. Embed the question with the same model and, if the model distinguishes them, the same query-versus-passage prefix it was trained with. And store the embedding model's name and version beside every vector, so an index can never silently mix two models.

Store

For one machine, the vector store should be boring. The candidates that earn a place on a single box are the ones that need no separate cluster to run:

StoreShapeFits when
pgvectorExtension to PostgresThe corpus already lives in Postgres, or metadata filtering and joins matter
sqlite-vecExtension to SQLiteAn embedded application or a desktop product; one file, no server
LanceDBEmbedded, columnar, on diskLarge corpora on one machine; versioned data; fast scans
Qdrant / ChromaLocal server or in-processYou want a purpose-built vector API and filtering without Postgres
Table 01 Options that run on one machine without a managed service. All support metadata filtering; the differences are in ecosystem and in how much other data you keep beside the vectors.

Index type matters less than it seems below a few million vectors. An exact search is often fast enough; an HNSW index buys speed at large scale at the cost of a small recall loss and more memory. Start exact, add the index when latency says so.

Retrieve: vectors and keywords together

Vector search finds passages that mean the same thing as the question. It is weak at exact strings: a part number, a name, an error code, a legal citation. Keyword search (BM25 or a full-text index) is the opposite. The standard fix is hybrid retrieval: run both, merge the ranked lists with reciprocal rank fusion, and pass the union to the reranker. Postgres and SQLite both have full-text search built in, so on a single box this costs one extra query.

retrieve.pypython
# Hybrid retrieval, merged by reciprocal rank fusion.
def rrf(rankings, k=60):
    scores = {}
    for ranked in rankings:                 # each is a list of chunk ids, best first
        for rank, cid in enumerate(ranked, start=1):
            scores[cid] = scores.get(cid, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

vec_hits = store.vector_search(embed(question), top=40, filter=acl_filter)
kw_hits  = store.keyword_search(question,       top=40, filter=acl_filter)
candidates = rrf([vec_hits, kw_hits])[:40]

Rerank

The first-stage retrievers are fast and approximate. A cross-encoder reranker reads the question and each candidate passage together and scores the pair directly, which is slower and much more accurate. Rerank the top forty candidates down to the five or eight that go in the prompt. This single step is usually the largest quality gain available in a RAG system, and it costs one small model call per candidate on the local GPU.

Generate and cite

The prompt carries the selected passages, each with an identifier, and an instruction to answer only from them and to reference the identifiers used. The model's job narrows to reading and composing, which is what small models are good at. Two disciplines make this work:

  • Refusal is a valid answer. If no passage supports an answer, the instruction says to say so. An ungrounded guess is the failure mode retrieval exists to prevent; the prompt has to make the honest answer the easy one.
  • Citations are checked, not trusted. After generation, verify that every cited identifier was in the prompt and that the cited passage plausibly supports the sentence beside it (a small model can score that). Drop or flag anything that fails.
prompt shapetext
Answer the question using only the passages below. Cite the passage id
in square brackets after each claim, like [p3]. If the passages do not
contain the answer, say "Not found in the provided documents."

[p1] (Handbook › Leave › Parental, p.12) ...
[p2] (Handbook › Leave › Sick, p.9) ...
...

Question: {question}

Evaluate from day one

A RAG system without an evaluation set is a demo. Build the set before the pipeline, and grow it as real questions arrive. Three measurements cover most of what goes wrong:

  1. Retrieval hit rate. For each question with a known source passage, is that passage in the top-k after reranking? This isolates the retriever from the model, and it is where most failures live.
  2. Faithfulness. Does every claim in the answer trace to a cited passage? Score with a rubric or with a second local model acting as a judge, and spot-check the judge.
  3. Answer quality. Against a reference answer, does the response cover what it should? Exact match for factual lookups; a rubric for open questions.

Run the set on every change: a new embedding model, a different chunk size, a new generator, a quantization change. The numbers will tell you which stage moved, and the set becomes the regression suite that lets the stack evolve without regressing.

The reference stack, in one table

StageRuns onNotes
IngestParser of choice (per format)Preserve structure; attach metadata and access groups
De-identifyRules + local NER modelBefore embedding, if the corpus is regulated
Embedllama.cpp or sentence-transformersOne model, versioned, stored beside the vectors
StorePostgres + pgvector, or SQLite + sqlite-vecVectors, full-text index, and metadata in one place
RetrieveVector + keyword, RRF mergeFilter by access group at query time, never after
RerankLocal cross-encoderTop 40 → top 6; the biggest quality lever
Generatellama-server / Ollama / vLLM8B–14B at Q4 on 24 GB; larger on unified memory
Cite & checkPost-processing + small modelEvery citation verified against the prompt
EvaluateHeld-out set, run on every changeHit rate, faithfulness, quality
Table 02 One machine, no external calls at any stage. The generator's size follows the hardware guide.

Strygon builds this shape for operators whose documents cannot leave the building, and it is the retrieval layer underneath its own in-house products. The Local & private AI service describes how that is scoped as an engagement.

Questions people ask about this.

Answered in full, in plain text — the same words the page's structured data carries.

  1. Do you need a vector database to build RAG locally?

    Not a separate one. Postgres with pgvector or SQLite with sqlite-vec holds vectors, metadata, and a full-text index in the same database, which is simpler to run and makes hybrid retrieval one extra query. A dedicated vector store earns its place at scales of many millions of vectors or when you want its specific API; below that, an extension to the database you already have is the better call.

  2. Which embedding model should you use for local RAG?

    One that runs locally, that you have tested on your own corpus, and that you then never change without re-indexing. The open families in the bge, e5, nomic-embed, and gte lines are all serviceable and small enough to run on any GPU. Evaluate two or three on a hundred real questions, measure retrieval hit rate, and pick the winner; the difference between models is smaller than the difference between having a reranker and not.

  3. How is RAG different from fine-tuning?

    RAG puts facts in the prompt at query time; fine-tuning changes the model's weights. Use retrieval for anything that changes, anything that needs a citation, and anything the model should not have memorized. Use fine-tuning for style, format, and domain language. Most production systems use retrieval alone; a few add a light fine-tune on top to make the model follow the citation format reliably.

Next step

Send the stack you have.

Describe the model, the data and the hardware in a few lines. Strygon replies with what the architecture likely needs, before any talk of price.