Guide · Layer 03 · Models
Fine-tuning small models with LoRA and QLoRA: when it pays, and how
Fine-tuning is the most over-reached-for tool in local AI and, in the right place, the most effective one. This guide sets out the decision first (prompt, retrieve, or tune), then the mechanics: what a low-rank adapter actually changes, how much memory it takes on the hardware you have, how to build a dataset that helps rather than harms, and how to get the result back into a runtime you can serve.
- Series
- 06 of 09
- Sections
- 07
- Updated
- Sep 14, 2026
- Reading time
- 11 min
- Written for
- developers
Prompt, retrieve, or tune
Three tools change what a model produces, and they are not interchangeable. A prompt changes behaviour for one call, costs nothing to iterate, and is the right first move for almost everything. Retrieval puts facts in front of the model at query time and is the right tool for anything that changes or needs a citation, as the RAG guide covers. Fine-tuning changes the weights, and it is the right tool for a narrow set of problems that the other two cannot reach:
- Format discipline. The model must emit a strict structure (a schema, a report layout, a house style) every time, and prompting gets it to ninety-five percent but not a hundred.
- Domain language. The corpus uses vocabulary, abbreviations, or conventions the base model handles poorly, and retrieval cannot fix comprehension.
- Consistent tone or persona across thousands of outputs where a long system prompt is expensive to re-send on every call.
- Distillation. A large model produces the outputs you want; a small one is fast enough to ship. Train the small one on the large one's outputs.
- Prompt compression. A task that needs a two-thousand-token system prompt to work can often be tuned into the weights, cutting per-call cost and latency.
What fine-tuning does badly is add knowledge. Teaching a model new facts through supervised tuning is unreliable, hard to update, and impossible to cite. If the goal is for the model to know something, retrieve it.
What LoRA changes
Full fine-tuning updates every weight in the model and needs memory for the weights, their gradients, and the optimizer's state, roughly sixteen bytes per parameter with a standard AdamW setup in mixed precision. For a 7B model that is over a hundred gigabytes before activations. It is not a single-GPU job.
LoRA (low-rank adaptation) freezes the base model and trains a pair of small matrices beside each targeted weight matrix, whose product is a low-rank update added at inference. The number of trainable parameters drops by orders of magnitude, the base weights never change, and the result is a small adapter file that can be swapped, stacked, or merged. Rank (r) sets the adapter's capacity; 8 to 64 covers most uses, and the attention projections plus the feed-forward layers are the usual targets.
QLoRA goes one step further: the frozen base is held in 4-bit (NF4) while the adapters train in higher precision, with paged optimizer state to survive memory spikes. That is what puts a 7B–14B fine-tune on a single consumer card.
| Method | Trainable params | Approximate memory, 7B–8B model | Fits on |
|---|---|---|---|
| Full fine-tune, mixed precision | All | ~16 bytes/param → 110+ GB plus activations | Multi-GPU only |
| LoRA, BF16 base | ~0.1–1% | Base ~16 GB + activations + adapter | 24 GB with short sequences; 48 GB comfortably |
| QLoRA, 4-bit base | ~0.1–1% | Base ~4–5 GB + activations + adapter | 12–16 GB with care; 24 GB comfortably |
The dataset is the work
Every hour spent on the dataset saves ten on the training run. For style, format, and domain-language objectives, a few hundred to a few thousand high-quality examples is the working range; more examples of lower quality make the model worse, not better.
- Collect real inputs. The prompts the model will actually receive, not idealized ones. Diversity across the input distribution matters more than volume.
- Write or curate the outputs you want. Every example is a demonstration the model will imitate, including its mistakes. If a large model drafts them, a person reviews them.
- Deduplicate and clean. Near-duplicates teach the model to overfit one shape. Strip anything that should not end up in weights: personal data, secrets, content you do not have rights to train on.
- Hold out a test split before you look at any result, and never train on it. Fifty to a hundred examples is enough to see movement.
- Use the chat template the base model was trained with. Mismatched templates are the most common reason a tuned model behaves worse than the base.
{"messages": [
{"role": "system", "content": "You extract structured fields from intake notes."},
{"role": "user", "content": "Pt reports 3 wks of intermittent chest tightness..."},
{"role": "assistant", "content": "{\"duration_weeks\": 3, \"symptom\": \"chest tightness\", \"pattern\": \"intermittent\"}"}
]}If the corpus is regulated, the de-identification layer runs over the dataset before training with the same rigour as before embedding. A model memorizes; an identifier that enters the weights can be recovered by the right prompt.
Tooling
The training stack has consolidated around a few options, all of which run on a single local GPU:
- PEFT and TRL (Hugging Face) are the reference libraries: PEFT implements LoRA and QLoRA, TRL wraps supervised fine-tuning and preference training. Everything else builds on them.
- Unsloth patches the training path for large speed and memory improvements on consumer NVIDIA cards, with the same API surface.
- Axolotl and LLaMA-Factory are config-driven front ends: a YAML file describes the base model, dataset, adapter settings, and run.
- MLX LM is Apple's native path on M-series machines and runs LoRA and QLoRA against unified memory.
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
lora = LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
)
trainer = SFTTrainer(
model=model, # loaded in 4-bit for QLoRA
train_dataset=dataset, # the messages-format JSONL above
peft_config=lora,
args=SFTConfig(
output_dir="out", num_train_epochs=2,
per_device_train_batch_size=2, gradient_accumulation_steps=8,
learning_rate=2e-4, bf16=True, gradient_checkpointing=True,
max_length=2048, logging_steps=10, eval_strategy="steps",
),
)
trainer.train()Evaluate before, then after
Run the held-out split through the base model first and record the score with the same rubric you will use afterwards: exact match for extraction, a compile-and-test pass for code, a checklist for prose. Then train, then run it again. If the number did not move, the dataset did not teach what you thought. If it moved on the training examples but not on the held-out ones, the model memorized instead of learning.
Also run a handful of general prompts unrelated to the task. Narrow tuning can degrade general ability, and a model that formats invoices perfectly and can no longer summarize an email may not be the one you wanted. A low learning rate, one or two epochs, and a modest rank keep the change local.
From adapter to served model
The adapter is a small file; serving it means either loading it alongside the base at runtime (vLLM and several other servers support hot-loading LoRA adapters, which is ideal when many adapters share one base) or merging it into the weights to produce a standalone model. For a single-model deployment on llama.cpp or Ollama, merging is the usual path:
# 1. merge the adapter into the base (PEFT)
python merge.py --base ./base --adapter ./out --dst ./merged
# 2. convert to GGUF and quantize (llama.cpp tools)
python convert_hf_to_gguf.py ./merged --outfile merged-f16.gguf
llama-quantize merged-f16.gguf merged-Q4_K_M.gguf Q4_K_M
# 3. re-run the held-out eval on the QUANTIZED file, not the merged one
# (quantization can move the number — see the quantization guide)That last step is the one most often skipped. The evaluation belongs on the exact artifact you will serve, at the exact quantization, because a tuned model's gains can be sensitive to the bits it is served at. The quantization guide explains why.
Risks, named
- Catastrophic forgetting. The model loses general ability while gaining the narrow one. Mitigate with a small learning rate, few epochs, and general examples mixed into the dataset.
- Overfitting. Perfect on the training set, worse on new inputs. Mitigate with a real held-out split, early stopping, and more diverse data rather than more epochs.
- Data leakage into weights. Anything in the dataset can come back out. De-identify first; never include secrets or content you cannot ship.
- Template drift. Training with a different chat template than the runtime uses. Check the template before and after.
- Licence terms. The base model's licence governs the tuned one. Read it before the run, especially for anything that will be sold or redistributed.