RAG, fine-tuning or long context?

The options

Better prompts and examples. Always first. Clear instructions, a few good examples and structured output solve more problems than people expect, cost nothing to build and change in minutes.

Long context. Put the whole document, codebase section or conversation into the prompt. Simple, no index to maintain, and the model sees everything at once, which helps tasks that need the whole picture (summarising a contract, comparing two reports). The costs: every request pays for all those tokens, latency grows with input size, the window has a hard limit, and models may use details in the middle of very long inputs less reliably. Prompt caching softens the cost when the same material is reused across many requests.

Retrieval (RAG). Index a large or changing body of documents and fetch only the relevant passages per request. Scales to far more material than any window, keeps facts current by re-indexing, supports per-user permissions and citations. The costs: a pipeline to build and evaluate (chunking, embeddings, search, re-ranking), and answers can only be as good as what retrieval finds.

Fine-tuning. Train a model further on your examples so that a behaviour becomes the default: a house format, a classification scheme, a tone, a domain's way of reasoning, or a smaller model imitating a larger one on one narrow task. Parameter-efficient methods such as LoRA make this cheaper by training a small set of added weights. The costs: you need a good dataset (often hundreds to thousands of high-quality examples), a training and evaluation process, and retraining when the base model or the target behaviour changes. It is a poor store for facts: hard to update or delete, no citations.

A comparison

These combine. A common production setup is retrieval for knowledge, a well-tested prompt for behaviour, and, at high volume, a fine-tuned smaller model for a narrow step such as classification or extraction.

Deciding in practice

1. Write the eval set first, so each option is measured on the same cases.
2. Try prompt improvements, then long context if the material fits. Measure quality, cost per request and latency.
3. If the material does not fit, is too costly to send every time, changes often, or needs per-user permissions, build retrieval and evaluate retrieval and answers separately.
4. If quality plateaus on format, style or a narrow decision, and you have the volume to justify it, try fine-tuning against the same eval, and keep the prompt-only version as the baseline.

Re-run the comparison when models change. Context windows, caching prices and base-model quality have moved quickly, so a choice that was right a year ago may not be the cheapest or best today.