Retrieval-augmented generation

Indexing: chunk, embed, store

• Chunk. Split documents into passages small enough to be specific and large enough to make sense alone; a few hundred tokens is a common starting point. Split on structure (headings, paragraphs) rather than at fixed character counts, overlap neighbours slightly, and keep the title or heading path with each chunk so it has context.
• Embed. An embedding model turns text into a vector of numbers such that texts with related meanings end up close together. Use the same embedding model for documents and questions.
• Store. Keep each vector with its text, source, and metadata (owner, permissions, date) in a vector index: a dedicated vector database or a vector extension of a database you already run.

Retrieval: search, re-rank, trim

At question time, embed the question and find the nearest chunks, usually by cosine similarity. Pure vector search can miss exact terms (product codes, error numbers, names), so many systems run keyword search (such as BM25) alongside and merge the results ("hybrid search"). A re-ranker, a slower model that reads the question and each candidate together, then reorders the top few dozen so only the best handful go into the prompt.

The toy below uses word-count vectors instead of a real embedding model, which is enough to show chunking, cosine similarity, top-k and citing sources.

Generation: answer from the evidence

Put the passages in clearly delimited, labelled blocks and instruct the model to use only them, to cite passage ids for each claim, and to say when the passages do not contain the answer. Some APIs have built-in citation features that return the exact quoted spans. Your code can then check that every cited id was actually supplied, and show users links to the sources.

Where RAG goes wrong

• Retrieval misses. The answer exists but was not retrieved: chunks too large or too small, missing keyword search, no re-ranking, or the question phrased very differently from the document. Build a retrieval test set of questions with the chunk ids that should come back, and track recall at k.
• Too much context. Stuffing in fifty passages raises cost and latency and can make answers worse; models tend to use the beginning and end of long inputs more reliably than the middle.
• Stale or forbidden data. Re-index when documents change, delete chunks when documents are deleted, and filter by the user's permissions at query time, before anything reaches the prompt.
• Poisoned documents. Retrieved text is untrusted input. A document can contain instructions aimed at the model (indirect prompt injection); see the security topic.