RAG: retrieve, ground, answer with sources

Retrieval-augmented generation finds the relevant pieces of your documents first, then makes the model answer from them.

The idea

A model only knows what it was trained on and what is in its prompt. RAG fixes that for your own data in two phases. Ingest (done ahead of time): split documents into chunks, embed each chunk, store the vectors. Answer (at question time): embed the question, retrieve the closest chunks, put them in the prompt as sources, and tell the model to answer only from them and cite which ones it used. The answer is grounded in text you can show the user.

Code

public sealed class RagService(IEmbedder embedder, IVectorStore store, ILlmClient llm)
{
    public async Task IngestAsync(string document)
    {
        var chunks = Chunker.Split(document, maxChars: 800, overlap: 100);
        foreach (var c in chunks)
            await store.AddAsync(c.Id, c.Text, await embedder.EmbedAsync(c.Text));
    }

    public async Task<Answer> AskAsync(string question)
    {
        float[] q = await embedder.EmbedAsync(question);
        var hits = await store.SearchAsync(q, topK: 2);
        string sources = string.Join("\n",
            hits.Select(h => $"<source id='{h.Id}'>{h.Text}</source>"));
        string prompt = "Answer only from the sources and cite their ids. If they do not contain the answer, say so.\n"
                      + sources + "\nQuestion: " + question;
        string text = await llm.CompleteAsync(prompt);
        return new Answer(text, hits.Select(h => h.Id).ToList());
    }
}

Tips

• Interview line: "RAG grounds the model in retrieved documents, which reduces hallucination and lets me use private or fresh data without retraining."
• It reduces made-up answers but does not remove them. If retrieval misses the right chunk, the model has nothing to ground on, so tell it to say "not found".
• Chunk size matters: too big and the relevant bit is drowned out, too small and it loses context. A small overlap between chunks helps.
• Return the source ids with the answer so users can check where it came from.

In the .NET library

Keep ingestion (a background job) and answering (a request path) separate. Both depend on interfaces such as IEmbedder and IVectorStore, so you can swap vector databases and unit test with in-memory fakes.