RAG Explained: Find Evidence Before Asking AI to Answer

You ask an assistant about a lesson that changed yesterday. A model may not know the update. Retrieval-augmented generation, or RAG, first finds relevant information and then uses it as context for a generated answer.

RAG does not retrain the model for each question. It connects retrieval and generation. Better evidence can help an answer, but retrieval can miss the right text and generation can still make mistakes.

Follow the full path

1. Collect documents you are allowed to use.
2. Split them into useful sections while keeping titles and source information.
3. Build a searchable index.
4. Search for relevant sections when a question arrives.
5. Check access and choose a small useful context.
6. Ask the model to answer from that evidence, with source references.
7. Check the result and allow an honest no-answer response.

Start with retrieval you can understand

This local example uses TF-IDF word matching and cosine similarity. It does not use a vector database or a model API. It demonstrates retrieval, not a complete generative RAG system. Install scikit-learn and save rag_search.py:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

DOCUMENTS = [
    {"id":"python-1","text":"Python lists store ordered items. Use append to add an item."},
    {"id":"docker-1","text":"A Docker image packages an app. A container runs from an image."},
    {"id":"mcp-1","text":"MCP connects an AI application with tools and data through a common protocol."}]

class Search:
    def __init__(self,documents):
        self.documents = documents
        self.vectorizer = TfidfVectorizer()
        self.matrix = self.vectorizer.fit_transform(d["text"] for d in documents)
    def retrieve(self,question,k=2):
        scores = cosine_similarity(self.vectorizer.transform([question]),self.matrix)[0]
        order = scores.argsort()[::-1][:k]
        return [dict(self.documents[i],score=round(float(scores[i]),3))
                for i in order if scores[i] > 0]

if __name__ == "__main__":
    search = Search(DOCUMENTS)
    for query in ["How do Python lists add an item?","ocean tide forecast"]:
        hits = search.retrieve(query)
        print(query)
        print(hits if hits else "No matching evidence. Do not invent an answer.")

Run python rag_search.py. The Python question should retrieve python-1 first. The unrelated ocean question should return no matching evidence. The score is a text similarity score, not a probability that an answer is correct.

Add generation as a separate step

Send the question and the retrieved source IDs and text to a chosen model using the API lesson. An instruction could say: “Use only these passages. Cite their source IDs. If they do not support the answer, say what information is missing.” Do not claim that the retrieval code above already generates an answer.

A citation must support the sentence it is attached to. The model can invent an ID or cite an irrelevant passage, so validate IDs and review support. Show the source title and link where possible so a reader can check it.

Embeddings and hybrid search

Dense embeddings can help find related meaning even when words differ. You encode documents and questions with a compatible embedding model and compare their vectors. A vector database stores and searches those vectors. It does not automatically solve document quality or permissions.

Keyword search can be strong for product codes and exact names. Hybrid search combines signals, and a reranker can reorder candidates. Measure whether these additions improve your actual questions before adding them.

Keep access with the documents

A user must not receive a private document just because it appears in the index. Enforce authorisation in the retrieval path before content reaches the model. Keep document permissions and deletions up to date. Caches must respect the same access boundaries.

Retrieved text can contain instructions from an attacker. Treat it as evidence, not as trusted application policy. Do not let a document grant a tool new permissions or request that secrets be revealed.

Evaluate the two parts

Make a question set with expected source documents, including unanswerable questions. First check whether retrieval finds the needed evidence. Then check whether the answer is correct, supported and complete. This tells you whether to improve search or generation.

Track document versions, index freshness, response time and cost. A stale source can produce a well-cited but outdated answer. Different chunk sizes and overlap need testing; there is no universal setting for every collection.

Practice

Add a document with a similar title but a different meaning. Add two sources that disagree. Define how you would show the date and uncertainty rather than selecting a convenient answer. Then compare keyword retrieval with a chosen embedding model on the same question set.

Official references

Reviewed on 25 October 2026. Use documentation for your installed versions.

• LlamaIndex RAG concepts
• TF-IDF text representation
• Security boundaries for connected tools

All AI & Machine Learning tutorials