AI & Machine Learning

RAG Explained: How to Give an LLM Access to Your Own Knowledge

Learn how RAG works, explore embeddings and vector databases, and build a working AI-powered document question-answering system using Python, FAISS, and Ollama.

RAG Explained: How to Give an LLM Access to Your Own Knowledge

Introduction

Large Language Models (LLMs) have changed how we interact with information. We can ask questions in natural language, generate code, summarize documents, and even build AI assistants.

But there is one important limitation.

What happens when an LLM doesn't know about the information you want it to use?

Imagine building an AI assistant for a company. The company has thousands of internal documents containing employee policies, technical documentation, product information, and customer support instructions.

You cannot expect a general-purpose LLM to know all this private information.

One possible solution is to train or fine-tune a model, but that isn't always necessary or practical.

Instead, we can use something called Retrieval-Augmented Generation (RAG).

RAG allows an LLM to retrieve relevant information from an external knowledge source and use that information when generating an answer.

Let's understand how it works from an engineering perspective.

intro_rag.jpeg

1. What is Retrieval-Augmented Generation?

Retrieval-Augmented Generation is an architecture that combines information retrieval with the text-generation capabilities of an LLM.

Rather than depending entirely on the knowledge encoded in its model parameters, an LLM receives relevant external information at inference time.

There are three fundamental operations:

  • Retrieval: Find information relevant to the user's question.

  • Augmentation: Add the retrieved information to the model's input.

  • Generation: Let the LLM generate an answer using the supplied context.

Think about building a chatbot that answers questions using your PDF documents.

Without RAG, the model might not know anything about your documents.

With RAG, your application searches those documents, extracts relevant information, and provides it to the model before asking it to answer.

The LLM doesn't need to memorize the entire document collection.

2. Understanding the RAG Architecture

A RAG application generally has two major workflows:

  1. Document ingestion (preparing the knowledge base).

  2. Query processing (answering user questions).

flow_chart.jpeg

Let's examine both.

A. Document Ingestion Pipeline

Before users can ask questions, we need to prepare our documents.

Step 1: Document Loading

First, collect information from different sources, such as:

  • PDF files

  • Text documents

  • Markdown files

  • Websites

  • Databases

The objective is to extract usable text and associated metadata.

Step 2: Text Chunking

Imagine having a document containing 100 pages.

Passing the entire document to an LLM for every question would be inefficient and could exceed its context window.

Instead, we divide the document into smaller pieces called chunks.

For example:

Original document:

"Machine learning is a field of artificial intelligence. It enables computers to learn patterns from data..."

After chunking:

  • Chunk 1: Introduction to machine learning.

  • Chunk 2: Types of machine learning.

  • Chunk 3: Supervised learning algorithms.

Chunking is an important engineering decision.

If chunks are too small, they may lose meaningful context. If they are too large, retrieval may return irrelevant information alongside useful content.

Step 3: Generating Embeddings

Now we need a way to search these chunks based on their meaning rather than just matching exact words.

This is where embeddings become useful.

An embedding model converts text into a numerical vector representing its semantic characteristics.

For example:

"How can I train a neural network?"

And:

"What are the steps involved in training a deep learning model?"

These sentences use different words but have similar meanings.

An embedding model can represent them as vectors that are relatively close in the embedding space.

This makes semantic search possible.

flow_chunk_embd.jpeg

Step 4: Vector Database

After generating embeddings, we store them in a vector index or vector database.

Alongside each vector, we typically store:

  • Original text chunk

  • Document identifier

  • Page number

  • Metadata

  • Other information needed for citations

Examples of vector databases and search systems include:

  • FAISS

  • Chroma

  • Qdrant

  • Pinecone

Now our knowledge base is ready for retrieval.

B. Query Processing Pipeline

Once the documents have been indexed, users can start asking questions.

Step 1: User Query

Suppose a user asks:

"Which machine learning algorithm is suitable for predicting house prices?"

We convert this question into an embedding using the same compatible embedding model.

Step 2: Similarity Search

We compare the query embedding against stored document embeddings.

A similarity function, such as cosine similarity, can help identify vectors representing semantically related content.

The retrieval system returns the most relevant chunks.

For example:

  • Chunk 12: Linear regression.

  • Chunk 18: Decision tree regression.

  • Chunk 24: Random forest regression.

how_retrival_works.jpeg

Step 3: Context Augmentation

We combine the retrieved information with the original question.

A simplified prompt could look like this:

"You are an AI assistant. Answer the question using the provided context.

Context:
[Retrieved document chunks]

Question:
Which machine learning algorithm is suitable for predicting house prices?

If the provided context does not contain enough information, clearly say so."

The retrieved context gives the LLM additional information to work with.

Step 4: Response Generation

Finally, the LLM processes the question and retrieved context to generate a response.

The answer is now grounded in the information retrieved from the knowledge base.

generation_flowchart.jpeg

That's the basic RAG pipeline.

3. Building a Working RAG Application Using Python and FAISS

Understanding the architecture is useful, but let's implement it.

We'll build a small RAG application using:

  • FAISS: For storing and retrieving document embeddings.

  • Sentence Transformers: For converting text into embeddings.

  • Ollama: For generating answers using a locally running LLM.

  • NumPy: For handling numerical vectors.

The implementation will use sample documents directly in Python, so you can run it without downloading a dataset.

Step 1: Install the dependencies

Install the required Python libraries:

pip install faiss-cpu sentence-transformers numpy requests

Install Ollama from https://ollama.com/.

Download the language model:

ollama pull llama3.2:3b

Make sure Ollama is running before executing the Python program.

Step 2: Complete implementation

Create a file named rag_faiss.py.

import faiss
import numpy as np
import requests

from sentence_transformers import SentenceTransformer


# ---------------------------------
# 1. Sample Knowledge Base
# ---------------------------------

documents = [
    {
        "source": "machine_learning.txt",
        "text": """
        Machine learning is a branch of artificial
        intelligence that enables computers to learn
        patterns from data without explicitly
        programming every rule.

        Supervised learning uses labeled datasets
        to train models. Common algorithms include
        linear regression, decision trees,
        random forests and support vector machines.
        """
    },
    {
        "source": "deep_learning.txt",
        "text": """
        Deep learning is a subset of machine learning
        that uses neural networks with multiple layers.

        Convolutional Neural Networks (CNNs) are
        commonly used in image processing and
        computer vision.

        Recurrent Neural Networks are designed
        to process sequential information.
        """
    },
    {
        "source": "nlp.txt",
        "text": """
        Natural Language Processing (NLP) enables
        computers to understand and process human
        language.

        Common NLP applications include sentiment
        analysis, text classification, machine
        translation and question answering.
        """
    }
]


# ---------------------------------
# 2. Text Chunking
# ---------------------------------

def create_chunks(text, chunk_size=300, overlap=50):

    chunks = []
    start = 0

    while start < len(text):

        end = start + chunk_size

        chunks.append(text[start:end])

        start += chunk_size - overlap

    return chunks


texts = []
metadata = []

for document in documents:

    chunks = create_chunks(document["text"])

    for chunk in chunks:

        texts.append(chunk)

        metadata.append({
            "source": document["source"],
            "text": chunk
        })


# ---------------------------------
# 3. Generate Embeddings
# ---------------------------------

print("Loading embedding model...")

embedding_model = SentenceTransformer(
    "sentence-transformers/all-MiniLM-L6-v2"
)

embeddings = embedding_model.encode(
    texts,
    convert_to_numpy=True,
    normalize_embeddings=True
)

embeddings = np.asarray(
    embeddings,
    dtype="float32"
)


# ---------------------------------
# 4. Create FAISS Index
# ---------------------------------

dimension = embeddings.shape[1]

index = faiss.IndexFlatIP(dimension)

index.add(embeddings)

print(f"Indexed {index.ntotal} document chunks.")


# ---------------------------------
# 5. Retrieve Relevant Documents
# ---------------------------------

def retrieve(query, top_k=3):

    query_embedding = embedding_model.encode(
        [query],
        convert_to_numpy=True,
        normalize_embeddings=True
    )

    query_embedding = np.asarray(
        query_embedding,
        dtype="float32"
    )

    k = min(top_k, index.ntotal)

    scores, indices = index.search(
        query_embedding,
        k
    )

    results = []

    for score, idx in zip(scores[0], indices[0]):

        if idx == -1:
            continue

        result = metadata[idx].copy()

        result["score"] = float(score)

        results.append(result)

    return results


# ---------------------------------
# 6. Generate Answer Using Ollama
# ---------------------------------

def generate_answer(query, retrieved_docs):

    context = "\n\n".join(
        f"Source: {doc['source']}\n{doc['text']}"
        for doc in retrieved_docs
    )

    prompt = f"""
    You are a helpful AI assistant.

    Answer the question using only the
    information provided in the context.

    If the answer is not present in the
    context, say that you don't have
    enough information.

    Context:
    {context}

    Question:
    {query}

    Answer:
    """

    response = requests.post(
        "http://localhost:11434/api/generate",
        json={
            "model": "llama3.2:3b",
            "prompt": prompt,
            "stream": False
        },
        timeout=120
    )

    response.raise_for_status()

    return response.json()["response"]


# ---------------------------------
# 7. Run the RAG Pipeline
# ---------------------------------

if __name__ == "__main__":

    query = input("\nAsk a question: ")

    print("\nSearching knowledge base...")

    retrieved_docs = retrieve(query)

    print("\nRetrieved Documents:")

    for doc in retrieved_docs:

        print(
            f"\nSource: {doc['source']}"
            f"\nSimilarity: {doc['score']:.4f}"
            f"\nContent: {doc['text']}"
        )

    print("\nGenerating answer...")

    answer = generate_answer(
        query,
        retrieved_docs
    )

    print("\nFinal Answer:")
    print(answer)

Step 3: Run the application

python rag_faiss.py

Try asking:

What is the difference between machine
learning and deep learning?

The application will:

  1. Convert your question into an embedding.

  2. Search the FAISS index.

  3. Retrieve the most relevant document chunks.

  4. Construct a prompt containing the retrieved context.

  5. Send the prompt to Ollama.

  6. Display the generated answer.

Step 4: Understanding the FAISS implementation

There is an interesting detail in this implementation:

index = faiss.IndexFlatIP(dimension)
index.add(embeddings)

IndexFlatIP performs exact inner-product similarity search.

But why are we using inner product instead of directly implementing cosine similarity?

Look at this part:

normalize_embeddings=True

We normalize both document embeddings and query embeddings.

For normalized vectors, their inner product is equal to cosine similarity

Because both vectors are normalized to have the same length, we can calculate their cosine similarity simply by taking their dot product.

This means FAISS can perform cosine similarity search using IndexFlatIP.

We also use:

scores, indices = index.search(
    query_embedding,
    k
)

This returns the similarity scores and indices of the nearest document vectors.

We use those indices to retrieve the original text chunks and their metadata.

Important: This is a small educational implementation. It uses exact similarity search, a simple character-based chunking strategy, and an in-memory FAISS index. A production application would need better document parsing, persistence, more sophisticated chunking, retrieval evaluation, and possibly reranking.

4. Why Does RAG Matter?

RAG solves several practical problems associated with building LLM applications.

Access to private information

An application can retrieve information from internal company documents without requiring that information to be part of the original model training data.

Updating knowledge

When documents change, we can update the knowledge base and its index instead of retraining the entire language model.

Source attribution

Because retrieved chunks can retain document metadata, applications can provide references to the original documents.

Reduced dependence on model memory

The model receives relevant external information instead of relying exclusively on what it learned during training.

However, RAG does not guarantee factual accuracy. Its effectiveness depends heavily on retrieval quality and generation behavior.

5. Is RAG a Perfect Solution?

No. Building a working RAG application is relatively straightforward. Building a reliable RAG application is much harder.

Here are some common problems.

Poor chunking strategy

If important information is divided incorrectly, the retrieval system may return incomplete context.

Incorrect retrieval

A vector database can return semantically similar content that does not actually answer the question.

Hallucinations

Even when relevant context is provided, an LLM can generate incorrect information or introduce unsupported details.

Context limitations

Retrieving too many chunks can increase token consumption, latency, and the amount of irrelevant information given to the model.

This is why production RAG systems often need additional techniques such as:

  • Hybrid search

  • Query rewriting

  • Metadata filtering

  • Reranking

  • Retrieval evaluation

These techniques improve different parts of the retrieval pipeline.

6. Where Should You Go After Learning Basic RAG?

Once you understand the basic architecture, there are several interesting areas to explore.

Advanced RAG: Improving chunking, retrieval, and context construction.

Hybrid Search: Combining keyword-based search with vector similarity search.

Reranking: Reordering retrieved documents using a more sophisticated relevance model.

Agentic RAG: Allowing an AI agent to decide when and how to retrieve additional information.

RAG Evaluation: Measuring retrieval quality and the factual relevance of generated answers.

These are important steps when moving from a simple document chatbot to a more sophisticated AI application.

Conclusion

Retrieval-Augmented Generation is more than just connecting a vector database to an LLM.

It is an architectural approach that combines information retrieval and language generation to build applications that can work with external knowledge.

Understanding the complete pipeline—from document ingestion and chunking to embeddings, retrieval, and generation—is essential for anyone interested in developing practical LLM applications.

The interesting engineering challenge isn't simply making an LLM answer questions.

It's making sure that the information provided to the LLM is relevant, reliable, and useful.

That is where good RAG engineering begins.


Further Reading

  1. Retrieval-Augmented Generation — Microsoft Learn

  2. Common RAG Techniques Explained — Microsoft

  3. FAISS Documentation

  4. Sentence Transformers Documentation

  5. Ollama Documentation

0

0 comments

Sign in to join the discussion.

Loading comments…

FF
Faizan FarhadSee everything by @faizanfarhad2211

Edited by Rajat Chauhan. How articles here are written and edited.