Exploring RAG architecture: How retrieval-augmented generation is revolutionizing GenAI

This article is written by Laura Meyer, an Engineer at a leading consultancy, specializing in AI, data science, and DevOps, with extensive experience in GenAI innovation and delivering technical training.

Bridging the gap: Overcoming LLM limitations with retrieval-augmented generation (RAG)

Large language models (LLMs) have revolutionized industries by enabling intelligent content creation, analysis, and automation. Their ability to learn from massive and diverse datasets has unlocked new possibilities in how we interact with technology, solve complex problems, and innovate. Yet, even these powerful tools have their challenges: LLMs can be resource-intensive, struggle with real-time information, lack access to proprietary or private data and may not be tailored to specific domains. Their reliance on static training data limits their ability to adapt quickly to changing environments, and fine-tuning them can inadvertently introduce errors (so-called “hallucinations”) or erase valuable prior knowledge. In dynamic, high-stakes industries where accuracy, confidentiality, and up-to-date insights are critical, these limitations highlight the need for a more agile solution.

Retrieval-augmented generation (RAG) emerges as a groundbreaking solution, redefining how we use language models. By integrating external knowledge retrieval into the generative process, RAG enables AI systems to access up-to-date, domain-specific information and provide more accurate, context-aware responses.

This innovation bridges the gap between static AI knowledge and the real-world need for timely, reliable and contextually-relevant information.

RAG combines two essential components:

1. Retrieval mechanism: This component pulls relevant, domain-specific information from external sources like knowledge bases, databases, or even the web.
2. Generative model: The retrieved information is then passed to a GenAI model (like GPT) to craft responses grounded in the retrieved data.

Imagine you’re taking an open-book exam. You have all the resources you need—notes, textbooks, and references—but the key to success isn’t memorizing everything. Instead, it’s knowing where to look, finding the right information quickly, and applying it effectively to answer complex questions. That’s exactly how RAG works.

In this scenario, the generative AI model is like the student. It has foundational knowledge (pre-trained data), but instead of storing every detail, it relies on an “open book”—an external knowledge base (like databases, websites, or even PDFs). The retrieval system acts like flipping through pages or searching your notes for the most relevant information. Once found, the model integrates this knowledge with its existing understanding to craft a clear and thoughtful response.

RAG, like a prepared student, thrives by combining what it knows with the ability to quickly access and use external resources, making its answers precise, dynamic, and context-aware.

Summarizing RAG architecture:

The flow diagram below illustrates the step-by-step process of a RAG system, where a user’s query is processed through embedding, retrieval, and generative modeling to produce context-aware and accurate responses.

1. The process begins when a user submits a query or request. This input could be a question, a search prompt, or any task requiring a context-aware response.
2. The query is converted into a numerical vector representation using an embedding model (e.g., text-embedding-ada-002). This representation captures the semantic meaning of the input and enables effective comparison with stored knowledge.
3. Relevant external knowledge sources, such as structured databases, unstructured documents, knowledge graphs, APIs, or websites, are identified as potential candidates for the query.
4. The raw information is broken into smaller, manageable chunks of text. This step maintains the context within each chunk while optimizing the search and indexing processes.
5. Each chunk is converted into a numerical vector representation using an embedding model. These vectors are stored in a vector database for fast retrieval, where they can be indexed and searched based on semantic similarity.
6. When the user query is received and transformed into a vector, this vector is passed to the retrieval system, which searches the vector database to find the most semantically similar chunks of text (or documents) based on the query vector.
7. The retrieval system employs techniques such as vector-based similarity search (embedding matching) or keyword-based filtering to locate the most pertinent and contextually relevant data.
8. The most relevant documents or chunks of text are retrieved and preprocessed, ensuring the generative model has access to accurate, up-to-date, and factual information.
9. The user’s query, retrieved chunks, and additional instructions are combined to create an enriched input for the generative model. This augmentation ensures the model has all the necessary context to deliver an accurate response.
10. The LLM (e.g., GPT-4, PaLM) processes the augmented input, blending the retrieved knowledge with its pre-trained understanding to generate a well-formed and context-aware response.
11. The final response is presented to the user, whether as a direct answer, a detailed explanation, or a summarized document tailored to the user’s request.

Let’s dive into what makes RAG so transformative and guide you through building your very own RAG system.

Building a simple RAG system: Hands-on guide

Basic pass-through prompt

Before exploring RAG functionality, let’s first examine a simple pass-through prompt. This demonstrates how a standalone LLM processes questions compared to a RAG architecture. A pass-through prompt directly sends a user query to the model without introducing any additional context or data retrieval, allowing us to understand how the model generates responses purely based on pre-existing knowledge from its training data.

We first need to import the necessary OpenAI components and initialize the client to interface with the LLM. The OpenAI() class provides access to the chat-based models, such as gpt-3.5-turbo .

Next, let’s define a user query to test the model’s response. Do you know how many times per year you should change your car’s oil? 🤔

This question is stored in the user_query variable and will be sent directly to the LLM without any auxiliary context or retrieval mechanisms.

The client.chat.completions.create() function is used to interact with the LLM. This function specifies the model, user input, and other parameters like temperature, which controls the response’s randomness—higher values lead to more creative and varied outputs, while lower values make responses more deterministic.

Once the response is received, response.choices[0].message.content extracts the actual generated text from the model’s response. This is the answer the model generates based on the query.

The LLM’s response demonstrates its ability to synthesize general knowledge and context. When asked, “How many times per year should a car’s oil be changed?” it provided the common recommendation: every 3,000 to 5,000 miles or every 3 to 6 months, whichever comes first.

Furthermore, it acknowledged contextual nuances by noting that the interval may vary depending on factors like the car’s make, model, and type of oil used. This showcases the LLM’s ability to incorporate general knowledge and reason through edge cases even without additional contextual inputs or retrieval-based context augmentation.

System prompt