The problem RAG solves
Imagine asking an ordinary Large Language Model about your company's internal vacation policy. What happens? At best, it honestly says: "I don't know that." At worst, it invents an answer that sounds plausible but is entirely made up. Why? Because the model only knows what was in its training data, and your company wiki was certainly not part of that training dataset.
This applies to many things: today's news, internal documents, your own notes, customer data, your product's manual, legal texts from a specific law firm. Anything that wasn't public or was too recent, the AI simply doesn't know.
One option would be to retrain the model. But that's expensive, slow, and massively overkill for most cases. There's a far more elegant way, and it's called RAG.
What RAG actually is
RAG stands for Retrieval Augmented Generation. It sounds clunky, but at its core it's a simple idea: before the AI answers, we search for the relevant information from a collection of documents and place it directly into the prompt. The AI then answers based on these concrete texts, not from the hazy memory of its training.
Picture the AI as a very bright student who has the world's general knowledge in their head, but never studied your specific field. In a normal exam, they'd guess. RAG is the moment you place the right cheat sheet on the table in front of every question. Suddenly they don't answer from gut feeling, but from the text right in front of them.
The beauty of it: the student isn't "retrained." Their entire knowledge of language, logic, and the world stays intact. We simply give them exactly the right hints at exactly the right time.
RAG turns a general-purpose AI into a specialist without retraining it even once.
How RAG works, in four steps
Behind this simple idea sits a small machine. It runs in two phases: once when building the knowledge base (happens once, or regularly), and once for every user question.
Step one. Cutting up documents
Your documents. PDFs, wiki pages, manuals, notes, get split into small, meaningful pieces. These pieces are called chunks. A chunk can be a paragraph, a section, or a few sentences. Important: they must be small enough to fit in the prompt, but large enough to make sense on their own.
Step two. Turning meaning into numbers
Each chunk is passed through a special model that turns the text into an embedding, a long list of numbers that represents the meaning of the text. Two texts with similar content get similar number lists.
Step three. Into the database
These number lists are stored together with the original texts in a vector database. This is a special database that doesn't search by words, but by similarity of meaning.
Step four. Ask a question, get an answer
When the user asks a question, that question is also turned into an embedding. The vector database finds the most similar chunks. These chunks are written into the prompt together with the original question, and the AI answers based on these concrete passages.
Embeddings, when meaning becomes coordinates
The term embedding sounds abstract at first. But the image behind it is a nice one: picture a huge space, not three dimensions like ours, but hundreds or thousands. In this space, every word, every sentence, every paragraph is assigned a fixed point. Texts with similar meaning land close together; texts with no relation land far apart.
The sentences "How do I request vacation?" and "Where do I find the vacation form?" use different words but have very similar meaning, their points in embedding space lie practically next to each other. A classic keyword search would miss that. An embedding system finds the connection instantly.
That's the magic: RAG doesn't just find word matches, it finds related ideas. Even if your wiki uses completely different phrasing than the user's question, the right passage ends up in the answer.
The vector database, a library for meanings
A vector database (also called a vector store) is optimized to find the most similar items among thousands, millions, or billions of embeddings in a flash. Well-known names in this space are Pinecone, Chroma, Qdrant, Weaviate, or Milvus. Some run in the cloud, others you can run on your own machine.
For the user, it doesn't matter which one you choose, the principle is always the same: it takes vectors in, gives vectors out, and does it fast.
A vector database doesn't replace your regular database. It complements it. Structured data (tables, numbers, IDs) stays exactly where it already lives. The vector database is responsible for text and meaning.
What RAG gets you
The benefits are enormous, and that's exactly why RAG is showing up everywhere right now:
Freshness
You can add, change, or delete documents at any time. The AI knows immediately, no retraining needed.
Company-specific knowledge
Internal documents, product manuals, customer history, everything the AI would otherwise never see, becomes usable.
No training
You don't have to touch a model, rent any GPUs, or wait weeks. RAG works with any standard LLM via the API.
Traceable sources
Because the AI answers based on concrete chunks, you can cite the sources, the user sees where the answer came from.
Where RAG hits its limits
The method is powerful, but it's no cure-all. There are typical pitfalls:
RAG is only as good as the underlying data. If your documents are outdated, contradictory, or incomplete, the AI answers exactly that way, just with the convincing tone of an LLM. Garbage in, garbage out applies doubly here.
A second problem is the chunking itself. If a chunk is too small, context is missing, the beginning of a thought sits in one piece, the end in another. If it's too large, the search becomes less precise. Good chunking is a small art.
Third: what the search doesn't find also can't make it into the answer. If the question is phrased in a way that finds no matching chunk. Tough luck. The AI then either doesn't answer at all or falls back on its training knowledge and possibly starts guessing. More on this on the next page, when we talk about hallucinations.
Source citations aren't decoration with RAG, they're mandatory. When the AI says "according to document XY...", people can verify it. Without source citations, it stays a matter of faith, and that's a risk for business-critical topics.
Where you already see RAG today
Practically every professional AI chatbot with "knowledge about my company" runs on RAG behind the scenes. The same goes for "chat with your PDF" tools, legal research AIs, code assistants that "know" your codebase, or support bots that have your help articles memorized. Many agents also use RAG as one of their tools, the agent calls the vector database like any other tool.
What you now know
- RAG (Retrieval Augmented Generation) gives an AI the matching "cheat sheet" from a document collection before every answer, without retraining.
- The method is based on embeddings (meaning as a list of numbers) and a vector database that searches by similarity of meaning.
- RAG makes AI current and company-specific, but it's only as good as the underlying data and the chunking.