What Is RAG? A Beginner’s Playbook on Answering from Your Own Documents

Meera is a developer in Pune. Her company has a 60-page leave policy, and every week people ask the same questions. “How many sick days do I get?” “Can I carry leave over?” Her manager says, “Just put an AI chatbot on it.”

Meera tries a chatbot. It answers fast and sounds sure. But it says 12 sick days when the policy says 8. The model never read her company’s policy. So she asks the question every beginner asks: what is RAG, and can it fix this?

This story is made up to teach the idea. Any numbers in it are illustrative.

Last verified: 9 November 2026. AI tools change fast; check the linked sources for anything newer.

Who this is for: beginners who know what an AI chatbot is and want it to answer from their own files. Time needed: about 30-40 minutes to read, and a weekend to try it.

What is RAG, in simple words

RAG stands for Retrieval-Augmented Generation. It sounds heavy, but the idea is small: look up the right pages first, then let the AI write the answer from those pages.

Think of an exam. A normal chatbot takes a closed-book exam. It answers from memory, and memory can be wrong or old. RAG is an open-book exam. The AI finds the right page in your book, reads it, and then writes.

The precise version: a retriever searches your documents, and a language model writes the answer using what the retriever found. The idea comes from a 2020 research paper by Patrick Lewis and colleagues, which joined a text generator with a search step over Wikipedia.

The playbook at a glance

1. Pick a small set of documents the AI should know.
2. Cut them into small chunks.
3. Turn each chunk into numbers and store it for search.
4. At question time, find the best chunks and hand them to the AI.
5. Test the answers and fix the weak spots.

Play 1: Pick a small set of documents

Goal: decide what the AI is allowed to answer from.

1. Choose one topic, such as the leave policy.
2. Collect the current version of each file. Remove old copies.
3. Write down 10 real questions people ask. You will test with them later.

Meera takes only the leave policy and its FAQ page. She skips the other 200 files for now.

You are done when: you have a clean folder and a list of 10 questions.

Play 2: Cut the documents into chunks

Goal: make pieces small enough to find, but big enough to make sense.

An AI can only read so much text at once. That limit is the context window, which we explain in our how LLMs work playbook. So we do not hand over all 60 pages. We hand over the few paragraphs that matter.

1. Split each document by headings first, then by paragraphs.
2. Aim for chunks of a few paragraphs each.
3. Let neighbouring chunks overlap a little, so a sentence is not cut in half.
4. Keep the file name and heading with every chunk, so you can show the source later.

Meera splits the policy by heading. “Sick leave” becomes one chunk, “Carry-over rules” becomes another.

You are done when: each chunk answers one idea and carries its source name.

Play 3: Make chunks searchable

Goal: let the computer find chunks by meaning, not just by exact words.

An embedding turns a piece of text into a list of numbers that captures its meaning. Chunks with similar meaning get similar numbers. A vector store is a place that keeps those numbers and finds the closest ones fast. If this is new, read our embeddings playbook first.

1. Pick an embedding model from your AI provider or an open one.
2. Turn every chunk into numbers.
3. Save the numbers, the chunk text and the source in a vector store.

Now when someone asks “Can I roll over unused days?”, the search finds the carry-over chunk even though the policy never uses the word “roll over”.

You are done when: a test search for one of your 10 questions returns the right chunk at the top.

Play 4: Retrieve, then ask the AI

Goal: give the AI the right pages and a clear rule to use only them.

1. Take the user’s question and turn it into numbers the same way.
2. Fetch the top 3 to 5 closest chunks.
3. Build a prompt: the rules, then the chunks, then the question.
4. Send it to the model and show the answer with the source names.

Here is the idea as a tiny illustration. Your tool’s names will differ.

question = "How many sick days do I get?"
chunks = vector_store.search(question, top=4)
prompt = (
  "Answer only from the context below. "
  "If the answer is not there, say you do not know.\n\n"
  "Context:\n" + join(chunks) + "\n\nQuestion: " + question
)
answer = model.ask(prompt)
show(answer, sources=chunks)

In plain steps: search, paste the best pages into the prompt, tell the model to stay inside them, and show where the answer came from. The rule “say you do not know” matters most. It stops confident guessing.

Meera’s chatbot now says “8 sick days” and shows the policy section. People can check it themselves.

You are done when: an answer cites its source, and a question outside your documents gets “I do not know”.

Play 5: Test and fix the weak spots

Goal: know when it fails and why.

1. Run your 10 questions and note the right answer for each.
2. For every wrong answer, check the chunks first. Was the right chunk found?
3. If not found, fix the chunking or the search. If found but ignored, fix the prompt.
4. Run the whole list again after each change.

Meera finds that “maternity leave” fails because the answer sits in two chunks. She makes the chunks larger for that section. For a deeper way to test, see our AI evals playbook.

You are done when: at least 9 of your 10 questions get a correct, sourced answer.

Do you need RAG?

Now that you know what is RAG, check whether your problem really needs it.

Not sure whether to use RAG or fine-tuning? Read our RAG vs fine-tuning playbook.

Mistakes to avoid

• Starting with every file you own. Start with one topic and grow.
• Chunks that are too tiny or too huge. Tiny chunks lose meaning, huge ones bury the answer. Test a few sizes.
• No “I do not know” rule. Without it the model fills gaps with guesses. Add the rule and test it.
• Old documents left in the store. The AI will quote outdated pages. Re-index when files change.
• Trusting every document blindly. A document can contain hidden instructions. See our prompt injection playbook.

How to know it is working

• The right chunk appears in the top results for most test questions.
• Every answer shows a source a person can open.
• Questions outside your documents get an honest “I do not know”.
• The score on your 10 questions stays steady after each change.

Your checklist

• ☐ One topic and clean, current documents
• ☐ Ten real test questions with correct answers
• ☐ Chunks with source and heading attached
• ☐ Embeddings stored and searchable
• ☐ Prompt that says “answer only from the context”
• ☐ Sources shown with every answer
• ☐ Test list rerun after every change

Frequently asked questions

What is RAG in one sentence?

RAG means searching your documents first and then letting an AI write an answer from what the search found, like an open-book exam.

Does RAG stop AI from making things up?

It reduces made-up answers, but it does not remove them. If the search finds the wrong page, the AI can still answer wrongly. That is why you test and show sources.

Do I need a vector database?

For a small set of documents, a simple in-memory store is enough to learn. A dedicated vector database helps when you have many documents or many users.

Is RAG the same as fine-tuning?

No. Fine-tuning changes the model itself. RAG leaves the model alone and gives it pages to read at question time, so updating knowledge is as easy as updating a file.

Can I build RAG in Java?

Yes. Our hands-on guide shows a RAG app with Spring AI.

Back to Meera: your next step

A month later, Meera’s chatbot answers most leave questions with a link to the exact policy section. Her manager stops asking about “putting AI on it” and asks which document to add next. She did not train a model. She taught it where to look. That is what is RAG at its simplest: an AI that checks the book before it answers.

Your next action: pick one small folder and your 10 questions today. Then try Play 1 to Play 4 in a weekend.

Sources: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020), plus beginner explainers such as the DEV Community RAG beginner’s guide.