How to Improve RAG Accuracy: A Beginner’s Playbook on Chunking, Hybrid Search and Reranking

Meera’s help-centre bot answers questions from her company’s documents. It worked fine in the demo. Then a customer typed an error code, ERR-4012, and the bot replied with a long, friendly answer about password resets. Wrong page, confident tone.

Meera did not need a bigger model. She needed to improve RAG accuracy in the part that finds the right pages. In this playbook you will follow her through four fixes, in the order that gives the most gain for the least work.

Last verified: 10 November 2026. AI tools change fast; check the linked sources for anything newer.

Who this is for: beginners who have a first RAG bot and see wrong or vague answers. Time needed: about 30-45 minutes to read, a few hours to try on your own data.

What does “better RAG answers” mean, in simple words

RAG (retrieval-augmented generation) means the bot first looks up pages from your documents, then asks the model to answer using those pages. If you have not read it yet, start with our What Is RAG playbook.

Think of an open-book exam. The student is clever, but if the wrong chapter is open on the desk, the answer will be wrong. Most bad RAG answers are a wrong chapter on the desk problem, not a clever-student problem.

So the precise version is this: answer quality depends mostly on retrieval, the step that picks which pieces of text reach the model. The plays below improve that step.

The playbook at a glance

1. Build a small test list so you can see if a change helps.
2. Fix your chunks (how documents are cut into pieces).
3. Add hybrid search (meaning plus exact words).
4. Add reranking (a second, careful look at the top results).
5. Measure, keep what helps and drop what does not.

Play 1: Build a small test list first

Goal: know whether each change really helps, instead of guessing.

1. Write 15-20 real questions users ask.
2. Next to each, write which document page holds the answer.
3. Run them all and count how often the right page appears in the top 5 results.

Meera picks 18 questions, including the ERR-4012 one. Her bot finds the right page for only 11 of them (illustrative numbers). That is her starting score. Our AI evals playbook goes deeper on test lists.

You are done when: you have a saved list of questions and a starting score.

Play 2: Fix your chunks

Goal: make each piece of text one complete idea.

A chunk is one piece of a document that gets stored and searched. Imagine cutting a recipe book with scissors every 500 letters. Some pieces would start in the middle of a sentence and lose the dish name. Search works the same way: a chunk without context is hard to match.

1. Cut along headings and paragraphs, not at a fixed character count.
2. Start with roughly 300-500 tokens per chunk. Practitioner guides suggest this as a starting point, so tune it on your own test list.
3. Add the page title and section heading to the top of every chunk.
4. Re-run your test list.

Meera finds her old chunks cut error tables in half. After cutting by heading, her score moves from 11 to 14 out of 18 (illustrative).

You are done when: no chunk starts mid-sentence and every chunk shows its page and heading.

Play 3: Add hybrid search

Goal: find pages by meaning and by exact words.

Vector search (see our embeddings playbook) finds text with a similar meaning. It is like asking a librarian who understands topics. But it can miss exact things such as ERR-4012 or a product code.

Keyword search (a common method is called BM25) is like the index at the back of a book. It is great at exact words and poor at meaning. Hybrid search runs both and merges the two result lists, because they fail on different questions.

1. Turn on keyword search in your database or search tool. Many vector databases and search engines offer it.
2. Merge the two lists. A common starting mix in guides is about 70% meaning and 30% keywords. Treat it as a starting point.
3. Re-run your test list and try a few other mixes.

Meera turns on keyword search and the ERR-4012 page jumps to the top. Her score becomes 16 out of 18 (illustrative).

You are done when: exact codes and names find their page, and meaning-based questions still work.

Play 4: Add reranking

Goal: put the best page first, not just somewhere in the top ten.

Searching is fast but rough, like a quick skim of many CVs. A reranker is a second model that reads the question together with each candidate and scores how well they fit, like a careful interview of the shortlist.

1. Fetch a wide set, for example the top 30-50 candidates.
2. Send them to a reranker (hosted services and open-source models exist).
3. Keep only the best 3-5 for the model.
4. Check the extra delay. Guides report roughly 100-300 ms, but measure your own.

Here is the idea in a few lines. It is an illustration, since each tool names things differently.

question = "What does ERR-4012 mean?"
keyword_hits = keyword_search(question, top=25)
meaning_hits = vector_search(question, top=25)
candidates = merge(keyword_hits, meaning_hits)
best = reranker(question, candidates)[:4]
answer = model(question, context=best)

In plain steps: search two ways, merge, let the reranker pick the best four, then ask the model. Meera’s score reaches 17 out of 18 (illustrative), but answers take a little longer, so she keeps it only for the help-centre bot.

You are done when: the right page is usually first, and the delay is acceptable to your users.

Play 5: Measure, keep what helps

Goal: stop adding parts that do not pay for themselves.

1. Write down the score after each play.
2. Note the extra cost and delay of each step.
3. Remove any step that adds little.
4. Add every new failing question to the test list.

Meera keeps a tiny table in her notes: chunks +3, hybrid +2, reranking +1. The table tells her where to spend her next hour.

You are done when: every part of your pipeline has a number that justifies it.

Do you need all of this?

Mistakes to avoid

• Changing three things at once. Fix: change one, re-run the test list, then the next.
• Blaming the model first. Fix: print the retrieved pages. If the right page is missing, the model cannot help.
• Copying someone’s chunk size or mix as a rule. Fix: treat numbers in blogs as starting points and tune on your data.
• Skipping the test list. Fix: 15 questions is enough to start.

How to know it is working

• Your test score goes up after each change.
• Exact codes and names find the right page.
• Users stop repeating the same question in other words.
• Answer time stays within what users accept.

Your checklist

• ☐ 15-20 test questions with the right page noted
• ☐ Chunks cut by heading, with titles added
• ☐ Keyword and meaning search merged
• ☐ Reranker tried, delay measured
• ☐ Score written down after every change

Frequently asked questions

What should I fix first to improve RAG accuracy?

Build a test list, then fix chunking. It is cheap and often gives the biggest gain. Add hybrid search next, then reranking.

What is hybrid search in RAG?

It runs keyword search and meaning-based vector search together and merges the results. Keywords catch exact terms, vectors catch similar meaning.

Is reranking worth the extra delay?

Sometimes. If the right page is already first most of the time, skip it. If it is found but ranked low, reranking usually helps. Measure on your own test list.

What is the best chunk size?

There is no single best size. Start near 300-500 tokens, cut along headings, and tune using your test list.

Meera’s bot, one week later

A customer types ERR-4012 again. This time the bot shows the right error page, with the fix in two lines. Meera did not change the model. She changed what gets put on the desk.

Your next action: write your 15 test questions today. Next on the roadmap is keeping RAG safe and cheap, so also read our prompt injection playbook.

Sources: practitioner guides on RAG retrieval, including Mastering RAG: From Chunking to Hybrid Search and Building Production RAG Systems. Chunk sizes, mixes and delays are starting points from such guides, not benchmarks. Meera’s story and her scores are illustrative.