Why RAG is inventing answers from company documents and how to find out why

A RAG system can make mistakes at several stages: when a document enters the index, when the search selects fragments, when these fragments form a context for the model, and when the model formulates an answer. An equally incorrect response to the user can have four different causes. Therefore, it is necessary to diagnose each stage separately. There is no proven benefit to changing a model, vector base, or prompt without understanding the reason: you change one part and the bug lives in another.

RAG (Retrieval Augmented Generation) stands for generation with search: the system first finds relevant fragments in your documents and then passes them to the model along with the question.

After the article, you will be able to diagnose your own system using a table of “question, verified answer, retrieved passages, response, and cause,” measure the effect of a fix on the same sample, and set up feedback so the same errors do not return.

First identify the type of error

The "bot makes up" complaint describes a symptom, not a cause. Before you change anything, analyze some real wrong answers and assign each one to one of the types.

The fact is not in the documents. There is simply no answer to the question in the knowledge base, but the model formulated it. Here, the problem is not in the search, but in the fact that the system does not know how to honestly say "this is not in the documents".

Outdated document. The fact is there, but in the old version of the regulation, and the new version either did not get into the index, or exists next to the old one.

Bad search. The required fragment exists in the index, but the search did not return it, or returned it among a large number of irrelevant fragments.

Incorrect conclusion. The model received the correct fragment, but did not answer according to it: mixed it with another, missed the condition, repeated general knowledge instead of the text of the document.

This division is consistent with OpenAI's methodology for model accuracy. She distinguishes between two sets of RAG problems: the search returned the wrong context or too much irrelevant stuff that drowns out the right information, and the model got the right context but used it wrong. For the first group, the technique advises to adjust the search and reduce noise, for the second, work with the model's instructions.

Case Study: Return Policy

The example is made up. The company has changed the product return policy: previously the term was 14 days, now it is 30. A week after the update, the support bot responds to customers with "14 days". The reason is not determined by the same answer. Consider three different failures, each of which gives exactly this answer.

Failure 1. An old file got into the index. The new version of the regulation was placed in a different folder, which is not covered by indexing. There is only a document with 14 days in the index. The model answers correctly according to what she received. Fix: source update process. The new document must replace the old one in the index, and the old one must be deleted or marked as invalid.

Failure 2. The search returned both undated versions. There is both an old and a new document in the index. The search returns both fragments, and the text of the fragments has no date or version number. The model sees two conflicting numbers and chooses one. Fix: metadata. Each fragment should carry an effective date and document status, and the search should filter out invalid versions before the fragments enter the model.

Failure 3. The model received a new version but repeated the old rule. The search worked correctly, only the new document with 30 days in the context. But the model answered "14 days", for example because it was answering from a general idea of ​​typical return conditions, not from the text. Fix: generation. Request a response with a quote from the given fragment, check that the number in the response matches the number in the fragment, and test this scenario separately.

Three crashes, three different fixes. If the team in all three cases simply replaced the model, the problem in the first and second would remain.

Check the source document

Diagnostics begins not with the model, but with the document on which the answer was to be based.

Version and date. Does a document with the correct answer exist and is it current? Aren't there several versions in the sources without marking which one is valid?

Tables. Conditions are often recorded in tables: tariffs, terms, limits. When converting a document to text, the table may lose its structure, and the values ​​will be detached from their headings. Open the text that actually got into the index and check if you can understand from it which number belongs to which.

Scans. If the document is an image, the text from it is obtained by recognition. Recognition errors are converted into answers. Check the recognized text of exactly those pages that relate to the problem question.

Conflicts. Sometimes two current documents really contradict each other: department regulations and general company policy. The source owner must resolve the conflict; before that, the system should mark the conflict and pass the question to the person, and not arbitrarily choose a rule.

Access rights. Does the system have access to the required document? Check if the current user has rights to the document. If there is no right, the correct behavior is not to open the document; such a case is not corrected by expanding access just for the sake of completeness of the answer.

If at this stage it turned out that the required fact is not in the index or it is distorted, you can not go further: no search or model settings will fix this.

Check search and build context

If the document is correct and is in the index, the next question is whether the search found it and whether the found fragment reached the model in a suitable form.

Search

For a problem question, look at what fragments the search returned, in what order, and with what similarity scores. The answer to three questions:

1. Is the desired fragment among the results? If not, the problem is in the search: in how the document is divided into fragments, in the formulation of the query, in the search settings.
2. In what position is it? If the desired fragment is there, but in tenth place after nine irrelevant ones, it may not enter the context due to the limitation of the number of fragments.
3. Isn't it drowned out by noise? The OpenAI methodology specifically calls out the situation when too much irrelevant context drowns out the correct information and causes contrived answers.

Building the context

Between the search and the model, there is another step that is often missed in diagnostics: the system assembles the query text from fragments. Here, too, you can lose the right answer.

• Cutting. Ten fragments were found, and five were matched. Was it necessary among them?
• Metadata loss. The document date and status were in the index, but not in the text seen by the model.
• Position in a long context. A study by Lost in the Middle (2023) on two tasks, searching for an answer in multiple documents and searching for a meaning by key, showed a U-shaped relationship. Quality was highest when the required information was at the beginning or end of the context, and dropped markedly when it was in the middle. The authors note that the decline persisted even for models specifically built for the long context. This is a result for the models and problems investigated at that time, and not a characteristic of all modern models. But this is an argument to check on your system whether the fragment position affects the response.

A practical way to check: save the full text that the system sent to the model for the problem question and read it with your eyes. Is it possible to give the correct answer from it, without any other knowledge? If the required fact is not there, the system does not have a sufficient source basis for the answer, even if the model guesses it by chance.

Check the generation

If the correct fragment reached the model in a suitable form, and the answer is incorrect, the problem is in the generation.

The test method is simple and time-consuming: match each significant statement of the answer with the text of the provided fragments. Numbers, terms, conditions, exceptions, names of documents are considered significant. For each, choose one of the three:

• confirmed: is in the fragment and transmitted correctly;
• distorted: is in the fragment, but passed with an error (different number, missing condition);
• unconfirmed: this is not in the snippets, the model added it herself.

Distorted and unconfirmed statements indicate generation problems. Typical Fixes: Clearer instructions to answer only given text and say when there is no answer; the requirement to cite a fragment; checking the format of the response with a code; separate tests for numbers and terms. OpenAI's methodology for this type of problem advises to work with instructions first, and to consider retraining if examples improve accuracy.

The instruction "answer only according to the documents" is useful, but does not guarantee the result. Its effect should be measured in the same way as any other correction.

Diagnostic table

Below is a table for a tutorial with a return policy. It can be used as a template for your own mistakes.

The Found Fragments column is populated from the system log, not from memory. Without saving what the search returned and what got into the query, it is impossible to distinguish the second failure from the third.

Measure the correction

A fix that "seems to help" three questions may worsen the answers to thirty others. Therefore, changes are measured on the same control sample before and after.

Control sample. A set of questions with reference answers that have been checked by people who know the subject. The OpenAI methodology advises building a set of questions with reference answers even before optimization and mentions a guideline of 20 questions for initial work with instructions. Include in the sample questions of all four types of errors, including questions that are not answered in the documents.

What counts. Not just the proportion of correct answers. Separately count correct refusals (the system honestly said that there was no answer), false refusals (there was an answer, but the system refused) and wrong answers. A fix that reduces guesswork at the cost of the bot rejecting half as many questions might be worse than the original state.

Cost. More fragments in context, larger model, additional response validation increases cost and latency. Write them next to the quality.

Automatic evaluation. Tools like Ragas offer to evaluate three aspects separately: whether the search finds relevant and focused fragments, whether the model uses them reliably, and the quality of the response itself. The authors describe these metrics as not requiring reference annotations from humans. This is convenient for frequent runs. In the Ragas study, assessment was implemented through inquiries to the LLM; it is not an independent proof of truth. In addition, faithfulness means conformity to the given context: if the source itself is outdated, a high consistency score does not make the answer correct. Check automated scores against reference answers and against manual spot-checking by humans, especially for questions where mistakes are costly: money, deadlines, legal terms.

Separately about expectations. The OpenAI methodology notes that there is no universal threshold of "accurate enough" and proposes to determine the target accuracy from business economics: the value of correct answers and the cost of each type of error. The error percentage of your system is not output from the external benchmark: it is only shown by your sample.

Organize feedback

Diagnosing a single error is useful. A process where every mistake becomes a test is more useful. This requires feedback where enough data is stored for analysis.

What to store for each error:

• user question and system answer;
• the fragments returned by the search and the text found in the query to the model;
• versions of documents in the index at the time of response;
• the correct answer, confirmed by a responsible person;
• the specified reason: document, search, context compilation or generation;
• correction made and date.

Storage of diagnostic samples needs to be minimized: mask personal data and secrets, limit access and storage period. For regression tests, use impersonal or synthetic copies whenever possible.

Who is responsible. The document owner corrects the error in the document, not the team supporting the bot. A search or generation error is corrected by the technical team. Distribute these roles in advance, otherwise every complaint will be treated as "bot broke".

Turn each bug into a test. Add every analyzed case with a verified answer to the regression set. Keep a separate holdout sample that you do not use for tuning; repeatedly optimizing against the same questions can produce an overly optimistic estimate. After each system change (a new model, new search settings, or updated instructions), run the full test set. This helps catch regressions and confirms that the fix still works.

What to do in the near future:

1. Collect 10 to 20 real wrong answers and assign each one to one of the four error types.
2. For each, open the text of the document in the index, the search results, and the full text of the query to the model.
3. Fill in the diagnostic table: question, standard, found fragments, answer, reason.
4. Form a control sample with reference responses, including unanswered questions in the documents.
5. Measure the current state, make one correction, and measure again on the same sample, taking into account failures and cost.
6. Configure data retention to parse each new error and the order in which it enters the control sample.

Frequently asked questions