How to Teach AI to Say “I Don’t Know” When the Knowledge Base Has No Answer

The short answer: a reliable “I don’t know” does not come from one sentence in a prompt. It is controlled behavior across the system: the system checks whether it found a document passage that actually answers the question and, if not, clarifies the request, states the limitation honestly, or hands the question to a person. Asking the model to rate its own confidence is not a guarantee; that rating is not a calibrated probability of correctness.

The key requirement is to optimize two errors at the same time: a false answer and an unnecessary refusal. If you reduce only the first, the assistant turns into a permanent “I don’t know.” After reading this article, you will be able to describe when an answer is truly unavailable, configure four assistant responses, and build an evaluation set from your real requests.

When an answer is truly unavailable

“The knowledge base has no answer” can describe at least four different situations. The right response depends on which one occurred.

There is no document. The question concerns a topic the knowledge base does not cover. For example, a customer asks about delivery to a country where the company does not operate, and the documents say nothing about it.

There is not enough information. A document covers the topic but not the required detail. A returns policy gives general rules, while the question concerns a particular product or contract type that it does not mention. The model may “fill in” an answer from general rules, making this the most dangerous case.

There is a version conflict. Two documents give different answers: an old and a new policy, or a general policy and an exception for a particular region. Without a precedence rule, the model may choose one at random or combine them.

Access restriction. Apply permissions during retrieval and do not pass restricted passages to the model. A user-facing response must not confirm whether a particular document exists or reveal its contents. The assistant can give a neutral reply and explain how to contact the access owner.

Ideally, the system distinguishes these states technically, not just through model wording. For example, retrieval can return signals as well as passages: how many passages were found, whether documents from different versions were included, and how many results were filtered by permissions. Then the response decision does not depend only on the model’s judgment.

Checking evidence

The key idea: an answer is allowed only when the system finds a passage that addresses the specific question. Topical similarity is not enough.

Most RAG systems use similarity search: they find passages that are close in meaning to the request. But closeness is not an answer. For the question “Can I return a product after 30 days if it has been used?”, search might easily find a passage about returns within 14 days. The topic is the same, but that passage does not answer the specific question.

It is therefore useful to separate two checks:

1. Was relevant context found? Are any of the retrieved passages about the question? This evaluates retrieval quality.
2. Is the answer grounded in the context? Is every claim supported by retrieved passages rather than the model’s general knowledge? This evaluates faithfulness to the source.

Practical techniques include:

• Ask the model to find a quotation first. An instruction such as “provide the passage that directly answers the question; if there is none, say so” makes the absence of evidence explicit.
• Use a structured result. Return fields for the status (answer, clarification, limitation, handoff), text, and IDs of the passages used. Code can check that an “answer” status has at least one passage reference. But structure does not prove the model selected the right status.
• Add a separate grounding check. For critical topics, add a second step that checks whether the answer’s claims appear in the cited passages.

Check permissions before the model, not after it. If a passage the user cannot access enters the context, an instruction such as “do not reveal it” is not reliable protection: the model may paraphrase it indirectly. Apply permission filters during retrieval and give the model only allowed passages. The system may use a signal that results were filtered by permissions only in its internal response logic. The user-facing message should remain neutral and must not confirm whether a particular source exists or what it contains.

One more point about similarity thresholds: a vector-search score measures how close texts are in a particular representation, not the probability that a passage contains the correct answer. There is no universal threshold. It depends on the embedding model, language, document style, and chunking method. A threshold can be one signal, but it must be tuned on your data.

Four responses

When a complete answer is unavailable, the assistant has at least four useful options.

Answer. A passage directly answers the question and the user has permission to see it. The assistant responds and, ideally, identifies the supporting document.

Clarify. The question is ambiguous and the answer depends on a detail that can be requested. For example, rules differ for individuals and companies, but it is unclear which one the user represents. Clarification is often more useful than refusal: it turns “I don’t know” into “I can answer if you tell me this.”

State the limitation. A document exists but does not cover the case. The assistant honestly says what is and is not known: “The policy describes returns within 14 days for unused products. No terms for used products were found.”

Hand off to a person. There is no answer, clarification will not help, and the issue matters. The assistant forwards it with context to a responsible employee so the customer does not have to repeat everything.

Tie the response to the situations in the first section. No document: state the limitation or hand it off. Not enough detail: clarify or state the limitation. Conflicting versions: apply the precedence rule, or if none exists, show the conflict and hand it off. For access restrictions, give a neutral reply that does not confirm whether a source exists, then explain how to contact the responsible person.

How to phrase a useful refusal

A bad “I don’t know” ends the conversation. A useful one gives the person a next step.

A useful refusal has three elements:

1. What exactly was not found. Avoid a generic “I can’t help.” Be specific: “The available documents do not specify the terms for this case.”
2. What related information is known, if useful. State the general rule with a clear note that it may not apply to this case.
3. The next step. Which detail to clarify, whom the request will go to, and what happens next.

Illustrative response:

“The available policy does not specify this case. Please clarify the contract type: standard or corporate. If we cannot find the applicable rule, we will send the question to a specialist.”

Avoid:

• Invented confidence. Customers may treat “probably” or “usually” before an unsupported assumption as an answer.
• Revealing restricted information. A refusal due to access rights must not hint at the document’s content. “I cannot tell you what the termination order says…” already reveals information.
• Promises the system cannot keep. If handoff is not configured or has no response-time commitment, do not say “someone will contact you within an hour.”

Two groups of errors

Any configuration of “I don’t know” behavior shifts the balance between two errors.

False answer. The assistant answers without evidence or with incorrect evidence. The cost depends on the topic: an inaccuracy in a general product description is less harmful than an incorrect contract term or refund amount.

Unnecessary refusal. The assistant says “I don’t know” even though the answer is in the documents. This has a real cost too: the customer goes to an operator, workload does not decrease, and trust in the assistant falls.

Research on models’ ability to abstain shows that even powerful models struggle with it, while some prompting strategies (strict instructions and step-by-step reasoning) can improve performance. Prompts matter, but do not solve the problem by themselves. Measure results on your data.

A practical rule is to measure both errors separately and always together. If stricter rules reduce false answers, check how many unnecessary refusals they add. For topics where a false answer is costly (finance, legal terms, safety), you may deliberately accept more refusals and handoffs. The balance may be different for general reference questions.

Tuning on real requests

Set rules and thresholds using evaluation examples, not intuition. The design below is illustrative; the numbers are a proposed test structure, not results of a completed study.

Illustrative 100-question evaluation set

Record the correct response for each question and, where available, a reference answer. After running the set, count correct answers, false answers, unnecessary refusals, correct refusals, and restricted-information leaks separately. The last measure must be zero regardless of all others.

How to work with the set:

1. Collect questions from real requests rather than inventing convenient ones. The most valuable are those the assistant has already mishandled.
2. Have someone who understands the process label the examples and record why each response is correct.
3. Change one rule at a time and compare results on the same set.
4. Do not add evaluation examples to the prompt; otherwise, you are testing memorization.
5. Review the set when documents, products, or common customer questions change.

After launch, add observations from real use to the evaluation set. It is useful to log each response with its status, retrieved passages, and outcome: did the customer get help, contact an operator, or have an operator correct the answer? Once a week or on another regular schedule, the owner reviews examples from the log:

• Refusals after which an operator quickly found the answer in the documents. These are unnecessary refusals and often point to retrieval or chunking problems.
• Answers corrected by an operator. These are false answers; each is a candidate for the evaluation set.
• Frequent unanswered questions. These are not assistant errors but signals to the knowledge-base owner that a document or section is missing.

The last item often provides the greatest practical value. Honest “I don’t know” responses, counted over time, reveal which company knowledge has not been documented and help prioritize what to write down first.

Conclusion

A reliable “I don’t know” combines several parts: distinguishing why an answer is unavailable, checking evidence, offering four responses instead of one, and measuring both groups of errors. The model’s confidence rating and a similarity threshold are signals, not guarantees.

What you can do in your process:

1. Collect 100 real questions in four groups: answer available, no answer, document conflict, and permission restriction.
2. Record the correct response for each: answer, clarify, state the limitation, or hand off to a person.
3. Configure the assistant to return a response status and references to retrieved passages; have code check that an answer has at least one reference.
4. Define precedence rules for conflicting documents and a handoff path that includes context.
5. Run the set and count false answers, unnecessary refusals, and restricted-information leaks separately.
6. Change rules one at a time and rerun the set. Review the set as documents change.

If it matters that your assistant answers honestly from company documents, the BotLabs team can discuss the process with you in a 15-minute consultation. We will review the questions received most often, where false answers and unnecessary refusals occur, and which evaluation set to start with. The next step will be to decide what to check on your real requests.

Frequently asked questions