How I AI #2: Why "Only Answer If You're Confident" Doesn't Stop Hallucinations
The obvious fix for a RAG bot that makes things up is a confidence threshold. I tried it. Here's why it fails and the two-layer approach that actually works.
Last post, my RAG app had a false-refusal problem. It was a weak model refusing questions it had the answer to. Fixing that got me to 5/5 answered, 5/5 refused. But along the way I hit a stranger problem, and it’s the one worth its own post, because the “obvious” solution is a trap almost everyone walks into first.
Quick recap of the setup: a RAG app has to refuse when the answer isn’t in your documents, otherwise it hallucinates confidently, which is the worst failure mode there is. So the question becomes: how does the code decide “this question isn’t covered by my docs”?
The obvious answer (that everyone tries first)
Retrieval already gives you a number. When you embed the question and search your vector store, each retrieved chunk comes back with a similarity score, which means how close the chunk is to the question in meaning, from 0 to 1. High score means relevant match. Low score means nothing good found.
So the obvious rule writes itself:
chunks, top_score = retrieve(question)
if top_score < THRESHOLD: # nothing relevant enough
return "I don't know based on the provided documents."
else:
return answer_from(chunks) # confident enough — answerLet’s pick a threshold, say 0.55 and that becomes our confidence gate. Below the line, refuse. Above it, answer. Clean. Deterministic. It’s the first thing I built.
It does not work. And the reason it doesn’t work is genuinely interesting.
The receipt: an off-topic question that scored higher than a real one
I logged the actual top similarity score for every question in my test set. Here’s the real data.
Questions that were in my docs (I want these answered):
0.789 What are the four strategies of context engineering?
0.753 Why do multi-step agent loops need verification gates?
0.674 What is context rot?
0.622 What are the three risk categories in a permission matrix?
0.585 What does the equation Agent = Model + Harness mean?Questions that were not in my docs (I want these refused):
0.612 What is the boiling point of water in Fahrenheit?
0.551 How do I make sourdough bread?
0.523 What year was the Eiffel Tower built?
0.501 Who won the 2018 FIFA World Cup?
0.488 What is the capital of Australia?Look at the overlap. My lowest real question “Agent = Model + Harness” scored 0.585. My highest off-topic question, the boiling point of water, which appears nowhere in my documents scored 0.612.
The irrelevant question scored higher than the relevant one.
There is no threshold that separates these two lists. Set the line at 0.60 and you refuse a real question while answering about boiling water. Set it at 0.58 and you answer more off-topic questions. The distributions overlap, and no single cutoff can split two lists that overlap. The knife-edge doesn’t exist.
Why this happens (the part worth understanding)
This surprised me, so I dug in, and the explanation is a useful mental model.
An embedding score measures semantic similarity, not factual containment. Those are different things. “What is the boiling point of water in Fahrenheit?” is a short, clean, technical-sounding factual question, and my documents are full of short, clean, technical-sounding factual sentences (they’re notes about AI systems). The shape of the question matches the shape of my content, even though the topic is completely different. The embedding model rewards that resemblance with a decent score.
Meanwhile, “What does the equation Agent = Model + Harness mean?” is phrased oddly, has an equation in it, and doesn’t look like a typical sentence. So even though the answer is sitting right there in a chunk, its similarity score comes out lower.
Similarity is a measure of “does this look related” and not “is the answer actually in here.” Confusing the two is the core mistake, and a single threshold bakes that confusion in. This is the same trap as judging a search result by how many keywords it shares with your query. Surface resemblance isn’t relevance.
The fix: stop asking one number to do two jobs
Once I saw it that way, the fix was obvious. I was asking the similarity score to answer a question it fundamentally can’t: “is the answer to this question contained in these chunks?” An embedding score can’t know that. But something else can, the model reading the chunks.
So I split the decision into two stages, each doing the job it’s actually good at:
RELEVANCE_THRESHOLD = 0.45 # deliberately LOW — a coarse garbage filter
def answer(question):
chunks, top_score = retrieve(question)
# Stage 1: cheap coarse filter. Only catches total garbage —
# a question so unrelated that retrieval returned nothing usable.
if not chunks or top_score < RELEVANCE_THRESHOLD:
return refuse()
# Stage 2: the real gate. Hand the chunks to the model with a
# strict instruction: answer ONLY if the answer is actually here,
# otherwise say you don't know.
return generate_grounded_answer(question, chunks)The two stages have completely different jobs:
Stage 1 (the threshold) is now a coarse filter, not a precision instrument. I dropped it from 0.55 to 0.45. Low enough that it never rejects a real question. Its only job is to catch the rare case where retrieval comes back with nothing remotely usable, so I don’t waste a model call. It’s not trying to make the fine-grained “is this relevant” call anymore, because it can’t.
Stage 2 (the model) is the real gate. The model can read the chunks and reason about whether they actually contain the answer. It is the exact judgment the similarity score can’t make. The system prompt from post #1 does the work: “Use ONLY facts found in the sources. If the sources do not contain the answer, reply “I don’t know based on the provided documents.”
So when the boiling-water question sneaks past the coarse filter (it scores 0.612, above 0.45), it hits the model, which looks at chunks about context engineering and harness design, sees no boiling point anywhere, and correctly refuses. The score got it in the door; the model threw it back out.
Result: 5/5 answered, 5/5 refused. Every off-topic question refused, every real question answered and not because I found the magic threshold, but because I stopped asking the threshold to be magic.
The takeaway
The lesson that stuck: use each tool for the job it’s actually good at, and don’t force one signal to make a decision it can’t make.
A similarity score is great at “rank these chunks by how related they look” and bad at “is the answer literally in here.” A language model is the reverse. It is expensive to call, but it can actually read. The bug wasn’t a wrong threshold value; it was using the wrong tool for the decision.
That’s a pattern I now see everywhere in these systems: the failure isn’t a bad parameter, it’s a good parameter being asked to do a job it was never capable of.
Next up: I take this same RAG agent and run it for 50 turns of conversation and watch one part of its “memory” quietly grow until it’s eating 60% of every request. It’s the moment “context engineering” stopped being a buzzword for me.
Building this stuff and writing down what breaks. Follow along.