How I AI #1: The 3B Model That Lied to Me By Refusing to Answer

Most of my "AI bugs" turned out to be systems-design decisions in disguise. Here's the first one.

I spent 10+ years building backend systems. A few months ago I started teaching myself AI engineering, and not the part where you train models. The part where you build systems around them: retrieval, context, agent loops, evals. This series is just me writing down what broke and what the numbers said. No “10 prompts that will change your life.” Just the stuff I actually hit.

Here’s the first one.

The setup: teaching a RAG app to say “I don’t know”

I was building a “chat with your docs” app. The classic RAG pipeline. You embed your documents, store the vectors in Postgres (pgvector), and at query time you retrieve the most relevant chunks and hand them to the model as context. Local and free the whole way: Ollama for inference, nomic-embed-text for embeddings.

The single most important feature of a RAG app isn’t answering questions. It’s refusing to answer when the answer isn’t in your documents. A RAG bot that confidently makes things up is worse than no bot at all. So the system prompt was strict:

SYSTEM_PROMPT = """You are a careful assistant that answers ONLY from the
provided numbered sources. Rules:
- Use ONLY facts found in the sources below. Do not use outside knowledge.
- Cite the sources you used inline like [1], [2].
- If the sources do not contain the answer, reply EXACTLY:
  "I don't know based on the provided documents."
- Be concise."""

To test it, I wrote a diagnostic: 5 questions whose answers are in the docs (should be answered + cited) and 5 that aren’t (should be refused). The scoreboard I wanted was 5/5 answered, 5/5 refused.

I got 4/5 answered, 5/5 refused.

The refusals were perfect. But one question that was clearly in my documents got refused too. The model looked at a chunk containing the exact answer and said: “I don’t know based on the provided documents”.

That’s not a hallucination. It’s the opposite. It’s a false refusal. And in a system where “refuse when unsure” is the safety rule, a model that refuses when it shouldn’t is quietly breaking the product. It’s lying about what it knows.

The debugging: was it retrieval, the prompt, or the model?

My systems-engineering instinct kicked in: isolate the layer. A RAG answer has three failure points: retrieval, the prompt, or the model, and you debug them in order.

Was it retrieval? Was the right chunk even reaching the model? I logged the similarity scores. The question was “What are the three risk categories in a permission matrix?” and the top chunk literally read: “The permission matrix classifies every tool as read_only, financial, or destructive”. Cosine similarity 0.62. Comfortably retrieved. Retrieval was fine.

Was it the prompt? Maybe “reply EXACTLY this string when unsure” was making it trigger-happy. Plausible! but the same prompt handled the other 4 in-doc questions correctly. If the prompt were the problem, it’d fail more consistently.

That left the model. So I ran the exact same context and prompt through two different models and diffed the output:

q = 'What are the three risk categories in a permission matrix?'
chunks, sim = retrieve(q)
user = f'Sources:\n{format_sources(chunks)}\n\nQuestion: {q}'

for m in ['llama3.2:latest', 'llama3.1:latest']:
    r = LLMClient(model=m).chat([{'role': 'user', 'content': user}],
                                system=SYSTEM_PROMPT)
    print(f'--- {m} (similarity {sim:.2f}) ---')
    print(r.text.strip())

Output:

--- llama3.2:latest (similarity 0.62) ---
I don't know based on the provided documents.

--- llama3.1:latest (similarity 0.62) ---
According to [1], the permission matrix classifies every tool as
**read_only**, **financial**, or **destructive**.

Same context. Same prompt. Same retrieved chunk with the answer sitting right there. llama3.2 (3 billion parameters) refused. llama3.1 (8 billion) answered correctly and cited its source.

The bug wasn’t in my code. It was in the model. The 3B model wasn’t capable enough to follow “answer from the source if it’s there, otherwise refuse”. It over-indexed on the “refuse” half of the instruction. Grounded refusal is a genuinely hard instruction-following task, and 3B just wasn’t up to it.

The fix: model choice is an engineering decision

Here’s the part that reframed how I think about this whole field. I was treating “which model” like a default. Grab the small fast one, move on. But model selection is an architectural decision, the same as picking a database or a queue. Different jobs want different models.

The catch: llama3.1 (8B) is correct but slow, ~28 seconds per call on my Mac (Ollama runs on CPU in Docker, no GPU). llama3.2 (3B) is ~4 seconds but not reliable enough for grounded RAG. And my agents which make many calls in a loop needed something fast that still follows instructions well.

So instead of one global model, I split by job, all driven from config:

# .env
OLLAMA_CHAT_MODEL=llama3.2:latest    # fast default
OLLAMA_RAG_MODEL=llama3.1:latest     # RAG generation — accuracy over speed
OLLAMA_AGENT_MODEL=qwen2.5:3b        # agent loops — fast AND follows format

The RAG pipeline pins itself to the stronger model where correctness matters:

client = LLMClient(model=settings.ollama_rag_model)

Re-ran the diagnostic. 5/5 answered, 5/5 refused. Fixed! not by touching a line of retrieval or prompt logic, but by putting the right model on the right job.

A bonus bug: the embedding model needed a magic word

While I was in there, I found a second, subtler issue. My similarity scores were all bunched up, relevant and irrelevant questions were scoring too close together to separate cleanly. At one point “what is the boiling point of water in Fahrenheit” (not in my docs) scored 0.61, while a real in-doc question scored 0.585. The off-topic question scored higher than the real one.

Turns out nomic-embed-text is an asymmetric model: it expects you to prefix queries and documents differently.

def embed_query(text):
    return embed(f"search_query: {text}")      # queries get this prefix

def embed_documents(texts):
    return embed([f"search_document: {t}" for t in texts])  # docs get this one

Adding the prefixes widened the gap between relevant and irrelevant enough to make the whole thing work. (The threshold-overlap problem deserves its own post. A single similarity cutoff genuinely cannot separate these cases, and the fix is a two-stage gate. More on that later.)

What I’d do differently

Being honest about the ceiling here:

1. This runs on local Ollama, free, private, great for learning. But on real hardware or a hosted model, the accuracy/speed tradeoff that forced the model-split largely disappears. The lesson (match the model to the job) stays; the specific split is an artifact of running 8B models on a laptop CPU.

2. I’d add a re-ranker. A cross-encoder between retrieval and generation would catch “plausible but wrong” chunks that similarity alone lets through.

3. I’d use the model’s native token counting and citations rather than approximating.

The takeaway

The thing that stuck with me: most of my “AI bugs” weren’t AI bugs at all. They were systems-design decisions wearing a costume. “Which model” is a capacity-planning question. “When does it refuse” is a validation-gate question. “Why do the scores overlap” is a data-representation question. The probabilistic component in the middle is new; the engineering around it is the same discipline I’ve done for a decade.

That’s the whole thesis of this series, and it’s why I think experienced engineers have a real edge in AI engineering, not despite skipping the ML-training rabbit hole, but because they can treat the model as a component and put good systems around it.

Next up: remember that off-topic question about boiling water that scored higher than a real one? That’s not a fluke, it’s why you can’t rely on a similarity score alone to decide when your RAG bot should answer. Next post: why that approach fails, and the two-layer fix that actually works.

Building this stuff and writing down what breaks. Follow along.