When Similarity Is Not Relevance

Relevance was never a search problem. It was a context problem wearing a search problem's clothes.

A customer emails support: "We need to cancel our Enterprise plan, we were acquired, and the new parent company has its own vendor contracts."

The agent embeds the query and pulls its top matches from the vector store.

The top result is "How to cancel your subscription." It describes the self-serve flow: click a button, done. But Enterprise accounts can't cancel that way. They have to go through their CSM (Customer Success Manager) and give 30 days' written notice.

Right below it: a blog post on "Switching Plans After an Acquisition," a churn-prevention FAQ offering a discount to "stay," and the enterprise onboarding doc, which just happens to repeat the word "contract" six times.

Every one of these is semantically close to the query. None of them is the actual answer.

The real answer, that Enterprise cancellations require written notice to the account's CSM under Section 4.2 of the MSA, lives in a contracts doc that never even uses the word "cancel." It doesn't even make to the top TEN.

How Similarity Diverges From Relevance

Embeddings capture meaning at the linguistic level. "Cancel subscription" and "end membership" sit close together in a vector space. So does almost every document that mentions cancellation. That's exactly why the agent surfaced four cancellation-adjacent chunks and missed the one that mattered. The self-serve article, the acquisition blog post, the retention FAQ, the onboarding doc, all of them were semantically close to the query. The MSA clause wasn’t, because it never uses the word “cancel” anywhere in the document, it talks about notice periods and CSMs. Similarity had no way to connect it to the question being asked.

That's the gap, relevance depends on things embeddings don't encode. Here, it depended on knowing the customer was on Enterprise plan, not a self-serve one. A system that had that context: plan type, account tier, what "cancellation" even means for this specific customer: could have ranked the MSA clause first and the self-serve article not at all. distance of vectors alone can't make that call. It only sees that two pieces of text use similar words, not which one actually applies to this account.

The problem gets worse as the knowledge base grows. With a handful of documents, the closest match is often the right one, there's nothing else nearby to compete with it. Add a few thousand more docs, and you get more wrong answers.

Read more about why agent memory is really a relevance problem

Why Top-K Makes This Worse

The obvious fix could be is to just retrieve more chunks, if k=10 didn't surface the MSA clause, try k=20. Unfortunately it doesn't work. Increasing k often makes accuracy worse. It's also expensive, stuffing twenty chunks into the context window for what should be a simple lookup burns tokens and latency on a query that didn't need any of it.

Every retrieved chunk competes for the model’s attention. In our cancellation example, most of the top results are already wrong but look relevant: the self-serve guide, the acquisition blog, the retention FAQ, and the onboarding doc. Increasing k does not necessarily help. It often just adds more similar-looking noise. So even if the correct MSA clause is somewhere in the context, the model may still miss it because several other chunks use the customer’s exact words and look more relevant.

The Cost in Production

The agent still does not have the MSA clause. It only has a few chunks that look related to cancellation, so it gives the wrong answer. The customer asks again. The agent searches again, retrieves more chunks, and makes another LLM call. Maybe it rephrases the query, reranks the results, or loops through retrieval one more time.

What should have been one retrieval + one model call can turn into several retrieval steps and multiple API calls. Across thousands of support requests, that means more tokens, more inference, higher latency, and much higher compute cost, all because the right context was not retrieved in the first place.

And adding more chunks does not necessarily fix it. More chunks mean more context for the model to process, while repeated retries mean even more computation. So poor retrieval is not only an accuracy problem. It creates a compute problem too:

Moving From Similarity to Relevance

Bridging the gap requires retrieval systems that incorporate context beyond the query embedding, the things a human support agent would just know.

Take the MSA clause. Vector search missed it because nothing about "Section 4.2" sounds like "cancel." But the clause isn't just a floating chunk of text, it's connected to other things: it belongs to the Enterprise plan entity, it's a type of cancellation policy, and it's linked to the CSM role, not the self-serve flow.

If the retrieval system knew those relationships: plan → policy → clause, self-serve ≠ enterprise
it wouldn't need the word "cancel" to appear anywhere near "Section 4.2." It could follow the connection, this customer is on an Enterprise plan, Enterprise plans route through a CSM, here's the clause that governs that path.

That's not a similarity match. That's a relationship the system should navigate.

The same goes for context outside the document itself. Knowing the customer's plan tier, that this is their first message on the topic, and that "we were acquired" signals urgency, none of that lives in an embedding, but all of it changes which chunk is actually useful. This means combining vector search with structured filtering, relationship graphs, and reranking, instead of relying on similarity alone.

Conclusions

The customer’s question was not difficult. It was just specific. The answer existed in one document which doesn’t have “cancel” word and hence vector search based similarity never found it because it is designed to find similar vectors. Sounding close and being right turned out to be different things. A model that knows this customer is on an Enterprise plan, that Enterprise plans route through a CSM, and that the MSA clause is connected to that policy, not just similar in wording to "cancel", can find the right answer instead of the closest-sounding one. Relevance was never a search problem. It was a context problem wearing a search problem's clothes.