Embeddings: similar in what way?

Which of these four shapes is the odd one out? Pick one, and decide why.

Each answer is defensible in some way. A is the only red shape; B is the only square; C is the only outline; D is the only small shape. You just have to supply the criterion that the question left in the dark.

Now imagine these are parts on a production line. Which needs a different paint colour? A. Which needs a square mould? B. Once we know what we are doing, some differences become decisive and others become irrelevant.

The same issue appears when we ask a search engine to find “similar” documents. Similar in subject, wording, purpose, underlying cause, or the action we should take next?

When you ask for the nearest document, what determines the meaning of “near”?

How a model gives distance a meaning, and what does it mean

First, a quick reminder on the nature of embeddings.

An embedding represents an piece of data (text, image, …) as a list of numbers, or vector in vector space. This is where the terms “embeddings” come from: the content is embedded in the 3D space as say, the vector [11, 2, 9] (often, these spaces are much larger than three-dimensional : up to thousands of dimensions).

A search system then compares these vectors using a scoring rule (a metric). Most often, that rule is something like “vectors pointing in the same direction are more related” (cosine similarity).

So the trivial (and wrong) answer to our opening question is: the metric tells us what is near what.

A better (and true) answer is: the embedding model that determines which input maps to which vector — and implicitly, their relative distances.

An more important (and still true) answer is: someone else. The people that selected the dataset, model size, training parameters… All their choices have determined what the model gives you.

This is not particularly bad in and of itself, but this often means you are using very generalist models that are are optimized for everything and nothing at the same time. Such a model would tell us there is no outlier in our shape example: most shapes looks like most others. Not choosing may seems like the right choice, but remember: if your question is “which needs a different paint colour?” you want your search system to single one out.

This shape example make the missing decision visible. With text however, the word “semantic” is sometimes magically hand-waved to pretend there is an universally useful way to organise meaning — but the problem stands.

Suppose you ask: “How do I disable guest access in V4?” Your documentation contains an overview of guest access, a detailed procedure for V3, and a V4 installation reference. All three concern the right subject — none are right.

Embeddings in the age of agents

The emergence of highly agentic AI has added yet another drawback to embeddings: they lack feedback.

Agent nowadays iterate and explore over many turns. To do so properly, they require interpretable environements. For instance, an agent searching for earlier guest access procedure might go something like:

1. Notice there documentation folder root is made of 4 folders : v1/ to v4/.
2. It opens v4/ and see the folders “management”, “security”.
3. Since it’s looking for access privilege, the agent navigates to security.
4. it notices the security folder is mostly about MFA and endpoint protection, so it navigates back and goes it management/
5. There, it finds a document “guest_acess.docx”, Bingo !

Being able to navigate this way is crucial to agent performances, and simple embedding don’t allow for that. If your embedding search produces irrelevant results, what do you do? How do you change your search query? The agent cannot rebound on that negative result.

Are embeddings still good for anything, then?

Of course. Having a model supply a useful notion of similarity can save us an enormous amount of work. We just need to decide what job we are giving it.

At very large scales and in unstructured contexts, they make a first pass affordable. You cannot ask an agent to read a billion documents before deciding which ones deserve a closer look. Precomputed embeddings and a suitable search index can reduce that collection to a manageable set of candidates; libraries such as Faiss support similarity search at this scale. The objective is to retain the useful material while making the next stage cheaper. Even reducing billions to millions could help an offline processing job, but an agent answering one question needs a much smaller shortlist. What matters is how much relevant evidence survives the cut.

For exploration, you may welcome someone else’s criterion. Sometimes you don’t yet know what you are looking for: you are browsing images for inspiration, or exploring an unfamiliar collection of articles. A learned grouping gives you somewhere to start and may reveal connections you would not have thought to ask about. It is also useful when there are no helpful folders, labels or captions to navigate. For a loose collection of images, a model trained to align text and images, such as CLIP, can let you search by description. The model supplies an organisation you can explore and question as your own criterion becomes clearer.

For simple recommendations, resemblance may be the job. “Show me other articles like this one” is a reasonable request. An embedding model that captures the relevant subject matter can provide a useful first recommendation system, including for new articles with no readership history. That does not automatically capture everything a reader wants—novelty, variety and personal preferences also matter—but a ranking of related items may be sufficient for the feature you are building. Check it against what your readers actually find useful.

For agents, embeddings can provide an entry point into a navigable environment. In our earlier example, a search result accompanied by its path might land the agent at v4/security/mfa.docx. The document does not answer the question, but the path gives the agent somewhere to start. If it can list neighbouring directories, it can discover v4/management/ and continue towards guest_access.docx. Return the source path alongside the passage, and provide tools to inspect the surrounding files. The search supplies a starting point; the environment supplies the next moves. This combination of retrieval and navigation fits the approach described in Anthropic’s context engineering guide.

With four shapes, you can inspect every difference yourself. With four million items, a useful learned arrangement becomes a considerable advantage. Its value depends on whether it helps you do the work—and that is something you can evaluate.