Cosine, dot product and distance: choose the metric your embedding model expects
Compare three similarity calculations on small vectors and see when normalization changes the ranking.
On this page
The short answer
Use the similarity metric and preprocessing recommended for the embedding model. Cosine compares direction, dot product also reflects vector magnitude, and Euclidean distance measures separation. For unit vectors, cosine and dot product coincide and squared Euclidean distance equals 2 minus twice cosine. That equivalence requires normalization; it does not make arbitrary service scores interchangeable. Check both document and query vectors before changing an index.
Start with the embedding contract
Record the model identifier, revision, vector dimension and recommended similarity measure. Query and document embeddings must belong to compatible representations; equal dimensions alone do not prove compatibility. Some retrieval models use different query and document instructions, so compatibility does not necessarily mean identical input formatting. Apply the documented preprocessing consistently and preserve it with the index configuration.
Distinguish direction and magnitude
For nonzero vectors q and x, cosine is their dot product divided by the product of their lengths. A positive rescaling of either vector leaves cosine unchanged. Dot product has no such invariance: making a vector longer can increase its score even when its direction stays the same. Decide whether length carries useful information in this particular model rather than treating normalization as a universally harmless cleanup.
Calculate three rankings by hand
Consider the constructed vectors q=(1,0), a=(2,0) and b=(1,1). Their dot products with q are 2 and 1, so dot product ranks a first. Cosines are 1 and approximately 0.707, also ranking a first. Raw Euclidean distances are both 1, producing a tie. This arithmetic example demonstrates different metric behavior; it is not a measured retrieval result or evidence that one metric is more accurate.
Understand the unit-vector equivalence
After normalization, a becomes (1,0), b becomes approximately (0.707,0.707), and q remains (1,0). Their distances from q become 0 and approximately 0.765. For two unit vectors, expanding the squared distance gives 1+1-2 times their dot product. Thus minimizing distance and maximizing cosine produce the same ordering in exact arithmetic under these assumptions. Approximate indexes and tie handling can still produce different returned candidate sets.
Reject invalid vectors before indexing
A zero vector cannot be normalized to unit length by division, and cosine is undefined when either length is zero. Nonfinite values and inconsistent dimensions should also be treated as data or pipeline problems. Define whether such records are rejected, quarantined or retried. Silently replacing every invalid vector with zeros creates a new representation without establishing that the embedding model or search service supports it.
Interpret the score returned by the service
An API can transform a distance or similarity into its own ranking score. Azure AI Search, for example, documents a transformed cosine score rather than returning raw cosine as its search score. Therefore a threshold copied from another service or from a hand calculation may be wrong. Read the score definition for the actual query mode, including whether hybrid retrieval combines multiple rankings.
Evaluate changes on meaningful queries
Before switching a production metric, compare a small set of representative queries with explicit relevance judgments. Keep the documents, model and preprocessing fixed when isolating a metric change. If using an approximate index, distinguish a candidate-recall problem from a scoring problem. Record missing relevant documents and unexpected high-ranked results instead of interpreting a visually larger score as a quality improvement.
Keep normalization versioned with the index
Changing normalization changes stored vectors and can require rebuilding an index. Normalizing only queries while leaving an incompatible document representation unchanged is not a general migration strategy. Save the model revision, metric and normalization settings together so a later deployment can reproduce the representation. Verify the model card for the model actually selected; a recommendation for one normalized embedding model does not establish a rule for all models.
Things to check
- Use compatible query and document representations.
- Confirm the model-specific metric recommendation.
- Reject zero and nonfinite vectors under a documented policy.
- Read the actual service score definition.
- Treat the small-vector values as illustrative arithmetic.
Where this applies
The equivalence applies to nonzero unit vectors in exact arithmetic. Approximate indexing, quantization, hybrid ranking and service score transformations can affect returned results. This guide does not establish retrieval accuracy for a particular corpus.