Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers

Table of Contents What are Multi-Vector Models? The MaxSim Operator What You Gain, and What It Costs Installation Loading a Model Inspecting What a Checkpoint Configured Encoding Queries and Documents Scoring with MaxSim Score Magnitude and MeanMaxSim Semantic Search Retrieve and Rerank Indexing Visual Document Retrieval Audio Retrieval Video Retrieval Interpretability Token Pooling Speeding Up Inference Evaluating a Model Coming from PyLate or colpali-engine Supported Models Text Retrieval Models Visual Document Retrieval Models Acknowledgements Additional Resources Documentation Example Scripts Training Hugging Face Hub Companion Blogposts Sentence Transformers is a Python library for using and training embedding and reranker models for applications like retrieval augmented generation, semantic search, and more.

With the v6.0 update, it gains a fourth model type: MultiVectorEncoder, for ColBERT-style late interaction retrieval. Any PyLate checkpoint and any Stanford-NLP ColBERT checkpoint loads straight into it, and colpali-engine models for visual document retrieval can be used too, through the same familiar API you already use for dense, sparse, and reranker models. Where a regular embedding model compresses a whole text into one vector, a multi-vector model keeps one vector per token and scores query against document with the MaxSim operator. That preserves token-level matching information that a single vector has to average away, which usually means stronger retrieval at the cost of a bigger index. It’s also the state of the art for visual document retrieval, where a text query is matched against page images directly, with no OCR step in between. In this blogpost, we’ll show you how to use these models: loading the various checkpoint formats, encoding and scoring, plugging them into a search stack, running them on page images, and keeping the index affordable. Everything below runs on a plain pip install -U sentence-transformers. A dense embedding model reads a text and returns a single fixed-size vector. Everything the model noticed has to fit in those 384, 768, or 1024 numbers, and similarity is one dot product between two such summaries. This works remarkably well, but the compression is lossy in a specific way: a rare entity, an exact identifier, or one crucial clause in a long passage all have to compete for room in the same vector. A query with several requirements at once runs into the same wall. For “green sofa with wooden legs and rounded cushions”, a single vector has to blend all four into one point, so a green sofa with the wrong legs ends up sitting close to the one you actually asked for. A multi-vector model (also called a late-interaction or ColBERT-style model, after the ColBERT paper) skips that compression. It runs the same transformer, but instead of pooling the token embeddings into one vector, it projects each token embedding down to a small dimension (classically 128) and keeps all of them. A 9-token document becomes a 9×128 matrix, not a 1×128 vector.