*NeoMME*: an efficient Multimodal-native and Multilingual Encoder
Hugging Face tL;DR Why another multimodal encoder? NeoMME encoder backbone One Transformer for images and text Learning from images through masked text NeoMME-Retriever A dual-head design for dense and late-interaction retrieval Competitive retrieval at compact model sizes Making high-resolution retrieval practical for late-interaction Try NeoMME-Retriever yourself!
Fine-tuning with Sentence Transformers From retrieval to visual RAG Conclusion Acknowledgements Citation We introduce NeoMME, a family of 260M and 800M multilingual multimodal encoders. Unlike many generative visual language models, NeoMME does not use a separate pretrained vision tower or a causal language model. A single bidirectional Transformer processes both text tokens and raw image patches, and we train the entire model from scratch with a masked discrete-diffusion objective. We fine-tuned NeoMME for visual document retrieval using ColPali’s page-image approach. NeoMME-Retriever returns dense and late-interaction embeddings in one forward pass. Both model sizes lie on the ViDoRe v3 Pareto frontier for nDCG@10 and model size. At a matched 2048×2048 image input size on an NVIDIA L40S GPU, the 260M model encodes about 51 pages per second, or about twice ColModernVBERT’s throughput. Hierarchical token pooling and asymmetric quantization reduce late-interaction index storage from roughly 1.5 MB to 6 kB per page (255× smaller) while retaining more than 95% of baseline nDCG@10. NeoMME is available in Hugging Face Transformers. We release all model checkpoints under the Apache 2.0 license. Many recent visual document retrievers are adapted from pretrained generative visual language models. A separately pretrained vision encoder produces visual features, which a projector maps into the language model’s input space. A causal decoder then processes the combined image and text representations. Retrieval, classification, and token labeling do not generate text autoregressively, so they do not require a causal decoder or the parameter and compute overhead of this architecture. ModernBERT brought efficient architecture and training improvements to bidirectional encoders.
