Revolutionizing Hybrid Search: Introducing BM42 Algorithm by Qdrant
Qdrant Introduces BM42 Algorithm to Revolutionize Hybrid Search
Qdrant, a leading provider of vector search technology, has recently unveiled BM42, a groundbreaking algorithm designed to transform hybrid search. For the past forty years, the industry standard algorithm for search engines has been BM25, used by major players like Google and Yahoo. However, with the emergence of vector search and the introduction of Retrieval-Augmented Generation (RAG), the need for a more advanced solution has become apparent. BM42 aims to bridge this gap by combining the strengths of BM25 with modern transformer models, offering a significant upgrade for search applications.
The Legacy of BM25
BM25 has maintained its relevance over the years due to its simple yet effective formula, which calculates document relevance based on term frequency and inverse document frequency (IDF). While this method excels in traditional web search environments with consistent document lengths and query structures, the rise of RAG systems has introduced challenges with shorter, more varied documents and queries. In these scenarios, BM25’s reliance on document statistics becomes less effective.
The Introduction of BM42
BM42 addresses these challenges by integrating the core principles of BM25 with transformer models. The key innovation in BM42 lies in using attention matrices from transformers to determine the importance of terms within documents. By leveraging attention matrices, BM42 can accurately gauge the significance of each token in a document, even for shorter texts typical in RAG applications.
Advantages of BM42
BM42 offers several advantages over BM25 and SPLADE, another modern alternative that uses transformers to create sparse embeddings. While SPLADE has shown superior performance in academic benchmarks, it faces challenges with computational resources, tokenization, and domain dependency. BM42 retains the interpretability and simplicity of BM25 while overcoming these limitations.
One of BM42’s primary benefits is its efficiency. The algorithm can quickly perform document and query inferences, making it suitable for real-time applications with a low memory footprint. BM42 supports multiple languages and domains, making it highly versatile.
Practical Implementation
BM42 can be seamlessly integrated into Qdrant’s vector search engine by setting up a collection for hybrid search with BM42 and using dense embeddings from models like jina.ai. This combination allows for a balanced approach, enhancing retrieval accuracy in modern search applications.
Encouraging Community Engagement
Qdrant’s release of BM42 encourages community engagement and innovation. Developers and researchers are invited to experiment with BM42, share their projects, and contribute to its ongoing development. By providing this powerful tool, Qdrant aims to empower its community to push the boundaries of search technology.
Conclusion
The introduction of BM42 by Qdrant marks a significant milestone in search algorithm evolution. By combining the robustness of BM25 with transformer intelligence, BM42 sets a new standard for hybrid search, offering a versatile, efficient, and highly accurate solution for today’s search applications.
