Mixedbread's Asymmetric Quantization claims 97% storage reduction with minimal accuracy loss

A new embedding quantization technique promises near-lossless information retrieval with massive storage and cost savings. We analyze the published benchmarks and assess its fit for production RAG systems.

The Answer Up Front

For teams building Retrieval-Augmented Generation (RAG) systems at scale, where the cost of embedding storage is a primary bottleneck, Asymmetric Quantization (AQ) is a technique to evaluate immediately. Teams that require the absolute highest retrieval accuracy and for whom cost is no object can likely wait for broader adoption and independent testing. The bottom line is that AQ, as presented by mixedbread.ai, appears to be a significant step in making high-performance semantic search economically viable for a wider range of applications, assuming the published benchmarks hold up in real-world scenarios.

Methodology

This is a v0 review based on the technical blog post “Asymmetric Quantization: Near-Lossless Retrieval with 97% Storage Reduction,” published by mixedbread.ai on July 2, 2024. All performance metrics and technical details are drawn directly from this source. We are analyzing the company's published claims and its benchmark results on the Massive Text Embedding Benchmark (MTEB) leaderboard. The analysis covers the technical approach of Asymmetric Quantization and its reported impact on storage size and retrieval performance.

This review does not include independent, hands-on benchmarking. We have not validated the claimed MTEB scores, measured retrieval latency, or tested the provided Hugging Face model (mxbai-embed-large-v1-aq) on a private dataset. The performance claims are presented as reported by the vendor. We plan to conduct independent tests for a future update.

What It Does

Asymmetric Quantization is a technique for compressing text embeddings to reduce their storage footprint while preserving most of their retrieval performance. The core value proposition rests on two main claims.

Reduces embedding storage by 32x

The primary function of AQ is compression. According to the post, it reduces the company's mxbai-embed-large-v1 model's vectors from 3072-byte float32 representations down to 96-byte quantized vectors. This is a 97% reduction in storage size. For a corpus of 100 million documents, this would reduce the storage requirement from approximately 307 GB to just 9.6 GB, a massive saving in both RAM for in-memory indexes and persistent disk storage.

Maintains near-lossless retrieval accuracy

Compression in vector search typically involves a trade-off with accuracy. The company claims its AQ method largely avoids this. Their quantized model reportedly achieves a score of 67.51 on the MTEB benchmark, which is 99.1% of the 68.12 score achieved by the original, uncompressed float32 model. This suggests a minimal performance penalty for the significant storage savings.

An asymmetric approach to search

The technique is named for its asymmetric handling of query and document vectors. A user's search query is kept as a high-precision float32 vector. The massive corpus of document vectors, however, is stored in the compressed low-precision format. During a search, the high-precision query is compared directly against the low-precision document vectors, which the authors claim is key to maintaining high accuracy without needing to decompress the entire document index.

What's Interesting / What's Not

The most interesting aspect is the direct impact on the economics of building with LLMs. The cost of generating, storing, and serving embeddings is a significant and recurring operational expense. A 32x reduction in storage directly attacks this problem, making large-scale semantic search accessible to teams with smaller budgets. Releasing the technique via a public model on Hugging Face is also a strong move, as it allows any team to immediately begin verifying the claims.

The asymmetric architecture is a clever solution. It focuses compression on the largest part of the system (the document store) while preserving the full information content of the query, where precision matters most for finding relevant results.

What's missing from the analysis is a detailed discussion of latency. The post focuses on storage and retrieval accuracy, but the computational cost of comparing a float32 query vector to a quantized document vector is not benchmarked. It is unclear how this process impacts end-to-end search speed compared to float32-to-float32 or other quantization methods like Binary or Scalar Quantization. Furthermore, all benchmarks use the public MTEB dataset. Performance on noisy, domain-specific enterprise data remains an open question.

Pricing

As of July 2024, Asymmetric Quantization is available as a technique implemented in a model, not a priced software product. The model, mixedbread-ai/mxbai-embed-large-v1-aq, is available on Hugging Face, presumably under an open-source license (an explicit license was not specified in the blog post). Use is free, aside from the computational cost of running the model.

Verdict

Asymmetric Quantization is a compelling technique for any team feeling the financial pressure of vector database costs at scale. The published results from mixedbread.ai present an almost ideal trade-off: massive (97%) storage reduction for a negligible (sub-1%) drop in retrieval accuracy. If these numbers are reproducible across different datasets and workloads, AQ could become a standard tool for building cost-effective RAG systems. While independent verification is a critical next step, this is one of the most promising and practical developments in vector compression this year. Teams building semantic search should put this on their short list for evaluation.

What We'd Test Next

For a v2 review, we would conduct independent benchmarks. First, we would measure end-to-end retrieval latency, comparing the AQ model against the float32 original and other popular quantized models. Second, we would evaluate retrieval performance on a private, domain-specific dataset, such as a corpus of technical documentation or customer support tickets, to see how it performs outside of a standard benchmark. Finally, we would analyze the integration overhead and performance within popular vector databases that support custom quantization schemes.

The investor read

Asymmetric Quantization is a pure 'picks and shovels' play for the AI market. The cost of storing and serving embeddings is a major, recurring operational expense for AI-native companies, and any technology that drastically reduces this cost is an enabling one. This technique directly attacks the storage cost component of vector database providers, potentially commoditizing a feature they might otherwise sell at a premium. While mixedbread.ai's commercial strategy isn't yet clear, the technique itself signals a large and durable market for efficiency tools in the AI stack. An investable company here would need a clear distribution model and a moat beyond a single algorithm, such as a suite of optimization tools or deep integrations into the data pipeline.

Pull quote: “The bottom line is that AQ, as presented by mixedbread.ai, appears to be a significant step in making high-performance semantic search economically viable for a wider range of applications, assuming the published benchmarks hold up in real-world scenarios.”