Vector search benchmarks suite evaluates FAISS, ScaNN, and USearch efficiency

An independent benchmarking repository tests popular vector search libraries across dataset sizes up to one million samples, measuring search speed, memory consumption, and accuracy against exact search.

For engineers choosing an in-memory vector search library, the vector-search-benchmarks suite by developer Mohamed provides a clean, reproducible framework to compare FAISS, ScaNN, and USearch. If you require absolute search speed at scale, ScaNN remains the top choice, though it demands a complex build environment. For lightweight, dependency-free deployments, USearch offers the best memory-to-performance ratio. Skip this benchmark suite if you are evaluating managed cloud vector databases, as it focuses strictly on local, in-memory library performance.

Methodology

This review evaluates the vector-search-benchmarks repository (commit state as of May 2026) published by developer Mohamed (GitHub user mohamed-em2m). This v0 review draws on the author's published claims and repository structure at https://github.com/mohamed-em2m/vector-search-benchmarks and the associated results page at https://mohamed-em2m.github.io/vector-search-benchmarks/. Independent execution of the benchmark suite is pending. The suite tests vector search libraries including FAISS, ScaNN, and USearch across dataset scales ranging from 500 to 1,000,000 samples. It measures index build time, search latency, memory footprint, and recall accuracy compared to an exact flat search. This review does not cover long-term memory leak testing, multi-threaded scaling under high concurrent write loads, or performance on non-x86 CPU architectures.

What the suite does

Reproducible vector library testing

The repository provides a structured Python framework to register, configure, and run vector search libraries under identical hardware constraints. By standardizing the input dataset, the suite ensures that FAISS, ScaNN, and USearch are evaluated on equal terms. The test runner automates dataset generation, index creation, and query execution.

Multi-scale dataset evaluation

The framework tests performance across a wide spectrum of dataset sizes, starting at a minimal 500 samples and scaling up to 1,000,000 samples. This scaling is critical because vector search behaviors change dramatically when index structures exceed CPU cache limits or system memory thresholds.

Three-dimensional metric tracking

Instead of focusing solely on queries per second, the suite tracks three primary metrics. First, it measures search speed (latency per query). Second, it records memory usage during indexing and search. Third, it calculates similarity recall accuracy by comparing approximate nearest neighbor results against an exact brute-force search.

What is interesting and what is not

What is interesting

The value of Mohamed's suite lies in its focus on local, in-memory libraries rather than managed cloud databases. Many modern benchmarks are published by cloud vendors and focus on network-bound API performance. This repository isolates raw algorithmic efficiency. The inclusion of USearch is particularly valuable. USearch is a smaller, single-header alternative to FAISS, and seeing it benchmarked directly against Google's ScaNN and Meta's FAISS helps developers understand if they can trade a few percentage points of recall for a significantly simpler deployment footprint. The framework's registration pattern also makes it easy to add other libraries like HNSWLIB or Milvus Lite.

What is not interesting

The suite relies on synthetic datasets, which do not always reflect real-world vector distributions. Real-world embeddings from models like Cohere or OpenAI have specific clustering behaviors that affect index traversal. Furthermore, the repository lacks automated hardware profiling. Because vector search libraries rely heavily on SIMD instructions like AVX-512 or ARM Neon, performance figures are highly hardware-dependent. Without explicit CPU and compiler flag logging in the results, comparing raw numbers across different machines remains difficult. The author's published results also lack deep analysis of the trade-offs between index build time and query latency.

Pricing

The vector-search-benchmarks suite is open-source and free, licensed under the MIT License. The underlying libraries (FAISS, ScaNN, USearch) are also free and open-source. This pricing snapshot is accurate as of May 2026.

Verdict

The vector-search-benchmarks suite is a highly useful tool for developers who need to select an in-memory vector library for edge deployment or local search tasks. If your application demands minimal memory usage and easy compilation, the benchmarks indicate USearch is the optimal choice. If your priority is maximum search throughput on large datasets and you can tolerate a heavier dependency chain, ScaNN is the clear winner. We recommend using this repository to run local tests on your specific target hardware rather than relying solely on the pre-compiled HTML results.

What we would test next

In a future evaluation, we would run this suite on a dedicated bare-metal instance to isolate CPU cache effects. We would replace the synthetic datasets with real-world embeddings, specifically 768-dimension text embeddings and 1536-dimension document embeddings. Finally, we would benchmark the impact of concurrent write operations during active search queries, which is a common failure mode for in-memory vector indexes.

The investor read

The vector database market has seen massive venture funding flow to managed services like Pinecone, Milvus, and Qdrant. However, Mohamed's benchmarks highlight a counter-trend: the high efficiency of local, in-memory libraries like USearch and ScaNN. For many enterprise applications, especially those operating at the edge or within strict privacy boundaries, a lightweight embedded library is both cheaper and faster than a network-bound cloud database. Investors should watch USearch as a signal that vector search is commoditizing down to the library level. If a single-header library can deliver comparable recall and latency to complex distributed systems for datasets under one million vectors, the addressable market for expensive, standalone vector database platforms may shrink to only the largest web-scale enterprises.

Pull quote: “For lightweight, dependency-free deployments, USearch offers the best memory-to-performance ratio.”