MongoDB Vector Search Benchmark Overview

Quantization

Quantization reduces the precision of vector embeddings to decrease memory usage and improve search speed, with trade-offs in search accuracy.

Vector Dimensionality

We used Voyage AI's voyage-3-large model to embed the medium (5.5M) and large (15.3M) vectors datasets. We chose this embedding model because of its outperformance on many IR benchmarks and because it is trained with both Matryoshka Representation Learning and quantization in mind. Therefore, it performs well at lower dimensions with quantization enabled, even at higher volumes of vectors.

We leveraged indexing on views to produce additional fields that slice the first N dimensions of the source 2048 dimension vector to produce 1024, 512, and 256 dimension vectors and index them as we would the source field.

db.createView(
  "all_dims_amazon_dataset",
  "2048d_amazon_dataset",
  [
    {
      $addFields: {
        "1024_embedding": { $slice: ["$embedding", 1024] },
        "512_embedding": { $slice: ["$embedding", 512] },
        "256_embedding": { $slice: ["$embedding", 256] }
      }
    }
  ]
)

db.all_dims_amazon_dataset.createSearchIndex(
  "all_dims_vector_index",
  "vectorSearch",
  {
    "fields": [
      {
        "numDimensions": 2048,
        "path": "embedding", // original 2048d embedding produced by voyage-3-large
        "quantization": "scalar", // adjust to binary when needed
        "similarity": "dotProduct",
        "type": "vector"
      },
      {
        "numDimensions": 1024,
        "path": "1024_embedding",
        "quantization": "scalar",
        "similarity": "cosine", // sliced embeddings aren't normalized, so must use cosine
        "type": "vector"
      },
      {
        "numDimensions": 512,
        "path": "512_embedding",
        "quantization": "scalar",
        "similarity": "cosine",
        "type": "vector"
      },
      {
        "numDimensions": 256,
        "path": "256_embedding",
        "quantization": "scalar",
        "similarity": "cosine",
        "type": "vector"
      }
    ]
  }
)

Similar to different representations at each position, the different dimensionalities impact the representational capacity of each vector. Consequently, you can achieve higher accuracy with 2048d vectors compared to 256d vectors, especially when you measure against a 2048d float ENN baseline.

In addition to requiring more storage and memory, higher dimensional vectors are somewhat slower to query compared to lower dimensional vectors, but this is mitigated significantly as MongoDB Vector Search leverages SIMD instructions when performing vector comparisons.

Filtering

We also created a separate index definition on the collection containing all 15.3M items, which includes filters on two fields to enable pre-filtered queries against this dataset.

db.large_amazon_dataset.createSearchIndex(
  "vectorSearch",
  "large_vector_index",
  {
    "fields": [
      {
        "numDimensions": 2048,
        "path": "embedding",
        "quantization": "scalar", // adjust to binary when needed
        "similarity": "dotProduct",
        "type": "vector"
      },
      {
        "path": "category",
        "type": "filter"
      },
      {
        "path": "price",
        "type": "filter"
      }
    ]
  }
)

We ran vector search queries, both unfiltered and filtered, against the large indexed dataset:

# unfiltered query
query = [
  {
    "$vectorSearch": {
      "index": "large_vector_index",
      "path": "embedding",
      "queryVector": embedding.tolist(),
      "limit": k,
      "numCandidates": candidates,
    }
  },
  {
    "$project": {"embedding": 0}
  }
]

# filtered query
query = [
  {
    "$vectorSearch": {
      "index": "large_vector_index",
      "path": "embedding",
      "queryVector": embedding.tolist(),
      "limit": k,
      "numCandidates": candidates,
      "filter": {"$and": [{'price': {'$lte': 1000}}, {'category': {'$eq': "Pet Supplies"}}]}
    }
  },
  {
    "$project": {"embedding": 0}
  }
]

Search Node Configuration

MongoDB Vector Search performance scales with dedicated Search Nodes, which handle vector computations separately from your primary database workload and make efficient use of dedicated hardware instances. All tests were conducted using an M20 base cluster, but depending on the type of test, we reconfigured the Search Nodes used to better fit our test case. All tests were run using Search Nodes on AWS us-east-1, with an EC2 instance also in us-east-1 making requests. There are three types of Search Nodes that you can provision on AWS, which vary in terms of disk, RAM, and vCPUs that they have available:

binData Vector Compression

For the large dataset, we leveraged an additional feature called vector compression, which reduces the footprint of each vector in the source collection by about 60%. This accelerates the step within a query when IDs are hydrated in the source collection, and this is a recommended step for all large workloads.

Concurrency

We assessed not only serial query latency, but also total throughput/QPS when 10 and 100 requests are issued concurrently.

Sharding

We assessed the impact of sharding our cluster and collection on the _id field on the system's throughput , focusing on request concurrency of 10 and 100 for the large binary quantized index.