mirror of
https://github.com/FerretDB/documentdb.git
synced 2026-11-06 19:12:50 -05:00
Zinan Ren
ac6f028c86
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
With the release of [pgvector 0.8.0](https://github.com/pgvector/pgvector/tree/5bc7937715c67add21c8bcc0d4284162c7a0174f?tab=readme-ov-file#iterative-index-scans), which supports iterative scan for HNSW and IVF, we enable it in filtering search to support pre-filtering(currently it is post-filtering)
- Pass the new parameter "iterativeScan" through the function bson_search_param from the coordinator to the worker.
Example : { "efSearch" : 10, "iterativeScan" : "relaxed_order"}
- Enable iterative scan for hnsw/ivf index on workers:
SET LOCAL hnsw.iterative_scan = relaxed_order;
SET LOCAL ivfflat.iterative_scan = relaxed_order;
relaxed_order by default, which allows results to be slightly out of order by distance, but provides better recall.
After iterative search, re-ordering will be applied to the results.
- Set iterative scan for diskann index to relaxed_order
SET LOCAL diskann.iterative_search = relaxed_order;
- Check the availability of iterative scan in pgvector
The pgvector iterative scan is available from version 0.8.0. Currently, if the pgvector extension has not been upgraded to 0.8.0, the iterative scan will be disabled in those clusters until pgvector is upgraded to 0.8.0.
###Related GUCs
**documentdb.vectorPreFilterIterativeScanMode**: enable iterative index scans, which will automatically scan more of the index until enough results are found (relaxed_order by default)
- off, disable iterative scan
- relaxed_order, allows results to be slightly out of order by distance, but provides better recall
- strict_order, ensures results are in the exact order by distance
####GUCs from pgvector
**hnsw.max_scan_tuples**: Specify the max number of tuples to visit (20,000 by default)
**hnsw.scan_mem_multiplier**: Specify the max amount of memory to use, as a multiple of work_mem (1 by default)
**ivfflat.max_probes**: Specify the max number of probes(32768 by default)
###Mongo Command(no change in this PR)
```
db.testing_hnsw_vector_collection.aggregate(
[
{
"$search": {
"cosmosSearch": {
"vector": [ 3.0, 4.9, 1.0 ],
"path": "vect",
"filter": {"meta.a": [ { "b" : 3 } ]},
"k": 4,
"efSearch": 100
},
}
},
{
"$project": {
"rank": {
"$floor": [{"$multiply": [{ "$meta": "searchScore"},100000]}]
}
}
}
])
```
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces iterative index scans for HNSW and IVF in filtering searches, enhancing the pre-filtering capabilities of vector searches...
26 lines
718 B
C
26 lines
718 B
C
/*-------------------------------------------------------------------------
|
|
* Copyright (c) Microsoft Corporation. All rights reserved.
|
|
*
|
|
* include/vector/vector_planner.h
|
|
*
|
|
* Function declarations for vector index used in planner and custom scan.
|
|
*
|
|
*-------------------------------------------------------------------------
|
|
*/
|
|
#ifndef VECTOR_PLANNER__H
|
|
#define VECTOR_PLANNER__H
|
|
|
|
#include <nodes/pathnodes.h>
|
|
#include <nodes/pg_list.h>
|
|
|
|
#include "io/bson_core.h"
|
|
#include "vector/vector_utilities.h"
|
|
|
|
|
|
pgbson * CalculateSearchParamBsonForIndexPath(IndexPath *vectorSearchPath,
|
|
pgbson *searchParamBson);
|
|
|
|
void SetSearchParametersToGUC(Oid vectorAccessMethodOid, pgbson *searchParamBson);
|
|
|
|
#endif
|