upstream/documentdb
2
0 Fork 0
mirror of https://github.com/FerretDB/documentdb.git synced 2026-11-07 03:22:46 -05:00
Code Releases 1 Activity
916 Commits 7 Branches 21 Tags
double-ok
Commit Graph
11 Commits
Author SHA1 Message Date
Zinan Ren f4896e9c81 Merged PR 1670376: [ReleaseValidation] Add release validation tests for hnsw/diskann/filtering/compression 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Testing
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
1. Check the max dimensions when creating the vector index, previously checked by pgvector
2. Add the following release validation tests:
- HNSW index (M40 and above)
- DiskANN index (M30 and above)
- Filtering search (GA)
- Exact search (GA)
- Compression half (GA)
- Compression PQ (GA)

----
#### AI description  (iteration 1)
#### PR Classification
This pull request implements new vector search release validation tests and updates the backend’s vector index dimension checks and error messaging.

#### PR Summary
The changes add comprehensive release validation tests for vector search covering various compression types (PQ, half), index kinds (ivf, hnsw, diskann), filtering, and exact match scenarios.
- `outerLoop/releaseValidation/testSpecs/` now includes multiple new JSON spec files (e.g. ValidateVectorSearchCompressionPQ.json, ValidateVectorSearchCompressionHalf.json, ValidateVectorSearchDiskANN.json, ValidateVectorSearchHNSW.json, ValidateVectorSearchFiltering.json, and ValidateVectorSearchExact.json) with updated expected errors.
- Test scripts in `ReleaseValidation.Tests.NodeJs/test/test_signoffTestsNodeJS.js` and C# signoff test file `SignoffTestsCSharp.cs` have been modified to enable and incorporate these new vector search tests.
- In `pg_documentdb/src/commands/create_indexes.c`, additional checks enforce maximum dimensions based on compression type (non-compressed: 2000, half: 4000, PQ: as defined) and update error messages accordingly.
- Several Postgres gateway and pgmongo files have been updated to remove obsolete regexes and return revised error codes (e.g. error code 67) and messages for exceeding vector dimension limits.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-05-29 12:12:31 +08:00
Zinan Ren 4b79ce38d4 Merged PR 1668682: [Vector Search] Add GUC for the default search parameter calculation 
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

Adding the GUC documentdb.enableVectorCalculateDefaultSearchParam to enable or disable the calculation of default search parameters. The default value is on
Adding feature counts for the usage of default search parameters.

----
#### AI description  (iteration 1)
#### PR Classification
This PR implements a new configuration option to enable or disable the dynamic calculation of default search parameters for vector search indexes.

#### PR Summary
The pull request adds a GUC flag that controls whether default search parameters (e.g., nProbes for IVF and efSearch for HNSW) are computed dynamically based on collection size. It updates internal calculation functions, test expectations, and feature counters accordingly.

- `src/vector/vector_index_kind_impl.c`: Revised the IVF and HNSW search parameter calculation functions to conditionally compute default parameters when the new GUC is enabled.
- `src/configs/feature_flag_configs.c` and `include/vector/vector_common.h`: Introduced the new boolean GUC `enableVectorCalculateDefaultSearchParam`.
- Test files (e.g., files under `/oss/internal/pg_documentdb_distributed/src/test/regress/expected/` and corresponding `pgmongo` tests): Updated expected outputs and query plans to reflect the new default parameter values.
- Feature counter outputs and related SQL tests: Modified to include metrics for the new default search parameter usage.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

Related work items: #4194082
 2025-05-29 12:12:31 +08:00
Zinan Ren ac6f028c86 Merged PR 1653287: [Vector Search] Enable iterative index scans for HNSW and IVF in filtering searches 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes

### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description
With the release of [pgvector 0.8.0](https://github.com/pgvector/pgvector/tree/5bc7937715c67add21c8bcc0d4284162c7a0174f?tab=readme-ov-file#iterative-index-scans), which supports iterative scan for HNSW and IVF, we enable it in filtering search to support pre-filtering(currently it is post-filtering)
- Pass the new parameter "iterativeScan" through the function bson_search_param from the coordinator to the worker.
Example : { "efSearch" : 10, "iterativeScan" : "relaxed_order"}
- Enable iterative scan for hnsw/ivf index on workers:
SET LOCAL  hnsw.iterative_scan = relaxed_order;
SET LOCAL  ivfflat.iterative_scan = relaxed_order;
relaxed_order by default, which allows results to be slightly out of order by distance, but provides better recall.
After iterative search, re-ordering will be applied to the results.
- Set iterative scan for diskann index to relaxed_order
SET LOCAL  diskann.iterative_search = relaxed_order;
- Check the availability of iterative scan in pgvector
The pgvector iterative scan is available from version 0.8.0. Currently, if the pgvector extension has not been upgraded to 0.8.0, the iterative scan will be disabled in those clusters until pgvector is upgraded to 0.8.0.

###Related GUCs
**documentdb.vectorPreFilterIterativeScanMode**: enable iterative index scans, which will automatically scan more of the index until enough results are found (relaxed_order by default)
- off, disable iterative scan
- relaxed_order,  allows results to be slightly out of order by distance, but provides better recall
- strict_order, ensures results are in the exact order by distance

####GUCs from pgvector
**hnsw.max_scan_tuples**: Specify the max number of tuples to visit (20,000 by default)
**hnsw.scan_mem_multiplier**: Specify the max amount of memory to use, as a multiple of work_mem (1 by default)
**ivfflat.max_probes**: Specify the max number of probes(32768 by default)

###Mongo Command(no change in this PR)
```
db.testing_hnsw_vector_collection.aggregate(
[
  {
    "$search": {
	  "cosmosSearch": {
        "vector": [ 3.0, 4.9, 1.0 ],
        "path": "vect",
        "filter": {"meta.a": [ { "b" : 3 } ]},
        "k": 4,
        "efSearch": 100
      },
    }
  },
  {
    "$project": {
      "rank": {
        "$floor": [{"$multiply": [{ "$meta": "searchScore"},100000]}]
      }
    }
  }
])
```

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request introduces iterative index scans for HNSW and IVF in filtering searches, enhancing the pre-filtering capabilities of vector searches...
 2025-05-09 14:08:53 +08:00
Zinan Ren e05e91ba51 Merged PR 1650017: [Vector Search] Add support for PQ compression in the Diskann index 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Adding support Product Quantization (PQ) compression in vector index, which is a popular method for dramatically compressing high-dimensional vectors to use less memory, and for making nearest-neighbor search faster.
Note: currently, this PQ compression is only supported by the Diskann index.

### Related GUCs
**documentdb.enableVectorCompressionPQ**: Enables vector PQ compression (enabled by default).

### Create index spec:
- **compression**(optional):
  - "half": half-precision vector, which using 16 bits floats.
  - "pq": product quantization compression.

- **pqCompressedDims**(optional): indicates the dimensions after compression. If not specified, it will be automatically calculated based on the original dimensions.
  - Minimum: 1
  - Maximum: 8000

- **pqSampleSize**(optional):Indicates the number of samples PQ uses to generate the initial centroids. The default value is 1000.
  - Minimum: 1000
  - Maximum: 100000

**Note**: It is recommended to create the index after the collection has some data.. If the collection is empty, random vectors within the range [-1, 1] are used to generate the initial centroids and create the index. If the number of documents is less than pqSampleSize, the training data is padded with random data within the range [min, max] from the sampled vector data.

### Command example
```
db.runCommand(
{ "createIndexes": "testing_vector_collection", "indexes":
[{
     "key": { "v": "cosmosSearch" },
     "name": "hsnw_index",
     "cosmosSearchOptions":
     {
         "kind": "vector-diskann",
         "similarity": "L2",
         "dimensions": 1536,
         "compression": "pq",
         "pqCompressedDims": 96,
         "pqSampleSize": 2000
     }
}]
} )
```
### Search spec:
No changes

----
#### AI description  (iteration 1)
#### PR Classification
New feature: This pull request adds support for Product Quantization (PQ) compression in the Diskann index.

#### PR Summary
This pull request introduces support for PQ compression in the Diskann index, enhancing vector search capabilities. It includes error handling for unsupported compression types and validation for PQ parameters.
- Updated `/pgmongo/src/test/regress/expected/bson_aggregation_pipeline_tests_vector_diskann.out` to include tests for PQ compression, error cases, and validation of PQ parameters such as `pqCompressedDims` and `pqSampleSize`.
- Added tests for vector search with PQ compression, including L2, cosine, and IP similarity measures.
- Implemented checks for maximum dimensions and resource limits during PQ index creation.
<!-- GitOpsUserAgent=GitOps.Apps.S...
 2025-04-30 14:57:09 +08:00
Zinan Ren 591a38b86d Merged PR 1606660: [Vector Search] Support half-precision vector indexing 
### Does this PR have any customer impact?
No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
With the release of pgvector 0.7.0, which supports [Half-Precision Indexing](https://github.com/pgvector/pgvector?tab=readme-ov-file#half-precision-indexing), vectors can have up to 4,000 dimensions

###Related GUCs
**documentdb.enableVectorCompressionHalf**: enable vector compression(enabled by default)
- off, disable vector compression
- on, enable vector compression

###New parameters
- Create index spec: "compression": string
"half": half-precision vector, which using 16 bits float

- Search spec: "oversampling": double
Oversampling factor: Oversampling retrieval from half index to compensate for the loss of precision.
Minimum value is 1. This parameter is only permitted when a compression method is used on the vector index.
 For example, if oversampling=1.5 and k=10, then oversampling*k=15 vectors from a compressed index. Then, the full 32-bits vector will be used for the subsequent ordering and scoring,

###Mongo Command
```
db.runCommand(
{ "createIndexes": "testing_vector_collection", "indexes":
[{
     "key": { "v": "cosmosSearch" },
     "name": "hsnw_index",
     "cosmosSearchOptions":
     {
         "kind": "vector-hnsw",
         "similarity": "L2",
         "dimensions": 3,
         "compression": "half"
     }
}]
} )
```
```
db.testing_hnsw_vector_collection.aggregate(
[
  {
    "$search": {
      "cosmosSearch": {
        "vector": [ 3.0, 4.9, 1.0 ],
        "path": "vect",
        "filter": {"meta.a": [ { "b" : 3 } ]},
        "k": 4,
        "efSearch": 100,
        "oversampling": 2.0
      },
    }
  },
  {
    "$project": {
      "rank": {
        "$floor": [{"$multiply": [{ "$meta": "searchScore"},100000]}]
      }
    }
  }
])
```

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request adds support for half-precision vector indexing in the vector search functionality.
- `metadata_cache.c`: Added functions for half-precision vector similarity operators and functions.
- `SearchTests.cs`: Added integration tests to validate half-precision vector search, including error cases and index creation scenarios.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-23 12:04:26 +08:00
Zinan Ren 1d3b47c814 Merged PR 1573602: [Vector] Add "exact" search parameter to enable enn search 
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature, Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

### Introduce new field "**exact**" in vector search spec to specify whether to run ENN or ANN search.
[exact field spec](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-stage/#fields)

### Value can be one of the following, defaults to false.
- false - to run ANN search
- true - to run ENN search

### Example
`{ "k": 1, "path": "vect", "filter": {"attr": {"$eq":1}}, "exact": true, "efSearch": 40 }`
----
#### AI description  (iteration 1)
#### PR Classification
Code enhancement for filter selectivity in vector search queries.

#### PR Summary
This pull request enhances the filter selectivity mechanism in vector search queries by adding new functions and improving existing ones.
- `bson_aggregation_vector_search.c`: Added functions to calculate filter selectivity and handle vector search options, including `AddEmptySearchParamFunctionToQuery` and `GetFilterSelectivity`.
- `index_support.c`: Added logic to identify vector search queries in the planner.
- `system_configs.c`: Introduced new configuration variables for maximum filter selectivity and document count.
- `documents_planner.c`: Enhanced planner to handle vector search queries based on selectivity.
- `vector_common.h`: Declared new configuration variables for vector filter runtime sort selectivity and document count.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-02-19 17:00:35 +08:00
Vinod Sridharan b0ddee67fe Merged PR 1577306: [Vector] Iterative search take 2: Parser changes 
### Does this PR have any customer impact?
No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
To do iterative search, this is a different approach and trying to see if we can do it better to load fewer documents. This change introduces only the parser changes behind a feature switch to do a different iterative search. The remaining planner and exec changes will be done in subsequent PRs.

----
#### AI description  (iteration 1)
#### PR Classification
New feature: Parser changes for iterative search using Vector DiskANN.

#### PR Summary
This pull request introduces parser changes to support iterative search using Vector DiskANN.
- Added `bson_aggregation_pipeline_vector_diskann_planner.out` with test cases for inserting and querying documents using Vector DiskANN.
- Added `bson_aggregation_pipeline_vector_diskann_planner.sql` with SQL scripts for creating functions, inserting documents, and performing search operations using Vector DiskANN.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-02-13 17:09:16 -08:00
Vinod Sridharan 8e6210ca46 Merged PR 1574183: [Vector] Clean up dead vector code 
### Does this PR have any customer impact?
No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Clean up dead vector code - the score calculation is now embedded in the document.

----
#### AI description  (iteration 1)
#### PR Classification
Code cleanup

#### PR Summary
This pull request removes dead vector code from the project, simplifying the codebase and improving maintainability.
- Removed `CreateFCInfoForScoreCalculation` and related logic from `src/vector/vector_utilities.c`.
- Deleted `SearchQueryEvalDataWorker` struct and associated functions from `include/vector/vector_utilities.h`.
- Cleaned up vector search data handling in `src/customscan/custom_query_scan.c`.
- Removed vector search query data extraction logic from `src/opclass/index_support.c`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-02-13 17:09:16 -08:00
Zinan Ren 0ad54c20b8 Merged PR 1540839: [OSS] Remove helio references from sources of documentdb and documentdb_distributed 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description
Remove helio references from documentdb
Remove helio references from documentdb_distributed

----
#### AI description  (iteration 1)
#### PR Classification
Code cleanup

#### PR Summary
This pull request removes references to "helio" and replaces them with "documentdb" in the sources of `documentdb` and `documentdb_distributed`.
- `metadata_cache.h` and `metadata_cache.c`: Replaced "Helio" with "DocumentDB" in variable names and function names.
- `pgmongo.c`: Updated include statements and function calls to use "documentdb" instead of "helio".
- `planner.c`: Updated planner function and related calls to use "DocumentDBApiPlanner".
- Moved several header files from `helio_*` to `documentdb_*` naming convention without code changes.
- `commands_common.c` and `create_indexes.c`: Updated function calls and variable names to use "DocumentDB" instead of "Helio".
 2025-01-15 07:48:40 +00:00
Zinan Ren 8e7bd960bf Merged PR 1540262: [OSS] Remove helio references from sources of documentdb_core 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description
Remove helio from file name
Remove helio references from source content except schema reference
Start local documentdb server with the documentdb extension
Regression tests using the documentdb  extension

----
#### AI description  (iteration 1)
#### PR Classification
Code cleanup

#### PR Summary
This pull request removes references to "helio" from the `documentdb_core` source code.
- `scripts/utils.sh`: Renamed variables from `preloadExtensions` to `preloadLibraries`.
- `pg_documentdb_core/include/utils/error_utils.h`: Updated function and macro names to replace "Helio" with "DocumentDB".
- `pg_documentdb_core/include/io/helio_bson_core.h` and `pg_documentdb_core/include/query/helio_bson_compare.h`: Moved to `pg_documentdb_core/include/io/bson_core.h` and `pg_documentdb_core/include/query/bson_compare.h`.
- `pg_documentdb_core/src/collation/collation.c` and `pg_documentdb_core/src/io/pgbson.c`: Replaced error codes and messages to use "DocumentDB" instead of "Helio".
- `scripts/validate_helio_extension_state.sh`: Added exceptions for certain files to skip validation for "helio" references.
 2025-01-15 07:48:40 +00:00
Vinod Sridharan 3f9a507562 Merged PR 1539950: [OSS] Move helio_api and helio_distributed sources under documentdb 
### Does this PR have any customer impact?
No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description
Move source files under new directory.

----
#### AI description  (iteration 1)
#### PR Classification
Code reorganization

#### PR Summary
This pull request moves the `helio_api` and `helio_distributed` sources under the `documentdb` directory, updating references and initialization logic accordingly.
- Added `pg_documentdb.c` for initializing the `pg_documentdb` extension.
- Added `documentdb_distributed.c` for initializing the `pg_documentdb_distributed` extension.
- Updated `validate_helio_extension_state.sh` to include new checks for `documentdb`.
- Moved `helio_api_extension_version.c` to `documentdb_extension_version.c` without code changes.
- Updated various `Makefile` configurations to reflect the new directory structure and dependencies.
 2025-01-15 07:48:39 +00:00