### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
No
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Cleanup vector tests
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) There were order by tests not running but had the sql files in there. This was catching contracts related to arrays with null values where 2 cases were ordered later than they should be. This fixes that. As part of that we noticed that this breaks when there's paths that have array indexes (e.g. a.b.0). This also fixes that contract.
2) Add checks to say all tests must run or exist in a schedule file and fix the output
3) There's flakeyness in currentop due to tracking the version refresh query - skip the version refresh query in currentop.
----
#### AI description (iteration 1)
#### PR Classification
This pull request fixes correctness issues in composite order-by functionality and improves associated testing.
#### PR Summary
The changes refine the handling of intermediate arrays in order-by operations by updating traversal and comparison logic, while enhancing the test suite and validation scripts for composite order-by scenarios.
- `pg_documentdb/src/query/bson_dollar_operators.c` and related files were modified to correctly process nested arrays and compare composite order-by values.
- Test files in `pgmongo/src/test/regress/expected/` and `pgmongo/src/test/regress/sql/` were updated to add new order-by cases and remove obsolete tests.
- Validation scripts (e.g. in `oss/internal/pg_documentdb_distributed/src/test/regress/validate_test_output.sh` and `oss/pg_documentdb/src/test/regress/validate_test_output.sh`) were enhanced for more reliable schedule checks and collection ID extraction.
- Several BSON traversal and update operator functions were updated to include an `isArrayIndexSearch` parameter for finer control during array element processing.
- `pg_documentdb/src/commands/current_op.c` and `pg_documentdb/src/utils/version_utils.c` were updated to skip version refresh queries appropriately.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Performance
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Currently, each round of the sproc that does create indexes does exactly 1 index build and bails. This means if a single client request queues 4 indexes and the job runs every 2 seconds, it'll take 8 seconds to build all 4 indexes. This is wasteful since we know all 4 indexes up front.
Add a loop to the index build process so that multiple indexes can be built without waiting the 2 seconds.
As part of this, there were a few places allocating in memory contexts that are unstable across executions. Fix this.
Additionally there was a loop that could cause index builds to starve for ~20 mins if it picked the wrong index. Fix this loop too.
Finally, Noticed that the version cache does not cache properly if the Major version is 0. Fix this (which can cause perf regressions in DocumentDB).
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes, the API schema
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
1.05
### Description
We are adding support for the Connection Status command, which exists in native Mongo, and allows users to see their authenticated user, authenticated roles, and user privileges. For more information, visit connectionStatus.md.
This PR is the second of three PRs for this feature. The first one added extensible role privileges to our `users_info` command, allowing users to retrieve role privileges with `showPrivileges: true`. This second PR adds a callable `connection_status` command to our API, that utilizes the role privileges. The last PR will add a gateway change that calls this `connection_status` command.
Related work items: #4287246
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
Enhancement and bug fix to improve the stability and management of file-based cursors.
#### PR Summary
This pull request refactors and bolsters file-based cursor management by introducing shared memory support, improved cleanup, error handling, and configuration changes. It also adds new API endpoints and logging for better cursor lifecycle control.
- **`cursor_store.c` and related files:** Added shared memory functions (e.g., `FileCursorShmemSize`, `InitializeFileCursorShmem`), cursor count regulation (with `IncrementCursorCount`/`DecrementCursorCount`), and cleanup routines to ensure stable cursor file operations.
- **Test and expected output files:** Updated regression tests and expected outputs to cover scenarios like duplicate cursor creation, cursor deletion limits, and error reporting when the file cursor capacity is reached.
- **API enhancements:** Introduced new UDFs (e.g., `command_delete_cursors`) and tenant client methods for killing and deleting cursors, improving session management.
- **Configuration updates:** Renamed and added GUC parameters (e.g., `MaxAllowedCursorIntermediateFileSizeMB`, `MaxCursorFileCount`, and cursor expiry settings) to control file-based cursor behavior.
- **Logging and metrics improvements:** Enhanced logging in `mdsd_logger.c` to report current cursor counts and related metrics for operational insight.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
More general improvements to order by and composite:
Create Indexes
1) Support creating a composite index side by side with a regular index
2) Block creation of descending or wildcard and unsupported set ups for composite indexes
Planner:
1) Support ordering by the Nth path of an index if the prefixes are equality.
2) Validate that if it's not the first column, prefixes are provided
3) Support incremental sort with index pushdown.
----
#### AI description (iteration 1)
#### PR Classification
This pull request delivers a new feature and API enhancement to enable order-by index pushdown and refined composite index support.
#### PR Summary
The update streamlines composite index processing by introducing stricter equivalency checks, new UDF functions for full-scan order-by pushdown, and enhanced validations during index creation while updating explain outputs and tests.
- `src/metadata/index.c` refines index options equivalency logic to correctly compare composite term settings.
- `src/opclass/bson_gin_composite_entrypoint.c` and `src/index_am/rum.c` add functions to support order-by pushdown and manage duplicate entries efficiently.
- `sql/udfs/query/bson_dollar_comparison--0.105-0.sql` (and related operator files) introduce new UDFs including `bson_dollar_fullscan` to enable full scan operations.
- `src/commands/create_indexes.c` enforces new composite index validations (restricting use to ascending, non-wildcard regular indexes) with precise error reporting.
- Test scripts and explain plan outputs in the `regress` and `pgmongo` directories are updated to confirm the new composite order-by behavior.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
This pull request adds documentation for the `$bucketAuto` aggregation stage in PostgreSQL, detailing its design, usage, and implementation.
- Adds `bucketAuto.md` with details on the `$bucketAuto` aggregation stage
- code implementation
- support all granularity types: POWERSOF2, 1-2-5, Renard numbers, E series
Related work items: #3234185
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
Adding support for custom index handler.
### Description
Adding support for custom index handler. Currently the default index access method to index bson data type is "rum". This change allows one to swap out "rum" for something else.
The use an alternate index handler one needs to specify two things
1. Firstly, register the index handler and its capabilities and relevant utility call backs. Currently, the system only allows a maximum of 5 alternate index handler.
2. Secondly, set the alternate_index_handler GUC to let the system chose the alternate index handler to index bson.
Following are a few notes on the mechanism
1. During index create if the alternate AM does not support certain kind of bson index (e.g, unique index, wildcard index) then the index is created via the default rum index. Otherwise, the alternate AM is chosen.
2. If an index has been created using alternate index AM but then reset the alternate_index_handler GUC or set it to another third alternate index AM, the old alternate index AM will still be in use, and long as both the old and new alternate index AM are in the registry.
Other: Fixed Densify and SetWindow fields flakey tests for sharded collection
----
Related work items: #3796361
Yes
Feature
Yes
No
- Support compact database administration command behind the GUC `enableCompact`.
- Add support for compact command in backend using VACUUM FULL which is a blocking operation.
- Only one compact command can be run at a time on a specified collection.
- VACUUM FULL is run using libpq in a top level transaction as it it not supported inside functions or procedures.
- Add support of tracking compact command in current_op.
- `dryRun` estimation is based on the query defined [here at pgexperts](https://github.com/pgexperts/pgx_scripts/blob/master/bloat/table_bloat_check.sql), this can be a rough estimation and doesn't provided details about the space reclaimed by vacuuming RUM indexes of the collection.
- Read only users can't perform `compact` command.
----
New feature implementation to support the backend compact command.
This PR adds complete backend support for a blocking compact command that estimates table bloat and conditionally performs a VACUUM FULL operation on collection tables.
- `src/utils/storage_utils.c` and `include/utils/storage_utils.h`: Introduced a new bloat estimation utility returning collection bloat stats.
- `src/commands/compact.c`: Implemented the compact command including BSON command parsing, dry-run handling, and conditional execution of VACUUM FULL.
- `internal/pg_documentdb_distributed/src/test/regress/sql/command_compact.sql`: Added regression tests covering various compact command scenarios.
- `include/utils/feature_counter.h` and `src/infrastructure/feature_counter.c`: Updated feature counter mapping to include the new compact command feature.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4237921
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Dev
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Add more feature counters for let, cursor types and top-level commands
----
#### AI description (iteration 1)
#### PR Classification
This pull request implements a new feature by adding and tracking additional feature counters in the engine.
#### PR Summary
This change adds new feature counter entries and instrumentation to track usage for several commands and top-level "let" support. The updates include increasing the feature count limit and inserting ReportFeatureUsage calls in relevant command and aggregation pipelines.
- `src/infrastructure/feature_counter.c` and `include/utils/feature_counter.h`: Added new feature counter mappings and enums for commands (e.g., agg_cursor_first_page, count, distinct, find_cursor_first_page, get_more, list_collections_cursor_first_page, list_indexes_cursor_first_page) and top-level "let" support.
- `src/aggregation/bson_aggregation_pipeline.c`: Integrated ReportFeatureUsage calls for "let" in aggregate and find queries.
- `src/commands/aggregation_commands.c`: Inserted ReportFeatureUsage calls in various command functions to track usage of the new feature counters.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4224518, #4228942
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
more
----
#### AI description (iteration 1)
#### PR Classification
This PR introduces a new feature by adding support for tracking the internal inhibit optimization feature usage.
#### PR Summary
The changes enhance the feature counter system to monitor the `_internalInhibitOptimization` usage in aggregation pipelines and SQL tests.
- `mongo_feature_counters.sql`: Added new SQL queries to test and display the feature counter for `_internalInhibitOptimization`.
- `feature_counter.h`: Added a new enum constant, `FEATURE_STAGE_INTERNAL_INHIBIT_OPTIMIZATION`, for the new feature.
- `bson_aggregation_nested_pipeline.c`: Updated the internal optimization handler to log feature usage.
- `feature_counter.c`: Extended the feature mapping to include `_internalInhibitOptimization`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Adding the GUC documentdb.enableVectorCalculateDefaultSearchParam to enable or disable the calculation of default search parameters. The default value is on
Adding feature counts for the usage of default search parameters.
----
#### AI description (iteration 1)
#### PR Classification
This PR implements a new configuration option to enable or disable the dynamic calculation of default search parameters for vector search indexes.
#### PR Summary
The pull request adds a GUC flag that controls whether default search parameters (e.g., nProbes for IVF and efSearch for HNSW) are computed dynamically based on collection size. It updates internal calculation functions, test expectations, and feature counters accordingly.
- `src/vector/vector_index_kind_impl.c`: Revised the IVF and HNSW search parameter calculation functions to conditionally compute default parameters when the new GUC is enabled.
- `src/configs/feature_flag_configs.c` and `include/vector/vector_common.h`: Introduced the new boolean GUC `enableVectorCalculateDefaultSearchParam`.
- Test files (e.g., files under `/oss/internal/pg_documentdb_distributed/src/test/regress/expected/` and corresponding `pgmongo` tests): Updated expected outputs and query plans to reflect the new default parameter values.
- Feature counter outputs and related SQL tests: Modified to include metrics for the new default search parameter usage.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4194082
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Adding support Product Quantization (PQ) compression in vector index, which is a popular method for dramatically compressing high-dimensional vectors to use less memory, and for making nearest-neighbor search faster.
Note: currently, this PQ compression is only supported by the Diskann index.
### Related GUCs
**documentdb.enableVectorCompressionPQ**: Enables vector PQ compression (enabled by default).
### Create index spec:
- **compression**(optional):
- "half": half-precision vector, which using 16 bits floats.
- "pq": product quantization compression.
- **pqCompressedDims**(optional): indicates the dimensions after compression. If not specified, it will be automatically calculated based on the original dimensions.
- Minimum: 1
- Maximum: 8000
- **pqSampleSize**(optional):Indicates the number of samples PQ uses to generate the initial centroids. The default value is 1000.
- Minimum: 1000
- Maximum: 100000
**Note**: It is recommended to create the index after the collection has some data.. If the collection is empty, random vectors within the range [-1, 1] are used to generate the initial centroids and create the index. If the number of documents is less than pqSampleSize, the training data is padded with random data within the range [min, max] from the sampled vector data.
### Command example
```
db.runCommand(
{ "createIndexes": "testing_vector_collection", "indexes":
[{
"key": { "v": "cosmosSearch" },
"name": "hsnw_index",
"cosmosSearchOptions":
{
"kind": "vector-diskann",
"similarity": "L2",
"dimensions": 1536,
"compression": "pq",
"pqCompressedDims": 96,
"pqSampleSize": 2000
}
}]
} )
```
### Search spec:
No changes
----
#### AI description (iteration 1)
#### PR Classification
New feature: This pull request adds support for Product Quantization (PQ) compression in the Diskann index.
#### PR Summary
This pull request introduces support for PQ compression in the Diskann index, enhancing vector search capabilities. It includes error handling for unsupported compression types and validation for PQ parameters.
- Updated `/pgmongo/src/test/regress/expected/bson_aggregation_pipeline_tests_vector_diskann.out` to include tests for PQ compression, error cases, and validation of PQ parameters such as `pqCompressedDims` and `pqSampleSize`.
- Added tests for vector search with PQ compression, including L2, cosine, and IP similarity measures.
- Implemented checks for maximum dimensions and resource limits during PQ index creation.
<!-- GitOpsUserAgent=GitOps.Apps.S...
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
- Add a new section for tracking feature flags
- Add a feature counter to track customers whoc an benefit from the new rum_index_scan.
----
#### AI description (iteration 1)
#### PR Classification
Refactoring and feature enhancement with added tracking capabilities.
#### PR Summary
This pull request refactors feature flags and introduces new tracking capabilities for feature usage. It reorganizes the feature flag sections and adds new flags for various functionalities.
- Refactored and added new feature flags in `feature_flag_configs.c`, including indexing, planner, and aggregation features.
- Updated `custom_scan_continuation.c` to improve index scan handling and added feature usage reporting.
- Modified `insert.c` to handle snapshot management more effectively.
- Enhanced feature usage tracking in `feature_counter.c` and `feature_counter.h` by adding a new feature counter for index scans with limits.
- Updated test expectations in multiple files to reflect the new feature tracking changes.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Currently bulk insert run as a function which means that the entire execution runs as a single transaction. Each batch insert is handled as a subtransaction and rolled back to a save point if it fails.
This means that document locks are held for longer, and we can hit limits w.r.t. max documents updated per transaction.
In multi-node setups this also means distributed transactions are held longer causing loss of connection pools.
This change introduces a stored procedure for bulk insert that periodically "Commits" the state to WAL and starts a new transaction. This more mirrors what the non-atomic bulk update is supposed to do. The old function is retained to be used within transactions.
----
#### AI description (iteration 1)
#### PR Classification
Performance improvement for write commands.
#### PR Summary
This pull request introduces a new procedure for handling write commands, enhancing performance by optimizing user-specific command execution. It includes the addition of new functions and procedures to manage MongoDB operations more efficiently.
- Added new SQL functions and procedures in `/pgmongo/sql/udfs/gateway_funcs/command_funcs--1.103-0.sql` for running commands, queries, DDL, and write operations as a specific user.
- Introduced new insert functions and procedures in `/oss/pg_documentdb/sql/udfs/commands_crud/insert--0.103-0.sql` to handle MongoDB insert operations.
- Implemented `pgmongo_run_write_procedure_as_user` and `ExecuteWriteCommandCore` in `/pgmongo/src/infrastructure/command_funcs.c` to manage write commands with user context.
- Added `CommitOuterInsertTransactions` function in `/oss/pg_documentdb/src/commands/insert.c` to handle transaction commits outside of transaction blocks.
- Introduced a feature flag `EnableWriteProcedure` in `/oss/pg_documentdb/src/configs/feature_flag_configs.c` to control the enabling of write procedures.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Currently bulk updates run as a function which means that the entire execution runs as a single transaction. Each batch update is handled as a subtransaction and rolled back to a save point if it fails.
This means that document locks are held for longer, and we can hit limits w.r.t. max documents updated per transaction.
In multi-node setups this also means distributed transactions are held longer causing loss of connection pools.
This change introduces a stored procedure for bulk update that periodically "Commits" the state to WAL and starts a new transaction. This more mirrors what the non-atomic bulk update is supposed to do. The old function is retained to be used within transactions.
Still TODO:
For unsharded collections in a multi-node setup, we need to send batches of updates to the worker. This is not wired up in this change.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces a new bulk update feature for handling non-transactional updates in the MongoDB wire-protocol. It adds new functions and procedures to support bulk updates and modifies existing update logic to accommodate this new functionality.
- Added `update_bulk` procedure and related functions in `sql/udfs/commands_crud/update--0.103-0.sql` for non-transactional bulk updates.
- Implemented `command_update_bulk` in `src/commands/update.c` to handle bulk updates outside of transactions.
- Enhanced `ProcessBatchUpdateCore` in `src/commands/update.c` to support transactional and non-transactional updates.
- Updated feature tracking in `include/utils/feature_counter.h` and `src/infrastructure/feature_counter.c` to include `FEATURE_COMMAND_UPDATE_BULK`.
- Modified SQL scripts `documentdb--0.102-1--0.103-0.sql` and `pgmongo--1.102-1--1.103-0.sql` to include the new update functionality.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
With the release of pgvector 0.7.0, which supports [Half-Precision Indexing](https://github.com/pgvector/pgvector?tab=readme-ov-file#half-precision-indexing), vectors can have up to 4,000 dimensions
###Related GUCs
**documentdb.enableVectorCompressionHalf**: enable vector compression(enabled by default)
- off, disable vector compression
- on, enable vector compression
###New parameters
- Create index spec: "compression": string
"half": half-precision vector, which using 16 bits float
- Search spec: "oversampling": double
Oversampling factor: Oversampling retrieval from half index to compensate for the loss of precision.
Minimum value is 1. This parameter is only permitted when a compression method is used on the vector index.
For example, if oversampling=1.5 and k=10, then oversampling*k=15 vectors from a compressed index. Then, the full 32-bits vector will be used for the subsequent ordering and scoring,
###Mongo Command
```
db.runCommand(
{ "createIndexes": "testing_vector_collection", "indexes":
[{
"key": { "v": "cosmosSearch" },
"name": "hsnw_index",
"cosmosSearchOptions":
{
"kind": "vector-hnsw",
"similarity": "L2",
"dimensions": 3,
"compression": "half"
}
}]
} )
```
```
db.testing_hnsw_vector_collection.aggregate(
[
{
"$search": {
"cosmosSearch": {
"vector": [ 3.0, 4.9, 1.0 ],
"path": "vect",
"filter": {"meta.a": [ { "b" : 3 } ]},
"k": 4,
"efSearch": 100,
"oversampling": 2.0
},
}
},
{
"$project": {
"rank": {
"$floor": [{"$multiply": [{ "$meta": "searchScore"},100000]}]
}
}
}
])
```
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for half-precision vector indexing in the vector search functionality.
- `metadata_cache.c`: Added functions for half-precision vector similarity operators and functions.
- `SearchTests.cs`: Added integration tests to validate half-precision vector search, including error cases and index creation scenarios.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces CRUD operations for Entra users in the DataPlane.
- `authentication.c`: Added functions to create users with external identity providers, validate identity provider info, and check if the current user is an Entra admin.
- `users.h`: Added new header file defining user CRUD functions and related structures.
- `users.c`: Updated user creation logic to handle external identity providers and added parsing for custom data.
- `api_hooks.c`: Added hooks for creating users with external identity providers and checking user privileges.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Altering creation_time column removing not null and default constraint.
----
#### AI description (iteration 1)
#### PR Classification
Performance improvement
#### PR Summary
This pull request alters the `creation_time` column in the database schema to improve performance.
- Added `AlterCreationTime` function in `cluster_operations.c` to modify the `creation_time` column.
- Updated `SetupCluster` function in `cluster_operations.c` to call `AlterCreationTime`.
- Created new SQL scripts `extension_upgrade_alter_creation_time_1.102-0.sql` and `extension_upgrade_alter_creation_time_1.102-0.out` for testing the changes.
- Modified multiple test output files to reflect the changes in the `creation_time` column constraints.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
support $toUUID operator
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces the `$toUUID` operator for MongoDB 8.0, enabling conversion of strings to UUIDs.
- Added `ProcessDollarToUUID` function in `bson_expression_type_operators.c` to handle the conversion logic.
- Updated `bson_expression_operators.h` to declare new parsing and handling functions for `$toUUID`.
- Included new test cases in `bson_aggregation_type_operators_tests.sql` and `bson_aggregation_type_operators_tests.out` to validate the `$toUUID` functionality.
- Modified `feature_counter.h` and `feature_counter.c` to include the new `$toUUID` feature in the feature count and mapping.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature counter
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
No
### Description
Adding feature counter for sort by id to make decisions about sort by id pushdown to primary key optimization.
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature, Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
### Introduce new field "**exact**" in vector search spec to specify whether to run ENN or ANN search.
[exact field spec](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-stage/#fields)
### Value can be one of the following, defaults to false.
- false - to run ANN search
- true - to run ENN search
### Example
`{ "k": 1, "path": "vect", "filter": {"attr": {"$eq":1}}, "exact": true, "efSearch": 40 }`
----
#### AI description (iteration 1)
#### PR Classification
Code enhancement for filter selectivity in vector search queries.
#### PR Summary
This pull request enhances the filter selectivity mechanism in vector search queries by adding new functions and improving existing ones.
- `bson_aggregation_vector_search.c`: Added functions to calculate filter selectivity and handle vector search options, including `AddEmptySearchParamFunctionToQuery` and `GetFilterSelectivity`.
- `index_support.c`: Added logic to identify vector search queries in the planner.
- `system_configs.c`: Introduced new configuration variables for maximum filter selectivity and document count.
- `documents_planner.c`: Enhanced planner to handle vector search queries based on selectivity.
- `vector_common.h`: Declared new configuration variables for vector filter runtime sort selectivity and document count.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
draft
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces new aggregation functions and support for t-digest in the project.
- Added `bson_tdigest.c` to implement t-digest helper functions.
- Created new SQL aggregate functions in `group_aggregates--1.23-0.sql` and `group_aggregates_support--1.23-0.sql`.
- Updated `bson_aggregation_pipeline.c` to handle the new `$percentile` accumulator.
- Modified `metadata_cache.c` and `metadata_cache.h` to include OID for the new `BsonPercentileAggregateFunction`.
- Updated migration scripts `pg_helio_api--1.22-0--1.23-0.sql` and `pgmongo--1.22-0--1.23-0.sql` to include new SQL files.
### Does this PR have any customer impact?
yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes, UDF
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Implement for stage redact
Related work items: #1798586
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
To enable top level query operators working in schema validation.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces support for the `$jsonSchema` query operator for schema validation.
- Added handling for `QUERY_OPERATOR_JSONSCHEMA` in `query_operator.c` and `mongo_query_operator.c`.
- Implemented `GetExpressionEvalStateForBsonInput` in `bson_expr_eval.c`.
- Added `BsonJsonSchemaFunctionId` and related OID cache in `metadata_cache.c`.
- Modified `PrepareForSchemaValidation` in `schema_validation.c` to use the new evaluation state function.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Move DocumentDB to schema version 0. To do this version checks now need to handle internal and external versions being different.
Tackle this in version_utils.c
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for DocumentDB on version 0.
- Modified `cluster_operations.c` to use `DocDB_V0` for version checks.
- Updated `version_utils.h` and `version_utils.c` to define and use `MajorVersion` enum for version handling.
- Adjusted various files in the `aggregation` directory to use `DocDB_V0` for version checks.
- Updated `create_indexes.c` to use `DocDB_V0` for version checks.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Remove helio from file name
Remove helio references from source content except schema reference
Start local documentdb server with the documentdb extension
Regression tests using the documentdb extension
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request removes references to "helio" from the `documentdb_core` source code.
- `scripts/utils.sh`: Renamed variables from `preloadExtensions` to `preloadLibraries`.
- `pg_documentdb_core/include/utils/error_utils.h`: Updated function and macro names to replace "Helio" with "DocumentDB".
- `pg_documentdb_core/include/io/helio_bson_core.h` and `pg_documentdb_core/include/query/helio_bson_compare.h`: Moved to `pg_documentdb_core/include/io/bson_core.h` and `pg_documentdb_core/include/query/bson_compare.h`.
- `pg_documentdb_core/src/collation/collation.c` and `pg_documentdb_core/src/io/pgbson.c`: Replaced error codes and messages to use "DocumentDB" instead of "Helio".
- `scripts/validate_helio_extension_state.sh`: Added exceptions for certain files to skip validation for "helio" references.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Move source files under new directory.
----
#### AI description (iteration 1)
#### PR Classification
Code reorganization
#### PR Summary
This pull request moves the `helio_api` and `helio_distributed` sources under the `documentdb` directory, updating references and initialization logic accordingly.
- Added `pg_documentdb.c` for initializing the `pg_documentdb` extension.
- Added `documentdb_distributed.c` for initializing the `pg_documentdb_distributed` extension.
- Updated `validate_helio_extension_state.sh` to include new checks for `documentdb`.
- Moved `helio_api_extension_version.c` to `documentdb_extension_version.c` without code changes.
- Updated various `Makefile` configurations to reflect the new directory structure and dependencies.
Zinan Ren
Vinod Sridharan
William Wang
Shuai Tian
Jayanta Mondal
Deepak Bisht
Cornelius Boateng
Parag Jain
Deepak Pemmaraju
Xing Fan
Marco Slot
Jia Hu