Yes
Feature
Yes
No
- Support compact database administration command behind the GUC `enableCompact`.
- Add support for compact command in backend using VACUUM FULL which is a blocking operation.
- Only one compact command can be run at a time on a specified collection.
- VACUUM FULL is run using libpq in a top level transaction as it it not supported inside functions or procedures.
- Add support of tracking compact command in current_op.
- `dryRun` estimation is based on the query defined [here at pgexperts](https://github.com/pgexperts/pgx_scripts/blob/master/bloat/table_bloat_check.sql), this can be a rough estimation and doesn't provided details about the space reclaimed by vacuuming RUM indexes of the collection.
- Read only users can't perform `compact` command.
----
New feature implementation to support the backend compact command.
This PR adds complete backend support for a blocking compact command that estimates table bloat and conditionally performs a VACUUM FULL operation on collection tables.
- `src/utils/storage_utils.c` and `include/utils/storage_utils.h`: Introduced a new bloat estimation utility returning collection bloat stats.
- `src/commands/compact.c`: Implemented the compact command including BSON command parsing, dry-run handling, and conditional execution of VACUUM FULL.
- `internal/pg_documentdb_distributed/src/test/regress/sql/command_compact.sql`: Added regression tests covering various compact command scenarios.
- `include/utils/feature_counter.h` and `src/infrastructure/feature_counter.c`: Updated feature counter mapping to include the new compact command feature.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4237921
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
PoC/Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Today, persisted cursors require the connections to be held. This is bad since Connections/Backends are a limited resource.
This introduces the option to save the cursor state in a file on disk and reuse that across backends.
This is behind a GUC due to the presence of the following:
1) Needs more testing and validation
2) Needs backpressure to ensure that disk consumption doesn't balloon up
3) There is a procedure added to clean up these files but it is not wired up
4) We need to handle ultra-large responses (beyond 4 GB it fails outright at the moment). This needs more graceful handling to split into chunks.
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a new feature implementation that introduces file‐based persisted cursors to allow query results to be “drained” and stored on disk without holding on to active database connections.
#### PR Summary
This PR adds support for persisting cursor state to the filesystem in both the pg_documentdb and pgmongo modules, enabling connections to be released while still allowing cursors to be later resumed. The changes include:
- **`infrastructure/cursor_store.c` and `cursor_store.h`** – New APIs for creating, writing to, reading from, and closing persisted cursor files.
- **Commands integration** – Modifications in `commands/cursors.c`, `commands/aggregation_commands.c`, and `cursor_private.h` to use file-based cursor functions for draining and continuation.
- **SQL test suites** – Addition of new test files (e.g. in `.../regress/sql/bson_aggregation_cursor_tests_file.sql`) to validate cursor behavior across various query scenarios.
- **Configuration updates** – New custom GUCs in `feature_flag_configs.c` and adjustments in test PostgreSQL configuration files to enable and control file-based persisted cursors.
- **Executor enhancements** – Updates in cursor draining and query plan execution functions to support both in-memory and file-based cursor paging.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes, yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support sorting with collation.
We define new custom sort operators which compare operands using applicable collation string.
`<<<` is for ascending order.
`>>>` is for descending order.
Example:
`SELECT * from <collection> ORDER BY <value> USING <<<`
This will further be used for partitioning, grouping, and related comparisons.
----
#### AI description (iteration 1)
#### PR Classification
New feature: Support for collation in sort operations.
#### PR Summary
This pull request adds support for collation in sort operations within the database engine.
- `src/test/regress/expected/bson_dollar_ops_collation_tests_runtime.out`: Added multiple test cases to verify collation behavior with various sort and order by operations, including case sensitivity and numeric ordering.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4014123
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
GW plumbing of compact command
1- Route the compact command to the backend.
2- Add a dummy C handler function for `compact` that does nothing and throws the same error.
Next PR is going to add support for compact in the backend with blocking writes by running VACUUM FULL
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for the `compact` command in the MongoDB Postgres runtime.
- Added `CompactAsync` method in `PostgresClient.cs` to handle the compact command.
- Implemented `command_compact` function in the new file `compact.c`.
- Updated `PostgresMongoClient.cs` to include the `CompactAsync` method.
- Modified `MongoRequestParser.cs` to parse the compact command.
- Added SQL function definitions for the compact command in `compact--0.103-0.sql` and `compact--latest.sql`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
BugFix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
Bugfix to override nested arrays in dotted path in `bson_dollar_merge_documents` UDF for few cases
### Description
bson_dollar_merge_documents function today recurses into nested arrays always and project in each element of array but that is not correct with mongo semantics.
This is only applicable if the projectetion is a dotted path.
The function is used in 7 different stages/features today:
1- list_collection: no dotted paths for projection, so this is okay to follow any projection.
2- graphlookup: Mongo does a shallow projection in this case.
3- lookup: Mongo does a shallow projection in this case.
4- inverseMatch: okay as no dotted path projection
5- count: okay as no dotted path projection
6- group : okay as no dotted paths are allowed
7- setWindowFields: only this stage does a full projection even inside nested arrays
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request addresses multiple scenarios for fixing projection with `bson_dollar_merge_documents`.
- Added new functions in `oss/pg_documentdb/sql/udfs/projection/bson_projection--0.102-0.sql` to handle various BSON document operations.
- Updated `bson_dollar_merge_documents` function to include an `overrideArray` parameter.
- Modified test expectations in `oss/internal/pg_documentdb_distributed/src/test/regress/expected/bson_aggregation_pipeline_tests_graphlookup.out` to reflect changes in the `bson_dollar_merge_documents` function.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Get background index builds to work for single-node documentdb.
This changeset fixes background index builds for OSS by:
1. Scheduling the background index build tasks
2. Moving the distributed functionality (which had not being refactored out of the indexing code during the move to OSS) into distributed hooks, with the base implementation for single-node.
3. Refactoring SQL tests to get even with the above changes.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix for background index builds in DocumentDB.
#### PR Summary
This pull request addresses issues with background index builds in DocumentDB by adding comprehensive tests and handling various edge cases.
- `commands_create_indexes_background.out`: Added extensive test cases to validate the behavior of `create_indexes_background` function, including handling of invalid inputs, unique constraints, and various index specifications.
- Introduced a new schema and function to manage index job statuses.
- Enhanced error handling and validation for index creation requests, ensuring proper error messages and statuses are returned for invalid operations.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
- add cursor result type for UDFs, this helps PG to not always try to figure out the `OUT` variables and form a tuple descriptor
----
#### AI description (iteration 1)
#### PR Classification
Performance improvement and API change.
#### PR Summary
This pull request introduces performance improvements for point reads by adding a new cursor type for SQL functions. It modifies existing functions to return a structured type instead of a record, enhancing efficiency and clarity.
- `query_cursors_aggregate--latest.sql`: Added `cursor_result` type and updated several functions to return this type instead of a record.
- `query_cursors_aggregate--0.103-0.sql`: Newly added file with similar changes as `query_cursors_aggregate--latest.sql`.
- `documentdb--0.102-1--0.103-0.sql` and `pgmongo--1.102-1--1.103-0.sql`: Updated to include the new `query_cursors_aggregate--0.103-0.sql` file.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4157027
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
Version update for the release of version 103.
#### PR Summary
This pull request updates the version numbers across several control files and adds release notes for version 1.103 and 1.104. It also includes changes to SQL files and their corresponding SHA1 checksums.
- `/oss/internal/pg_documentdb_distributed/documentdb_distributed.control`, `/oss/pg_documentdb/documentdb.control`, `/oss/pg_documentdb_core/documentdb_core.control`, and `/pgmongo/pgmongo.control`: Updated default version numbers.
- `/docs/release_notes/pgmongo_1.103.md` and `/docs/release_notes/pgmongo_1.104.md`: Added release notes for versions 1.103 and 1.104.
- `/pgmongo/sql/pgmongo--1.103-1--1.104-1.sql`: Added new SQL file for version transition.
- `/docs/release_notes/toc.yml`: Updated table of contents to include new release notes.
- `/oss/CHANGELOG.md`: Added entry for documentdb v0.104-0.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4140262
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Currently bulk insert run as a function which means that the entire execution runs as a single transaction. Each batch insert is handled as a subtransaction and rolled back to a save point if it fails.
This means that document locks are held for longer, and we can hit limits w.r.t. max documents updated per transaction.
In multi-node setups this also means distributed transactions are held longer causing loss of connection pools.
This change introduces a stored procedure for bulk insert that periodically "Commits" the state to WAL and starts a new transaction. This more mirrors what the non-atomic bulk update is supposed to do. The old function is retained to be used within transactions.
----
#### AI description (iteration 1)
#### PR Classification
Performance improvement for write commands.
#### PR Summary
This pull request introduces a new procedure for handling write commands, enhancing performance by optimizing user-specific command execution. It includes the addition of new functions and procedures to manage MongoDB operations more efficiently.
- Added new SQL functions and procedures in `/pgmongo/sql/udfs/gateway_funcs/command_funcs--1.103-0.sql` for running commands, queries, DDL, and write operations as a specific user.
- Introduced new insert functions and procedures in `/oss/pg_documentdb/sql/udfs/commands_crud/insert--0.103-0.sql` to handle MongoDB insert operations.
- Implemented `pgmongo_run_write_procedure_as_user` and `ExecuteWriteCommandCore` in `/pgmongo/src/infrastructure/command_funcs.c` to manage write commands with user context.
- Added `CommitOuterInsertTransactions` function in `/oss/pg_documentdb/src/commands/insert.c` to handle transaction commits outside of transaction blocks.
- Introduced a feature flag `EnableWriteProcedure` in `/oss/pg_documentdb/src/configs/feature_flag_configs.c` to control the enabling of write procedures.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Add a new bson_expression_get that appends the collationString to the got value, if a valid collationString is passed to it.
This will be used as a helper function to operations that rely on bson_expression_get to pass down collation such as bson_lookup_join_filter.
If no collation string is passed as an arg, we simply return the got value.
----
#### AI description (iteration 1)
#### PR Classification
New feature to support `$graphLookup` in the engine with collation.
#### PR Summary
This pull request adds support for `$graphLookup` with collation in the engine.
- Added new functions in `oss/pg_documentdb/sql/udfs/aggregation/bson_lookup_functions--0.103-0.sql` for `$graphLookup` operations.
- Modified `oss/pg_documentdb/src/aggregation/bson_aggregation_nested_pipeline.c` to handle collation in `$graphLookup` join filters.
- Updated `oss/pg_documentdb/src/operators/bson_expression.c` to remove collation handling from `bson_expression_get`.
- Added `BsonDollarLookupJoinFilterWithCollationFunctionOid` in `oss/pg_documentdb/src/metadata/metadata_cache.c`.
- Adjusted collation handling in `oss/pg_documentdb/src/query/bson_dollar_operators.c` for `$graphLookup`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4104899
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Currently bulk updates run as a function which means that the entire execution runs as a single transaction. Each batch update is handled as a subtransaction and rolled back to a save point if it fails.
This means that document locks are held for longer, and we can hit limits w.r.t. max documents updated per transaction.
In multi-node setups this also means distributed transactions are held longer causing loss of connection pools.
This change introduces a stored procedure for bulk update that periodically "Commits" the state to WAL and starts a new transaction. This more mirrors what the non-atomic bulk update is supposed to do. The old function is retained to be used within transactions.
Still TODO:
For unsharded collections in a multi-node setup, we need to send batches of updates to the worker. This is not wired up in this change.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces a new bulk update feature for handling non-transactional updates in the MongoDB wire-protocol. It adds new functions and procedures to support bulk updates and modifies existing update logic to accommodate this new functionality.
- Added `update_bulk` procedure and related functions in `sql/udfs/commands_crud/update--0.103-0.sql` for non-transactional bulk updates.
- Implemented `command_update_bulk` in `src/commands/update.c` to handle bulk updates outside of transactions.
- Enhanced `ProcessBatchUpdateCore` in `src/commands/update.c` to support transactional and non-transactional updates.
- Updated feature tracking in `include/utils/feature_counter.h` and `src/infrastructure/feature_counter.c` to include `FEATURE_COMMAND_UPDATE_BULK`.
- Modified SQL scripts `documentdb--0.102-1--0.103-0.sql` and `pgmongo--1.102-1--1.103-0.sql` to include the new update functionality.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Fix up the remaining operators - implementations are sub-optimal
2) Add type bracketing options for all types and use them in operators
3) Add support for truncation of terms
TODO:
- Support recheck on the index for operators than just bounds
- Support N boundaries for index terms.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request addresses issues with operators and type bracketing in BSON composite indexes.
- Added `bson_gin_index_types_core.c` to manage BSON type bounds for indexes.
- Updated `bson_gin_composite_entrypoint.c` to fix operator handling and apply type bracketing for various BSON index strategies.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
As part of testing out the composite operator classes, introduce a new composite operator class for Bson. This will be behind a test only GUC and will be for testing composite index pushdown.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces a new composite operator class for BSON GIN indexes in PostgreSQL.
- Added `bson_gin_composite_entrypoint.c` to implement GIN operator functions for BSON composite indexes.
- Created test cases in `bson_composite_index_tests_insert_terms.out` to validate term generation for composite paths.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix(ish)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
0.102-0 had changes that are non-idempotent. Given there were some incorrect dependencies taken on the 0.102 release (independently) introduce a 0.102-1 version that applies all the delta of 0.102-0 in an idempotent manner so that dependencies taken partially on 0.102 and dependencies taken on 0.102 fully work side-by-side.
----
#### AI description (iteration 1)
#### PR Classification
Hotfix for DocumentDB version upgrade.
#### PR Summary
This pull request introduces a hotfix for DocumentDB, ensuring an idempotent upgrade from version 0.102-0 to 0.102-1.
- Added `documentdb--0.102-0--0.102-1.sql` to handle partial upgrades and repeat UDF upgrades.
- Modified `data_table_utils.c` to check for both 0.102-0 and 0.102-1 versions during data table upgrades.
- Updated `pgmongo.control` to set the default version to 1.102-1.
- Added `pgmongo--1.102-1--1.103-0.sql` for version transition.
- Updated SHA1 checksums in multiple `pgmongo/sql/sha1` files to reflect changes.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Version update for 102 release
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support for pushdown of to PFE indexes
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for pushdown operations to PFE indexes, enhancing query performance and index utilization.
- Added new functions in `sql/udfs/index_mgmt/bson_indexbounds_functions--0.102-0.sql` for BSON index bounds operations.
- Introduced new operators in `sql/operators/bson_btree_pfe_operators--0.102-0.sql` for BSON and BSON query comparisons.
- Updated `src/query/query_operator.c` to handle new index bounds operators and expressions.
- Added new type definitions in `sql/types/bsonindexbounds--0.102-0.sql` for BSON index bounds.
- Modified `src/metadata/metadata_cache.c` to include OIDs and functions for BSON index bounds types and operators.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Altering creation_time column removing not null and default constraint.
----
#### AI description (iteration 1)
#### PR Classification
Performance improvement
#### PR Summary
This pull request alters the `creation_time` column in the database schema to improve performance.
- Added `AlterCreationTime` function in `cluster_operations.c` to modify the `creation_time` column.
- Updated `SetupCluster` function in `cluster_operations.c` to call `AlterCreationTime`.
- Created new SQL scripts `extension_upgrade_alter_creation_time_1.102-0.sql` and `extension_upgrade_alter_creation_time_1.102-0.out` for testing the changes.
- Modified multiple test output files to reflect the changes in the `creation_time` column constraints.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Remove all Citus references from pg_documentdb tests
2) Remove some citus references from documentdb c files (Comments)
3) Remove dead code for bson_out aggregates
4) Remove all References to IsClusterVersionAtLeast less than 0.24 (since those are now deployed)
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request introduces code cleanups and enhancements related to aggregation functions in DocumentDB.
- `oss/pg_documentdb/sql/udfs/aggregation/group_aggregates--0.102-0.sql`: Added new aggregation functions for BSON data types, including BSONSUM, BSONAVERAGE, BSONMAX, BSONMIN, BSONFIRST, BSONLAST, BSONFIRSTN, BSONLASTN, BSON_ARRAY_AGG, BSON_OBJECT_AGG, BSON_ADD_TO_SET, BSON_MERGE_OBJECTS, BSONSTDDEVPOP, BSONSTDDEVSAMP, BSONPERCENTILE, and BSONMEDIAN.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support collation with `find` queries.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for the `project find` functionality with collation in the engine.
- Added new SQL functions in `sql/udfs/projection/bson_projection--0.102-0.sql` for handling BSON projection with collation.
- Updated `metadata_cache.c` to include OID caching for the new `bson_dollar_project_find` function with collation.
- Modified `bson_project.c` to handle additional arguments for collation in the `bson_dollar_project_find` function.
- Updated `bson_projection--latest.sql` to include the new function definition.
- Added function declaration in `metadata_cache.h` for the new `bson_dollar_project_find` function with collation.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #3924476
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support collation for `replaceRoot` aggregation stage
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for the `$replaceRoot` operation with collation in the BSON aggregation pipeline.
- `bson_projection--0.102-0.sql` and `bson_projection--latest.sql`: Added new function `bson_dollar_replace_root` with collation support.
- `metadata_cache.c` and `metadata_cache.h`: Added OID cache and retrieval function for the new `bson_dollar_replace_root` with collation.
- `bson_project.c`: Updated `bson_dollar_replace_root` to handle collation.
- `bson_aggregation_pipeline.c`: Integrated collation handling in `HandleReplaceRoot`.
- Updated test files to include new test cases for `$replaceRoot` with collation.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support collation for `$addFields` and `$set` aggregation stages
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for the `addFields` operation with collation in the aggregation pipeline.
- `bson_project.c`: Added collation support to `bson_dollar_add_fields` and related functions.
- `bson_expression_comparison_operators.c`: Updated comparison operators to handle collation.
- `bson_dollar_operators.c`: Enhanced query state population to include collation.
- Test files `bson_dollar_ops_collation_tests_runtime.out` and `bson_dollar_ops_collation_tests_runtime.sql`: Added tests for collation with `$expr` and `$addFields`.
- `bson_projection--latest.sql`: Added new function definition for `bson_dollar_add_fields` with collation.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #3919349, #3919367
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
more
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for collation in BSON projection functions.
- Added new SQL functions in `sql/udfs/projection/bson_projection--0.102-0.sql` to handle collation.
- Updated `src/aggregation/bson_project.c` to include collation string handling in BSON projection.
- Modified `src/aggregation/bson_aggregation_pipeline.c` to support collation in projection stages.
- Enhanced `src/query/bson_dollar_operators.c` to populate collation string in query state.
- Added collation-related tests in `sql/bson_dollar_ops_collation_tests_runtime.sql`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #3918941
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
more
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for collation in the redact function.
- `bson_project.c`: Added handling for an additional collation string argument in the `bson_dollar_redact` function.
- `bson_aggregation_redact--0.24-0.sql` and `bson_aggregation_redact--latest.sql`: Updated to include the new collation string parameter in the `bson_dollar_redact` function definition.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #3919400
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
[schema validation] apply to stage $merge/$out
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces schema validation for the `$merge` and `$out` stages in MongoDB aggregation pipelines.
- Added `schema_validation_merge.md` to document the design and implementation details.
- Created new SQL functions in `bson_merge_functions--1.24-0.sql` to handle schema validation during the `$merge` stage.
- Updated `bson_aggregation_output_pipeline.c` to integrate schema validation logic into the `$merge` and `$out` stages.
- Modified `toc.yml` to include the new schema validation documentation.
- Updated `pgmongo--1.23-0--1.24-0.sql` and `pg_helio_api--1.23-0--1.24-0.sql` to include the new SQL functions.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Change signature of `bson_query_match` with `let` and `collation` support to match other internal functions.
Fix schema name for pgmongo.
----
#### AI description (iteration 1)
#### PR Classification
New feature: Collation support for query match in unsharded collections.
#### PR Summary
This pull request adds collation support for query match operations in unsharded collections, ensuring proper collation enforcement.
- Modified `bson_query_match` function calls in `/pgmongo/src/test/docdb_compat/expected/bson_dollar_ops_collation_tests_runtime.out` to adjust collation parameters.
- Updated collation parameters in `bson_query_match` function calls in `/oss/internal/pg_documentdb_distributed/src/test/regress/expected/bson_dollar_ops_collation_tests_runtime.out`.
- Adjusted SQL test cases in `/oss/internal/pg_documentdb_distributed/src/test/regress/sql/bson_dollar_ops_collation_tests_runtime.sql` to reflect new collation support.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #3914004
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes; yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support `collation` with query operator using an internal UDF:
```
bson_query_match(document bson, query bson, collationString text, variableSpec bson)
```
Enable feature with `EnableCollationAndLetForQueryMatch` GUC.
Collation on sharded collections with collation-aware shard keys not supported yet.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds collation support for query operators in the engine.
- `query_operator.c`: Added logic to handle collation strings in BSON query operators and expanded function to support collation and variable specifications.
- `metadata_cache.c`: Introduced a new function to get the OID of `bson_query_match` with collation and let arguments.
- `feature_flag_configs.c`: Added a new feature flag to enable collation and let for write commands.
- `bson_query_match--0.102.sql` and `bson_query_match--latest.sql`: Added new SQL functions to support collation in BSON queries.
- `documents_planner.c`: Updated planner to recognize and handle the new collation and let functionality in queries.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Features
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
- Add support for a top level current_op command that calls the aggregation function and forms a command response
- Add support for joining between regular tables and documentdb tables. In this mode, we'll still require a query to be exclusively for find/aggregate etc, but we can put those queries in a CTE/Subquery and join with other find/aggregate/postgres queries and apply regular postgres join semantics.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces new features for handling `current_op` and joining with DocumentDB tables, including support for nested aggregation functions.
- Added `BuildAggregateSpecFromCommandSpec` and `command_current_op` functions in `src/commands/current_op.c` to handle new aggregation specifications.
- Implemented `MutateQueryAggregatorFunction` and `ExpandNestedAggregationFunction` in `src/planner/documents_planner.c` to support nested aggregation queries.
- Added new test cases in `src/test/regress/sql/bson_aggregation_functions_nested_tests.sql` and `src/test/regress/expected/bson_aggregation_functions_nested_tests.out`.
- Introduced feature flag `allowNestedAggregationFunctionInQueries` in `src/configs/feature_flag_configs.c` to control nested aggregation functionality.
- Created new UDFs in `sql/udfs/commands_diagnostic/current_op--0.102-0.sql` for the `currentOp` aggregation stage.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
[OSS] Fix TTL indexes in documentdb
This moves the function to C and instead of querying the indexes and then getting the collection shards from pg_dist_shard table, it uses the C hook to get the table shards where for non-distributed it returns the base table and then uses that information to drop the expired records.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request addresses issues with TTL indexes in DocumentDB by refactoring and optimizing the deletion of expired rows.
- `ttl_index.c`: Refactored the function to delete expired rows, added a new function `delete_expired_rows`, and optimized the deletion process.
- `ttl_support_functions--latest.sql` and `delete_expired_rows--latest.sql`: Replaced PL/pgSQL procedures with C language implementations for better performance.
- `collection.c`: Moved the function `GetMongoCollectionShardOidsAndNames` from `coll_stats.c` to `collection.c` for better organization.
### Does this PR have any customer impact?
yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
yes, creating new UDFs:
- `mongo_api_internal.run_explain_as_user`
- `list_databases`
### Are you introducing any new config? If yes, do you have tests with and without them being set?
yes, gating the explain experience behind the `pgmongo.enableExplainsInSharedUserPool`
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Port the rest of the DML commands and explains to the backend.
Update all of the test coverage for explains for the new changes too.
Move ListDatabases command from Gateway to backend
Related work items: #3759098
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Release
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Bump release
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Description
Updating schema to release 1.101
----
#### AI description (iteration 1)
#### PR Classification
Schema update for new release version.
#### PR Summary
This pull request updates the schema to release version 1.101, including version bumps and minor SQL adjustments.
- Updated `default_version` in `documentdb_distributed.control`, `documentdb.control`, and `documentdb_core.control` to `0.102-0`.
- Updated `default_version` in `pgmongo.control` to `1.101-0`.
- Added new SQL file `pgmongo--1.101-0--1.102-0.sql` with search path settings.
- Updated SHA1 checksums for various `pgmongo` SQL files.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
BugFix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Because of N-1 tests we can't update the sql UDF and start using it in documentdb at the same release. This 1 of 2 PRs to add the schema changes in release 0.101 and next PR will start using these UDF in documentdb in 0.102
Bug description: `bson_dollar_merge_documents` UDF today doesn't override nested arrays for projection with conflicting path and add the new projection entities inside each element of array which is not required in few places
e.g. $lookup, $graphlookup etc
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces a new function to handle mutating nested arrays in BSON documents.
- Added `bson_dollar_merge_documents` function with an override flag in `bson_projection--0.101-0.sql`.
- Implemented `MergeDocumentWithArrayOverride` function in `bson_project.c` to support the new feature.
- Updated test cases in `bson_aggregation_stage_lookup_tests.sql` and corresponding expected output files.
- Included the new SQL file in `documentdb--0.100-0--0.101-0.sql` and `pgmongo--1.24-0--1.101-0.sql`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Enable support for current_op aggregation stage in OSS.
Still pending: support for getting locking pids and index builds.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request enables support for the `current_op` aggregation stage in DocumentDB.
- Added new functions in `current_op--0.101-0.sql` and `current_op--latest.sql` to support the `current_op` aggregation stage.
- Modified `current_op.c` to handle new attributes and improve the `current_op` command processing.
- Updated `system_configs.c` to include a new configuration variable for tracking the application name in `current_op`.
- Adjusted test cases in `bson_aggregation_pipeline_tests_coll_agnostic.sql` and related expected output files to validate the new feature.
- Updated `metadata_cache.c` to correctly reference the `current_op_aggregation` function.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Schema
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Snap 0.101 branch for OSS, 0.100 is already released and can no longer accept changes in schema
----
#### AI description (iteration 1)
#### PR Classification
Version update.
#### PR Summary
This pull request updates the version number for the DocumentDB API and core API surface for PostgreSQL.
- `oss/pg_documentdb/documentdb.control`: Updated `default_version` to '0.101-0'.
- `oss/pg_documentdb_core/documentdb_core.control`: Updated `default_version` to '0.101-0'.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Docs
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
This PR includes updates to add base Doxygen comments within SQL files, setting up initial documentation structure for functions. Additionally, it includes Doxygen configuration files (Doxyfile, etc.) tailored to support SQL file parsing and documentation generation.
----
#### AI description (iteration 1)
#### PR Classification
Documentation
#### PR Summary
This pull request adds Doxygen comments to various `helio_api` functions to improve code documentation.
- Added Doxygen comments to functions in `sql/udfs/commands_crud/query_cursors_single_page--latest.sql`
- Added Doxygen comments to functions in `sql/udfs/utils/extension_version_utils--latest.sql`
- Added Doxygen comments to functions in `sql/udfs/auth/auth_scram--latest.sql`
- Added Doxygen comments to functions in `sql/udfs/commands_crud/delete--latest.sql`
- Added Doxygen comments to functions in `sql/udfs/schema_mgmt/create_collection_view--latest.sql`
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Enable distributed tests in documentdb, and clean up helio references from documentdb_distributed
Add substitution to make pg_helio_distributed tests work
Add substitution to make pgmongo/helio_compat tests work
Move udfs bson_aggregation_redact to documentdb
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request enables distributed tests in DocumentDB by adding a new set of regression tests for BSON aggregation comparison operators.
- Added `bson_aggregation_comparison_operators_tests.out` with extensive test cases for BSON comparison operators (`$cmp`, `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`).
* Merged PR 1557887: Change error file mapping name
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Rename error file mapping
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request updates the name of the error mapping file and adjusts references accordingly.
- `Makefile`: Updated references from `mongoerrors.csv` to `error_mappings.csv`.
- `docs/pgmongo/design_docs/data_plane/backend_common/error_management.md`: Updated documentation to reflect the new file name.
- `src/utils/documentdb_conversion_codes.c`: Updated comment to reference the new file name.
- Moved `oss/mongoerrors.csv` to `oss/error_mappings.csv` without code changes.
* Merged PR 1555452: [OSS] Enable distributed tests in documentdb
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Enable distributed tests in documentdb, and clean up helio references from documentdb_distributed
Add substitution to make pg_helio_distributed tests work
Add substitution to make pgmongo/helio_compat tests work
Move udfs bson_aggregation_redact to documentdb
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request enables distributed tests in DocumentDB by adding a new set of regression tests for BSON aggregation comparison operators.
- Added `bson_aggregation_comparison_operators_tests.out` with extensive test cases for BSON comparison operators (`$cmp`, `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`).
* Merged PR 1555841: [OSS] fix regress sql test workflow
[OSS] fix regress sql test workflow
----
Bug fix for the SQL test workflow.
This pull request fixes the regression in the SQL test workflow by updating the GitHub Actions configuration and related scripts.
- Updated `.github/workflows/regress_tests.yml` to improve the build matrix, install dependencies, and run tests for multiple PostgreSQL versions and Ubuntu versions.
- Modified `scripts/start_oss_server.sh` to correct the help message and default directory.
- Minor cleanup in `scripts/install_setup_pg_cron.sh` to remove an unnecessary blank line.
---------
Co-authored-by: Vinod Sridharan <visridha@microsoft.com>
Co-authored-by: Zinan Ren <zinanren@microsoft.com>
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request moves the `auth_scram` functions back to the OSS directory and updates the relevant SQL files to reflect this change.
- Moved `auth_scram--1.14-0.sql` and `auth_scram--latest.sql` from `helio_api/pg_helio_api/sql/udfs/auth` to `oss/pg_documentdb/sql/udfs/auth`.
- Updated `pg_helio_api--1.12-0--1.13-0.sql` to include the new path for `auth_scram--1.14-0.sql`.
- Updated `pgmongo--1.13-2--1.14-0.sql` to include the new path for `auth_scram--1.14-0.sql`.
- Updated `documentdb--0.24-0.sql` to include the new path for `auth_scram--1.14-0.sql`.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
version update 1.24
----
#### AI description (iteration 1)
#### PR Classification
Version update
#### PR Summary
This pull request updates the version numbers for various modules and adds corresponding SQL migration files.
- Updated version numbers in `.control` files for `pg_helio_api`, `pg_helio_core`, `pg_helio_distributed`, `pg_documentdb`, `pg_documentdb_core`, and `pgmongo`.
- Added new SQL migration files for version updates in `pg_helio_api`, `pg_helio_core`, `pg_helio_distributed`, `pg_documentdb`, `pg_documentdb_core`, and `pgmongo`.
- Moved release notes file from `pgmongo_1.25.md` to `pgmongo_1.100.md`.
- Updated `toc.yml` to include the new release notes for `pgmongo 1.100`.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Move pre_consistent back to helio and remove from documentdb schema
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request moves the `pre_consistent` function back to the `helio` schema and removes it from the `documentdb` schema.
- `pgmongo/sql/pgmongo--1.4-0--1.4-1.sql`: Updated includes to reference `pg_helio_api` instead of `pg_documentdb`.
- `oss/pg_documentdb/sql/documentdb--0.24-0.sql`: Removed includes for `index_operator_classes_preconsistent` files.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
draft
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces new aggregation functions and support for t-digest in the project.
- Added `bson_tdigest.c` to implement t-digest helper functions.
- Created new SQL aggregate functions in `group_aggregates--1.23-0.sql` and `group_aggregates_support--1.23-0.sql`.
- Updated `bson_aggregation_pipeline.c` to handle the new `$percentile` accumulator.
- Modified `metadata_cache.c` and `metadata_cache.h` to include OID for the new `BsonPercentileAggregateFunction`.
- Updated migration scripts `pg_helio_api--1.22-0--1.23-0.sql` and `pgmongo--1.22-0--1.23-0.sql` to include new SQL files.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Porting udfs into documentdb
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup and feature enhancement.
#### PR Summary
This pull request continues the porting of UDFs into DocumentDB, updating and including various SQL files for better functionality and organization.
- Updated `documentdb--0.24-0.sql` to include new UDFs and operators, replacing old function references.
- Added new UDFs for query, projection, geospatial, aggregation, metadata, schema management, index management, CRUD commands, user management, schema validation, telemetry, TTL, vector search, extension version, diagnostic, and RBAC metadata.
- Removed outdated function definitions and replaced them with updated SQL includes.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
Aggregation performance pipeline optimization for $lookup and $unwind stage combination on same fields.
### Description
1. Optimize $lookup and $unwind stage combination on same field and lookup options are simple `localField` and `foreignField`
2. Delete dead code for $lookup
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup and optimization.
#### PR Summary
This pull request improves the combination of `$lookup` and `$unwind` aggregation stages by removing redundant and complex code, simplifying the logic, and optimizing performance.
- Removed the `ProcessLookupCore` function and its associated logic from `bson_aggregation_nested_pipeline.c`.
- Removed the `UnwindWalkerContext` struct and related functions for optimizing `$unwind` with `ARRAY_AGG` from `bson_aggregation_nested_pipeline.c`.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Start porting schemas for documentdb api.
Introducing API_SCHEMA_NAME_V2, API_SCHEMA_INTERNAL_NAME_V2, API_CATALOG_SCHEMA_NAME_V2
----
#### AI description (iteration 1)
#### PR Classification
Porting type/operator/schema into documentdb.
#### PR Summary
This pull request ports various types, operators, and schemas into documentdb as part of an ongoing effort.
- `pg_documentdb/sql/documentdb--0.24-0.sql`: Created multiple schemas and included various SQL files for query functions, operators, and metadata.
- `pg_documentdb/sql/operators/bson_dollar_operators--0.16-0.sql`: Updated operator definitions to use `__API_SCHEMA_INTERNAL_V2__`.
- `pgmongo/sql/operators/bson_dollar_negation_operators--1.15-1.sql` and `pg_documentdb/sql/operators/bson_dollar_negation_operators--0.16-0.sql`: Changed operator definitions to use `__API_SCHEMA_INTERNAL_V2__`.
- `Makefile.cflags`: Added checks for new schema variables `API_SCHEMA_NAME_V2`, `API_SCHEMA_INTERNAL_NAME_V2`, and `API_CATALOG_SCHEMA_NAME_V2`.
- `pg_documentdb/sql/schema/unique_shard_path_operator_class--0.24-0.sql`: Updated operator class definitions to use `__API_SCHEMA_INTERNAL_V2__`.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Move udf files from pg_helio_api -> pg_documentdb
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request moves User Defined Function (UDF) files from the `pg_helio_api` directory to the `pg_documentdb` directory.
- Updated `pgmongo/sql/pgmongo--1.23-0--1.24-0.sql` to include UDFs from `pg_documentdb` instead of `pg_helio_api`.
- Modified `oss/pg_helio_api/sql/pg_helio_api--1.15-0--1.16-0.sql` to reference UDFs from `pg_documentdb`.
- Adjusted `oss/pg_helio_api/sql/pg_helio_api--1.10-0.sql` to include UDFs from `pg_documentdb`.
- Updated `scripts/validate_helio_extension_state.sh` to add validation exceptions for the moved UDF files.
- Changed `pgmongo/sql/pgmongo--1.21-0--1.22-0.sql` to reference UDFs from `pg_documentdb`.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Move type/operator/schema files from pg_helio_api -> pg_documentdb
Except udf files
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request moves type, operator, and schema files from the `pg_helio_api` directory to the `pg_documentdb` directory.
- Updated SQL include paths in multiple `pg_helio_api` SQL files to reference `pg_documentdb` directory.
- Modified `pgmongo` SQL files to include paths from `pg_documentdb` instead of `pg_helio_api`.
- Adjusted `validate_helio_extension_state.sh` script to account for new file paths.
- Updated `Makefile` in `pg_helio_api` to include the new directory structure.
- Changed include paths in `pg_documentdb_distributed` and `pg_helio_distributed` C source files to reflect the new directory structure.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
- Redo GUCs for helio -> documentdb preserving back-compat for the service.
- Move Docdb version to 1.24 -> 0.24
----
#### AI description (iteration 1)
#### PR Classification
API change
#### PR Summary
This pull request updates the prefix for all GUCs (Grand Unified Configuration) to a neutral prefix.
- `documentdb_init.c`: Changed GUCs from "helio_api" to a dynamic prefix using `psprintf`.
- `documentdb_distributed_init.c` and `documentdb_distributed.c`: Updated GUCs to use a dynamic prefix.
- `pg_documentdb.c`, `pg_helio_api.c`, and `pgmongo.c`: Modified initialization functions to pass the new GUC prefix.
- `helio_api_init.h` and `helio_distributed_init.h`: Updated function signatures to include the new GUC prefix parameter.
- Updated control files to reflect the new version `0.24-0`.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Remove helio from file name
Remove helio references from source content except schema reference
Start local documentdb server with the documentdb extension
Regression tests using the documentdb extension
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request removes references to "helio" from the `documentdb_core` source code.
- `scripts/utils.sh`: Renamed variables from `preloadExtensions` to `preloadLibraries`.
- `pg_documentdb_core/include/utils/error_utils.h`: Updated function and macro names to replace "Helio" with "DocumentDB".
- `pg_documentdb_core/include/io/helio_bson_core.h` and `pg_documentdb_core/include/query/helio_bson_compare.h`: Moved to `pg_documentdb_core/include/io/bson_core.h` and `pg_documentdb_core/include/query/bson_compare.h`.
- `pg_documentdb_core/src/collation/collation.c` and `pg_documentdb_core/src/io/pgbson.c`: Replaced error codes and messages to use "DocumentDB" instead of "Helio".
- `scripts/validate_helio_extension_state.sh`: Added exceptions for certain files to skip validation for "helio" references.
Deepak Bisht
Vinod Sridharan
Cornelius Boateng
Amit Prabhudesai
Manish Gaur
Parag Jain
Jia Hu
Santiago Fernandez Madero
Andrew Khoma
Javier Figueroa
Vinod Sridharan
Sena Gungor
Zinan Ren
Siddhesh Vethe
Shuai Tian
Xing Fan
