### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Comments
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
All the previous version SQL changes should be merged into this PR so that we can update the hashes in this PR
----
#### AI description (iteration 1)
#### PR Classification
This PR is a documentation update that cleans up inline SQL comments to align with the latest contributing guidelines.
#### PR Summary
The changes refine the SQL file comments for geospatial operator classes and functions by removing outdated references to Mongo indexing and validations.
- `oss/pg_documentdb/sql/operators/bson_gist_geospatial_op_classes--0.16-0.sql`: Updated comments by replacing "mongo 2d index options" and "mongo 2dsphjere index options" with simplified descriptions.
- `oss/pg_documentdb/sql/udfs/geospatial/bson_gist_extensibility_functions--0.16-0.sql` and `oss/pg_documentdb/sql/udfs/geospatial/bson_gist_extensibility_functions--latest.sql`: Revised function comments to remove "mongo db validations" for a more general description of geometry validations.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4354111
This PR updates the extension version for the new release 0.106.
Changes included:
- Control files updated to new version
- SQL files generated for version upgrades
- CHANGELOG.md updated with new version entry
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
The cron job for cursor cleanup on disk was broken since it was declared as a function but the call was done as a procedure. Fix the job command.
----
#### AI description (iteration 1)
#### PR Classification
This pull request fixes a bug related to the cursor cleanup cron job.
#### PR Summary
The changes address incorrect scheduling and command syntax for cursor cleanup tasks in different modules.
- `/pgmongo/sql/pgmongo--1.104-1--1.105-0.sql`: Added a plpgsql block to conditionally schedule the `helio_cursor_cleanup_task` if it does not exist.
- `/oss/pg_documentdb/sql/documentdb--0.104-0--0.104-1.sql`: Updated the cron job syntax for `documentdb_cursor_cleanup_task` to call the correct cleanup function.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
[OSS] fix 0.104-1 hotfix upgrade scripts
----
#### AI description (iteration 1)
#### PR Classification
Hotfix update to upgrade SQL upgrade scripts correcting version references.
#### PR Summary
This pull request updates SQL upgrade scripts to use the correct UDF and documentdb version numbers.
- In `pgmongo/sql/pgmongo--1.103-0--1.103-1.sql`, the include directive was updated from version 0.105-0 to 0.104-1.
- In `oss/pg_documentdb/sql/documentdb--0.104-0--0.104-1.sql`, the UDF include reference was changed to use version 0.104-1.
- The file `oss/pg_documentdb/sql/documentdb--0.104-0--0.105-0.sql` was moved to `oss/pg_documentdb/sql/documentdb--0.104-1--0.105-0.sql` with no code changes.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes, the API schema
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
1.05
### Description
We are adding support for the Connection Status command, which exists in native Mongo, and allows users to see their authenticated user, authenticated roles, and user privileges. For more information, visit connectionStatus.md.
This PR is the second of three PRs for this feature. The first one added extensible role privileges to our `users_info` command, allowing users to retrieve role privileges with `showPrivileges: true`. This second PR adds a callable `connection_status` command to our API, that utilizes the role privileges. The last PR will add a gateway change that calls this `connection_status` command.
Related work items: #4287246
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
Enhancement and bug fix to improve the stability and management of file-based cursors.
#### PR Summary
This pull request refactors and bolsters file-based cursor management by introducing shared memory support, improved cleanup, error handling, and configuration changes. It also adds new API endpoints and logging for better cursor lifecycle control.
- **`cursor_store.c` and related files:** Added shared memory functions (e.g., `FileCursorShmemSize`, `InitializeFileCursorShmem`), cursor count regulation (with `IncrementCursorCount`/`DecrementCursorCount`), and cleanup routines to ensure stable cursor file operations.
- **Test and expected output files:** Updated regression tests and expected outputs to cover scenarios like duplicate cursor creation, cursor deletion limits, and error reporting when the file cursor capacity is reached.
- **API enhancements:** Introduced new UDFs (e.g., `command_delete_cursors`) and tenant client methods for killing and deleting cursors, improving session management.
- **Configuration updates:** Renamed and added GUC parameters (e.g., `MaxAllowedCursorIntermediateFileSizeMB`, `MaxCursorFileCount`, and cursor expiry settings) to control file-based cursor behavior.
- **Logging and metrics improvements:** Enhanced logging in `mdsd_logger.c` to report current cursor counts and related metrics for operational insight.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Removing unnecessary in/out from bson_update_document which were preset eariler due to older change_stream implementation.
1. For now moving udf to helio_api_internal and same time removing useless in/out params
TODO:
In next release will modify this for oss with updating test and minimal c changes will update --latest files also.
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a code cleanup and update to simplify the integration of the `bson_update_document` function into OSS.
#### PR Summary
The PR introduces a new C-based SQL function for `bson_update_document` with streamlined parameters, updates the main SQL script to include this new file, and refreshes the API schema output to reflect the change.
- `pgmongo/sql/udfs/crud/bson_update_document--1.105-0.sql`: Adds the new function definition with cleaned-up parameter handling.
- `pgmongo/sql/pgmongo--1.104-0--1.105-0.sql`: Updates the script to include the newly added UDF file.
- `pgmongo/src/test/regress/expected/public_api_schema.out`: Modifies the schema output to include the new function entry.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Add support of altrernate index handler in 1.103
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
More general improvements to order by and composite:
Create Indexes
1) Support creating a composite index side by side with a regular index
2) Block creation of descending or wildcard and unsupported set ups for composite indexes
Planner:
1) Support ordering by the Nth path of an index if the prefixes are equality.
2) Validate that if it's not the first column, prefixes are provided
3) Support incremental sort with index pushdown.
----
#### AI description (iteration 1)
#### PR Classification
This pull request delivers a new feature and API enhancement to enable order-by index pushdown and refined composite index support.
#### PR Summary
The update streamlines composite index processing by introducing stricter equivalency checks, new UDF functions for full-scan order-by pushdown, and enhanced validations during index creation while updating explain outputs and tests.
- `src/metadata/index.c` refines index options equivalency logic to correctly compare composite term settings.
- `src/opclass/bson_gin_composite_entrypoint.c` and `src/index_am/rum.c` add functions to support order-by pushdown and manage duplicate entries efficiently.
- `sql/udfs/query/bson_dollar_comparison--0.105-0.sql` (and related operator files) introduce new UDFs including `bson_dollar_fullscan` to enable full scan operations.
- `src/commands/create_indexes.c` enforces new composite index validations (restricting use to ascending, non-wildcard regular indexes) with precise error reporting.
- Test scripts and explain plan outputs in the `regress` and `pgmongo` directories are updated to confirm the new composite order-by behavior.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Move bucketAuto to schema 105
----
#### AI description (iteration 1)
#### PR Classification
This pull request introduces a schema migration for the `$bucketAuto` aggregation stage, moving its support from schema 104 to schema 105.
#### PR Summary
The changes update documentation, SQL scripts, and test configuration to reflect the migration of `$bucketAuto` support to the new schema.
- `pgmongo/sql/pgmongo--1.104-0--1.105-0.sql`: Added inclusion of the new `$bucketAuto` UDF file for schema 105.
- `oss/pg_documentdb/sql/documentdb--0.104-0--0.105-0.sql`: New file added to include `$bucketAuto` support for the updated schema.
- `oss/CHANGELOG.md` and release notes files: Updated to move the `$bucketAuto` aggregation stage support from the previous schema version to the new release notes for schema 105.
- Several test CSV files under `/gateway/managed/Tests.Integration.Shell.Postgres` were updated to reflect the new schema version.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
This pull request adds documentation for the `$bucketAuto` aggregation stage in PostgreSQL, detailing its design, usage, and implementation.
- Adds `bucketAuto.md` with details on the `$bucketAuto` aggregation stage
- code implementation
- support all granularity types: POWERSOF2, 1-2-5, Renard numbers, E series
Related work items: #3234185
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Version update for 104
Related work items: #4281831
Yes
Feature
Yes
No
- Support compact database administration command behind the GUC `enableCompact`.
- Add support for compact command in backend using VACUUM FULL which is a blocking operation.
- Only one compact command can be run at a time on a specified collection.
- VACUUM FULL is run using libpq in a top level transaction as it it not supported inside functions or procedures.
- Add support of tracking compact command in current_op.
- `dryRun` estimation is based on the query defined [here at pgexperts](https://github.com/pgexperts/pgx_scripts/blob/master/bloat/table_bloat_check.sql), this can be a rough estimation and doesn't provided details about the space reclaimed by vacuuming RUM indexes of the collection.
- Read only users can't perform `compact` command.
----
New feature implementation to support the backend compact command.
This PR adds complete backend support for a blocking compact command that estimates table bloat and conditionally performs a VACUUM FULL operation on collection tables.
- `src/utils/storage_utils.c` and `include/utils/storage_utils.h`: Introduced a new bloat estimation utility returning collection bloat stats.
- `src/commands/compact.c`: Implemented the compact command including BSON command parsing, dry-run handling, and conditional execution of VACUUM FULL.
- `internal/pg_documentdb_distributed/src/test/regress/sql/command_compact.sql`: Added regression tests covering various compact command scenarios.
- `include/utils/feature_counter.h` and `src/infrastructure/feature_counter.c`: Updated feature counter mapping to include the new compact command feature.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4237921
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
PoC/Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Today, persisted cursors require the connections to be held. This is bad since Connections/Backends are a limited resource.
This introduces the option to save the cursor state in a file on disk and reuse that across backends.
This is behind a GUC due to the presence of the following:
1) Needs more testing and validation
2) Needs backpressure to ensure that disk consumption doesn't balloon up
3) There is a procedure added to clean up these files but it is not wired up
4) We need to handle ultra-large responses (beyond 4 GB it fails outright at the moment). This needs more graceful handling to split into chunks.
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a new feature implementation that introduces file‐based persisted cursors to allow query results to be “drained” and stored on disk without holding on to active database connections.
#### PR Summary
This PR adds support for persisting cursor state to the filesystem in both the pg_documentdb and pgmongo modules, enabling connections to be released while still allowing cursors to be later resumed. The changes include:
- **`infrastructure/cursor_store.c` and `cursor_store.h`** – New APIs for creating, writing to, reading from, and closing persisted cursor files.
- **Commands integration** – Modifications in `commands/cursors.c`, `commands/aggregation_commands.c`, and `cursor_private.h` to use file-based cursor functions for draining and continuation.
- **SQL test suites** – Addition of new test files (e.g. in `.../regress/sql/bson_aggregation_cursor_tests_file.sql`) to validate cursor behavior across various query scenarios.
- **Configuration updates** – New custom GUCs in `feature_flag_configs.c` and adjustments in test PostgreSQL configuration files to enable and control file-based persisted cursors.
- **Executor enhancements** – Updates in cursor draining and query plan execution functions to support both in-memory and file-based cursor paging.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes, yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support sorting with collation.
We define new custom sort operators which compare operands using applicable collation string.
`<<<` is for ascending order.
`>>>` is for descending order.
Example:
`SELECT * from <collection> ORDER BY <value> USING <<<`
This will further be used for partitioning, grouping, and related comparisons.
----
#### AI description (iteration 1)
#### PR Classification
New feature: Support for collation in sort operations.
#### PR Summary
This pull request adds support for collation in sort operations within the database engine.
- `src/test/regress/expected/bson_dollar_ops_collation_tests_runtime.out`: Added multiple test cases to verify collation behavior with various sort and order by operations, including case sensitivity and numeric ordering.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4014123
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
GW plumbing of compact command
1- Route the compact command to the backend.
2- Add a dummy C handler function for `compact` that does nothing and throws the same error.
Next PR is going to add support for compact in the backend with blocking writes by running VACUUM FULL
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for the `compact` command in the MongoDB Postgres runtime.
- Added `CompactAsync` method in `PostgresClient.cs` to handle the compact command.
- Implemented `command_compact` function in the new file `compact.c`.
- Updated `PostgresMongoClient.cs` to include the `CompactAsync` method.
- Modified `MongoRequestParser.cs` to parse the compact command.
- Added SQL function definitions for the compact command in `compact--0.103-0.sql` and `compact--latest.sql`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
BugFix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
Bugfix to override nested arrays in dotted path in `bson_dollar_merge_documents` UDF for few cases
### Description
bson_dollar_merge_documents function today recurses into nested arrays always and project in each element of array but that is not correct with mongo semantics.
This is only applicable if the projectetion is a dotted path.
The function is used in 7 different stages/features today:
1- list_collection: no dotted paths for projection, so this is okay to follow any projection.
2- graphlookup: Mongo does a shallow projection in this case.
3- lookup: Mongo does a shallow projection in this case.
4- inverseMatch: okay as no dotted path projection
5- count: okay as no dotted path projection
6- group : okay as no dotted paths are allowed
7- setWindowFields: only this stage does a full projection even inside nested arrays
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request addresses multiple scenarios for fixing projection with `bson_dollar_merge_documents`.
- Added new functions in `oss/pg_documentdb/sql/udfs/projection/bson_projection--0.102-0.sql` to handle various BSON document operations.
- Updated `bson_dollar_merge_documents` function to include an `overrideArray` parameter.
- Modified test expectations in `oss/internal/pg_documentdb_distributed/src/test/regress/expected/bson_aggregation_pipeline_tests_graphlookup.out` to reflect changes in the `bson_dollar_merge_documents` function.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Get background index builds to work for single-node documentdb.
This changeset fixes background index builds for OSS by:
1. Scheduling the background index build tasks
2. Moving the distributed functionality (which had not being refactored out of the indexing code during the move to OSS) into distributed hooks, with the base implementation for single-node.
3. Refactoring SQL tests to get even with the above changes.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix for background index builds in DocumentDB.
#### PR Summary
This pull request addresses issues with background index builds in DocumentDB by adding comprehensive tests and handling various edge cases.
- `commands_create_indexes_background.out`: Added extensive test cases to validate the behavior of `create_indexes_background` function, including handling of invalid inputs, unique constraints, and various index specifications.
- Introduced a new schema and function to manage index job statuses.
- Enhanced error handling and validation for index creation requests, ensuring proper error messages and statuses are returned for invalid operations.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
- add cursor result type for UDFs, this helps PG to not always try to figure out the `OUT` variables and form a tuple descriptor
----
#### AI description (iteration 1)
#### PR Classification
Performance improvement and API change.
#### PR Summary
This pull request introduces performance improvements for point reads by adding a new cursor type for SQL functions. It modifies existing functions to return a structured type instead of a record, enhancing efficiency and clarity.
- `query_cursors_aggregate--latest.sql`: Added `cursor_result` type and updated several functions to return this type instead of a record.
- `query_cursors_aggregate--0.103-0.sql`: Newly added file with similar changes as `query_cursors_aggregate--latest.sql`.
- `documentdb--0.102-1--0.103-0.sql` and `pgmongo--1.102-1--1.103-0.sql`: Updated to include the new `query_cursors_aggregate--0.103-0.sql` file.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4157027
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
Version update for the release of version 103.
#### PR Summary
This pull request updates the version numbers across several control files and adds release notes for version 1.103 and 1.104. It also includes changes to SQL files and their corresponding SHA1 checksums.
- `/oss/internal/pg_documentdb_distributed/documentdb_distributed.control`, `/oss/pg_documentdb/documentdb.control`, `/oss/pg_documentdb_core/documentdb_core.control`, and `/pgmongo/pgmongo.control`: Updated default version numbers.
- `/docs/release_notes/pgmongo_1.103.md` and `/docs/release_notes/pgmongo_1.104.md`: Added release notes for versions 1.103 and 1.104.
- `/pgmongo/sql/pgmongo--1.103-1--1.104-1.sql`: Added new SQL file for version transition.
- `/docs/release_notes/toc.yml`: Updated table of contents to include new release notes.
- `/oss/CHANGELOG.md`: Added entry for documentdb v0.104-0.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4140262
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Currently bulk insert run as a function which means that the entire execution runs as a single transaction. Each batch insert is handled as a subtransaction and rolled back to a save point if it fails.
This means that document locks are held for longer, and we can hit limits w.r.t. max documents updated per transaction.
In multi-node setups this also means distributed transactions are held longer causing loss of connection pools.
This change introduces a stored procedure for bulk insert that periodically "Commits" the state to WAL and starts a new transaction. This more mirrors what the non-atomic bulk update is supposed to do. The old function is retained to be used within transactions.
----
#### AI description (iteration 1)
#### PR Classification
Performance improvement for write commands.
#### PR Summary
This pull request introduces a new procedure for handling write commands, enhancing performance by optimizing user-specific command execution. It includes the addition of new functions and procedures to manage MongoDB operations more efficiently.
- Added new SQL functions and procedures in `/pgmongo/sql/udfs/gateway_funcs/command_funcs--1.103-0.sql` for running commands, queries, DDL, and write operations as a specific user.
- Introduced new insert functions and procedures in `/oss/pg_documentdb/sql/udfs/commands_crud/insert--0.103-0.sql` to handle MongoDB insert operations.
- Implemented `pgmongo_run_write_procedure_as_user` and `ExecuteWriteCommandCore` in `/pgmongo/src/infrastructure/command_funcs.c` to manage write commands with user context.
- Added `CommitOuterInsertTransactions` function in `/oss/pg_documentdb/src/commands/insert.c` to handle transaction commits outside of transaction blocks.
- Introduced a feature flag `EnableWriteProcedure` in `/oss/pg_documentdb/src/configs/feature_flag_configs.c` to control the enabling of write procedures.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Add a new bson_expression_get that appends the collationString to the got value, if a valid collationString is passed to it.
This will be used as a helper function to operations that rely on bson_expression_get to pass down collation such as bson_lookup_join_filter.
If no collation string is passed as an arg, we simply return the got value.
----
#### AI description (iteration 1)
#### PR Classification
New feature to support `$graphLookup` in the engine with collation.
#### PR Summary
This pull request adds support for `$graphLookup` with collation in the engine.
- Added new functions in `oss/pg_documentdb/sql/udfs/aggregation/bson_lookup_functions--0.103-0.sql` for `$graphLookup` operations.
- Modified `oss/pg_documentdb/src/aggregation/bson_aggregation_nested_pipeline.c` to handle collation in `$graphLookup` join filters.
- Updated `oss/pg_documentdb/src/operators/bson_expression.c` to remove collation handling from `bson_expression_get`.
- Added `BsonDollarLookupJoinFilterWithCollationFunctionOid` in `oss/pg_documentdb/src/metadata/metadata_cache.c`.
- Adjusted collation handling in `oss/pg_documentdb/src/query/bson_dollar_operators.c` for `$graphLookup`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4104899
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Currently bulk updates run as a function which means that the entire execution runs as a single transaction. Each batch update is handled as a subtransaction and rolled back to a save point if it fails.
This means that document locks are held for longer, and we can hit limits w.r.t. max documents updated per transaction.
In multi-node setups this also means distributed transactions are held longer causing loss of connection pools.
This change introduces a stored procedure for bulk update that periodically "Commits" the state to WAL and starts a new transaction. This more mirrors what the non-atomic bulk update is supposed to do. The old function is retained to be used within transactions.
Still TODO:
For unsharded collections in a multi-node setup, we need to send batches of updates to the worker. This is not wired up in this change.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces a new bulk update feature for handling non-transactional updates in the MongoDB wire-protocol. It adds new functions and procedures to support bulk updates and modifies existing update logic to accommodate this new functionality.
- Added `update_bulk` procedure and related functions in `sql/udfs/commands_crud/update--0.103-0.sql` for non-transactional bulk updates.
- Implemented `command_update_bulk` in `src/commands/update.c` to handle bulk updates outside of transactions.
- Enhanced `ProcessBatchUpdateCore` in `src/commands/update.c` to support transactional and non-transactional updates.
- Updated feature tracking in `include/utils/feature_counter.h` and `src/infrastructure/feature_counter.c` to include `FEATURE_COMMAND_UPDATE_BULK`.
- Modified SQL scripts `documentdb--0.102-1--0.103-0.sql` and `pgmongo--1.102-1--1.103-0.sql` to include the new update functionality.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Fix up the remaining operators - implementations are sub-optimal
2) Add type bracketing options for all types and use them in operators
3) Add support for truncation of terms
TODO:
- Support recheck on the index for operators than just bounds
- Support N boundaries for index terms.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request addresses issues with operators and type bracketing in BSON composite indexes.
- Added `bson_gin_index_types_core.c` to manage BSON type bounds for indexes.
- Updated `bson_gin_composite_entrypoint.c` to fix operator handling and apply type bracketing for various BSON index strategies.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
As part of testing out the composite operator classes, introduce a new composite operator class for Bson. This will be behind a test only GUC and will be for testing composite index pushdown.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces a new composite operator class for BSON GIN indexes in PostgreSQL.
- Added `bson_gin_composite_entrypoint.c` to implement GIN operator functions for BSON composite indexes.
- Created test cases in `bson_composite_index_tests_insert_terms.out` to validate term generation for composite paths.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix(ish)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
0.102-0 had changes that are non-idempotent. Given there were some incorrect dependencies taken on the 0.102 release (independently) introduce a 0.102-1 version that applies all the delta of 0.102-0 in an idempotent manner so that dependencies taken partially on 0.102 and dependencies taken on 0.102 fully work side-by-side.
----
#### AI description (iteration 1)
#### PR Classification
Hotfix for DocumentDB version upgrade.
#### PR Summary
This pull request introduces a hotfix for DocumentDB, ensuring an idempotent upgrade from version 0.102-0 to 0.102-1.
- Added `documentdb--0.102-0--0.102-1.sql` to handle partial upgrades and repeat UDF upgrades.
- Modified `data_table_utils.c` to check for both 0.102-0 and 0.102-1 versions during data table upgrades.
- Updated `pgmongo.control` to set the default version to 1.102-1.
- Added `pgmongo--1.102-1--1.103-0.sql` for version transition.
- Updated SHA1 checksums in multiple `pgmongo/sql/sha1` files to reflect changes.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Version update for 102 release
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support for pushdown of to PFE indexes
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for pushdown operations to PFE indexes, enhancing query performance and index utilization.
- Added new functions in `sql/udfs/index_mgmt/bson_indexbounds_functions--0.102-0.sql` for BSON index bounds operations.
- Introduced new operators in `sql/operators/bson_btree_pfe_operators--0.102-0.sql` for BSON and BSON query comparisons.
- Updated `src/query/query_operator.c` to handle new index bounds operators and expressions.
- Added new type definitions in `sql/types/bsonindexbounds--0.102-0.sql` for BSON index bounds.
- Modified `src/metadata/metadata_cache.c` to include OIDs and functions for BSON index bounds types and operators.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Altering creation_time column removing not null and default constraint.
----
#### AI description (iteration 1)
#### PR Classification
Performance improvement
#### PR Summary
This pull request alters the `creation_time` column in the database schema to improve performance.
- Added `AlterCreationTime` function in `cluster_operations.c` to modify the `creation_time` column.
- Updated `SetupCluster` function in `cluster_operations.c` to call `AlterCreationTime`.
- Created new SQL scripts `extension_upgrade_alter_creation_time_1.102-0.sql` and `extension_upgrade_alter_creation_time_1.102-0.out` for testing the changes.
- Modified multiple test output files to reflect the changes in the `creation_time` column constraints.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Remove all Citus references from pg_documentdb tests
2) Remove some citus references from documentdb c files (Comments)
3) Remove dead code for bson_out aggregates
4) Remove all References to IsClusterVersionAtLeast less than 0.24 (since those are now deployed)
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request introduces code cleanups and enhancements related to aggregation functions in DocumentDB.
- `oss/pg_documentdb/sql/udfs/aggregation/group_aggregates--0.102-0.sql`: Added new aggregation functions for BSON data types, including BSONSUM, BSONAVERAGE, BSONMAX, BSONMIN, BSONFIRST, BSONLAST, BSONFIRSTN, BSONLASTN, BSON_ARRAY_AGG, BSON_OBJECT_AGG, BSON_ADD_TO_SET, BSON_MERGE_OBJECTS, BSONSTDDEVPOP, BSONSTDDEVSAMP, BSONPERCENTILE, and BSONMEDIAN.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support collation with `find` queries.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for the `project find` functionality with collation in the engine.
- Added new SQL functions in `sql/udfs/projection/bson_projection--0.102-0.sql` for handling BSON projection with collation.
- Updated `metadata_cache.c` to include OID caching for the new `bson_dollar_project_find` function with collation.
- Modified `bson_project.c` to handle additional arguments for collation in the `bson_dollar_project_find` function.
- Updated `bson_projection--latest.sql` to include the new function definition.
- Added function declaration in `metadata_cache.h` for the new `bson_dollar_project_find` function with collation.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #3924476
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support collation for `replaceRoot` aggregation stage
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for the `$replaceRoot` operation with collation in the BSON aggregation pipeline.
- `bson_projection--0.102-0.sql` and `bson_projection--latest.sql`: Added new function `bson_dollar_replace_root` with collation support.
- `metadata_cache.c` and `metadata_cache.h`: Added OID cache and retrieval function for the new `bson_dollar_replace_root` with collation.
- `bson_project.c`: Updated `bson_dollar_replace_root` to handle collation.
- `bson_aggregation_pipeline.c`: Integrated collation handling in `HandleReplaceRoot`.
- Updated test files to include new test cases for `$replaceRoot` with collation.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support collation for `$addFields` and `$set` aggregation stages
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for the `addFields` operation with collation in the aggregation pipeline.
- `bson_project.c`: Added collation support to `bson_dollar_add_fields` and related functions.
- `bson_expression_comparison_operators.c`: Updated comparison operators to handle collation.
- `bson_dollar_operators.c`: Enhanced query state population to include collation.
- Test files `bson_dollar_ops_collation_tests_runtime.out` and `bson_dollar_ops_collation_tests_runtime.sql`: Added tests for collation with `$expr` and `$addFields`.
- `bson_projection--latest.sql`: Added new function definition for `bson_dollar_add_fields` with collation.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #3919349, #3919367
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
more
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for collation in BSON projection functions.
- Added new SQL functions in `sql/udfs/projection/bson_projection--0.102-0.sql` to handle collation.
- Updated `src/aggregation/bson_project.c` to include collation string handling in BSON projection.
- Modified `src/aggregation/bson_aggregation_pipeline.c` to support collation in projection stages.
- Enhanced `src/query/bson_dollar_operators.c` to populate collation string in query state.
- Added collation-related tests in `sql/bson_dollar_ops_collation_tests_runtime.sql`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #3918941
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
more
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for collation in the redact function.
- `bson_project.c`: Added handling for an additional collation string argument in the `bson_dollar_redact` function.
- `bson_aggregation_redact--0.24-0.sql` and `bson_aggregation_redact--latest.sql`: Updated to include the new collation string parameter in the `bson_dollar_redact` function definition.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #3919400
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
[schema validation] apply to stage $merge/$out
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces schema validation for the `$merge` and `$out` stages in MongoDB aggregation pipelines.
- Added `schema_validation_merge.md` to document the design and implementation details.
- Created new SQL functions in `bson_merge_functions--1.24-0.sql` to handle schema validation during the `$merge` stage.
- Updated `bson_aggregation_output_pipeline.c` to integrate schema validation logic into the `$merge` and `$out` stages.
- Modified `toc.yml` to include the new schema validation documentation.
- Updated `pgmongo--1.23-0--1.24-0.sql` and `pg_helio_api--1.23-0--1.24-0.sql` to include the new SQL functions.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Change signature of `bson_query_match` with `let` and `collation` support to match other internal functions.
Fix schema name for pgmongo.
----
#### AI description (iteration 1)
#### PR Classification
New feature: Collation support for query match in unsharded collections.
#### PR Summary
This pull request adds collation support for query match operations in unsharded collections, ensuring proper collation enforcement.
- Modified `bson_query_match` function calls in `/pgmongo/src/test/docdb_compat/expected/bson_dollar_ops_collation_tests_runtime.out` to adjust collation parameters.
- Updated collation parameters in `bson_query_match` function calls in `/oss/internal/pg_documentdb_distributed/src/test/regress/expected/bson_dollar_ops_collation_tests_runtime.out`.
- Adjusted SQL test cases in `/oss/internal/pg_documentdb_distributed/src/test/regress/sql/bson_dollar_ops_collation_tests_runtime.sql` to reflect new collation support.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #3914004
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes; yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support `collation` with query operator using an internal UDF:
```
bson_query_match(document bson, query bson, collationString text, variableSpec bson)
```
Enable feature with `EnableCollationAndLetForQueryMatch` GUC.
Collation on sharded collections with collation-aware shard keys not supported yet.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds collation support for query operators in the engine.
- `query_operator.c`: Added logic to handle collation strings in BSON query operators and expanded function to support collation and variable specifications.
- `metadata_cache.c`: Introduced a new function to get the OID of `bson_query_match` with collation and let arguments.
- `feature_flag_configs.c`: Added a new feature flag to enable collation and let for write commands.
- `bson_query_match--0.102.sql` and `bson_query_match--latest.sql`: Added new SQL functions to support collation in BSON queries.
- `documents_planner.c`: Updated planner to recognize and handle the new collation and let functionality in queries.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Features
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
- Add support for a top level current_op command that calls the aggregation function and forms a command response
- Add support for joining between regular tables and documentdb tables. In this mode, we'll still require a query to be exclusively for find/aggregate etc, but we can put those queries in a CTE/Subquery and join with other find/aggregate/postgres queries and apply regular postgres join semantics.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces new features for handling `current_op` and joining with DocumentDB tables, including support for nested aggregation functions.
- Added `BuildAggregateSpecFromCommandSpec` and `command_current_op` functions in `src/commands/current_op.c` to handle new aggregation specifications.
- Implemented `MutateQueryAggregatorFunction` and `ExpandNestedAggregationFunction` in `src/planner/documents_planner.c` to support nested aggregation queries.
- Added new test cases in `src/test/regress/sql/bson_aggregation_functions_nested_tests.sql` and `src/test/regress/expected/bson_aggregation_functions_nested_tests.out`.
- Introduced feature flag `allowNestedAggregationFunctionInQueries` in `src/configs/feature_flag_configs.c` to control nested aggregation functionality.
- Created new UDFs in `sql/udfs/commands_diagnostic/current_op--0.102-0.sql` for the `currentOp` aggregation stage.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
[OSS] Fix TTL indexes in documentdb
This moves the function to C and instead of querying the indexes and then getting the collection shards from pg_dist_shard table, it uses the C hook to get the table shards where for non-distributed it returns the base table and then uses that information to drop the expired records.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request addresses issues with TTL indexes in DocumentDB by refactoring and optimizing the deletion of expired rows.
- `ttl_index.c`: Refactored the function to delete expired rows, added a new function `delete_expired_rows`, and optimized the deletion process.
- `ttl_support_functions--latest.sql` and `delete_expired_rows--latest.sql`: Replaced PL/pgSQL procedures with C language implementations for better performance.
- `collection.c`: Moved the function `GetMongoCollectionShardOidsAndNames` from `coll_stats.c` to `collection.c` for better organization.
### Does this PR have any customer impact?
yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
yes, creating new UDFs:
- `mongo_api_internal.run_explain_as_user`
- `list_databases`
### Are you introducing any new config? If yes, do you have tests with and without them being set?
yes, gating the explain experience behind the `pgmongo.enableExplainsInSharedUserPool`
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Port the rest of the DML commands and explains to the backend.
Update all of the test coverage for explains for the new changes too.
Move ListDatabases command from Gateway to backend
Related work items: #3759098
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Release
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Bump release
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Description
Updating schema to release 1.101
----
#### AI description (iteration 1)
#### PR Classification
Schema update for new release version.
#### PR Summary
This pull request updates the schema to release version 1.101, including version bumps and minor SQL adjustments.
- Updated `default_version` in `documentdb_distributed.control`, `documentdb.control`, and `documentdb_core.control` to `0.102-0`.
- Updated `default_version` in `pgmongo.control` to `1.101-0`.
- Added new SQL file `pgmongo--1.101-0--1.102-0.sql` with search path settings.
- Updated SHA1 checksums for various `pgmongo` SQL files.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
BugFix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Because of N-1 tests we can't update the sql UDF and start using it in documentdb at the same release. This 1 of 2 PRs to add the schema changes in release 0.101 and next PR will start using these UDF in documentdb in 0.102
Bug description: `bson_dollar_merge_documents` UDF today doesn't override nested arrays for projection with conflicting path and add the new projection entities inside each element of array which is not required in few places
e.g. $lookup, $graphlookup etc
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces a new function to handle mutating nested arrays in BSON documents.
- Added `bson_dollar_merge_documents` function with an override flag in `bson_projection--0.101-0.sql`.
- Implemented `MergeDocumentWithArrayOverride` function in `bson_project.c` to support the new feature.
- Updated test cases in `bson_aggregation_stage_lookup_tests.sql` and corresponding expected output files.
- Included the new SQL file in `documentdb--0.100-0--0.101-0.sql` and `pgmongo--1.24-0--1.101-0.sql`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Enable support for current_op aggregation stage in OSS.
Still pending: support for getting locking pids and index builds.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request enables support for the `current_op` aggregation stage in DocumentDB.
- Added new functions in `current_op--0.101-0.sql` and `current_op--latest.sql` to support the `current_op` aggregation stage.
- Modified `current_op.c` to handle new attributes and improve the `current_op` command processing.
- Updated `system_configs.c` to include a new configuration variable for tracking the application name in `current_op`.
- Adjusted test cases in `bson_aggregation_pipeline_tests_coll_agnostic.sql` and related expected output files to validate the new feature.
- Updated `metadata_cache.c` to correctly reference the `current_op_aggregation` function.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Schema
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Snap 0.101 branch for OSS, 0.100 is already released and can no longer accept changes in schema
----
#### AI description (iteration 1)
#### PR Classification
Version update.
#### PR Summary
This pull request updates the version number for the DocumentDB API and core API surface for PostgreSQL.
- `oss/pg_documentdb/documentdb.control`: Updated `default_version` to '0.101-0'.
- `oss/pg_documentdb_core/documentdb_core.control`: Updated `default_version` to '0.101-0'.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Docs
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
This PR includes updates to add base Doxygen comments within SQL files, setting up initial documentation structure for functions. Additionally, it includes Doxygen configuration files (Doxyfile, etc.) tailored to support SQL file parsing and documentation generation.
----
#### AI description (iteration 1)
#### PR Classification
Documentation
#### PR Summary
This pull request adds Doxygen comments to various `helio_api` functions to improve code documentation.
- Added Doxygen comments to functions in `sql/udfs/commands_crud/query_cursors_single_page--latest.sql`
- Added Doxygen comments to functions in `sql/udfs/utils/extension_version_utils--latest.sql`
- Added Doxygen comments to functions in `sql/udfs/auth/auth_scram--latest.sql`
- Added Doxygen comments to functions in `sql/udfs/commands_crud/delete--latest.sql`
- Added Doxygen comments to functions in `sql/udfs/schema_mgmt/create_collection_view--latest.sql`
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Enable distributed tests in documentdb, and clean up helio references from documentdb_distributed
Add substitution to make pg_helio_distributed tests work
Add substitution to make pgmongo/helio_compat tests work
Move udfs bson_aggregation_redact to documentdb
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request enables distributed tests in DocumentDB by adding a new set of regression tests for BSON aggregation comparison operators.
- Added `bson_aggregation_comparison_operators_tests.out` with extensive test cases for BSON comparison operators (`$cmp`, `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`).
* Merged PR 1557887: Change error file mapping name
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Rename error file mapping
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request updates the name of the error mapping file and adjusts references accordingly.
- `Makefile`: Updated references from `mongoerrors.csv` to `error_mappings.csv`.
- `docs/pgmongo/design_docs/data_plane/backend_common/error_management.md`: Updated documentation to reflect the new file name.
- `src/utils/documentdb_conversion_codes.c`: Updated comment to reference the new file name.
- Moved `oss/mongoerrors.csv` to `oss/error_mappings.csv` without code changes.
* Merged PR 1555452: [OSS] Enable distributed tests in documentdb
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Enable distributed tests in documentdb, and clean up helio references from documentdb_distributed
Add substitution to make pg_helio_distributed tests work
Add substitution to make pgmongo/helio_compat tests work
Move udfs bson_aggregation_redact to documentdb
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request enables distributed tests in DocumentDB by adding a new set of regression tests for BSON aggregation comparison operators.
- Added `bson_aggregation_comparison_operators_tests.out` with extensive test cases for BSON comparison operators (`$cmp`, `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`).
* Merged PR 1555841: [OSS] fix regress sql test workflow
[OSS] fix regress sql test workflow
----
Bug fix for the SQL test workflow.
This pull request fixes the regression in the SQL test workflow by updating the GitHub Actions configuration and related scripts.
- Updated `.github/workflows/regress_tests.yml` to improve the build matrix, install dependencies, and run tests for multiple PostgreSQL versions and Ubuntu versions.
- Modified `scripts/start_oss_server.sh` to correct the help message and default directory.
- Minor cleanup in `scripts/install_setup_pg_cron.sh` to remove an unnecessary blank line.
---------
Co-authored-by: Vinod Sridharan <visridha@microsoft.com>
Co-authored-by: Zinan Ren <zinanren@microsoft.com>
Deepak Bisht
CosmosDB Build Service (msdata)
Vinod Sridharan
Santiago Fernandez Madero
William Wang
Parag Jain
Shuai Tian
Manish Gaur
Cornelius Boateng
Amit Prabhudesai
Jia Hu
Andrew Khoma
Javier Figueroa
Vinod Sridharan
Sena Gungor
Zinan Ren
Siddhesh Vethe
