### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Improve pushdown of $elemMatch for composite indexes.
When there's an elemMatch of the form `{ "a": { "$elemMatch": { "$gt": 10, "$lt": 15 }}`
This can scan the index only from 10 -> 15
However given the multi-key index requirement we were doing 2 scans:
- One from 10 to infinity and the other from -infinity to 15.
Optimize this path for $elemMatch.
----
#### AI description (iteration 1)
#### PR Classification
This PR introduces a new feature that optimizes the pushdown of `$elemMatch` queries for composite indexes.
#### PR Summary
The pull request implements optimized handling of `$elemMatch` predicates on composite indexes by converting them into specialized index pushdown expressions and updating the corresponding test expectations.
- **`oss/pg_documentdb/src/opclass/bson_gin_composite_core.c` & `index_support.c`**: Added logic to detect `$elemMatch` in range queries and convert them into optimized pushdown expressions (via functions like `GetElemMatchIndexPushdownOperator`).
- **Configuration Changes**: Introduced a new GUC (`documentdb.useNewElemMatchIndexOperatorOnPushdown`) in `pg_documentdb/src/configs/feature_flag_configs.c` to control the new behavior.
- **Test Updates**: Modified expected explain outputs in both `/oss/internal/pg_documentdb_distributed/src/test/regress/` and `/pgmongo/src/test/docdb_compat/expected/` to reflect the revised index condition format using `elemMatchIndexOp`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Order by expressions have a filter for full scan that makes it so it can push down to the index. However the default selectivity of 1% kicks in at the time. Instead ensure that It has a selectivity of 100%
Also fix list collections with distribution failing on views
----
#### AI description (iteration 1)
#### PR Classification
Bug fix addressing errors in BSON range operator selectivity and index pushdown behavior.
#### PR Summary
This PR fixes issues with full scan detection and selectivity handling in BSON range queries while enhancing index pushdown and metadata listing tests. The changes improve the correctness of query planning and operator behavior in DocumentDB integrations.
- **`src/query/bson_dollar_selectivity.c` & `src/opclass/index_support.c`**: Added functions to detect full scans and correctly set selectivity (e.g., returning 1.0) for `$range` queries.
- **Test Files (`bson_composite_order_by_index_tests.out`, `list_metadata_cursor_tests.out`, and SQL scripts)**: Extended tests for composite order-by indexes and distributed metadata, ensuring proper index selection and query plans.
- **Configuration and Startup Scripts (`feature_flag_configs.c`, `start_oss_server.sh`)**: Introduced the `setSelectivityForFullScan` flag and updated extended RUM extension settings.
- **SQL Migration Scripts**: Altered operator restrict definitions to use updated selectivity functions for consistent operator handling.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
DevOps
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Move the PG18 build container to the latest image
----
#### AI description (iteration 1)
#### PR Classification
Configuration update for build container images.
#### PR Summary
This pull request updates the PG18 build container image to use the latest normalized tag, ensuring consistency with the PG17 configuration.
- Updated ``/.pipelines/templates/common_variables.yml`` by replacing the hard-coded tag for PG18 with ``variables.NormalizedContainerTag``.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Move pg_documentdb_extended_rum to the top level folder from the internal folder since documentdb (tests and product) now has a dependency on it.
pg_documentdb_extended_rum is a fork of the RUM Extension that provides extensions that were needed for documentdb. This includes the ability to push down order by and group by to the index; The ability to do index only scans; improvements on vacuum and handling of page splits; index scan improvements while retaining the original inverted index for text search capabilities that the RUM index had.
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
no
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
no
### Are you introducing any new config? If yes, do you have tests with and without them being set?
no
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Use GA version of PG18
----
#### AI description (iteration 1)
#### PR Classification
This pull request updates dependency references to use the GA version of PG18 and adjusts related version settings.
#### PR Summary
The changes update version references to switch from a release candidate to the GA version of PG18 while also bumping the CITUS version. Key modifications include:
- `oss/scripts/setup_versions.sh`: Changed `POSTGRES_18_REF` from "REL_18_RC1" to "REL_18_0" with the corresponding new commit hash and updated `CITUS_13_VERSION` from v13.1.0 to v13.2.0.
- `cgmanifest.json`: Updated the postgres repository commit hash to align with the GA version.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Continous TTL was not taking into consideration the max continous timeout but instead the single ttl task so it was always capped at 20 seconds.
TTL descending needed to use the order by function and fullrange so that it goes through the order by push down logic rather than PGs logic.
Also, add an entry to the log to indicate if it is descending or not.
----
#### AI description (iteration 1)
#### PR Classification
This PR fixes TTL purging issues by correcting the descending sort behavior and ensuring continuous purging.
#### PR Summary
The changes update TTL deletion logic to use descending order when enabled and adjust associated logging and configuration options. This enhances purging consistency and aligns test expectations.
- `pgmongo/src/test/regress/expected/pgmongo_internal_ttl_tests.out` and related test outputs now include the `use_desc_sort` flag in log messages.
- `oss/pg_documentdb_distributed/src/test/regress/expected/commands_create_ttl_indexes.out` and SQL test files have been modified to set descending sort and disable sequential scans.
- `oss/pg_documentdb/src/ttl/ttl_index.c` has been updated to conditionally append descending order queries using index orderby pushdown.
- Configuration files such as `oss/pg_documentdb/src/configs/background_job_configs.c` now define and enable TTL descending sort options (e.g., `EnableTTLDescSort` and `EnableIndexOrderbyPushdown`).
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Turn on some Vacuum cleanup GUCs and Parallel build GUCs by default for RUM
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a configuration update that enables key RUM features by adjusting default GUC settings.
#### PR Summary
The changes update default settings in RUM header files to enable parallel index building and vacuuming of entry items.
- `pgmongo_rum/src/rum.h`: Changed `RUM_DEFAULT_ENABLE_PARALLEL_INDEX_BUILD` from false to true and `RUM_DEFAULT_VACUUM_ENTRY_ITEMS` from false to true.
- `oss/internal/pg_documentdb_extended_rum/src/pg_documentdb_rum.h`: Changed `RUM_DEFAULT_ENABLE_PARALLEL_INDEX_BUILD` from false to true.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Have the OSS PGRX install script build release by default for install
----
#### AI description (iteration 1)
#### PR Classification
Bug fix to correctly set build profiles in the pgrx installation script.
#### PR Summary
This pull request adjusts the pgrx installation script to default to a release build when no profile is specified and to separate profile arguments for installation and packaging.
- In `/oss/scripts/build_and_install_with_pgrx.sh`, the script now introduces `packageProfileArg` and `installProfileArg`, using `--release` for installation when no profile is provided.
- The cargo commands in the script are updated to use the new profile arguments for both the install and package operations.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
no
### Are you introducing any new config? If yes, do you have tests with and without them being set?
no
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
- enable building both locally & in docker
- fix the hardcoded paths
----
#### AI description (iteration 1)
#### PR Classification
Enhancement: Updates to build scripts and Docker configuration to enable building the gateway both locally and in Docker.
#### PR Summary
This pull request revises several scripts and configuration files to integrate local and containerized build processes for the gateway.
- **`/oss/scripts/emulator_entrypoint.sh`**: Refactors the configuration file handling by defining a variable and updating file permissions and copy paths.
- **`/oss/scripts/build_and_start_gateway.sh`**: Introduces a local build process using Cargo (clean, build, and binary copy) and adjusts the binary execution path.
- **`/oss/.github/containers/Build-Ubuntu/Dockerfile_gateway`**: Updates copy paths to place the gateway binary and configuration into the new subfolder.
- **`/oss/rust-toolchain.toml` & `/oss/pg_documentdb_gw/.gitignore`**: Adds the Cargo component to the toolchain and ensures the built binary is ignored.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Testing
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Currently documentdb tests only run with default rum and non-composite op-class.
While with PG18 they run with ex_rum as a binary they still don't leverage the extended_rum index am and op-class.
Add an option to add backend tests that test the combo until we make this the default path.
----
#### AI description (iteration 1)
#### PR Classification
This pull request introduces a new feature by adding support for running backend tests that leverage extended_rum in DocumentDB.
#### PR Summary
The changes integrate an extended_rum test suite into the DocumentDB backend testing infrastructure, ensuring that extended index handlers are properly configured and validated.
- **`src/test/extended_rum_tests`**: Added new SQL test files, expected output files, scheduling files, a Makefile, a shell script (`mutate_schedule.sh`), a custom `postgresql.conf`, and a `.gitignore` to support and isolate extended_rum tests.
- **`src/test/regress/validate_test_output.sh`**: Modified to correctly resolve directories for validating test outputs.
- **Top-level and project Makefiles**: Updated (`Makefile`, `oss/pg_documentdb/Makefile`, and `src/test/Makefile`) with new targets to run extended_rum tests alongside the existing regression tests.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
Adding a warmpath aggregate 5M flow for the Rust gateway by following the original design and approach. In summary, we store the information we need in memory and flush every 5 minutes. The data is aggregated based on a key hash of static fields and the values like count, max and min per dimension are upserted.
Related work items: #3761267, #4673400
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Add support for hidden indexes with distributed tables.
- Add function that supports per node distribution with shards: Select all shards for the table and pick one shard per node.
- Add a planner worker function that executes on every shard
- On the shard skip execution if it's not the selected shard
- Run the local function on the worker function.
- Plumb in hooks to execute the index functions remotely.
----
#### AI description (iteration 1)
#### PR Classification
This pull request implements a new feature to support hidden indexes in a distributed environment.
#### PR Summary
The changes integrate hidden index support into distributed index update operations by introducing new per-node functions, updating API hooks, and modifying planner and UDF logic for proper shard routing and logging.
- **`node_distribution_operations.c`**: New file that adds functions (e.g., `UpdateDistributedPostgresIndex` and its worker) to execute index update commands per node in a distributed setting.
- **`diagnostic_commands_common.c`**: Introduces `command_node_worker` to ensure that update commands target the appropriate shard based on table identity.
- **`coll_mod.c` and `coll_mod.h`**: Updated to use `UpdatePostgresIndexWithOverride` and correctly incorporate hidden index modifications with distributed operations.
- **API hooks and metadata files** (e.g., `api_hooks.c`, `api_hooks_def.h`, `api_hooks.h`, and `metadata_cache.c`): Enhanced to support new function pointers and caching mechanisms for hidden index updates via the new command worker.
- **SQL UDF scripts**: New UDF definitions for `command_node_worker` and `update_postgres_index_worker` enable proper registration and execution of hidden index operations in the distributed framework.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature/Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
- Add tracking of parallel workers in currentOp.
- Ensure index build progress works for blocking index builds
- Fix crash in parallel rum index builds
- Allow RUM parallel index build path even for serial build (so we can unify to one codepath)
- Turn on amcanparalllelbuild for PG >= 17 based on the GUC
----
#### AI description (iteration 1)
#### PR Classification
Bug fix addressing logging inaccuracies for parallel build operations.
#### PR Summary
This pull request improves logging for current operations and index build progress by incorporating additional metadata and refining SQL queries to better handle parallel workers.
- In `oss/pg_documentdb/src/commands/current_op.c`, new attributes (backend type and leader PID) are extracted from query results and appended to BSON output for accurate identification of parallel workers.
- In `oss/pg_documentdb/src/commands/current_op.c`, progress logging is enhanced by including a "concurrent" flag and updating phase-to-message mappings for index build initialization.
- In `pgmongo/src/infrastructure/index_stats_logging.c`, the index progress query now uses left joins with COALESCE to reliably capture index and table names, improving stats logging for parallel builds.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfixes
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Fix bugs around post-recovery cleanup and incomplete splits in some cases causing Lost path
2) The retry around the failure was also not working due to case differences in errors
3) Ensure tracking vs fixing are tracked by different GUCs.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix addressing lost index paths in RUM indexes by ensuring incomplete page splits are properly handled.
#### PR Summary
This pull request fixes lost path errors in RUM indexes by enhancing the insertion and page split logic to detect, complete, and clear incomplete splits.
- **`pgmongo_rum/src/rumbtree.c`**: Introduced new functions (e.g., `rumPlaceToPage`, `rumFinishSplit`, `rumFinishOldSplit`) to complete pending page splits during insertions and update parent links appropriately.
- **`pgmongo_rum/src/rum.h`, `rumdatapage.c`, and `rumentrypage.c`**: Updated to track and clear the `RUM_INCOMPLETE_SPLIT` flag and to integrate new callbacks for incomplete split scenarios.
- **`pgmongo_rum/src/rumutil.c` and related configuration files**: Added custom GUC variables (`track_incomplete_split`, `fix_incomplete_split`, `enable_inject_page_split_incomplete`) to control the behavior of incomplete split handling.
- **Test files in `/pgmongo/src/test/regress/`**: Added new regression tests to validate the correct handling of incomplete splits and the associated lost path conditions.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
* added documentation on how to compile and install the binary
Signed-off-by: Vamsi Konakanchi <contact@vamsi-k.com>
* added seperate document for info on building from scratch
Signed-off-by: Vamsi Konakanchi <contact@vamsi-k.com>
* changed release to release-with-symbols for better debugging
Signed-off-by: Vamsi Konakanchi <contact@vamsi-k.com>
* updated the port and added the proper build command for gw
Signed-off-by: Vamsi Konakanchi <contact@vamsi-k.com>
---------
Signed-off-by: Vamsi Konakanchi <contact@vamsi-k.com>
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
This new option for create indexes creates an index as if it was a UNIQUE composite term index, but without actually creating the constraint on the table, so it is a regular index.
This is the first phase of 3 needed.
We introduce a three-step workflow that lets customers build the index structure in the background (non-blocking), then progressively enable uniqueness enforcement:
1. buildAsUnique (createIndex) — build the index as if unique but without registering any table constraint (non-blocking)
2. prepareUnique (collMod) — register the uniqueness constraint in the system catalogs but mark it unvalidated so it starts enforcing uniqueness for new writes only
3. convertUnique (collMod) — validate/enforce uniqueness for existing rows (walk the table or short-circuit if an existing validated unique index already proves uniqueness)
## Goals
- Allow background construction of unique-capable indexes without blocking writes
- Provide a fast metadata-only step to enable uniqueness for new writes
- Provide a validation step that enforces uniqueness for pre-existing rows only when it is safe
- Preserve data integrity and PostgreSQL metadata correctness
- Expose these steps through familiar MongoDB commands (`createIndex` and `collMod`)
- Migrating non ordered unique indexes to ordered indexes without blocking production writes.
## High-level Workflow
Step 1 — buildAsUnique (createIndex):
- Customer requests an index with storageEngine.buildAsUnique=true.
- The system creates a regular index physically (non-blocking) but records metadata indicating the index is intended to become unique
Step 2 — prepareUnique (collMod):
- Customer requests convert-to-unique preparation via `collMod` (prepareUnique)
- The system registers a constraint in the PostgreSQL catalogs but does not validate it against existing data; it enforces uniqueness on new writes going forward
- This is a fast, mostly metadata-only operation (convalidated = false)
Step 3 — convertUnique (collMod):
- Customer requests final conversion via `collMod` (convertUnique)
- The system validates uniqueness for existing rows by walking the table (or aborts if duplicates are found), or it can skip the full walk when an existing validated unique index already proves uniqueness
----
#### AI description (iteration 1)
#### PR Classification
This pull request implements a new feature that adds support for the "prepareForUnique" option in index creation.
#### PR Summary
The changes extend the index creation mechanism to allow a non-table unique index to be prepared via the "prepareForUnique" flag, with appropriate error handling and ...
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Move order by support into 106.1 schema as a hotfix option
----
#### AI description (iteration 1)
#### PR Classification
This pull request implements a new feature by adding reverse order support for the 106.1 schema.
#### PR Summary
The changes refine reverse sorting by updating conditional checks in the C code and aligning SQL migration scripts with the new version.
- `oss/pg_documentdb/src/commands/current_op.c`: Added a null check for `detail` before allocating and returning the index specification, ensuring proper handling.
- `oss/pg_documentdb/src/opclass/index_support.c`: Modified reverse order conditions to include the 106.1 patch check along with the existing 107.0 version check.
- `pgmongo/sql/`: Moved the SQL migration file (renamed from `pgmongo--1.106-0--1.107-0.sql` to `pgmongo--1.106-0--1.106-1.sql`) and updated the corresponding sha1 checksum files.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Remove default arg value from the new update function.
Function has not yet been used in C code.
----
#### AI description (iteration 1)
#### PR Classification
This pull request is an API change that revises internal UDF functions for update operations by removing default argument values.
#### PR Summary
The changes introduce a new SQL file for bson update functions and update existing function signatures to eliminate default values—resulting in stricter parameter handling and an added parameter for V2 functions.
- `oss/pg_documentdb/sql/udfs/commands_crud/bson_update_document--0.108-0.sql`: Added a new file defining the update functions.
- `oss/pg_documentdb/sql/udfs/commands_crud/bson_update_document--latest.sql`: Modified the `__API_SCHEMA_INTERNAL_V2__.update_bson_document` function by removing default values from `arrayFilters` and `variableSpec` and adding the `collationString text` parameter.
- `pgmongo/sql/pgmongo--1.107-0--1.108-0.sql` and `oss/pg_documentdb/sql/documentdb--0.107-0--0.108-0.sql`: Updated include directives to load the new update document SQL file.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
- Add a gateway host extension that starts up and maintains the documentdb_gw as a postgres background worker.
- Fix up start_oss_server.sh to support also launching the documentdb_gw as a background worker
- Fix up start_oss_server.sh to support using documentdb_extended_rum as a binary replacement for rum
- Fix up build_and_install_with_pgrx to support packaging a pgrx extension as well as ensuring pgrx is installed correctly.
- Add sanity test to validate the gateway host.
----
#### AI description (iteration 23)
#### PR Classification
New feature implementation.
#### PR Summary
This PR adds support for a DocumentDB Gateway Host extension that runs as a PostgreSQL background worker, enhancing the OSS extension capabilities. Key changes include:
- **`/.pipelines/templates/oss_documentdb_build.yml`**: Added conditional build jobs for OSS Gateway Host and OSS Gateway with new parameters.
- **`/.pipelines/templates/oss_documentdb_tests.yml`**: Introduced a new testing pipeline to validate the DocumentDB extension with the Gateway Host.
- **`/oss/scripts/build_and_install_with_pgrx.sh`**: Updated cargo-pgrx installation logic with enhanced options and help support.
- **`/oss/scripts/start_oss_server.sh`**: Enhanced the startup script to configure gateway worker mode and extended rum options.
- **File reorganization**: Moved the `pg_documentdb_gw_host` source and control files from `/gateway` to `/oss` and updated the changelog accordingly.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Observability
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Add feature counters for all public query operators
----
#### AI description (iteration 1)
#### PR Classification
This PR implements a new feature by adding comprehensive feature counters for all query operators to enhance observability.
#### PR Summary
The changes extend the observability framework by introducing new feature counter definitions, mappings, and usage in query operator implementations.
- Updated `oss/pg_documentdb/include/utils/feature_counter.h` and `oss/pg_documentdb/src/infrastructure/feature_counter.c` to add new query operator feature enum values and corresponding string mappings with an increased `MAX_FEATURE_COUNT`.
- Modified `oss/pg_documentdb/src/planner/mongo_query_operator.c` to assign the new feature counters to each query operator.
- Revised test expected outputs in `/oss/internal/pg_documentdb_distributed/src/test/regress/expected/feature_counters.out` and `/pgmongo/src/test/docdb_compat/expected/feature_counters.out` to reflect the updated counters.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
BugFix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
- Today before building the complete pipeline query of lookup we initialize the varlevelsup of `Var` to be set to INT_MAX but if there is an aggregate such as $sum it then tries to get the parseState of those many levels of parent and hence crashes out.
The fix is we earlier set varlevelsup to int max so that we can uniquely identify these Vars after building the query for pipeline but instead of using varlevelsup now we use location field to uniquely identify these Vars and adjust the levelsup later.
----
#### AI description (iteration 1)
#### PR Classification
This pull request implements a bug fix to resolve a crash when building aggregate queries that include $group aggregates within $lookup.
#### PR Summary
The changes introduce a new test case to validate the correct query plan for lookup queries with aggregates and adjust internal variable handling in the aggregation pipeline builder to prevent crash dumps.
- **`bson_aggregation_nested_pipeline.c`**: Updated variable resolution by setting `varlevelsup` to 0 and adjusting `location` values to ensure proper handling in nested pipeline contexts.
- **Test files (`bson_aggregation_stage_lookup_pg17_tests.out`, `bson_aggregation_stage_lookup_tests.out`, and related SQL tests)**: Added new expected outputs for lookup queries incorporating aggregates to verify the fix across different test suites.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4114368, #4696693
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
DevOps
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Add Version Logging in the backend metrics collection.
This makes it so we're not reliant on the gateway to log version information to metrics.
----
#### AI description (iteration 1)
#### PR Classification
This pull request introduces a new feature to log backend version metrics.
#### PR Summary
The changes add functionality to periodically log version information from various components, ensuring updated logging when version changes or timer intervals trigger a new record.
- `pgmongo/src/infrastructure/mdsd_logger.c`: Added new functions (`append_version_metric_data` and `LogVersionMetricData`) to construct and send version metrics, and integrated these into the main logging workflow.
- `pgmongo/src/pgmongo_extension_version.c`: Introduced `PgmongoGetExtendedVersionString` and updated the version-returning function to use this extended string.
- `oss/pg_documentdb/src/utils/version_utils.c`: Added `GetCurrentShortVersionStringForLogging` to format a concise version string for backend logging.
- `pgmongo/src/metadata/oid_cache.c`: Added the `CitusVersionFunctionOid` function and updated the OidCache structure to support version logging.
- Header updates in `oss/pg_documentdb/include/utils/version_utils_private.h` and `pgmongo/include/metadata/oid_cache.h` to declare the new version utility and Oid functions.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
no
### Are you introducing any new config? If yes, do you have tests with and without them being set?
no
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
Provide Rust dev environment in devcontainer
### Description
- install toolchain from rustup
- ensure we run the proper profile in dockerfile
- make sure tests are working
----
#### AI description (iteration 1)
#### PR Classification
This pull request introduces a new feature to enhance the Rust development environment in the OSS repository.
#### PR Summary
The changes streamline the Rust dev experience by updating the build infrastructure, adding Rust-specific tools, and fine-tuning test configurations. Key updates include:
- **`Dockerfile_gateway`**: Adjusts the Docker workflow by switching to a non-root user, installing Rust via a new `install_rustup.sh` script, and building with the custom `release-with-symbols` profile.
- **`devcontainer.json`**: Incorporates the Rust analyzer extension and detailed Rust settings for improved performance, code formatting, and project linking.
- **`Cargo.toml`**: Introduces a new release profile (`release-with-symbols`) with tailored debug and optimization settings.
- **Test and common modules**: Updates test assertions and runtime configuration to correctly handle error messages and dynamically set the Postgres system user.
- **New files and minor updates**: Adds `install_rustup.sh` and `rust-toolchain.toml` for robust Rust toolchain management, and updates `.gitignore` by removing `.vscode/`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf/Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Delete the old vacuum posting tree path (it's been unused for a year)
2) Port in more changes to minimize delta between internal and documentdb_rum - enable pruning empty entries in the entry tree.
----
#### AI description (iteration 1)
#### PR Classification
This pull request performs a code cleanup and synchronization of vacuum routines for RUM indexes across internal and documentdb_rum modules.
#### PR Summary
This update refactors the vacuum functionality by removing legacy functions and unifying the cleanup logic for empty entries and posting trees, while also enhancing logging and configuration consistency. Key changes include:
- **`/oss/internal/pg_documentdb_extended_rum/src/rumvacuum.c`**: Removed obsolete functions (e.g., rumVacuumPostingTreeLeaves and rumVacuumPostingTree) and refactored vacuum routines to use the new rumCleanupEmptyEntries mechanism.
- **`/pgmongo_rum/src/rumvacuum.c`**: Made similar removals and simplifications in vacuum scanning, streamlining bulk delete processing and ensuring consistent handling of empty entries.
- **Logging Enhancements**: Updated bulk delete routines to report detailed metrics (empty pages, empty entries, pruned pages/entries, etc.) using unredacted logging.
- **GUC and Header Updates**: Modified custom variable definitions in `rumutil.c` and adjusted header files (`pg_documentdb_rum.h` and `rum.h`) to remove deprecated flags and introduce a new vacuum cleanup entries setting.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Fix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Pick up newest version of Citus 12 which has the fix for the crash on EnsureSnapshotRegistered
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a version update and bug fix that integrates a new citus release containing a crash fix.
#### PR Summary
The changes update the citus dependency to a commit that fixes snapshot handling crashes and improve related installation scripts while removing an obsolete hook.
- In `oss/scripts/setup_versions.sh`, the CITUS_VERSION is updated to the commit with the crash fix.
- In `scripts/install_setup_pg_diskann.sh`, logic is added to derive the Postgres library directory using pg_config for a more robust setup.
- In `cgmanifest.json`, the commit hash for the citus dependency is updated.
- In `.devcontainer/postcreate.sh`, the call to the rust hooks setup script is removed.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
Introduce a struct to parse client information from the connection context that will be used for useragent logging
----
#### AI description (iteration 1)
#### PR Classification
New feature – introduces a dedicated structure for parsing and formatting client information from BSON documents.
#### PR Summary
This PR adds a new module that defines a `ClientInformation` struct and associated helper functions to extract client details from BSON, and it integrates this functionality into telemetry logging.
- `oss/pg_documentdb_gw/src/client_info.rs`: Introduces the `ClientInformation` struct with parsing and formatting functions alongside comprehensive tests.
- `gateway/native/src/telemetry/failure.rs`: Updates the failure logging function to use the new client information parsing.
- `gateway/native/tests/telemetry.rs`: Enhances tests to verify the correct structure and content of the client info output.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Reduce deltas between private and documentdb_rum - port more changes between internal and documentdb_rum.
----
#### AI description (iteration 1)
#### PR Classification
This pull request ports internal changes to the documentdb_rum extension, introducing new unredacted logging hook integration and enhanced scan logic.
#### PR Summary
The changes integrate a new hook setter for unredacted logging, add support for pre-consistent scanning using function pointers, and update TS function exports and minor comments for clarity.
- **`pgmongo_rum/src/rumutil.c` & `pgmongo_rum/src/rum.h`**: Removed the legacy unredacted logging function and added a new exported hook setter (`SetRumUnredactedLogEmitHook`) along with macros for unredacted logging.
- **`oss/internal/pg_documentdb_extended_rum/src/rumget.c` & related files**: Enhanced scan routines by incorporating pre-consistent function pointers and bounds checking to determine the appropriate scan type.
- **`oss/pg_documentdb/src/index_am/rum.c`**: Modified the initialization routine to load and invoke the new unredacted logging hook from the external library.
- **TS utility files (`rum_ts_utils.c` and others)**: Added PGDLLEXPORT declarations for TS functions and made minor comment updates for improved clarity.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Enable more builds and tests with PG18
Enable backend tests for documentdb with pg18
----
#### AI description (iteration 7)
#### PR Classification
This pull request implements PG18 support by updating APIs, tests, and build pipelines.
#### PR Summary
The changes introduce PG18‑specific logic across regression tests, build scripts, and core GIN indexing code to enhance compatibility and performance with PostgreSQL 18.
- **`oss/pg_documentdb/src/test/regress/expected/`**: New PG18 output files (e.g. `collection_management_pg18.out` and inverse match explain tests) were added while obsolete PG17/PG15 files were removed or moved.
- **`/.pipelines/templates/extension_build_flex_core.yml`**: Build pipelines were updated with new parameters (`buildExtensionPackage`, `buildDiskANN`) and conditional steps to support PG18 builds.
- **`helio_gin` sources (e.g. `ginget.c`, `ginutil.c`, `ginordered.c`)**: Core GIN code was refactored using `#if PG_VERSION_NUM >= 180000` conditionals to utilize new PG18 functions and adapt to API changes.
- **Test scheduling and mutation scripts**: Updated to include PG18 tokens and adjust SQL file references for proper test execution on PostgreSQL 18.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
no
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
no
### Are you introducing any new config? If yes, do you have tests with and without them being set?
no
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
* Use the appropriate GUC for the user_crud_commands.sql
### Description
change `documentdb_api.enableUsersInfoPrivileges` to `documentdb.enableUsersInfoPrivileges` in user_crud_commands.sql
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
Embed the requestId in the activity_id making it easier to trace a complete request from the moment the connection was created all the way up until it is closed.
Also, we were not including the activity_id in all the places that we could making tracing difficult. As well, we performed some log cleanup to only include useful information in the messages.
```
2025-09-17T20:28:34.582Z New TCP connection established with activity id: '77a5e6b3-17f2-4102-9f73-8cd16786a9db'
2025-09-17T20:28:36.482Z Processing connection: '77a5e6b3-17f2-4102-9f73-8cd100000013'. Client IP: '[::ffff:127.0.0.1]:60126', HostName: 'localhost'
2025-09-17T20:28:36.483Z Connection with id '77a5e6b3-17f2-4102-9f73-8cd100000013' closed.
```
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a bug fix that improves logging and connection tracking by correctly embedding the original requestId in the activity_id.
Related work items: #4584662
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1. replaced GetObjectIdFilterFromQueryDocument with GetObjectIdFilterFromQueryDocumentValue We have already seen this in updateOne path in cpu and fixed it , missed it updatemany and delete so added in this PR
2. In delete path at once place we were calling GetObjectIdFilterFromQueryDocument early where it not used so pushed it down
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Add an extension that can work as a postgres background worker that launches the DocumentDB Gateway.
Allow registration of this worker in start_oss_server.sh as well.
This simplifies management of the DocumentDB extension by letting PG manage the lifetime of all components (documentdb and the gateway as well).
----
#### AI description (iteration 9)
#### PR Classification
New feature: This PR introduces a new PostgreSQL extension that hosts the DocumentDB gateway using pgrx and a background worker.
#### PR Summary
The changes add a Rust-based extension (`pg_documentdb_gw_host`) that registers a background worker running on a Tokio runtime to manage DocumentDB gateway operations, and updates the build, installation, and CI scripts to support it.
- `gateway/pg_documentdb_gw_host/src/bgworker.rs`: Implements the background worker that initializes the gateway, configures TLS, and manages connection pools.
- `gateway/pg_documentdb_gw_host/src/lib.rs` and `src/gucs.rs`: Define extension initialization and custom GUC settings for database and configuration file path.
- `gateway/pg_documentdb_gw_host/Cargo.toml`, control, and embed files: Set up the build configuration and packaging for the new extension.
- Pipeline YAML and build scripts: Update CI/CD and installation processes to build and deploy the new gateway host extension.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Make DocumentDB not explicitly depend on the RUM extension (it's done implicitly) on the binary.
Instead pull in the SQL dependency into documentdb and reference the C functions internally. This allows us to swap between rum.so and extended_rum.
For Pg18 (where there isn't yet official RUM support) default to using extended_rum for the .so file and remove all dependencies on the "rum.so"
----
#### AI description (iteration 2)
#### PR Classification
This PR is an API and dependency update that removes the explicit dependency on the external RUM extension by integrating its functionality internally.
#### PR Summary
The update integrates the RUM text-search capabilities directly into DocumentDB and cleans up the external linkage.
- `oss/internal/pg_documentdb_extended_rum/src/rum_ts_utils.c` is newly added with core text-search functions.
- `oss/pg_documentdb/src/index_am/rum.c` is modified to load internal wrappers that call the new RUM functions.
- SQL UDF adapter files and operator class definitions are updated to reference the internal functions instead of public RUM functions.
- Build scripts, metadata, and control files are adjusted to remove the explicit dependency on the external RUM extension.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
no
### Are you introducing any new config? If yes, do you have tests with and without them being set?
no
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
- enable the default TLS 1.3 mozilla intermediate acceptor to enable the latest ciphersuites
- dispatch new thread as soon as we've got the tcp connection
- because of this change we can now reload the whole SslAcceptor and not just cert bundle
Related work items: #4689774
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Use prepared statements with UpdateAllMatchingDocuments
2) Don't pass shard_key_value by default if using the `@@` operator
3) Do something similar for SELECT update candidate.
----
#### AI description (iteration 1)
#### PR Classification
Performance optimization and bug fix for update query execution.
#### PR Summary
This PR enhances update performance by refining shard key filter handling and optimizing SPI plan selection, while also introducing new query plan cache keys and a configuration flag for legacy behavior.
- `oss/pg_documentdb/src/commands/update.c`: Refactors update candidate selection by conditionally adding shard key filters, adjusting argument indexing, and using the full collection object instead of separate identifiers.
- `oss/pg_documentdb/include/infrastructure/documentdb_plan_cache.h`: Introduces new constants for query plan caching to support optimized update queries.
- `oss/pg_documentdb/src/configs/feature_flag_configs.c`: Adds the `UseLegacyShardKeyFilterOnUpdate` flag to offer backward compatibility with the previous shard key filtering behavior.
- Test files in the distributed module: Update expected SQL query strings to match the revamped query construction logic.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Remove feature flags that are default on with the v106:
- EnableNewCompositeIndexOpclass
- EnableDescendingCompositeIndex
- EnableCompositeUniqueIndex
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup aimed at removing redundant composite feature flags that are now always default.
#### PR Summary
This PR simplifies the codebase by eliminating conditional logic, configuration variables, and test settings related to composite feature flags such as descending composite index, composite unique index, and the new composite index opclass. The changes remove obsolete definitions and related conditional checks while updating tests and config files accordingly.
- In `configs/feature_flag_configs.c`, removed custom variable definitions and configuration settings for composite flags and introduced new default flags like `EnableIndexOnlyScan` and `EnableRangeOptimizationForComposite`.
- In files such as `commands/create_indexes.c`, `index_am/rum.c`, and others, removed conditional logic and external declarations dependent on composite feature flags.
- Updated numerous test scripts and configuration files to remove legacy settings (e.g., removing `enableNewCompositeIndexOpClass` and `enableDescendingCompositeIndex`) and adjust expected outputs.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Minor perf fixes for persistent cursor
- Don't generate cursorId in persistent cursor creation
- Don't copy the persistent cursor plan on generation
- Don't eagerly create a hold store unless the cursor is not terminated.
----
#### AI description (iteration 1)
#### PR Classification
Performance improvement for persistent cursors through enhanced cursor ID generation and optimized portal hold management.
#### PR Summary
This PR refines persistent cursor handling by introducing conditional logic based on a new feature flag, resulting in faster and more efficient cursor operations.
- `/oss/pg_documentdb/src/commands/aggregation_commands.c`: Adjusted cursor ID generation and continuation document creation with support for delayed portal hold via a new conditional path.
- `/oss/pg_documentdb/src/commands/cursors.c`: Optimized portal management by selectively holding portals and modifying cursor options according to the delayed hold flag.
- `/oss/pg_documentdb/src/configs/feature_flag_configs.c`: Introduced the `EnableDelayedHoldPortal` flag to control and tune portal hold behavior.
- `/scripts/build_dev_flex_container.sh`: Updated container cleanup logic to respect the SKIP_CONTAINER_CLEANUP environment variable.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
no
### Are you introducing any new config? If yes, do you have tests with and without them being set?
no
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
- drop unused environment variable `ENFORCE_SSL` in dockerfile *[Bugfix]* [issue 313](https://github.com/documentdb/documentdb/issues/313)
### Description
drop unused environment variable
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Add support for hidden indexes with collMod
This will mark the index as not ready in the postgres catalog. Handles distributed setups as well.
However, this is currently restricted to single node (sharded/unsharded).
A subsequent change can help multi-node set ups.
----
#### AI description (iteration 1)
#### PR Classification
This pull request implements a new feature by adding support for hidden indexes in DocumentDB.
#### PR Summary
The pull request extends the collMod command and associated index management logic to recognize and update a new “hidden” flag in index options, while enforcing constraints (e.g. disallowing hidden modifications for primary or unique indexes) in both single-node and distributed environments.
- **`coll_mod.c`**: Introduces helper functions (e.g. GetHiddenFlagFromOptions, UpdateHiddenInIndexOptions, UpdatePostgresIndex) and SPI query updates to process the hidden flag properly.
- **Test Files**: Adds and updates multiple test cases (e.g. in `commands_collmod_tests_hidden.out` and SQL test scripts) to verify hidden index creation, query planning, and error handling.
- **Distributed Hooks and API**: Updates in files like `distributed_hooks.c` and `api_hooks.c` (with corresponding header changes) enable proper handling of hidden index updates across shard indexes in distributed environments.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
1.108
### Description
We are adding support for the RolesInfo command in Engine
----
#### AI description (iteration 1)
#### PR Classification
New feature enhancement: introducing the `rolesInfo` command to retrieve detailed role information including inherited roles and privileges.
#### PR Summary
This pull request adds comprehensive support for the `rolesInfo` command in the engine, implementing new parsing, processing, and BSON output functions while updating tests and auxiliary query strings.
- `oss/pg_documentdb/src/commands/roles.c`: Added functions (e.g., `ParseRolesInfoSpec`, `WriteRoleDocumentWithInheritedRoles`, `WriteInheritedRolesToArray`) to parse rolesInfo parameters and construct role documents with inherited roles and optional privileges.
- `oss/pg_documentdb/src/test/regress/expected/role_crud_commands.out` and `pgmongo/src/test/regress/expected/role_crud_commands.out`: Expanded test suites with numerous rolesInfo scenarios, error conditions, and inheritance verification tests.
- `oss/pg_documentdb/include/roles.h`: Introduced new types and function prototypes to support rolesInfo and role CRUD operations.
- `pgmongo/src/pgmongo.c`: Updated query string definitions to handle role name substitutions for OSS compatibility.
- `oss/pg_documentdb/src/commands/users.c`: Modified role checking and privilege extraction utilities to align with the new rolesInfo functionality.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4552244
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
Vinod Sridharan
Shuai Tian
Shuai Tian
Santiago Fernandez Madero
Zhe Li
Andrew Khoma
Juan Carlos Aguilera Mendez
Vamsi Konakanchi
Cornelius Boateng
Deepak Bisht
Parag Jain
Viktor Tarassov
William Wang