### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Now that 105 is done, delete any version checks < 102 and remove obsolete feature flags
----
#### AI description (iteration 1)
#### PR Classification
This PR performs code cleanup by removing obsolete feature flag checks and legacy version conditionals.
#### PR Summary
The changes simplify function implementations and configuration management by eliminating outdated feature flags and redundant version checks.
- **`/oss/pg_documentdb/src/metadata/metadata_cache.c` and aggregation files**: Removed conditional branches based on cluster versions (e.g., always using 5 parameters) to simplify BSON merge and projection functions.
- **`/oss/pg_documentdb/src/configs/feature_flag_configs.c` and related config files**: Cleaned up obsolete feature flag variables and custom variable definitions (such as those for index term truncation, index operator bounds, and RUM fast path).
- **Test scripts and expected output files (under `/oss/internal/pg_documentdb_distributed` and `/pgmongo`)**: Updated query plan expectations and removed toggles for index term truncation, reflecting the new default behavior.
- **Other code areas (e.g., in `query_operator.c`, `opclass/gin_index_term.c`, and `oid_cache.c`)**: Eliminated outdated version checks and conditional logic, resulting in cleaner and more consistent code paths.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
Bug fix to enforce that the "roles" field is passed as an array.
#### PR Summary
This pull request adds a validation to ensure the "roles" field is an array during user creation, throwing a specific error if it is not, and updates the corresponding test cases to verify this behavior.
- `oss/pg_documentdb/src/commands/users.c`: Added a check to throw an error when the "roles" field is not an array.
- `oss/pg_documentdb/src/test/regress/sql/user_crud_commands.sql` and `/expected/user_crud_commands.out`: Updated tests to expect the error message on invalid "roles" input.
- `pgmongo/src/test/regress/sql/user_crud_commands.sql` and `/expected/user_crud_commands.out`: Applied similar test updates to enforce the array requirement for the "roles" field.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Fix rum crash
----
#### AI description (iteration 1)
#### PR Classification
Bug fix that resolves crashes in rum index operations.
#### PR Summary
This pull request fixes a crash by enhancing error handling and implementing recovery logic when page splits occur during index scans. The changes improve robustness and stability during query operations.
- `oss/pg_documentdb/src/commands/commands_common.c`: Expanded error handling to detect both "Lost Path" and "invalid offset on rumpage" errors, updating error codes and messages before throwing errors.
- `pgmongo_rum/src/rumutil.c`: Added two custom boolean configuration variables to control refinding leaf pages and throwing errors on invalid data pages.
- `pgmongo_rum/src/rumget.c`: Inserted a loop in `entryGetNextItem` to refind the correct leaf page when a page is split.
- `pgmongo_rum/src/rum.h`: Integrated a check for invalid offsets with error logging and defined new macros and extern declarations for the additional configuration options.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
GUC
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
GUC
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
1.05
### Description
This PR adds a GUC, pgmongo.EnableNativeAuthentication, and code that will block customer requests if 1.) they are native, and 2.) native authentication is disabled.
----
#### AI description (iteration 1)
#### PR Classification
New feature implementation to add a configurable toggle for native authentication.
#### PR Summary
This pull request introduces support for disabling native authentication by adding a new GUC variable and enforcing checks in user management commands.
- `/pgmongo/src/pgmongo.c`: Defines the custom GUC `pgmongo.enableNativeAuthentication` and initializes the `EnableNativeAuthentication` boolean with its default value.
- `/oss/pg_documentdb/src/commands/users.c`: Integrates checks in user creation, update, drop, and retrieval functions to error out when native authentication is disabled for internal users.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #3978592, #4311471
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Comments
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
All the previous version SQL changes should be merged into this PR so that we can update the hashes in this PR
----
#### AI description (iteration 1)
#### PR Classification
This PR is a documentation update that cleans up inline SQL comments to align with the latest contributing guidelines.
#### PR Summary
The changes refine the SQL file comments for geospatial operator classes and functions by removing outdated references to Mongo indexing and validations.
- `oss/pg_documentdb/sql/operators/bson_gist_geospatial_op_classes--0.16-0.sql`: Updated comments by replacing "mongo 2d index options" and "mongo 2dsphjere index options" with simplified descriptions.
- `oss/pg_documentdb/sql/udfs/geospatial/bson_gist_extensibility_functions--0.16-0.sql` and `oss/pg_documentdb/sql/udfs/geospatial/bson_gist_extensibility_functions--latest.sql`: Revised function comments to remove "mongo db validations" for a more general description of geometry validations.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4354111
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Build Custom Plan for single insert for shard table so we skip PG planning phase and directly execute query insert query.
this change improves 500 TPS majored with pgbench and we are not matching native PG for single node , insert one cases
1. created a local PG table same as our data table (even for that I haven't created btree index so we expect operation to be faster) ran the Pgbench using INSERT INTO command
2. ran the pgbench with our insert() udf
This PR updates the extension version for the new release 0.106.
Changes included:
- Control files updated to new version
- SQL files generated for version upgrades
- CHANGELOG.md updated with new version entry
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
graphlookup stage .sql .out
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
For Operator Selectivity, we have a feature flag and function that does better operator selecitivty for operator expressions. However, when doing planning, the query operator generates mostly FuncExprs for index operators. This means the selectivity logic for funcitons defaults to 0.3333 even if opr selectivity can do better.
For Functions, when opr selectivity is enabled, use the same logic to determine selectivity enabling better index pushdowns.
----
#### AI description (iteration 1)
#### PR Classification
This pull request implements a new feature to align function selectivity with operator selectivity for index operations in the DocumentDB Postgres extension.
#### PR Summary
The PR refactors selectivity calculations to ensure consistency between function and operator estimations, updates associated test cases, and introduces new configuration flags. Key changes include:
- **`pg_documentdb/src/opclass/index_support.c`**: Revised the `dollar_support` function to route selectivity requests correctly using a new pointer variable and support for selectivity mode.
- **`pg_documentdb/src/query/bson_dollar_selectivity.c` and `include/query/bson_dollar_selectivity.h`**: Updated the selectivity computation routines to accept a default selectivity parameter and adjust calculations based on the new operator selectivity mode.
- **`configs/feature_flag_configs.c` and `system_configs.c`**: Added new configuration flags (`DisableDollarSupportFuncSelectivity` and `EnableNewOperatorSelectivityMode`) and adjusted default values to control the new selectivity behavior.
- **Test files in `/oss/internal/pg_documentdb_distributed/src/test/regress/` and `/pgmongo/src/test/docdb_compat/expected/`**: Extended and modified SQL regression tests to verify expected index query plans and selectivity estimates.
- **Configuration scripts (`/scripts/utils.sh` and related test configs)**: Made minor adjustments such as reducing the index build schedule interval to better support the new functionality.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Cleanup tests for index creation
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix/Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Re-enable lookup left join and make inner join behind a GUC
----
#### AI description (iteration 1)
#### PR Classification
This PR re-enables lookup left join functionality while making inner join behavior optional via a new configuration flag.
#### PR Summary
The pull request refactors lookup join processing so that by default the system uses left join (adding a null check filter) while inner join is enabled only when the new GUC is turned on. Key changes include:
- **`pg_documentdb/src/aggregation/bson_aggregation_nested_pipeline.c`**: Updated join logic to use LEFT JOIN LATERAL with an added null filter and replaced the deprecated `useInnerJoin` flag with `preserveNullAndEmptyArrays`.
- **`pg_documentdb/src/configs/feature_flag_configs.c`**: Introduced the new boolean GUC `enableLookupInnerJoin` (with a default of false) to control join type.
- **Test Files (e.g., expected output and SQL tests under `pg_documentdb` and `pgmongo` directories)**: Modified expected query plans and test scripts to validate the updated lookup join behavior.
- **`pg_documentdb/src/customscan/custom_explain_scan.c`**: Propagated parameter information to custom explain scan outputs.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
The cron job for cursor cleanup on disk was broken since it was declared as a function but the call was done as a procedure. Fix the job command.
----
#### AI description (iteration 1)
#### PR Classification
This pull request fixes a bug related to the cursor cleanup cron job.
#### PR Summary
The changes address incorrect scheduling and command syntax for cursor cleanup tasks in different modules.
- `/pgmongo/sql/pgmongo--1.104-1--1.105-0.sql`: Added a plpgsql block to conditionally schedule the `helio_cursor_cleanup_task` if it does not exist.
- `/oss/pg_documentdb/sql/documentdb--0.104-0--0.104-1.sql`: Updated the cron job syntax for `documentdb_cursor_cleanup_task` to call the correct cleanup function.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Code cleanup and documentation updates according to the latest contribution guidelines
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Support order by descending as a feature under a feature flag if the underlying index can support it
2) Fix up memory performance issues causing OOMs with composite index with large numbers of bounds
3) On doing this noticed that row_get_bson was busted for int arrays, fix this behavior to not crash
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
cleanup statics file
----
#### AI description (iteration 1)
#### PR Classification
This pull request performs a code cleanup by standardizing and simplifying comments in the statistics operator functions.
#### PR Summary
The PR updates in-code comments to remove redundant references and streamline wording regarding null and numeric return behaviors in statistical functions.
- Updated `oss/pg_documentdb/src/aggregation/bson_aggregation_statistics.c` to replace mentions of "Mongo returns" with "Returns" across multiple functions, clarifying behavior for empty sets, single numeric values, and non-numeric inputs.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
No
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Cleanup vector tests
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
[oss] Cleanup to conform with contributing guidelines
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a code cleanup that standardizes naming conventions to conform with the contributing guidelines.
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) There were order by tests not running but had the sql files in there. This was catching contracts related to arrays with null values where 2 cases were ordered later than they should be. This fixes that. As part of that we noticed that this breaks when there's paths that have array indexes (e.g. a.b.0). This also fixes that contract.
2) Add checks to say all tests must run or exist in a schedule file and fix the output
3) There's flakeyness in currentop due to tracking the version refresh query - skip the version refresh query in currentop.
----
#### AI description (iteration 1)
#### PR Classification
This pull request fixes correctness issues in composite order-by functionality and improves associated testing.
#### PR Summary
The changes refine the handling of intermediate arrays in order-by operations by updating traversal and comparison logic, while enhancing the test suite and validation scripts for composite order-by scenarios.
- `pg_documentdb/src/query/bson_dollar_operators.c` and related files were modified to correctly process nested arrays and compare composite order-by values.
- Test files in `pgmongo/src/test/regress/expected/` and `pgmongo/src/test/regress/sql/` were updated to add new order-by cases and remove obsolete tests.
- Validation scripts (e.g. in `oss/internal/pg_documentdb_distributed/src/test/regress/validate_test_output.sh` and `oss/pg_documentdb/src/test/regress/validate_test_output.sh`) were enhanced for more reliable schedule checks and collection ID extraction.
- Several BSON traversal and update operator functions were updated to include an `isArrayIndexSearch` parameter for finer control during array element processing.
- `pg_documentdb/src/commands/current_op.c` and `pg_documentdb/src/utils/version_utils.c` were updated to skip version refresh queries appropriately.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
[oss] Cleanup to follow documentdb coding guidelines
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a code cleanup aimed at aligning the codebase with DocumentDB coding guidelines.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Testing
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
CurrentOp tests have flakeyness due to some background process. Log the SQL query for tests for currentOp so that we can track the source of flakeyness
----
#### AI description (iteration 1)
#### PR Classification
This pull request implements a new feature that adds SQL command details to the currentOp output for tests.
#### PR Summary
The changes update the expected test outputs and the server logic to include a new "sql" field in currentOp responses, which displays the actual SQL command executed for each operation.
- **`pgmongo/src/test/regress/expected/isolation-currentop.out`**: Updated expected outputs to include the "sql" field for various operations.
- **`oss/pg_documentdb/src/commands/current_op.c`**: Modified to append the SQL command in the output when the feature flag is enabled.
- **`oss/pg_documentdb/src/configs/testing_configs.c`**: Added a new configuration variable (`CurrentOpAddSqlCommand`) to control SQL command inclusion.
- **`pgmongo/src/test/regress/isolation_postgresql.conf`**: Enabled the SQL command logging feature in the test configuration.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
- Change .c files to align with contributing guidelines
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup and refactoring to align naming conventions with contributing guidelines.
#### PR Summary
This pull request standardizes naming and comments across several C source files by replacing outdated "mongo" terminology with "documentdb" or "collection" as appropriate. The changes improve consistency and clarity in code and documentation.
- **`oss/pg_documentdb/src/aggregation/bson_aggregation_window_operators.c`**: Renamed enums, macros, and validation checks from FRAMEOPTION_MONGO_* to FRAMEOPTION_DOCUMENTDB_* and updated related comments.
- **`oss/pg_documentdb/src/aggregation/index_stats.c`**: Updated variable names and output messages to use “collection” terminology and reflect wire compatible formats.
- **`oss/pg_documentdb/src/aggregation/bson_densify.c`**: Replaced isMongoDataTable with isCollectionDataTable and revised related comments and function calls.
- **Other files (`bson_tree.c`, `bson_unwind.c`, and testing_configs.c)**: Modified comments and error messages to remove Mongo-specific references, ensuring consistency with the new naming guidelines.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Lookup/$graphLookup pushdown to index was broken. An earlier attempt to fix it removed inner joins as a way since it reproed this. However, this could still repro in other cases, and was recently found in graphLookup too. The real root cause seems to be due to the removal of paramPath when converting index scan to bitmap scan- fix this.
----
#### AI description (iteration 1)
#### PR Classification
This PR fixes a bug causing crashes in $lookup/$graphLookup operations by switching to inner join semantics when null preservation is not required.
#### PR Summary
The changes refactor the lookup join processing to eliminate the crash by introducing a new boolean flag (`useInnerJoin`) in the lookup context and adjusting the join type accordingly, while also updating the tests to validate the new behavior.
- **`src/aggregation/bson_aggregation_nested_pipeline.c`**: Removed redundant null filtering logic and added the `useInnerJoin` flag to control join behavior.
- **`src/aggregation/bson_aggregation_nested_pipeline.c`**: Updated the `MakeLookupJoinRte` function to set the join type to INNER JOIN if `useInnerJoin` is true.
- **Test files in `/oss/internal/pg_documentdb_distributed/src/test/regress/expected/` and `/pgmongo/src/test/docdb_compat/expected/`**: Added new lookup inner join test cases and updated expected output files.
- **`src/customscan/custom_scan_continuation.c`**: Adjusted planner path handling to correctly propagate parameter path info in lookup-related queries.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Description
Code clean up. Finish checking for fill linearfill locf in .c .sql .out
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup improving inline comment clarity in a core aggregation function.
#### PR Summary
This pull request refines the inline comments in the BSON aggregation window function to remove redundant references and clarify expected numeric behaviors.
- `oss/pg_documentdb/src/aggregation/bson_aggregation_window_functions.c`: Updated comments in the `bson_linear_fill` function to remove "As Mongo does," and clearly describe that the filled value should be numeric or nullish, as well as outlining type expectations for return values.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
finish cleanning up of densify .c .sql
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup and refactoring.
#### PR Summary
This PR cleans up naming and comments in the densify module to enhance clarity and consistency by replacing Mongo-specific terminology with generic collection terminology.
- In `oss/pg_documentdb/src/aggregation/bson_densify.c`, renamed the variable from `isMongoDataTable` to `isCollectionDataTable` and updated its usage.
- In the same file, updated comments to remove outdated Mongo references and clarify functionality.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
update tests
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
comments
----
#### AI description (iteration 1)
#### PR Classification
This pull request performs code cleanup by revising comment texts.
#### PR Summary
The changes update comments in `bson_expression_type_operators.c` to replace references to "native mongo" with "protocol," ensuring the comments accurately reflect protocol behavior.
- `bson_expression_type_operators.c`: Updated comments in ValidateStringIsNotHexBase, ProcessDollarType, and ProcessDollarToDate to clarify protocol-specific behavior regarding hexadecimal validation, missing field paths, and unsupported int32 to date conversions.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Cleanup as per new change guideline, part -3
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a code cleanup that standardizes and clarifies inline comments across multiple command files.
#### PR Summary
The changes update comment wording and remove redundant explanations to align with new guidelines, improving overall code clarity and consistency. Key changes include:
- Updates in `coll_mod.c` to clarify lock acquisition and index TTL constraints.
- Revised comments in `coll_stats.c` and `db_stats.c` to standardize responses and scale conversion details.
- Simplified comment phrasing in `current_op.c` and `delete.c` by removing unnecessary Mongo-specific references.
- Cleaned up prevalidation explanations in `drop_indexes.c` and support option comments in `commands_common.c`.
- Removed redundant commentary in `find_and_modify.c` regarding field casing.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
[oss] Cleanup as per new change guideline
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a test cleanup update designed to standardize test collection names and expected outputs in both aggregation pipeline and set operator tests.
#### PR Summary
The changes update and rename test data to comply with new guidelines. Key modifications include:
- Updated expected output files for BSON aggregation “merge objects” tests in `/oss/internal/pg_documentdb_distributed/src/test/regress/expected/` and `/pgmongo/src/test/docdb_compat/expected/` by renaming collections (e.g. from `objColl1` to `mergeObjectsGroupColl`) and adjusting document content.
- Modified SQL test files in `/oss/pg_documentdb/src/test/regress/sql/` and `/oss/internal/pg_documentdb_distributed/src/test/regress/sql/` to reflect new collection names and query parameters.
- Revised BSON aggregation set operators tests to update date values (switching from 2019 timestamps to 2013 values) and associated expected outputs.
- Updated drop and shard collection commands accordingly to maintain consistency across tests.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
clean up bson_aggregation_pipeline comments
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
[OSS] Cleanup test files, aggregation and insert one
----
#### AI description (iteration 1)
#### PR Classification
Test cleanup and expected output update for aggregation pipelines and insert commands.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Other
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Aligning to contributing guideline for geospatial module
----
#### AI description (iteration 1)
#### PR Classification
This pull request performs a code cleanup in the geospatial module to improve clarity and consistency across comments, error messages, and test outputs.
#### PR Summary
The changes refine inline documentation and output messages to accurately describe the intended behavior of geospatial operators—especially for the $geoWithin operator—and standardize legacy processing with Postgis expectations.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Code cleanup as per new codiing guideline
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Fix up some more documentation and comments around the code.
----
#### AI description (iteration 1)
#### PR Classification
This PR performs a code and documentation cleanup by refactoring naming conventions and removing deprecated staging functions to align with the contributing guide.
#### PR Summary
The pull request removes obsolete C wrappers for staging collection operations and standardizes internal API naming from "MongoCatalog" to "ApiCatalog" while refining inline comments for clarity and consistency.
- **`src/metadata/collection.c` & `include/metadata/collection.h`**: Removed unused functions for handling staging collections during $out operations.
- **`src/metadata/metadata_cache.c` & `include/metadata/metadata_cache.h`**: Renamed functions and variables from "MongoCatalog" to "ApiCatalog" to standardize internal identifiers.
- **`src/aggregation/bson_aggregation_config_queries.c` & `src/aggregation/bson_aggregation_metadata_queries.c`**: Updated BSON aggregation queries to reference the new API naming.
- **`src/infrastructure/cursor_store.c`**: Enhanced documentation by updating comments and adding a shared memory structure for cursor tracking.
- **`src/metadata/index.c` & `src/index_am/rum.c`**: Applied minor comment corrections and removed outdated TODO notes for clarity.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Fixes a bug hit in production
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
TODO
### Description
1. Fix facet internal error on multiple facet.
2. Fix a minor potential bug on facet where alias name of a facet stage can conflict
3. Adds a faq on how the repro query was achieved
----
#### AI description (iteration 1)
#### PR Classification
Bug fix addressing an internal error in facet query planning.
#### PR Summary
This pull request fixes an internal error ("could not find plan for CTE 'facet_base_0'") by updating the Common Table Expressions (CTE) naming scheme and related query construction in facet aggregation pipelines.
- `oss/pg_documentdb/src/aggregation/bson_aggregation_nested_pipeline.c` now builds CTE names using both stage number and nested pipeline level, and the query creation call is adjusted accordingly.
- Test expectation files under `oss/internal/pg_documentdb_distributed/src/test/regress/expected/` and `pgmongo/src/test/docdb_compat/expected/` are updated to reflect the new CTE naming (e.g. from `facet_base_0` to `facet_base_0_0`).
- Additional test queries and SQL scripts in `oss/internal/pg_documentdb_distributed/src/test/regress/sql/` are added with the corrected CTE names.
- Documentation (`docs/faqs.md`) is updated with refined instructions for debugging queries with prepared statements.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #3741094
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Code cleanup and documentation updates to match contribution guidelines
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a code cleanup and documentation update.
#### PR Summary
The updates refine inline comments and documentation across update, error handling, and collation components to improve clarity and consistency, while replacing obsolete Mongo references with DocumentDB terminology. Key changes include:
- **`oss/pg_documentdb/src/update/bson_update_operators.c`**: Revised comments for update operators (e.g., `$currentDate`, `$push`, `$mul`) to clarify behavior and update examples.
- **`oss/pg_documentdb/src/utils/error_utils.c`**: Updated documentation to reference DocumentDB errors instead of Mongo errors.
- **`oss/pg_documentdb/src/update/bson_update.c` and `oss/pg_documentdb/src/update/bson_update_aggregation.c`**: Improved inline comment wording to accurately describe update specifications and error conditions.
- **`oss/pg_documentdb_core/src/collation/collation.c`**: Cleaned up collation conversion comments and locale handling explanations.
- **Test files under `oss/internal/pg_documentdb_distributed/src/test` and `pgmongo/src/test/docdb_compat/expected`**: Updated expected output comments to align with the revised error message tests.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Fix up docs across some files to match contributing guidelines.
----
#### AI description (iteration 1)
#### PR Classification
This PR focuses on documentation improvements and in-code comment cleanup to comply with contributing guidelines.
#### PR Summary
The pull request refines comment texts and error messages across several source files, removes unused code documentation, and updates internal clarifications.
- `oss/pg_documentdb/src/aggregation/bson_aggregates.c`: Removed the unused `BsonOutAggregateState` typedef and updated aggregation comments (e.g., changing "HASHTBL" to "HASHCTL") while eliminating redundant code comments.
- `oss/pg_documentdb/src/commands/create_collection_view.c`: Removed several unused fields from the `CreateSpec` structure and updated header comments to reflect the correct file path.
- `oss/pg_documentdb/src/aggregation/bson_aggregation_pipeline.c`: Revised comment formatting and error messages in aggregation and query generation functions to ensure consistent terminology.
- `oss/pg_documentdb/src/auth/scram256.c` & `oss/pg_documentdb/src/commands/update.c`: Updated documentation comments for authentication responses and update error handling for improved clarity.
- Other files in `/oss/pg_documentdb/src/opclass/` and `/oss/pg_documentdb_core/`: Corrected typographical errors and adjusted error code references to align with the project’s guidelines.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
3 c files
----
#### AI description (iteration 1)
#### PR Classification
This pull request focuses on code cleanup and documentation updates.
#### PR Summary
The changes standardize and simplify documentation comments and examples across several operator implementations, removing obsolete references and unnecessary details.
- `oss/pg_documentdb/src/aggregation/bson_aggregation_output_pipeline.c`: Updated doc examples for `$merge` and `$out` to use generic placeholders and removed temporary test-related comments.
- `oss/pg_documentdb/src/query/bson_dollar_operators.c`: Cleaned up comments for multiple BSON dollar operators by eliminating "Mongo's" references and redundant test notes.
- `oss/pg_documentdb_core/src/types/pcre_regex.c`: Refined regex compile option comments and removed legacy test reference lines.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Performance
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Currently, each round of the sproc that does create indexes does exactly 1 index build and bails. This means if a single client request queues 4 indexes and the job runs every 2 seconds, it'll take 8 seconds to build all 4 indexes. This is wasteful since we know all 4 indexes up front.
Add a loop to the index build process so that multiple indexes can be built without waiting the 2 seconds.
As part of this, there were a few places allocating in memory contexts that are unstable across executions. Fix this.
Additionally there was a loop that could cause index builds to starve for ~20 mins if it picked the wrong index. Fix this loop too.
Finally, Noticed that the version cache does not cache properly if the Major version is 0. Fix this (which can cause perf regressions in DocumentDB).
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
[OSS] fix 0.104-1 hotfix upgrade scripts
----
#### AI description (iteration 1)
#### PR Classification
Hotfix update to upgrade SQL upgrade scripts correcting version references.
#### PR Summary
This pull request updates SQL upgrade scripts to use the correct UDF and documentdb version numbers.
- In `pgmongo/sql/pgmongo--1.103-0--1.103-1.sql`, the include directive was updated from version 0.105-0 to 0.104-1.
- In `oss/pg_documentdb/sql/documentdb--0.104-0--0.104-1.sql`, the UDF include reference was changed to use version 0.104-1.
- The file `oss/pg_documentdb/sql/documentdb--0.104-0--0.105-0.sql` was moved to `oss/pg_documentdb/sql/documentdb--0.104-1--0.105-0.sql` with no code changes.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes, the API schema
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
1.05
### Description
We are adding support for the Connection Status command, which exists in native Mongo, and allows users to see their authenticated user, authenticated roles, and user privileges. For more information, visit connectionStatus.md.
This PR is the second of three PRs for this feature. The first one added extensible role privileges to our `users_info` command, allowing users to retrieve role privileges with `showPrivileges: true`. This second PR adds a callable `connection_status` command to our API, that utilizes the role privileges. The last PR will add a gateway change that calls this `connection_status` command.
Related work items: #4287246
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
Enhancement and bug fix to improve the stability and management of file-based cursors.
#### PR Summary
This pull request refactors and bolsters file-based cursor management by introducing shared memory support, improved cleanup, error handling, and configuration changes. It also adds new API endpoints and logging for better cursor lifecycle control.
- **`cursor_store.c` and related files:** Added shared memory functions (e.g., `FileCursorShmemSize`, `InitializeFileCursorShmem`), cursor count regulation (with `IncrementCursorCount`/`DecrementCursorCount`), and cleanup routines to ensure stable cursor file operations.
- **Test and expected output files:** Updated regression tests and expected outputs to cover scenarios like duplicate cursor creation, cursor deletion limits, and error reporting when the file cursor capacity is reached.
- **API enhancements:** Introduced new UDFs (e.g., `command_delete_cursors`) and tenant client methods for killing and deleting cursors, improving session management.
- **Configuration updates:** Renamed and added GUC parameters (e.g., `MaxAllowedCursorIntermediateFileSizeMB`, `MaxCursorFileCount`, and cursor expiry settings) to control file-based cursor behavior.
- **Logging and metrics improvements:** Enhanced logging in `mdsd_logger.c` to report current cursor counts and related metrics for operational insight.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Removing unnecessary in/out from bson_update_document which were preset eariler due to older change_stream implementation.
1. For now moving udf to helio_api_internal and same time removing useless in/out params
TODO:
In next release will modify this for oss with updating test and minimal c changes will update --latest files also.
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a code cleanup and update to simplify the integration of the `bson_update_document` function into OSS.
#### PR Summary
The PR introduces a new C-based SQL function for `bson_update_document` with streamlined parameters, updates the main SQL script to include this new file, and refreshes the API schema output to reflect the change.
- `pgmongo/sql/udfs/crud/bson_update_document--1.105-0.sql`: Adds the new function definition with cleaned-up parameter handling.
- `pgmongo/sql/pgmongo--1.104-0--1.105-0.sql`: Updates the script to include the newly added UDF file.
- `pgmongo/src/test/regress/expected/public_api_schema.out`: Modifies the schema output to include the new function entry.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
setting guc enableDataTableWithoutCreationTime to on so that all the new tables will not have creation_time column
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a new feature change that enables omitting the creation_time field from query results.
#### PR Summary
The PR updates the behavior of the system by setting the GUC flag enableDataTableWithoutCreationTime to true, so that output from various commands no longer includes the creation_time column.
- In `configs/feature_flag_configs.c`, the default for enableDataTableWithoutCreationTime is switched to true.
- Expected output files in multiple test folders (`oss/pg_documentdb`, `oss/internal/pg_documentdb_distributed`, and `pgmongo`) have been revised to remove the creation_time field.
- Test SQL scripts have been updated to set the flag as needed, ensuring consistency between the new behavior and test expectations.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Add support of altrernate index handler in 1.103
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
`deleteOne` requires `_id` or shard key value filter. If only `_id` is used, we query for the minimum shard key value and use it.
The querying for that shard key must be able to use the `variableSpec`.
----
#### AI description (iteration 1)
#### PR Classification
API change to enable proper handling of deleteOne commands on sharded collections using only an _id filter by incorporating variable specification support.
#### PR Summary
This pull request enhances deleteOne functionality by integrating a variableSpec (let variable) into query construction for sharded collections, ensuring that _id-only filters are handled correctly. The changes update core command functions, test expectations, and configuration flags to support the new behavior.
- **`commands_common.c`**: Revised the `FindShardKeyValueForDocumentId` function to accept a new variableSpec parameter and conditionally modify the query.
- **`delete.c` and `update.c`**: Updated deletion and update operations to process the variableSpec for write commands.
- **Test and Configuration Files**: Renamed and updated GUC flags (from enableLet... to enableVariablesAndCollationSupportForQueryMatch) and adjusted expected outputs to reflect the new deleteOne behavior.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
This pull request implements a new feature that enhances explain plan output and composite index functionality by adding order‑by pushdown and related index option validations.
#### PR Summary
The changes add new metrics in explain plans and strengthen composite index handling for order-by pushdown across the system. Key changes include:
- **`/gateway/managed/Mongo.Service.Postgres/Helpers/PostgresExplainHelper.cs` & `/oss/pg_documentdb_gw/src/explain/mod.rs`** – Added fields like `sortKeysCount`, `sortKey`, `presortedKey(s)`, and `hasOrderBy` to enhance explain output.
- **`/oss/pg_documentdb/src/metadata/index.c` and related index management files** – Implemented improved composite index option checks and support functions such as `CompositeIndexSupportsOrderByPushdown` and equality-range predicate validation.
- **UDF and operator files (e.g., `bson_dollar_comparison--*.sql`, `/oss/pg_documentdb/src/query/bson_dollar_operators.c`)** – Introduced new functions like `bson_dollar_fullscan` to support full scan pushdowns in combination with composite indexes.
- **`/pgmongo_rum/src/rumget.c`** – Enhanced ordered scan handling and duplicate management for multi-key indexes.
- **Pipeline and configuration files (e.g., `.pipelines/templates/single_node_tests.yml`)** – Updated test scripts and experimental configuration parameters to enable the new composite opclass and order‑by pushdown features.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
[DocumentDB issus 178](https://github.com/microsoft/documentdb/issues/178)
When the document size exceeds 2000, the tuple is toasted. We need to detoast the datum before converting it to pgbson.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix addressing improper detoasting of pgbson values.
#### PR Summary
This pull request replaces direct pointer extraction and SPI_datumTransfer calls with dedicated DatumGetPgBson macros to ensure both toasted and non-toasted bson documents are correctly handled. It also enhances test coverage for find-and-modify operations on large documents.
- `commands_find_and_modify.out` and SQL test files: Added new queries for deleting large (toasted) and non-toasted documents.
- `retryable_writes.c`, `delete.c`, metadata, index, and query operator files: Replaced manual SPI_datumTransfer and DatumGetPointer calls with DatumGetPgBson (and DatumGetPgBson_MAYBE_NULL) to properly detoast and copy pgbson objects.
- `pgbson.h` and `bson_io.c`: Introduced and integrated new macros to streamline detoasting of pgbson data.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
More general improvements to order by and composite:
Create Indexes
1) Support creating a composite index side by side with a regular index
2) Block creation of descending or wildcard and unsupported set ups for composite indexes
Planner:
1) Support ordering by the Nth path of an index if the prefixes are equality.
2) Validate that if it's not the first column, prefixes are provided
3) Support incremental sort with index pushdown.
----
#### AI description (iteration 1)
#### PR Classification
This pull request delivers a new feature and API enhancement to enable order-by index pushdown and refined composite index support.
#### PR Summary
The update streamlines composite index processing by introducing stricter equivalency checks, new UDF functions for full-scan order-by pushdown, and enhanced validations during index creation while updating explain outputs and tests.
- `src/metadata/index.c` refines index options equivalency logic to correctly compare composite term settings.
- `src/opclass/bson_gin_composite_entrypoint.c` and `src/index_am/rum.c` add functions to support order-by pushdown and manage duplicate entries efficiently.
- `sql/udfs/query/bson_dollar_comparison--0.105-0.sql` (and related operator files) introduce new UDFs including `bson_dollar_fullscan` to enable full scan operations.
- `src/commands/create_indexes.c` enforces new composite index validations (restricting use to ascending, non-wildcard regular indexes) with precise error reporting.
- Test scripts and explain plan outputs in the `regress` and `pgmongo` directories are updated to confirm the new composite order-by behavior.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
- Fix $match filters not applied after preserving null on $lookup and $unwind combinations
### Description
This fixes the OSS issue: https://github.com/microsoft/documentdb/issues/198
- Starting PG 16 the the right join query that will have NULL results need to assign `varnullingrels` index of RTE which can replace the NULL values for outer JOIN conditions, if the VAR expression is a result of JOIN.
This is seen as the diff in the parse plan of not working and working case of the same query in PG 16
The final query described in the bug has below format and the last `WHERE` filter can merge with the inner right query if `varnullingrels` is not specified which yields incorrect results
```SQL
SELECT document
FROM ( WITT debug_print_parse to on;
LECT document
FROM ( WITH "lookupLeftCte_0" AS (
SELECT collection.document,
bson_dollar_lookup_extract_filter_expression(collection.document, '{ "name" : "author" }'::bson) AS lookup_filter
FROM documentdb_data.documents_3 collection
WHERE collection.shard_key_value = '3'::bigint
)
SELECT bson_dollar_merge_documents_at_path(lookup_stage_1.document, COALESCE("lookupRight_stage_1".document, '{ }'::bson), 'author-names'::text) AS document
FROM "lookupLeftCte_0" lookup_stage_1
LEFT JOIN LATERAL ( WITH lookup_right_query AS (
SELECT collection_0_1.document
FROM documentdb_data.documents_2 collection_0_1
WHERE collection_0_1.shard_key_value = '2'::bigint
)
SELECT lookup_right_query_stage_0.document
FROM lookup_right_query lookup_right_query_stage_0
WHERE bson_dollar_lookup_join_filter(lookup_right_query_stage_0.document, lookup_stage_1.lookup_filter, 'name'::text)) "lookupRight_stage_1" ON true) agg_stage_1
WHERE document #= '{ "title" : "Emma" }'::bsonquery;
```
----
#### AI description (iteration 1)
#### PR Classification
Bug fix addressing the incorrect application of match filters in aggregation pipelines combining $lookup and $unwind stages.
#### PR Summary
This pull request fixes the issue by updating the aggregation pipeline processing logic for null-preserving $lookup/$unwind operations and enhancing test coverage to verify correct filtering.
- **`oss/p...
Vinod Sridharan
Deepak Pemmaraju
William Wang
Deepak Bisht
Parag Jain
CosmosDB Build Service (msdata)
Duan Pei
Zinan Ren
Jayanta Mondal
Xing Fan
Santiago Fernandez Madero
Jinrui Zhang
Shuai Tian
Jia Hu
Cornelius Boateng
