### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Move Location13103 error to documentdb_codes.txt
----
#### AI description (iteration 1)
#### PR Classification
Bug fix: Moving an error code definition to the correct file.
#### PR Summary
This pull request corrects the placement of the "Location13103" error by removing it from one file and adding it into the proper location.
- `pg_documentdb_gw/include/gateway_errors.txt`: Removed the "Location13103" error entry.
- `pg_documentdb_core/include/utils/documentdb_codes.txt`: Added the "Location13103" error entry.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) There were order by tests not running but had the sql files in there. This was catching contracts related to arrays with null values where 2 cases were ordered later than they should be. This fixes that. As part of that we noticed that this breaks when there's paths that have array indexes (e.g. a.b.0). This also fixes that contract.
2) Add checks to say all tests must run or exist in a schedule file and fix the output
3) There's flakeyness in currentop due to tracking the version refresh query - skip the version refresh query in currentop.
----
#### AI description (iteration 1)
#### PR Classification
This pull request fixes correctness issues in composite order-by functionality and improves associated testing.
#### PR Summary
The changes refine the handling of intermediate arrays in order-by operations by updating traversal and comparison logic, while enhancing the test suite and validation scripts for composite order-by scenarios.
- `pg_documentdb/src/query/bson_dollar_operators.c` and related files were modified to correctly process nested arrays and compare composite order-by values.
- Test files in `pgmongo/src/test/regress/expected/` and `pgmongo/src/test/regress/sql/` were updated to add new order-by cases and remove obsolete tests.
- Validation scripts (e.g. in `oss/internal/pg_documentdb_distributed/src/test/regress/validate_test_output.sh` and `oss/pg_documentdb/src/test/regress/validate_test_output.sh`) were enhanced for more reliable schedule checks and collection ID extraction.
- Several BSON traversal and update operator functions were updated to include an `isArrayIndexSearch` parameter for finer control during array element processing.
- `pg_documentdb/src/commands/current_op.c` and `pg_documentdb/src/utils/version_utils.c` were updated to skip version refresh queries appropriately.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
Enhancement and bug fix to improve the stability and management of file-based cursors.
#### PR Summary
This pull request refactors and bolsters file-based cursor management by introducing shared memory support, improved cleanup, error handling, and configuration changes. It also adds new API endpoints and logging for better cursor lifecycle control.
- **`cursor_store.c` and related files:** Added shared memory functions (e.g., `FileCursorShmemSize`, `InitializeFileCursorShmem`), cursor count regulation (with `IncrementCursorCount`/`DecrementCursorCount`), and cleanup routines to ensure stable cursor file operations.
- **Test and expected output files:** Updated regression tests and expected outputs to cover scenarios like duplicate cursor creation, cursor deletion limits, and error reporting when the file cursor capacity is reached.
- **API enhancements:** Introduced new UDFs (e.g., `command_delete_cursors`) and tenant client methods for killing and deleting cursors, improving session management.
- **Configuration updates:** Renamed and added GUC parameters (e.g., `MaxAllowedCursorIntermediateFileSizeMB`, `MaxCursorFileCount`, and cursor expiry settings) to control file-based cursor behavior.
- **Logging and metrics improvements:** Enhanced logging in `mdsd_logger.c` to report current cursor counts and related metrics for operational insight.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
[DocumentDB issus 178](https://github.com/microsoft/documentdb/issues/178)
When the document size exceeds 2000, the tuple is toasted. We need to detoast the datum before converting it to pgbson.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix addressing improper detoasting of pgbson values.
#### PR Summary
This pull request replaces direct pointer extraction and SPI_datumTransfer calls with dedicated DatumGetPgBson macros to ensure both toasted and non-toasted bson documents are correctly handled. It also enhances test coverage for find-and-modify operations on large documents.
- `commands_find_and_modify.out` and SQL test files: Added new queries for deleting large (toasted) and non-toasted documents.
- `retryable_writes.c`, `delete.c`, metadata, index, and query operator files: Replaced manual SPI_datumTransfer and DatumGetPointer calls with DatumGetPgBson (and DatumGetPgBson_MAYBE_NULL) to properly detoast and copy pgbson objects.
- `pgbson.h` and `bson_io.c`: Introduced and integrated new macros to streamline detoasting of pgbson data.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
This pull request adds documentation for the `$bucketAuto` aggregation stage in PostgreSQL, detailing its design, usage, and implementation.
- Adds `bucketAuto.md` with details on the `$bucketAuto` aggregation stage
- code implementation
- support all granularity types: POWERSOF2, 1-2-5, Renard numbers, E series
Related work items: #3234185
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix/Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Orderby on null/undefined documents was incorrect - fix up the ordering
2) Equality on null/undefined was also incorrect fix this up
3) When traversing the document, we were doing an extra traversal into arrays when it was not needed. fix this up
4) Order by on composite index didn't handle nulls correctly - fix this too.
5) Support pushdown of $elemMatch into composite indexes without array terms.
----
#### AI description (iteration 1)
#### PR Classification
This pull request fixes bugs in the composite index and orderby functionality.
#### PR Summary
The changes correct the orderby comparison logic and improve composite index term generation in BSON processing. The pull request also adds new functions to handle intermediate array paths and introduces legacy behavior flags, while supplementing the fixes with comprehensive integration tests.
- **`src/query/bson_dollar_operators.c`**: Modified `OrderByVisitArrayField` to pass the correct BSON value and added new functions for array path handling.
- **`src/opclass/gin_index_term.c` and related opclass files**: Introduced `GenerateValueMaybeUndefinedTerm` and updated index term comparison to distinguish between undefined and partially undefined values.
- **`pgbson_utils.c`**: Adjusted array traversal logic with a new callback for missing intermediate array paths and a flag to control array traverse optimization.
- **Test files under `/oss/pg_documentdb/src/test/regress/expected/` and `/pgmongo/src/test/regress/expected/`**: Added extensive composite ordering and filtering tests to validate the corrected behavior.
- **`configs/feature_flag_configs.c`**: Added custom GUCs for `useLegacyOrderByBehavior` and `useLegacyNullEqualityBehavior` to manage legacy ordering and null equality rules.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Currently for Update/Delete, we currently parse the update spec, query spec, etc into pgbsons on startup. Then reuse it per commands.
This has a few issues - for large document requests, this causes us to duplicate memory and persist it for the duration of the update and also transcode all up front - this is worse for multi-node when all we need to do is build yet another update spec and push it to the worker.
Instead, we now only cache pointers to the spec document; then when we need it we convert to pgbson only if necessary.
A consequence of this is that on a single node update, if we're doing a simple updateOne on _id then we would never serialize the updateSpec at all (for instance).
We also more eagerly free memory on the intermediate update specs.
----
#### AI description (iteration 1)
#### PR Classification
Performance optimization for update parsing by refactoring BSON value handling.
#### PR Summary
This PR streamlines update parsing by replacing heavy BSON initialization routines with lightweight wrappers and converting update-related types from `pgbson*` to `bson_value_t*`, which reduces overhead and improves coordinator/worker compatibility.
- `src/commands/update.c`: Replaced multiple `PgbsonInitFromDocumentBsonValue` and similar calls with efficient wrappers like `CreateBsonValueFrom` and updated serialization (including back-compatible array filter and update document formats).
- `include/commands/update.h`: Updated update structure fields to use `bson_value_t*` instead of `pgbson*`, aligning interfaces with the new parsing approach.
- `src/update/bson_update.c` and `bson_update_operators_workflow.c`: Refactored update type determination and replace document logic to use new BSON value iterators and helper functions.
- Other modules (e.g. `commands/find_and_modify.c`, `commands/delete.c`, query operators, sharding) were updated to leverage the lighter-weight BSON representation and its associated helper routines.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
When servers have default_readonly_transaction set, the current behavior is wonky - it will cause failures in underlying queries and pollute the errors, and it will go through a bunch of work only to fail.
Additionally, background jobs like TTL will fail ungracefully.
Handle this GUC by failing write commands early with a more useful error message.
Additionally, in cases like TTL, add an opt-in way to prune rows even in the presence of this GUC if it's opted into.
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a bug fix that improves error handling for write operations attempted during default read-only transactions.
#### PR Summary
The changes introduce a unified mechanism to check and enforce read-only transaction restrictions, ensuring graceful errors when write operations are attempted while in a read-only state. Key changes include:
- **`include/commands/commands_common.h`**: Added the inline function `ThrowIfServerOrTransactionReadOnly` to validate transaction state and throw appropriate errors.
- **Test files**: New regression tests under `/oss/pg_documentdb/src/test/regress/expected/` and `/oss/pg_documentdb/src/test/regress/sql/readonly_transaction_tests.sql` validate the expected error responses under read-only conditions.
- **`ttl_index.c`**: Modified TTL job logic to handle read-only transactions, allowing jobs to commit and restart when the corresponding flag is enabled.
- **Gateway and error handling**: Updated configuration in `TenantConfigurationBase.cs`, error augmentation in `PostgresMongoResultExtensions.cs`, and error mappings (e.g., adding error code 10107 for `NotWritablePrimary`) to propagate clear error messages.
- **Command modules**: Added calls to `ThrowIfServerOrTransactionReadOnly` in several command implementations (e.g., in `insert.c`, `update.c`, `delete.c`, `create_indexes.c`) to enforce read-only restrictions uniformly.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support collation with comparison of `array` and `document` values.
Also adds support for `$in` on nested arrays and documents.
----
#### AI description (iteration 1)
#### PR Classification
New feature: Adds support for collation-aware hashing of arrays and documents in BSON.
#### PR Summary
This pull request introduces collation-aware hashing for BSON arrays and documents, enhancing the handling of collation in hash sets. It refactors existing hash functions to accommodate collation strings and recursive hashing of nested BSON structures.
- `hashset_utils.c`: Implements collation-aware hashing functions for BSON values, including recursive handling of arrays and documents.
- `bson_expression_set_operators.c`: Replaces existing hash set creation with collation-aware hash set creation.
- `bson_dollar_operators.c`: Updates logic to use collation-aware hash sets for `$in` queries, removing unsupported scenarios.
- `hashset_utils.h`: Adds a new struct `BsonValueHashEntry` to store BSON values with collation strings.
- `collation.h`: Moves `IsBsonTypeCollationAware` function to handle collation applicability checks.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4111861, #4115239
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
This pr adds a Rust gateway for documentdb in the pg_documentdb_gw folder
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) InitializeBsonIndexTerm called BsonValueToPgbsonElement that was using the bson_iter_next a lot. This is a lot of overhead when we know it's a single element. Use an unsafe variant of it to generate a pgbsonelement
2) Introduce a MatchAndRecurse option for index paths that allows the term generation logic to know whether it's an exact match or if the child paths also match. This is currently unused, but will be used in a subsequent change.
----
#### AI description (iteration 1)
#### PR Classification
Code improvement for term generation in wildcard index.
#### PR Summary
This pull request enhances the term generation process for wildcard indexes by introducing an unsafe transformation method and refining index traversal options.
- `pgbsonelement.c`: Added `FillPgbsonElementUnsafe` and `BsonValueToPgbsonElementUnsafe` functions for unsafe BSON element transformation.
- `bson_gin_entrypoint.c`: Introduced `IndexTraverse_MatchAndRecurse` option to improve path matching and recursion.
- `feature_flag_configs.c`: Added feature flag `indexTermUseUnsafeTransform` to enable unsafe transformation.
- `gin_index_term.c`: Utilized the unsafe transformation method based on the new feature flag.
- `bson_gin_core.c`: Updated term path generation to handle the new traversal option.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces CRUD operations for Entra users in the DataPlane.
- `authentication.c`: Added functions to create users with external identity providers, validate identity provider info, and check if the current user is an Entra admin.
- `users.h`: Added new header file defining user CRUD functions and related structures.
- `users.c`: Updated user creation logic to handle external identity providers and added parsing for custom data.
- `api_hooks.c`: Added hooks for creating users with external identity providers and checking user privileges.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support conversion to binData from string and vice versa.
$convert to BinData
```
"$convert": {
"input": <string>,
"to": {
"type": "binData",
"subtype": <bson_subtype_t>
},
"format": <"base64"/"string"/"hex"/"uuid"/"base64url">
}
```
OR
```
--- subtype will default to 0
"$convert": {
"input": <string>,
"to": "binData",
"format": <"base64"/"string"/"hex"/"uuid"/"base64url">
}
```
BinData $toString
```
{ $toString: <binData expression> }
```
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces support for converting data to `BinData` using the `$convert` operator.
- `src/operators/bson_expression_type_operators.c`: Added new functions and modified existing ones to handle `BinData` conversion, including `ProcessDollarToBinData`, `GetToTypeAndSubTypeForBinDataConversion`, and `ValidateAndGetBinDataConversionSubTypeAndFormat`.
- Refactored `ParseDollarConvert` and `ApplyDollarConvert` to accommodate the new `BinData` conversion logic.
- Introduced a new struct `ConvertToTypeArguments` to manage conversion arguments.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support new syntax of $getField
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for the extended `$getField` operator for version 8.0.
- Added new test file `postgresjs/expression_get_field_80_custom.js` to validate `$getField` operator behavior.
- Updated `bson_expression_object_operators.c` to handle new error codes and validation for `$getField`.
- Modified `bson_aggregation_object_operators_tests.sql` and related expected output files to include new test cases for `$getField`.
- Added new error code mappings in `documentdb_errors.csv` and `error_mappings.csv`.
- Updated `documentdb_codes.txt` and `mongo_codes.csv` to include new error codes.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Log Connection open events in the GW
2) Add current backend PID to notices from BE in GW
3) Add ErrDetail_log support for lock statements in GW
4) Don't continue Insert/Update/Delete on Query cancellation.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request addresses livesite issues by adding logging and fixing continuation on query cancellation.
- `error_utils.h`: Added `IsOperatorInterventionError` function to handle specific error codes.
- `update.c`, `delete.c`, `insert.c`: Integrated `IsOperatorInterventionError` to rethrow specific errors.
- `PostgresClientProvider.cs`: Added logging for physical connection opens.
- `PostgresClient.cs`: Enhanced logging to include process ID and hint details.
- `logger_hooks.c`: Modified logging behavior to promote detail logs to hints.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
- Support top level let variables in $geoNear stage
### Description
Support top level let variables in $geoNear stage
- Today there are 2 scenarios where `let` is used with $geoNear
1- `top level`: Any `let` provided in the top level command such as `aggregate`, the variables used in geonear spec should result into a `constant`, so we evaluate and rewrite the geoNear spec before parsing it to replace any expressions/let variables from top level let.
2- `nested let`: This scenario occurs with lookup pipeline and let is a combination of top level let and lookup let, this case is not optimized in this PR and will still return in error
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for top-level `let` expressions within the `$geoNear` stage in aggregation pipelines.
- `oss/pg_documentdb/src/aggregation/bson_aggregation_pipeline.c`: Added logic to handle `let` expressions by updating the `$geoNear` specification if a `CONST let` expression is present in the context.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Other
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
`documentdb.aggregation_stages_limit`- sets the max number of pipeline stages allowed in aggregations. (validates even the nested pipelines)
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
- Limit aggregation stages to 1000
### Description
Limit aggregation stages to 1000 via the GUC introduced.
This limit is enforced in newer server version > 7.1 , making it similar would have needed us to introduce api hooks as document db doesn't have server version. so we went ahead with simple GUC based approach to apply the limit, because this is rare limit to hit
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces a limit on the number of aggregation stages in a pipeline to 1000 stages.
- `bson_aggregation_pipeline_private.h`: Added a helper function `CheckMaxAllowedAggregationStages` to enforce the stage limit.
- `bson_aggregation_pipeline.c`: Integrated the stage limit check within the `ExtractAggregationStages` function.
- `system_configs.c`: Defined a new configuration variable `aggregation_stages_limit` to set the maximum allowed stages.
- `bson_aggregation_pipeline_tests.sql` and `bson_aggregation_pipeline_tests.out`: Added tests to verify the stage limit enforcement.
- `documentdb_errors.h` and related error mapping files: Introduced a new error code for exceeding the stage limit.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No
### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Store the value of $$NOW in the variable spec as `now`.
Pass it through the ParseAggregationContext to be utilized by operators that implement the new aggregation framework.
Persist value in the cursor's continuation document to be used for get_more queries.
Enable/Disable with `EnableNowSystemVariable` GUC.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces the `NOW_SYS_VARIABLE` to the BSON aggregation pipeline, allowing the use of the current timestamp in aggregation operations.
- Modified `bson_aggregation_pipeline_tests_add_to_set_group.out` to include `NOW_SYS_VARIABLE` in the BSON expressions for various aggregation operations.
Related work items: #3613264, #3613268
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
$mergeObjects Operator performance Optimization. Earlier for merging two object we were creating bson tree and populating values on it which was time complex but now we are using hashmap to update existing values and linklist to maintain the order.
Numbers with collection has 1 document and array has 2 objects with different numbers of element in it.
 ----
#### AI description (iteration 1)
#### PR Classification
Performance optimization
#### PR Summary
This pull request optimizes the `$mergeObject` operation by introducing a new ordered hash set for BSON elements.
- `oss/pg_documentdb_core/src/utils/hashset_utils.c`: Added functions to create and manage an ordered hash set for BSON elements.
- `oss/pg_documentdb/src/aggregation/bson_aggregation_output_pipeline.c`: Removed redundant hash set creation and related functions.
- `oss/pg_documentdb/src/operators/bson_expression_object_operators.c`: Refactored to use the new ordered hash set for BSON elements, improving the `$mergeObject` operation.
- `oss/pg_documentdb_core/include/utils/hashset_utils.h`: Added definitions for the new ordered hash set structure and functions.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
Bug fix to remove Personally Identifiable Information (PII) from user CRUD error messages.
#### PR Summary
This pull request ensures that PII is not leaked in error messages during user CRUD operations by only logging error hints for specific internal errors.
- `PostgresMongoClient.cs`: Added checks to log only error hints for error code "42710" to prevent PII leakage in `CreateUserAsync`, `UpdateUserAsync`, `DropUserAsync`, and `UsersInfoAsync` methods.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
draft
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces new aggregation functions and support for t-digest in the project.
- Added `bson_tdigest.c` to implement t-digest helper functions.
- Created new SQL aggregate functions in `group_aggregates--1.23-0.sql` and `group_aggregates_support--1.23-0.sql`.
- Updated `bson_aggregation_pipeline.c` to handle the new `$percentile` accumulator.
- Modified `metadata_cache.c` and `metadata_cache.h` to include OID for the new `BsonPercentileAggregateFunction`.
- Updated migration scripts `pg_helio_api--1.22-0--1.23-0.sql` and `pgmongo--1.22-0--1.23-0.sql` to include new SQL files.
### Does this PR have any customer impact?
yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes, UDF
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Implement for stage redact
Related work items: #1798586
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
the PR to fix some cases of jstests/core/server6238.js, 
15 line of case cannot be fixed yet, because `$a` is not a operator, in group stage should report `InvalidPipelineOperator`, in project stage, should report error code `31325` . so to fix the case, we need to distinguish the group stage and the project stage in ParseAggregationExpressionData Function.
fix some cases of jstest.
when field is start with "$", ErrorCode should is 16410.
example:
```
db.coll1.aggregate([
{
$project: {
$a: "$a"
}
}
]);
```
when field is invalidOperator, ErrorCode should is 40236
```
db.jstest1.aggregate([
{
$group: {
_id: "$_id",
$a: {
$sum: 1
}
}
}
]);
```
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request addresses a bug in the jstest_server by adding a new error code and updating related error handling logic.
- `helio_conversion_codes.c`: Added handling for `ERRCODE_HELIO_LOCATION40236`.
- `bson_tree.c` and `bson_io.c`: Updated error codes to use `ERRCODE_HELIO_LOCATION40236`.
- `helio_errors.h`, `helio_codes.txt`, `helioerrors.csv`, `mongoerrors.csv`, and `mongo_codes.csv`: Added definitions and entries for `ERRCODE_HELIO_LOCATION40236`.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
This is the PR to fix js error of jstests/core/server1470.js
In upsert case, TraverseQueryDocumentAndProcess skips processing $ref or $id, for example, given sample query:
`db.server1470.update({
"name": "first",
"pic": {"$ref": "foo", "$id": ObjectId("4c48d04cd33a5a92628c9af6")}
}, {$set: {x: 1}}, true, true);`
"pic" field is skipped and not inserted in the collection.
Changes includes:
1. Support $ref and $id in upsert case
2. handle error cases
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for DBRef in upsert operations.
- Modified `bson_query.c` to handle `$ref` and `$id` operators during upsert.
- Updated `query_operator.c` to include error handling for DBRef fields.
- Added new test cases in `commands_update.sql` and `commands_update.out` to validate DBRef upsert functionality.
- Introduced new error codes in `helio_errors.h`, `helio_codes.txt`, and related CSV files for DBRef validation errors.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Start porting schemas for SQL in documentdb_core.
Maintain back-compat by introducing CORE_SCHEMA_v2 to handle cases where the existing logic already depends on there being 2 core schemas
----
#### AI description (iteration 1)
#### PR Classification
Code refactoring to port schemas into `documentdb_core`.
#### PR Summary
This pull request refactors the code to port BSON type definitions and related functions into `documentdb_core`.
- `pg_helio_core/sql/types/bson--1.10-0.sql`: Replaced BSON type definitions with an include directive.
- Added `pg_documentdb_core/sql/types/bson--0.24-0.sql` and `pg_documentdb_core/sql/types/bson_typanalyze--0.24-0.sql` for BSON type and analysis functions.
- `pg_documentdb_core/sql/documentdb_core--0.24-0.sql`: Created `documentdb_core` schema and included BSON type definitions.
- Updated Makefiles (`Makefile.cflags`, `pg_helio_api/Makefile`, `pg_helio_core/Makefile`, `internal/pg_helio_distributed/Makefile`) to include new schema definitions.
- Updated `postgresql.conf` files to preload `pg_documentdb_core`.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Remove helio references from documentdb
Remove helio references from documentdb_distributed
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request removes references to "helio" and replaces them with "documentdb" in the sources of `documentdb` and `documentdb_distributed`.
- `metadata_cache.h` and `metadata_cache.c`: Replaced "Helio" with "DocumentDB" in variable names and function names.
- `pgmongo.c`: Updated include statements and function calls to use "documentdb" instead of "helio".
- `planner.c`: Updated planner function and related calls to use "DocumentDBApiPlanner".
- Moved several header files from `helio_*` to `documentdb_*` naming convention without code changes.
- `commands_common.c` and `create_indexes.c`: Updated function calls and variable names to use "DocumentDB" instead of "Helio".
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Remove helio from file name
Remove helio references from source content except schema reference
Start local documentdb server with the documentdb extension
Regression tests using the documentdb extension
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request removes references to "helio" from the `documentdb_core` source code.
- `scripts/utils.sh`: Renamed variables from `preloadExtensions` to `preloadLibraries`.
- `pg_documentdb_core/include/utils/error_utils.h`: Updated function and macro names to replace "Helio" with "DocumentDB".
- `pg_documentdb_core/include/io/helio_bson_core.h` and `pg_documentdb_core/include/query/helio_bson_compare.h`: Moved to `pg_documentdb_core/include/io/bson_core.h` and `pg_documentdb_core/include/query/bson_compare.h`.
- `pg_documentdb_core/src/collation/collation.c` and `pg_documentdb_core/src/io/pgbson.c`: Replaced error codes and messages to use "DocumentDB" instead of "Helio".
- `scripts/validate_helio_extension_state.sh`: Added exceptions for certain files to skip validation for "helio" references.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
[OSS] remove dependencies on helio errors
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
- Move c files from pg_helio_core -> pg_documentdb_core
- Make helio_core, and other extensions depend on pg_documentdb_core as a lib dependency
----
#### AI description (iteration 1)
#### PR Classification
Code reorganization
#### PR Summary
This pull request moves source files from `pg_helio_core` to `pg_documentdb_core` to better organize the project structure.
- Deleted `oss/pg_helio_core/include/utils/helio_codes.txt`.
- Added `oss/pg_documentdb_core/src/pg_documentdb_core.c` for initializing the shared library.
- Updated `oss/Makefile.cflags` to include new paths and installation rules for `pg_documentdb_core`.
- Moved `oss/pg_helio_core/src/bson_init.c` and `oss/pg_helio_core/include/bson_init.h` to `oss/pg_documentdb_core/src/bson_init.c` and `oss/pg_documentdb_core/include/bson_init.h` respectively.
- Modified various `Makefile` files to set `USE_DOCUMENTDB_CORE` and adjust include paths.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Not yet
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Initial setup for documentdb extensions.
Create Makefiles and set up build scripts around documentdb.
This does not yet touch the actual code or make anything deployable.
----
#### AI description (iteration 1)
#### PR Classification
Initial setup for DocumentDB integration.
#### PR Summary
This pull request sets up the initial structure and configuration for DocumentDB integration.
- Added `documentdb_errors.h` for defining DocumentDB-specific error codes.
- Created `Makefile` for `pg_documentdb`, `pg_documentdb_core`, and `pg_documentdb_distributed` to handle build configurations.
- Updated `validate_helio_extension_state.sh` and `validate_headers.sh` scripts to include checks for DocumentDB.
- Modified `bson_aggregation_nested_pipeline.c`, `bson_aggregation_vector_search.c`, and other source files to replace references from Helio to DocumentDB.
- Added control files for `pg_documentdb`, `pg_documentdb_core`, and `pg_documentdb_distributed` extensions.
Juan Carlos Aguilera Mendez
Vinod Sridharan
Zinan Ren
Shuai Tian
Cornelius Boateng
Alexander Laye
Deepak Pemmaraju
Xing Fan
Deepak Bisht
Parag Jain
Duan Pei
Xiaochun Liang
