### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
no
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support building documentdb for pg18
This sets up DocumentDB for PG18 builds.
Note that execution and tests are not validated yet (just compilation).
Also bumps up extension dependencies to support the builds of pg18.
----
#### AI description (iteration 7)
#### PR Classification
This pull request adds PG18 support by implementing the necessary compatibility and feature changes in DocumentDB.
#### PR Summary
The update enables DocumentDB builds for PG18 by adapting ICU collation handling, updating version and pipeline configurations, and incorporating compatibility macros and API adjustments across the codebase. Key changes include:
- **`oss/pg_documentdb_core/src/collation/collation.c`**: Introduced new ICU converter functions and updated collation conversion logic.
- **`oss/scripts/setup_versions.sh` and pipelines**: Adjusted dependency versions and container build configurations to support PG18.
- **`oss/pg_documentdb_core/include/utils/documentdb_pg_compatibility.h`**: Added a new compatibility header with macros to bridge PG18 API differences.
- **Extended RUM modules (`oss/internal/pg_documentdb_extended_rum` and `pgmongo_rum`)**: Updated type definitions (e.g., replacing `PostingItem` with `RumPostingItem`) and switched from `GinStatsData` to `RumStatsData`.
- **Various DocumentDB source files**: Modified executor calls, background worker logic, and tuple attribute access to comply with PG18’s API changes.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Simple locale must use binary comparison and not necessarily case-insensitive comparison.
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a bug fix to correctly handle the simple collation locale.
#### PR Summary
The changes ensure that when the collation locale is specified as `simple`, all additional collation options are ignored and binary comparison is applied.
- `oss/pg_documentdb_core/src/collation/collation.c`: Added logic to bypass extra collation options when the locale is `simple`.
- `oss/pg_documentdb_core/src/query/bson_compare.c`: Updated string comparison to use binary comparison for the `simple` collation.
- Test files under `oss/internal/pg_documentdb_distributed/src/test` and `pgmongo/src/test` were updated to add regression checks for simple collation behavior.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Description
1. Fixing issues identified by similarity checker
2. Fix the corresponding sql unit tests
3. Fix JSTests failure due to above changes
4. Fix corresponding C# test failure
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a code cleanup that standardizes and improves error message wording across the codebase.
#### PR Summary
The changes update error messages throughout the repository to provide clearer, more consistent, and more detailed feedback when operations fail.
- Updates in geospatial operator files (e.g. in `$geoNear`, `$near`, `$nearSphere`) replace informal phrasing with precise messages.
- Numerous error messages in aggregation and projection test files have been reworded for consistent phrasing
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
draft
update error msg
----
#### AI description (iteration 1)
#### PR Classification
This is a code cleanup and bug‑fix pull request that standardizes and clarifies error messages and log details across multiple modules.
#### PR Summary
The changes improve the consistency, clarity, and accuracy of user‑facing messages and logging throughout the code base. Key changes include:
- In files under directories such as `oss/internal/pg_documentdb_distributed/src/test/regress/expected/` and `pgmongo/src/test/docdb_compat/expected/`, error messages for geospatial query operators (e.g. geoNear, $center, $within) and aggregation stage operators have been rewritten for clarity (e.g. replacing vague “Too many geoNear expressions” with “Excessive number of geoNear query expressions” and clarifying numeric parameter validations).
- Several operators’ messages (e.g. `$cond`, `$not`, `$abs`, `$floor`, `$divide`, `$log`, etc.) have been updated to state exactly how many arguments are required, using
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
draft
----
#### AI description (iteration 1)
#### PR Classification
Bug fix: This PR revises error messages and inline comments to improve clarity and consistency across the code and test outputs.
#### PR Summary
The changes refactor and update a large number of error messages across various modules—ranging from aggregation and update operators to geospatial, schema validation, and command handling—to provide more precise diagnostic feedback.
- **Tests (e.g. in `tests/expected/...
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
update errmsg
----
#### AI description (iteration 1)
#### PR Classification
This pull request focuses on updating and standardizing error messages across various operators, aggregation stages, and test expectation files.
#### PR Summary
The changes reword error messages throughout the codebase and test outputs to provide clearer, more consistent, and more informative feedback for users. Key updates include:
- **`/oss/internal/pg_documentdb_distributed/src/test/regress/expected/*.out`** – Revised expected error outputs for aggregation, update, and geospatial tests to use clearer phrasing (e.g., for `$merge`, `$pop`, `$indexOfArray`, and `$geoNear`).
- **Core operator files (e.g., `bson_aggregation_output_pipeline.c`, `bson_expression_arithmetic_operators.c`, `bson_expression_conditional_operators.c`)** – Updated error message texts to explicitly state operator requirements and types.
- **Geospatial and vector modules (e.g., files under `bson_geojson_utils.c`, `bson_geospatial_*`, and vector search/index configuration files)** – Standardized validations and error reporting for GeoJSON, CRS, and vector search parameters.
- **Additional modules (e.g., collection view creation and user management files)** – Improved error handling messages to better guide users on input misconfigurations.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
similarity issue
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a code cleanup aimed at standardizing and clarifying error and diagnostic messages throughout the codebase.
#### PR Summary
The changes update error messages and logging across many modules (aggregation, query, update, index creation, collation, etc.) to use clearer, more precise, and consistent language.
- **Aggregation Operators:** Many files in the aggregation code (e.g. in `bson_aggregation_*` and related operator implementations) now report errors such as division by zero, invalid inputs, and argument count in a formal and uniform manner.
- **Index & Projection Validation
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Features
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Port a few more changes into the public documentdb_rum to reduce diff to internal RUM
2) Add more inlining of computation of RumItem from the pages (behind a GUC defaulted on)
3) Add option for bson_build_document and fix bson_get_value_text to not quote strings unnecessarily to be compatible with `::text` casts
----
#### AI description (iteration 1)
#### PR Classification
This PR implements API updates and performance improvements by introducing a new item pointer decoding mechanism in RUM indexes and extending BSON UDF functionalities.
#### PR Summary
The pull request refactors core RUM index scan routines and enhances BSON API functions to improve performance and API consistency. Key changes include:
- **`oss/internal/pg_documentdb_extended_rum` and `pgmongo_rum`**: Introduced new macros and inline functions (e.g., `decode_varbyte_blocknumber`, `rumDataPageLeafReadWithBlockNumberIncr`) along with the boolean flag `RumUseNewItemPtrDecoding` to optimize item pointer decoding.
- **`rumget.c`, `rumentrypage.c`, and `rumdatapage.c`**: Updated the posting tree scan logic to conditionally apply the new decoding approach, streamlining data page processing.
- **`pg_documentdb_core/sql/udfs/bson_io`**: Added new BSON UDFs (such as `bson_build_document`) and overloads (e.g., a three-argument version of `bson_get_value_text`) to expand API capabilities.
- **`pgbson.c` and `bson_io.c`**: Enhanced BSON logging functions with optional quoting support for JSON conversion, improving logging clarity and consistency.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
This PR updates the extension version for the new release 1.108.
Changes included:
- Control files updated to version 1.107-0
- SQL files generated for version upgrades
- Release notes file created for version 1.108
- TOC.yml updated with new release notes entry
- CHANGELOG.md updated with new version entry
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Logging
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Port support for unredacted log messages in RUM
----
#### AI description (iteration 1)
#### PR Classification
This PR introduces a new feature to support unredacted log messages for improved diagnostics in RUM.
#### PR Summary
The pull request adds a custom logging hook and macro to output unredacted messages, enhancing diagnostic details in the RUM module.
- `rumutil.c`: Added `PgmongoRumUnredactedLogEmitHook`, which formats log messages, sanitizes percent signs, and emits unredacted logs.
- `rumvacuum.c`: Updated logging by replacing the standard `ereport` with the new `elog_rum_unredacted` macro for detailed vacuum diagnostics.
- `rum.h`: Declared the new logging function and defined the `elog_rum_unredacted` macro with a specific SQL state code.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Improve performance of indexscans and indexonlyscans
IndexTerm:
>1) BsonIndexTerm is used in several places - reduce the size of this struct by removing isTruncated bool from the struct.
>2) Add options to check metadata state without parsing the index term at all.
>3) InitializeBsonIndexTerm - do not create a wrapper bson_value_t - just initialize the term directly.
OrderedScans:
>1) For every page we visit, we free the prior buffer and call PageGetTempPageCopy. Instead, keep one buffer per scan and just copy the buffer into the static buffer created per scan
>2) When doing indexonlyscan, don't recreate an indextuple very time - instead reuse the memory and just fill the tuple buffer with what was previously allocated
PgbsonWriter:
>1) Add support for reusing the same writer across invocations by calling reset on the buffer
>2) Add options to retrieve the value from a pgbsonwriter
Composite:
>1) Use a case dispatch for strategies rather than multiple if/else
>2) Work on the serialized bson index terms as much as possible and lazy load the index terms
>3) Skip Compare on bounds calls if there's no bounds declared
----
#### AI description (iteration 1)
#### PR Classification
This pull request delivers performance improvements and code refactoring for index scan operations.
#### PR Summary
The changes optimize index scans by refactoring tuple construction and buffer page handling for RUM indexes, while also unifying and streamlining BSON composite index term processing.
- `pgmongo_rum/src/rumget.c`: Introduced `IndexBuildTupleDynamic` to build tuples without compression and reuse memory, and refactored page copy logic using `CopyPageContents` and an added `isPageValid` flag.
- `oss/pg_documentdb/src/opclass/bson_gin_composite_entrypoint.c` (and related files): Replaced direct truncation checks with function calls (e.g., `IsIndexTermTruncated`), updated serialized composite index term initialization, and adjusted ordering transform logic.
- `oss/pg_documentdb/src/opclass/bson_gin_index_term.c` (and accompanying headers): Added functions for handling serialized composite index terms and metadata, consolidating truncation flag logic.
- `oss/pg_documentdb_core/src/io/pgbson.c`/`pgbson_writer.h`: Added new writer functions (e.g., reset and value retrieval) to improve memory reuse and reduce allocation overhead.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
DevOps
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Add a hook that enables the service to dump unredacted logs to the server.
Add a macro that can be called in this case.
The default implementation does nothing, but the custom impl can provide a way to ship `safe` unredacted logs to a telemetry sink.
----
#### AI description (iteration 1)
#### PR Classification
This pull request is an API change that introduces unredacted logging support in the extension.
#### PR Summary
The changes add new logging hooks and functions to output unredacted server logs by sanitizing log messages and replacing standard logging calls in critical workflows.
- **`pgmongo/src/api_hook_impl.c`**: Added `PgmongoUnredactedLogEmitHook` to sanitize messages (by replacing `%` with `p`) and register it via a new logging hook.
- **`pg_documentdb/src/commands/create_indexes_background.c`**: Replaced multiple `ereport(LOG, ...)` calls with the `elog_unredacted` macro to log unredacted messages during index operations.
- **`pg_documentdb/src/utils/error_utils.c` & `pg_documentdb_core/include/utils/error_utils.h`**: Introduced `errmsg_unredacted` and defined the `elog_unredacted` macro to support the new logging mechanism.
- **Miscellaneous**: Adjusted logging verbosity in files like `distribution/distributed_hooks.c` and `create_collection_core.c` by adding contextual flags for hiding statements and contexts.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
This PR updates the extension version for the new release 1.107.
Changes included:
- Control files updated to version 1.106-0
- SQL files generated for version upgrades
- Release notes file created for version 1.107
- TOC.yml updated with new release notes entry
- CHANGELOG.md updated with new version entry
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
[Compat] Fixing output pipeline namespace validations
----
#### AI description (iteration 1)
#### PR Classification
Bug fix and compatibility update.
#### PR Summary
This pull request enforces strict namespace validations for aggregation pipeline output stages by preventing writes to internal databases and system collections. It introduces a dedicated validation function for both $out and $merge stages and integrates new error codes across the codebase and mapping files.
- `oss/pg_documentdb/src/aggregation/bson_aggregation_output_pipeline.c`: Added the `ValidateTargetNameSpaceForOutputStage` function and updated calls in both the $out and $merge handlers.
- `pgmongo/src/utils/documentdb_conversion_codes.c` and `oss/pg_documentdb_core/include/utils/documentdb_errors.h`: Introduced and mapped new error codes for invalid namespace targets.
- Updated multiple error mapping files (`documentdb_errors.csv`, `error_mappings.csv`, `documentdb_codes.txt`, `external_error_mapping.csv`, and `mongo_codes.csv`) and test CSV files under `gateway/managed/Tests.Integration.Shell.Postgres` to reflect the new validations and error messages.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
cleanup documentdb_core
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Move Location13103 error to documentdb_codes.txt
----
#### AI description (iteration 1)
#### PR Classification
Bug fix: Moving an error code definition to the correct file.
#### PR Summary
This pull request corrects the placement of the "Location13103" error by removing it from one file and adding it into the proper location.
- `pg_documentdb_gw/include/gateway_errors.txt`: Removed the "Location13103" error entry.
- `pg_documentdb_core/include/utils/documentdb_codes.txt`: Added the "Location13103" error entry.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
This PR updates the extension version for the new release 0.106.
Changes included:
- Control files updated to new version
- SQL files generated for version upgrades
- CHANGELOG.md updated with new version entry
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Support order by descending as a feature under a feature flag if the underlying index can support it
2) Fix up memory performance issues causing OOMs with composite index with large numbers of bounds
3) On doing this noticed that row_get_bson was busted for int arrays, fix this behavior to not crash
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) There were order by tests not running but had the sql files in there. This was catching contracts related to arrays with null values where 2 cases were ordered later than they should be. This fixes that. As part of that we noticed that this breaks when there's paths that have array indexes (e.g. a.b.0). This also fixes that contract.
2) Add checks to say all tests must run or exist in a schedule file and fix the output
3) There's flakeyness in currentop due to tracking the version refresh query - skip the version refresh query in currentop.
----
#### AI description (iteration 1)
#### PR Classification
This pull request fixes correctness issues in composite order-by functionality and improves associated testing.
#### PR Summary
The changes refine the handling of intermediate arrays in order-by operations by updating traversal and comparison logic, while enhancing the test suite and validation scripts for composite order-by scenarios.
- `pg_documentdb/src/query/bson_dollar_operators.c` and related files were modified to correctly process nested arrays and compare composite order-by values.
- Test files in `pgmongo/src/test/regress/expected/` and `pgmongo/src/test/regress/sql/` were updated to add new order-by cases and remove obsolete tests.
- Validation scripts (e.g. in `oss/internal/pg_documentdb_distributed/src/test/regress/validate_test_output.sh` and `oss/pg_documentdb/src/test/regress/validate_test_output.sh`) were enhanced for more reliable schedule checks and collection ID extraction.
- Several BSON traversal and update operator functions were updated to include an `isArrayIndexSearch` parameter for finer control during array element processing.
- `pg_documentdb/src/commands/current_op.c` and `pg_documentdb/src/utils/version_utils.c` were updated to skip version refresh queries appropriately.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Code cleanup and documentation updates to match contribution guidelines
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a code cleanup and documentation update.
#### PR Summary
The updates refine inline comments and documentation across update, error handling, and collation components to improve clarity and consistency, while replacing obsolete Mongo references with DocumentDB terminology. Key changes include:
- **`oss/pg_documentdb/src/update/bson_update_operators.c`**: Revised comments for update operators (e.g., `$currentDate`, `$push`, `$mul`) to clarify behavior and update examples.
- **`oss/pg_documentdb/src/utils/error_utils.c`**: Updated documentation to reference DocumentDB errors instead of Mongo errors.
- **`oss/pg_documentdb/src/update/bson_update.c` and `oss/pg_documentdb/src/update/bson_update_aggregation.c`**: Improved inline comment wording to accurately describe update specifications and error conditions.
- **`oss/pg_documentdb_core/src/collation/collation.c`**: Cleaned up collation conversion comments and locale handling explanations.
- **Test files under `oss/internal/pg_documentdb_distributed/src/test` and `pgmongo/src/test/docdb_compat/expected`**: Updated expected output comments to align with the revised error message tests.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Fix up docs across some files to match contributing guidelines.
----
#### AI description (iteration 1)
#### PR Classification
This PR focuses on documentation improvements and in-code comment cleanup to comply with contributing guidelines.
#### PR Summary
The pull request refines comment texts and error messages across several source files, removes unused code documentation, and updates internal clarifications.
- `oss/pg_documentdb/src/aggregation/bson_aggregates.c`: Removed the unused `BsonOutAggregateState` typedef and updated aggregation comments (e.g., changing "HASHTBL" to "HASHCTL") while eliminating redundant code comments.
- `oss/pg_documentdb/src/commands/create_collection_view.c`: Removed several unused fields from the `CreateSpec` structure and updated header comments to reflect the correct file path.
- `oss/pg_documentdb/src/aggregation/bson_aggregation_pipeline.c`: Revised comment formatting and error messages in aggregation and query generation functions to ensure consistent terminology.
- `oss/pg_documentdb/src/auth/scram256.c` & `oss/pg_documentdb/src/commands/update.c`: Updated documentation comments for authentication responses and update error handling for improved clarity.
- Other files in `/oss/pg_documentdb/src/opclass/` and `/oss/pg_documentdb_core/`: Corrected typographical errors and adjusted error code references to align with the project’s guidelines.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
3 c files
----
#### AI description (iteration 1)
#### PR Classification
This pull request focuses on code cleanup and documentation updates.
#### PR Summary
The changes standardize and simplify documentation comments and examples across several operator implementations, removing obsolete references and unnecessary details.
- `oss/pg_documentdb/src/aggregation/bson_aggregation_output_pipeline.c`: Updated doc examples for `$merge` and `$out` to use generic placeholders and removed temporary test-related comments.
- `oss/pg_documentdb/src/query/bson_dollar_operators.c`: Cleaned up comments for multiple BSON dollar operators by eliminating "Mongo's" references and redundant test notes.
- `oss/pg_documentdb_core/src/types/pcre_regex.c`: Refined regex compile option comments and removed legacy test reference lines.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
Enhancement and bug fix to improve the stability and management of file-based cursors.
#### PR Summary
This pull request refactors and bolsters file-based cursor management by introducing shared memory support, improved cleanup, error handling, and configuration changes. It also adds new API endpoints and logging for better cursor lifecycle control.
- **`cursor_store.c` and related files:** Added shared memory functions (e.g., `FileCursorShmemSize`, `InitializeFileCursorShmem`), cursor count regulation (with `IncrementCursorCount`/`DecrementCursorCount`), and cleanup routines to ensure stable cursor file operations.
- **Test and expected output files:** Updated regression tests and expected outputs to cover scenarios like duplicate cursor creation, cursor deletion limits, and error reporting when the file cursor capacity is reached.
- **API enhancements:** Introduced new UDFs (e.g., `command_delete_cursors`) and tenant client methods for killing and deleting cursors, improving session management.
- **Configuration updates:** Renamed and added GUC parameters (e.g., `MaxAllowedCursorIntermediateFileSizeMB`, `MaxCursorFileCount`, and cursor expiry settings) to control file-based cursor behavior.
- **Logging and metrics improvements:** Enhanced logging in `mdsd_logger.c` to report current cursor counts and related metrics for operational insight.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
[DocumentDB issus 178](https://github.com/microsoft/documentdb/issues/178)
When the document size exceeds 2000, the tuple is toasted. We need to detoast the datum before converting it to pgbson.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix addressing improper detoasting of pgbson values.
#### PR Summary
This pull request replaces direct pointer extraction and SPI_datumTransfer calls with dedicated DatumGetPgBson macros to ensure both toasted and non-toasted bson documents are correctly handled. It also enhances test coverage for find-and-modify operations on large documents.
- `commands_find_and_modify.out` and SQL test files: Added new queries for deleting large (toasted) and non-toasted documents.
- `retryable_writes.c`, `delete.c`, metadata, index, and query operator files: Replaced manual SPI_datumTransfer and DatumGetPointer calls with DatumGetPgBson (and DatumGetPgBson_MAYBE_NULL) to properly detoast and copy pgbson objects.
- `pgbson.h` and `bson_io.c`: Introduced and integrated new macros to streamline detoasting of pgbson data.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
more
----
#### AI description (iteration 1)
#### PR Classification
Performance optimization.
#### PR Summary
This PR eliminates an unnecessary memory allocation during BSON parsing by omitting the call to `pstrdup` for the collation string.
- `/oss/pg_documentdb_core/src/io/pgbsonelement.c`: Replaced `pstrdup(bson_iter_utf8(&iter, NULL))` with a direct pointer assignment from `bson_iter_utf8(&iter, NULL)`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
This pull request adds documentation for the `$bucketAuto` aggregation stage in PostgreSQL, detailing its design, usage, and implementation.
- Adds `bucketAuto.md` with details on the `$bucketAuto` aggregation stage
- code implementation
- support all granularity types: POWERSOF2, 1-2-5, Renard numbers, E series
Related work items: #3234185
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix/Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Orderby on null/undefined documents was incorrect - fix up the ordering
2) Equality on null/undefined was also incorrect fix this up
3) When traversing the document, we were doing an extra traversal into arrays when it was not needed. fix this up
4) Order by on composite index didn't handle nulls correctly - fix this too.
5) Support pushdown of $elemMatch into composite indexes without array terms.
----
#### AI description (iteration 1)
#### PR Classification
This pull request fixes bugs in the composite index and orderby functionality.
#### PR Summary
The changes correct the orderby comparison logic and improve composite index term generation in BSON processing. The pull request also adds new functions to handle intermediate array paths and introduces legacy behavior flags, while supplementing the fixes with comprehensive integration tests.
- **`src/query/bson_dollar_operators.c`**: Modified `OrderByVisitArrayField` to pass the correct BSON value and added new functions for array path handling.
- **`src/opclass/gin_index_term.c` and related opclass files**: Introduced `GenerateValueMaybeUndefinedTerm` and updated index term comparison to distinguish between undefined and partially undefined values.
- **`pgbson_utils.c`**: Adjusted array traversal logic with a new callback for missing intermediate array paths and a flag to control array traverse optimization.
- **Test files under `/oss/pg_documentdb/src/test/regress/expected/` and `/pgmongo/src/test/regress/expected/`**: Added extensive composite ordering and filtering tests to validate the corrected behavior.
- **`configs/feature_flag_configs.c`**: Added custom GUCs for `useLegacyOrderByBehavior` and `useLegacyNullEqualityBehavior` to manage legacy ordering and null equality rules.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Version update for 104
Related work items: #4281831
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Currently for Update/Delete, we currently parse the update spec, query spec, etc into pgbsons on startup. Then reuse it per commands.
This has a few issues - for large document requests, this causes us to duplicate memory and persist it for the duration of the update and also transcode all up front - this is worse for multi-node when all we need to do is build yet another update spec and push it to the worker.
Instead, we now only cache pointers to the spec document; then when we need it we convert to pgbson only if necessary.
A consequence of this is that on a single node update, if we're doing a simple updateOne on _id then we would never serialize the updateSpec at all (for instance).
We also more eagerly free memory on the intermediate update specs.
----
#### AI description (iteration 1)
#### PR Classification
Performance optimization for update parsing by refactoring BSON value handling.
#### PR Summary
This PR streamlines update parsing by replacing heavy BSON initialization routines with lightweight wrappers and converting update-related types from `pgbson*` to `bson_value_t*`, which reduces overhead and improves coordinator/worker compatibility.
- `src/commands/update.c`: Replaced multiple `PgbsonInitFromDocumentBsonValue` and similar calls with efficient wrappers like `CreateBsonValueFrom` and updated serialization (including back-compatible array filter and update document formats).
- `include/commands/update.h`: Updated update structure fields to use `bson_value_t*` instead of `pgbson*`, aligning interfaces with the new parsing approach.
- `src/update/bson_update.c` and `bson_update_operators_workflow.c`: Refactored update type determination and replace document logic to use new BSON value iterators and helper functions.
- Other modules (e.g. `commands/find_and_modify.c`, `commands/delete.c`, query operators, sharding) were updated to leverage the lighter-weight BSON representation and its associated helper routines.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
When servers have default_readonly_transaction set, the current behavior is wonky - it will cause failures in underlying queries and pollute the errors, and it will go through a bunch of work only to fail.
Additionally, background jobs like TTL will fail ungracefully.
Handle this GUC by failing write commands early with a more useful error message.
Additionally, in cases like TTL, add an opt-in way to prune rows even in the presence of this GUC if it's opted into.
----
#### AI description (iteration 1)
#### PR Classification
This pull request is a bug fix that improves error handling for write operations attempted during default read-only transactions.
#### PR Summary
The changes introduce a unified mechanism to check and enforce read-only transaction restrictions, ensuring graceful errors when write operations are attempted while in a read-only state. Key changes include:
- **`include/commands/commands_common.h`**: Added the inline function `ThrowIfServerOrTransactionReadOnly` to validate transaction state and throw appropriate errors.
- **Test files**: New regression tests under `/oss/pg_documentdb/src/test/regress/expected/` and `/oss/pg_documentdb/src/test/regress/sql/readonly_transaction_tests.sql` validate the expected error responses under read-only conditions.
- **`ttl_index.c`**: Modified TTL job logic to handle read-only transactions, allowing jobs to commit and restart when the corresponding flag is enabled.
- **Gateway and error handling**: Updated configuration in `TenantConfigurationBase.cs`, error augmentation in `PostgresMongoResultExtensions.cs`, and error mappings (e.g., adding error code 10107 for `NotWritablePrimary`) to propagate clear error messages.
- **Command modules**: Added calls to `ThrowIfServerOrTransactionReadOnly` in several command implementations (e.g., in `insert.c`, `update.c`, `delete.c`, `create_indexes.c`) to enforce read-only restrictions uniformly.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support collation with comparison of `array` and `document` values.
Also adds support for `$in` on nested arrays and documents.
----
#### AI description (iteration 1)
#### PR Classification
New feature: Adds support for collation-aware hashing of arrays and documents in BSON.
#### PR Summary
This pull request introduces collation-aware hashing for BSON arrays and documents, enhancing the handling of collation in hash sets. It refactors existing hash functions to accommodate collation strings and recursive hashing of nested BSON structures.
- `hashset_utils.c`: Implements collation-aware hashing functions for BSON values, including recursive handling of arrays and documents.
- `bson_expression_set_operators.c`: Replaces existing hash set creation with collation-aware hash set creation.
- `bson_dollar_operators.c`: Updates logic to use collation-aware hash sets for `$in` queries, removing unsupported scenarios.
- `hashset_utils.h`: Adds a new struct `BsonValueHashEntry` to store BSON values with collation strings.
- `collation.h`: Moves `IsBsonTypeCollationAware` function to handle collation applicability checks.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4111861, #4115239
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
Version update for the release of version 103.
#### PR Summary
This pull request updates the version numbers across several control files and adds release notes for version 1.103 and 1.104. It also includes changes to SQL files and their corresponding SHA1 checksums.
- `/oss/internal/pg_documentdb_distributed/documentdb_distributed.control`, `/oss/pg_documentdb/documentdb.control`, `/oss/pg_documentdb_core/documentdb_core.control`, and `/pgmongo/pgmongo.control`: Updated default version numbers.
- `/docs/release_notes/pgmongo_1.103.md` and `/docs/release_notes/pgmongo_1.104.md`: Added release notes for versions 1.103 and 1.104.
- `/pgmongo/sql/pgmongo--1.103-1--1.104-1.sql`: Added new SQL file for version transition.
- `/docs/release_notes/toc.yml`: Updated table of contents to include new release notes.
- `/oss/CHANGELOG.md`: Added entry for documentdb v0.104-0.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4140262
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
- pgbsonelement today copies the bson_value_t buffer always, use the referrence in critical places to avoid copy
Before the fix
After the fix
----
#### AI description (iteration 1)
#### PR Classification
Code optimization
#### PR Summary
This pull request introduces optimizations for point reads by avoiding the copying of BSON value buffers.
- `include/io/pgbsonelement.h`: Added `pgbsonelementref` struct and `BsonIterToPgbsonElementRef` function.
- `src/query/bson_compare.c`: Updated comparisons to use `pgbsonelementref` instead of `pgbsonelement`.
- `src/io/pgbsonelement.c`: Implemented `BsonIterToPgbsonElementRef` function.
- `src/query/bsonquery_compare.c`: Modified query comparison logic to utilize `pgbsonelementref`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
This pr adds a Rust gateway for documentdb in the pg_documentdb_gw folder
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) InitializeBsonIndexTerm called BsonValueToPgbsonElement that was using the bson_iter_next a lot. This is a lot of overhead when we know it's a single element. Use an unsafe variant of it to generate a pgbsonelement
2) Introduce a MatchAndRecurse option for index paths that allows the term generation logic to know whether it's an exact match or if the child paths also match. This is currently unused, but will be used in a subsequent change.
----
#### AI description (iteration 1)
#### PR Classification
Code improvement for term generation in wildcard index.
#### PR Summary
This pull request enhances the term generation process for wildcard indexes by introducing an unsafe transformation method and refining index traversal options.
- `pgbsonelement.c`: Added `FillPgbsonElementUnsafe` and `BsonValueToPgbsonElementUnsafe` functions for unsafe BSON element transformation.
- `bson_gin_entrypoint.c`: Introduced `IndexTraverse_MatchAndRecurse` option to improve path matching and recursion.
- `feature_flag_configs.c`: Added feature flag `indexTermUseUnsafeTransform` to enable unsafe transformation.
- `gin_index_term.c`: Utilized the unsafe transformation method based on the new feature flag.
- `bson_gin_core.c`: Updated term path generation to handle the new traversal option.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces CRUD operations for Entra users in the DataPlane.
- `authentication.c`: Added functions to create users with external identity providers, validate identity provider info, and check if the current user is an Entra admin.
- `users.h`: Added new header file defining user CRUD functions and related structures.
- `users.c`: Updated user creation logic to handle external identity providers and added parsing for custom data.
- `api_hooks.c`: Added hooks for creating users with external identity providers and checking user privileges.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support advanced locales
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces a collation add-on to enhance locale and collation type handling in the collation module.
- `collation.c`: Added support for parsing and handling collation types within locale strings, including replacing underscores with hyphens.
- `bson_dollar_ops_collation_tests_runtime.out`: Added new test cases to validate collation functionality with various locales and collation types.
- `bson_dollar_ops_collation_tests_runtime.sql`: Updated SQL test scripts to include new collation scenarios and validate query matching with different collation settings.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support conversion to binData from string and vice versa.
$convert to BinData
```
"$convert": {
"input": <string>,
"to": {
"type": "binData",
"subtype": <bson_subtype_t>
},
"format": <"base64"/"string"/"hex"/"uuid"/"base64url">
}
```
OR
```
--- subtype will default to 0
"$convert": {
"input": <string>,
"to": "binData",
"format": <"base64"/"string"/"hex"/"uuid"/"base64url">
}
```
BinData $toString
```
{ $toString: <binData expression> }
```
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces support for converting data to `BinData` using the `$convert` operator.
- `src/operators/bson_expression_type_operators.c`: Added new functions and modified existing ones to handle `BinData` conversion, including `ProcessDollarToBinData`, `GetToTypeAndSubTypeForBinDataConversion`, and `ValidateAndGetBinDataConversionSubTypeAndFormat`.
- Refactored `ParseDollarConvert` and `ApplyDollarConvert` to accommodate the new `BinData` conversion logic.
- Introduced a new struct `ConvertToTypeArguments` to manage conversion arguments.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix(ish)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
0.102-0 had changes that are non-idempotent. Given there were some incorrect dependencies taken on the 0.102 release (independently) introduce a 0.102-1 version that applies all the delta of 0.102-0 in an idempotent manner so that dependencies taken partially on 0.102 and dependencies taken on 0.102 fully work side-by-side.
----
#### AI description (iteration 1)
#### PR Classification
Hotfix for DocumentDB version upgrade.
#### PR Summary
This pull request introduces a hotfix for DocumentDB, ensuring an idempotent upgrade from version 0.102-0 to 0.102-1.
- Added `documentdb--0.102-0--0.102-1.sql` to handle partial upgrades and repeat UDF upgrades.
- Modified `data_table_utils.c` to check for both 0.102-0 and 0.102-1 versions during data table upgrades.
- Updated `pgmongo.control` to set the default version to 1.102-1.
- Added `pgmongo--1.102-1--1.103-0.sql` for version transition.
- Updated SHA1 checksums in multiple `pgmongo/sql/sha1` files to reflect changes.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Version update for 102 release
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support new syntax of $getField
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for the extended `$getField` operator for version 8.0.
- Added new test file `postgresjs/expression_get_field_80_custom.js` to validate `$getField` operator behavior.
- Updated `bson_expression_object_operators.c` to handle new error codes and validation for `$getField`.
- Modified `bson_aggregation_object_operators_tests.sql` and related expected output files to include new test cases for `$getField`.
- Added new error code mappings in `documentdb_errors.csv` and `error_mappings.csv`.
- Updated `documentdb_codes.txt` and `mongo_codes.csv` to include new error codes.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Remove all Citus references from pg_documentdb tests
2) Remove some citus references from documentdb c files (Comments)
3) Remove dead code for bson_out aggregates
4) Remove all References to IsClusterVersionAtLeast less than 0.24 (since those are now deployed)
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request introduces code cleanups and enhancements related to aggregation functions in DocumentDB.
- `oss/pg_documentdb/sql/udfs/aggregation/group_aggregates--0.102-0.sql`: Added new aggregation functions for BSON data types, including BSONSUM, BSONAVERAGE, BSONMAX, BSONMIN, BSONFIRST, BSONLAST, BSONFIRSTN, BSONLASTN, BSON_ARRAY_AGG, BSON_OBJECT_AGG, BSON_ADD_TO_SET, BSON_MERGE_OBJECTS, BSONSTDDEVPOP, BSONSTDDEVSAMP, BSONPERCENTILE, and BSONMEDIAN.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Log Connection open events in the GW
2) Add current backend PID to notices from BE in GW
3) Add ErrDetail_log support for lock statements in GW
4) Don't continue Insert/Update/Delete on Query cancellation.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request addresses livesite issues by adding logging and fixing continuation on query cancellation.
- `error_utils.h`: Added `IsOperatorInterventionError` function to handle specific error codes.
- `update.c`, `delete.c`, `insert.c`: Integrated `IsOperatorInterventionError` to rethrow specific errors.
- `PostgresClientProvider.cs`: Added logging for physical connection opens.
- `PostgresClient.cs`: Enhanced logging to include process ID and hint details.
- `logger_hooks.c`: Modified logging behavior to promote detail logs to hints.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
- Support top level let variables in $geoNear stage
### Description
Support top level let variables in $geoNear stage
- Today there are 2 scenarios where `let` is used with $geoNear
1- `top level`: Any `let` provided in the top level command such as `aggregate`, the variables used in geonear spec should result into a `constant`, so we evaluate and rewrite the geoNear spec before parsing it to replace any expressions/let variables from top level let.
2- `nested let`: This scenario occurs with lookup pipeline and let is a combination of top level let and lookup let, this case is not optimized in this PR and will still return in error
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for top-level `let` expressions within the `$geoNear` stage in aggregation pipelines.
- `oss/pg_documentdb/src/aggregation/bson_aggregation_pipeline.c`: Added logic to handle `let` expressions by updating the `$geoNear` specification if a `CONST let` expression is present in the context.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support collation with `$setEquals`, `$setUnion`, `$setIntersection`, `$setDifference`, `$setIsSubet` in the aggregation pipeline.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for collation in set operators.
- `collation.c`: Added `HashStringWithCollation` function to hash strings with collation.
- `bson_expression_set_operators.c`: Introduced collation handling in set operator states and comparison functions.
- `bson_dollar_ops_collation_tests_runtime.sql`: Added tests for `$setEquals` with collation.
- `collation.h`: Declared `HashStringWithCollation` function.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Vinod Sridharan
Cornelius Boateng
Shuai Tian
Jinrui Zhang
Siwen Tang
Xing Fan
Duan Pei
Jia Hu
CosmosDB Build Service (msdata)
Vinod Sridharan
Xing Guo
Parag Jain
Zinan Ren
Juan Carlos Aguilera Mendez
Jayanta Mondal
Shuai Tian
Manish Gaur
Deepak Bisht
Alexander Laye
Deepak Pemmaraju
yangqiaoandYang Qiao
Michael Banck
