### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Issue1:
Our lastN has some errors. When taking the last N elements, the original order is destroyed.
For example:
for[31,33,99,1], if want to get last3, it should be [33,99,1], our current lastN will get [1,99,33]:
left is native mongo, right one is our pgmongo
Fix:
When there is no sort, the result is reversed in the transit function.
When there is sort, the result is reversed in the final function.
Issue2:
BottomN also has similar errors as mentioned above.
left is native mongo, right one is our pgmongo
#### AI description (iteration 1)
#### PR Classification
Bug fix addressing incorrect accumulation logic for the LastN accumulator.
#### PR Summary
This PR fixes issues in the LastN accumulator by streamlining memory copy operations and adding robust error checking for BSON data integrity, while also correcting a misconfigured boolean flag in related functions.
- `oss/pg_documentdb/src/aggregation/bson_sorted_accumulator.c`: Simplified the copy routine by replacing conditional branches with a unified memmove and added error checks to detect and report insufficient or corrupted BSON entries.
- `oss/pg_documentdb/src/aggregation/bson_first_last.c`: Updated the `isLast` flag from true to false in both transition and combine functions to ensure correct accumulator behavior.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Comments
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
All the previous version SQL changes should be merged into this PR so that we can update the hashes in this PR
----
#### AI description (iteration 1)
#### PR Classification
This PR is a documentation update that cleans up inline SQL comments to align with the latest contributing guidelines.
#### PR Summary
The changes refine the SQL file comments for geospatial operator classes and functions by removing outdated references to Mongo indexing and validations.
- `oss/pg_documentdb/sql/operators/bson_gist_geospatial_op_classes--0.16-0.sql`: Updated comments by replacing "mongo 2d index options" and "mongo 2dsphjere index options" with simplified descriptions.
- `oss/pg_documentdb/sql/udfs/geospatial/bson_gist_extensibility_functions--0.16-0.sql` and `oss/pg_documentdb/sql/udfs/geospatial/bson_gist_extensibility_functions--latest.sql`: Revised function comments to remove "mongo db validations" for a more general description of geometry validations.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4354111
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Move bucketAuto to schema 105
----
#### AI description (iteration 1)
#### PR Classification
This pull request introduces a schema migration for the `$bucketAuto` aggregation stage, moving its support from schema 104 to schema 105.
#### PR Summary
The changes update documentation, SQL scripts, and test configuration to reflect the migration of `$bucketAuto` support to the new schema.
- `pgmongo/sql/pgmongo--1.104-0--1.105-0.sql`: Added inclusion of the new `$bucketAuto` UDF file for schema 105.
- `oss/pg_documentdb/sql/documentdb--0.104-0--0.105-0.sql`: New file added to include `$bucketAuto` support for the updated schema.
- `oss/CHANGELOG.md` and release notes files: Updated to move the `$bucketAuto` aggregation stage support from the previous schema version to the new release notes for schema 105.
- Several test CSV files under `/gateway/managed/Tests.Integration.Shell.Postgres` were updated to reflect the new schema version.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
This pull request adds documentation for the `$bucketAuto` aggregation stage in PostgreSQL, detailing its design, usage, and implementation.
- Adds `bucketAuto.md` with details on the `$bucketAuto` aggregation stage
- code implementation
- support all granularity types: POWERSOF2, 1-2-5, Renard numbers, E series
Related work items: #3234185
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Remove all Citus references from pg_documentdb tests
2) Remove some citus references from documentdb c files (Comments)
3) Remove dead code for bson_out aggregates
4) Remove all References to IsClusterVersionAtLeast less than 0.24 (since those are now deployed)
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request introduces code cleanups and enhancements related to aggregation functions in DocumentDB.
- `oss/pg_documentdb/sql/udfs/aggregation/group_aggregates--0.102-0.sql`: Added new aggregation functions for BSON data types, including BSONSUM, BSONAVERAGE, BSONMAX, BSONMIN, BSONFIRST, BSONLAST, BSONFIRSTN, BSONLASTN, BSON_ARRAY_AGG, BSON_OBJECT_AGG, BSON_ADD_TO_SET, BSON_MERGE_OBJECTS, BSONSTDDEVPOP, BSONSTDDEVSAMP, BSONPERCENTILE, and BSONMEDIAN.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
more
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for collation in the redact function.
- `bson_project.c`: Added handling for an additional collation string argument in the `bson_dollar_redact` function.
- `bson_aggregation_redact--0.24-0.sql` and `bson_aggregation_redact--latest.sql`: Updated to include the new collation string parameter in the `bson_dollar_redact` function definition.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #3919400
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
[schema validation] apply to stage $merge/$out
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces schema validation for the `$merge` and `$out` stages in MongoDB aggregation pipelines.
- Added `schema_validation_merge.md` to document the design and implementation details.
- Created new SQL functions in `bson_merge_functions--1.24-0.sql` to handle schema validation during the `$merge` stage.
- Updated `bson_aggregation_output_pipeline.c` to integrate schema validation logic into the `$merge` and `$out` stages.
- Modified `toc.yml` to include the new schema validation documentation.
- Updated `pgmongo--1.23-0--1.24-0.sql` and `pg_helio_api--1.23-0--1.24-0.sql` to include the new SQL functions.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Enable distributed tests in documentdb, and clean up helio references from documentdb_distributed
Add substitution to make pg_helio_distributed tests work
Add substitution to make pgmongo/helio_compat tests work
Move udfs bson_aggregation_redact to documentdb
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request enables distributed tests in DocumentDB by adding a new set of regression tests for BSON aggregation comparison operators.
- Added `bson_aggregation_comparison_operators_tests.out` with extensive test cases for BSON comparison operators (`$cmp`, `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`).
* Merged PR 1557887: Change error file mapping name
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Rename error file mapping
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request updates the name of the error mapping file and adjusts references accordingly.
- `Makefile`: Updated references from `mongoerrors.csv` to `error_mappings.csv`.
- `docs/pgmongo/design_docs/data_plane/backend_common/error_management.md`: Updated documentation to reflect the new file name.
- `src/utils/documentdb_conversion_codes.c`: Updated comment to reference the new file name.
- Moved `oss/mongoerrors.csv` to `oss/error_mappings.csv` without code changes.
* Merged PR 1555452: [OSS] Enable distributed tests in documentdb
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Enable distributed tests in documentdb, and clean up helio references from documentdb_distributed
Add substitution to make pg_helio_distributed tests work
Add substitution to make pgmongo/helio_compat tests work
Move udfs bson_aggregation_redact to documentdb
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request enables distributed tests in DocumentDB by adding a new set of regression tests for BSON aggregation comparison operators.
- Added `bson_aggregation_comparison_operators_tests.out` with extensive test cases for BSON comparison operators (`$cmp`, `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`).
* Merged PR 1555841: [OSS] fix regress sql test workflow
[OSS] fix regress sql test workflow
----
Bug fix for the SQL test workflow.
This pull request fixes the regression in the SQL test workflow by updating the GitHub Actions configuration and related scripts.
- Updated `.github/workflows/regress_tests.yml` to improve the build matrix, install dependencies, and run tests for multiple PostgreSQL versions and Ubuntu versions.
- Modified `scripts/start_oss_server.sh` to correct the help message and default directory.
- Minor cleanup in `scripts/install_setup_pg_cron.sh` to remove an unnecessary blank line.
---------
Co-authored-by: Vinod Sridharan <visridha@microsoft.com>
Co-authored-by: Zinan Ren <zinanren@microsoft.com>
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
draft
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces new aggregation functions and support for t-digest in the project.
- Added `bson_tdigest.c` to implement t-digest helper functions.
- Created new SQL aggregate functions in `group_aggregates--1.23-0.sql` and `group_aggregates_support--1.23-0.sql`.
- Updated `bson_aggregation_pipeline.c` to handle the new `$percentile` accumulator.
- Modified `metadata_cache.c` and `metadata_cache.h` to include OID for the new `BsonPercentileAggregateFunction`.
- Updated migration scripts `pg_helio_api--1.22-0--1.23-0.sql` and `pgmongo--1.22-0--1.23-0.sql` to include new SQL files.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Porting udfs into documentdb
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup and feature enhancement.
#### PR Summary
This pull request continues the porting of UDFs into DocumentDB, updating and including various SQL files for better functionality and organization.
- Updated `documentdb--0.24-0.sql` to include new UDFs and operators, replacing old function references.
- Added new UDFs for query, projection, geospatial, aggregation, metadata, schema management, index management, CRUD commands, user management, schema validation, telemetry, TTL, vector search, extension version, diagnostic, and RBAC metadata.
- Removed outdated function definitions and replaced them with updated SQL includes.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Move udf files from pg_helio_api -> pg_documentdb
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request moves User Defined Function (UDF) files from the `pg_helio_api` directory to the `pg_documentdb` directory.
- Updated `pgmongo/sql/pgmongo--1.23-0--1.24-0.sql` to include UDFs from `pg_documentdb` instead of `pg_helio_api`.
- Modified `oss/pg_helio_api/sql/pg_helio_api--1.15-0--1.16-0.sql` to reference UDFs from `pg_documentdb`.
- Adjusted `oss/pg_helio_api/sql/pg_helio_api--1.10-0.sql` to include UDFs from `pg_documentdb`.
- Updated `scripts/validate_helio_extension_state.sh` to add validation exceptions for the moved UDF files.
- Changed `pgmongo/sql/pgmongo--1.21-0--1.22-0.sql` to reference UDFs from `pg_documentdb`.
Duan Pei
Deepak Bisht
Shuai Tian
Vinod Sridharan
Cornelius Boateng
Jia Hu
Zinan Ren
Siddhesh Vethe
Xing Fan
