### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Use OSS RUM source in OSS build
----
#### AI description (iteration 1)
#### PR Classification
This pull request implements a feature update to enable the OSS build to use the OSS RUM source through configurable parameters and script modifications.
#### PR Summary
The PR refactors build scripts and extension routines to support toggling between private and OSS RUM sources. Key changes include:
- **`scripts/pipeline_build_extension_core.sh`**: Converted positional parameters to flag-based arguments using getopts and added a conditional to select OSS RUM installation.
- **`pipelines/templates/extension_build_core.yml` & `.pipelines/templates/oss_documentdb_build.yml`**: Introduced the `useOssRumRepo` parameter to control whether the OSS RUM source is used.
- **`/oss/pg_documentdb/src/index_am/rum.c`**: Updated index routine functions to directly call OSS-specific implementations when the new composite index feature is enabled.
- **`scripts/build_dev_flex_container.sh`**: Adjusted the call to the pipeline build script to use the new flag-based argument format.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Optimize the fix for postgis 3.5
Related postgis issue: https://trac.osgeo.org/postgis/ticket/5652
----
This pull request is a bug fix that updates the PostGIS 3.5 build process.
The changes modify the installation script and update Docker container configurations to resolve build issues for PostGIS 3.5 by removing unused GDAL components and avoiding parallel build race conditions.
- `oss/scripts/install_setup_postgis.sh`: Added a call to `./autogen.sh`, updated configuration options by removing GDAL use and adding `--without-raster`, and replaced parallel `make` with serial execution.
- Multiple Dockerfiles (`.pipelines/Containers/Container-Base-Mariner/Dockerfile`, `/.devcontainer/Dockerfile`, `/.pipelines/Containers/Container-Base-Ubuntu/Dockerfile`, `/oss/.devcontainer/Dockerfile`, and `/oss/.github/containers/Build-Ubuntu/Dockerfile`): Removed GDAL development dependencies to align with the updated build process.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
* init
* fix
* revert
* Create docker-image.yml
* Update docker-image.yml
* Update docker-image.yml
* set image
* clean up
* image tag and build
* typo
* fix naming
* rename
* arm64
* add arm64
* update
* set default data path
* generate cert files in start up scripts
* set up for cloud native PG
* init
* gateway
* using ENV to set arguments
* remove password
* set name
* add citus back
* update trigger
* push to GHCR
* generate manifest to GHCR
* update image name
* Revert "update image name"
This reverts commit 0279d40a56adce11920f512854190c19edfaa13b.
* rename
* rename
* set image name and tags
* set input for manifest tag
* make documentdb version configurable
* add port check
* use matrix for clearance
* set default value
* set env
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Bumoing citus version to 12.1.7
----
#### AI description (iteration 1)
#### PR Classification
Version update.
#### PR Summary
This pull request updates the Citus version from 12.1.6 to 12.1.7, ensuring the project uses the latest stable release.
- Updated `CITUS_VERSION` in `/oss/scripts/setup_versions.sh` to `v12.1.7`.
- Updated `commitHash` in `/cgmanifest.json` to match the new Citus version.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4170021
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Get background index builds to work for single-node documentdb.
This changeset fixes background index builds for OSS by:
1. Scheduling the background index build tasks
2. Moving the distributed functionality (which had not being refactored out of the indexing code during the move to OSS) into distributed hooks, with the base implementation for single-node.
3. Refactoring SQL tests to get even with the above changes.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix for background index builds in DocumentDB.
#### PR Summary
This pull request addresses issues with background index builds in DocumentDB by adding comprehensive tests and handling various edge cases.
- `commands_create_indexes_background.out`: Added extensive test cases to validate the behavior of `create_indexes_background` function, including handling of invalid inputs, unique constraints, and various index specifications.
- Introduced a new schema and function to manage index job statuses.
- Enhanced error handling and validation for index creation requests, ensuring proper error messages and statuses are returned for invalid operations.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
This pr adds a Rust gateway for documentdb in the pg_documentdb_gw folder
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) InitializeBsonIndexTerm called BsonValueToPgbsonElement that was using the bson_iter_next a lot. This is a lot of overhead when we know it's a single element. Use an unsafe variant of it to generate a pgbsonelement
2) Introduce a MatchAndRecurse option for index paths that allows the term generation logic to know whether it's an exact match or if the child paths also match. This is currently unused, but will be used in a subsequent change.
----
#### AI description (iteration 1)
#### PR Classification
Code improvement for term generation in wildcard index.
#### PR Summary
This pull request enhances the term generation process for wildcard indexes by introducing an unsafe transformation method and refining index traversal options.
- `pgbsonelement.c`: Added `FillPgbsonElementUnsafe` and `BsonValueToPgbsonElementUnsafe` functions for unsafe BSON element transformation.
- `bson_gin_entrypoint.c`: Introduced `IndexTraverse_MatchAndRecurse` option to improve path matching and recursion.
- `feature_flag_configs.c`: Added feature flag `indexTermUseUnsafeTransform` to enable unsafe transformation.
- `gin_index_term.c`: Utilized the unsafe transformation method based on the new feature flag.
- `bson_gin_core.c`: Updated term path generation to handle the new traversal option.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
In this pull request, update the Makefile.versions and scripts/generate_extension_version.sh to avoid build failures when using a tarball from tags.
In release tarball there is no .git/index file, so we need to check if the file exists before using it. If the file does not exist, we will set some variables to unknown and use the version from the control file.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix to avoid build failure with tarball from tags.
#### PR Summary
This pull request updates the build scripts to handle cases where the build is performed from a tarball without a Git repository.
- `oss/scripts/generate_extension_version.sh`: Added checks to handle builds outside a Git repository by using version information from a control file.
- `oss/Makefile.versions`: Modified to pass the Git index directory to the version generation script and removed dependency on the Git index file.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Upgrade dependency
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
- Upgrade Postgis to 3.5.2
### Description
Upgrade Postgis to 3.5.2
----
#### AI description (iteration 1)
#### PR Classification
Version upgrade
#### PR Summary
This pull request upgrades PostGIS to version 3.5.2.
- Updated `cgmanifest.json` to reflect the new PostGIS version and download URL.
- Modified `oss/scripts/setup_versions.sh` to set the PostGIS version to 3.5.2.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Error message:
HEAD is now at 8c80bd84d Fix configure with CMake < 3.x
CMake Error at CMakeLists.txt:1 (cmake_minimum_required):
Compatibility with CMake < 3.5 has been removed from CMake.
Reference of this fix:
Reference: https://github.com/citusdata/citus/blob/main/STYLEGUIDE.md#using-citus_indent
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request fixes a CMake version error when installing Uncrustify by switching from `git clone` to `curl` for downloading the source.
- `oss/scripts/install_citus_indent.sh`: Replaced `git clone` with `curl` to download Uncrustify and updated installation commands.
- `cgmanifest.json`: Updated Uncrustify component type and download URL.
- `oss/scripts/setup_versions.sh`: Added function to get Uncrustify version and defined `CRUSTIFY_VERSION`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Support users with setting up on arm64
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
DevOps
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
Support users with setting up DocumentDB on arm64
### Description
tested with Azure VM on ARM64
----
#### AI description (iteration 1)
#### PR Classification
Code update to support ARM64 architecture in Dockerfile.
#### PR Summary
This pull request updates the Dockerfile to support ARM64 architecture and upgrades the base image to Ubuntu 22.04. It also refactors installation scripts and environment variables.
- `Dockerfile`: Upgraded base image to Ubuntu 22.04, added support for ARM64, refactored installation scripts.
- `install_setup_intel_decimal_math_lib.sh`: Changed source retrieval method from tarball to git.
- `setup_versions.sh`: Updated version for Intel Decimal Math Library.
- `.devcontainer/Dockerfile`: Upgraded base image to Ubuntu 22.04, added support for ARM64.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
[Infra] adding support for documentdb_distributed extension in start_oss_server
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for the `documentdb_distributed` extension in the `start_oss_server` script.
- `start_oss_server.sh`: Added a new option `-x` to start the server with the `documentdb_distributed` extension and updated the script to handle this new option.
- `utils.sh`: Added a new function `AddNodeToCluster` to add nodes to the cluster when the `documentdb_distributed` extension is used.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
citus script:
line 46 throws error when ProcessCitusMakefileGlobal does not exist
libbson script:
save the downloaded file locally to avoid run tar command with incompleted file
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Infra
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Setup initial infrastructure for building with Pg17.
Set up TODOs for work that's remaining
Disable tests and builds that don't yet work with Pg17.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for building container images with PostgreSQL 17.
- `.pipelines/pgmongo_build_containers.yml`: Added a new stage `Build_PG17` to build PG17 images, including a job to build the Mariner base container image with PostgreSQL 17.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
[OSS] Fix TTL indexes in documentdb
This moves the function to C and instead of querying the indexes and then getting the collection shards from pg_dist_shard table, it uses the C hook to get the table shards where for non-distributed it returns the base table and then uses that information to drop the expired records.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request addresses issues with TTL indexes in DocumentDB by refactoring and optimizing the deletion of expired rows.
- `ttl_index.c`: Refactored the function to delete expired rows, added a new function `delete_expired_rows`, and optimized the deletion process.
- `ttl_support_functions--latest.sql` and `delete_expired_rows--latest.sql`: Replaced PL/pgSQL procedures with C language implementations for better performance.
- `collection.c`: Moved the function `GetMongoCollectionShardOidsAndNames` from `coll_stats.c` to `collection.c` for better organization.
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Enable support for current_op aggregation stage in OSS.
Still pending: support for getting locking pids and index builds.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request enables support for the `current_op` aggregation stage in DocumentDB.
- Added new functions in `current_op--0.101-0.sql` and `current_op--latest.sql` to support the `current_op` aggregation stage.
- Modified `current_op.c` to handle new attributes and improve the `current_op` command processing.
- Updated `system_configs.c` to include a new configuration variable for tracking the application name in `current_op`.
- Adjusted test cases in `bson_aggregation_pipeline_tests_coll_agnostic.sql` and related expected output files to validate the new feature.
- Updated `metadata_cache.c` to correctly reference the `current_op_aggregation` function.
* Merged PR 1565985: [OSS] Enable DBStats and CollStats in documentdb
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Enable DBStats, CollStats and indexStats in DocumentDB. These were partially broken due to their dependence on Citus. Add appropriate hooks to ensure that Distributed continues to work.
Also clean up the code reuse of run_command_on_all_nodes across the 4 diagnostic commands.
CurrentOp and index builds are also broken but will be tackled in a separate change.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request enables the `DBStats` and `CollStats` commands in DocumentDB.
- `db_stats.c` and `coll_stats.c`: Refactored functions to retrieve shard OIDs and names, returning boolean status.
- `distributed_hooks.c`: Added core function to get shard IDs and names for a collection.
- Added new regression tests in `bson_aggregation_pipeline_diagnostic_command_tests.out` to validate `DBStats` and `CollStats` functionality.
* Merged PR 1565678: Updated README to remove architecture images per PGCA guidelines
### Does this PR have any customer impact?
No.
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Documentation
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Per PGCA guidelines, we are temporarily removing the architecture image and replacing it with an approved one soon.
----
#### AI description (iteration 1)
#### PR Classification
Documentation update
#### PR Summary
This pull request updates the README to comply with PGCA guidelines by removing architecture images.
- `oss/README.md`: Removed the architecture image.
* DocumentDB sync sha=f3b8a6ec5
---------
Co-authored-by: Vinod Sridharan <visridha@microsoft.com>
Co-authored-by: Patty Chow <patty.chow@microsoft.com>
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Infra
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
We have validation scripts we run internally. Run a subset of them on OSS
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds validation scripts to the OSS setup to ensure code quality and consistency.
- Added `oss/scripts/validate_extension_file_state.sh` to validate no Citus references and check SQL file versions.
- Updated `/.pipelines/templates/oss_documentdb_build.yml` to include the new validation script in the build process.
- Modified `/oss/.github/workflows/regress_tests.yml` to run the new validation script during regression tests.
- Updated messages in `oss/scripts/validate_headers.sh` for clarity.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Fixed OSS issue: https://github.com/microsoft/documentdb/issues/40
`start_oss_server script -c` option is broken.
Once the `search_path` is set in ~/.psqlrc that is used for creating the extensions until unless an override is provided the install location for dependent extension started deviating from `public`
```SQL
SELECT e.extname AS extension_name,
n.nspname AS schema_name
FROM pg_extension e
JOIN pg_namespace n ON e.extnamespace = n.oid
WHERE e.extname IN ('documentdb', 'documentdb_core', 'pg_cron', 'tsm_system_rows', 'vector', 'postgis', 'rum');
extension_name | schema_name
-----------------+-----------------
documentdb_core | public
pg_cron | pg_catalog
tsm_system_rows | documentdb_core
vector | documentdb_core
postgis | documentdb_core
rum | documentdb_core
documentdb | public
(7 rows)
```
Always `CREATE EXTENSION` without psqlrc shell overrides for deterministic behavior.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request fixes the `-c` option in the `scripts/start_oss_server` script which was causing issues.
- `oss/scripts/utils.sh`: Added `SCHEMA public` to the `CREATE EXTENSION` command to ensure proper schema usage.
[OSS] fix regress sql test workflow
----
Bug fix for the SQL test workflow.
This pull request fixes the regression in the SQL test workflow by updating the GitHub Actions configuration and related scripts.
- Updated `.github/workflows/regress_tests.yml` to improve the build matrix, install dependencies, and run tests for multiple PostgreSQL versions and Ubuntu versions.
- Modified `scripts/start_oss_server.sh` to correct the help message and default directory.
- Minor cleanup in `scripts/install_setup_pg_cron.sh` to remove an unnecessary blank line.
* Merged PR 1557887: Change error file mapping name
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Rename error file mapping
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request updates the name of the error mapping file and adjusts references accordingly.
- `Makefile`: Updated references from `mongoerrors.csv` to `error_mappings.csv`.
- `docs/pgmongo/design_docs/data_plane/backend_common/error_management.md`: Updated documentation to reflect the new file name.
- `src/utils/documentdb_conversion_codes.c`: Updated comment to reference the new file name.
- Moved `oss/mongoerrors.csv` to `oss/error_mappings.csv` without code changes.
* Merged PR 1555452: [OSS] Enable distributed tests in documentdb
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Enable distributed tests in documentdb, and clean up helio references from documentdb_distributed
Add substitution to make pg_helio_distributed tests work
Add substitution to make pgmongo/helio_compat tests work
Move udfs bson_aggregation_redact to documentdb
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request enables distributed tests in DocumentDB by adding a new set of regression tests for BSON aggregation comparison operators.
- Added `bson_aggregation_comparison_operators_tests.out` with extensive test cases for BSON comparison operators (`$cmp`, `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`).
* Merged PR 1555841: [OSS] fix regress sql test workflow
[OSS] fix regress sql test workflow
----
Bug fix for the SQL test workflow.
This pull request fixes the regression in the SQL test workflow by updating the GitHub Actions configuration and related scripts.
- Updated `.github/workflows/regress_tests.yml` to improve the build matrix, install dependencies, and run tests for multiple PostgreSQL versions and Ubuntu versions.
- Modified `scripts/start_oss_server.sh` to correct the help message and default directory.
- Minor cleanup in `scripts/install_setup_pg_cron.sh` to remove an unnecessary blank line.
---------
Co-authored-by: Vinod Sridharan <visridha@microsoft.com>
Co-authored-by: Zinan Ren <zinanren@microsoft.com>
Move test files from pg_helio_distributed to pg_documentdb_distributed
Clean up pg_documentdb
----
Code cleanup
This pull request cleans up the `helio_api` references and updates them to `documentdb`.
- `custom_query_scan.c` and `custom_scan_continuation.c`: Updated custom scan names from `HelioApi` to `DocumentDBApi`.
- Deleted `commands_coll_stats.out` and `commands_coll_stats.sql` test files.
- `start_oss_server.sh`: Removed `helio` server type and related logic.
- `setup_psqlrc.sh`: Updated search path and settings from `helio` to `documentdb`.
- `.devcontainer/Dockerfile` and `.github/containers/Build-Ubuntu/Dockerfile`: Updated user and labels from `heliodb` to `documentdb`.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
In oss, we noticed frequently sql test result difference introduced by newer package minor versions (postgres, pgvector, postgis, etc).
To make oss CI integration stable and to make test result consistent in oss and pgmongo, we should specify fixed versions for packages in oss, and align with pgmongo.
Will depend on manual package version update after verifying the compatibility and meanwhile update test results.
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup and enhancement.
#### PR Summary
This pull request declares and uses fixed versions of various packages to ensure consistency and reliability in the build environment.
- Updated `/oss/.devcontainer/Dockerfile` to declare and use fixed versions for multiple dependencies including PostgreSQL, Citus, libbson, PG_CRON, PG_VECTOR, Intel Decimal Math Library, PCRE2, and PostGIS.
- Modified `/.devcontainer/Dockerfile` to align with the fixed versioning approach and added necessary environment variables.
- Added `GetPostgresSourceRef` function in `oss/scripts/utils.sh` to map PostgreSQL versions to specific source references.
- Updated `oss/scripts/install_setup_intel_decimal_math_lib.sh` and `oss/scripts/install_setup_pcre2.sh` to accept version parameters.
- Moved several scripts from `/scripts` to `/oss/scripts` for better organization.
Related work items: #3571853
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Move most of helio_api out of OSS into internal.
Keep internal/pg_helio_distributed/src/test/regress for now until we port distributed tests over.
----
#### AI description (iteration 1)
#### PR Classification
Code reorganization to move most of `helio_api` out of the OSS directory.
#### PR Summary
This pull request reorganizes the project structure by moving the `helio_api` components out of the OSS directory to a new `helio_api` directory.
- Deleted files `diff-filter`, `diff`, `normalize.sed`, and `expect_normalize.sed` from `oss/internal/pg_helio_distributed/src/test/regress/bin`.
- Updated paths in `.pipelines/templates/oss_build.yml` and `.pipelines/templates/oss_documentdb_build.yml` to reflect the new `helio_api` directory structure.
- Modified `Makefile` and `oss/Makefile` to update paths and add new targets for `helio_api`.
- Moved `pg_helio_api`, `pg_helio_core`, and `pg_helio_distributed` directories from `oss` to `helio_api`.
- Added new `helio_api/Makefile` and `helio_api/Makefile.inc` to manage the build process for the moved components.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Porting udfs into documentdb
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup and feature enhancement.
#### PR Summary
This pull request continues the porting of UDFs into DocumentDB, updating and including various SQL files for better functionality and organization.
- Updated `documentdb--0.24-0.sql` to include new UDFs and operators, replacing old function references.
- Added new UDFs for query, projection, geospatial, aggregation, metadata, schema management, index management, CRUD commands, user management, schema validation, telemetry, TTL, vector search, extension version, diagnostic, and RBAC metadata.
- Removed outdated function definitions and replaced them with updated SQL includes.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Remove helio from file name
Remove helio references from source content except schema reference
Start local documentdb server with the documentdb extension
Regression tests using the documentdb extension
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request removes references to "helio" from the `documentdb_core` source code.
- `scripts/utils.sh`: Renamed variables from `preloadExtensions` to `preloadLibraries`.
- `pg_documentdb_core/include/utils/error_utils.h`: Updated function and macro names to replace "Helio" with "DocumentDB".
- `pg_documentdb_core/include/io/helio_bson_core.h` and `pg_documentdb_core/include/query/helio_bson_compare.h`: Moved to `pg_documentdb_core/include/io/bson_core.h` and `pg_documentdb_core/include/query/bson_compare.h`.
- `pg_documentdb_core/src/collation/collation.c` and `pg_documentdb_core/src/io/pgbson.c`: Replaced error codes and messages to use "DocumentDB" instead of "Helio".
- `scripts/validate_helio_extension_state.sh`: Added exceptions for certain files to skip validation for "helio" references.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Move source files under new directory.
----
#### AI description (iteration 1)
#### PR Classification
Code reorganization
#### PR Summary
This pull request moves the `helio_api` and `helio_distributed` sources under the `documentdb` directory, updating references and initialization logic accordingly.
- Added `pg_documentdb.c` for initializing the `pg_documentdb` extension.
- Added `documentdb_distributed.c` for initializing the `pg_documentdb_distributed` extension.
- Updated `validate_helio_extension_state.sh` to include new checks for `documentdb`.
- Moved `helio_api_extension_version.c` to `documentdb_extension_version.c` without code changes.
- Updated various `Makefile` configurations to reflect the new directory structure and dependencies.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Add [-t] - optional argument. serverType is the type of server to start.
Supported values are helio and documentdb, default is helio"
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds an option for starting a DocumentDB server and includes some script adjustments.
- Moved `oss/scripts/start_helio_server.sh` to `oss/scripts/start_oss_server.sh`.
- Updated `oss/scripts/utils.sh` to support additional preload extensions for PostgreSQL configurations.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Not yet
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Initial setup for documentdb extensions.
Create Makefiles and set up build scripts around documentdb.
This does not yet touch the actual code or make anything deployable.
----
#### AI description (iteration 1)
#### PR Classification
Initial setup for DocumentDB integration.
#### PR Summary
This pull request sets up the initial structure and configuration for DocumentDB integration.
- Added `documentdb_errors.h` for defining DocumentDB-specific error codes.
- Created `Makefile` for `pg_documentdb`, `pg_documentdb_core`, and `pg_documentdb_distributed` to handle build configurations.
- Updated `validate_helio_extension_state.sh` and `validate_headers.sh` scripts to include checks for DocumentDB.
- Modified `bson_aggregation_nested_pipeline.c`, `bson_aggregation_vector_search.c`, and other source files to replace references from Helio to DocumentDB.
- Added control files for `pg_documentdb`, `pg_documentdb_core`, and `pg_documentdb_distributed` extensions.
### Does this PR have any customer impact?
No.
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
N/A.
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No.
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No.
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Upgrading libbson dependency.
----
#### AI description (iteration 1)
#### PR Classification
Dependency upgrade
#### PR Summary
This pull request upgrades the `libbson` library to version 1.28.0 and updates related initialization code.
- Updated `MONGO_DRIVER_VERSION` to 1.28.0 in `oss/scripts/install_setup_libbson.sh`.
- Added a `NULL` entry to the `bson_mem_vtable_t` structure in `oss/pg_helio_core/src/bson_init.c`.
Related work items: #3468225
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Move Error map reference into a helio and mongo set of values. This splits error mapping to helio and error for other implementations like mongo wire protocol.
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request moves the error map reference to a generated folder, cleaning up the source directory.
- Removed Mongo error code definitions from `oss/pg_helio_core/include/utils/helio_errors.h`.
- Added `oss/pg_helio_core/generated/include/utils/helio_error_conversion.h` with Mongo error code definitions.
- Updated `oss/scripts/generate_error_values.sh` to handle the new generated header file.
- Moved `oss/pg_helio_core/src/utils/helio_errors.c` to `oss/pg_helio_core/generated/src/utils/helio_error_conversion.c`.
- Updated `pgmongo/Makefile` to include the new generated directory in the build process.
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request moves the error map reference to a generated folder, separating it from the source directory.
- `oss/pg_helio_core/include/utils/helio_errors.h`: Removed Mongo error code definitions.
- `oss/pg_helio_core/generated/include/utils/helio_error_conversion.h`: Added new file for Mongo error code definitions.
- `oss/scripts/generate_error_values.sh`: Updated script to generate new header and source files for error conversion.
- `pgmongo/Makefile`: Updated include paths to reference the new generated directory.
- Moved `oss/pg_helio_core/src/utils/helio_errors.c` to `oss/pg_helio_core/generated/src/utils/helio_error_conversion.c` without code changes.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Features
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Add code generated helper function to convert helio errors to mongo error codes.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces a new helper function to convert Helio error codes to Mongo error codes.
- Added `helio_errors.c` to implement the conversion logic.
- Updated `generate_error_values.sh` to generate the new conversion file.
- Modified `Makefile` to include the new conversion file in the build process.
- Updated `helio_errors.h` to declare the new conversion function.
### Does this PR have any customer impact?
no
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
no
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
na
### Description
1. Move all legacy errors to new helio errors
2. Remove all refs to mongo_errors.h excluding error_utils.c , there is function which needs backword compatibility
3. Update `scripts/validate_helio_extension_state.sh` to enforce all validations on oss/pg_helio_api parent directory instead of individual sub directories
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup and deprecation.
#### PR Summary
This pull request moves all error handling to `helio_errors` and removes references to `mongo_errors.h`, deprecating certain functions.
- `error_utils.c`: Deprecated `command_throw_mongo_error` and `command_convert_mongo_error_to_postgres` functions.
- `mongo_errors.h`: Removed Mongo error handling code and redefined error code checks.
- `validate_helio_extension_state.sh`: Updated validation script to remove checks for `mongo_errors.h`.
- Added new SQL functions for database commands in `kill_op--1.22-0.sql`, `get_shard_key_value--1.22-0.sql`, and `reindex--1.22-0.sql`.
- Updated test files to reflect deprecation changes in error handling functions.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Move to helio error codes in pg-helio-core
Enforce hygiene in helio core
Move to helio errors
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request refactors error handling by replacing Mongo-specific error codes with Helio-specific error codes and updates error messages accordingly.
- Added `CheckNoMongoErrorCodes` function in `/scripts/validate_helio_extension_state.sh` to ensure no Mongo-specific error codes are used.
- Updated error codes and messages in `oss/pg_helio_core/src/query/bson_compare.c` to use Helio-specific codes.
- Modified error handling in `oss/pg_helio_core/src/types/pcre_regex.c` to replace Mongo error codes with Helio error codes.
- Replaced Mongo error codes with Helio error codes in `oss/pg_helio_core/src/io/pgbsonsequence.c`.
- Updated error codes in `oss/pg_helio_core/src/types/decimal128.c` to use Helio-specific codes.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No(ish)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Start using the new Helio error codes. These are of the format M0000->M0ZZZ
Convert helio_core to use M0001 for BadValue
With the new error syntax, the errmsg is the user facing error message.
Any PII Safe messages intended for the server will now be placed under errdetail_log (rather than errhint).
----
#### AI description (iteration 1)
#### PR Classification
New feature: Introduces a new error format and usage.
#### PR Summary
This pull request implements a new error format and updates the usage of error handling across multiple components.
- `logger_hooks.c`: Adds logic to transform client log data and ensure error details are properly set.
- `PostgresMongoResultExtensions.cs`: Refactors error handling to include sanitized error hints and details.
- `error.rs` and `pg.rs`: Adds support for known PostgreSQL error codes and refines error handling.
- `PostgresError.cs` and `PostgresClient.cs`: Updates error properties to include error hints and details.
- Multiple files in `pg_helio_core`: Replaces `ERRCODE_INVALID_TEXT_REPRESENTATION` with `ERRCODE_HELIO_BADVALUE` for better error categorization.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Use new error codes and move to source generators for errors for the C# GW
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup and new feature.
#### PR Summary
This pull request ports CodeGen and new error definitions to the managed Gateway.
- Added comprehensive error definitions in `include/utils/helio_errors.h`.
- Removed `EtwEventWriterGenerator` class and related code from `Cosmos.Core.Trace.EtwEventWriterGenerator/EtwEventWriterGenerator.cs`.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Generate error codes that are PG compatible
- Code-gen header file for Mongo Errors
- Generate a text file that can be used for other languages.
- In subsequent changes the existing codebase will be updated to consume these errors.
----
#### AI description (iteration 1)
#### PR Classification
New feature: Generate error codes that are PostgreSQL (PG) compatible.
#### PR Summary
This pull request introduces a new feature to generate error codes that are compatible with PostgreSQL.
- Added `helio_errors.h` with definitions for various Helio error codes and their corresponding MongoDB error codes.
- Created `mongoerrors.csv` to map error names to MongoDB error codes and their ordinal positions.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
DevOps
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
N/A
### Description
Updated libbson build to remove libicu as a dependency.
----
#### AI description (iteration 1)
#### PR Classification
Build configuration update to remove `libicu` as a dependency.
#### PR Summary
This pull request updates the build configuration for the `libbson` library to remove `libicu` as a dependency.
- `oss/scripts/install_setup_libbson.sh`: Updated build script to disable ICU support by adding `-DMONGOC_ENABLE_ICU=OFF -DENABLE_ICU=OFF` flags.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes(ish)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Introduce pg_helio_distributed for distribution hooks and make pgmongo depend on that.
Also makes the rum_join_scan loading of private rum bits optional
----
#### AI description (iteration 1)
#### PR Classification
New feature: Introduce Helio Distributed Extension
#### PR Summary
This pull request introduces the Helio Distributed Extension, which adds distributed execution capabilities to the Helio system.
- Removed `shard_colocation.c` from `/pgmongo/src/distribution/` and added `distributed_hooks.c` in `/oss/private/pg_helio_distributed/src/distribution/`.
- Updated `api_hook_impl.c` to use new distributed hooks.
- Added new Makefile and build scripts for the Helio Distributed Extension in `/oss/private/pg_helio_distributed/`.
- Introduced new test validation scripts and configurations in `/oss/private/pg_helio_distributed/src/test/regress/`.
- Updated `pgmongo` Makefile to include the new distributed extension.
Guanzhou Song
Vinod Sridharan
Shuai Tian
Parag Jain
Amit Prabhudesai
Alexander Laye
yangqiaoandroot
Yang Qiao
Shuai Tian
Deepak Bisht
Santiago Fernandez Madero
James Broadhead
Álvaro Hernández Tortosa
Lucas Borges Fernandes
John Pao