Improve log infrastructure to output postgres, gateway and other logs to docker log for better visibility.
----
#### AI description (iteration 1)
#### PR Classification
This pull request improves the log infrastructure by centralizing log management, enhancing log streaming, and adding graceful shutdown handling.
#### PR Summary
The changes standardize logging across components by implementing dynamic log streaming, centralized log directory setup, and proper process cleanup on container shutdown.
- `oss/scripts/emulator_entrypoint.sh`: Added a `cleanup` function and signal traps to gracefully terminate log streaming and gateway processes.
- `oss/scripts/emulator_entrypoint.sh`: Introduced the `start_log_streaming` function to stream logs for PostgreSQL, OSS server, and gateway with configurable prefixes.
- `oss/scripts/emulator_entrypoint.sh`: Configured centralized log directory creation, symlink creation for PostgreSQL logs, and updated log redirections using environment variables.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No.
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature.
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No.
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes.
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Add OSS background_worker initial jobs framework. For this initial version, only schedulable jobs are allowed (i.e. jobs that run on timed intervals). The background worker loop has a latch timeout with a predefined duration of 10 seconds that is configured through a GUC. Every time this timeout occurs, it iterates on the JobRegistry array.
1. Checks if the job is running and if it timed out (i.e. running longer than timeoutInSeconds). If yes, cancels the query and finishes libpq connection.
2. Checks if the job finished running (i.e. libpq connection isn't busy). If yes, finishes the libpq connection.
3. Runs the job if its scheduled interval has reached (i.e. currentTime - lastStartTime >= scheduledInterval). Each job execution has its own libpq connection to the database.
In order to add a new job callers must pass a BackgroundWorkerJob structure to RegisterBackgroundWorkerJob.
----
#### AI description (iteration 1)
#### PR Classification
This pull request introduces a new feature by adding an OSS background worker jobs framework.
#### PR Summary
The changes implement a framework for scheduling, executing, and managing background jobs within the background worker process. This provides structured job processing with timeout, cancellation, and configuration support.
- **`src/background_worker/background_worker.c`**: Added job management functions (e.g., ManageJobs, ExecuteJob, CanExecuteJob, CheckJobCompletion, CancelJobIfTimeIsUp, ValidateJob) and integrated job scheduling in the main loop.
- **`include/background_worker/background_worker_job.h`**: Introduced new structures for defining background worker jobs and their execution objects.
- **`src/api_hook_impl.c`**: Added a hook function to retrieve background worker jobs for registration.
- **`src/configs/background_job_configs.c`**: Defined new configuration variables (GUCs) to enable background worker jobs.
- **API Hooks and Connection Management Files**: Updated declarations and functions (e.g., in `include/api_hooks_def.h` and connection management files) to support connection handling necessary for job execution.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4225917
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
OS
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
PgFlex currently uses Azure Linux AZL 2.0 (internally referred as Mariner 2.0) to deploy the PgFlex VM instances.
AZL 2.0 is reaching EOL by July 2025, so we need to update / move all the flex components to AZL 3.0 (Mariner 3.0)
This is Part 1 where i am just building the base image with AZL3.0. The subsequent Pr will cover the flex-build and the backend-pipeline. Since the changes are big, so breaking the change into smaller Pr .
The subsequent PR is here: https://msdata.visualstudio.com/CosmosDB/_git/pgmongo/pullrequest/1727166
Here is the successful run: https://msdata.visualstudio.com/CosmosDB/_build/results?buildId=179278861&view=results
----
#### AI description (iteration 1)
#### PR Classification
This pull request updates the build configuration and container base images to support the migration to AZL3.0.
#### PR Summary
The changes introduce a new base Docker image built on AzureLinux 3.0 along with pipeline and script adjustments to support AZL3.0 across build, deployment, and OS detection.
- `/.pipelines/Containers/Container-Base-AzureLinux-3.0/Dockerfile`: Added a new Dockerfile that establishes the AZL3.0 base image and installs necessary build dependencies and tools.
- `/.pipelines/pgmongo_build_containers.yml` and `/.pipelines/templates/pgmongo_build_base_containers.yml`: Updated pipeline configurations to include new build jobs and tagging logic for the AZL3.0 SKU.
- Updated OS detection logic in `oss/scripts/utils.sh`, `scripts/install_setup_pgaadauth.sh`, `scripts/pipeline_deploy_utils.sh`, and `scripts/utils.sh` to recognize the `azurelinux` identifier.
- `scripts/install_setup_openssl_1_1.sh` and `/.pipelines/FlexContainers/Deployment-AzureLinx-3.0/Dockerfile`: Added new files for installing OpenSSL 1.1.1 from source and for deploying PgMongo artifacts using AZL3.0.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4330937
Require local auth if external IP is flag is enabled:
If the allowExternalAccess flag is enabled on the Postgres server, this will require to use username password to connect.
Require Password to Start Docker Container
password should always be provided to protect the database
$ docker run doc:001 --username guanzhou
Error: PASSWORD is required. Please provide a password using --password argument or PASSWORD environment variable.
----
#### AI description (iteration 1)
#### PR Classification
This pull request implements a security enhancement by enforcing mandatory password authentication and stronger connection methods for external access.
#### PR Summary
The pull request ensures that a password must be provided at startup and updates the authentication configuration for PostgreSQL in external access scenarios.
- `oss/scripts/emulator_entrypoint.sh`: Added a validation check that exits when PASSWORD is missing and updated the CLI help to mark the password as REQUIRED.
- `oss/scripts/start_oss_server.sh`: Changed PostgreSQL settings to use scram-sha-256 for external IP connections instead of the "trust" method.
- `oss/.github/containers/Build-Ubuntu/Dockerfile_gateway`: Removed the default PASSWORD value, requiring explicit configuration.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
NO
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Cleanup
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
NO
### Are you introducing any new config? If yes, do you have tests with and without them being set?
NO
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
cleanup OSS scripts
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Use OSS RUM source in OSS build
----
#### AI description (iteration 1)
#### PR Classification
This pull request implements a feature update to enable the OSS build to use the OSS RUM source through configurable parameters and script modifications.
#### PR Summary
The PR refactors build scripts and extension routines to support toggling between private and OSS RUM sources. Key changes include:
- **`scripts/pipeline_build_extension_core.sh`**: Converted positional parameters to flag-based arguments using getopts and added a conditional to select OSS RUM installation.
- **`pipelines/templates/extension_build_core.yml` & `.pipelines/templates/oss_documentdb_build.yml`**: Introduced the `useOssRumRepo` parameter to control whether the OSS RUM source is used.
- **`/oss/pg_documentdb/src/index_am/rum.c`**: Updated index routine functions to directly call OSS-specific implementations when the new composite index feature is enabled.
- **`scripts/build_dev_flex_container.sh`**: Adjusted the call to the pipeline build script to use the new flag-based argument format.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Optimize the fix for postgis 3.5
Related postgis issue: https://trac.osgeo.org/postgis/ticket/5652
----
This pull request is a bug fix that updates the PostGIS 3.5 build process.
The changes modify the installation script and update Docker container configurations to resolve build issues for PostGIS 3.5 by removing unused GDAL components and avoiding parallel build race conditions.
- `oss/scripts/install_setup_postgis.sh`: Added a call to `./autogen.sh`, updated configuration options by removing GDAL use and adding `--without-raster`, and replaced parallel `make` with serial execution.
- Multiple Dockerfiles (`.pipelines/Containers/Container-Base-Mariner/Dockerfile`, `/.devcontainer/Dockerfile`, `/.pipelines/Containers/Container-Base-Ubuntu/Dockerfile`, `/oss/.devcontainer/Dockerfile`, and `/oss/.github/containers/Build-Ubuntu/Dockerfile`): Removed GDAL development dependencies to align with the updated build process.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
* init
* fix
* revert
* Create docker-image.yml
* Update docker-image.yml
* Update docker-image.yml
* set image
* clean up
* image tag and build
* typo
* fix naming
* rename
* arm64
* add arm64
* update
* set default data path
* generate cert files in start up scripts
* set up for cloud native PG
* init
* gateway
* using ENV to set arguments
* remove password
* set name
* add citus back
* update trigger
* push to GHCR
* generate manifest to GHCR
* update image name
* Revert "update image name"
This reverts commit 0279d40a56adce11920f512854190c19edfaa13b.
* rename
* rename
* set image name and tags
* set input for manifest tag
* make documentdb version configurable
* add port check
* use matrix for clearance
* set default value
* set env
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Bumoing citus version to 12.1.7
----
#### AI description (iteration 1)
#### PR Classification
Version update.
#### PR Summary
This pull request updates the Citus version from 12.1.6 to 12.1.7, ensuring the project uses the latest stable release.
- Updated `CITUS_VERSION` in `/oss/scripts/setup_versions.sh` to `v12.1.7`.
- Updated `commitHash` in `/cgmanifest.json` to match the new Citus version.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4170021
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Get background index builds to work for single-node documentdb.
This changeset fixes background index builds for OSS by:
1. Scheduling the background index build tasks
2. Moving the distributed functionality (which had not being refactored out of the indexing code during the move to OSS) into distributed hooks, with the base implementation for single-node.
3. Refactoring SQL tests to get even with the above changes.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix for background index builds in DocumentDB.
#### PR Summary
This pull request addresses issues with background index builds in DocumentDB by adding comprehensive tests and handling various edge cases.
- `commands_create_indexes_background.out`: Added extensive test cases to validate the behavior of `create_indexes_background` function, including handling of invalid inputs, unique constraints, and various index specifications.
- Introduced a new schema and function to manage index job statuses.
- Enhanced error handling and validation for index creation requests, ensuring proper error messages and statuses are returned for invalid operations.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
This pr adds a Rust gateway for documentdb in the pg_documentdb_gw folder
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) InitializeBsonIndexTerm called BsonValueToPgbsonElement that was using the bson_iter_next a lot. This is a lot of overhead when we know it's a single element. Use an unsafe variant of it to generate a pgbsonelement
2) Introduce a MatchAndRecurse option for index paths that allows the term generation logic to know whether it's an exact match or if the child paths also match. This is currently unused, but will be used in a subsequent change.
----
#### AI description (iteration 1)
#### PR Classification
Code improvement for term generation in wildcard index.
#### PR Summary
This pull request enhances the term generation process for wildcard indexes by introducing an unsafe transformation method and refining index traversal options.
- `pgbsonelement.c`: Added `FillPgbsonElementUnsafe` and `BsonValueToPgbsonElementUnsafe` functions for unsafe BSON element transformation.
- `bson_gin_entrypoint.c`: Introduced `IndexTraverse_MatchAndRecurse` option to improve path matching and recursion.
- `feature_flag_configs.c`: Added feature flag `indexTermUseUnsafeTransform` to enable unsafe transformation.
- `gin_index_term.c`: Utilized the unsafe transformation method based on the new feature flag.
- `bson_gin_core.c`: Updated term path generation to handle the new traversal option.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
In this pull request, update the Makefile.versions and scripts/generate_extension_version.sh to avoid build failures when using a tarball from tags.
In release tarball there is no .git/index file, so we need to check if the file exists before using it. If the file does not exist, we will set some variables to unknown and use the version from the control file.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix to avoid build failure with tarball from tags.
#### PR Summary
This pull request updates the build scripts to handle cases where the build is performed from a tarball without a Git repository.
- `oss/scripts/generate_extension_version.sh`: Added checks to handle builds outside a Git repository by using version information from a control file.
- `oss/Makefile.versions`: Modified to pass the Git index directory to the version generation script and removed dependency on the Git index file.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Upgrade dependency
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
- Upgrade Postgis to 3.5.2
### Description
Upgrade Postgis to 3.5.2
----
#### AI description (iteration 1)
#### PR Classification
Version upgrade
#### PR Summary
This pull request upgrades PostGIS to version 3.5.2.
- Updated `cgmanifest.json` to reflect the new PostGIS version and download URL.
- Modified `oss/scripts/setup_versions.sh` to set the PostGIS version to 3.5.2.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Error message:
HEAD is now at 8c80bd84d Fix configure with CMake < 3.x
CMake Error at CMakeLists.txt:1 (cmake_minimum_required):
Compatibility with CMake < 3.5 has been removed from CMake.
Reference of this fix:
Reference: https://github.com/citusdata/citus/blob/main/STYLEGUIDE.md#using-citus_indent
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request fixes a CMake version error when installing Uncrustify by switching from `git clone` to `curl` for downloading the source.
- `oss/scripts/install_citus_indent.sh`: Replaced `git clone` with `curl` to download Uncrustify and updated installation commands.
- `cgmanifest.json`: Updated Uncrustify component type and download URL.
- `oss/scripts/setup_versions.sh`: Added function to get Uncrustify version and defined `CRUSTIFY_VERSION`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
Support users with setting up on arm64
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
DevOps
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
Support users with setting up DocumentDB on arm64
### Description
tested with Azure VM on ARM64
----
#### AI description (iteration 1)
#### PR Classification
Code update to support ARM64 architecture in Dockerfile.
#### PR Summary
This pull request updates the Dockerfile to support ARM64 architecture and upgrades the base image to Ubuntu 22.04. It also refactors installation scripts and environment variables.
- `Dockerfile`: Upgraded base image to Ubuntu 22.04, added support for ARM64, refactored installation scripts.
- `install_setup_intel_decimal_math_lib.sh`: Changed source retrieval method from tarball to git.
- `setup_versions.sh`: Updated version for Intel Decimal Math Library.
- `.devcontainer/Dockerfile`: Upgraded base image to Ubuntu 22.04, added support for ARM64.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
[Infra] adding support for documentdb_distributed extension in start_oss_server
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for the `documentdb_distributed` extension in the `start_oss_server` script.
- `start_oss_server.sh`: Added a new option `-x` to start the server with the `documentdb_distributed` extension and updated the script to handle this new option.
- `utils.sh`: Added a new function `AddNodeToCluster` to add nodes to the cluster when the `documentdb_distributed` extension is used.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
citus script:
line 46 throws error when ProcessCitusMakefileGlobal does not exist
libbson script:
save the downloaded file locally to avoid run tar command with incompleted file
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Infra
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Setup initial infrastructure for building with Pg17.
Set up TODOs for work that's remaining
Disable tests and builds that don't yet work with Pg17.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for building container images with PostgreSQL 17.
- `.pipelines/pgmongo_build_containers.yml`: Added a new stage `Build_PG17` to build PG17 images, including a job to build the Mariner base container image with PostgreSQL 17.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
[OSS] Fix TTL indexes in documentdb
This moves the function to C and instead of querying the indexes and then getting the collection shards from pg_dist_shard table, it uses the C hook to get the table shards where for non-distributed it returns the base table and then uses that information to drop the expired records.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request addresses issues with TTL indexes in DocumentDB by refactoring and optimizing the deletion of expired rows.
- `ttl_index.c`: Refactored the function to delete expired rows, added a new function `delete_expired_rows`, and optimized the deletion process.
- `ttl_support_functions--latest.sql` and `delete_expired_rows--latest.sql`: Replaced PL/pgSQL procedures with C language implementations for better performance.
- `collection.c`: Moved the function `GetMongoCollectionShardOidsAndNames` from `coll_stats.c` to `collection.c` for better organization.
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Enable support for current_op aggregation stage in OSS.
Still pending: support for getting locking pids and index builds.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request enables support for the `current_op` aggregation stage in DocumentDB.
- Added new functions in `current_op--0.101-0.sql` and `current_op--latest.sql` to support the `current_op` aggregation stage.
- Modified `current_op.c` to handle new attributes and improve the `current_op` command processing.
- Updated `system_configs.c` to include a new configuration variable for tracking the application name in `current_op`.
- Adjusted test cases in `bson_aggregation_pipeline_tests_coll_agnostic.sql` and related expected output files to validate the new feature.
- Updated `metadata_cache.c` to correctly reference the `current_op_aggregation` function.
* Merged PR 1565985: [OSS] Enable DBStats and CollStats in documentdb
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Enable DBStats, CollStats and indexStats in DocumentDB. These were partially broken due to their dependence on Citus. Add appropriate hooks to ensure that Distributed continues to work.
Also clean up the code reuse of run_command_on_all_nodes across the 4 diagnostic commands.
CurrentOp and index builds are also broken but will be tackled in a separate change.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request enables the `DBStats` and `CollStats` commands in DocumentDB.
- `db_stats.c` and `coll_stats.c`: Refactored functions to retrieve shard OIDs and names, returning boolean status.
- `distributed_hooks.c`: Added core function to get shard IDs and names for a collection.
- Added new regression tests in `bson_aggregation_pipeline_diagnostic_command_tests.out` to validate `DBStats` and `CollStats` functionality.
* Merged PR 1565678: Updated README to remove architecture images per PGCA guidelines
### Does this PR have any customer impact?
No.
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Documentation
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Per PGCA guidelines, we are temporarily removing the architecture image and replacing it with an approved one soon.
----
#### AI description (iteration 1)
#### PR Classification
Documentation update
#### PR Summary
This pull request updates the README to comply with PGCA guidelines by removing architecture images.
- `oss/README.md`: Removed the architecture image.
* DocumentDB sync sha=f3b8a6ec5
---------
Co-authored-by: Vinod Sridharan <visridha@microsoft.com>
Co-authored-by: Patty Chow <patty.chow@microsoft.com>
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Infra
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
We have validation scripts we run internally. Run a subset of them on OSS
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds validation scripts to the OSS setup to ensure code quality and consistency.
- Added `oss/scripts/validate_extension_file_state.sh` to validate no Citus references and check SQL file versions.
- Updated `/.pipelines/templates/oss_documentdb_build.yml` to include the new validation script in the build process.
- Modified `/oss/.github/workflows/regress_tests.yml` to run the new validation script during regression tests.
- Updated messages in `oss/scripts/validate_headers.sh` for clarity.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Fixed OSS issue: https://github.com/microsoft/documentdb/issues/40
`start_oss_server script -c` option is broken.
Once the `search_path` is set in ~/.psqlrc that is used for creating the extensions until unless an override is provided the install location for dependent extension started deviating from `public`
```SQL
SELECT e.extname AS extension_name,
n.nspname AS schema_name
FROM pg_extension e
JOIN pg_namespace n ON e.extnamespace = n.oid
WHERE e.extname IN ('documentdb', 'documentdb_core', 'pg_cron', 'tsm_system_rows', 'vector', 'postgis', 'rum');
extension_name | schema_name
-----------------+-----------------
documentdb_core | public
pg_cron | pg_catalog
tsm_system_rows | documentdb_core
vector | documentdb_core
postgis | documentdb_core
rum | documentdb_core
documentdb | public
(7 rows)
```
Always `CREATE EXTENSION` without psqlrc shell overrides for deterministic behavior.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request fixes the `-c` option in the `scripts/start_oss_server` script which was causing issues.
- `oss/scripts/utils.sh`: Added `SCHEMA public` to the `CREATE EXTENSION` command to ensure proper schema usage.
[OSS] fix regress sql test workflow
----
Bug fix for the SQL test workflow.
This pull request fixes the regression in the SQL test workflow by updating the GitHub Actions configuration and related scripts.
- Updated `.github/workflows/regress_tests.yml` to improve the build matrix, install dependencies, and run tests for multiple PostgreSQL versions and Ubuntu versions.
- Modified `scripts/start_oss_server.sh` to correct the help message and default directory.
- Minor cleanup in `scripts/install_setup_pg_cron.sh` to remove an unnecessary blank line.
* Merged PR 1557887: Change error file mapping name
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Rename error file mapping
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request updates the name of the error mapping file and adjusts references accordingly.
- `Makefile`: Updated references from `mongoerrors.csv` to `error_mappings.csv`.
- `docs/pgmongo/design_docs/data_plane/backend_common/error_management.md`: Updated documentation to reflect the new file name.
- `src/utils/documentdb_conversion_codes.c`: Updated comment to reference the new file name.
- Moved `oss/mongoerrors.csv` to `oss/error_mappings.csv` without code changes.
* Merged PR 1555452: [OSS] Enable distributed tests in documentdb
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Enable distributed tests in documentdb, and clean up helio references from documentdb_distributed
Add substitution to make pg_helio_distributed tests work
Add substitution to make pgmongo/helio_compat tests work
Move udfs bson_aggregation_redact to documentdb
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request enables distributed tests in DocumentDB by adding a new set of regression tests for BSON aggregation comparison operators.
- Added `bson_aggregation_comparison_operators_tests.out` with extensive test cases for BSON comparison operators (`$cmp`, `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`).
* Merged PR 1555841: [OSS] fix regress sql test workflow
[OSS] fix regress sql test workflow
----
Bug fix for the SQL test workflow.
This pull request fixes the regression in the SQL test workflow by updating the GitHub Actions configuration and related scripts.
- Updated `.github/workflows/regress_tests.yml` to improve the build matrix, install dependencies, and run tests for multiple PostgreSQL versions and Ubuntu versions.
- Modified `scripts/start_oss_server.sh` to correct the help message and default directory.
- Minor cleanup in `scripts/install_setup_pg_cron.sh` to remove an unnecessary blank line.
---------
Co-authored-by: Vinod Sridharan <visridha@microsoft.com>
Co-authored-by: Zinan Ren <zinanren@microsoft.com>
Move test files from pg_helio_distributed to pg_documentdb_distributed
Clean up pg_documentdb
----
Code cleanup
This pull request cleans up the `helio_api` references and updates them to `documentdb`.
- `custom_query_scan.c` and `custom_scan_continuation.c`: Updated custom scan names from `HelioApi` to `DocumentDBApi`.
- Deleted `commands_coll_stats.out` and `commands_coll_stats.sql` test files.
- `start_oss_server.sh`: Removed `helio` server type and related logic.
- `setup_psqlrc.sh`: Updated search path and settings from `helio` to `documentdb`.
- `.devcontainer/Dockerfile` and `.github/containers/Build-Ubuntu/Dockerfile`: Updated user and labels from `heliodb` to `documentdb`.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
In oss, we noticed frequently sql test result difference introduced by newer package minor versions (postgres, pgvector, postgis, etc).
To make oss CI integration stable and to make test result consistent in oss and pgmongo, we should specify fixed versions for packages in oss, and align with pgmongo.
Will depend on manual package version update after verifying the compatibility and meanwhile update test results.
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup and enhancement.
#### PR Summary
This pull request declares and uses fixed versions of various packages to ensure consistency and reliability in the build environment.
- Updated `/oss/.devcontainer/Dockerfile` to declare and use fixed versions for multiple dependencies including PostgreSQL, Citus, libbson, PG_CRON, PG_VECTOR, Intel Decimal Math Library, PCRE2, and PostGIS.
- Modified `/.devcontainer/Dockerfile` to align with the fixed versioning approach and added necessary environment variables.
- Added `GetPostgresSourceRef` function in `oss/scripts/utils.sh` to map PostgreSQL versions to specific source references.
- Updated `oss/scripts/install_setup_intel_decimal_math_lib.sh` and `oss/scripts/install_setup_pcre2.sh` to accept version parameters.
- Moved several scripts from `/scripts` to `/oss/scripts` for better organization.
Related work items: #3571853
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Move most of helio_api out of OSS into internal.
Keep internal/pg_helio_distributed/src/test/regress for now until we port distributed tests over.
----
#### AI description (iteration 1)
#### PR Classification
Code reorganization to move most of `helio_api` out of the OSS directory.
#### PR Summary
This pull request reorganizes the project structure by moving the `helio_api` components out of the OSS directory to a new `helio_api` directory.
- Deleted files `diff-filter`, `diff`, `normalize.sed`, and `expect_normalize.sed` from `oss/internal/pg_helio_distributed/src/test/regress/bin`.
- Updated paths in `.pipelines/templates/oss_build.yml` and `.pipelines/templates/oss_documentdb_build.yml` to reflect the new `helio_api` directory structure.
- Modified `Makefile` and `oss/Makefile` to update paths and add new targets for `helio_api`.
- Moved `pg_helio_api`, `pg_helio_core`, and `pg_helio_distributed` directories from `oss` to `helio_api`.
- Added new `helio_api/Makefile` and `helio_api/Makefile.inc` to manage the build process for the moved components.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Porting udfs into documentdb
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup and feature enhancement.
#### PR Summary
This pull request continues the porting of UDFs into DocumentDB, updating and including various SQL files for better functionality and organization.
- Updated `documentdb--0.24-0.sql` to include new UDFs and operators, replacing old function references.
- Added new UDFs for query, projection, geospatial, aggregation, metadata, schema management, index management, CRUD commands, user management, schema validation, telemetry, TTL, vector search, extension version, diagnostic, and RBAC metadata.
- Removed outdated function definitions and replaced them with updated SQL includes.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Remove helio from file name
Remove helio references from source content except schema reference
Start local documentdb server with the documentdb extension
Regression tests using the documentdb extension
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request removes references to "helio" from the `documentdb_core` source code.
- `scripts/utils.sh`: Renamed variables from `preloadExtensions` to `preloadLibraries`.
- `pg_documentdb_core/include/utils/error_utils.h`: Updated function and macro names to replace "Helio" with "DocumentDB".
- `pg_documentdb_core/include/io/helio_bson_core.h` and `pg_documentdb_core/include/query/helio_bson_compare.h`: Moved to `pg_documentdb_core/include/io/bson_core.h` and `pg_documentdb_core/include/query/bson_compare.h`.
- `pg_documentdb_core/src/collation/collation.c` and `pg_documentdb_core/src/io/pgbson.c`: Replaced error codes and messages to use "DocumentDB" instead of "Helio".
- `scripts/validate_helio_extension_state.sh`: Added exceptions for certain files to skip validation for "helio" references.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Move source files under new directory.
----
#### AI description (iteration 1)
#### PR Classification
Code reorganization
#### PR Summary
This pull request moves the `helio_api` and `helio_distributed` sources under the `documentdb` directory, updating references and initialization logic accordingly.
- Added `pg_documentdb.c` for initializing the `pg_documentdb` extension.
- Added `documentdb_distributed.c` for initializing the `pg_documentdb_distributed` extension.
- Updated `validate_helio_extension_state.sh` to include new checks for `documentdb`.
- Moved `helio_api_extension_version.c` to `documentdb_extension_version.c` without code changes.
- Updated various `Makefile` configurations to reflect the new directory structure and dependencies.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Add [-t] - optional argument. serverType is the type of server to start.
Supported values are helio and documentdb, default is helio"
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds an option for starting a DocumentDB server and includes some script adjustments.
- Moved `oss/scripts/start_helio_server.sh` to `oss/scripts/start_oss_server.sh`.
- Updated `oss/scripts/utils.sh` to support additional preload extensions for PostgreSQL configurations.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Not yet
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Initial setup for documentdb extensions.
Create Makefiles and set up build scripts around documentdb.
This does not yet touch the actual code or make anything deployable.
----
#### AI description (iteration 1)
#### PR Classification
Initial setup for DocumentDB integration.
#### PR Summary
This pull request sets up the initial structure and configuration for DocumentDB integration.
- Added `documentdb_errors.h` for defining DocumentDB-specific error codes.
- Created `Makefile` for `pg_documentdb`, `pg_documentdb_core`, and `pg_documentdb_distributed` to handle build configurations.
- Updated `validate_helio_extension_state.sh` and `validate_headers.sh` scripts to include checks for DocumentDB.
- Modified `bson_aggregation_nested_pipeline.c`, `bson_aggregation_vector_search.c`, and other source files to replace references from Helio to DocumentDB.
- Added control files for `pg_documentdb`, `pg_documentdb_core`, and `pg_documentdb_distributed` extensions.
### Does this PR have any customer impact?
No.
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
N/A.
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No.
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No.
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Upgrading libbson dependency.
----
#### AI description (iteration 1)
#### PR Classification
Dependency upgrade
#### PR Summary
This pull request upgrades the `libbson` library to version 1.28.0 and updates related initialization code.
- Updated `MONGO_DRIVER_VERSION` to 1.28.0 in `oss/scripts/install_setup_libbson.sh`.
- Added a `NULL` entry to the `bson_mem_vtable_t` structure in `oss/pg_helio_core/src/bson_init.c`.
Related work items: #3468225
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Move Error map reference into a helio and mongo set of values. This splits error mapping to helio and error for other implementations like mongo wire protocol.
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request moves the error map reference to a generated folder, cleaning up the source directory.
- Removed Mongo error code definitions from `oss/pg_helio_core/include/utils/helio_errors.h`.
- Added `oss/pg_helio_core/generated/include/utils/helio_error_conversion.h` with Mongo error code definitions.
- Updated `oss/scripts/generate_error_values.sh` to handle the new generated header file.
- Moved `oss/pg_helio_core/src/utils/helio_errors.c` to `oss/pg_helio_core/generated/src/utils/helio_error_conversion.c`.
- Updated `pgmongo/Makefile` to include the new generated directory in the build process.
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request moves the error map reference to a generated folder, separating it from the source directory.
- `oss/pg_helio_core/include/utils/helio_errors.h`: Removed Mongo error code definitions.
- `oss/pg_helio_core/generated/include/utils/helio_error_conversion.h`: Added new file for Mongo error code definitions.
- `oss/scripts/generate_error_values.sh`: Updated script to generate new header and source files for error conversion.
- `pgmongo/Makefile`: Updated include paths to reference the new generated directory.
- Moved `oss/pg_helio_core/src/utils/helio_errors.c` to `oss/pg_helio_core/generated/src/utils/helio_error_conversion.c` without code changes.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Features
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Add code generated helper function to convert helio errors to mongo error codes.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces a new helper function to convert Helio error codes to Mongo error codes.
- Added `helio_errors.c` to implement the conversion logic.
- Updated `generate_error_values.sh` to generate the new conversion file.
- Modified `Makefile` to include the new conversion file in the build process.
- Updated `helio_errors.h` to declare the new conversion function.
### Does this PR have any customer impact?
no
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
no
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
na
### Description
1. Move all legacy errors to new helio errors
2. Remove all refs to mongo_errors.h excluding error_utils.c , there is function which needs backword compatibility
3. Update `scripts/validate_helio_extension_state.sh` to enforce all validations on oss/pg_helio_api parent directory instead of individual sub directories
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup and deprecation.
#### PR Summary
This pull request moves all error handling to `helio_errors` and removes references to `mongo_errors.h`, deprecating certain functions.
- `error_utils.c`: Deprecated `command_throw_mongo_error` and `command_convert_mongo_error_to_postgres` functions.
- `mongo_errors.h`: Removed Mongo error handling code and redefined error code checks.
- `validate_helio_extension_state.sh`: Updated validation script to remove checks for `mongo_errors.h`.
- Added new SQL functions for database commands in `kill_op--1.22-0.sql`, `get_shard_key_value--1.22-0.sql`, and `reindex--1.22-0.sql`.
- Updated test files to reflect deprecation changes in error handling functions.
Guanzhou Song
Alexey Palazhchenko
Lucas Borges Fernandes
Manish Kumar
Yang Qiao
Guanzhou Song
Vinod Sridharan
Shuai Tian
Parag Jain
Amit Prabhudesai
Alexander Laye
yangqiaoandroot
Shuai Tian
Deepak Bisht
Santiago Fernandez Madero
James Broadhead
Álvaro Hernández Tortosa
