### Does this PR have any customer impact?
No.
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature.
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No.
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes.
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Add OSS background_worker initial jobs framework. For this initial version, only schedulable jobs are allowed (i.e. jobs that run on timed intervals). The background worker loop has a latch timeout with a predefined duration of 10 seconds that is configured through a GUC. Every time this timeout occurs, it iterates on the JobRegistry array.
1. Checks if the job is running and if it timed out (i.e. running longer than timeoutInSeconds). If yes, cancels the query and finishes libpq connection.
2. Checks if the job finished running (i.e. libpq connection isn't busy). If yes, finishes the libpq connection.
3. Runs the job if its scheduled interval has reached (i.e. currentTime - lastStartTime >= scheduledInterval). Each job execution has its own libpq connection to the database.
In order to add a new job callers must pass a BackgroundWorkerJob structure to RegisterBackgroundWorkerJob.
----
#### AI description (iteration 1)
#### PR Classification
This pull request introduces a new feature by adding an OSS background worker jobs framework.
#### PR Summary
The changes implement a framework for scheduling, executing, and managing background jobs within the background worker process. This provides structured job processing with timeout, cancellation, and configuration support.
- **`src/background_worker/background_worker.c`**: Added job management functions (e.g., ManageJobs, ExecuteJob, CanExecuteJob, CheckJobCompletion, CancelJobIfTimeIsUp, ValidateJob) and integrated job scheduling in the main loop.
- **`include/background_worker/background_worker_job.h`**: Introduced new structures for defining background worker jobs and their execution objects.
- **`src/api_hook_impl.c`**: Added a hook function to retrieve background worker jobs for registration.
- **`src/configs/background_job_configs.c`**: Defined new configuration variables (GUCs) to enable background worker jobs.
- **API Hooks and Connection Management Files**: Updated declarations and functions (e.g., in `include/api_hooks_def.h` and connection management files) to support connection handling necessary for job execution.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4225917
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
OS
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
PgFlex currently uses Azure Linux AZL 2.0 (internally referred as Mariner 2.0) to deploy the PgFlex VM instances.
AZL 2.0 is reaching EOL by July 2025, so we need to update / move all the flex components to AZL 3.0 (Mariner 3.0)
This is Part 1 where i am just building the base image with AZL3.0. The subsequent Pr will cover the flex-build and the backend-pipeline. Since the changes are big, so breaking the change into smaller Pr .
The subsequent PR is here: https://msdata.visualstudio.com/CosmosDB/_git/pgmongo/pullrequest/1727166
Here is the successful run: https://msdata.visualstudio.com/CosmosDB/_build/results?buildId=179278861&view=results
----
#### AI description (iteration 1)
#### PR Classification
This pull request updates the build configuration and container base images to support the migration to AZL3.0.
#### PR Summary
The changes introduce a new base Docker image built on AzureLinux 3.0 along with pipeline and script adjustments to support AZL3.0 across build, deployment, and OS detection.
- `/.pipelines/Containers/Container-Base-AzureLinux-3.0/Dockerfile`: Added a new Dockerfile that establishes the AZL3.0 base image and installs necessary build dependencies and tools.
- `/.pipelines/pgmongo_build_containers.yml` and `/.pipelines/templates/pgmongo_build_base_containers.yml`: Updated pipeline configurations to include new build jobs and tagging logic for the AZL3.0 SKU.
- Updated OS detection logic in `oss/scripts/utils.sh`, `scripts/install_setup_pgaadauth.sh`, `scripts/pipeline_deploy_utils.sh`, and `scripts/utils.sh` to recognize the `azurelinux` identifier.
- `scripts/install_setup_openssl_1_1.sh` and `/.pipelines/FlexContainers/Deployment-AzureLinx-3.0/Dockerfile`: Added new files for installing OpenSSL 1.1.1 from source and for deploying PgMongo artifacts using AZL3.0.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4330937
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
[Infra] adding support for documentdb_distributed extension in start_oss_server
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for the `documentdb_distributed` extension in the `start_oss_server` script.
- `start_oss_server.sh`: Added a new option `-x` to start the server with the `documentdb_distributed` extension and updated the script to handle this new option.
- `utils.sh`: Added a new function `AddNodeToCluster` to add nodes to the cluster when the `documentdb_distributed` extension is used.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
[OSS] Fix TTL indexes in documentdb
This moves the function to C and instead of querying the indexes and then getting the collection shards from pg_dist_shard table, it uses the C hook to get the table shards where for non-distributed it returns the base table and then uses that information to drop the expired records.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request addresses issues with TTL indexes in DocumentDB by refactoring and optimizing the deletion of expired rows.
- `ttl_index.c`: Refactored the function to delete expired rows, added a new function `delete_expired_rows`, and optimized the deletion process.
- `ttl_support_functions--latest.sql` and `delete_expired_rows--latest.sql`: Replaced PL/pgSQL procedures with C language implementations for better performance.
- `collection.c`: Moved the function `GetMongoCollectionShardOidsAndNames` from `coll_stats.c` to `collection.c` for better organization.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Fixed OSS issue: https://github.com/microsoft/documentdb/issues/40
`start_oss_server script -c` option is broken.
Once the `search_path` is set in ~/.psqlrc that is used for creating the extensions until unless an override is provided the install location for dependent extension started deviating from `public`
```SQL
SELECT e.extname AS extension_name,
n.nspname AS schema_name
FROM pg_extension e
JOIN pg_namespace n ON e.extnamespace = n.oid
WHERE e.extname IN ('documentdb', 'documentdb_core', 'pg_cron', 'tsm_system_rows', 'vector', 'postgis', 'rum');
extension_name | schema_name
-----------------+-----------------
documentdb_core | public
pg_cron | pg_catalog
tsm_system_rows | documentdb_core
vector | documentdb_core
postgis | documentdb_core
rum | documentdb_core
documentdb | public
(7 rows)
```
Always `CREATE EXTENSION` without psqlrc shell overrides for deterministic behavior.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request fixes the `-c` option in the `scripts/start_oss_server` script which was causing issues.
- `oss/scripts/utils.sh`: Added `SCHEMA public` to the `CREATE EXTENSION` command to ensure proper schema usage.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
In oss, we noticed frequently sql test result difference introduced by newer package minor versions (postgres, pgvector, postgis, etc).
To make oss CI integration stable and to make test result consistent in oss and pgmongo, we should specify fixed versions for packages in oss, and align with pgmongo.
Will depend on manual package version update after verifying the compatibility and meanwhile update test results.
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup and enhancement.
#### PR Summary
This pull request declares and uses fixed versions of various packages to ensure consistency and reliability in the build environment.
- Updated `/oss/.devcontainer/Dockerfile` to declare and use fixed versions for multiple dependencies including PostgreSQL, Citus, libbson, PG_CRON, PG_VECTOR, Intel Decimal Math Library, PCRE2, and PostGIS.
- Modified `/.devcontainer/Dockerfile` to align with the fixed versioning approach and added necessary environment variables.
- Added `GetPostgresSourceRef` function in `oss/scripts/utils.sh` to map PostgreSQL versions to specific source references.
- Updated `oss/scripts/install_setup_intel_decimal_math_lib.sh` and `oss/scripts/install_setup_pcre2.sh` to accept version parameters.
- Moved several scripts from `/scripts` to `/oss/scripts` for better organization.
Related work items: #3571853
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Remove helio from file name
Remove helio references from source content except schema reference
Start local documentdb server with the documentdb extension
Regression tests using the documentdb extension
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request removes references to "helio" from the `documentdb_core` source code.
- `scripts/utils.sh`: Renamed variables from `preloadExtensions` to `preloadLibraries`.
- `pg_documentdb_core/include/utils/error_utils.h`: Updated function and macro names to replace "Helio" with "DocumentDB".
- `pg_documentdb_core/include/io/helio_bson_core.h` and `pg_documentdb_core/include/query/helio_bson_compare.h`: Moved to `pg_documentdb_core/include/io/bson_core.h` and `pg_documentdb_core/include/query/bson_compare.h`.
- `pg_documentdb_core/src/collation/collation.c` and `pg_documentdb_core/src/io/pgbson.c`: Replaced error codes and messages to use "DocumentDB" instead of "Helio".
- `scripts/validate_helio_extension_state.sh`: Added exceptions for certain files to skip validation for "helio" references.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Add [-t] - optional argument. serverType is the type of server to start.
Supported values are helio and documentdb, default is helio"
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds an option for starting a DocumentDB server and includes some script adjustments.
- Moved `oss/scripts/start_helio_server.sh` to `oss/scripts/start_oss_server.sh`.
- Updated `oss/scripts/utils.sh` to support additional preload extensions for PostgreSQL configurations.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Chmod +x all scripts
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
TSG
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
This change introduces a section in the SOP that includes a script enabling simultaneous updates across multiple accounts under the same subscription.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
We defaulted to PG16, but some parts were still left as 15.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
building the project under debian 12 failed with dangling pointer warning. and error existing memory usage
optimization was using compiler dependent feature that should be removed.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
DevOps
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Fix pg_ctl path for heliodb start/stop operations
balinorLiang
Lucas Borges Fernandes
Manish Kumar
Guanzhou Song
yangqiaoandroot
Parag Jain
Santiago Fernandez Madero
Deepak Bisht
Shuai Tian
Zinan Ren
Aditya Kotalwar
Murat Tuncer
Vinod Sridharan