### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Make DocumentDB not explicitly depend on the RUM extension (it's done implicitly) on the binary.
Instead pull in the SQL dependency into documentdb and reference the C functions internally. This allows us to swap between rum.so and extended_rum.
For Pg18 (where there isn't yet official RUM support) default to using extended_rum for the .so file and remove all dependencies on the "rum.so"
----
#### AI description (iteration 2)
#### PR Classification
This PR is an API and dependency update that removes the explicit dependency on the external RUM extension by integrating its functionality internally.
#### PR Summary
The update integrates the RUM text-search capabilities directly into DocumentDB and cleans up the external linkage.
- `oss/internal/pg_documentdb_extended_rum/src/rum_ts_utils.c` is newly added with core text-search functions.
- `oss/pg_documentdb/src/index_am/rum.c` is modified to load internal wrappers that call the new RUM functions.
- SQL UDF adapter files and operator class definitions are updated to reference the internal functions instead of public RUM functions.
- Build scripts, metadata, and control files are adjusted to remove the explicit dependency on the external RUM extension.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support order and composite extensibility for documentdb_rum via index am extensibility in documentdb.
----
#### AI description (iteration 1)
#### PR Classification
This pull request introduces a new feature by adding support for documentdb_rum via index extensibility.
#### PR Summary
The changes integrate a new index access method for documentdb_rum, updating core indexing interfaces, schema definitions, and build/test scripts to enable extended RUM indexing functionality. Key updates include:
- **`internal/pg_documentdb_rum/src/documentdb_rum_adapter.c`**: Introduces the new `documentdb_extended_rum` handler with functions for scan, build, insert, and others.
- **`include/index_am/index_am_exports.h` and related utility files**: Add and update APIs for index extensibility (including multikey status support) to accommodate the new handler.
- **`pg_documentdb/src/index_am/create_indexes.c`**: Modifies index creation logic to use new opclass schemas and integrate the extended rum index handler.
- **SQL schema files (e.g., `documentdb_rum--1.0.sql` and operator class scripts)**: Define the new extension schema, operator classes, and functions required for extended RUM.
- **Build scripts and regression test files**: Update Makefiles and expected test outputs to load, build, and validate the documentdb_rum extension functionality.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No.
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature.
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No.
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes.
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Add OSS background_worker initial jobs framework. For this initial version, only schedulable jobs are allowed (i.e. jobs that run on timed intervals). The background worker loop has a latch timeout with a predefined duration of 10 seconds that is configured through a GUC. Every time this timeout occurs, it iterates on the JobRegistry array.
1. Checks if the job is running and if it timed out (i.e. running longer than timeoutInSeconds). If yes, cancels the query and finishes libpq connection.
2. Checks if the job finished running (i.e. libpq connection isn't busy). If yes, finishes the libpq connection.
3. Runs the job if its scheduled interval has reached (i.e. currentTime - lastStartTime >= scheduledInterval). Each job execution has its own libpq connection to the database.
In order to add a new job callers must pass a BackgroundWorkerJob structure to RegisterBackgroundWorkerJob.
----
#### AI description (iteration 1)
#### PR Classification
This pull request introduces a new feature by adding an OSS background worker jobs framework.
#### PR Summary
The changes implement a framework for scheduling, executing, and managing background jobs within the background worker process. This provides structured job processing with timeout, cancellation, and configuration support.
- **`src/background_worker/background_worker.c`**: Added job management functions (e.g., ManageJobs, ExecuteJob, CanExecuteJob, CheckJobCompletion, CancelJobIfTimeIsUp, ValidateJob) and integrated job scheduling in the main loop.
- **`include/background_worker/background_worker_job.h`**: Introduced new structures for defining background worker jobs and their execution objects.
- **`src/api_hook_impl.c`**: Added a hook function to retrieve background worker jobs for registration.
- **`src/configs/background_job_configs.c`**: Defined new configuration variables (GUCs) to enable background worker jobs.
- **API Hooks and Connection Management Files**: Updated declarations and functions (e.g., in `include/api_hooks_def.h` and connection management files) to support connection handling necessary for job execution.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4225917
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Comments
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
All the previous version SQL changes should be merged into this PR so that we can update the hashes in this PR
----
#### AI description (iteration 1)
#### PR Classification
This PR is a documentation update that cleans up inline SQL comments to align with the latest contributing guidelines.
#### PR Summary
The changes refine the SQL file comments for geospatial operator classes and functions by removing outdated references to Mongo indexing and validations.
- `oss/pg_documentdb/sql/operators/bson_gist_geospatial_op_classes--0.16-0.sql`: Updated comments by replacing "mongo 2d index options" and "mongo 2dsphjere index options" with simplified descriptions.
- `oss/pg_documentdb/sql/udfs/geospatial/bson_gist_extensibility_functions--0.16-0.sql` and `oss/pg_documentdb/sql/udfs/geospatial/bson_gist_extensibility_functions--latest.sql`: Revised function comments to remove "mongo db validations" for a more general description of geometry validations.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4354111
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes, yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support sorting with collation.
We define new custom sort operators which compare operands using applicable collation string.
`<<<` is for ascending order.
`>>>` is for descending order.
Example:
`SELECT * from <collection> ORDER BY <value> USING <<<`
This will further be used for partitioning, grouping, and related comparisons.
----
#### AI description (iteration 1)
#### PR Classification
New feature: Support for collation in sort operations.
#### PR Summary
This pull request adds support for collation in sort operations within the database engine.
- `src/test/regress/expected/bson_dollar_ops_collation_tests_runtime.out`: Added multiple test cases to verify collation behavior with various sort and order by operations, including case sensitivity and numeric ordering.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4014123
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Fix up the remaining operators - implementations are sub-optimal
2) Add type bracketing options for all types and use them in operators
3) Add support for truncation of terms
TODO:
- Support recheck on the index for operators than just bounds
- Support N boundaries for index terms.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request addresses issues with operators and type bracketing in BSON composite indexes.
- Added `bson_gin_index_types_core.c` to manage BSON type bounds for indexes.
- Updated `bson_gin_composite_entrypoint.c` to fix operator handling and apply type bracketing for various BSON index strategies.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
As part of testing out the composite operator classes, introduce a new composite operator class for Bson. This will be behind a test only GUC and will be for testing composite index pushdown.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces a new composite operator class for BSON GIN indexes in PostgreSQL.
- Added `bson_gin_composite_entrypoint.c` to implement GIN operator functions for BSON composite indexes.
- Created test cases in `bson_composite_index_tests_insert_terms.out` to validate term generation for composite paths.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Move pre_consistent back to helio and remove from documentdb schema
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request moves the `pre_consistent` function back to the `helio` schema and removes it from the `documentdb` schema.
- `pgmongo/sql/pgmongo--1.4-0--1.4-1.sql`: Updated includes to reference `pg_helio_api` instead of `pg_documentdb`.
- `oss/pg_documentdb/sql/documentdb--0.24-0.sql`: Removed includes for `index_operator_classes_preconsistent` files.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Porting udfs into documentdb
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup and feature enhancement.
#### PR Summary
This pull request continues the porting of UDFs into DocumentDB, updating and including various SQL files for better functionality and organization.
- Updated `documentdb--0.24-0.sql` to include new UDFs and operators, replacing old function references.
- Added new UDFs for query, projection, geospatial, aggregation, metadata, schema management, index management, CRUD commands, user management, schema validation, telemetry, TTL, vector search, extension version, diagnostic, and RBAC metadata.
- Removed outdated function definitions and replaced them with updated SQL includes.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Start porting schemas for documentdb api.
Introducing API_SCHEMA_NAME_V2, API_SCHEMA_INTERNAL_NAME_V2, API_CATALOG_SCHEMA_NAME_V2
----
#### AI description (iteration 1)
#### PR Classification
Porting type/operator/schema into documentdb.
#### PR Summary
This pull request ports various types, operators, and schemas into documentdb as part of an ongoing effort.
- `pg_documentdb/sql/documentdb--0.24-0.sql`: Created multiple schemas and included various SQL files for query functions, operators, and metadata.
- `pg_documentdb/sql/operators/bson_dollar_operators--0.16-0.sql`: Updated operator definitions to use `__API_SCHEMA_INTERNAL_V2__`.
- `pgmongo/sql/operators/bson_dollar_negation_operators--1.15-1.sql` and `pg_documentdb/sql/operators/bson_dollar_negation_operators--0.16-0.sql`: Changed operator definitions to use `__API_SCHEMA_INTERNAL_V2__`.
- `Makefile.cflags`: Added checks for new schema variables `API_SCHEMA_NAME_V2`, `API_SCHEMA_INTERNAL_NAME_V2`, and `API_CATALOG_SCHEMA_NAME_V2`.
- `pg_documentdb/sql/schema/unique_shard_path_operator_class--0.24-0.sql`: Updated operator class definitions to use `__API_SCHEMA_INTERNAL_V2__`.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Move type/operator/schema files from pg_helio_api -> pg_documentdb
Except udf files
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request moves type, operator, and schema files from the `pg_helio_api` directory to the `pg_documentdb` directory.
- Updated SQL include paths in multiple `pg_helio_api` SQL files to reference `pg_documentdb` directory.
- Modified `pgmongo` SQL files to include paths from `pg_documentdb` instead of `pg_helio_api`.
- Adjusted `validate_helio_extension_state.sh` script to account for new file paths.
- Updated `Makefile` in `pg_helio_api` to include the new directory structure.
- Changed include paths in `pg_documentdb_distributed` and `pg_helio_distributed` C source files to reflect the new directory structure.
Vinod Sridharan
Lucas Borges Fernandes
Deepak Bisht
Cornelius Boateng
Zinan Ren