### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
For Operator Selectivity, we have a feature flag and function that does better operator selecitivty for operator expressions. However, when doing planning, the query operator generates mostly FuncExprs for index operators. This means the selectivity logic for funcitons defaults to 0.3333 even if opr selectivity can do better.
For Functions, when opr selectivity is enabled, use the same logic to determine selectivity enabling better index pushdowns.
----
#### AI description (iteration 1)
#### PR Classification
This pull request implements a new feature to align function selectivity with operator selectivity for index operations in the DocumentDB Postgres extension.
#### PR Summary
The PR refactors selectivity calculations to ensure consistency between function and operator estimations, updates associated test cases, and introduces new configuration flags. Key changes include:
- **`pg_documentdb/src/opclass/index_support.c`**: Revised the `dollar_support` function to route selectivity requests correctly using a new pointer variable and support for selectivity mode.
- **`pg_documentdb/src/query/bson_dollar_selectivity.c` and `include/query/bson_dollar_selectivity.h`**: Updated the selectivity computation routines to accept a default selectivity parameter and adjust calculations based on the new operator selectivity mode.
- **`configs/feature_flag_configs.c` and `system_configs.c`**: Added new configuration flags (`DisableDollarSupportFuncSelectivity` and `EnableNewOperatorSelectivityMode`) and adjusted default values to control the new selectivity behavior.
- **Test files in `/oss/internal/pg_documentdb_distributed/src/test/regress/` and `/pgmongo/src/test/docdb_compat/expected/`**: Extended and modified SQL regression tests to verify expected index query plans and selectivity estimates.
- **Configuration scripts (`/scripts/utils.sh` and related test configs)**: Made minor adjustments such as reducing the index build schedule interval to better support the new functionality.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes, yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support sorting with collation.
We define new custom sort operators which compare operands using applicable collation string.
`<<<` is for ascending order.
`>>>` is for descending order.
Example:
`SELECT * from <collection> ORDER BY <value> USING <<<`
This will further be used for partitioning, grouping, and related comparisons.
----
#### AI description (iteration 1)
#### PR Classification
New feature: Support for collation in sort operations.
#### PR Summary
This pull request adds support for collation in sort operations within the database engine.
- `src/test/regress/expected/bson_dollar_ops_collation_tests_runtime.out`: Added multiple test cases to verify collation behavior with various sort and order by operations, including case sensitivity and numeric ordering.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4014123
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Currently for Update/Delete, we currently parse the update spec, query spec, etc into pgbsons on startup. Then reuse it per commands.
This has a few issues - for large document requests, this causes us to duplicate memory and persist it for the duration of the update and also transcode all up front - this is worse for multi-node when all we need to do is build yet another update spec and push it to the worker.
Instead, we now only cache pointers to the spec document; then when we need it we convert to pgbson only if necessary.
A consequence of this is that on a single node update, if we're doing a simple updateOne on _id then we would never serialize the updateSpec at all (for instance).
We also more eagerly free memory on the intermediate update specs.
----
#### AI description (iteration 1)
#### PR Classification
Performance optimization for update parsing by refactoring BSON value handling.
#### PR Summary
This PR streamlines update parsing by replacing heavy BSON initialization routines with lightweight wrappers and converting update-related types from `pgbson*` to `bson_value_t*`, which reduces overhead and improves coordinator/worker compatibility.
- `src/commands/update.c`: Replaced multiple `PgbsonInitFromDocumentBsonValue` and similar calls with efficient wrappers like `CreateBsonValueFrom` and updated serialization (including back-compatible array filter and update document formats).
- `include/commands/update.h`: Updated update structure fields to use `bson_value_t*` instead of `pgbson*`, aligning interfaces with the new parsing approach.
- `src/update/bson_update.c` and `bson_update_operators_workflow.c`: Refactored update type determination and replace document logic to use new BSON value iterators and helper functions.
- Other modules (e.g. `commands/find_and_modify.c`, `commands/delete.c`, query operators, sharding) were updated to leverage the lighter-weight BSON representation and its associated helper routines.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support collation with comparison of `array` and `document` values.
Also adds support for `$in` on nested arrays and documents.
----
#### AI description (iteration 1)
#### PR Classification
New feature: Adds support for collation-aware hashing of arrays and documents in BSON.
#### PR Summary
This pull request introduces collation-aware hashing for BSON arrays and documents, enhancing the handling of collation in hash sets. It refactors existing hash functions to accommodate collation strings and recursive hashing of nested BSON structures.
- `hashset_utils.c`: Implements collation-aware hashing functions for BSON values, including recursive handling of arrays and documents.
- `bson_expression_set_operators.c`: Replaces existing hash set creation with collation-aware hash set creation.
- `bson_dollar_operators.c`: Updates logic to use collation-aware hash sets for `$in` queries, removing unsupported scenarios.
- `hashset_utils.h`: Adds a new struct `BsonValueHashEntry` to store BSON values with collation strings.
- `collation.h`: Moves `IsBsonTypeCollationAware` function to handle collation applicability checks.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #4111861, #4115239
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support collation for aggregation and query match queries on sharded collections.
If the `shard key filter` is collation-aware, we make the query distributed and push it to all shards.
----
#### AI description (iteration 1)
#### PR Classification
New feature: Collation support for aggregation and query match on sharded collections.
#### PR Summary
This pull request adds collation support for aggregation and query match operations on sharded collections.
- `bson_dollar_ops_collation_tests_runtime.sql`: Added tests for collation-aware shard key queries.
- `sharding.c`: Introduced logic to track collation-aware shard key fields and updated related functions.
- `query_operator.c`: Modified to include shard key filters only if they are not collation-sensitive.
- `bson_aggregation_pipeline.c`: Added collation-aware checks for shard key filters in aggregation pipelines.
- `bson_query.h`: Added utility function `IsBsonTypeCollationAware` to check if BSON types are collation-aware.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #3915171, #3922023
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
To enable top level query operators working in schema validation.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces support for the `$jsonSchema` query operator for schema validation.
- Added handling for `QUERY_OPERATOR_JSONSCHEMA` in `query_operator.c` and `mongo_query_operator.c`.
- Implemented `GetExpressionEvalStateForBsonInput` in `bson_expr_eval.c`.
- Added `BsonJsonSchemaFunctionId` and related OID cache in `metadata_cache.c`.
- Modified `PrepareForSchemaValidation` in `schema_validation.c` to use the new evaluation state function.
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Improve performance for a point read set up for unsharded collections.
In the scenario where we have a point read in a local unsharded collection with colocation, we can bypass a lot of the planner and go directly to the PK index. Use this to simplify the planner.
With this change, point reads on YCSB are able to hit ~16k on Marlin and ~17.5-18k on flex
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
Introduces a custom planner for point read operations to optimize query performance.
- Added `documents_custom_planner.c` to implement the custom point read planner.
- Updated `aggregation_commands.c` to handle the new point read cursor type.
- Modified `planner.c` to integrate the custom point read planner and expand aggregation functions.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Remove helio references from documentdb
Remove helio references from documentdb_distributed
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request removes references to "helio" and replaces them with "documentdb" in the sources of `documentdb` and `documentdb_distributed`.
- `metadata_cache.h` and `metadata_cache.c`: Replaced "Helio" with "DocumentDB" in variable names and function names.
- `pgmongo.c`: Updated include statements and function calls to use "documentdb" instead of "helio".
- `planner.c`: Updated planner function and related calls to use "DocumentDBApiPlanner".
- Moved several header files from `helio_*` to `documentdb_*` naming convention without code changes.
- `commands_common.c` and `create_indexes.c`: Updated function calls and variable names to use "DocumentDB" instead of "Helio".
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Remove helio from file name
Remove helio references from source content except schema reference
Start local documentdb server with the documentdb extension
Regression tests using the documentdb extension
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup
#### PR Summary
This pull request removes references to "helio" from the `documentdb_core` source code.
- `scripts/utils.sh`: Renamed variables from `preloadExtensions` to `preloadLibraries`.
- `pg_documentdb_core/include/utils/error_utils.h`: Updated function and macro names to replace "Helio" with "DocumentDB".
- `pg_documentdb_core/include/io/helio_bson_core.h` and `pg_documentdb_core/include/query/helio_bson_compare.h`: Moved to `pg_documentdb_core/include/io/bson_core.h` and `pg_documentdb_core/include/query/bson_compare.h`.
- `pg_documentdb_core/src/collation/collation.c` and `pg_documentdb_core/src/io/pgbson.c`: Replaced error codes and messages to use "DocumentDB" instead of "Helio".
- `scripts/validate_helio_extension_state.sh`: Added exceptions for certain files to skip validation for "helio" references.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Move source files under new directory.
----
#### AI description (iteration 1)
#### PR Classification
Code reorganization
#### PR Summary
This pull request moves the `helio_api` and `helio_distributed` sources under the `documentdb` directory, updating references and initialization logic accordingly.
- Added `pg_documentdb.c` for initializing the `pg_documentdb` extension.
- Added `documentdb_distributed.c` for initializing the `pg_documentdb_distributed` extension.
- Updated `validate_helio_extension_state.sh` to include new checks for `documentdb`.
- Moved `helio_api_extension_version.c` to `documentdb_extension_version.c` without code changes.
- Updated various `Makefile` configurations to reflect the new directory structure and dependencies.
Vinod Sridharan
Cornelius Boateng
Jia Hu
Zinan Ren