mirror of
https://github.com/FerretDB/documentdb.git
synced 2026-11-07 03:22:46 -05:00
main
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Siwen Tang
|
be11798bef |
Merged PR 1778507: [Refector] Update to match coding standards - part 5
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description draft update error msg ---- #### AI description (iteration 1) #### PR Classification This is a code cleanup and bug‑fix pull request that standardizes and clarifies error messages and log details across multiple modules. #### PR Summary The changes improve the consistency, clarity, and accuracy of user‑facing messages and logging throughout the code base. Key changes include: - In files under directories such as `oss/internal/pg_documentdb_distributed/src/test/regress/expected/` and `pgmongo/src/test/docdb_compat/expected/`, error messages for geospatial query operators (e.g. geoNear, $center, $within) and aggregation stage operators have been rewritten for clarity (e.g. replacing vague “Too many geoNear expressions” with “Excessive number of geoNear query expressions” and clarifying numeric parameter validations). - Several operators’ messages (e.g. `$cond`, `$not`, `$abs`, `$floor`, `$divide`, `$log`, etc.) have been updated to state exactly how many arguments are required, using <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Deepak Bisht
|
b1471da24d |
Merged PR 1757273: Fix $lookup index pushdown
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) BugFix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes `documentdb.forceBitmapScanForLookup` now controls if we need force bitmap index scan paths for lookup JOIN queries ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) - $lookup JOIN condition pushed to index more reliably and avoids bitmap heap scan conversion. ### Description Fix index pushdown for $lookup with more data. Problem description: 1- Today when the lookup index scan paths are converted to bitmap heap scan path and while doing so we loose the parameterization info already build by PG planner, and the converted path considers all rows to be returned, this impacts index selectivity as the path becomes expensive. Solution: - Fixed the bitmap heap scan path by adjusting the estimated rows returned from the parameterized info. - Skip force conversion of index path to bitmap heap scan path for $lookup JOIN cases behind a GUC which should be default behavior going forward ---- #### AI description (iteration 1) #### PR Classification Bug fix addressing incorrect $lookup index pushdown and lookup join behavior. #### PR Summary This PR refines the planner and execution paths for $lookup queries by improving bitmap scan conversion, selectivity estimation, and associated configuration while updating regression tests to validate the new behavior. - `pgmongo/src/test/docdb_compat/expected/bson_aggregation_tests_lookup_inner_join.out` and `oss/internal/pg_documentdb_distributed/src/test/regress/sql/bson_aggregation_tests_lookup_inner_join.sql`: Updated test cases and expected outputs to exercise various lookup join scenarios and index usage modes. - `oss/pg_documentdb/src/planner/documents_planner.c`: Introduced the helper `IsBitmapHeapConversionSupported` to centralize conditions for pushing down bitmap heap conversion, replacing the older merge query check. - `oss/pg_documentdb/src/customscan/custom_scan_continuation.c`: Improved handling of parameterization in index pushdown paths for lookup queries to restore accurate row estimates. - `oss/pg_documentdb/src/opclass/index_support.c` and `oss/pg_documentdb/src/query/bson_dollar_selectivity.c`: Modified lookup filter support functions to enforce low selectivity for lookup-modified index quals. - `oss/pg_documentdb/src/configs/feature_flag_configs.c` and `oss/pg_documentdb/include/query/bson_dollar_selectivity.h`: Added and centralized the `forceBitmapScanForLookup` flag along with selectivity constants for consistent behavior. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4522655 |
||
 Vinod Sridharan
|
528b744a4f |
Merged PR 1717610: [Selectivity] Make Function Selectivity Match Opr Selectivity for Index
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description For Operator Selectivity, we have a feature flag and function that does better operator selecitivty for operator expressions. However, when doing planning, the query operator generates mostly FuncExprs for index operators. This means the selectivity logic for funcitons defaults to 0.3333 even if opr selectivity can do better. For Functions, when opr selectivity is enabled, use the same logic to determine selectivity enabling better index pushdowns. ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature to align function selectivity with operator selectivity for index operations in the DocumentDB Postgres extension. #### PR Summary The PR refactors selectivity calculations to ensure consistency between function and operator estimations, updates associated test cases, and introduces new configuration flags. Key changes include: - **`pg_documentdb/src/opclass/index_support.c`**: Revised the `dollar_support` function to route selectivity requests correctly using a new pointer variable and support for selectivity mode. - **`pg_documentdb/src/query/bson_dollar_selectivity.c` and `include/query/bson_dollar_selectivity.h`**: Updated the selectivity computation routines to accept a default selectivity parameter and adjust calculations based on the new operator selectivity mode. - **`configs/feature_flag_configs.c` and `system_configs.c`**: Added new configuration flags (`DisableDollarSupportFuncSelectivity` and `EnableNewOperatorSelectivityMode`) and adjusted default values to control the new selectivity behavior. - **Test files in `/oss/internal/pg_documentdb_distributed/src/test/regress/` and `/pgmongo/src/test/docdb_compat/expected/`**: Extended and modified SQL regression tests to verify expected index query plans and selectivity estimates. - **Configuration scripts (`/scripts/utils.sh` and related test configs)**: Made minor adjustments such as reducing the index build schedule interval to better support the new functionality. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Cornelius Boateng
|
b204b58b82 |
Merged PR 1619783: [engine][collation] Support for sort in runtime
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes, yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Support sorting with collation. We define new custom sort operators which compare operands using applicable collation string. `<<<` is for ascending order. `>>>` is for descending order. Example: `SELECT * from <collection> ORDER BY <value> USING <<<` This will further be used for partitioning, grouping, and related comparisons. ---- #### AI description (iteration 1) #### PR Classification New feature: Support for collation in sort operations. #### PR Summary This pull request adds support for collation in sort operations within the database engine. - `src/test/regress/expected/bson_dollar_ops_collation_tests_runtime.out`: Added multiple test cases to verify collation behavior with various sort and order by operations, including case sensitivity and numeric ordering. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4014123 |
||
|
Vinod Sridharan
|
8261e772f4 |
Merged PR 1672946: [Update] Make update parsing cheaper
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Perf ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Currently for Update/Delete, we currently parse the update spec, query spec, etc into pgbsons on startup. Then reuse it per commands. This has a few issues - for large document requests, this causes us to duplicate memory and persist it for the duration of the update and also transcode all up front - this is worse for multi-node when all we need to do is build yet another update spec and push it to the worker. Instead, we now only cache pointers to the spec document; then when we need it we convert to pgbson only if necessary. A consequence of this is that on a single node update, if we're doing a simple updateOne on _id then we would never serialize the updateSpec at all (for instance). We also more eagerly free memory on the intermediate update specs. ---- #### AI description (iteration 1) #### PR Classification Performance optimization for update parsing by refactoring BSON value handling. #### PR Summary This PR streamlines update parsing by replacing heavy BSON initialization routines with lightweight wrappers and converting update-related types from `pgbson*` to `bson_value_t*`, which reduces overhead and improves coordinator/worker compatibility. - `src/commands/update.c`: Replaced multiple `PgbsonInitFromDocumentBsonValue` and similar calls with efficient wrappers like `CreateBsonValueFrom` and updated serialization (including back-compatible array filter and update document formats). - `include/commands/update.h`: Updated update structure fields to use `bson_value_t*` instead of `pgbson*`, aligning interfaces with the new parsing approach. - `src/update/bson_update.c` and `bson_update_operators_workflow.c`: Refactored update type determination and replace document logic to use new BSON value iterators and helper functions. - Other modules (e.g. `commands/find_and_modify.c`, `commands/delete.c`, query operators, sharding) were updated to leverage the lighter-weight BSON representation and its associated helper routines. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Cornelius Boateng
|
f6d20fbf8a |
Merged PR 1646920: [engine][collation] Support for Arrays and Documents
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Support collation with comparison of `array` and `document` values. Also adds support for `$in` on nested arrays and documents. ---- #### AI description (iteration 1) #### PR Classification New feature: Adds support for collation-aware hashing of arrays and documents in BSON. #### PR Summary This pull request introduces collation-aware hashing for BSON arrays and documents, enhancing the handling of collation in hash sets. It refactors existing hash functions to accommodate collation strings and recursive hashing of nested BSON structures. - `hashset_utils.c`: Implements collation-aware hashing functions for BSON values, including recursive handling of arrays and documents. - `bson_expression_set_operators.c`: Replaces existing hash set creation with collation-aware hash set creation. - `bson_dollar_operators.c`: Updates logic to use collation-aware hash sets for `$in` queries, removing unsupported scenarios. - `hashset_utils.h`: Adds a new struct `BsonValueHashEntry` to store BSON values with collation strings. - `collation.h`: Moves `IsBsonTypeCollationAware` function to handle collation applicability checks. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4111861, #4115239 |
||
|
Cornelius Boateng
|
4da81c67e6 |
Merged PR 1588632: [engine][collation] Support for Aggregation, Find, Query Match on Sharded Collections
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Support collation for aggregation and query match queries on sharded collections. If the `shard key filter` is collation-aware, we make the query distributed and push it to all shards. ---- #### AI description (iteration 1) #### PR Classification New feature: Collation support for aggregation and query match on sharded collections. #### PR Summary This pull request adds collation support for aggregation and query match operations on sharded collections. - `bson_dollar_ops_collation_tests_runtime.sql`: Added tests for collation-aware shard key queries. - `sharding.c`: Introduced logic to track collation-aware shard key fields and updated related functions. - `query_operator.c`: Modified to include shard key filters only if they are not collation-sensitive. - `bson_aggregation_pipeline.c`: Added collation-aware checks for shard key filters in aggregation pipelines. - `bson_query.h`: Added utility function `IsBsonTypeCollationAware` to check if BSON types are collation-aware. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #3915171, #3922023 |
||
 Jia Hu
|
55f6fa3840 |
Merged PR 1485956: [schema validation] support schema rule included top level operator
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md)) ### Description To enable top level query operators working in schema validation. ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request introduces support for the `$jsonSchema` query operator for schema validation. - Added handling for `QUERY_OPERATOR_JSONSCHEMA` in `query_operator.c` and `mongo_query_operator.c`. - Implemented `GetExpressionEvalStateForBsonInput` in `bson_expr_eval.c`. - Added `BsonJsonSchemaFunctionId` and related OID cache in `metadata_cache.c`. - Modified `PrepareForSchemaValidation` in `schema_validation.c` to use the new evaluation state function. |
||
|
Vinod Sridharan
|
0cc46eca57 |
Merged PR 1542496: [Perf] Custom planner for point read
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Perf ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md)) ### Description Improve performance for a point read set up for unsharded collections. In the scenario where we have a point read in a local unsharded collection with colocation, we can bypass a lot of the planner and go directly to the PK index. Use this to simplify the planner. With this change, point reads on YCSB are able to hit ~16k on Marlin and ~17.5-18k on flex ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary Introduces a custom planner for point read operations to optimize query performance. - Added `documents_custom_planner.c` to implement the custom point read planner. - Updated `aggregation_commands.c` to handle the new point read cursor type. - Modified `planner.c` to integrate the custom point read planner and expand aggregation functions. |
||
 Zinan Ren
|
0ad54c20b8 |
Merged PR 1540839: [OSS] Remove helio references from sources of documentdb and documentdb_distributed
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactoring ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md)) ### Description Remove helio references from documentdb Remove helio references from documentdb_distributed ---- #### AI description (iteration 1) #### PR Classification Code cleanup #### PR Summary This pull request removes references to "helio" and replaces them with "documentdb" in the sources of `documentdb` and `documentdb_distributed`. - `metadata_cache.h` and `metadata_cache.c`: Replaced "Helio" with "DocumentDB" in variable names and function names. - `pgmongo.c`: Updated include statements and function calls to use "documentdb" instead of "helio". - `planner.c`: Updated planner function and related calls to use "DocumentDBApiPlanner". - Moved several header files from `helio_*` to `documentdb_*` naming convention without code changes. - `commands_common.c` and `create_indexes.c`: Updated function calls and variable names to use "DocumentDB" instead of "Helio". |
||
|
Zinan Ren
|
8e7bd960bf |
Merged PR 1540262: [OSS] Remove helio references from sources of documentdb_core
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactoring ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md)) ### Description Remove helio from file name Remove helio references from source content except schema reference Start local documentdb server with the documentdb extension Regression tests using the documentdb extension ---- #### AI description (iteration 1) #### PR Classification Code cleanup #### PR Summary This pull request removes references to "helio" from the `documentdb_core` source code. - `scripts/utils.sh`: Renamed variables from `preloadExtensions` to `preloadLibraries`. - `pg_documentdb_core/include/utils/error_utils.h`: Updated function and macro names to replace "Helio" with "DocumentDB". - `pg_documentdb_core/include/io/helio_bson_core.h` and `pg_documentdb_core/include/query/helio_bson_compare.h`: Moved to `pg_documentdb_core/include/io/bson_core.h` and `pg_documentdb_core/include/query/bson_compare.h`. - `pg_documentdb_core/src/collation/collation.c` and `pg_documentdb_core/src/io/pgbson.c`: Replaced error codes and messages to use "DocumentDB" instead of "Helio". - `scripts/validate_helio_extension_state.sh`: Added exceptions for certain files to skip validation for "helio" references. |
||
|
Vinod Sridharan
|
3f9a507562 |
Merged PR 1539950: [OSS] Move helio_api and helio_distributed sources under documentdb
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactor ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md)) ### Description Move source files under new directory. ---- #### AI description (iteration 1) #### PR Classification Code reorganization #### PR Summary This pull request moves the `helio_api` and `helio_distributed` sources under the `documentdb` directory, updating references and initialization logic accordingly. - Added `pg_documentdb.c` for initializing the `pg_documentdb` extension. - Added `documentdb_distributed.c` for initializing the `pg_documentdb_distributed` extension. - Updated `validate_helio_extension_state.sh` to include new checks for `documentdb`. - Moved `helio_api_extension_version.c` to `documentdb_extension_version.c` without code changes. - Updated various `Makefile` configurations to reflect the new directory structure and dependencies. |