mirror of
https://github.com/FerretDB/documentdb.git
synced 2026-11-06 19:12:50 -05:00
main
184
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Vinod Sridharan
|
84b999ae88 |
Merged PR 1815453: [Composite] Support optimized elemMatch pushdown
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Improve pushdown of $elemMatch for composite indexes.
When there's an elemMatch of the form `{ "a": { "$elemMatch": { "$gt": 10, "$lt": 15 }}`
This can scan the index only from 10 -> 15
However given the multi-key index requirement we were doing 2 scans:
- One from 10 to infinity and the other from -infinity to 15.
Optimize this path for $elemMatch.
----
#### AI description (iteration 1)
#### PR Classification
This PR introduces a new feature that optimizes the pushdown of `$elemMatch` queries for composite indexes.
#### PR Summary
The pull request implements optimized handling of `$elemMatch` predicates on composite indexes by converting them into specialized index pushdown expressions and updating the corresponding test expectations.
- **`oss/pg_documentdb/src/opclass/bson_gin_composite_core.c` & `index_support.c`**: Added logic to detect `$elemMatch` in range queries and convert them into optimized pushdown expressions (via functions like `GetElemMatchIndexPushdownOperator`).
- **Configuration Changes**: Introduced a new GUC (`documentdb.useNewElemMatchIndexOperatorOnPushdown`) in `pg_documentdb/src/configs/feature_flag_configs.c` to control the new behavior.
- **Test Updates**: Modified expected explain outputs in both `/oss/internal/pg_documentdb_distributed/src/test/regress/` and `/pgmongo/src/test/docdb_compat/expected/` to reflect the revised index condition format using `elemMatchIndexOp`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
|
||
|
Vinod Sridharan
|
2108af32e6 |
Merged PR 1812986: [Hidden Index] Support Hidden Indexes with Distribution
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add support for hidden indexes with distributed tables. - Add function that supports per node distribution with shards: Select all shards for the table and pick one shard per node. - Add a planner worker function that executes on every shard - On the shard skip execution if it's not the selected shard - Run the local function on the worker function. - Plumb in hooks to execute the index functions remotely. ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature to support hidden indexes in a distributed environment. #### PR Summary The changes integrate hidden index support into distributed index update operations by introducing new per-node functions, updating API hooks, and modifying planner and UDF logic for proper shard routing and logging. - **`node_distribution_operations.c`**: New file that adds functions (e.g., `UpdateDistributedPostgresIndex` and its worker) to execute index update commands per node in a distributed setting. - **`diagnostic_commands_common.c`**: Introduces `command_node_worker` to ensure that update commands target the appropriate shard based on table identity. - **`coll_mod.c` and `coll_mod.h`**: Updated to use `UpdatePostgresIndexWithOverride` and correctly incorporate hidden index modifications with distributed operations. - **API hooks and metadata files** (e.g., `api_hooks.c`, `api_hooks_def.h`, `api_hooks.h`, and `metadata_cache.c`): Enhanced to support new function pointers and caching mechanisms for hidden index updates via the new command worker. - **SQL UDF scripts**: New UDF definitions for `command_node_worker` and `update_postgres_index_worker` enable proper registration and execution of hidden index operations in the distributed framework. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
 Santiago Fernandez Madero
|
9ac8147799 |
Merged PR 1786779: [background unique build 1/3] Create index as unique support
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description This new option for create indexes creates an index as if it was a UNIQUE composite term index, but without actually creating the constraint on the table, so it is a regular index. This is the first phase of 3 needed. We introduce a three-step workflow that lets customers build the index structure in the background (non-blocking), then progressively enable uniqueness enforcement: 1. buildAsUnique (createIndex) — build the index as if unique but without registering any table constraint (non-blocking) 2. prepareUnique (collMod) — register the uniqueness constraint in the system catalogs but mark it unvalidated so it starts enforcing uniqueness for new writes only 3. convertUnique (collMod) — validate/enforce uniqueness for existing rows (walk the table or short-circuit if an existing validated unique index already proves uniqueness) ## Goals - Allow background construction of unique-capable indexes without blocking writes - Provide a fast metadata-only step to enable uniqueness for new writes - Provide a validation step that enforces uniqueness for pre-existing rows only when it is safe - Preserve data integrity and PostgreSQL metadata correctness - Expose these steps through familiar MongoDB commands (`createIndex` and `collMod`) - Migrating non ordered unique indexes to ordered indexes without blocking production writes. ## High-level Workflow Step 1 — buildAsUnique (createIndex): - Customer requests an index with storageEngine.buildAsUnique=true. - The system creates a regular index physically (non-blocking) but records metadata indicating the index is intended to become unique Step 2 — prepareUnique (collMod): - Customer requests convert-to-unique preparation via `collMod` (prepareUnique) - The system registers a constraint in the PostgreSQL catalogs but does not validate it against existing data; it enforces uniqueness on new writes going forward - This is a fast, mostly metadata-only operation (convalidated = false) Step 3 — convertUnique (collMod): - Customer requests final conversion via `collMod` (convertUnique) - The system validates uniqueness for existing rows by walking the table (or aborts if duplicates are found), or it can skip the full walk when an existing validated unique index already proves uniqueness ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature that adds support for the "prepareForUnique" option in index creation. #### PR Summary The changes extend the index creation mechanism to allow a non-table unique index to be prepared via the "prepareForUnique" flag, with appropriate error handling and ... Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
 Deepak Bisht
|
1a695bdf9f |
Merged PR 1811981: [Observability] Add feature counters for all query operators
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Observability ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add feature counters for all public query operators ---- #### AI description (iteration 1) #### PR Classification This PR implements a new feature by adding comprehensive feature counters for all query operators to enhance observability. #### PR Summary The changes extend the observability framework by introducing new feature counter definitions, mappings, and usage in query operator implementations. - Updated `oss/pg_documentdb/include/utils/feature_counter.h` and `oss/pg_documentdb/src/infrastructure/feature_counter.c` to add new query operator feature enum values and corresponding string mappings with an increased `MAX_FEATURE_COUNT`. - Modified `oss/pg_documentdb/src/planner/mongo_query_operator.c` to assign the new feature counters to each query operator. - Revised test expected outputs in `/oss/internal/pg_documentdb_distributed/src/test/regress/expected/feature_counters.out` and `/pgmongo/src/test/docdb_compat/expected/feature_counters.out` to reflect the updated counters. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
ef8f7ae41c |
Merged PR 1809984: Add version logging to the backend
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) DevOps ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add Version Logging in the backend metrics collection. This makes it so we're not reliant on the gateway to log version information to metrics. ---- #### AI description (iteration 1) #### PR Classification This pull request introduces a new feature to log backend version metrics. #### PR Summary The changes add functionality to periodically log version information from various components, ensuring updated logging when version changes or timer intervals trigger a new record. - `pgmongo/src/infrastructure/mdsd_logger.c`: Added new functions (`append_version_metric_data` and `LogVersionMetricData`) to construct and send version metrics, and integrated these into the main logging workflow. - `pgmongo/src/pgmongo_extension_version.c`: Introduced `PgmongoGetExtendedVersionString` and updated the version-returning function to use this extended string. - `oss/pg_documentdb/src/utils/version_utils.c`: Added `GetCurrentShortVersionStringForLogging` to format a concise version string for backend logging. - `pgmongo/src/metadata/oid_cache.c`: Added the `CitusVersionFunctionOid` function and updated the OidCache structure to support version logging. - Header updates in `oss/pg_documentdb/include/utils/version_utils_private.h` and `pgmongo/include/metadata/oid_cache.h` to declare the new version utility and Oid functions. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
95d7a4b6a6 |
Merged PR 1806432: [OSS] Remove explicit RUM dependency on the documentdb extension
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactoring ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Make DocumentDB not explicitly depend on the RUM extension (it's done implicitly) on the binary. Instead pull in the SQL dependency into documentdb and reference the C functions internally. This allows us to swap between rum.so and extended_rum. For Pg18 (where there isn't yet official RUM support) default to using extended_rum for the .so file and remove all dependencies on the "rum.so" ---- #### AI description (iteration 2) #### PR Classification This PR is an API and dependency update that removes the explicit dependency on the external RUM extension by integrating its functionality internally. #### PR Summary The update integrates the RUM text-search capabilities directly into DocumentDB and cleans up the external linkage. - `oss/internal/pg_documentdb_extended_rum/src/rum_ts_utils.c` is newly added with core text-search functions. - `oss/pg_documentdb/src/index_am/rum.c` is modified to load internal wrappers that call the new RUM functions. - SQL UDF adapter files and operator class definitions are updated to reference the internal functions instead of public RUM functions. - Build scripts, metadata, and control files are adjusted to remove the explicit dependency on the external RUM extension. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
4bcd774983 |
Merged PR 1804966: [Update performance] Improve update performance in a few codepaths
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Perf ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description 1) Use prepared statements with UpdateAllMatchingDocuments 2) Don't pass shard_key_value by default if using the `@@` operator 3) Do something similar for SELECT update candidate. ---- #### AI description (iteration 1) #### PR Classification Performance optimization and bug fix for update query execution. #### PR Summary This PR enhances update performance by refining shard key filter handling and optimizing SPI plan selection, while also introducing new query plan cache keys and a configuration flag for legacy behavior. - `oss/pg_documentdb/src/commands/update.c`: Refactors update candidate selection by conditionally adding shard key filters, adjusting argument indexing, and using the full collection object instead of separate identifiers. - `oss/pg_documentdb/include/infrastructure/documentdb_plan_cache.h`: Introduces new constants for query plan caching to support optimized update queries. - `oss/pg_documentdb/src/configs/feature_flag_configs.c`: Adds the `UseLegacyShardKeyFilterOnUpdate` flag to offer backward compatibility with the previous shard key filtering behavior. - Test files in the distributed module: Update expected SQL query strings to match the revamped query construction logic. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
080ea66e2b |
Merged PR 1803246: [Hidden Index] Support for hidden indexes in DocumentDB
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add support for hidden indexes with collMod This will mark the index as not ready in the postgres catalog. Handles distributed setups as well. However, this is currently restricted to single node (sharded/unsharded). A subsequent change can help multi-node set ups. ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature by adding support for hidden indexes in DocumentDB. #### PR Summary The pull request extends the collMod command and associated index management logic to recognize and update a new “hidden” flag in index options, while enforcing constraints (e.g. disallowing hidden modifications for primary or unique indexes) in both single-node and distributed environments. - **`coll_mod.c`**: Introduces helper functions (e.g. GetHiddenFlagFromOptions, UpdateHiddenInIndexOptions, UpdatePostgresIndex) and SPI query updates to process the hidden flag properly. - **Test Files**: Adds and updates multiple test cases (e.g. in `commands_collmod_tests_hidden.out` and SQL test scripts) to verify hidden index creation, query planning, and error handling. - **Distributed Hooks and API**: Updates in files like `distributed_hooks.c` and `api_hooks.c` (with corresponding header changes) enable proper handling of hidden index updates across shard indexes in distributed environments. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
 William Wang
|
190c5f7710 |
Merged PR 1750825: [Full RBAC] Adding Support for RolesInfo Command in Engine (2/2, RolesInfo Impl.)
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) 1.108 ### Description We are adding support for the RolesInfo command in Engine ---- #### AI description (iteration 1) #### PR Classification New feature enhancement: introducing the `rolesInfo` command to retrieve detailed role information including inherited roles and privileges. #### PR Summary This pull request adds comprehensive support for the `rolesInfo` command in the engine, implementing new parsing, processing, and BSON output functions while updating tests and auxiliary query strings. - `oss/pg_documentdb/src/commands/roles.c`: Added functions (e.g., `ParseRolesInfoSpec`, `WriteRoleDocumentWithInheritedRoles`, `WriteInheritedRolesToArray`) to parse rolesInfo parameters and construct role documents with inherited roles and optional privileges. - `oss/pg_documentdb/src/test/regress/expected/role_crud_commands.out` and `pgmongo/src/test/regress/expected/role_crud_commands.out`: Expanded test suites with numerous rolesInfo scenarios, error conditions, and inheritance verification tests. - `oss/pg_documentdb/include/roles.h`: Introduced new types and function prototypes to support rolesInfo and role CRUD operations. - `pgmongo/src/pgmongo.c`: Updated query string definitions to handle role name substitutions for OSS compatibility. - `oss/pg_documentdb/src/commands/users.c`: Modified role checking and privilege extraction utilities to align with the new rolesInfo functionality. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4552244 Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Deepak Bisht
|
0f390b2cbb |
Merged PR 1798239: Fix Segmentation faults in ordered aggregate
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) BugFix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description - For aggregates such as `$last` or `$first` where the total number of aggregate values is 1 and existing aggregate state is more than the space required we try to use the same memory space while also updating it on the go. This will overwrite some later referred variable from the state. - The changes also fixes a memory leak in ordered aggregates where the new transition state was allocate in top level aggregate context and PG makes a copy and discards the original value (without freeing it), now we allocate the new transition state only in the CurrentContext which is the per tuple execution context So always create a new memory space for transition in ordered aggregates ---- #### AI description (iteration 1) #### PR Classification This pull request fixes a bug causing segmentation faults in ordered aggregate operations involving the `$last` operator used with the `$setWindowFields` stage. #### PR Summary This update resolves memory issues by removing unsafe byte array reuse in the state serialization for ordered aggregates and enhances regression tests to verify correct behavior. - `oss/pg_documentdb/src/aggregation/bson_sorted_accumulator.c`: Modified `SerializeOrderState` to always allocate new memory and removed the old byte array reuse logic. - `oss/pg_documentdb/include/aggregation/bson_sorted_accumulator.h`: Updated the function signature of `SerializeOrderState` to reflect the new implementation. - Test files in `oss/internal/pg_documentdb_distributed/src/test/regress` and `pgmongo/src/test/docdb_compat`: Added new tests for `$last` to check ordered aggregate functionality with `$setWindowFields`. - Documentation files (`docs/release_notes/pgmongo_1.108.md` and `oss/CHANGELOG.md`): Updated release notes and changelog to document the segmentation fault fix. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4687821 Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
32b11ab18f |
Merged PR 1799781: [Reindex] Support internally handling Reindex with background index processor
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add infrastructure to handle REINDEX CONCURRENTLY using hte background index queue. This doesn't expose any new public API at this time, and is meant as a first pass to enable administrators to enqueue reindex concurrently in the background. Progress can be tracked by currentOp ---- #### AI description (iteration 2) #### PR Classification This PR introduces a new feature that adds asynchronous reindex operations using the background index processor. #### PR Summary This update extends the background index processor to support reindexing by implementing new C functions and updating index queue management, SQL UDF definitions, and tests. Key changes include: - `src/commands/create_indexes_background.c`: Added `command_reindex_index_background` and `command_reindex_index_background_internal` functions and refactored common command core logic. - `src/metadata/index.c`: Modified index queue functions and metadata handling to support the new REINDEX command type. - `sql/udfs/index_mgmt/create_index_background--0.108-0.sql`: Added new SQL UDFs for reindexing indexes. - Test files under `src/test/regress` and `/pgmongo/src/test/regress`: Updated to validate the background reindexing functionality. - Minor updates in logging and distributed hook functions to accommodate reindex operations in the background processor. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
 Jia Hu
|
2a8a93311a |
Merged PR 1554321: [json schema] support keyword required
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md)) ### Description support keyword `required` in `$jsonSchema` ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request adds support for the `required` keyword in JSON schema validation. - Added `ParseRequired` function to handle the `required` keyword in `bson_json_schema_tree.c`. - Updated `BuildSchemaTreeCoreOnNode` to call `ParseRequired` when the `required` keyword is encountered. - Modified `ValidateBsonValueObject` in `bson_json_schema_validator.c` to validate required fields. - Added test cases for the `required` keyword in multiple test files (`bson_dollar_ops_json_schema_query_tests.out`, `bson_dollar_ops_json_schema_build_tree_tests.out`). - Updated `ValidationsObject` structure in `bson_json_schema_tree.h` to include a `required` field. Related work items: #4638803 Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
 Cornelius Boateng
|
ce0cb575be |
Merged PR 1794050: [engine][collation] Support for $sortArray
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Support for the `$sortArray` aggregation operator. ---- #### AI description (iteration 1) #### PR Classification This pull request implements collation support for the $sortArray operator as a new feature. #### PR Summary This change integrates collation awareness into the $sortArray operator so that array sorting properly respects collation rules. The modifications update both the underlying sorting logic and the tests to reflect these new capabilities. - `pg_documentdb/src/operators/bson_expression_array_operators.c`: Updated parsing of the $sortArray operator to copy the collation string into the sort context. - `pg_documentdb/src/utils/sort_utils.c` and `pg_documentdb/include/utils/sort_utils.h`: Switched to collation-aware comparison functions by replacing calls to CompareBsonValueAndType with CompareBsonValueAndTypeWithCollation. - Test expectation files under `/oss/internal/pg_documentdb_distributed/` and `/pgmongo/` were updated to include new $sortArray outputs with different collation settings. - Integration CSV files were modified to mark the previously failing `$sortArray` tests as passing. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
William Wang
|
5a37471d7d |
Merged PR 1784924: Utility File and Query Refactoring for Role/User CRUD APIs
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description ---- #### AI description (iteration 1) #### PR Classification This PR refactors and centralizes role/user CRUD utility functions. #### PR Summary The changes migrate and consolidate the logic for user and role management into a new utility module and header file, streamlining the codebase and removing redundant implementations from command files. - `src/utils/role_utils.c` adds core functions for password hashing, privilege consolidation, and user/role document writing. - `include/utils/role_utils.h` provides declarations for the newly centralized role/user utilities. - `src/commands/users.c` and `src/commands/roles.c` are cleaned up by removing duplicated role/user functionality now handled in the new module. - `pgmongo.c` and related metadata files are updated to use the new role replacement variable and adjust related constants. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4639026, #4639027 |
||
|
Deepak Bisht
|
2570b9c884 |
Merged PR 1788297: [OSS] Fix concurrent upsert bug, reapply the update spec on conflict while instead of replacing with new document
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Bugfix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) BugFix, data loss in concurrent upserts. ### Description On conflict for upserts we were replacing the document with newer document which is not the right behavior, we need to apply the second update on top of the first document for upserts. ---- #### AI description (iteration 1) #### PR Classification Bug fix: Resolve a concurrent upsert issue by reapplying the update spec on conflict instead of replacing the document. #### PR Summary This pull request corrects the upsert behavior by modifying the queries and function interfaces to reapply the update specification. It also adds new tests to validate the corrected concurrent upsert logic. - `oss/pg_documentdb/src/commands/insert.c` and `oss/pg_documentdb/include/commands/insert.h`: Modified the upsert SQL query to use `bson_update_document` and updated the function signature to require an additional update document parameter. - `oss/pg_documentdb/src/commands/update.c`: Updated calls to the upsert logic to pass the update spec, ensuring the update is reapplied on conflict. - `oss/pg_documentdb/src/update/bson_update.c`: Added logic to build update metadata when none is provided. - `pgmongo/src/test/regress/specs/isolation-concurrent-upserts.spec` and the corresponding expected output file: Introduced new tests and updated the test schedule to verify the concurrent upsert behavior. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4644372 |
||
|
William Wang
|
26ac331188 |
Merged PR 1758648: Adding Support for DropRole Command in Engine
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) 1.107 ### Description This PR adds the dropRole command to Engine. ---- #### AI description (iteration 1) #### PR Classification New feature: implementing support for the dropRole command. #### PR Summary This pull request adds the dropRole command to the engine, enabling deletion of custom roles with proper validations and dependency checks. It includes comprehensive tests and updates to documentation and feature tracking. - `oss/pg_documentdb/src/commands/roles.c`: Added functions to parse dropRole specifications and execute the drop operation with dependency and error handling. - `oss/pg_documentdb/include/roles.h` and feature counter files: Updated interfaces and feature mappings to expose dropRole support. - Test files under `oss/pg_documentdb/src/test/regress` and `pgmongo/src/test/regress`: Introduced extensive regression tests for dropRole, covering both success scenarios and various error cases. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4333807 |
||
 Jinrui Zhang
|
3ccd656663 |
Merged PR 1777067: [Refector] Update to match coding standards - part 0
### Description 1. Fixing issues identified by similarity checker 2. Fix the corresponding sql unit tests 3. Fix JSTests failure due to above changes 4. Fix corresponding C# test failure ---- #### AI description (iteration 1) #### PR Classification This pull request is a code cleanup that standardizes and improves error message wording across the codebase. #### PR Summary The changes update error messages throughout the repository to provide clearer, more consistent, and more detailed feedback when operations fail. - Updates in geospatial operator files (e.g. in `$geoNear`, `$near`, `$nearSphere`) replace informal phrasing with precise messages. - Numerous error messages in aggregation and projection test files have been reworded for consistent phrasing <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Siwen Tang
|
be11798bef |
Merged PR 1778507: [Refector] Update to match coding standards - part 5
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description draft update error msg ---- #### AI description (iteration 1) #### PR Classification This is a code cleanup and bug‑fix pull request that standardizes and clarifies error messages and log details across multiple modules. #### PR Summary The changes improve the consistency, clarity, and accuracy of user‑facing messages and logging throughout the code base. Key changes include: - In files under directories such as `oss/internal/pg_documentdb_distributed/src/test/regress/expected/` and `pgmongo/src/test/docdb_compat/expected/`, error messages for geospatial query operators (e.g. geoNear, $center, $within) and aggregation stage operators have been rewritten for clarity (e.g. replacing vague “Too many geoNear expressions” with “Excessive number of geoNear query expressions” and clarifying numeric parameter validations). - Several operators’ messages (e.g. `$cond`, `$not`, `$abs`, `$floor`, `$divide`, `$log`, etc.) have been updated to state exactly how many arguments are required, using <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Xing Fan
|
3b6061e6ea |
Merged PR 1777085: [Refector] Update to match coding standards - part 6
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description draft ---- #### AI description (iteration 1) #### PR Classification Bug fix: This PR revises error messages and inline comments to improve clarity and consistency across the code and test outputs. #### PR Summary The changes refactor and update a large number of error messages across various modules—ranging from aggregation and update operators to geospatial, schema validation, and command handling—to provide more precise diagnostic feedback. - **Tests (e.g. in `tests/expected/... <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Duan Pei
|
6a129514a9 |
Merged PR 1777887: [Refector] Update to match coding standards - part 1
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description update errmsg ---- #### AI description (iteration 1) #### PR Classification This pull request focuses on updating and standardizing error messages across various operators, aggregation stages, and test expectation files. #### PR Summary The changes reword error messages throughout the codebase and test outputs to provide clearer, more consistent, and more informative feedback for users. Key updates include: - **`/oss/internal/pg_documentdb_distributed/src/test/regress/expected/*.out`** – Revised expected error outputs for aggregation, update, and geospatial tests to use clearer phrasing (e.g., for `$merge`, `$pop`, `$indexOfArray`, and `$geoNear`). - **Core operator files (e.g., `bson_aggregation_output_pipeline.c`, `bson_expression_arithmetic_operators.c`, `bson_expression_conditional_operators.c`)** – Updated error message texts to explicitly state operator requirements and types. - **Geospatial and vector modules (e.g., files under `bson_geojson_utils.c`, `bson_geospatial_*`, and vector search/index configuration files)** – Standardized validations and error reporting for GeoJSON, CRS, and vector search parameters. - **Additional modules (e.g., collection view creation and user management files)** – Improved error handling messages to better guide users on input misconfigurations. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Jia Hu
|
7db51ff43b |
Merged PR 1777927: [Refector] Update to match coding standards - part 2
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) similarity issue ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description ---- #### AI description (iteration 1) #### PR Classification This pull request is a code cleanup aimed at standardizing and clarifying error and diagnostic messages throughout the codebase. #### PR Summary The changes update error messages and logging across many modules (aggregation, query, update, index creation, collation, etc.) to use clearer, more precise, and consistent language. - **Aggregation Operators:** Many files in the aggregation code (e.g. in `bson_aggregation_*` and related operator implementations) now report errors such as division by zero, invalid inputs, and argument count in a formal and uniform manner. - **Index & Projection Validation <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Yang Qiao
|
7f0c061b83 |
Merged PR 1778033: [Refactor] Update to match coding standards - part 4
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Testing ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Update to match coding standards ---- |
||
 Parag Jain
|
fa57f40720 |
Merged PR 1779662: [Observability] advanced feature usage for write commands
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description [Insert] Feature Usage Insert ---- #### AI description (iteration 1) #### PR Classification New feature addition that implements feature usage tracking for insert and update commands. #### PR Summary This pull request adds inline functions to report feature usage based on batch sizes for insert and update operations and updates the associated feature enum definitions. - `/oss/pg_documentdb/src/commands/insert.c`: Added `ReportInsertFeatureUsage` to report usage metrics based on batch size and integrated its call during batch insertion processing. - `/oss/pg_documentdb/src/commands/update.c`: Introduced `ReportUpdateFeatureUsage` with similar threshold-based reporting, integrated into the batch update process. - `/oss/pg_documentdb/include/utils/feature_counter.h`: Extended the feature counter enums with new constants for detailed insert and update reporting, and increased `MAX_FEATURE_COUNT`. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
William Wang
|
665098c897 |
Merged PR 1761487: Adding Support for RolesInfo Command in Engine (1/2, RolesInfo Parsing)
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) 1.107 ### Description We are implementing the RolesInfo command in Engine with three separate PRs. The first one will include the parsing of the input, the second one will be the core implementation of the command, and the third one will be the implementation of showBuiltInRoles. ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature by adding support for the rolesInfo command in the engine. #### PR Summary This update introduces functions to parse and handle the rolesInfo command, supporting various input formats and validations while updating error messages and tests. - **`roles.c`**: Added new parsing functions (`ParseRolesInfoSpec`, `ParseSingleRoleFromArray`, and `ParseRoleDocument`) and integrated distributed query logic in `roles_info`. - **`command_roles_info` & `create_role`**: Updated command wrappers and role validation to use the new logic and the `IS_SUPPORTED_BUILTIN_ROLE` macro. - **`roles.h`**: Introduced the `RolesInfoSpec` structure and defined the `IS_SUPPORTED_BUILTIN_ROLE` macro. - **Test Files**: Extended regression tests and expected outputs in both DocumentDB and Helio API test suites to cover multiple rolesInfo scenarios and error cases. - **`users.c` & Feature Counters**: Removed the redundant built-in role support function and added a new feature counter for rolesInfo on built-in roles. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4333805 |
||
|
Vinod Sridharan
|
c7613604d7 |
Merged PR 1777606: [Bugfix] Fixes for point read planner and lookup
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Bugfix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description For $lookup we were ignoring the joinquals for point read pushdown - this happened to work before if the planner chose a point read plan before. But now that we're more explicit in looking for this we need to handle this explicitly. Also ensure point read plan generation considers outer_relids (this wasn't done before). ---- #### AI description (iteration 1) #### PR Classification Bugfix: This PR corrects issues in the point read planner and lookup functionality. #### PR Summary The changes fix bugs related to index selection and query planning for multi-layer lookups by improving how join and restriction information is handled, and by extending test coverage for point read scenarios. - **`oss/pg_documentdb/src/opclass/index_support.c`**: Updated handling of outerRelids and join restrictions to ensure proper index path identification. - **`oss/pg_documentdb/src/planner/documents_planner.c` and `oss/pg_documentdb/include/opclass/bson_index_support.h`**: Modified calls to pass join info into restriction path processing for accurate planning. - Test files under **`oss/internal/pg_documentdb_distributed`** and **`pgmongo`**: Added new multi-layer lookup queries and expected outputs to verify that lookups consistently pick the _id index. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
a7b1076f31 |
Merged PR 1773571: Improve IndexScan and IndexOnlyScan Performance
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Improve performance of indexscans and indexonlyscans IndexTerm: >1) BsonIndexTerm is used in several places - reduce the size of this struct by removing isTruncated bool from the struct. >2) Add options to check metadata state without parsing the index term at all. >3) InitializeBsonIndexTerm - do not create a wrapper bson_value_t - just initialize the term directly. OrderedScans: >1) For every page we visit, we free the prior buffer and call PageGetTempPageCopy. Instead, keep one buffer per scan and just copy the buffer into the static buffer created per scan >2) When doing indexonlyscan, don't recreate an indextuple very time - instead reuse the memory and just fill the tuple buffer with what was previously allocated PgbsonWriter: >1) Add support for reusing the same writer across invocations by calling reset on the buffer >2) Add options to retrieve the value from a pgbsonwriter Composite: >1) Use a case dispatch for strategies rather than multiple if/else >2) Work on the serialized bson index terms as much as possible and lazy load the index terms >3) Skip Compare on bounds calls if there's no bounds declared ---- #### AI description (iteration 1) #### PR Classification This pull request delivers performance improvements and code refactoring for index scan operations. #### PR Summary The changes optimize index scans by refactoring tuple construction and buffer page handling for RUM indexes, while also unifying and streamlining BSON composite index term processing. - `pgmongo_rum/src/rumget.c`: Introduced `IndexBuildTupleDynamic` to build tuples without compression and reuse memory, and refactored page copy logic using `CopyPageContents` and an added `isPageValid` flag. - `oss/pg_documentdb/src/opclass/bson_gin_composite_entrypoint.c` (and related files): Replaced direct truncation checks with function calls (e.g., `IsIndexTermTruncated`), updated serialized composite index term initialization, and adjusted ordering transform logic. - `oss/pg_documentdb/src/opclass/bson_gin_index_term.c` (and accompanying headers): Added functions for handling serialized composite index terms and metadata, consolidating truncation flag logic. - `oss/pg_documentdb_core/src/io/pgbson.c`/`pgbson_writer.h`: Added new writer functions (e.g., reset and value retrieval) to improve memory reuse and reduce allocation overhead. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
10b395a75b |
Merged PR 1773514: [Orderby] Improve order by pushdown behavior & selectivity
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Pushing down order by and filters to a composite index should prefer the smallest index with the highest number of order by columns matching. To do this we need to participate in the core add_path set up as opposed to after. Consequently, we mutate the indexPath when its created in create_index_path as opposed to modifying it in the relpathlist_hook. We also reorder the indexes by paths so that for filters, we prefer indexes with fewer paths over many that can match the same paths. ---- #### AI description (iteration 1) #### PR Classification This pull request refines the ORDER BY pushdown mechanism in index scans by adding new processing logic and improving index selectivity while introducing a feature flag for the legacy behavior. #### PR Summary The changes enhance query performance by introducing distinct pushdown paths for ORDER BY clauses, refining composite index handling, and updating test expectations to reflect the new "orderByScan" operator. The following key modifications were made: - **`oss/pg_documentdb/src/opclass/index_support.c`**: Added new functions (e.g., `ConsiderIndexOrderByPushdownNew`, `ProcessOrderByStatements`) to split new versus legacy ORDER BY pushdown logic. - **Test Files** (under `/oss/internal/pg_documentdb_distributed/src/test/regress/expected/` and `/pgmongo/src/test/docdb_compat/expected/`): Updated expected query plans to use the new "orderByScan" predicate instead of "fullScan". - **Index and Planner Modules** (e.g., `oss/pg_documentdb/src/index_am/rum.c` and `planner/documents_planner.c`): Enhanced composite index ranking to factor in the number of index paths. - **Configuration Files** (e.g., `oss/pg_documentdb/src/configs/feature_flag_configs.c`): Introduced the `EnableIndexOrderbyPushdownLegacy` flag for toggling the legacy behavior. - **Supporting Modules** (in aggregation and query operators): Adjusted functions in files like `bson_query_common.c` and related headers to handle new order-by scan directions consistently. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
df1e6f4e61 |
Merged PR 1773287: [Explain] A few explain scan improvements based on recent investigations
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description A few explain improvements: 1) Push index usage under each index IXSCAN node (rather than at the top level aggregated) 2) Add a bool for indexonlyscan and add number of heap fetches under indexonlyscan 3) Only mark a scan as _id_ and COLLSCAN if it's an unsharded lookup with no object_id filter 4) For BitmapAnd, don't add an unnecessary FETCH stage before the bitmapindexscan IXSCAN 5) Don't print the index name in the parent FETCH stage of an ixscan 6) For aggregation scans, push the indexdetails to the $cursor stage (as opposed to next level stages). 7) In the backend - handle BitmapAnd for explain scan. Other bugs found in this: 1) Don't skip writing WAL for multikey status after index build (since the whole page image is already marked for WAL before that point) 2) Don't call ginUpdateStats (call rumUpdateStats) in parallel index build. ---- #### AI description (iteration 1) #### PR Classification This PR enhances explain scan functionality by improving index usage reporting and scan diagnostics across multiple components. #### PR Summary The changes refine the explain plan output for various scan types by better handling index conditions, propagating index details, and introducing new diagnostic fields. - `PostgresExplainHelper.cs`: Refactored index condition processing, improved index usage reporting, and added flags like `isIndexOnlyScan` and handling for heap fetches. - `mod.rs` (in pg_documentdb_gw explain): Updated query planning and execution stats logic to support Bitmap Index Scan and propagate index details into inner plans. - C source files (`index_am/rum_adapter.c` and `custom_explain_scan.c`): Simplified WAL logging, improved multi-key index status updating, and added handling for BitmapAndState in scan explanations. - `utf8.strings.csv`: Added new keys (`isIndexOnlyScan` and `totalDocsAnalyzed`) to support the enhanced explain output. - `CHANGELOG.md`: Documented the explain scan improvements as a new feature. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
c01cd9bf6c |
Merged PR 1768176: [Index Policies] Add a few more metadata lines in index policy logging
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Logging ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description 1) Track whether or not the index is TTL 2) Track numPaths different from numKeys (needed for composite) 3) Track whether it's a composite index or not Also add a check that checks that the PG index is valid as part of updating the stats before marking the index as valid in the documentdb Catalog ---- #### AI description (iteration 1) #### PR Classification Enhancement: The pull request updates index policy logging by adding new metadata fields and refining index property validations. #### PR Summary This pull request enriches index policy logging by incorporating new utility functions and metadata details to better capture index characteristics and ensure index integrity. - **`src/metadata/index.c`**: Added `IndexSpecIsWildcardIndex` and `IndexSpecIsOrderedIndex` functions to determine index types. - **`src/utils/index_policies_utils.c`**: Enhanced `BuildIndexMetricLog` by logging additional fields such as `numPaths`, `hasTtl`, `hasPfe`, and computing the wildcard and ordered index flags. - **`include/metadata/index.h`**: Introduced new inline functions for boolean conversion with default behaviors. - **Commands modules** (`create_indexes.c`, `coll_mod.c`, and `drop_indexes.c`): Updated validations (e.g., checking index liveliness) and adjusted boolean handling to align with the new metadata scheme. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
5a7b36d6ec |
Merged PR 1767749: Port roaring bitmap implementation to DocumentDB
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Port code into OSS - for multi-key we were using roaring bitmaps - port this change into OSS and make it the default implementation. ---- #### AI description (iteration 1) #### PR Classification This PR introduces a new feature by integrating the CRoaring bitmap library into DocumentDB. #### PR Summary The changes port the roaring bitmap implementation to DocumentDB by moving and updating key components, adjusting build settings, and adding required licensing documentation. - **`oss/pg_documentdb/src/index_am/rum.c`**: Updates include registering the roaring bitmap state functions (setting `IndexArrayStateFuncs` to `&RoaringStateFuncs`) with improved error handling on duplicate registration. - **File moves**: The adapter file was moved from `pgmongo/src/bitmap_utils/roaring_bitmap_adapter.c` to `oss/pg_documentdb/src/index_am/roaring_bitmap_adapter.c`, with corresponding include path updates in several files. - **`oss/pg_documentdb/Makefile`**: Modified to add the build flag `-DCROARING_ATOMIC_IMPL=3` for using the C atomics implementation, while the same flag was removed from `pgmongo/Makefile`. - **Initialization changes**: In `oss/pg_documentdb/src/pg_documentdb.c`, a new hook `RegisterRoaringBitmapHooks()` is now registered during module initialization. - **Licensing and notice updates**: A new CRoaring license file was added and the `oss/NOTICE` file was updated to include CRoaring attribution. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Jayanta Mondal
|
6b95258b24 |
Merged PR 1751463: [TTL] use index hints to force ordered index scan on composite TTL indexes
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
In this PR, for TTL indexes, created as a composite index, we force an ordered scan while querying for eligible rows using index hints.
Composite indexes support ordered scan on the index allowing the scan to be really efficient on the index. Without composite index, the planner uses a bitmap index scan which requires fetching all index pages and creating a bitmap. If there are a lot of documents to deleted, this can put a lot of pressure on the disk IOPS due to repeated fetching of large amount of index pages.
When TTL index is created as a composite index, we effectively bypass the planner via the use of index hints and forces postgres to use the composite index. When index hint is provided the planner pins an ordered scan on the index provided as a hint.
Below is an example of how an additional condition is added, as an index hint, to the query for fetching purge eligible rows:
```sql
SELECT ctid FROM mongo_data.documents_1963502_19635007
WHERE bson_dollar_lt(document, '{ "ttl" : { "$date" : { "$numberLong" : "1657900030775" } } }'::bson)
AND helio_api_internal.bson_dollar_index_hint(document, 'ttl_index'::text, '{"key": {"ttl": 1}}'::bson, isSparse)
```
|
||
|
Santiago Fernandez Madero
|
dbdaee55ea |
Merged PR 1764519: [Index Only Scan] Add support for index only scans at the planner and rum layer
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
This adds index only scan support for simple aggregate queries that don't project anything. i.e: a count query.
```
db.index_only_scan.explain("allPlansExecution").aggregate([
{ $match: { country: { $gte: "Brazil" } } },
{ $group: { _id: "1", n: { $sum: 1 } } }
]);
```
Next steps:
1. Support on sharded collections
2. Support with index hints
3. Support when projection matches index
4. Check null equality support
5. Add explain entry for truncated terms
----
#### AI description (iteration 1)
#### PR Classification
This pull request implements a new feature to support index only scans across the planner and rum layers.
#### PR Summary
The changes enable queries to be satisfied solely by the index, reducing table heap access and potentially improving performance.
- **Test Files:** New composite index only scan tests were added in files under `oss/internal/pg_documentdb_distributed/src/test/regress` and `pgmongo/src/test/docdb_compat`, along with new SQL test scripts.
- **Index Support Code:** Modifications in `oss/pg_documentdb/src/opclass/index_support.c` (and related files) add functions such as `ConsiderIndexOnlyScan` and update projection logic to assess and enable index only scan pathways.
- **Rum Layer Adjustments:** Changes in files like `rumget.c` and `rumscan.c` (including their pgmongo counterparts) implement support for projecting index tuple results and adjust cost computations for index only scans.
- **Configuration Updates:** New GUC settings and feature flag changes (e.g. in `/oss/pg_documentdb/src/configs/feature_flag_configs.c`) introduce and control the `EnableIndexOnlyScan` parameter.
- **Planner Integration:** Updates in `documents_planner.c` now invoke the index only scan consideration function during path list construction to integrate the new feature into query planning.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
|
||
|
Jia Hu
|
aee4a8feed |
Merged PR 1584197: [Encrypt] [json schema] support CSFLE schema enforcement and schema rule check
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description enable customer to configure CSFLE schema on the server side add key word 'encrypt'/'encryptMetadata' support for `$jsonSchema` ---- #### AI description (iteration 1) #### PR Classification New feature: client-side field level encryption schema support. #### PR Summary This pull request introduces support for client-side field level encryption in the schema validation. - `bson_json_schema_validator.c`: Added `ValidateBsonValueBinary` function to handle binary value validation for encryption. - `bson_json_schema_tree.c`: Added parsing logic for `encrypt` and `encryptMetadata` keywords in the schema. - `bson_json_schema_tree.h`: Introduced `BinaryValidationTypes` enum and `ValidationsBinary` struct to support encryption validation. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
90b70c07f2 |
Merged PR 1760321: [Sort] Support descending sort on an ascending index
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Today, sorts require the index and sort direction to match exactly. However, RUM indexes can support backward scan as well, so we can push descending sorts to ascending indexes. Enable this capability. However, this required a few changes to the ordered scan to support moving backward in the tree in RUM This also required us to use a new order by descending operator since Postgres by default only supports ascending sorts on orderbyop. ---- #### AI description (iteration 1) #### PR Classification New feature enhancement to add descending sort capabilities on ascending indexes. #### PR Summary This pull request introduces comprehensive support for descending sort—integrating reverse flags into comparator functions, scan direction adjustments, and new operator metadata—to enable proper pushdown and execution of reverse order queries. - `oss/pg_documentdb/src/query/bson_dollar_operators.c`: Refactored sort input and comparator functions to incorporate reverse flags (`isReverse`) and adjust value comparisons for descending order. - `oss/pg_documentdb/src/opclass/bson_gin_composite_entrypoint.c` and related index modules: Added logic for backward scans, including new functions to determine scan direction and reverse comparison adjustments. - Test SQL files (e.g. `bson_composite_order_by_index_tests.sql`): Extended the test suite with descending order queries and validate explain plans for reverse scan ordering. - Metadata and operator definitions: Introduced new reverse order operator functions (e.g. `BsonOrderByReverseIndexOperatorId`) and updated index support utilities to correctly process backward scans. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Deepak Bisht
|
b1471da24d |
Merged PR 1757273: Fix $lookup index pushdown
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) BugFix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes `documentdb.forceBitmapScanForLookup` now controls if we need force bitmap index scan paths for lookup JOIN queries ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) - $lookup JOIN condition pushed to index more reliably and avoids bitmap heap scan conversion. ### Description Fix index pushdown for $lookup with more data. Problem description: 1- Today when the lookup index scan paths are converted to bitmap heap scan path and while doing so we loose the parameterization info already build by PG planner, and the converted path considers all rows to be returned, this impacts index selectivity as the path becomes expensive. Solution: - Fixed the bitmap heap scan path by adjusting the estimated rows returned from the parameterized info. - Skip force conversion of index path to bitmap heap scan path for $lookup JOIN cases behind a GUC which should be default behavior going forward ---- #### AI description (iteration 1) #### PR Classification Bug fix addressing incorrect $lookup index pushdown and lookup join behavior. #### PR Summary This PR refines the planner and execution paths for $lookup queries by improving bitmap scan conversion, selectivity estimation, and associated configuration while updating regression tests to validate the new behavior. - `pgmongo/src/test/docdb_compat/expected/bson_aggregation_tests_lookup_inner_join.out` and `oss/internal/pg_documentdb_distributed/src/test/regress/sql/bson_aggregation_tests_lookup_inner_join.sql`: Updated test cases and expected outputs to exercise various lookup join scenarios and index usage modes. - `oss/pg_documentdb/src/planner/documents_planner.c`: Introduced the helper `IsBitmapHeapConversionSupported` to centralize conditions for pushing down bitmap heap conversion, replacing the older merge query check. - `oss/pg_documentdb/src/customscan/custom_scan_continuation.c`: Improved handling of parameterization in index pushdown paths for lookup queries to restore accurate row estimates. - `oss/pg_documentdb/src/opclass/index_support.c` and `oss/pg_documentdb/src/query/bson_dollar_selectivity.c`: Modified lookup filter support functions to enforce low selectivity for lookup-modified index quals. - `oss/pg_documentdb/src/configs/feature_flag_configs.c` and `oss/pg_documentdb/include/query/bson_dollar_selectivity.h`: Added and centralized the `forceBitmapScanForLookup` flag along with selectivity constants for consistent behavior. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4522655 |
||
|
Vinod Sridharan
|
de0ba5546b |
Merged PR 1758786: [RUM] Code Cleanup & Porting more changes to OSS
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
1) Porting some more code to OSS from the private rum fork to minimize deltas more
a) Clean up usage of setting flags to use the Macros instead
b) Initialize scanEntries to 0 for explain
c) Port comment on cleanup of left walk of RUM tree
d) Add support for reporting index build phase (this is in prep for parallel index build)
2) Remove unused hooks and deprecated hooks
a) MultiAndJoin (never exercised)
b) Overriding the RUM am handler - this is superceded with index_am extensibility.
----
#### AI description (iteration 1)
#### PR Classification
This pull request performs code cleanup by removing deprecated legacy functions and consolidating routines while porting essential changes to the OSS release.
#### PR Summary
The changes streamline the RUM code by eliminating obsolete routines and legacy hooks, consolidating scan initialization and bitmap handling, and enhancing progress reporting and page deletion reliability.
- `pgmongo/src/customscan/rum_custom_join_scan.c` was deleted to remove unused functionality.
- `pgmongo_rum/src/rumget.c` refactored scan initialization by merging functions and removing the legacy multiandgetbitmap implementation.
- `pgmongo/src/helio_gin_am/helio_gin_utils.c` removed outdated helio gin routines and associated hook functions.
- `oss/internal/pg_documentdb_extended_rum/src/rumutil.c` and `pg_documentdb_rum.h` added a new `rumbuildphasename` function with progress phase constants.
- Changes in `rumvacuum.c` (and related files) introduced retry logic and replaced direct flag manipulation with new macros (`RumPageIsNotDeleted`, `RumPageForceSetDeleted`) for improved page deletion handling.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
|
||
|
Vinod Sridharan
|
0c509dc131 |
Merged PR 1754381: [Sort] Support sort by id pushdown to the index
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Support pushdown of sort of _id to the primary key btree index if available. ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature to push down sort on the _id field into the index, optimizing query ordering for primary key lookups. #### PR Summary The changes introduce new logic to detect and push down sort-by-_id operations by leveraging primary key indexes, update sort expression creation to use collation-aware functions, and remove outdated configuration flags. - `src/opclass/index_support.c`: Added and refactored code to identify primary key indexes and create new index paths when sorting by _id. - `src/aggregation/bson_aggregation_pipeline.c`: Updated sort function logic to apply collation and proper pushdown criteria, including full-scan clause insertion when necessary. - Test and expected output files: Adjusted explain plan outputs and renamed feature counters (switching from `sort_by_id_pushdownable` to `sort_by_id`) along with modifying configuration flags. - `include/opclass/bson_index_support.h` and config files: Introduced a new context structure (`PlannerQueryOrderByData`) and removed obsolete variables related to sort-by-_id pushdown. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
b7d5595c98 |
Merged PR 1753652: [Feature Flags] Clean up older feature flags post 106 schema change
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Cleanup ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Remove dead feature flags, and set some to true now that 106 schema has upgraded. ---- #### AI description (iteration 1) #### PR Classification This PR performs a code cleanup and deprecation of legacy feature flags following the v106 schema change. #### PR Summary The changes remove obsolete feature flag checks and legacy code paths, streamlining the codebase to always use the new behaviors. Notable updates include: - **`pg_documentdb/src/infrastructure/cursor_store.c`**: Introduces file-based persisted cursor storage. - **`pg_documentdb/src/commands/cursors.c`**: Eliminates the conditional branch for `UseRawExecutorForQueryPlan`, unifying query execution. - **`pg_documentdb/src/configs/feature_flag_configs.c`**: Removes several deprecated feature flag definitions (e.g., raw executor, unsafe index term transform, legacy order by behavior). - **`pg_documentdb/src/sharding/sharding.c`**: Switches to the new shard key calculation path by removing legacy conditional checks. - **Other modules (opclass, create_collection, commands_common, bson operators)**: Clean up legacy feature flag conditions to enforce the new, consistent behavior. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Santiago Fernandez Madero
|
3245167027 |
Merged PR 1749636: [Composite] Prefer composite indexes over non composite on the planer layer
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description [Composite] Prefer composite indexes over non composite on the planer layer This implements the get_rel_info_hook and reorders the rel indexlist based on the following ordering: 1. primary key index 2. composite index 3. regular indexes 4. wildcard indexes 5. Any other index This also renames enableCompositeTerm to enableOrderedIndex option on the metadata options we return to the user ---- #### AI description (iteration 1) #### PR Classification Feature enhancement: this PR updates the planner to preferentially use composite indexes over non-composite ones. #### PR Summary The pull request implements index ordering logic in the planner to favor composite indexes by introducing new sorting and logging functions, replacing the legacy “enableCompositeTerm” with “enableOrderedIndex”, and updating the associated feature flags and tests. - **`src/planner/documents_planner.c`**: Added functions (e.g., `GetIndexOptInfoSortOrder`, `CompareIndexOptionsFunc`, and `ExtensionGetRelationInfoHook*`) to prioritize indexes so that composite indexes rank higher. - **`src/configs/feature_flag_configs.c` and `src/configs/testing_configs.c`**: Introduced new boolean flags (`enableIndexPriorityOrdering` and `logRelationIndexesOrder`) to control the index ordering and logging behavior. - **`src/metadata/index.c` and `src/commands/create_indexes.c`**: Updated index creation and comparison logic to replace `enableCompositeTerm` with `enableOrderedIndex` for consistent composite index handling. - **Test files (under `/oss/pg_documentdb/src/test/regress/` and `/pgmongo/src/test/`)**: Revised expected outputs and SQL tests to validate that composite indexes are now preferred in the query planner. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Parag Jain
|
07788048f4 |
Merged PR 1749844: [BugFix] lock fixes for bulk insert procedure
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description We should lock shard table after each commit. ---- #### AI description (iteration 1) #### PR Classification Bug fix targeting issues in bulk insert operations. #### PR Summary This pull request fixes bulk insert bugs by updating test expectations and core commit-lock logic for multi-batch operations. The changes ensure proper handling of duplicate keys and intermediate commits by introducing tighter shard table locking and updating batch configuration parameters. - **Tests**: Updated expected outputs in `command_insert_procedure.out` and related SQL files (both in `/oss/internal/pg_documentdb_distributed` and `/pgmongo`) to include multi-batch scenarios and duplicate insertion handling. - **Update Commands**: Modified update bulk tests in expected output files to reflect new debug messages and adjusted batch commit counts. - **Core Code**: Updated `commands_common.c`, `insert.c`, and `update.c` to pass a new `shardTableOid` parameter to `CommitWriteProcedureAndReacquireCollectionLock`, ensuring the shard table is locked when valid. - **Test Substitutions**: Revised `apply_helio_test_substitutions.sh` to update the batch write transaction parameter from `documentdb.batchWriteSubTransactionCount`/`helio_api.batchWriteSubTransactionCount` to `pgmongo.batchWriteSubTransactionCount`. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
5bf241a175 |
Merged PR 1745338: Add better support for custom createIndex options
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Mongo drivers don't pass through all the options to createIndex requiring customers to use runCommand() whenever there's custom overrides in index builds. However, mongo drivers pass-through storageEngine through to the server. Use that object to pass in custom options (like blocking index create, compositeTerm etc) to the server. E.g.:  ---- #### AI description (iteration 1) #### PR Classification This pull request implements a feature enhancement to improve support for custom createIndex options. #### PR Summary The changes refactor and extend index creation logic by centralizing custom option parsing and introducing a blocking flag to control concurrent index creation. The modifications also update test expectations to validate the new behavior. - **`oss/pg_documentdb/src/commands/create_indexes.c`**: Moved custom option parsing (e.g., for `enableLargeIndexKeys`, composite term options, and collation) into a dedicated function (`ParseCustomIndexDefOption`) and integrated processing of custom `storageEngine` options. - **`oss/pg_documentdb/src/commands/create_indexes_background.c`**: Adjusted index creation logic to set the index as non-concurrent when a blocking flag is specified. - **`oss/pg_documentdb/include/commands/create_indexes.h`**: Extended the `IndexDef` structure by adding a new `blocking` field. - **Test files under `oss/internal/pg_documentdb_distributed/src/test/regress/` and `pgmongo/src/test/docdb_compat/expected/`**: Updated expected outputs to reflect the new custom index options and blocking index creation behavior. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
df40dd8ec6 |
Merged PR 1743184: [Index] Support Prefered ordered scans when the index supports ordering
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description ---- #### AI description (iteration 1) #### PR Classification This PR introduces a new feature to enable preferential ordered index scans when the index supports ordering. #### PR Summary The pull request enhances index scan behavior by selecting an optimized ordered scan path when only one index entry is present and the new configuration flag is enabled, resulting in improved query planning. Key changes include: - **`rumget.c` (both `pg_documentdb_extended_rum` and `pgmongo_rum`):** Added a branch to perform an ordered scan when `RumPreferOrderedIndexScan` is true and a single entry is provided. - **`rumutil.c` (both implementations):** Introduced a new GUC variable to control the preference for ordered index scans. - **Header files (`pg_documentdb_rum.h` and `rum.h`):** Declared and defined the new `RumPreferOrderedIndexScan` flag with a default value. - **Test files:** Added new expected output and SQL regression tests to validate composite index scans under the preferred ordered scan configuration. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
623ef26f85 |
Merged PR 1742378: [Hints] Add support for index hints in DocumentDB
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Index hints are a mongo feature that allows customers to specify which index they would like to target for a query. Typically this is used in diagnostic scenarios when testing query performance. With the new composite index op class coming in, we’re introducing side by side support for composite which makes it so we would want customers to be able to test which index is better for planning. In this spirit, index hints is introduced which follows the same infra as the force pushdown for geonear and text and adds a new option for index hints. As part of this the code is also refactored to cleanup some of the logic in the relpathlist and fix correctness issues around text index force pushdown (which was needed to handle index hints) ---- #### AI description (iteration 1) #### PR Classification This pull request introduces a new feature to support index hints in DocumentDB. #### PR Summary The changes integrate index hint support into query planning and execution, allowing users to force a specific index during query processing. Key updates include: - **`src/opclass/index_support.c`**: Added new functions and a force index operator type (`ForceIndexOpType_IndexHint`) for filtering available indexes and generating custom operator expressions based on index hints. - **`src/planner/documents_planner.c`**: Modified base table query planning logic to detect index hints and adjust index pushdown, including updates to path and restriction handling. - New UDFs (e.g., `bson_dollar_index_hint`) and configuration variables (`EnableIndexHintSupport`, `UseLegacyForcePushdownBehavior`) were added to support the hint mechanism. - Test files and SQL scripts (e.g., in `/oss/internal/pg_documentdb_distributed/src/test/regress/`) were introduced or updated to validate index hint pushdown behavior. - Documentation and release notes (e.g., `docs/pgmongo/design_docs/data_plane/indexing/index_hints.md`, `pgmongo_1.106.md`, and `CHANGELOG.md`) were updated to describe the index hint feature. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Lucas Borges Fernandes
|
4800068a67 |
Merged PR 1741680: [OSS] Improving index categorization api
### Does this PR have any customer impact? No. ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactoring. ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No. ### Are you introducing any new config? If yes, do you have tests with and without them being set? No. ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) N/A ### Description The purpose of this change is to add infrastructure to improve categorization and parsing of index specifications. This PR adds a function named `GetIndexTypeFromKeyDocument` to extract the index type from a pgbson key document. The index types can be any of regular, hashed, 2d, text, 2dsphere and cosmosSearch. There are no schema changes and no new configs. ---- #### AI description (iteration 1) #### PR Classification This pull request improves the index categorization API by refactoring utility functions and consolidating index type definitions. #### PR Summary The update adds a common helper to compare index types and a new function to determine an index’s type from a key document, while unifying index type definitions across headers and source files. - `src/metadata/index.c`: Introduced `IsIndexOfType` and `GetIndexTypeFromKeyDocument` to centralize index type comparisons. - `include/metadata/index.h`: Added the `MongoIndexKind` enum and `MongoIndexSupport` struct, with declarations for global index type arrays. - `include/commands/create_indexes.h` and `src/commands/create_indexes.c`: Removed duplicate index type definitions and updated the index support list for consistency. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
William Wang
|
e4024ad3ad |
Merged PR 1721770: [OSS] [Engine] Adding Support for Create Role Command and API Schema for Role CRUD
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) 105 ### Description As part of Full RBAC, we are adding support for role management CRUD operations. This PR adds support for the createRole Command and API Schema for dropRole, rolesInfo, updateRole commands Related work items: #4053600, #4338364 |
||
 Amit Prabhudesai
|
e25146b223 |
Merged PR 1739781: [OSS] Fix background worker tests
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Testing ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Fix OSS background worker tests ---- #### AI description (iteration 1) #### PR Classification This pull request fixes failing background worker tests by ensuring proper worker initialization and improving error handling. #### PR Summary The changes add waiting procedures in test helper SQL scripts to guarantee that background workers are fully launched before tests run, while also enhancing error logging and renaming configuration functions for clarity. - `documentdb_distributed_test_helpers.sql`, `documentdb_test_helpers.sql`, and `pgmongo_test_helpers.sql`: Introduced a stored procedure that loops until the appropriate background worker is detected, with corresponding calls added in the test scripts. - Expected output files (`documentdb_distributed_test_helpers.out`, `documentdb_test_helpers.out`, and `pgmongo_test_helpers.out`) now include these wait procedures to align with the new test logic. - `background_worker.c`: Modified the error handling in the background worker command generation to capture and log detailed error metadata (filename, line number, message_id). - `documentdb_init.c` and `background_job_configs.c` (along with header changes): Renamed and updated configuration functions from `InitDocumentDBBackgroundWorkerGucs` to `InitDocumentDBBackgroundWorkerConfigurations` and adjusted naming from `build_index_background` to `command_build_index_background`. - `postgresql.conf` files in multiple directories: Updated background worker settings by setting the background worker database name to 'regression'. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
William Wang
|
331d692a63 |
Merged PR 1686552: Adding Support for All Role Querying in UsersInfo/ConnectionStatus and User Query Optimization
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No. ### Are you introducing any new config? If yes, do you have tests with and without them being set? No. ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) 1.106 ### Description This PR introduces the following changes: - UsersInfo and ConnectionStatus commands to return all supported roles, instead of only ReadOnly and Admin roles; specifically, this PR adds the supported roles of ReadWrite, UserAdmin, and Root roles, along with privilege sets dbAdmin, userAdmin, clusterMonitor, clusterManager, and hostManager. - Modifies get user logic to use a hash table to record users and their roles, with role filtering. - Optimizes and beautifies queries within users.c. Related work items: #4158026, #4315160 |
||
|
Cornelius Boateng
|
f983efd1da |
Merged PR 1683401: [engine][let] Support for update
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support let and `$$NOW` variables with update queries.
The values of the variables can be accessed in the query `q` with `$expr` and 'u' with an aggregation pipeline.
Example:
```
{
update: <collection>,
updates: [
{
q: {$expr: {$eq: ["$_id", "$$varRef"]}},
u: [{$set: {"a": "$$NOW"}}],
multi: false, // updateOne
}
],
let: {"varRef": 1},
...
}
```
We parse the let value (together with the time system variables - $$NOW) and propagate it through the aggregation and variable contexts.
To enable feature, turn on the `EnableVariablesSupportForWriteCommands` GUC.
----
#### AI description (iteration 1)
#### PR Classification
This PR implements a new feature by adding support for the "let" variable specification in update commands.
#### PR Summary
The changes integrate "let" support into update operations, allowing dynamic variable specifications to be applied during update candidate selection, serialization, and execution. This enhancement improves the flexibility and expressiveness of update queries in the engine.
- In `src/commands/update.c`, the code now parses and applies a new `variableSpec` (under the "let" field) when building update queries and managing SQL arguments.
- In `include/infrastructure/documentdb_plan_cache.h`, new plan cache macros are added to account for update candidates using let and collation.
- In `include/commands/update.h`, the `UpdateOneParams` structure is updated to include the `variableSpec` parameter.
- In `src/commands/delete.c`, references to the unsupported "let" field in delete commands are removed.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #3423115, #3613875
|
||
|
Vinod Sridharan
|
7688125032 |
Merged PR 1737756: [RUM] Support ordered scans without explicit order by
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description 1) Move forceRumOrderedScan from the documentdb rum layer to the rum index layer 2) Support having ordered scans even without there being an explicit order by in the query 3) Push this order by information to the op class using the GIN_SEARCH_MODE This is part one of many to support cleaning up the rum.c in the documentdb layer and push the extensibility needed into the index am directly. ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature to support ordered scans in RUM without requiring an explicit "order by" clause. #### PR Summary The changes refactor and enhance the RUM scan logic to support ordered scans by introducing a new ordering mechanism and related configuration. This improves the flexibility of index scanning operations. - **`/oss/internal/pg_documentdb_extended_rum/src/rumget.c` & `/pgmongo_rum/src/rumget.c`**: Refactored scan functions to use a dedicated `orderByEntry` (and updated recheck flags) instead of relying on a key from the keys array. - **`/oss/internal/pg_documentdb_extended_rum/src/rumscan.c` & `/pgmongo_rum/src/rumscan.c`**: Updated functions such as `MoveScanForward` and `startScanEntryOrderedCore` to handle ordered scanning through the newly defined `orderByEntry` and centralized `orderStack`, simplifying signature parameters. - **Configuration and Initialization Files**: Introduced the GUC variable `RumForceOrderedIndexScan` and modified scan key initialization in `rumutil.c` and related header files to support forced ordered index scans. - **Test and Opclass Updates**: Adjusted test configurations and expected outputs, and updated composite index-related code (e.g., removal of the `BuildCompositeOrderByScanKeyArgument` function) to reflect the new order scan behavior. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Amit Prabhudesai
|
6ac8a8c563 |
Merged PR 1734798: [Background worker] Dynamically scheduled job intervals via hooks
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Bugfix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Dynamically scheduled job intervals via hooks ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature that enables dynamically scheduled job intervals for background worker jobs via hook functions. #### PR Summary The pull request refactors background worker job scheduling by replacing the static schedule interval with a hook that dynamically determines the interval. A default hook is provided if none is set, and all related validations and logging have been updated accordingly. - `oss/pg_documentdb/src/background_worker/background_worker.c`: Updated functions (ValidateJob, RegisterBackgroundWorkerJob, and CanExecuteJob) to retrieve and validate the schedule interval using a hook, including logging when using the default interval. - `oss/pg_documentdb/include/background_worker/background_worker_job.h`: Replaced the static schedule interval field with a hook function pointer and added a typedef for the scheduling hook. - Added a default hook function, `GetDefaultScheduleIntervalInSeconds`, that returns a default interval of 60 seconds for jobs without a custom hook. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
