mirror of
https://github.com/FerretDB/documentdb.git
synced 2026-11-08 03:52:41 -05:00
a04ec8ea005be94de9ed28683198f7e1c1dd664b
121
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Vinod Sridharan
|
528b744a4f |
Merged PR 1717610: [Selectivity] Make Function Selectivity Match Opr Selectivity for Index
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description For Operator Selectivity, we have a feature flag and function that does better operator selecitivty for operator expressions. However, when doing planning, the query operator generates mostly FuncExprs for index operators. This means the selectivity logic for funcitons defaults to 0.3333 even if opr selectivity can do better. For Functions, when opr selectivity is enabled, use the same logic to determine selectivity enabling better index pushdowns. ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature to align function selectivity with operator selectivity for index operations in the DocumentDB Postgres extension. #### PR Summary The PR refactors selectivity calculations to ensure consistency between function and operator estimations, updates associated test cases, and introduces new configuration flags. Key changes include: - **`pg_documentdb/src/opclass/index_support.c`**: Revised the `dollar_support` function to route selectivity requests correctly using a new pointer variable and support for selectivity mode. - **`pg_documentdb/src/query/bson_dollar_selectivity.c` and `include/query/bson_dollar_selectivity.h`**: Updated the selectivity computation routines to accept a default selectivity parameter and adjust calculations based on the new operator selectivity mode. - **`configs/feature_flag_configs.c` and `system_configs.c`**: Added new configuration flags (`DisableDollarSupportFuncSelectivity` and `EnableNewOperatorSelectivityMode`) and adjusted default values to control the new selectivity behavior. - **Test files in `/oss/internal/pg_documentdb_distributed/src/test/regress/` and `/pgmongo/src/test/docdb_compat/expected/`**: Extended and modified SQL regression tests to verify expected index query plans and selectivity estimates. - **Configuration scripts (`/scripts/utils.sh` and related test configs)**: Made minor adjustments such as reducing the index build schedule interval to better support the new functionality. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
4c190c5f02 |
Merged PR 1713276: [POC: Order by] Support order by descending pushdown to the index if available
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description 1) Support order by descending as a feature under a feature flag if the underlying index can support it 2) Fix up memory performance issues causing OOMs with composite index with large numbers of bounds 3) On doing this noticed that row_get_bson was busted for int arrays, fix this behavior to not crash |
||
 Zinan Ren
|
b11753f8d2 |
Merged PR 1712315: [oss][test cleanup] Vector search related cleanup
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) No ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Cleanup vector tests |
||
 Santiago Fernandez Madero
|
c0b63a1fd2 |
Merged PR 1713045: [oss] Cleanup to conform with contributing guidelines
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description [oss] Cleanup to conform with contributing guidelines ---- #### AI description (iteration 1) #### PR Classification This pull request is a code cleanup that standardizes naming conventions to conform with the contributing guidelines. |
||
|
Vinod Sridharan
|
1c433c472b |
Merged PR 1714381: [Order by + Test] Fix up more correctness of order by + Testing
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Bugfix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description 1) There were order by tests not running but had the sql files in there. This was catching contracts related to arrays with null values where 2 cases were ordered later than they should be. This fixes that. As part of that we noticed that this breaks when there's paths that have array indexes (e.g. a.b.0). This also fixes that contract. 2) Add checks to say all tests must run or exist in a schedule file and fix the output 3) There's flakeyness in currentop due to tracking the version refresh query - skip the version refresh query in currentop. ---- #### AI description (iteration 1) #### PR Classification This pull request fixes correctness issues in composite order-by functionality and improves associated testing. #### PR Summary The changes refine the handling of intermediate arrays in order-by operations by updating traversal and comparison logic, while enhancing the test suite and validation scripts for composite order-by scenarios. - `pg_documentdb/src/query/bson_dollar_operators.c` and related files were modified to correctly process nested arrays and compare composite order-by values. - Test files in `pgmongo/src/test/regress/expected/` and `pgmongo/src/test/regress/sql/` were updated to add new order-by cases and remove obsolete tests. - Validation scripts (e.g. in `oss/internal/pg_documentdb_distributed/src/test/regress/validate_test_output.sh` and `oss/pg_documentdb/src/test/regress/validate_test_output.sh`) were enhanced for more reliable schedule checks and collection ID extraction. - Several BSON traversal and update operator functions were updated to include an `isArrayIndexSearch` parameter for finer control during array element processing. - `pg_documentdb/src/commands/current_op.c` and `pg_documentdb/src/utils/version_utils.c` were updated to skip version refresh queries appropriately. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Deepak Bisht
|
5250964838 |
Merged PR 1711674: Geospatial module cleanup
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Other ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Aligning to contributing guideline for geospatial module ---- #### AI description (iteration 1) #### PR Classification This pull request performs a code cleanup in the geospatial module to improve clarity and consistency across comments, error messages, and test outputs. #### PR Summary The changes refine inline documentation and output messages to accurately describe the intended behavior of geospatial operators—especially for the $geoWithin operator—and standardize legacy processing with Postgis expectations. |
||
|
Vinod Sridharan
|
c5b5c71596 |
Merged PR 1712147: [Docs] Fix up docs and such to match contributing guide
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactor ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Fix up some more documentation and comments around the code. ---- #### AI description (iteration 1) #### PR Classification This PR performs a code and documentation cleanup by refactoring naming conventions and removing deprecated staging functions to align with the contributing guide. #### PR Summary The pull request removes obsolete C wrappers for staging collection operations and standardizes internal API naming from "MongoCatalog" to "ApiCatalog" while refining inline comments for clarity and consistency. - **`src/metadata/collection.c` & `include/metadata/collection.h`**: Removed unused functions for handling staging collections during $out operations. - **`src/metadata/metadata_cache.c` & `include/metadata/metadata_cache.h`**: Renamed functions and variables from "MongoCatalog" to "ApiCatalog" to standardize internal identifiers. - **`src/aggregation/bson_aggregation_config_queries.c` & `src/aggregation/bson_aggregation_metadata_queries.c`**: Updated BSON aggregation queries to reference the new API naming. - **`src/infrastructure/cursor_store.c`**: Enhanced documentation by updating comments and adding a shared memory structure for cursor tracking. - **`src/metadata/index.c` & `src/index_am/rum.c`**: Applied minor comment corrections and removed outdated TODO notes for clarity. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Parag Jain
|
2dc64d2081 |
Merged PR 1711000: [oss] Cleanup as per new change guideline, part -1
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description 3 c files ---- #### AI description (iteration 1) #### PR Classification This pull request focuses on code cleanup and documentation updates. #### PR Summary The changes standardize and simplify documentation comments and examples across several operator implementations, removing obsolete references and unnecessary details. - `oss/pg_documentdb/src/aggregation/bson_aggregation_output_pipeline.c`: Updated doc examples for `$merge` and `$out` to use generic placeholders and removed temporary test-related comments. - `oss/pg_documentdb/src/query/bson_dollar_operators.c`: Cleaned up comments for multiple BSON dollar operators by eliminating "Mongo's" references and redundant test notes. - `oss/pg_documentdb_core/src/types/pcre_regex.c`: Refined regex compile option comments and removed legacy test reference lines. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
a547931808 |
Merged PR 1706921: [Background Index] Improve create indexes performance
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Performance ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Currently, each round of the sproc that does create indexes does exactly 1 index build and bails. This means if a single client request queues 4 indexes and the job runs every 2 seconds, it'll take 8 seconds to build all 4 indexes. This is wasteful since we know all 4 indexes up front. Add a loop to the index build process so that multiple indexes can be built without waiting the 2 seconds. As part of this, there were a few places allocating in memory contexts that are unstable across executions. Fix this. Additionally there was a loop that could cause index builds to starve for ~20 mins if it picked the wrong index. Fix this loop too. Finally, Noticed that the version cache does not cache properly if the Major version is 0. Fix this (which can cause perf regressions in DocumentDB). |
||
 William Wang
|
d339399b36 |
Merged PR 1694202: Adding Support for Connection Status Command in OSS Engine
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes, the API schema ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) 1.05 ### Description We are adding support for the Connection Status command, which exists in native Mongo, and allows users to see their authenticated user, authenticated roles, and user privileges. For more information, visit connectionStatus.md. This PR is the second of three PRs for this feature. The first one added extensible role privileges to our `users_info` command, allowing users to retrieve role privileges with `showPrivileges: true`. This second PR adds a callable `connection_status` command to our API, that utilizes the role privileges. The last PR will add a gateway change that calls this `connection_status` command. Related work items: #4287246 |
||
|
Vinod Sridharan
|
ccc471e169 |
Merged PR 1703619: [Cursors] Improve stability of file based cursors
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description ---- #### AI description (iteration 1) #### PR Classification Enhancement and bug fix to improve the stability and management of file-based cursors. #### PR Summary This pull request refactors and bolsters file-based cursor management by introducing shared memory support, improved cleanup, error handling, and configuration changes. It also adds new API endpoints and logging for better cursor lifecycle control. - **`cursor_store.c` and related files:** Added shared memory functions (e.g., `FileCursorShmemSize`, `InitializeFileCursorShmem`), cursor count regulation (with `IncrementCursorCount`/`DecrementCursorCount`), and cleanup routines to ensure stable cursor file operations. - **Test and expected output files:** Updated regression tests and expected outputs to cover scenarios like duplicate cursor creation, cursor deletion limits, and error reporting when the file cursor capacity is reached. - **API enhancements:** Introduced new UDFs (e.g., `command_delete_cursors`) and tenant client methods for killing and deleting cursors, improving session management. - **Configuration updates:** Renamed and added GUC parameters (e.g., `MaxAllowedCursorIntermediateFileSizeMB`, `MaxCursorFileCount`, and cursor expiry settings) to control file-based cursor behavior. - **Logging and metrics improvements:** Enhanced logging in `mdsd_logger.c` to report current cursor counts and related metrics for operational insight. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Cornelius Boateng
|
19ccc295bf |
Merged PR 1686340: [engine][let] Support for deleteOne on sharded collections with only _id filter
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description `deleteOne` requires `_id` or shard key value filter. If only `_id` is used, we query for the minimum shard key value and use it. The querying for that shard key must be able to use the `variableSpec`. ---- #### AI description (iteration 1) #### PR Classification API change to enable proper handling of deleteOne commands on sharded collections using only an _id filter by incorporating variable specification support. #### PR Summary This pull request enhances deleteOne functionality by integrating a variableSpec (let variable) into query construction for sharded collections, ensuring that _id-only filters are handled correctly. The changes update core command functions, test expectations, and configuration flags to support the new behavior. - **`commands_common.c`**: Revised the `FindShardKeyValueForDocumentId` function to accept a new variableSpec parameter and conditionally modify the query. - **`delete.c` and `update.c`**: Updated deletion and update operations to process the variableSpec for write commands. - **Test and Configuration Files**: Renamed and updated GUC flags (from enableLet... to enableVariablesAndCollationSupportForQueryMatch) and adjusted expected outputs to reflect the new deleteOne behavior. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
71e7d304b1 |
Merged PR 1695552: [Composite|Order by] Enable JSTest validation for these features
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature that enhances explain plan output and composite index functionality by adding order‑by pushdown and related index option validations. #### PR Summary The changes add new metrics in explain plans and strengthen composite index handling for order-by pushdown across the system. Key changes include: - **`/gateway/managed/Mongo.Service.Postgres/Helpers/PostgresExplainHelper.cs` & `/oss/pg_documentdb_gw/src/explain/mod.rs`** – Added fields like `sortKeysCount`, `sortKey`, `presortedKey(s)`, and `hasOrderBy` to enhance explain output. - **`/oss/pg_documentdb/src/metadata/index.c` and related index management files** – Implemented improved composite index option checks and support functions such as `CompositeIndexSupportsOrderByPushdown` and equality-range predicate validation. - **UDF and operator files (e.g., `bson_dollar_comparison--*.sql`, `/oss/pg_documentdb/src/query/bson_dollar_operators.c`)** – Introduced new functions like `bson_dollar_fullscan` to support full scan pushdowns in combination with composite indexes. - **`/pgmongo_rum/src/rumget.c`** – Enhanced ordered scan handling and duplicate management for multi-key indexes. - **Pipeline and configuration files (e.g., `.pipelines/templates/single_node_tests.yml`)** – Updated test scripts and experimental configuration parameters to enable the new composite opclass and order‑by pushdown features. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
c6bc2bcb73 |
Merged PR 1694458: [POC|Order by] Order by Index & Pushdown improvements
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description More general improvements to order by and composite: Create Indexes 1) Support creating a composite index side by side with a regular index 2) Block creation of descending or wildcard and unsupported set ups for composite indexes Planner: 1) Support ordering by the Nth path of an index if the prefixes are equality. 2) Validate that if it's not the first column, prefixes are provided 3) Support incremental sort with index pushdown. ---- #### AI description (iteration 1) #### PR Classification This pull request delivers a new feature and API enhancement to enable order-by index pushdown and refined composite index support. #### PR Summary The update streamlines composite index processing by introducing stricter equivalency checks, new UDF functions for full-scan order-by pushdown, and enhanced validations during index creation while updating explain outputs and tests. - `src/metadata/index.c` refines index options equivalency logic to correctly compare composite term settings. - `src/opclass/bson_gin_composite_entrypoint.c` and `src/index_am/rum.c` add functions to support order-by pushdown and manage duplicate entries efficiently. - `sql/udfs/query/bson_dollar_comparison--0.105-0.sql` (and related operator files) introduce new UDFs including `bson_dollar_fullscan` to enable full scan operations. - `src/commands/create_indexes.c` enforces new composite index validations (restricting use to ascending, non-wildcard regular indexes) with precise error reporting. - Test scripts and explain plan outputs in the `regress` and `pgmongo` directories are updated to confirm the new composite order-by behavior. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Shuai Tian
|
17d4306a28 |
Merged PR 1422337: [pgmongo] Aggregate stage $bucketAuto implementation
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md)) ### Description This pull request adds documentation for the `$bucketAuto` aggregation stage in PostgreSQL, detailing its design, usage, and implementation. - Adds `bucketAuto.md` with details on the `$bucketAuto` aggregation stage - code implementation - support all granularity types: POWERSOF2, 1-2-5, Renard numbers, E series Related work items: #3234185 |
||
|
Vinod Sridharan
|
06a96c8975 |
Merged PR 1691723: Fix up correctness of composite and orderby
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Bugfix/Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description 1) Orderby on null/undefined documents was incorrect - fix up the ordering 2) Equality on null/undefined was also incorrect fix this up 3) When traversing the document, we were doing an extra traversal into arrays when it was not needed. fix this up 4) Order by on composite index didn't handle nulls correctly - fix this too. 5) Support pushdown of $elemMatch into composite indexes without array terms. ---- #### AI description (iteration 1) #### PR Classification This pull request fixes bugs in the composite index and orderby functionality. #### PR Summary The changes correct the orderby comparison logic and improve composite index term generation in BSON processing. The pull request also adds new functions to handle intermediate array paths and introduces legacy behavior flags, while supplementing the fixes with comprehensive integration tests. - **`src/query/bson_dollar_operators.c`**: Modified `OrderByVisitArrayField` to pass the correct BSON value and added new functions for array path handling. - **`src/opclass/gin_index_term.c` and related opclass files**: Introduced `GenerateValueMaybeUndefinedTerm` and updated index term comparison to distinguish between undefined and partially undefined values. - **`pgbson_utils.c`**: Adjusted array traversal logic with a new callback for missing intermediate array paths and a flag to control array traverse optimization. - **Test files under `/oss/pg_documentdb/src/test/regress/expected/` and `/pgmongo/src/test/regress/expected/`**: Added extensive composite ordering and filtering tests to validate the corrected behavior. - **`configs/feature_flag_configs.c`**: Added custom GUCs for `useLegacyOrderByBehavior` and `useLegacyNullEqualityBehavior` to manage legacy ordering and null equality rules. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
0ddb09436c |
Merged PR 1690832: [Composite] Remove array term generation for composite indexes
### Does this PR have any customer impact? Not yet ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature/POC ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description This adds more changes for orderby support in the index 1) Fixes the correctness issue arising from indexing arrays for order by - we remove the top level array term from indexing. To handle this, we move the query operators to support querying the index correctly when the query is on an array 2) For `$elemMatch` this temporarily makes elemMatch just a full scan for now (a subsequent change wil tackle updating the logic) 3) Adds orderby to the top level schema for the internal extension 4) For exists false/true, we were pushing a lot more terms to the runtime recheck. Ensure that we're hitting the right index terms and evaluate correctly against the index. ---- #### AI description (iteration 1) #### PR Classification This pull request is a code cleanup and enhancement that removes array term generation for composite indexes and updates related index term handling. #### PR Summary The changes streamline composite index generation by eliminating dedicated array terms, and they adjust boundary, metadata, and recheck logic to ensure consistent behavior. Key updates include: - **`oss/pg_documentdb/src/opclass/bson_gin_composite_core.c`**: Removed redundant array term generation and introduced functions like `SetArrayEqualityBound` to properly handle equality bounds for arrays. - **`oss/pg_documentdb/src/opclass/gin_index_term.c`**: Updated index term serialization and metadata management (including handling of undefined values via `GenerateValueUndefinedTerm`) to align with the new composite index behavior. - **Test and expected output files**: Revised expected composite index term outputs and SQL tests to reflect the removal of array-specific terms. - **Documentation and configuration**: Updated design docs and removed the configuration flag for skipping array term generation, consolidating the new composite index strategy. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Jayanta Mondal
|
4b419d9b40 |
Merged PR 1671857: Adding support for custom index handler
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) Adding support for custom index handler. ### Description Adding support for custom index handler. Currently the default index access method to index bson data type is "rum". This change allows one to swap out "rum" for something else. The use an alternate index handler one needs to specify two things 1. Firstly, register the index handler and its capabilities and relevant utility call backs. Currently, the system only allows a maximum of 5 alternate index handler. 2. Secondly, set the alternate_index_handler GUC to let the system chose the alternate index handler to index bson. Following are a few notes on the mechanism 1. During index create if the alternate AM does not support certain kind of bson index (e.g, unique index, wildcard index) then the index is created via the default rum index. Otherwise, the alternate AM is chosen. 2. If an index has been created using alternate index AM but then reset the alternate_index_handler GUC or set it to another third alternate index AM, the old alternate index AM will still be in use, and long as both the old and new alternate index AM are in the registry. Other: Fixed Densify and SetWindow fields flakey tests for sharded collection ---- Related work items: #3796361 |
||
|
William Wang
|
fe2f9a7ee7 |
Merged PR 1639930: Add Extensible Support for Role Privileges in UsersInfo Command
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description We've implemented several key improvements to DocumentDB's user management system, with a focus on enhancing the `usersInfo` command to support privilege information display. ### Feature Overview - Added support for the `showPrivileges` flag in `usersInfo` command - When enabled, user information responses now include detailed privilege documents - Privileges are organized by resource and action, similar to MongoDB's privilege format ### Implementation Details 1. **Privilege Data Structures** - Created predefined privilege sets for each role type (readOnly, readWrite, clusterAdmin) - Designed efficient merging of privilege sets with hash-based deduplication 2. **GUC Configuration** - Added `EnableUsersInfoPrivileges` boolean GUC to control feature availability 3. **Command Flow** - Parse `showPrivileges` flag from incoming BSON documents - When flag is true and GUC is enabled, append privilege array to user documents - Privileges include resource targets (cluster/database/collection) and action lists 4. **Memory Optimization** - Implemented hash-based privilege merging to avoid duplicate entries - Proper memory management with PostgreSQL allocation functions This functionality supports the `connectionStatus` command, which requires authenticated user privileges. We will also support `dbowner` and `root` roles, which will come in a follow-up PR. Related work items: #3852887 |
||
|
Deepak Bisht
|
96924b8eee |
Merged PR 1672438: Support Compact in the vCore Backend engine
Yes Feature Yes No - Support compact database administration command behind the GUC `enableCompact`. - Add support for compact command in backend using VACUUM FULL which is a blocking operation. - Only one compact command can be run at a time on a specified collection. - VACUUM FULL is run using libpq in a top level transaction as it it not supported inside functions or procedures. - Add support of tracking compact command in current_op. - `dryRun` estimation is based on the query defined [here at pgexperts](https://github.com/pgexperts/pgx_scripts/blob/master/bloat/table_bloat_check.sql), this can be a rough estimation and doesn't provided details about the space reclaimed by vacuuming RUM indexes of the collection. - Read only users can't perform `compact` command. ---- New feature implementation to support the backend compact command. This PR adds complete backend support for a blocking compact command that estimates table bloat and conditionally performs a VACUUM FULL operation on collection tables. - `src/utils/storage_utils.c` and `include/utils/storage_utils.h`: Introduced a new bloat estimation utility returning collection bloat stats. - `src/commands/compact.c`: Implemented the compact command including BSON command parsing, dry-run handling, and conditional execution of VACUUM FULL. - `internal/pg_documentdb_distributed/src/test/regress/sql/command_compact.sql`: Added regression tests covering various compact command scenarios. - `include/utils/feature_counter.h` and `src/infrastructure/feature_counter.c`: Updated feature counter mapping to include the new compact command feature. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4237921 |
||
|
Vinod Sridharan
|
c8fdba1168 |
Merged PR 1684751: [Composite|Order] Supporting order by on composite index
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description ---- #### AI description (iteration 1) #### PR Classification This pull request introduces a new feature to support “order by” queries on composite indexes and refines how composite query operators are handled. #### PR Summary The changes enable ordering by composite indexes by expanding the $all operator into an AND of expressions, adjusting index pushdown and term generation logic, and enforcing index-only paths when sequential scans are disabled. Key changes include: - **`src/query/query_operator.c`**: Added the `ExpandExprForDollarAll` function to decompose $all into composite AND expressions, and integrated its use controlled by a new feature flag. - **`src/planner/documents_planner.c`**: Introduced helper functions (e.g. `TrimPathListForSeqTypeScans` and `ForceExcludeNonIndexPaths`) to eliminate sequential scan paths when `ForceDisableSeqScan` is enabled, ensuring better index-based execution. - **`src/configs/feature_flag_configs.c` & `src/configs/testing_configs.c`**: Added new configuration flags like `ExpandDollarAllInQueryOperator`, `SkipGeneratingArrayTermForCompositeIndex`, and `ForceDisableSeqScan` to control composite index behavior. - **Composite Index Code (e.g. in `bson_gin_composite_entrypoint.c` and related files)**: Updated index strategy handling and term generation to support composite ordering and correct treatment of array terms. - **Test Files**: Numerous expected output and SQL regression tests (both in the distributed and pgmongo modules) were updated to reflect the new composite index ordering and query operator changes. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Cornelius Boateng
|
11fab5ffc6 |
Merged PR 1683133: [engine] More Feature Counters
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Dev ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add more feature counters for let, cursor types and top-level commands ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature by adding and tracking additional feature counters in the engine. #### PR Summary This change adds new feature counter entries and instrumentation to track usage for several commands and top-level "let" support. The updates include increasing the feature count limit and inserting ReportFeatureUsage calls in relevant command and aggregation pipelines. - `src/infrastructure/feature_counter.c` and `include/utils/feature_counter.h`: Added new feature counter mappings and enums for commands (e.g., agg_cursor_first_page, count, distinct, find_cursor_first_page, get_more, list_collections_cursor_first_page, list_indexes_cursor_first_page) and top-level "let" support. - `src/aggregation/bson_aggregation_pipeline.c`: Integrated ReportFeatureUsage calls for "let" in aggregate and find queries. - `src/commands/aggregation_commands.c`: Inserted ReportFeatureUsage calls in various command functions to track usage of the new feature counters. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4224518, #4228942 |
||
|
Vinod Sridharan
|
d75debabb4 |
Merged PR 1683166: [POC| Orderby] Support order by index pushdown with truncation
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) PoC ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add support for orderby runtime recheck and plumb checks for truncation into the index With this we can have the runtime recheck the orderby and the index handle the base walking of the tree. Additionally, add support for transforms from the index key to the runtime key so that we can pass down the representation. ---- #### AI description (iteration 1) #### PR Classification This pull request introduces a new feature by adding support for order-by index pushdown with truncation. #### PR Summary The changes overhaul the ordering and scan logic in the RUM index to correctly handle pushdown of order-by operations when truncation occurs. - In `pgmongo_rum/src/rumget.c`, scan functions were refactored to incorporate new recheck logic and a helper (`PrepareOrderedMatchedEntry`) for processing ordered entries. - In `oss/pg_documentdb/src/opclass/bson_gin_composite_entrypoint.c`, a new ordering transform function (`gin_bson_composite_ordering_transform`) was added and comparison logic modified to signal truncation. - In `pgmongo_rum/src/rumscan.c` and `pgmongo_rum/src/rumvalidate.c`, the handling and validation of ordering keys were updated to support raw ordering keys with new function signatures. - In regression test files under `/pgmongo/src/test/regress`, tests were enhanced to verify ordering with truncation. - In `pgmongo_rum/src/rum.h`, default settings and structure fields were updated to enable order-by raw keys and track recheck status for ordering. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Cornelius Boateng
|
addf8f18e0 |
Merged PR 1592678: [engine][let] Support for delete
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support `let` variables and `$$NOW` in delete queries.
The values of the variables can be accessed in the query by using `$expr`.
Example:
```
{
delete: <collection>,
deletes: [
{
q: {$expr: {$eq: ["$_id", "$$varRef"]}},
limit: 0, // multi
}
],
let: {"varRef": 1},
...
}
```
We parse the `let` value (together with the time system variables - `$$NOW`) and propagate it through the aggregation and variable contexts.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request adds support for the `let` clause in delete operations.
- `delete.c`: Implemented logic to handle `let` clause in delete queries, including variable specification and collation.
- `bson_aggregation_pipeline_tests_let.out`: Added extensive test cases to validate the new `let` clause functionality in delete operations.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #3423118, #3613877
|
||
|
Zinan Ren
|
f4896e9c81 |
Merged PR 1670376: [ReleaseValidation] Add release validation tests for hnsw/diskann/filtering/compression
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Testing ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description 1. Check the max dimensions when creating the vector index, previously checked by pgvector 2. Add the following release validation tests: - HNSW index (M40 and above) - DiskANN index (M30 and above) - Filtering search (GA) - Exact search (GA) - Compression half (GA) - Compression PQ (GA) ---- #### AI description (iteration 1) #### PR Classification This pull request implements new vector search release validation tests and updates the backend’s vector index dimension checks and error messaging. #### PR Summary The changes add comprehensive release validation tests for vector search covering various compression types (PQ, half), index kinds (ivf, hnsw, diskann), filtering, and exact match scenarios. - `outerLoop/releaseValidation/testSpecs/` now includes multiple new JSON spec files (e.g. ValidateVectorSearchCompressionPQ.json, ValidateVectorSearchCompressionHalf.json, ValidateVectorSearchDiskANN.json, ValidateVectorSearchHNSW.json, ValidateVectorSearchFiltering.json, and ValidateVectorSearchExact.json) with updated expected errors. - Test scripts in `ReleaseValidation.Tests.NodeJs/test/test_signoffTestsNodeJS.js` and C# signoff test file `SignoffTestsCSharp.cs` have been modified to enable and incorporate these new vector search tests. - In `pg_documentdb/src/commands/create_indexes.c`, additional checks enforce maximum dimensions based on compression type (non-compressed: 2000, half: 4000, PQ: as defined) and update error messages accordingly. - Several Postgres gateway and pgmongo files have been updated to remove obsolete regexes and return revised error codes (e.g. error code 67) and messages for exceeding vector dimension limits. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
7658398f9b |
Merged PR 1679366: [POC] Persisted cursors files without holding connections
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) PoC/Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Today, persisted cursors require the connections to be held. This is bad since Connections/Backends are a limited resource. This introduces the option to save the cursor state in a file on disk and reuse that across backends. This is behind a GUC due to the presence of the following: 1) Needs more testing and validation 2) Needs backpressure to ensure that disk consumption doesn't balloon up 3) There is a procedure added to clean up these files but it is not wired up 4) We need to handle ultra-large responses (beyond 4 GB it fails outright at the moment). This needs more graceful handling to split into chunks. ---- #### AI description (iteration 1) #### PR Classification This pull request is a new feature implementation that introduces file‐based persisted cursors to allow query results to be “drained” and stored on disk without holding on to active database connections. #### PR Summary This PR adds support for persisting cursor state to the filesystem in both the pg_documentdb and pgmongo modules, enabling connections to be released while still allowing cursors to be later resumed. The changes include: - **`infrastructure/cursor_store.c` and `cursor_store.h`** – New APIs for creating, writing to, reading from, and closing persisted cursor files. - **Commands integration** – Modifications in `commands/cursors.c`, `commands/aggregation_commands.c`, and `cursor_private.h` to use file-based cursor functions for draining and continuation. - **SQL test suites** – Addition of new test files (e.g. in `.../regress/sql/bson_aggregation_cursor_tests_file.sql`) to validate cursor behavior across various query scenarios. - **Configuration updates** – New custom GUCs in `feature_flag_configs.c` and adjustments in test PostgreSQL configuration files to enable and control file-based persisted cursors. - **Executor enhancements** – Updates in cursor draining and query plan execution functions to support both in-memory and file-based cursor paging. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Santiago Fernandez Madero
|
0d616cdcde |
Merged PR 1679451: Make first page cursor batch size configurable via GUC
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Make first page cursor batch size configurable via GUC ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature by introducing a configurable GUC for the first page cursor batch size. #### PR Summary The changes enable users to control the default batch size for the first page of cursors through a custom GUC, ensuring it respects underlying limitations. - `src/configs/system_configs.c`: Added a custom integer variable (`defaultCursorFirstPageBatchSize`) with a default value and proper bounds via the GUC interface. - `src/commands/aggregation_commands.c`: Updated the first page query data generation to use the new configurable variable instead of a hardcoded value. - Test files under `/oss/pg_documentdb/src/test/regress/`: Extended both SQL and expected output tests to verify the behavior of the GUC under various configurations. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4225119 |
||
|
Cornelius Boateng
|
b204b58b82 |
Merged PR 1619783: [engine][collation] Support for sort in runtime
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes, yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Support sorting with collation. We define new custom sort operators which compare operands using applicable collation string. `<<<` is for ascending order. `>>>` is for descending order. Example: `SELECT * from <collection> ORDER BY <value> USING <<<` This will further be used for partitioning, grouping, and related comparisons. ---- #### AI description (iteration 1) #### PR Classification New feature: Support for collation in sort operations. #### PR Summary This pull request adds support for collation in sort operations within the database engine. - `src/test/regress/expected/bson_dollar_ops_collation_tests_runtime.out`: Added multiple test cases to verify collation behavior with various sort and order by operations, including case sensitivity and numeric ordering. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4014123 |
||
|
Vinod Sridharan
|
8261e772f4 |
Merged PR 1672946: [Update] Make update parsing cheaper
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Perf ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Currently for Update/Delete, we currently parse the update spec, query spec, etc into pgbsons on startup. Then reuse it per commands. This has a few issues - for large document requests, this causes us to duplicate memory and persist it for the duration of the update and also transcode all up front - this is worse for multi-node when all we need to do is build yet another update spec and push it to the worker. Instead, we now only cache pointers to the spec document; then when we need it we convert to pgbson only if necessary. A consequence of this is that on a single node update, if we're doing a simple updateOne on _id then we would never serialize the updateSpec at all (for instance). We also more eagerly free memory on the intermediate update specs. ---- #### AI description (iteration 1) #### PR Classification Performance optimization for update parsing by refactoring BSON value handling. #### PR Summary This PR streamlines update parsing by replacing heavy BSON initialization routines with lightweight wrappers and converting update-related types from `pgbson*` to `bson_value_t*`, which reduces overhead and improves coordinator/worker compatibility. - `src/commands/update.c`: Replaced multiple `PgbsonInitFromDocumentBsonValue` and similar calls with efficient wrappers like `CreateBsonValueFrom` and updated serialization (including back-compatible array filter and update document formats). - `include/commands/update.h`: Updated update structure fields to use `bson_value_t*` instead of `pgbson*`, aligning interfaces with the new parsing approach. - `src/update/bson_update.c` and `bson_update_operators_workflow.c`: Refactored update type determination and replace document logic to use new BSON value iterators and helper functions. - Other modules (e.g. `commands/find_and_modify.c`, `commands/delete.c`, query operators, sharding) were updated to leverage the lighter-weight BSON representation and its associated helper routines. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Jayanta Mondal
|
fa75e56ac1 |
Merged PR 1669686: Adding feature counter for _internalInhibitOptimization
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description more ---- #### AI description (iteration 1) #### PR Classification This PR introduces a new feature by adding support for tracking the internal inhibit optimization feature usage. #### PR Summary The changes enhance the feature counter system to monitor the `_internalInhibitOptimization` usage in aggregation pipelines and SQL tests. - `mongo_feature_counters.sql`: Added new SQL queries to test and display the feature counter for `_internalInhibitOptimization`. - `feature_counter.h`: Added a new enum constant, `FEATURE_STAGE_INTERNAL_INHIBIT_OPTIMIZATION`, for the new feature. - `bson_aggregation_nested_pipeline.c`: Updated the internal optimization handler to log feature usage. - `feature_counter.c`: Extended the feature mapping to include `_internalInhibitOptimization`. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Zinan Ren
|
4b79ce38d4 |
Merged PR 1668682: [Vector Search] Add GUC for the default search parameter calculation
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Perf ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Adding the GUC documentdb.enableVectorCalculateDefaultSearchParam to enable or disable the calculation of default search parameters. The default value is on Adding feature counts for the usage of default search parameters. ---- #### AI description (iteration 1) #### PR Classification This PR implements a new configuration option to enable or disable the dynamic calculation of default search parameters for vector search indexes. #### PR Summary The pull request adds a GUC flag that controls whether default search parameters (e.g., nProbes for IVF and efSearch for HNSW) are computed dynamically based on collection size. It updates internal calculation functions, test expectations, and feature counters accordingly. - `src/vector/vector_index_kind_impl.c`: Revised the IVF and HNSW search parameter calculation functions to conditionally compute default parameters when the new GUC is enabled. - `src/configs/feature_flag_configs.c` and `include/vector/vector_common.h`: Introduced the new boolean GUC `enableVectorCalculateDefaultSearchParam`. - Test files (e.g., files under `/oss/internal/pg_documentdb_distributed/src/test/regress/expected/` and corresponding `pgmongo` tests): Updated expected outputs and query plans to reflect the new default parameter values. - Feature counter outputs and related SQL tests: Modified to include metrics for the new default search parameter usage. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4194082 |
||
|
Vinod Sridharan
|
c5bbda3241 |
Merged PR 1668237: [Livesite] Handle default_readonly_transaction more gracefully
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Bugfix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description When servers have default_readonly_transaction set, the current behavior is wonky - it will cause failures in underlying queries and pollute the errors, and it will go through a bunch of work only to fail. Additionally, background jobs like TTL will fail ungracefully. Handle this GUC by failing write commands early with a more useful error message. Additionally, in cases like TTL, add an opt-in way to prune rows even in the presence of this GUC if it's opted into. ---- #### AI description (iteration 1) #### PR Classification This pull request is a bug fix that improves error handling for write operations attempted during default read-only transactions. #### PR Summary The changes introduce a unified mechanism to check and enforce read-only transaction restrictions, ensuring graceful errors when write operations are attempted while in a read-only state. Key changes include: - **`include/commands/commands_common.h`**: Added the inline function `ThrowIfServerOrTransactionReadOnly` to validate transaction state and throw appropriate errors. - **Test files**: New regression tests under `/oss/pg_documentdb/src/test/regress/expected/` and `/oss/pg_documentdb/src/test/regress/sql/readonly_transaction_tests.sql` validate the expected error responses under read-only conditions. - **`ttl_index.c`**: Modified TTL job logic to handle read-only transactions, allowing jobs to commit and restart when the corresponding flag is enabled. - **Gateway and error handling**: Updated configuration in `TenantConfigurationBase.cs`, error augmentation in `PostgresMongoResultExtensions.cs`, and error mappings (e.g., adding error code 10107 for `NotWritablePrimary`) to propagate clear error messages. - **Command modules**: Added calls to `ThrowIfServerOrTransactionReadOnly` in several command implementations (e.g., in `insert.c`, `update.c`, `delete.c`, `create_indexes.c`) to enforce read-only restrictions uniformly. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Deepak Bisht
|
8b7b93ced7 |
Merged PR 1578129: [Pending release cut 1.102]- Fix projection with bson_dollar_merge_documents for multiple scrnarios
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) BugFix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) Bugfix to override nested arrays in dotted path in `bson_dollar_merge_documents` UDF for few cases ### Description bson_dollar_merge_documents function today recurses into nested arrays always and project in each element of array but that is not correct with mongo semantics. This is only applicable if the projectetion is a dotted path. The function is used in 7 different stages/features today: 1- list_collection: no dotted paths for projection, so this is okay to follow any projection. 2- graphlookup: Mongo does a shallow projection in this case. 3- lookup: Mongo does a shallow projection in this case. 4- inverseMatch: okay as no dotted path projection 5- count: okay as no dotted path projection 6- group : okay as no dotted paths are allowed 7- setWindowFields: only this stage does a full projection even inside nested arrays ---- #### AI description (iteration 1) #### PR Classification Bug fix #### PR Summary This pull request addresses multiple scenarios for fixing projection with `bson_dollar_merge_documents`. - Added new functions in `oss/pg_documentdb/sql/udfs/projection/bson_projection--0.102-0.sql` to handle various BSON document operations. - Updated `bson_dollar_merge_documents` function to include an `overrideArray` parameter. - Modified test expectations in `oss/internal/pg_documentdb_distributed/src/test/regress/expected/bson_aggregation_pipeline_tests_graphlookup.out` to reflect changes in the `bson_dollar_merge_documents` function. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Amit Prabhudesai
|
84ba0c99d8 |
Merged PR 1572685: [OSS] Fix background index builds for documentdb
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Bugfix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Get background index builds to work for single-node documentdb. This changeset fixes background index builds for OSS by: 1. Scheduling the background index build tasks 2. Moving the distributed functionality (which had not being refactored out of the indexing code during the move to OSS) into distributed hooks, with the base implementation for single-node. 3. Refactoring SQL tests to get even with the above changes. ---- #### AI description (iteration 1) #### PR Classification Bug fix for background index builds in DocumentDB. #### PR Summary This pull request addresses issues with background index builds in DocumentDB by adding comprehensive tests and handling various edge cases. - `commands_create_indexes_background.out`: Added extensive test cases to validate the behavior of `create_indexes_background` function, including handling of invalid inputs, unique constraints, and various index specifications. - Introduced a new schema and function to manage index job statuses. - Enhanced error handling and validation for index creation requests, ensuring proper error messages and statuses are returned for invalid operations. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
1153e6b6ac |
Merged PR 1661728: [Performance] Read & Query Perf Wave 2
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Perf ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description 1) In the Gateway, when preparing queries, embed the commandKey and userOid. That way a given prepared query is unique for a given user and command. This can be used for a future optimization to cache ACL check results 2) Reduce serialization costs from GW -> BE: Don't send the database name in the query. This is already in the request as a '$db' field. Support suppressing this (add a GUC to rollback just in case) 3) In the backend start parsing the $db if it's not sent by the GW 4) From the commandFuncs -> Query we were going through a layer of fmgr. We don't need this. Instead call directly into the functions in C to avoid the fmgr overhead. Additionally for GetMore this means we don't need to repack the heap tuple. As part of this we discovered that registering functions in pg_dist_object can regress perf for dispatch functions. Add a test to validate which ones of these are public. ---- #### AI description (iteration 1) #### PR Classification Performance improvement. #### PR Summary This pull request focuses on enhancing the performance of read and query operations by optimizing function calls and parameter handling. - Refactored `ExecuteQueryCore` in `command_funcs.c` to improve query execution by directly calling specific functions and adding ACL checks. - Changed function signatures in `bson_aggregation_pipeline.h` to use `text` instead of `Datum` for database parameters, improving type consistency. - Added `aggregation_commands.h` to export new aggregation command functions. - Updated `PostgresClient.cs` to conditionally include database names in query parameters, enhancing query flexibility. - Introduced new configuration option `SendDatabaseNameInParams` in `PostgresTenantConfiguration.cs` to control database name parameterization. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
58d0dfae08 |
Merged PR 1660651: [Perf] Miscellaneous Read Perf Improvements
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Perf ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description A few miscellaneous perf changes focused on Point reads 1) When parsing the find spec, we do a bunch of if/elses which means we're doing 10ish ifs for the keys that are lower in the if. Use a switch and dispatch on the first character to speed this up 2) When doing a point read and local execution, we can skip setting up yet another Portal and Transaction since we're already in a func context and we can leverage the same snapshot for the reads. Also we don't need to hold a SPI Destination and copy it to the target pgbson_array_writer, use a custom DestReceiver that takes hte Tuple and directly writes it to the target. 3) Cleanup the Gateway Dispatch functions and remove an unnecessary PG_TRY/PG_FINALLY (this is already handled by PG). ---- #### AI description (iteration 1) #### PR Classification Performance improvement. #### PR Summary This pull request introduces various performance enhancements to the read operations in the codebase, focusing on optimizing BSON aggregation and cursor handling. - `bson_aggregation_pipeline.c`: Refactored the `GenerateFindQuery` function to use a switch-case structure for improved performance and readability. - `cursors.c`: Added new functions `DrainStatementViaExecutor` and `DrainStatementViaPortal` to optimize query execution and result handling. - `command_funcs.c`: Simplified argument handling in `HelioExternalFunctionCall` and introduced `DoAclCheck` for access control. - `PostgresMongoCursorBuilder.BackendCursor.cs`: Simplified cursor page statement generation by removing unnecessary parameters. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Cornelius Boateng
|
f6d20fbf8a |
Merged PR 1646920: [engine][collation] Support for Arrays and Documents
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Support collation with comparison of `array` and `document` values. Also adds support for `$in` on nested arrays and documents. ---- #### AI description (iteration 1) #### PR Classification New feature: Adds support for collation-aware hashing of arrays and documents in BSON. #### PR Summary This pull request introduces collation-aware hashing for BSON arrays and documents, enhancing the handling of collation in hash sets. It refactors existing hash functions to accommodate collation strings and recursive hashing of nested BSON structures. - `hashset_utils.c`: Implements collation-aware hashing functions for BSON values, including recursive handling of arrays and documents. - `bson_expression_set_operators.c`: Replaces existing hash set creation with collation-aware hash set creation. - `bson_dollar_operators.c`: Updates logic to use collation-aware hash sets for `$in` queries, removing unsupported scenarios. - `hashset_utils.h`: Adds a new struct `BsonValueHashEntry` to store BSON values with collation strings. - `collation.h`: Moves `IsBsonTypeCollationAware` function to handle collation applicability checks. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4111861, #4115239 |
||
|
Vinod Sridharan
|
efa2b802f6 |
Merged PR 1659003: [Composite OpClass] Enable Runtime Functions and add test coverage
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature/POC ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Fix up runtime index recheck function support to reduce loading documents. This should reduce the number of runtime rechecks that are needed for documents and trust the index more as we will evaluate these clauses directly on the index first. Also add option to trigger jstest coverage for the new opclass - and set them to run on the MX jstests - this is still disabled right now since there are some failing tests but will be enabled separately. ---- #### AI description (iteration 1) #### PR Classification New feature: This pull request introduces runtime functions for composite index operations and enhances test coverage. #### PR Summary This pull request enables runtime functions for composite index operations, improving the flexibility and efficiency of index handling. It also adds test coverage to ensure the reliability of these new features. - `bson_gin_composite_entrypoint.c`: Refactored comparison logic to support runtime functions for index rechecks. - `bson_gin_composite_core.c`: Implemented `IsValidForIndexValue` function to handle various BSON index strategies with runtime rechecks. - `create_indexes.c`: Modified index creation logic to conditionally enable new composite index opclass based on configuration. - `bson_gin_composite_private.h`: Added `IndexRecheckArgs` structure to support runtime recheck strategies. - `testing_configs.c`: Introduced configuration options for enabling the new composite index opclass by default. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Deepak Bisht
|
cd19288dbc |
Merged PR 1647200: [Perf]- Point read improvement, remove PG inference of type and create the tuple descriptor directly
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Perf ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description - add cursor result type for UDFs, this helps PG to not always try to figure out the `OUT` variables and form a tuple descriptor  ---- #### AI description (iteration 1) #### PR Classification Performance improvement and API change. #### PR Summary This pull request introduces performance improvements for point reads by adding a new cursor type for SQL functions. It modifies existing functions to return a structured type instead of a record, enhancing efficiency and clarity. - `query_cursors_aggregate--latest.sql`: Added `cursor_result` type and updated several functions to return this type instead of a record. - `query_cursors_aggregate--0.103-0.sql`: Newly added file with similar changes as `query_cursors_aggregate--latest.sql`. - `documentdb--0.102-1--0.103-0.sql` and `pgmongo--1.102-1--1.103-0.sql`: Updated to include the new `query_cursors_aggregate--0.103-0.sql` file. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4157027 |
||
|
Zinan Ren
|
ac6f028c86 |
Merged PR 1653287: [Vector Search] Enable iterative index scans for HNSW and IVF in filtering searches
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
With the release of [pgvector 0.8.0](https://github.com/pgvector/pgvector/tree/5bc7937715c67add21c8bcc0d4284162c7a0174f?tab=readme-ov-file#iterative-index-scans), which supports iterative scan for HNSW and IVF, we enable it in filtering search to support pre-filtering(currently it is post-filtering)
- Pass the new parameter "iterativeScan" through the function bson_search_param from the coordinator to the worker.
Example : { "efSearch" : 10, "iterativeScan" : "relaxed_order"}
- Enable iterative scan for hnsw/ivf index on workers:
SET LOCAL hnsw.iterative_scan = relaxed_order;
SET LOCAL ivfflat.iterative_scan = relaxed_order;
relaxed_order by default, which allows results to be slightly out of order by distance, but provides better recall.
After iterative search, re-ordering will be applied to the results.
- Set iterative scan for diskann index to relaxed_order
SET LOCAL diskann.iterative_search = relaxed_order;
- Check the availability of iterative scan in pgvector
The pgvector iterative scan is available from version 0.8.0. Currently, if the pgvector extension has not been upgraded to 0.8.0, the iterative scan will be disabled in those clusters until pgvector is upgraded to 0.8.0.
###Related GUCs
**documentdb.vectorPreFilterIterativeScanMode**: enable iterative index scans, which will automatically scan more of the index until enough results are found (relaxed_order by default)
- off, disable iterative scan
- relaxed_order, allows results to be slightly out of order by distance, but provides better recall
- strict_order, ensures results are in the exact order by distance
####GUCs from pgvector
**hnsw.max_scan_tuples**: Specify the max number of tuples to visit (20,000 by default)
**hnsw.scan_mem_multiplier**: Specify the max amount of memory to use, as a multiple of work_mem (1 by default)
**ivfflat.max_probes**: Specify the max number of probes(32768 by default)
###Mongo Command(no change in this PR)
```
db.testing_hnsw_vector_collection.aggregate(
[
{
"$search": {
"cosmosSearch": {
"vector": [ 3.0, 4.9, 1.0 ],
"path": "vect",
"filter": {"meta.a": [ { "b" : 3 } ]},
"k": 4,
"efSearch": 100
},
}
},
{
"$project": {
"rank": {
"$floor": [{"$multiply": [{ "$meta": "searchScore"},100000]}]
}
}
}
])
```
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces iterative index scans for HNSW and IVF in filtering searches, enhancing the pre-filtering capabilities of vector searches...
|
||
 Jinrui Zhang
|
356361bceb |
Merged PR 1628037: Add string case support for '$toDate' operator
### Does this PR have any customer impact? Yes, by this PR, $toDate could process string case ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Enhancement for exsiting operator ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) N/A ### Are you introducing any new config? If yes, do you have tests with and without them being set? N/A ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) Add string case support for '$toDate' operator ### Description Before this PR, `$toDate` could not parse a string to datetime. With this PR, we leverage the ability of `$dateFromString` to finish the support of string case for `$toDate` operator ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request adds support for string case conversion in the `$toDate` operator. - Added `StringToDateWithDefaultFormat` function in `bson_expression_date_operators.c` to handle string to date conversion. - Updated `ProcessDollarToDate` in `bson_expression_type_operators.c` to utilize the new string conversion function. - Added tests for string case conversion in `$toDate` operator in `bson_aggregation_type_operators_tests.sql`. - Updated documentation and release notes to reflect the new feature. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Zinan Ren
|
e05e91ba51 |
Merged PR 1650017: [Vector Search] Add support for PQ compression in the Diskann index
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Adding support Product Quantization (PQ) compression in vector index, which is a popular method for dramatically compressing high-dimensional vectors to use less memory, and for making nearest-neighbor search faster.
Note: currently, this PQ compression is only supported by the Diskann index.
### Related GUCs
**documentdb.enableVectorCompressionPQ**: Enables vector PQ compression (enabled by default).
### Create index spec:
- **compression**(optional):
- "half": half-precision vector, which using 16 bits floats.
- "pq": product quantization compression.
- **pqCompressedDims**(optional): indicates the dimensions after compression. If not specified, it will be automatically calculated based on the original dimensions.
- Minimum: 1
- Maximum: 8000
- **pqSampleSize**(optional):Indicates the number of samples PQ uses to generate the initial centroids. The default value is 1000.
- Minimum: 1000
- Maximum: 100000
**Note**: It is recommended to create the index after the collection has some data.. If the collection is empty, random vectors within the range [-1, 1] are used to generate the initial centroids and create the index. If the number of documents is less than pqSampleSize, the training data is padded with random data within the range [min, max] from the sampled vector data.
### Command example
```
db.runCommand(
{ "createIndexes": "testing_vector_collection", "indexes":
[{
"key": { "v": "cosmosSearch" },
"name": "hsnw_index",
"cosmosSearchOptions":
{
"kind": "vector-diskann",
"similarity": "L2",
"dimensions": 1536,
"compression": "pq",
"pqCompressedDims": 96,
"pqSampleSize": 2000
}
}]
} )
```
### Search spec:
No changes
----
#### AI description (iteration 1)
#### PR Classification
New feature: This pull request adds support for Product Quantization (PQ) compression in the Diskann index.
#### PR Summary
This pull request introduces support for PQ compression in the Diskann index, enhancing vector search capabilities. It includes error handling for unsupported compression types and validation for PQ parameters.
- Updated `/pgmongo/src/test/regress/expected/bson_aggregation_pipeline_tests_vector_diskann.out` to include tests for PQ compression, error cases, and validation of PQ parameters such as `pqCompressedDims` and `pqSampleSize`.
- Added tests for vector search with PQ compression, including L2, cosine, and IP similarity measures.
- Implemented checks for maximum dimensions and resource limits during PQ index creation.
<!-- GitOpsUserAgent=GitOps.Apps.S...
|
||
|
William Wang
|
b2624497d9 |
Merged PR 1633683: Add Support for Username/Password Constraint Validation for DataPlane User CRUD with Extensibility Hooks
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description We are adding the following constraints for username and password: **Username** - The username length must be greater than 0 and less than or equal to 63 characters. **Password** - The password length must be between 8 and 256 characters (inclusive). - The password must satisfy at least 3 of the following 4 conditions: - Contains at least one lowercase letter. - Contains at least one uppercase letter. - Contains at least one digit. - Contains at least one special (non-alphanumeric) character. - The password must not contain any 3-character substring of the username (case-insensitive). Related work items: #3916282 |
||
|
Parag Jain
|
833c4677af |
Merged PR 1655313: [Cleanup] Removing Insert dead code for old implementation of $out
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Removing Insert dead code ---- #### AI description (iteration 1) #### PR Classification Code cleanup #### PR Summary This pull request removes dead code related to the `TryInsertOne` function, which is no longer needed in the codebase. - Removed the `TryInsertOne` function from `src/commands/insert.c`. - Deleted the `TryInsertOne` function declaration from `include/commands/insert.h`. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
5a19661e8b |
Merged PR 1654645: [Feature Flags] Refactoring features and add some tracking
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactor ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description - Add a new section for tracking feature flags - Add a feature counter to track customers whoc an benefit from the new rum_index_scan. ---- #### AI description (iteration 1) #### PR Classification Refactoring and feature enhancement with added tracking capabilities. #### PR Summary This pull request refactors feature flags and introduces new tracking capabilities for feature usage. It reorganizes the feature flag sections and adds new flags for various functionalities. - Refactored and added new feature flags in `feature_flag_configs.c`, including indexing, planner, and aggregation features. - Updated `custom_scan_continuation.c` to improve index scan handling and added feature usage reporting. - Modified `insert.c` to handle snapshot management more effectively. - Enhanced feature usage tracking in `feature_counter.c` and `feature_counter.h` by adding a new feature counter for index scans with limits. - Updated test expectations in multiple files to reflect the new feature tracking changes. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Parag Jain
|
4ed646dfb9 |
Merged PR 1647233: [Bulk Insert][Perf] support for bulk insert via procedures
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Currently bulk insert run as a function which means that the entire execution runs as a single transaction. Each batch insert is handled as a subtransaction and rolled back to a save point if it fails. This means that document locks are held for longer, and we can hit limits w.r.t. max documents updated per transaction. In multi-node setups this also means distributed transactions are held longer causing loss of connection pools. This change introduces a stored procedure for bulk insert that periodically "Commits" the state to WAL and starts a new transaction. This more mirrors what the non-atomic bulk update is supposed to do. The old function is retained to be used within transactions. ---- #### AI description (iteration 1) #### PR Classification Performance improvement for write commands. #### PR Summary This pull request introduces a new procedure for handling write commands, enhancing performance by optimizing user-specific command execution. It includes the addition of new functions and procedures to manage MongoDB operations more efficiently. - Added new SQL functions and procedures in `/pgmongo/sql/udfs/gateway_funcs/command_funcs--1.103-0.sql` for running commands, queries, DDL, and write operations as a specific user. - Introduced new insert functions and procedures in `/oss/pg_documentdb/sql/udfs/commands_crud/insert--0.103-0.sql` to handle MongoDB insert operations. - Implemented `pgmongo_run_write_procedure_as_user` and `ExecuteWriteCommandCore` in `/pgmongo/src/infrastructure/command_funcs.c` to manage write commands with user context. - Added `CommitOuterInsertTransactions` function in `/oss/pg_documentdb/src/commands/insert.c` to handle transaction commits outside of transaction blocks. - Introduced a feature flag `EnableWriteProcedure` in `/oss/pg_documentdb/src/configs/feature_flag_configs.c` to control the enabling of write procedures. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Cornelius Boateng
|
bec1aa3562 |
Merged PR 1638286: [engine][collation][enhancement] Extend bson_expression_get for collation support
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add a new bson_expression_get that appends the collationString to the got value, if a valid collationString is passed to it. This will be used as a helper function to operations that rely on bson_expression_get to pass down collation such as bson_lookup_join_filter. If no collation string is passed as an arg, we simply return the got value. ---- #### AI description (iteration 1) #### PR Classification New feature to support `$graphLookup` in the engine with collation. #### PR Summary This pull request adds support for `$graphLookup` with collation in the engine. - Added new functions in `oss/pg_documentdb/sql/udfs/aggregation/bson_lookup_functions--0.103-0.sql` for `$graphLookup` operations. - Modified `oss/pg_documentdb/src/aggregation/bson_aggregation_nested_pipeline.c` to handle collation in `$graphLookup` join filters. - Updated `oss/pg_documentdb/src/operators/bson_expression.c` to remove collation handling from `bson_expression_get`. - Added `BsonDollarLookupJoinFilterWithCollationFunctionOid` in `oss/pg_documentdb/src/metadata/metadata_cache.c`. - Adjusted collation handling in `oss/pg_documentdb/src/query/bson_dollar_operators.c` for `$graphLookup`. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4104899 |
||
|
Vinod Sridharan
|
36bb3e2920 |
Merged PR 1650400: [Bulk Update] Setup bulk update infrastructure
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Currently bulk updates run as a function which means that the entire execution runs as a single transaction. Each batch update is handled as a subtransaction and rolled back to a save point if it fails. This means that document locks are held for longer, and we can hit limits w.r.t. max documents updated per transaction. In multi-node setups this also means distributed transactions are held longer causing loss of connection pools. This change introduces a stored procedure for bulk update that periodically "Commits" the state to WAL and starts a new transaction. This more mirrors what the non-atomic bulk update is supposed to do. The old function is retained to be used within transactions. Still TODO: For unsharded collections in a multi-node setup, we need to send batches of updates to the worker. This is not wired up in this change. ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request introduces a new bulk update feature for handling non-transactional updates in the MongoDB wire-protocol. It adds new functions and procedures to support bulk updates and modifies existing update logic to accommodate this new functionality. - Added `update_bulk` procedure and related functions in `sql/udfs/commands_crud/update--0.103-0.sql` for non-transactional bulk updates. - Implemented `command_update_bulk` in `src/commands/update.c` to handle bulk updates outside of transactions. - Enhanced `ProcessBatchUpdateCore` in `src/commands/update.c` to support transactional and non-transactional updates. - Updated feature tracking in `include/utils/feature_counter.h` and `src/infrastructure/feature_counter.c` to include `FEATURE_COMMAND_UPDATE_BULK`. - Modified SQL scripts `documentdb--0.102-1--0.103-0.sql` and `pgmongo--1.102-1--1.103-0.sql` to include the new update functionality. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
71f084dca8 |
Merged PR 1644618: [Index] Improve wildcard index performance
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Improve index build and insert performance (especially for wildcard indexes).
The current wildcard indexes are heavily bloated - for 50 GB of data that is very array/document heavy, we could end up producing a 300 GB index in some cases.
To improve this, there's considerations of parallel index builds (introduced in PG17+), but also there's something to be said about reducing index bloat. This PR does that in two ways:
1) Given that index truncation happens at ~2500 bytes (or less), the possibility of compression for index terms is low. Therefore, even if the primary document is compressable, the index isn't. This introduces compression for index entries with a custom GUC that manages that. Especially for larger arrays and documents this yields a non-trivial saving.
2) There's a realization that with wildcard indexes and arrays, there's term duplication: An path `"a": [ 1, 2, 3 ]` generates
```
{ "a": [ 1, 2, 3 ] }
{ "a": 1 }
{ "a": 2 }
{ "a": 3 }
{ "a.0": 1 }
{ "a.1": 2 }
{ "a.2": 3 }
```
where the last 3 terms can be made redundant with the first 3. this attempts to remove those extra terms by now only generating:
```
{ "a": [ 1, 2, 3 ] }
{ "a": 1 }
{ "a": 2 }
{ "a": 3 }
```
Query plans that query a.0 will not be pushed to this index currently.
However, in the future this can be removed by querying a.0 as 'a' and running a runtime recheck.
Given that this only impacts queries that explicitly query by integer keys, and on wildcard indexes, this is deemed acceptable for the improvement in size reduction on the index.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces support for wildcard terms in the indexing system, enhancing query capabilities and performance.
- Refactored `SerializeBsonIndexTermCore` in `gin_index_term.c` to use a new helper function `BuildSerializedIndexTerm`.
- Added `SerializeBsonIndexTermWithCompression` function in `gin_index_term.c` for handling compressed index terms.
- Implemented `QueryPathHasDigits` and `FailIfQueryPathHasDigitsForWildcard` functions in `bson_gin_entrypoint.c` to validate query paths for wildcard indexes.
- Updated various query extraction functions in `bson_gin_core.c` to use `filterElement` from `BsonExtractQueryArgs` instead of directly accessing the query.
- Enhanced `GenerateIndexExprStr` in `create_indexes.c` to include options for reduced wildcard terms and generating not-found terms.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
|
||
|
Zinan Ren
|
591a38b86d |
Merged PR 1606660: [Vector Search] Support half-precision vector indexing
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description With the release of pgvector 0.7.0, which supports [Half-Precision Indexing](https://github.com/pgvector/pgvector?tab=readme-ov-file#half-precision-indexing), vectors can have up to 4,000 dimensions ###Related GUCs **documentdb.enableVectorCompressionHalf**: enable vector compression(enabled by default) - off, disable vector compression - on, enable vector compression ###New parameters - Create index spec: "compression": string "half": half-precision vector, which using 16 bits float - Search spec: "oversampling": double Oversampling factor: Oversampling retrieval from half index to compensate for the loss of precision. Minimum value is 1. This parameter is only permitted when a compression method is used on the vector index. For example, if oversampling=1.5 and k=10, then oversampling*k=15 vectors from a compressed index. Then, the full 32-bits vector will be used for the subsequent ordering and scoring, ###Mongo Command ``` db.runCommand( { "createIndexes": "testing_vector_collection", "indexes": [{ "key": { "v": "cosmosSearch" }, "name": "hsnw_index", "cosmosSearchOptions": { "kind": "vector-hnsw", "similarity": "L2", "dimensions": 3, "compression": "half" } }] } ) ``` ``` db.testing_hnsw_vector_collection.aggregate( [ { "$search": { "cosmosSearch": { "vector": [ 3.0, 4.9, 1.0 ], "path": "vect", "filter": {"meta.a": [ { "b" : 3 } ]}, "k": 4, "efSearch": 100, "oversampling": 2.0 }, } }, { "$project": { "rank": { "$floor": [{"$multiply": [{ "$meta": "searchScore"},100000]}] } } } ]) ``` ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request adds support for half-precision vector indexing in the vector search functionality. - `metadata_cache.c`: Added functions for half-precision vector similarity operators and functions. - `SearchTests.cs`: Added integration tests to validate half-precision vector search, including error cases and index creation scenarios. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
