mirror of
https://github.com/FerretDB/documentdb.git
synced 2026-11-06 19:12:50 -05:00
main
41
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Deepak Bisht
|
1a695bdf9f |
Merged PR 1811981: [Observability] Add feature counters for all query operators
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Observability ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add feature counters for all public query operators ---- #### AI description (iteration 1) #### PR Classification This PR implements a new feature by adding comprehensive feature counters for all query operators to enhance observability. #### PR Summary The changes extend the observability framework by introducing new feature counter definitions, mappings, and usage in query operator implementations. - Updated `oss/pg_documentdb/include/utils/feature_counter.h` and `oss/pg_documentdb/src/infrastructure/feature_counter.c` to add new query operator feature enum values and corresponding string mappings with an increased `MAX_FEATURE_COUNT`. - Modified `oss/pg_documentdb/src/planner/mongo_query_operator.c` to assign the new feature counters to each query operator. - Revised test expected outputs in `/oss/internal/pg_documentdb_distributed/src/test/regress/expected/feature_counters.out` and `/pgmongo/src/test/docdb_compat/expected/feature_counters.out` to reflect the updated counters. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
 Vinod Sridharan
|
ef8f7ae41c |
Merged PR 1809984: Add version logging to the backend
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) DevOps ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add Version Logging in the backend metrics collection. This makes it so we're not reliant on the gateway to log version information to metrics. ---- #### AI description (iteration 1) #### PR Classification This pull request introduces a new feature to log backend version metrics. #### PR Summary The changes add functionality to periodically log version information from various components, ensuring updated logging when version changes or timer intervals trigger a new record. - `pgmongo/src/infrastructure/mdsd_logger.c`: Added new functions (`append_version_metric_data` and `LogVersionMetricData`) to construct and send version metrics, and integrated these into the main logging workflow. - `pgmongo/src/pgmongo_extension_version.c`: Introduced `PgmongoGetExtendedVersionString` and updated the version-returning function to use this extended string. - `oss/pg_documentdb/src/utils/version_utils.c`: Added `GetCurrentShortVersionStringForLogging` to format a concise version string for backend logging. - `pgmongo/src/metadata/oid_cache.c`: Added the `CitusVersionFunctionOid` function and updated the OidCache structure to support version logging. - Header updates in `oss/pg_documentdb/include/utils/version_utils_private.h` and `pgmongo/include/metadata/oid_cache.h` to declare the new version utility and Oid functions. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
080ea66e2b |
Merged PR 1803246: [Hidden Index] Support for hidden indexes in DocumentDB
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add support for hidden indexes with collMod This will mark the index as not ready in the postgres catalog. Handles distributed setups as well. However, this is currently restricted to single node (sharded/unsharded). A subsequent change can help multi-node set ups. ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature by adding support for hidden indexes in DocumentDB. #### PR Summary The pull request extends the collMod command and associated index management logic to recognize and update a new “hidden” flag in index options, while enforcing constraints (e.g. disallowing hidden modifications for primary or unique indexes) in both single-node and distributed environments. - **`coll_mod.c`**: Introduces helper functions (e.g. GetHiddenFlagFromOptions, UpdateHiddenInIndexOptions, UpdatePostgresIndex) and SPI query updates to process the hidden flag properly. - **Test Files**: Adds and updates multiple test cases (e.g. in `commands_collmod_tests_hidden.out` and SQL test scripts) to verify hidden index creation, query planning, and error handling. - **Distributed Hooks and API**: Updates in files like `distributed_hooks.c` and `api_hooks.c` (with corresponding header changes) enable proper handling of hidden index updates across shard indexes in distributed environments. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
 William Wang
|
190c5f7710 |
Merged PR 1750825: [Full RBAC] Adding Support for RolesInfo Command in Engine (2/2, RolesInfo Impl.)
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) 1.108 ### Description We are adding support for the RolesInfo command in Engine ---- #### AI description (iteration 1) #### PR Classification New feature enhancement: introducing the `rolesInfo` command to retrieve detailed role information including inherited roles and privileges. #### PR Summary This pull request adds comprehensive support for the `rolesInfo` command in the engine, implementing new parsing, processing, and BSON output functions while updating tests and auxiliary query strings. - `oss/pg_documentdb/src/commands/roles.c`: Added functions (e.g., `ParseRolesInfoSpec`, `WriteRoleDocumentWithInheritedRoles`, `WriteInheritedRolesToArray`) to parse rolesInfo parameters and construct role documents with inherited roles and optional privileges. - `oss/pg_documentdb/src/test/regress/expected/role_crud_commands.out` and `pgmongo/src/test/regress/expected/role_crud_commands.out`: Expanded test suites with numerous rolesInfo scenarios, error conditions, and inheritance verification tests. - `oss/pg_documentdb/include/roles.h`: Introduced new types and function prototypes to support rolesInfo and role CRUD operations. - `pgmongo/src/pgmongo.c`: Updated query string definitions to handle role name substitutions for OSS compatibility. - `oss/pg_documentdb/src/commands/users.c`: Modified role checking and privilege extraction utilities to align with the new rolesInfo functionality. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4552244 Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
 Cornelius Boateng
|
ce0cb575be |
Merged PR 1794050: [engine][collation] Support for $sortArray
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Support for the `$sortArray` aggregation operator. ---- #### AI description (iteration 1) #### PR Classification This pull request implements collation support for the $sortArray operator as a new feature. #### PR Summary This change integrates collation awareness into the $sortArray operator so that array sorting properly respects collation rules. The modifications update both the underlying sorting logic and the tests to reflect these new capabilities. - `pg_documentdb/src/operators/bson_expression_array_operators.c`: Updated parsing of the $sortArray operator to copy the collation string into the sort context. - `pg_documentdb/src/utils/sort_utils.c` and `pg_documentdb/include/utils/sort_utils.h`: Switched to collation-aware comparison functions by replacing calls to CompareBsonValueAndType with CompareBsonValueAndTypeWithCollation. - Test expectation files under `/oss/internal/pg_documentdb_distributed/` and `/pgmongo/` were updated to include new $sortArray outputs with different collation settings. - Integration CSV files were modified to mark the previously failing `$sortArray` tests as passing. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
William Wang
|
5a37471d7d |
Merged PR 1784924: Utility File and Query Refactoring for Role/User CRUD APIs
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description ---- #### AI description (iteration 1) #### PR Classification This PR refactors and centralizes role/user CRUD utility functions. #### PR Summary The changes migrate and consolidate the logic for user and role management into a new utility module and header file, streamlining the codebase and removing redundant implementations from command files. - `src/utils/role_utils.c` adds core functions for password hashing, privilege consolidation, and user/role document writing. - `include/utils/role_utils.h` provides declarations for the newly centralized role/user utilities. - `src/commands/users.c` and `src/commands/roles.c` are cleaned up by removing duplicated role/user functionality now handled in the new module. - `pgmongo.c` and related metadata files are updated to use the new role replacement variable and adjust related constants. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4639026, #4639027 |
||
 Duan Pei
|
6a129514a9 |
Merged PR 1777887: [Refector] Update to match coding standards - part 1
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description update errmsg ---- #### AI description (iteration 1) #### PR Classification This pull request focuses on updating and standardizing error messages across various operators, aggregation stages, and test expectation files. #### PR Summary The changes reword error messages throughout the codebase and test outputs to provide clearer, more consistent, and more informative feedback for users. Key updates include: - **`/oss/internal/pg_documentdb_distributed/src/test/regress/expected/*.out`** – Revised expected error outputs for aggregation, update, and geospatial tests to use clearer phrasing (e.g., for `$merge`, `$pop`, `$indexOfArray`, and `$geoNear`). - **Core operator files (e.g., `bson_aggregation_output_pipeline.c`, `bson_expression_arithmetic_operators.c`, `bson_expression_conditional_operators.c`)** – Updated error message texts to explicitly state operator requirements and types. - **Geospatial and vector modules (e.g., files under `bson_geojson_utils.c`, `bson_geospatial_*`, and vector search/index configuration files)** – Standardized validations and error reporting for GeoJSON, CRS, and vector search parameters. - **Additional modules (e.g., collection view creation and user management files)** – Improved error handling messages to better guide users on input misconfigurations. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Parag Jain
|
fa57f40720 |
Merged PR 1779662: [Observability] advanced feature usage for write commands
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description [Insert] Feature Usage Insert ---- #### AI description (iteration 1) #### PR Classification New feature addition that implements feature usage tracking for insert and update commands. #### PR Summary This pull request adds inline functions to report feature usage based on batch sizes for insert and update operations and updates the associated feature enum definitions. - `/oss/pg_documentdb/src/commands/insert.c`: Added `ReportInsertFeatureUsage` to report usage metrics based on batch size and integrated its call during batch insertion processing. - `/oss/pg_documentdb/src/commands/update.c`: Introduced `ReportUpdateFeatureUsage` with similar threshold-based reporting, integrated into the batch update process. - `/oss/pg_documentdb/include/utils/feature_counter.h`: Extended the feature counter enums with new constants for detailed insert and update reporting, and increased `MAX_FEATURE_COUNT`. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
0c509dc131 |
Merged PR 1754381: [Sort] Support sort by id pushdown to the index
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Support pushdown of sort of _id to the primary key btree index if available. ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature to push down sort on the _id field into the index, optimizing query ordering for primary key lookups. #### PR Summary The changes introduce new logic to detect and push down sort-by-_id operations by leveraging primary key indexes, update sort expression creation to use collation-aware functions, and remove outdated configuration flags. - `src/opclass/index_support.c`: Added and refactored code to identify primary key indexes and create new index paths when sorting by _id. - `src/aggregation/bson_aggregation_pipeline.c`: Updated sort function logic to apply collation and proper pushdown criteria, including full-scan clause insertion when necessary. - Test and expected output files: Adjusted explain plan outputs and renamed feature counters (switching from `sort_by_id_pushdownable` to `sort_by_id`) along with modifying configuration flags. - `include/opclass/bson_index_support.h` and config files: Introduced a new context structure (`PlannerQueryOrderByData`) and removed obsolete variables related to sort-by-_id pushdown. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
623ef26f85 |
Merged PR 1742378: [Hints] Add support for index hints in DocumentDB
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Index hints are a mongo feature that allows customers to specify which index they would like to target for a query. Typically this is used in diagnostic scenarios when testing query performance. With the new composite index op class coming in, we’re introducing side by side support for composite which makes it so we would want customers to be able to test which index is better for planning. In this spirit, index hints is introduced which follows the same infra as the force pushdown for geonear and text and adds a new option for index hints. As part of this the code is also refactored to cleanup some of the logic in the relpathlist and fix correctness issues around text index force pushdown (which was needed to handle index hints) ---- #### AI description (iteration 1) #### PR Classification This pull request introduces a new feature to support index hints in DocumentDB. #### PR Summary The changes integrate index hint support into query planning and execution, allowing users to force a specific index during query processing. Key updates include: - **`src/opclass/index_support.c`**: Added new functions and a force index operator type (`ForceIndexOpType_IndexHint`) for filtering available indexes and generating custom operator expressions based on index hints. - **`src/planner/documents_planner.c`**: Modified base table query planning logic to detect index hints and adjust index pushdown, including updates to path and restriction handling. - New UDFs (e.g., `bson_dollar_index_hint`) and configuration variables (`EnableIndexHintSupport`, `UseLegacyForcePushdownBehavior`) were added to support the hint mechanism. - Test files and SQL scripts (e.g., in `/oss/internal/pg_documentdb_distributed/src/test/regress/`) were introduced or updated to validate index hint pushdown behavior. - Documentation and release notes (e.g., `docs/pgmongo/design_docs/data_plane/indexing/index_hints.md`, `pgmongo_1.106.md`, and `CHANGELOG.md`) were updated to describe the index hint feature. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
William Wang
|
e4024ad3ad |
Merged PR 1721770: [OSS] [Engine] Adding Support for Create Role Command and API Schema for Role CRUD
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) 105 ### Description As part of Full RBAC, we are adding support for role management CRUD operations. This PR adds support for the createRole Command and API Schema for dropRole, rolesInfo, updateRole commands Related work items: #4053600, #4338364 |
||
|
Deepak Bisht
|
66e684c662 |
Merged PR 1719410: [Freespace] Part2 - Add pgmongo UDFs to calculate freespace metric for all collections
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Logging ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) NA ### Description - Add Pgmongo UDF to collect freespace metrics using `pg_freespamcemap` extension for all collection. - Use shared memory to store the next timestamp when freespace metrics should be collected which is 8 hours on the PG restarts and after one collection then set to next 24 hours, this is done to make sure we handle cases where PG might restart frequently. - The metrics are published per collection only if freespace available is more than the threshold defined by `pgmongo.freespace_metric_collection_threshold` which is defaulted to 30% - Each freespace metric collection happens for at most 60 secs and then the function returns so that we don't have long running metric collectors. The next log collection timestamp and collectionId are stored in shared memory.  ---- #### AI description (iteration 1) #### PR Classification This pull request introduces a new feature by adding UDFs and associated infrastructure to calculate and log freespace metrics for all collections. #### PR Summary The changes add C-based UDF functions that compute freespace metrics using the pg_freespacemap extension, integrate these metrics into telemetry logging, and provide tests and installation scripts to support the feature. - `pgmongo/src/utils/freespace_utils.c`: Adds C UDF implementations for calculating freespace metrics on both sharded and unsharded collections. - `pgmongo/src/infrastructure/mdsd_logger.c`: Implements functions to collect and log freespace metrics to the telemetry system. - `pgmongo/sql/udfs/telemetry/pgmongo_freespace_metrics--*.sql`: Introduces new SQL UDF definitions to expose freespace metric functionality. - `pgmongo/src/test/regress` and configuration files: Add regression tests, expected outputs, and update installation scripts (including Makefile and extension setup) to enable and verify the pg_freespacemap extension. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4129539 |
||
 Zinan Ren
|
b11753f8d2 |
Merged PR 1712315: [oss][test cleanup] Vector search related cleanup
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) No ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Cleanup vector tests |
||
|
Vinod Sridharan
|
1c433c472b |
Merged PR 1714381: [Order by + Test] Fix up more correctness of order by + Testing
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Bugfix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description 1) There were order by tests not running but had the sql files in there. This was catching contracts related to arrays with null values where 2 cases were ordered later than they should be. This fixes that. As part of that we noticed that this breaks when there's paths that have array indexes (e.g. a.b.0). This also fixes that contract. 2) Add checks to say all tests must run or exist in a schedule file and fix the output 3) There's flakeyness in currentop due to tracking the version refresh query - skip the version refresh query in currentop. ---- #### AI description (iteration 1) #### PR Classification This pull request fixes correctness issues in composite order-by functionality and improves associated testing. #### PR Summary The changes refine the handling of intermediate arrays in order-by operations by updating traversal and comparison logic, while enhancing the test suite and validation scripts for composite order-by scenarios. - `pg_documentdb/src/query/bson_dollar_operators.c` and related files were modified to correctly process nested arrays and compare composite order-by values. - Test files in `pgmongo/src/test/regress/expected/` and `pgmongo/src/test/regress/sql/` were updated to add new order-by cases and remove obsolete tests. - Validation scripts (e.g. in `oss/internal/pg_documentdb_distributed/src/test/regress/validate_test_output.sh` and `oss/pg_documentdb/src/test/regress/validate_test_output.sh`) were enhanced for more reliable schedule checks and collection ID extraction. - Several BSON traversal and update operator functions were updated to include an `isArrayIndexSearch` parameter for finer control during array element processing. - `pg_documentdb/src/commands/current_op.c` and `pg_documentdb/src/utils/version_utils.c` were updated to skip version refresh queries appropriately. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
a547931808 |
Merged PR 1706921: [Background Index] Improve create indexes performance
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Performance ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Currently, each round of the sproc that does create indexes does exactly 1 index build and bails. This means if a single client request queues 4 indexes and the job runs every 2 seconds, it'll take 8 seconds to build all 4 indexes. This is wasteful since we know all 4 indexes up front. Add a loop to the index build process so that multiple indexes can be built without waiting the 2 seconds. As part of this, there were a few places allocating in memory contexts that are unstable across executions. Fix this. Additionally there was a loop that could cause index builds to starve for ~20 mins if it picked the wrong index. Fix this loop too. Finally, Noticed that the version cache does not cache properly if the Major version is 0. Fix this (which can cause perf regressions in DocumentDB). |
||
|
William Wang
|
d339399b36 |
Merged PR 1694202: Adding Support for Connection Status Command in OSS Engine
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes, the API schema ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) 1.05 ### Description We are adding support for the Connection Status command, which exists in native Mongo, and allows users to see their authenticated user, authenticated roles, and user privileges. For more information, visit connectionStatus.md. This PR is the second of three PRs for this feature. The first one added extensible role privileges to our `users_info` command, allowing users to retrieve role privileges with `showPrivileges: true`. This second PR adds a callable `connection_status` command to our API, that utilizes the role privileges. The last PR will add a gateway change that calls this `connection_status` command. Related work items: #4287246 |
||
|
Vinod Sridharan
|
ccc471e169 |
Merged PR 1703619: [Cursors] Improve stability of file based cursors
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description ---- #### AI description (iteration 1) #### PR Classification Enhancement and bug fix to improve the stability and management of file-based cursors. #### PR Summary This pull request refactors and bolsters file-based cursor management by introducing shared memory support, improved cleanup, error handling, and configuration changes. It also adds new API endpoints and logging for better cursor lifecycle control. - **`cursor_store.c` and related files:** Added shared memory functions (e.g., `FileCursorShmemSize`, `InitializeFileCursorShmem`), cursor count regulation (with `IncrementCursorCount`/`DecrementCursorCount`), and cleanup routines to ensure stable cursor file operations. - **Test and expected output files:** Updated regression tests and expected outputs to cover scenarios like duplicate cursor creation, cursor deletion limits, and error reporting when the file cursor capacity is reached. - **API enhancements:** Introduced new UDFs (e.g., `command_delete_cursors`) and tenant client methods for killing and deleting cursors, improving session management. - **Configuration updates:** Renamed and added GUC parameters (e.g., `MaxAllowedCursorIntermediateFileSizeMB`, `MaxCursorFileCount`, and cursor expiry settings) to control file-based cursor behavior. - **Logging and metrics improvements:** Enhanced logging in `mdsd_logger.c` to report current cursor counts and related metrics for operational insight. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
c6bc2bcb73 |
Merged PR 1694458: [POC|Order by] Order by Index & Pushdown improvements
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description More general improvements to order by and composite: Create Indexes 1) Support creating a composite index side by side with a regular index 2) Block creation of descending or wildcard and unsupported set ups for composite indexes Planner: 1) Support ordering by the Nth path of an index if the prefixes are equality. 2) Validate that if it's not the first column, prefixes are provided 3) Support incremental sort with index pushdown. ---- #### AI description (iteration 1) #### PR Classification This pull request delivers a new feature and API enhancement to enable order-by index pushdown and refined composite index support. #### PR Summary The update streamlines composite index processing by introducing stricter equivalency checks, new UDF functions for full-scan order-by pushdown, and enhanced validations during index creation while updating explain outputs and tests. - `src/metadata/index.c` refines index options equivalency logic to correctly compare composite term settings. - `src/opclass/bson_gin_composite_entrypoint.c` and `src/index_am/rum.c` add functions to support order-by pushdown and manage duplicate entries efficiently. - `sql/udfs/query/bson_dollar_comparison--0.105-0.sql` (and related operator files) introduce new UDFs including `bson_dollar_fullscan` to enable full scan operations. - `src/commands/create_indexes.c` enforces new composite index validations (restricting use to ascending, non-wildcard regular indexes) with precise error reporting. - Test scripts and explain plan outputs in the `regress` and `pgmongo` directories are updated to confirm the new composite order-by behavior. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Shuai Tian
|
17d4306a28 |
Merged PR 1422337: [pgmongo] Aggregate stage $bucketAuto implementation
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md)) ### Description This pull request adds documentation for the `$bucketAuto` aggregation stage in PostgreSQL, detailing its design, usage, and implementation. - Adds `bucketAuto.md` with details on the `$bucketAuto` aggregation stage - code implementation - support all granularity types: POWERSOF2, 1-2-5, Renard numbers, E series Related work items: #3234185 |
||
 Jayanta Mondal
|
4b419d9b40 |
Merged PR 1671857: Adding support for custom index handler
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) Adding support for custom index handler. ### Description Adding support for custom index handler. Currently the default index access method to index bson data type is "rum". This change allows one to swap out "rum" for something else. The use an alternate index handler one needs to specify two things 1. Firstly, register the index handler and its capabilities and relevant utility call backs. Currently, the system only allows a maximum of 5 alternate index handler. 2. Secondly, set the alternate_index_handler GUC to let the system chose the alternate index handler to index bson. Following are a few notes on the mechanism 1. During index create if the alternate AM does not support certain kind of bson index (e.g, unique index, wildcard index) then the index is created via the default rum index. Otherwise, the alternate AM is chosen. 2. If an index has been created using alternate index AM but then reset the alternate_index_handler GUC or set it to another third alternate index AM, the old alternate index AM will still be in use, and long as both the old and new alternate index AM are in the registry. Other: Fixed Densify and SetWindow fields flakey tests for sharded collection ---- Related work items: #3796361 |
||
|
Deepak Bisht
|
96924b8eee |
Merged PR 1672438: Support Compact in the vCore Backend engine
Yes Feature Yes No - Support compact database administration command behind the GUC `enableCompact`. - Add support for compact command in backend using VACUUM FULL which is a blocking operation. - Only one compact command can be run at a time on a specified collection. - VACUUM FULL is run using libpq in a top level transaction as it it not supported inside functions or procedures. - Add support of tracking compact command in current_op. - `dryRun` estimation is based on the query defined [here at pgexperts](https://github.com/pgexperts/pgx_scripts/blob/master/bloat/table_bloat_check.sql), this can be a rough estimation and doesn't provided details about the space reclaimed by vacuuming RUM indexes of the collection. - Read only users can't perform `compact` command. ---- New feature implementation to support the backend compact command. This PR adds complete backend support for a blocking compact command that estimates table bloat and conditionally performs a VACUUM FULL operation on collection tables. - `src/utils/storage_utils.c` and `include/utils/storage_utils.h`: Introduced a new bloat estimation utility returning collection bloat stats. - `src/commands/compact.c`: Implemented the compact command including BSON command parsing, dry-run handling, and conditional execution of VACUUM FULL. - `internal/pg_documentdb_distributed/src/test/regress/sql/command_compact.sql`: Added regression tests covering various compact command scenarios. - `include/utils/feature_counter.h` and `src/infrastructure/feature_counter.c`: Updated feature counter mapping to include the new compact command feature. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4237921 |
||
|
Cornelius Boateng
|
11fab5ffc6 |
Merged PR 1683133: [engine] More Feature Counters
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Dev ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add more feature counters for let, cursor types and top-level commands ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature by adding and tracking additional feature counters in the engine. #### PR Summary This change adds new feature counter entries and instrumentation to track usage for several commands and top-level "let" support. The updates include increasing the feature count limit and inserting ReportFeatureUsage calls in relevant command and aggregation pipelines. - `src/infrastructure/feature_counter.c` and `include/utils/feature_counter.h`: Added new feature counter mappings and enums for commands (e.g., agg_cursor_first_page, count, distinct, find_cursor_first_page, get_more, list_collections_cursor_first_page, list_indexes_cursor_first_page) and top-level "let" support. - `src/aggregation/bson_aggregation_pipeline.c`: Integrated ReportFeatureUsage calls for "let" in aggregate and find queries. - `src/commands/aggregation_commands.c`: Inserted ReportFeatureUsage calls in various command functions to track usage of the new feature counters. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4224518, #4228942 |
||
|
Jayanta Mondal
|
fa75e56ac1 |
Merged PR 1669686: Adding feature counter for _internalInhibitOptimization
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description more ---- #### AI description (iteration 1) #### PR Classification This PR introduces a new feature by adding support for tracking the internal inhibit optimization feature usage. #### PR Summary The changes enhance the feature counter system to monitor the `_internalInhibitOptimization` usage in aggregation pipelines and SQL tests. - `mongo_feature_counters.sql`: Added new SQL queries to test and display the feature counter for `_internalInhibitOptimization`. - `feature_counter.h`: Added a new enum constant, `FEATURE_STAGE_INTERNAL_INHIBIT_OPTIMIZATION`, for the new feature. - `bson_aggregation_nested_pipeline.c`: Updated the internal optimization handler to log feature usage. - `feature_counter.c`: Extended the feature mapping to include `_internalInhibitOptimization`. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Zinan Ren
|
4b79ce38d4 |
Merged PR 1668682: [Vector Search] Add GUC for the default search parameter calculation
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Perf ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Adding the GUC documentdb.enableVectorCalculateDefaultSearchParam to enable or disable the calculation of default search parameters. The default value is on Adding feature counts for the usage of default search parameters. ---- #### AI description (iteration 1) #### PR Classification This PR implements a new configuration option to enable or disable the dynamic calculation of default search parameters for vector search indexes. #### PR Summary The pull request adds a GUC flag that controls whether default search parameters (e.g., nProbes for IVF and efSearch for HNSW) are computed dynamically based on collection size. It updates internal calculation functions, test expectations, and feature counters accordingly. - `src/vector/vector_index_kind_impl.c`: Revised the IVF and HNSW search parameter calculation functions to conditionally compute default parameters when the new GUC is enabled. - `src/configs/feature_flag_configs.c` and `include/vector/vector_common.h`: Introduced the new boolean GUC `enableVectorCalculateDefaultSearchParam`. - Test files (e.g., files under `/oss/internal/pg_documentdb_distributed/src/test/regress/expected/` and corresponding `pgmongo` tests): Updated expected outputs and query plans to reflect the new default parameter values. - Feature counter outputs and related SQL tests: Modified to include metrics for the new default search parameter usage. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4194082 |
||
|
Zinan Ren
|
e05e91ba51 |
Merged PR 1650017: [Vector Search] Add support for PQ compression in the Diskann index
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Adding support Product Quantization (PQ) compression in vector index, which is a popular method for dramatically compressing high-dimensional vectors to use less memory, and for making nearest-neighbor search faster.
Note: currently, this PQ compression is only supported by the Diskann index.
### Related GUCs
**documentdb.enableVectorCompressionPQ**: Enables vector PQ compression (enabled by default).
### Create index spec:
- **compression**(optional):
- "half": half-precision vector, which using 16 bits floats.
- "pq": product quantization compression.
- **pqCompressedDims**(optional): indicates the dimensions after compression. If not specified, it will be automatically calculated based on the original dimensions.
- Minimum: 1
- Maximum: 8000
- **pqSampleSize**(optional):Indicates the number of samples PQ uses to generate the initial centroids. The default value is 1000.
- Minimum: 1000
- Maximum: 100000
**Note**: It is recommended to create the index after the collection has some data.. If the collection is empty, random vectors within the range [-1, 1] are used to generate the initial centroids and create the index. If the number of documents is less than pqSampleSize, the training data is padded with random data within the range [min, max] from the sampled vector data.
### Command example
```
db.runCommand(
{ "createIndexes": "testing_vector_collection", "indexes":
[{
"key": { "v": "cosmosSearch" },
"name": "hsnw_index",
"cosmosSearchOptions":
{
"kind": "vector-diskann",
"similarity": "L2",
"dimensions": 1536,
"compression": "pq",
"pqCompressedDims": 96,
"pqSampleSize": 2000
}
}]
} )
```
### Search spec:
No changes
----
#### AI description (iteration 1)
#### PR Classification
New feature: This pull request adds support for Product Quantization (PQ) compression in the Diskann index.
#### PR Summary
This pull request introduces support for PQ compression in the Diskann index, enhancing vector search capabilities. It includes error handling for unsupported compression types and validation for PQ parameters.
- Updated `/pgmongo/src/test/regress/expected/bson_aggregation_pipeline_tests_vector_diskann.out` to include tests for PQ compression, error cases, and validation of PQ parameters such as `pqCompressedDims` and `pqSampleSize`.
- Added tests for vector search with PQ compression, including L2, cosine, and IP similarity measures.
- Implemented checks for maximum dimensions and resource limits during PQ index creation.
<!-- GitOpsUserAgent=GitOps.Apps.S...
|
||
|
Vinod Sridharan
|
5a19661e8b |
Merged PR 1654645: [Feature Flags] Refactoring features and add some tracking
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactor ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description - Add a new section for tracking feature flags - Add a feature counter to track customers whoc an benefit from the new rum_index_scan. ---- #### AI description (iteration 1) #### PR Classification Refactoring and feature enhancement with added tracking capabilities. #### PR Summary This pull request refactors feature flags and introduces new tracking capabilities for feature usage. It reorganizes the feature flag sections and adds new flags for various functionalities. - Refactored and added new feature flags in `feature_flag_configs.c`, including indexing, planner, and aggregation features. - Updated `custom_scan_continuation.c` to improve index scan handling and added feature usage reporting. - Modified `insert.c` to handle snapshot management more effectively. - Enhanced feature usage tracking in `feature_counter.c` and `feature_counter.h` by adding a new feature counter for index scans with limits. - Updated test expectations in multiple files to reflect the new feature tracking changes. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Parag Jain
|
4ed646dfb9 |
Merged PR 1647233: [Bulk Insert][Perf] support for bulk insert via procedures
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Currently bulk insert run as a function which means that the entire execution runs as a single transaction. Each batch insert is handled as a subtransaction and rolled back to a save point if it fails. This means that document locks are held for longer, and we can hit limits w.r.t. max documents updated per transaction. In multi-node setups this also means distributed transactions are held longer causing loss of connection pools. This change introduces a stored procedure for bulk insert that periodically "Commits" the state to WAL and starts a new transaction. This more mirrors what the non-atomic bulk update is supposed to do. The old function is retained to be used within transactions. ---- #### AI description (iteration 1) #### PR Classification Performance improvement for write commands. #### PR Summary This pull request introduces a new procedure for handling write commands, enhancing performance by optimizing user-specific command execution. It includes the addition of new functions and procedures to manage MongoDB operations more efficiently. - Added new SQL functions and procedures in `/pgmongo/sql/udfs/gateway_funcs/command_funcs--1.103-0.sql` for running commands, queries, DDL, and write operations as a specific user. - Introduced new insert functions and procedures in `/oss/pg_documentdb/sql/udfs/commands_crud/insert--0.103-0.sql` to handle MongoDB insert operations. - Implemented `pgmongo_run_write_procedure_as_user` and `ExecuteWriteCommandCore` in `/pgmongo/src/infrastructure/command_funcs.c` to manage write commands with user context. - Added `CommitOuterInsertTransactions` function in `/oss/pg_documentdb/src/commands/insert.c` to handle transaction commits outside of transaction blocks. - Introduced a feature flag `EnableWriteProcedure` in `/oss/pg_documentdb/src/configs/feature_flag_configs.c` to control the enabling of write procedures. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
36bb3e2920 |
Merged PR 1650400: [Bulk Update] Setup bulk update infrastructure
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Currently bulk updates run as a function which means that the entire execution runs as a single transaction. Each batch update is handled as a subtransaction and rolled back to a save point if it fails. This means that document locks are held for longer, and we can hit limits w.r.t. max documents updated per transaction. In multi-node setups this also means distributed transactions are held longer causing loss of connection pools. This change introduces a stored procedure for bulk update that periodically "Commits" the state to WAL and starts a new transaction. This more mirrors what the non-atomic bulk update is supposed to do. The old function is retained to be used within transactions. Still TODO: For unsharded collections in a multi-node setup, we need to send batches of updates to the worker. This is not wired up in this change. ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request introduces a new bulk update feature for handling non-transactional updates in the MongoDB wire-protocol. It adds new functions and procedures to support bulk updates and modifies existing update logic to accommodate this new functionality. - Added `update_bulk` procedure and related functions in `sql/udfs/commands_crud/update--0.103-0.sql` for non-transactional bulk updates. - Implemented `command_update_bulk` in `src/commands/update.c` to handle bulk updates outside of transactions. - Enhanced `ProcessBatchUpdateCore` in `src/commands/update.c` to support transactional and non-transactional updates. - Updated feature tracking in `include/utils/feature_counter.h` and `src/infrastructure/feature_counter.c` to include `FEATURE_COMMAND_UPDATE_BULK`. - Modified SQL scripts `documentdb--0.102-1--0.103-0.sql` and `pgmongo--1.102-1--1.103-0.sql` to include the new update functionality. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Zinan Ren
|
591a38b86d |
Merged PR 1606660: [Vector Search] Support half-precision vector indexing
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description With the release of pgvector 0.7.0, which supports [Half-Precision Indexing](https://github.com/pgvector/pgvector?tab=readme-ov-file#half-precision-indexing), vectors can have up to 4,000 dimensions ###Related GUCs **documentdb.enableVectorCompressionHalf**: enable vector compression(enabled by default) - off, disable vector compression - on, enable vector compression ###New parameters - Create index spec: "compression": string "half": half-precision vector, which using 16 bits float - Search spec: "oversampling": double Oversampling factor: Oversampling retrieval from half index to compensate for the loss of precision. Minimum value is 1. This parameter is only permitted when a compression method is used on the vector index. For example, if oversampling=1.5 and k=10, then oversampling*k=15 vectors from a compressed index. Then, the full 32-bits vector will be used for the subsequent ordering and scoring, ###Mongo Command ``` db.runCommand( { "createIndexes": "testing_vector_collection", "indexes": [{ "key": { "v": "cosmosSearch" }, "name": "hsnw_index", "cosmosSearchOptions": { "kind": "vector-hnsw", "similarity": "L2", "dimensions": 3, "compression": "half" } }] } ) ``` ``` db.testing_hnsw_vector_collection.aggregate( [ { "$search": { "cosmosSearch": { "vector": [ 3.0, 4.9, 1.0 ], "path": "vect", "filter": {"meta.a": [ { "b" : 3 } ]}, "k": 4, "efSearch": 100, "oversampling": 2.0 }, } }, { "$project": { "rank": { "$floor": [{"$multiply": [{ "$meta": "searchScore"},100000]}] } } } ]) ``` ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request adds support for half-precision vector indexing in the vector search functionality. - `metadata_cache.c`: Added functions for half-precision vector similarity operators and functions. - `SearchTests.cs`: Added integration tests to validate half-precision vector search, including error cases and index creation scenarios. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Deepak Pemmaraju
|
cdb390d2eb |
Merged PR 1600780: DataPlane External Identity Provider User CRUD
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request introduces CRUD operations for Entra users in the DataPlane. - `authentication.c`: Added functions to create users with external identity providers, validate identity provider info, and check if the current user is an Entra admin. - `users.h`: Added new header file defining user CRUD functions and related structures. - `users.c`: Updated user creation logic to handle external identity providers and added parsing for custom data. - `api_hooks.c`: Added hooks for creating users with external identity providers and checking user privileges. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Parag Jain
|
80df5a1509 |
Merged PR 1604750: [perf][creation_time] Alter creation time : part 1
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Altering creation_time column removing not null and default constraint. ---- #### AI description (iteration 1) #### PR Classification Performance improvement #### PR Summary This pull request alters the `creation_time` column in the database schema to improve performance. - Added `AlterCreationTime` function in `cluster_operations.c` to modify the `creation_time` column. - Updated `SetupCluster` function in `cluster_operations.c` to call `AlterCreationTime`. - Created new SQL scripts `extension_upgrade_alter_creation_time_1.102-0.sql` and `extension_upgrade_alter_creation_time_1.102-0.out` for testing the changes. - Modified multiple test output files to reflect the changes in the `creation_time` column constraints. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Xing Fan
|
399e01d954 |
Merged PR 1539417: [Operator] $toUUID in Mongo 8.0
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md)) ### Description support $toUUID operator ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request introduces the `$toUUID` operator for MongoDB 8.0, enabling conversion of strings to UUIDs. - Added `ProcessDollarToUUID` function in `bson_expression_type_operators.c` to handle the conversion logic. - Updated `bson_expression_operators.h` to declare new parsing and handling functions for `$toUUID`. - Included new test cases in `bson_aggregation_type_operators_tests.sql` and `bson_aggregation_type_operators_tests.out` to validate the `$toUUID` functionality. - Modified `feature_counter.h` and `feature_counter.c` to include the new `$toUUID` feature in the feature count and mapping. |
||
|
Jayanta Mondal
|
4d5d15992c |
Merged PR 1582153: [Engine] Adding feature counter for sort by id
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature counter ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) No ### Description Adding feature counter for sort by id to make decisions about sort by id pushdown to primary key optimization. |
||
|
Zinan Ren
|
1d3b47c814 |
Merged PR 1573602: [Vector] Add "exact" search parameter to enable enn search
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature, Perf ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description ### Introduce new field "**exact**" in vector search spec to specify whether to run ENN or ANN search. [exact field spec](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-stage/#fields) ### Value can be one of the following, defaults to false. - false - to run ANN search - true - to run ENN search ### Example `{ "k": 1, "path": "vect", "filter": {"attr": {"$eq":1}}, "exact": true, "efSearch": 40 }` ---- #### AI description (iteration 1) #### PR Classification Code enhancement for filter selectivity in vector search queries. #### PR Summary This pull request enhances the filter selectivity mechanism in vector search queries by adding new functions and improving existing ones. - `bson_aggregation_vector_search.c`: Added functions to calculate filter selectivity and handle vector search options, including `AddEmptySearchParamFunctionToQuery` and `GetFilterSelectivity`. - `index_support.c`: Added logic to identify vector search queries in the planner. - `system_configs.c`: Introduced new configuration variables for maximum filter selectivity and document count. - `documents_planner.c`: Enhanced planner to handle vector search queries based on selectivity. - `vector_common.h`: Declared new configuration variables for vector filter runtime sort selectivity and document count. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Marco Slot
|
a05adcfd20 | Build pg_documentdb on PostgreSQL 17 (#13) | ||
|
Xing Fan
|
722ef60784 |
Merged PR 1479858: $percentile & $median for $group accumulator
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md)) ### Description draft ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request introduces new aggregation functions and support for t-digest in the project. - Added `bson_tdigest.c` to implement t-digest helper functions. - Created new SQL aggregate functions in `group_aggregates--1.23-0.sql` and `group_aggregates_support--1.23-0.sql`. - Updated `bson_aggregation_pipeline.c` to handle the new `$percentile` accumulator. - Modified `metadata_cache.c` and `metadata_cache.h` to include OID for the new `BsonPercentileAggregateFunction`. - Updated migration scripts `pg_helio_api--1.22-0--1.23-0.sql` and `pgmongo--1.22-0--1.23-0.sql` to include new SQL files. |
||
|
Shuai Tian
|
e3dc6c889e |
Merged PR 1404676: stage implementation: redact
### Does this PR have any customer impact? yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes, UDF ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md)) ### Description Implement for stage redact Related work items: #1798586 |
||
 Jia Hu
|
55f6fa3840 |
Merged PR 1485956: [schema validation] support schema rule included top level operator
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md)) ### Description To enable top level query operators working in schema validation. ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request introduces support for the `$jsonSchema` query operator for schema validation. - Added handling for `QUERY_OPERATOR_JSONSCHEMA` in `query_operator.c` and `mongo_query_operator.c`. - Implemented `GetExpressionEvalStateForBsonInput` in `bson_expr_eval.c`. - Added `BsonJsonSchemaFunctionId` and related OID cache in `metadata_cache.c`. - Modified `PrepareForSchemaValidation` in `schema_validation.c` to use the new evaluation state function. |
||
|
Vinod Sridharan
|
584cc387f0 |
Merged PR 1545267: [OSS] Support DocumentDB on V0
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactor ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md)) ### Description Move DocumentDB to schema version 0. To do this version checks now need to handle internal and external versions being different. Tackle this in version_utils.c ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request adds support for DocumentDB on version 0. - Modified `cluster_operations.c` to use `DocDB_V0` for version checks. - Updated `version_utils.h` and `version_utils.c` to define and use `MajorVersion` enum for version handling. - Adjusted various files in the `aggregation` directory to use `DocDB_V0` for version checks. - Updated `create_indexes.c` to use `DocDB_V0` for version checks. |
||
|
Zinan Ren
|
8e7bd960bf |
Merged PR 1540262: [OSS] Remove helio references from sources of documentdb_core
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactoring ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md)) ### Description Remove helio from file name Remove helio references from source content except schema reference Start local documentdb server with the documentdb extension Regression tests using the documentdb extension ---- #### AI description (iteration 1) #### PR Classification Code cleanup #### PR Summary This pull request removes references to "helio" from the `documentdb_core` source code. - `scripts/utils.sh`: Renamed variables from `preloadExtensions` to `preloadLibraries`. - `pg_documentdb_core/include/utils/error_utils.h`: Updated function and macro names to replace "Helio" with "DocumentDB". - `pg_documentdb_core/include/io/helio_bson_core.h` and `pg_documentdb_core/include/query/helio_bson_compare.h`: Moved to `pg_documentdb_core/include/io/bson_core.h` and `pg_documentdb_core/include/query/bson_compare.h`. - `pg_documentdb_core/src/collation/collation.c` and `pg_documentdb_core/src/io/pgbson.c`: Replaced error codes and messages to use "DocumentDB" instead of "Helio". - `scripts/validate_helio_extension_state.sh`: Added exceptions for certain files to skip validation for "helio" references. |
||
|
Vinod Sridharan
|
3f9a507562 |
Merged PR 1539950: [OSS] Move helio_api and helio_distributed sources under documentdb
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactor ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md)) ### Description Move source files under new directory. ---- #### AI description (iteration 1) #### PR Classification Code reorganization #### PR Summary This pull request moves the `helio_api` and `helio_distributed` sources under the `documentdb` directory, updating references and initialization logic accordingly. - Added `pg_documentdb.c` for initializing the `pg_documentdb` extension. - Added `documentdb_distributed.c` for initializing the `pg_documentdb_distributed` extension. - Updated `validate_helio_extension_state.sh` to include new checks for `documentdb`. - Moved `helio_api_extension_version.c` to `documentdb_extension_version.c` without code changes. - Updated various `Makefile` configurations to reflect the new directory structure and dependencies. |
