mirror of
https://github.com/FerretDB/documentdb.git
synced 2026-11-06 11:02:50 -05:00
main
80
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Vinod Sridharan
|
8514e44ac2 |
Merged PR 1817415: Fix up planning cost estimate for order by
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Order by expressions have a filter for full scan that makes it so it can push down to the index. However the default selectivity of 1% kicks in at the time. Instead ensure that It has a selectivity of 100% Also fix list collections with distribution failing on views ---- #### AI description (iteration 1) #### PR Classification Bug fix addressing errors in BSON range operator selectivity and index pushdown behavior. #### PR Summary This PR fixes issues with full scan detection and selectivity handling in BSON range queries while enhancing index pushdown and metadata listing tests. The changes improve the correctness of query planning and operator behavior in DocumentDB integrations. - **`src/query/bson_dollar_selectivity.c` & `src/opclass/index_support.c`**: Added functions to detect full scans and correctly set selectivity (e.g., returning 1.0) for `$range` queries. - **Test Files (`bson_composite_order_by_index_tests.out`, `list_metadata_cursor_tests.out`, and SQL scripts)**: Extended tests for composite order-by indexes and distributed metadata, ensuring proper index selection and query plans. - **Configuration and Startup Scripts (`feature_flag_configs.c`, `start_oss_server.sh`)**: Introduced the `setSelectivityForFullScan` flag and updated extended RUM extension settings. - **SQL Migration Scripts**: Altered operator restrict definitions to use updated selectivity functions for consistent operator handling. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
2108af32e6 |
Merged PR 1812986: [Hidden Index] Support Hidden Indexes with Distribution
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add support for hidden indexes with distributed tables. - Add function that supports per node distribution with shards: Select all shards for the table and pick one shard per node. - Add a planner worker function that executes on every shard - On the shard skip execution if it's not the selected shard - Run the local function on the worker function. - Plumb in hooks to execute the index functions remotely. ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature to support hidden indexes in a distributed environment. #### PR Summary The changes integrate hidden index support into distributed index update operations by introducing new per-node functions, updating API hooks, and modifying planner and UDF logic for proper shard routing and logging. - **`node_distribution_operations.c`**: New file that adds functions (e.g., `UpdateDistributedPostgresIndex` and its worker) to execute index update commands per node in a distributed setting. - **`diagnostic_commands_common.c`**: Introduces `command_node_worker` to ensure that update commands target the appropriate shard based on table identity. - **`coll_mod.c` and `coll_mod.h`**: Updated to use `UpdatePostgresIndexWithOverride` and correctly incorporate hidden index modifications with distributed operations. - **API hooks and metadata files** (e.g., `api_hooks.c`, `api_hooks_def.h`, `api_hooks.h`, and `metadata_cache.c`): Enhanced to support new function pointers and caching mechanisms for hidden index updates via the new command worker. - **SQL UDF scripts**: New UDF definitions for `command_node_worker` and `update_postgres_index_worker` enable proper registration and execution of hidden index operations in the distributed framework. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
 Cornelius Boateng
|
abafc44b4f |
Merged PR 1804989: [update] Internal Funcs Should Have No Defaults Arg Values
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Remove default arg value from the new update function. Function has not yet been used in C code. ---- #### AI description (iteration 1) #### PR Classification This pull request is an API change that revises internal UDF functions for update operations by removing default argument values. #### PR Summary The changes introduce a new SQL file for bson update functions and update existing function signatures to eliminate default values—resulting in stricter parameter handling and an added parameter for V2 functions. - `oss/pg_documentdb/sql/udfs/commands_crud/bson_update_document--0.108-0.sql`: Added a new file defining the update functions. - `oss/pg_documentdb/sql/udfs/commands_crud/bson_update_document--latest.sql`: Modified the `__API_SCHEMA_INTERNAL_V2__.update_bson_document` function by removing default values from `arrayFilters` and `variableSpec` and adding the `collationString text` parameter. - `pgmongo/sql/pgmongo--1.107-0--1.108-0.sql` and `oss/pg_documentdb/sql/documentdb--0.107-0--0.108-0.sql`: Updated include directives to load the new update document SQL file. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
95d7a4b6a6 |
Merged PR 1806432: [OSS] Remove explicit RUM dependency on the documentdb extension
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactoring ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Make DocumentDB not explicitly depend on the RUM extension (it's done implicitly) on the binary. Instead pull in the SQL dependency into documentdb and reference the C functions internally. This allows us to swap between rum.so and extended_rum. For Pg18 (where there isn't yet official RUM support) default to using extended_rum for the .so file and remove all dependencies on the "rum.so" ---- #### AI description (iteration 2) #### PR Classification This PR is an API and dependency update that removes the explicit dependency on the external RUM extension by integrating its functionality internally. #### PR Summary The update integrates the RUM text-search capabilities directly into DocumentDB and cleans up the external linkage. - `oss/internal/pg_documentdb_extended_rum/src/rum_ts_utils.c` is newly added with core text-search functions. - `oss/pg_documentdb/src/index_am/rum.c` is modified to load internal wrappers that call the new RUM functions. - SQL UDF adapter files and operator class definitions are updated to reference the internal functions instead of public RUM functions. - Build scripts, metadata, and control files are adjusted to remove the explicit dependency on the external RUM extension. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
080ea66e2b |
Merged PR 1803246: [Hidden Index] Support for hidden indexes in DocumentDB
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add support for hidden indexes with collMod This will mark the index as not ready in the postgres catalog. Handles distributed setups as well. However, this is currently restricted to single node (sharded/unsharded). A subsequent change can help multi-node set ups. ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature by adding support for hidden indexes in DocumentDB. #### PR Summary The pull request extends the collMod command and associated index management logic to recognize and update a new “hidden” flag in index options, while enforcing constraints (e.g. disallowing hidden modifications for primary or unique indexes) in both single-node and distributed environments. - **`coll_mod.c`**: Introduces helper functions (e.g. GetHiddenFlagFromOptions, UpdateHiddenInIndexOptions, UpdatePostgresIndex) and SPI query updates to process the hidden flag properly. - **Test Files**: Adds and updates multiple test cases (e.g. in `commands_collmod_tests_hidden.out` and SQL test scripts) to verify hidden index creation, query planning, and error handling. - **Distributed Hooks and API**: Updates in files like `distributed_hooks.c` and `api_hooks.c` (with corresponding header changes) enable proper handling of hidden index updates across shard indexes in distributed environments. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
32b11ab18f |
Merged PR 1799781: [Reindex] Support internally handling Reindex with background index processor
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add infrastructure to handle REINDEX CONCURRENTLY using hte background index queue. This doesn't expose any new public API at this time, and is meant as a first pass to enable administrators to enqueue reindex concurrently in the background. Progress can be tracked by currentOp ---- #### AI description (iteration 2) #### PR Classification This PR introduces a new feature that adds asynchronous reindex operations using the background index processor. #### PR Summary This update extends the background index processor to support reindexing by implementing new C functions and updating index queue management, SQL UDF definitions, and tests. Key changes include: - `src/commands/create_indexes_background.c`: Added `command_reindex_index_background` and `command_reindex_index_background_internal` functions and refactored common command core logic. - `src/metadata/index.c`: Modified index queue functions and metadata handling to support the new REINDEX command type. - `sql/udfs/index_mgmt/create_index_background--0.108-0.sql`: Added new SQL UDFs for reindexing indexes. - Test files under `src/test/regress` and `/pgmongo/src/test/regress`: Updated to validate the background reindexing functionality. - Minor updates in logging and distributed hook functions to accommodate reindex operations in the background processor. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
 CosmosDB Build Service (msdata)
|
72fb39d9a9 |
Merged PR 1775074: [tagv.1.108] Update extension version for new release 1.108
This PR updates the extension version for the new release 1.108. Changes included: - Control files updated to version 1.107-0 - SQL files generated for version upgrades - Release notes file created for version 1.108 - TOC.yml updated with new release notes entry - CHANGELOG.md updated with new version entry |
||
|
Vinod Sridharan
|
52d51e19d3 |
Merged PR 1765581: RUM: Port support for skipping parts of the tree when scanning disjoint sets of the tree
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description When scanning disjoint sets of the tree e.g. on a composite index of (a, b, c) a query of a = 1 && c > 3 && c < 5 we have to scan for every value of B. In this case, we don't skip intermediate parts of hte tree that don't match. This adds support for ascending scans to skip portions of the tree that don't match. Since we don't know what next value is available, uses a new extensibility support function in the index to ask the operator class what the next should be. Additionally adds an option in comparePartial to return "-2" or "-3" which means doesn't match, but also has the potential to skip parts of the tree. This is fully backwards compatible with older RUM installs since the code just checks for < 0 vs > 0 for skipping vs termination. Use that to skip entries within the page when we know that the entry is not a match but something at a later point of the tree may be a match for the query. |
||
|
Vinod Sridharan
|
5289ac451b |
Merged PR 1758714: [Reverse Order] Schema changes to support reverse index order
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add schema mappings to support reverse order against the index This requires us to have a reverse order by operator and associated sort support. This registers the order by functions - the implementation wil be flushed out in a separate change. This is due to the fact that amorderbyop only supports order by forwards when added with index recheck. Index Recheck is required in our case since orderby on the index is lossy. To overcome this, we introduce a forward walk index operator and a reverse walk index operator as 2 separate operators that can be pushed to the index. ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new reverse ordering feature through API changes and database schema updates. #### PR Summary The changes introduce reverse order functionality for BSON queries by adding new C functions and corresponding SQL deployment scripts. - `pg_documentdb/src/query/bson_dollar_operators.c` adds `command_bson_orderby_reverse` and `bson_orderby_compare_sort_support` functions with appropriate PG function registrations. - SQL files under `/oss/pg_documentdb/sql/udfs/` and `/oss/pg_documentdb/sql/operators/` provide new UDFs and operators (including a reverse order operator and btree comparator support) to enable the feature. - Expected API schema files in both `/oss/pg_documentdb/src/test/regress/expected/` and `/pgmongo/src/test/regress/expected/` are updated to include the new functions. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
CosmosDB Build Service (msdata)
|
c02db96c0c |
Merged PR 1750749: [tagv.1.107] Update extension version for new release 1.107
This PR updates the extension version for the new release 1.107. Changes included: - Control files updated to version 1.106-0 - SQL files generated for version upgrades - Release notes file created for version 1.107 - TOC.yml updated with new release notes entry - CHANGELOG.md updated with new version entry |
||
|
Vinod Sridharan
|
623ef26f85 |
Merged PR 1742378: [Hints] Add support for index hints in DocumentDB
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Index hints are a mongo feature that allows customers to specify which index they would like to target for a query. Typically this is used in diagnostic scenarios when testing query performance. With the new composite index op class coming in, we’re introducing side by side support for composite which makes it so we would want customers to be able to test which index is better for planning. In this spirit, index hints is introduced which follows the same infra as the force pushdown for geonear and text and adds a new option for index hints. As part of this the code is also refactored to cleanup some of the logic in the relpathlist and fix correctness issues around text index force pushdown (which was needed to handle index hints) ---- #### AI description (iteration 1) #### PR Classification This pull request introduces a new feature to support index hints in DocumentDB. #### PR Summary The changes integrate index hint support into query planning and execution, allowing users to force a specific index during query processing. Key updates include: - **`src/opclass/index_support.c`**: Added new functions and a force index operator type (`ForceIndexOpType_IndexHint`) for filtering available indexes and generating custom operator expressions based on index hints. - **`src/planner/documents_planner.c`**: Modified base table query planning logic to detect index hints and adjust index pushdown, including updates to path and restriction handling. - New UDFs (e.g., `bson_dollar_index_hint`) and configuration variables (`EnableIndexHintSupport`, `UseLegacyForcePushdownBehavior`) were added to support the hint mechanism. - Test files and SQL scripts (e.g., in `/oss/internal/pg_documentdb_distributed/src/test/regress/`) were introduced or updated to validate index hint pushdown behavior. - Documentation and release notes (e.g., `docs/pgmongo/design_docs/data_plane/indexing/index_hints.md`, `pgmongo_1.106.md`, and `CHANGELOG.md`) were updated to describe the index hint feature. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 William Wang
|
e4024ad3ad |
Merged PR 1721770: [OSS] [Engine] Adding Support for Create Role Command and API Schema for Role CRUD
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) 105 ### Description As part of Full RBAC, we are adding support for role management CRUD operations. This PR adds support for the createRole Command and API Schema for dropRole, rolesInfo, updateRole commands Related work items: #4053600, #4338364 |
||
 Amit Prabhudesai
|
e25146b223 |
Merged PR 1739781: [OSS] Fix background worker tests
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Testing ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Fix OSS background worker tests ---- #### AI description (iteration 1) #### PR Classification This pull request fixes failing background worker tests by ensuring proper worker initialization and improving error handling. #### PR Summary The changes add waiting procedures in test helper SQL scripts to guarantee that background workers are fully launched before tests run, while also enhancing error logging and renaming configuration functions for clarity. - `documentdb_distributed_test_helpers.sql`, `documentdb_test_helpers.sql`, and `pgmongo_test_helpers.sql`: Introduced a stored procedure that loops until the appropriate background worker is detected, with corresponding calls added in the test scripts. - Expected output files (`documentdb_distributed_test_helpers.out`, `documentdb_test_helpers.out`, and `pgmongo_test_helpers.out`) now include these wait procedures to align with the new test logic. - `background_worker.c`: Modified the error handling in the background worker command generation to capture and log detailed error metadata (filename, line number, message_id). - `documentdb_init.c` and `background_job_configs.c` (along with header changes): Renamed and updated configuration functions from `InitDocumentDBBackgroundWorkerGucs` to `InitDocumentDBBackgroundWorkerConfigurations` and adjusted naming from `build_index_background` to `command_build_index_background`. - `postgresql.conf` files in multiple directories: Updated background worker settings by setting the background worker database name to 'regression'. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Cornelius Boateng
|
f983efd1da |
Merged PR 1683401: [engine][let] Support for update
### Does this PR have any customer impact? No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No
### Are you introducing any new config? If yes, do you have tests with and without them being set? No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Support let and `$$NOW` variables with update queries.
The values of the variables can be accessed in the query `q` with `$expr` and 'u' with an aggregation pipeline.
Example:
```
{
update: <collection>,
updates: [
{
q: {$expr: {$eq: ["$_id", "$$varRef"]}},
u: [{$set: {"a": "$$NOW"}}],
multi: false, // updateOne
}
],
let: {"varRef": 1},
...
}
```
We parse the let value (together with the time system variables - $$NOW) and propagate it through the aggregation and variable contexts.
To enable feature, turn on the `EnableVariablesSupportForWriteCommands` GUC.
----
#### AI description (iteration 1)
#### PR Classification
This PR implements a new feature by adding support for the "let" variable specification in update commands.
#### PR Summary
The changes integrate "let" support into update operations, allowing dynamic variable specifications to be applied during update candidate selection, serialization, and execution. This enhancement improves the flexibility and expressiveness of update queries in the engine.
- In `src/commands/update.c`, the code now parses and applies a new `variableSpec` (under the "let" field) when building update queries and managing SQL arguments.
- In `include/infrastructure/documentdb_plan_cache.h`, new plan cache macros are added to account for update candidates using let and collation.
- In `include/commands/update.h`, the `UpdateOneParams` structure is updated to include the `variableSpec` parameter.
- In `src/commands/delete.c`, references to the unsupported "let" field in delete commands are removed.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Related work items: #3423115, #3613875
|
||
 Duan Pei
|
c550bfba32 |
Merged PR 1688875: [Fix] LastN and BottomN Accumulator
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Issue1: Our lastN has some errors. When taking the last N elements, the original order is destroyed. For example: for[31,33,99,1], if want to get last3, it should be [33,99,1], our current lastN will get [1,99,33]: left is native mongo, right one is our pgmongo  Fix: When there is no sort, the result is reversed in the transit function. When there is sort, the result is reversed in the final function. Issue2: BottomN also has similar errors as mentioned above. left is native mongo, right one is our pgmongo  #### AI description (iteration 1) #### PR Classification Bug fix addressing incorrect accumulation logic for the LastN accumulator. #### PR Summary This PR fixes issues in the LastN accumulator by streamlining memory copy operations and adding robust error checking for BSON data integrity, while also correcting a misconfigured boolean flag in related functions. - `oss/pg_documentdb/src/aggregation/bson_sorted_accumulator.c`: Simplified the copy routine by replacing conditional branches with a unified memmove and added error checks to detect and report insufficient or corrupted BSON entries. - `oss/pg_documentdb/src/aggregation/bson_first_last.c`: Updated the `isLast` flag from true to false in both transition and combine functions to ensure correct accumulator behavior. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
09af8231c2 |
Merged PR 1733803: [OSS] Support documentdb_rum via index extensibility
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Support order and composite extensibility for documentdb_rum via index am extensibility in documentdb. ---- #### AI description (iteration 1) #### PR Classification This pull request introduces a new feature by adding support for documentdb_rum via index extensibility. #### PR Summary The changes integrate a new index access method for documentdb_rum, updating core indexing interfaces, schema definitions, and build/test scripts to enable extended RUM indexing functionality. Key updates include: - **`internal/pg_documentdb_rum/src/documentdb_rum_adapter.c`**: Introduces the new `documentdb_extended_rum` handler with functions for scan, build, insert, and others. - **`include/index_am/index_am_exports.h` and related utility files**: Add and update APIs for index extensibility (including multikey status support) to accommodate the new handler. - **`pg_documentdb/src/index_am/create_indexes.c`**: Modifies index creation logic to use new opclass schemas and integrate the extended rum index handler. - **SQL schema files (e.g., `documentdb_rum--1.0.sql` and operator class scripts)**: Define the new extension schema, operator classes, and functions required for extended RUM. - **Build scripts and regression test files**: Update Makefiles and expected test outputs to load, build, and validate the documentdb_rum extension functionality. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Parag Jain
|
60dc82a14c |
Merged PR 1727776: [oss] Fixing collection/empty_data_table API without creation_time
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Fixing collection/empty API for new tables which has 3 columns only. |
||
 Lucas Borges Fernandes
|
1cf6873426 |
Merged PR 1661393: Add OSS background_worker initial jobs framework
### Does this PR have any customer impact? No. ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature. ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No. ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes. ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add OSS background_worker initial jobs framework. For this initial version, only schedulable jobs are allowed (i.e. jobs that run on timed intervals). The background worker loop has a latch timeout with a predefined duration of 10 seconds that is configured through a GUC. Every time this timeout occurs, it iterates on the JobRegistry array. 1. Checks if the job is running and if it timed out (i.e. running longer than timeoutInSeconds). If yes, cancels the query and finishes libpq connection. 2. Checks if the job finished running (i.e. libpq connection isn't busy). If yes, finishes the libpq connection. 3. Runs the job if its scheduled interval has reached (i.e. currentTime - lastStartTime >= scheduledInterval). Each job execution has its own libpq connection to the database. In order to add a new job callers must pass a BackgroundWorkerJob structure to RegisterBackgroundWorkerJob. ---- #### AI description (iteration 1) #### PR Classification This pull request introduces a new feature by adding an OSS background worker jobs framework. #### PR Summary The changes implement a framework for scheduling, executing, and managing background jobs within the background worker process. This provides structured job processing with timeout, cancellation, and configuration support. - **`src/background_worker/background_worker.c`**: Added job management functions (e.g., ManageJobs, ExecuteJob, CanExecuteJob, CheckJobCompletion, CancelJobIfTimeIsUp, ValidateJob) and integrated job scheduling in the main loop. - **`include/background_worker/background_worker_job.h`**: Introduced new structures for defining background worker jobs and their execution objects. - **`src/api_hook_impl.c`**: Added a hook function to retrieve background worker jobs for registration. - **`src/configs/background_job_configs.c`**: Defined new configuration variables (GUCs) to enable background worker jobs. - **API Hooks and Connection Management Files**: Updated declarations and functions (e.g., in `include/api_hooks_def.h` and connection management files) to support connection handling necessary for job execution. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4225917 |
||
|
Vinod Sridharan
|
f7b8631c1f |
Merged PR 1726287: [documentdb_rum] More feature ports to the documentdb_rum index
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description This change ports more customizations of RUM into the documentdb_rum fork 1) Support Fast rumgettuple when there's no order by clauses or it's a simple order by. This ensures that we don't fetch all the rows, put it into a tuplestore, sort it and return it and instead just rely on the index to return it in TID order *OR* the raw index order if specified. 2) For Compare partial entries, instead of storing the FULL rum item that has the addInfo -> If there's no addinfo just store the tid in the tuplestore 3) Support order by on the raw index key operator ---- #### AI description (iteration 1) #### PR Classification This PR implements new ordering and scan functionalities and enhances partial-match processing in the documentdb_rum index. #### PR Summary The changes port additional features to the documentdb_rum index, introducing full support for order-by ordered scans, rechecking mechanisms, and improved handling of minimal tuple matches while updating configuration and testing outputs. Key changes include: - **`oss/internal/pg_documentdb_rum/src/rumget.c`**: Added new functions (e.g. for ordered scans, rechecking order-by keys, and minimal tuple handling) to support ordered index scans. - **`oss/internal/pg_documentdb_rum/src/rumscan.c`**: Implemented support for ordering entries, validated ordering function signatures, and enforced error conditions for multi-column order-by. - **`pg_documentdb_rum/src/pg_documentdb_rum.h`**: Updated scan structure definitions with new fields for ordered scan support and the simple scan toggle. - Updated output expectations (e.g. `innerScanLoops` values) and configuration (GUC variable `RumAllowOrderByRawKeys`) in related test scripts and utility files. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> ---- #### AI description (iteration 1) #### PR Classification This pull request introduces new feature ports to support ordered index scan (order‑by functionality) in the documentdb_rum index. #### PR Summary The changes add extensive support for ordered scans in the documentdb_rum index by porting and adapting functionality originally developed for pgmongo. High-level updates include: - **`/oss/internal/pg_documentdb_rum/src/rumget.c`**: New and updated functions (e.g. for moving scan pointers, obtaining ordered items, and preparing matched entries) enable order‑by processing during index scans. - **`/oss/internal/pg_documentdb_rum/src/rumscan.c`**: Enhancements in handling order‑by keys and validating ordering function signatures allow use of raw ordering keys, with error checks added for multicolumn indexes. -... |
||
 Deepak Bisht
|
19c65dd504 |
Merged PR 1712186: [SQL] update coments for all SQL upgrade files
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Comments ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description All the previous version SQL changes should be merged into this PR so that we can update the hashes in this PR ---- #### AI description (iteration 1) #### PR Classification This PR is a documentation update that cleans up inline SQL comments to align with the latest contributing guidelines. #### PR Summary The changes refine the SQL file comments for geospatial operator classes and functions by removing outdated references to Mongo indexing and validations. - `oss/pg_documentdb/sql/operators/bson_gist_geospatial_op_classes--0.16-0.sql`: Updated comments by replacing "mongo 2d index options" and "mongo 2dsphjere index options" with simplified descriptions. - `oss/pg_documentdb/sql/udfs/geospatial/bson_gist_extensibility_functions--0.16-0.sql` and `oss/pg_documentdb/sql/udfs/geospatial/bson_gist_extensibility_functions--latest.sql`: Revised function comments to remove "mongo db validations" for a more general description of geometry validations. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4354111 |
||
|
CosmosDB Build Service (msdata)
|
7c62c43e51 |
Merged PR 1720001: Update extension version for new release 1.106
This PR updates the extension version for the new release 0.106. Changes included: - Control files updated to new version - SQL files generated for version upgrades - CHANGELOG.md updated with new version entry |
||
|
Vinod Sridharan
|
91e673939e |
Merged PR 1715796: [OSS] Fix cursor cleanup cron job
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Bugfix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description The cron job for cursor cleanup on disk was broken since it was declared as a function but the call was done as a procedure. Fix the job command. ---- #### AI description (iteration 1) #### PR Classification This pull request fixes a bug related to the cursor cleanup cron job. #### PR Summary The changes address incorrect scheduling and command syntax for cursor cleanup tasks in different modules. - `/pgmongo/sql/pgmongo--1.104-1--1.105-0.sql`: Added a plpgsql block to conditionally schedule the `helio_cursor_cleanup_task` if it does not exist. - `/oss/pg_documentdb/sql/documentdb--0.104-0--0.104-1.sql`: Updated the cron job syntax for `documentdb_cursor_cleanup_task` to call the correct cleanup function. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Santiago Fernandez Madero
|
49dcd421df |
Merged PR 1705670: [OSS] fix 0.104-1 hotfix upgrade scripts
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description [OSS] fix 0.104-1 hotfix upgrade scripts ---- #### AI description (iteration 1) #### PR Classification Hotfix update to upgrade SQL upgrade scripts correcting version references. #### PR Summary This pull request updates SQL upgrade scripts to use the correct UDF and documentdb version numbers. - In `pgmongo/sql/pgmongo--1.103-0--1.103-1.sql`, the include directive was updated from version 0.105-0 to 0.104-1. - In `oss/pg_documentdb/sql/documentdb--0.104-0--0.104-1.sql`, the UDF include reference was changed to use version 0.104-1. - The file `oss/pg_documentdb/sql/documentdb--0.104-0--0.105-0.sql` was moved to `oss/pg_documentdb/sql/documentdb--0.104-1--0.105-0.sql` with no code changes. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
William Wang
|
d339399b36 |
Merged PR 1694202: Adding Support for Connection Status Command in OSS Engine
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes, the API schema ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) 1.05 ### Description We are adding support for the Connection Status command, which exists in native Mongo, and allows users to see their authenticated user, authenticated roles, and user privileges. For more information, visit connectionStatus.md. This PR is the second of three PRs for this feature. The first one added extensible role privileges to our `users_info` command, allowing users to retrieve role privileges with `showPrivileges: true`. This second PR adds a callable `connection_status` command to our API, that utilizes the role privileges. The last PR will add a gateway change that calls this `connection_status` command. Related work items: #4287246 |
||
|
Vinod Sridharan
|
ccc471e169 |
Merged PR 1703619: [Cursors] Improve stability of file based cursors
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description ---- #### AI description (iteration 1) #### PR Classification Enhancement and bug fix to improve the stability and management of file-based cursors. #### PR Summary This pull request refactors and bolsters file-based cursor management by introducing shared memory support, improved cleanup, error handling, and configuration changes. It also adds new API endpoints and logging for better cursor lifecycle control. - **`cursor_store.c` and related files:** Added shared memory functions (e.g., `FileCursorShmemSize`, `InitializeFileCursorShmem`), cursor count regulation (with `IncrementCursorCount`/`DecrementCursorCount`), and cleanup routines to ensure stable cursor file operations. - **Test and expected output files:** Updated regression tests and expected outputs to cover scenarios like duplicate cursor creation, cursor deletion limits, and error reporting when the file cursor capacity is reached. - **API enhancements:** Introduced new UDFs (e.g., `command_delete_cursors`) and tenant client methods for killing and deleting cursors, improving session management. - **Configuration updates:** Renamed and added GUC parameters (e.g., `MaxAllowedCursorIntermediateFileSizeMB`, `MaxCursorFileCount`, and cursor expiry settings) to control file-based cursor behavior. - **Logging and metrics improvements:** Enhanced logging in `mdsd_logger.c` to report current cursor counts and related metrics for operational insight. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Parag Jain
|
303bf37644 |
Merged PR 1699587: [update][cleanup] Removing unnecessary in/out from bson_update_document
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Removing unnecessary in/out from bson_update_document which were preset eariler due to older change_stream implementation. 1. For now moving udf to helio_api_internal and same time removing useless in/out params TODO: In next release will modify this for oss with updating test and minimal c changes will update --latest files also. ---- #### AI description (iteration 1) #### PR Classification This pull request is a code cleanup and update to simplify the integration of the `bson_update_document` function into OSS. #### PR Summary The PR introduces a new C-based SQL function for `bson_update_document` with streamlined parameters, updates the main SQL script to include this new file, and refreshes the API schema output to reflect the change. - `pgmongo/sql/udfs/crud/bson_update_document--1.105-0.sql`: Adds the new function definition with cleaned-up parameter handling. - `pgmongo/sql/pgmongo--1.104-0--1.105-0.sql`: Updates the script to include the newly added UDF file. - `pgmongo/src/test/regress/expected/public_api_schema.out`: Modifies the schema output to include the new function entry. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Santiago Fernandez Madero
|
e9f3864686 |
Merged PR 1695711: Add support of alternate index handler in 1.103
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add support of altrernate index handler in 1.103 |
||
|
Vinod Sridharan
|
c6bc2bcb73 |
Merged PR 1694458: [POC|Order by] Order by Index & Pushdown improvements
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description More general improvements to order by and composite: Create Indexes 1) Support creating a composite index side by side with a regular index 2) Block creation of descending or wildcard and unsupported set ups for composite indexes Planner: 1) Support ordering by the Nth path of an index if the prefixes are equality. 2) Validate that if it's not the first column, prefixes are provided 3) Support incremental sort with index pushdown. ---- #### AI description (iteration 1) #### PR Classification This pull request delivers a new feature and API enhancement to enable order-by index pushdown and refined composite index support. #### PR Summary The update streamlines composite index processing by introducing stricter equivalency checks, new UDF functions for full-scan order-by pushdown, and enhanced validations during index creation while updating explain outputs and tests. - `src/metadata/index.c` refines index options equivalency logic to correctly compare composite term settings. - `src/opclass/bson_gin_composite_entrypoint.c` and `src/index_am/rum.c` add functions to support order-by pushdown and manage duplicate entries efficiently. - `sql/udfs/query/bson_dollar_comparison--0.105-0.sql` (and related operator files) introduce new UDFs including `bson_dollar_fullscan` to enable full scan operations. - `src/commands/create_indexes.c` enforces new composite index validations (restricting use to ascending, non-wildcard regular indexes) with precise error reporting. - Test scripts and explain plan outputs in the `regress` and `pgmongo` directories are updated to confirm the new composite order-by behavior. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Shuai Tian
|
597852a2c6 |
Merged PR 1694353: Move $bucketAuto to schema 105
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Move bucketAuto to schema 105 ---- #### AI description (iteration 1) #### PR Classification This pull request introduces a schema migration for the `$bucketAuto` aggregation stage, moving its support from schema 104 to schema 105. #### PR Summary The changes update documentation, SQL scripts, and test configuration to reflect the migration of `$bucketAuto` support to the new schema. - `pgmongo/sql/pgmongo--1.104-0--1.105-0.sql`: Added inclusion of the new `$bucketAuto` UDF file for schema 105. - `oss/pg_documentdb/sql/documentdb--0.104-0--0.105-0.sql`: New file added to include `$bucketAuto` support for the updated schema. - `oss/CHANGELOG.md` and release notes files: Updated to move the `$bucketAuto` aggregation stage support from the previous schema version to the new release notes for schema 105. - Several test CSV files under `/gateway/managed/Tests.Integration.Shell.Postgres` were updated to reflect the new schema version. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Shuai Tian
|
17d4306a28 |
Merged PR 1422337: [pgmongo] Aggregate stage $bucketAuto implementation
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md)) ### Description This pull request adds documentation for the `$bucketAuto` aggregation stage in PostgreSQL, detailing its design, usage, and implementation. - Adds `bucketAuto.md` with details on the `$bucketAuto` aggregation stage - code implementation - support all granularity types: POWERSOF2, 1-2-5, Renard numbers, E series Related work items: #3234185 |
||
 Manish Gaur
|
2c5ab611d7 |
Merged PR 1692296: Version update for 104
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Version update for 104 Related work items: #4281831 |
||
|
Deepak Bisht
|
96924b8eee |
Merged PR 1672438: Support Compact in the vCore Backend engine
Yes Feature Yes No - Support compact database administration command behind the GUC `enableCompact`. - Add support for compact command in backend using VACUUM FULL which is a blocking operation. - Only one compact command can be run at a time on a specified collection. - VACUUM FULL is run using libpq in a top level transaction as it it not supported inside functions or procedures. - Add support of tracking compact command in current_op. - `dryRun` estimation is based on the query defined [here at pgexperts](https://github.com/pgexperts/pgx_scripts/blob/master/bloat/table_bloat_check.sql), this can be a rough estimation and doesn't provided details about the space reclaimed by vacuuming RUM indexes of the collection. - Read only users can't perform `compact` command. ---- New feature implementation to support the backend compact command. This PR adds complete backend support for a blocking compact command that estimates table bloat and conditionally performs a VACUUM FULL operation on collection tables. - `src/utils/storage_utils.c` and `include/utils/storage_utils.h`: Introduced a new bloat estimation utility returning collection bloat stats. - `src/commands/compact.c`: Implemented the compact command including BSON command parsing, dry-run handling, and conditional execution of VACUUM FULL. - `internal/pg_documentdb_distributed/src/test/regress/sql/command_compact.sql`: Added regression tests covering various compact command scenarios. - `include/utils/feature_counter.h` and `src/infrastructure/feature_counter.c`: Updated feature counter mapping to include the new compact command feature. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4237921 |
||
|
Vinod Sridharan
|
7658398f9b |
Merged PR 1679366: [POC] Persisted cursors files without holding connections
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) PoC/Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Today, persisted cursors require the connections to be held. This is bad since Connections/Backends are a limited resource. This introduces the option to save the cursor state in a file on disk and reuse that across backends. This is behind a GUC due to the presence of the following: 1) Needs more testing and validation 2) Needs backpressure to ensure that disk consumption doesn't balloon up 3) There is a procedure added to clean up these files but it is not wired up 4) We need to handle ultra-large responses (beyond 4 GB it fails outright at the moment). This needs more graceful handling to split into chunks. ---- #### AI description (iteration 1) #### PR Classification This pull request is a new feature implementation that introduces file‐based persisted cursors to allow query results to be “drained” and stored on disk without holding on to active database connections. #### PR Summary This PR adds support for persisting cursor state to the filesystem in both the pg_documentdb and pgmongo modules, enabling connections to be released while still allowing cursors to be later resumed. The changes include: - **`infrastructure/cursor_store.c` and `cursor_store.h`** – New APIs for creating, writing to, reading from, and closing persisted cursor files. - **Commands integration** – Modifications in `commands/cursors.c`, `commands/aggregation_commands.c`, and `cursor_private.h` to use file-based cursor functions for draining and continuation. - **SQL test suites** – Addition of new test files (e.g. in `.../regress/sql/bson_aggregation_cursor_tests_file.sql`) to validate cursor behavior across various query scenarios. - **Configuration updates** – New custom GUCs in `feature_flag_configs.c` and adjustments in test PostgreSQL configuration files to enable and control file-based persisted cursors. - **Executor enhancements** – Updates in cursor draining and query plan execution functions to support both in-memory and file-based cursor paging. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Cornelius Boateng
|
b204b58b82 |
Merged PR 1619783: [engine][collation] Support for sort in runtime
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes, yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Support sorting with collation. We define new custom sort operators which compare operands using applicable collation string. `<<<` is for ascending order. `>>>` is for descending order. Example: `SELECT * from <collection> ORDER BY <value> USING <<<` This will further be used for partitioning, grouping, and related comparisons. ---- #### AI description (iteration 1) #### PR Classification New feature: Support for collation in sort operations. #### PR Summary This pull request adds support for collation in sort operations within the database engine. - `src/test/regress/expected/bson_dollar_ops_collation_tests_runtime.out`: Added multiple test cases to verify collation behavior with various sort and order by operations, including case sensitivity and numeric ordering. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4014123 |
||
|
Deepak Bisht
|
a2876ad93a |
Merged PR 1644508: [Compact] GW plumbing for the compact command support
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description GW plumbing of compact command 1- Route the compact command to the backend. 2- Add a dummy C handler function for `compact` that does nothing and throws the same error. Next PR is going to add support for compact in the backend with blocking writes by running VACUUM FULL ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request adds support for the `compact` command in the MongoDB Postgres runtime. - Added `CompactAsync` method in `PostgresClient.cs` to handle the compact command. - Implemented `command_compact` function in the new file `compact.c`. - Updated `PostgresMongoClient.cs` to include the `CompactAsync` method. - Modified `MongoRequestParser.cs` to parse the compact command. - Added SQL function definitions for the compact command in `compact--0.103-0.sql` and `compact--latest.sql`. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Deepak Bisht
|
8b7b93ced7 |
Merged PR 1578129: [Pending release cut 1.102]- Fix projection with bson_dollar_merge_documents for multiple scrnarios
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) BugFix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) Bugfix to override nested arrays in dotted path in `bson_dollar_merge_documents` UDF for few cases ### Description bson_dollar_merge_documents function today recurses into nested arrays always and project in each element of array but that is not correct with mongo semantics. This is only applicable if the projectetion is a dotted path. The function is used in 7 different stages/features today: 1- list_collection: no dotted paths for projection, so this is okay to follow any projection. 2- graphlookup: Mongo does a shallow projection in this case. 3- lookup: Mongo does a shallow projection in this case. 4- inverseMatch: okay as no dotted path projection 5- count: okay as no dotted path projection 6- group : okay as no dotted paths are allowed 7- setWindowFields: only this stage does a full projection even inside nested arrays ---- #### AI description (iteration 1) #### PR Classification Bug fix #### PR Summary This pull request addresses multiple scenarios for fixing projection with `bson_dollar_merge_documents`. - Added new functions in `oss/pg_documentdb/sql/udfs/projection/bson_projection--0.102-0.sql` to handle various BSON document operations. - Updated `bson_dollar_merge_documents` function to include an `overrideArray` parameter. - Modified test expectations in `oss/internal/pg_documentdb_distributed/src/test/regress/expected/bson_aggregation_pipeline_tests_graphlookup.out` to reflect changes in the `bson_dollar_merge_documents` function. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Amit Prabhudesai
|
84ba0c99d8 |
Merged PR 1572685: [OSS] Fix background index builds for documentdb
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Bugfix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Get background index builds to work for single-node documentdb. This changeset fixes background index builds for OSS by: 1. Scheduling the background index build tasks 2. Moving the distributed functionality (which had not being refactored out of the indexing code during the move to OSS) into distributed hooks, with the base implementation for single-node. 3. Refactoring SQL tests to get even with the above changes. ---- #### AI description (iteration 1) #### PR Classification Bug fix for background index builds in DocumentDB. #### PR Summary This pull request addresses issues with background index builds in DocumentDB by adding comprehensive tests and handling various edge cases. - `commands_create_indexes_background.out`: Added extensive test cases to validate the behavior of `create_indexes_background` function, including handling of invalid inputs, unique constraints, and various index specifications. - Introduced a new schema and function to manage index job statuses. - Enhanced error handling and validation for index creation requests, ensuring proper error messages and statuses are returned for invalid operations. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Deepak Bisht
|
cd19288dbc |
Merged PR 1647200: [Perf]- Point read improvement, remove PG inference of type and create the tuple descriptor directly
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Perf ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description - add cursor result type for UDFs, this helps PG to not always try to figure out the `OUT` variables and form a tuple descriptor  ---- #### AI description (iteration 1) #### PR Classification Performance improvement and API change. #### PR Summary This pull request introduces performance improvements for point reads by adding a new cursor type for SQL functions. It modifies existing functions to return a structured type instead of a record, enhancing efficiency and clarity. - `query_cursors_aggregate--latest.sql`: Added `cursor_result` type and updated several functions to return this type instead of a record. - `query_cursors_aggregate--0.103-0.sql`: Newly added file with similar changes as `query_cursors_aggregate--latest.sql`. - `documentdb--0.102-1--0.103-0.sql` and `pgmongo--1.102-1--1.103-0.sql`: Updated to include the new `query_cursors_aggregate--0.103-0.sql` file. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4157027 |
||
|
Manish Gaur
|
c7e25699df |
Merged PR 1656145: version update for 103 release.
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description ---- #### AI description (iteration 1) #### PR Classification Version update for the release of version 103. #### PR Summary This pull request updates the version numbers across several control files and adds release notes for version 1.103 and 1.104. It also includes changes to SQL files and their corresponding SHA1 checksums. - `/oss/internal/pg_documentdb_distributed/documentdb_distributed.control`, `/oss/pg_documentdb/documentdb.control`, `/oss/pg_documentdb_core/documentdb_core.control`, and `/pgmongo/pgmongo.control`: Updated default version numbers. - `/docs/release_notes/pgmongo_1.103.md` and `/docs/release_notes/pgmongo_1.104.md`: Added release notes for versions 1.103 and 1.104. - `/pgmongo/sql/pgmongo--1.103-1--1.104-1.sql`: Added new SQL file for version transition. - `/docs/release_notes/toc.yml`: Updated table of contents to include new release notes. - `/oss/CHANGELOG.md`: Added entry for documentdb v0.104-0. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4140262 |
||
|
Parag Jain
|
4ed646dfb9 |
Merged PR 1647233: [Bulk Insert][Perf] support for bulk insert via procedures
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Currently bulk insert run as a function which means that the entire execution runs as a single transaction. Each batch insert is handled as a subtransaction and rolled back to a save point if it fails. This means that document locks are held for longer, and we can hit limits w.r.t. max documents updated per transaction. In multi-node setups this also means distributed transactions are held longer causing loss of connection pools. This change introduces a stored procedure for bulk insert that periodically "Commits" the state to WAL and starts a new transaction. This more mirrors what the non-atomic bulk update is supposed to do. The old function is retained to be used within transactions. ---- #### AI description (iteration 1) #### PR Classification Performance improvement for write commands. #### PR Summary This pull request introduces a new procedure for handling write commands, enhancing performance by optimizing user-specific command execution. It includes the addition of new functions and procedures to manage MongoDB operations more efficiently. - Added new SQL functions and procedures in `/pgmongo/sql/udfs/gateway_funcs/command_funcs--1.103-0.sql` for running commands, queries, DDL, and write operations as a specific user. - Introduced new insert functions and procedures in `/oss/pg_documentdb/sql/udfs/commands_crud/insert--0.103-0.sql` to handle MongoDB insert operations. - Implemented `pgmongo_run_write_procedure_as_user` and `ExecuteWriteCommandCore` in `/pgmongo/src/infrastructure/command_funcs.c` to manage write commands with user context. - Added `CommitOuterInsertTransactions` function in `/oss/pg_documentdb/src/commands/insert.c` to handle transaction commits outside of transaction blocks. - Introduced a feature flag `EnableWriteProcedure` in `/oss/pg_documentdb/src/configs/feature_flag_configs.c` to control the enabling of write procedures. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Cornelius Boateng
|
bec1aa3562 |
Merged PR 1638286: [engine][collation][enhancement] Extend bson_expression_get for collation support
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add a new bson_expression_get that appends the collationString to the got value, if a valid collationString is passed to it. This will be used as a helper function to operations that rely on bson_expression_get to pass down collation such as bson_lookup_join_filter. If no collation string is passed as an arg, we simply return the got value. ---- #### AI description (iteration 1) #### PR Classification New feature to support `$graphLookup` in the engine with collation. #### PR Summary This pull request adds support for `$graphLookup` with collation in the engine. - Added new functions in `oss/pg_documentdb/sql/udfs/aggregation/bson_lookup_functions--0.103-0.sql` for `$graphLookup` operations. - Modified `oss/pg_documentdb/src/aggregation/bson_aggregation_nested_pipeline.c` to handle collation in `$graphLookup` join filters. - Updated `oss/pg_documentdb/src/operators/bson_expression.c` to remove collation handling from `bson_expression_get`. - Added `BsonDollarLookupJoinFilterWithCollationFunctionOid` in `oss/pg_documentdb/src/metadata/metadata_cache.c`. - Adjusted collation handling in `oss/pg_documentdb/src/query/bson_dollar_operators.c` for `$graphLookup`. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4104899 |
||
|
Vinod Sridharan
|
36bb3e2920 |
Merged PR 1650400: [Bulk Update] Setup bulk update infrastructure
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Currently bulk updates run as a function which means that the entire execution runs as a single transaction. Each batch update is handled as a subtransaction and rolled back to a save point if it fails. This means that document locks are held for longer, and we can hit limits w.r.t. max documents updated per transaction. In multi-node setups this also means distributed transactions are held longer causing loss of connection pools. This change introduces a stored procedure for bulk update that periodically "Commits" the state to WAL and starts a new transaction. This more mirrors what the non-atomic bulk update is supposed to do. The old function is retained to be used within transactions. Still TODO: For unsharded collections in a multi-node setup, we need to send batches of updates to the worker. This is not wired up in this change. ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request introduces a new bulk update feature for handling non-transactional updates in the MongoDB wire-protocol. It adds new functions and procedures to support bulk updates and modifies existing update logic to accommodate this new functionality. - Added `update_bulk` procedure and related functions in `sql/udfs/commands_crud/update--0.103-0.sql` for non-transactional bulk updates. - Implemented `command_update_bulk` in `src/commands/update.c` to handle bulk updates outside of transactions. - Enhanced `ProcessBatchUpdateCore` in `src/commands/update.c` to support transactional and non-transactional updates. - Updated feature tracking in `include/utils/feature_counter.h` and `src/infrastructure/feature_counter.c` to include `FEATURE_COMMAND_UPDATE_BULK`. - Modified SQL scripts `documentdb--0.102-1--0.103-0.sql` and `pgmongo--1.102-1--1.103-0.sql` to include the new update functionality. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
5ad889feeb |
Merged PR 1629302: [Composite POC] Fix up operators and type bracketing
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description 1) Fix up the remaining operators - implementations are sub-optimal 2) Add type bracketing options for all types and use them in operators 3) Add support for truncation of terms TODO: - Support recheck on the index for operators than just bounds - Support N boundaries for index terms. ---- #### AI description (iteration 1) #### PR Classification Bug fix #### PR Summary This pull request addresses issues with operators and type bracketing in BSON composite indexes. - Added `bson_gin_index_types_core.c` to manage BSON type bounds for indexes. - Updated `bson_gin_composite_entrypoint.c` to fix operator handling and apply type bracketing for various BSON index strategies. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
db6a3de7fd |
Merged PR 1627378: [Composite POC] Create single composite operator class
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description As part of testing out the composite operator classes, introduce a new composite operator class for Bson. This will be behind a test only GUC and will be for testing composite index pushdown. ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request introduces a new composite operator class for BSON GIN indexes in PostgreSQL. - Added `bson_gin_composite_entrypoint.c` to implement GIN operator functions for BSON composite indexes. - Created test cases in `bson_composite_index_tests_insert_terms.out` to validate term generation for composite paths. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
2c07ff0014 |
Merged PR 1625609: [DocumentDB] Add documentdb version 0.102-1 hotfix that is an idempotent upgrade of 0.102-0
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Bugfix(ish) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description 0.102-0 had changes that are non-idempotent. Given there were some incorrect dependencies taken on the 0.102 release (independently) introduce a 0.102-1 version that applies all the delta of 0.102-0 in an idempotent manner so that dependencies taken partially on 0.102 and dependencies taken on 0.102 fully work side-by-side. ---- #### AI description (iteration 1) #### PR Classification Hotfix for DocumentDB version upgrade. #### PR Summary This pull request introduces a hotfix for DocumentDB, ensuring an idempotent upgrade from version 0.102-0 to 0.102-1. - Added `documentdb--0.102-0--0.102-1.sql` to handle partial upgrades and repeat UDF upgrades. - Modified `data_table_utils.c` to check for both 0.102-0 and 0.102-1 versions during data table upgrades. - Updated `pgmongo.control` to set the default version to 1.102-1. - Added `pgmongo--1.102-1--1.103-0.sql` for version transition. - Updated SHA1 checksums in multiple `pgmongo/sql/sha1` files to reflect changes. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Manish Gaur
|
580a0a89b6 |
Merged PR 1615448: Version update for 102 release
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Version update for 102 release |
||
|
Vinod Sridharan
|
f31e0b6cad |
Merged PR 1615832: Support for pushdown of in to PFE indexes
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Support for pushdown of to PFE indexes ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request adds support for pushdown operations to PFE indexes, enhancing query performance and index utilization. - Added new functions in `sql/udfs/index_mgmt/bson_indexbounds_functions--0.102-0.sql` for BSON index bounds operations. - Introduced new operators in `sql/operators/bson_btree_pfe_operators--0.102-0.sql` for BSON and BSON query comparisons. - Updated `src/query/query_operator.c` to handle new index bounds operators and expressions. - Added new type definitions in `sql/types/bsonindexbounds--0.102-0.sql` for BSON index bounds. - Modified `src/metadata/metadata_cache.c` to include OIDs and functions for BSON index bounds types and operators. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Parag Jain
|
80df5a1509 |
Merged PR 1604750: [perf][creation_time] Alter creation time : part 1
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Altering creation_time column removing not null and default constraint. ---- #### AI description (iteration 1) #### PR Classification Performance improvement #### PR Summary This pull request alters the `creation_time` column in the database schema to improve performance. - Added `AlterCreationTime` function in `cluster_operations.c` to modify the `creation_time` column. - Updated `SetupCluster` function in `cluster_operations.c` to call `AlterCreationTime`. - Created new SQL scripts `extension_upgrade_alter_creation_time_1.102-0.sql` and `extension_upgrade_alter_creation_time_1.102-0.out` for testing the changes. - Modified multiple test output files to reflect the changes in the `creation_time` column constraints. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
b405e3d291 |
Merged PR 1610601: [Cleanup] Code Cleanups on Versions and DocumentDB
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactoring ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description 1) Remove all Citus references from pg_documentdb tests 2) Remove some citus references from documentdb c files (Comments) 3) Remove dead code for bson_out aggregates 4) Remove all References to IsClusterVersionAtLeast less than 0.24 (since those are now deployed) ---- #### AI description (iteration 1) #### PR Classification Code cleanup #### PR Summary This pull request introduces code cleanups and enhancements related to aggregation functions in DocumentDB. - `oss/pg_documentdb/sql/udfs/aggregation/group_aggregates--0.102-0.sql`: Added new aggregation functions for BSON data types, including BSONSUM, BSONAVERAGE, BSONMAX, BSONMIN, BSONFIRST, BSONLAST, BSONFIRSTN, BSONLASTN, BSON_ARRAY_AGG, BSON_OBJECT_AGG, BSON_ADD_TO_SET, BSON_MERGE_OBJECTS, BSONSTDDEVPOP, BSONSTDDEVSAMP, BSONPERCENTILE, and BSONMEDIAN. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Cornelius Boateng
|
98a5603d64 |
Merged PR 1594069: [engine][collation] Support for find
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Support collation with `find` queries. ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request adds support for the `project find` functionality with collation in the engine. - Added new SQL functions in `sql/udfs/projection/bson_projection--0.102-0.sql` for handling BSON projection with collation. - Updated `metadata_cache.c` to include OID caching for the new `bson_dollar_project_find` function with collation. - Modified `bson_project.c` to handle additional arguments for collation in the `bson_dollar_project_find` function. - Updated `bson_projection--latest.sql` to include the new function definition. - Added function declaration in `metadata_cache.h` for the new `bson_dollar_project_find` function with collation. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #3924476 |
