mirror of
https://github.com/FerretDB/documentdb.git
synced 2026-11-07 11:32:47 -05:00
main
411
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Vinod Sridharan
|
84b999ae88 |
Merged PR 1815453: [Composite] Support optimized elemMatch pushdown
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
### Description
Improve pushdown of $elemMatch for composite indexes.
When there's an elemMatch of the form `{ "a": { "$elemMatch": { "$gt": 10, "$lt": 15 }}`
This can scan the index only from 10 -> 15
However given the multi-key index requirement we were doing 2 scans:
- One from 10 to infinity and the other from -infinity to 15.
Optimize this path for $elemMatch.
----
#### AI description (iteration 1)
#### PR Classification
This PR introduces a new feature that optimizes the pushdown of `$elemMatch` queries for composite indexes.
#### PR Summary
The pull request implements optimized handling of `$elemMatch` predicates on composite indexes by converting them into specialized index pushdown expressions and updating the corresponding test expectations.
- **`oss/pg_documentdb/src/opclass/bson_gin_composite_core.c` & `index_support.c`**: Added logic to detect `$elemMatch` in range queries and convert them into optimized pushdown expressions (via functions like `GetElemMatchIndexPushdownOperator`).
- **Configuration Changes**: Introduced a new GUC (`documentdb.useNewElemMatchIndexOperatorOnPushdown`) in `pg_documentdb/src/configs/feature_flag_configs.c` to control the new behavior.
- **Test Updates**: Modified expected explain outputs in both `/oss/internal/pg_documentdb_distributed/src/test/regress/` and `/pgmongo/src/test/docdb_compat/expected/` to reflect the revised index condition format using `elemMatchIndexOp`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
|
||
|
Vinod Sridharan
|
8514e44ac2 |
Merged PR 1817415: Fix up planning cost estimate for order by
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Order by expressions have a filter for full scan that makes it so it can push down to the index. However the default selectivity of 1% kicks in at the time. Instead ensure that It has a selectivity of 100% Also fix list collections with distribution failing on views ---- #### AI description (iteration 1) #### PR Classification Bug fix addressing errors in BSON range operator selectivity and index pushdown behavior. #### PR Summary This PR fixes issues with full scan detection and selectivity handling in BSON range queries while enhancing index pushdown and metadata listing tests. The changes improve the correctness of query planning and operator behavior in DocumentDB integrations. - **`src/query/bson_dollar_selectivity.c` & `src/opclass/index_support.c`**: Added functions to detect full scans and correctly set selectivity (e.g., returning 1.0) for `$range` queries. - **Test Files (`bson_composite_order_by_index_tests.out`, `list_metadata_cursor_tests.out`, and SQL scripts)**: Extended tests for composite order-by indexes and distributed metadata, ensuring proper index selection and query plans. - **Configuration and Startup Scripts (`feature_flag_configs.c`, `start_oss_server.sh`)**: Introduced the `setSelectivityForFullScan` flag and updated extended RUM extension settings. - **SQL Migration Scripts**: Altered operator restrict definitions to use updated selectivity functions for consistent operator handling. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
b38e58c455 |
Merged PR 1818535: Move the PG18 build container to the latest image
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) DevOps ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Move the PG18 build container to the latest image ---- #### AI description (iteration 1) #### PR Classification Configuration update for build container images. #### PR Summary This pull request updates the PG18 build container image to use the latest normalized tag, ensuring consistency with the PG17 configuration. - Updated ``/.pipelines/templates/common_variables.yml`` by replacing the hard-coded tag for PG18 with ``variables.NormalizedContainerTag``. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
14b27c08e9 |
Merged PR 1817813: [ExtendedRum] Move pg_documentdb_extended_rum to the top level folder
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Move pg_documentdb_extended_rum to the top level folder from the internal folder since documentdb (tests and product) now has a dependency on it. pg_documentdb_extended_rum is a fork of the RUM Extension that provides extensions that were needed for documentdb. This includes the ability to push down order by and group by to the index; The ability to do index only scans; improvements on vacuum and handling of page splits; index scan improvements while retaining the original inverted index for text search capabilities that the RUM index had. Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
 Santiago Fernandez Madero
|
6f19dbec81 |
Merged PR 1817238: TTL fix descending and continous purging
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Continous TTL was not taking into consideration the max continous timeout but instead the single ttl task so it was always capped at 20 seconds. TTL descending needed to use the order by function and fullrange so that it goes through the order by push down logic rather than PGs logic. Also, add an entry to the log to indicate if it is descending or not. ---- #### AI description (iteration 1) #### PR Classification This PR fixes TTL purging issues by correcting the descending sort behavior and ensuring continuous purging. #### PR Summary The changes update TTL deletion logic to use descending order when enabled and adjust associated logging and configuration options. This enhances purging consistency and aligns test expectations. - `pgmongo/src/test/regress/expected/pgmongo_internal_ttl_tests.out` and related test outputs now include the `use_desc_sort` flag in log messages. - `oss/pg_documentdb_distributed/src/test/regress/expected/commands_create_ttl_indexes.out` and SQL test files have been modified to set descending sort and disable sequential scans. - `oss/pg_documentdb/src/ttl/ttl_index.c` has been updated to conditionally append descending order queries using index orderby pushdown. - Configuration files such as `oss/pg_documentdb/src/configs/background_job_configs.c` now define and enable TTL descending sort options (e.g., `EnableTTLDescSort` and `EnableIndexOrderbyPushdown`). <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
61980a6b10 |
Merged PR 1816565: [RUM] Turn on Parallel Build & Other GUCs by default
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Turn on some Vacuum cleanup GUCs and Parallel build GUCs by default for RUM ---- #### AI description (iteration 1) #### PR Classification This pull request is a configuration update that enables key RUM features by adjusting default GUC settings. #### PR Summary The changes update default settings in RUM header files to enable parallel index building and vacuuming of entry items. - `pgmongo_rum/src/rum.h`: Changed `RUM_DEFAULT_ENABLE_PARALLEL_INDEX_BUILD` from false to true and `RUM_DEFAULT_VACUUM_ENTRY_ITEMS` from false to true. - `oss/internal/pg_documentdb_extended_rum/src/pg_documentdb_rum.h`: Changed `RUM_DEFAULT_ENABLE_PARALLEL_INDEX_BUILD` from false to true. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
2108af32e6 |
Merged PR 1812986: [Hidden Index] Support Hidden Indexes with Distribution
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add support for hidden indexes with distributed tables. - Add function that supports per node distribution with shards: Select all shards for the table and pick one shard per node. - Add a planner worker function that executes on every shard - On the shard skip execution if it's not the selected shard - Run the local function on the worker function. - Plumb in hooks to execute the index functions remotely. ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature to support hidden indexes in a distributed environment. #### PR Summary The changes integrate hidden index support into distributed index update operations by introducing new per-node functions, updating API hooks, and modifying planner and UDF logic for proper shard routing and logging. - **`node_distribution_operations.c`**: New file that adds functions (e.g., `UpdateDistributedPostgresIndex` and its worker) to execute index update commands per node in a distributed setting. - **`diagnostic_commands_common.c`**: Introduces `command_node_worker` to ensure that update commands target the appropriate shard based on table identity. - **`coll_mod.c` and `coll_mod.h`**: Updated to use `UpdatePostgresIndexWithOverride` and correctly incorporate hidden index modifications with distributed operations. - **API hooks and metadata files** (e.g., `api_hooks.c`, `api_hooks_def.h`, `api_hooks.h`, and `metadata_cache.c`): Enhanced to support new function pointers and caching mechanisms for hidden index updates via the new command worker. - **SQL UDF scripts**: New UDF definitions for `command_node_worker` and `update_postgres_index_worker` enable proper registration and execution of hidden index operations in the distributed framework. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
4a38f05454 |
Merged PR 1813645: Fix up currentOp, index stats logging for Parallel Builds & a few index fixes
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature/Bugfix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description - Add tracking of parallel workers in currentOp. - Ensure index build progress works for blocking index builds - Fix crash in parallel rum index builds - Allow RUM parallel index build path even for serial build (so we can unify to one codepath) - Turn on amcanparalllelbuild for PG >= 17 based on the GUC ---- #### AI description (iteration 1) #### PR Classification Bug fix addressing logging inaccuracies for parallel build operations. #### PR Summary This pull request improves logging for current operations and index build progress by incorporating additional metadata and refining SQL queries to better handle parallel workers. - In `oss/pg_documentdb/src/commands/current_op.c`, new attributes (backend type and leader PID) are extracted from query results and appended to BSON output for accurate identification of parallel workers. - In `oss/pg_documentdb/src/commands/current_op.c`, progress logging is enhanced by including a "concurrent" flag and updating phase-to-message mappings for index build initialization. - In `pgmongo/src/infrastructure/index_stats_logging.c`, the index progress query now uses left joins with COALESCE to reliably capture index and table names, improving stats logging for parallel builds. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Santiago Fernandez Madero
|
9ac8147799 |
Merged PR 1786779: [background unique build 1/3] Create index as unique support
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description This new option for create indexes creates an index as if it was a UNIQUE composite term index, but without actually creating the constraint on the table, so it is a regular index. This is the first phase of 3 needed. We introduce a three-step workflow that lets customers build the index structure in the background (non-blocking), then progressively enable uniqueness enforcement: 1. buildAsUnique (createIndex) — build the index as if unique but without registering any table constraint (non-blocking) 2. prepareUnique (collMod) — register the uniqueness constraint in the system catalogs but mark it unvalidated so it starts enforcing uniqueness for new writes only 3. convertUnique (collMod) — validate/enforce uniqueness for existing rows (walk the table or short-circuit if an existing validated unique index already proves uniqueness) ## Goals - Allow background construction of unique-capable indexes without blocking writes - Provide a fast metadata-only step to enable uniqueness for new writes - Provide a validation step that enforces uniqueness for pre-existing rows only when it is safe - Preserve data integrity and PostgreSQL metadata correctness - Expose these steps through familiar MongoDB commands (`createIndex` and `collMod`) - Migrating non ordered unique indexes to ordered indexes without blocking production writes. ## High-level Workflow Step 1 — buildAsUnique (createIndex): - Customer requests an index with storageEngine.buildAsUnique=true. - The system creates a regular index physically (non-blocking) but records metadata indicating the index is intended to become unique Step 2 — prepareUnique (collMod): - Customer requests convert-to-unique preparation via `collMod` (prepareUnique) - The system registers a constraint in the PostgreSQL catalogs but does not validate it against existing data; it enforces uniqueness on new writes going forward - This is a fast, mostly metadata-only operation (convalidated = false) Step 3 — convertUnique (collMod): - Customer requests final conversion via `collMod` (convertUnique) - The system validates uniqueness for existing rows by walking the table (or aborts if duplicates are found), or it can skip the full walk when an existing validated unique index already proves uniqueness ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature that adds support for the "prepareForUnique" option in index creation. #### PR Summary The changes extend the index creation mechanism to allow a non-table unique index to be prepared via the "prepareForUnique" flag, with appropriate error handling and ... Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
 Deepak Bisht
|
1a695bdf9f |
Merged PR 1811981: [Observability] Add feature counters for all query operators
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Observability ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add feature counters for all public query operators ---- #### AI description (iteration 1) #### PR Classification This PR implements a new feature by adding comprehensive feature counters for all query operators to enhance observability. #### PR Summary The changes extend the observability framework by introducing new feature counter definitions, mappings, and usage in query operator implementations. - Updated `oss/pg_documentdb/include/utils/feature_counter.h` and `oss/pg_documentdb/src/infrastructure/feature_counter.c` to add new query operator feature enum values and corresponding string mappings with an increased `MAX_FEATURE_COUNT`. - Modified `oss/pg_documentdb/src/planner/mongo_query_operator.c` to assign the new feature counters to each query operator. - Revised test expected outputs in `/oss/internal/pg_documentdb_distributed/src/test/regress/expected/feature_counters.out` and `/pgmongo/src/test/docdb_compat/expected/feature_counters.out` to reflect the updated counters. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Deepak Bisht
|
9cec4df623 |
Merged PR 1807532: Crash Dump fix while building aggregate queries with $group aggregates inside $lookup
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) BugFix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description - Today before building the complete pipeline query of lookup we initialize the varlevelsup of `Var` to be set to INT_MAX but if there is an aggregate such as $sum it then tries to get the parseState of those many levels of parent and hence crashes out. The fix is we earlier set varlevelsup to int max so that we can uniquely identify these Vars after building the query for pipeline but instead of using varlevelsup now we use location field to uniquely identify these Vars and adjust the levelsup later. ---- #### AI description (iteration 1) #### PR Classification This pull request implements a bug fix to resolve a crash when building aggregate queries that include $group aggregates within $lookup. #### PR Summary The changes introduce a new test case to validate the correct query plan for lookup queries with aggregates and adjust internal variable handling in the aggregation pipeline builder to prevent crash dumps. - **`bson_aggregation_nested_pipeline.c`**: Updated variable resolution by setting `varlevelsup` to 0 and adjusting `location` values to ensure proper handling in nested pipeline contexts. - **Test files (`bson_aggregation_stage_lookup_pg17_tests.out`, `bson_aggregation_stage_lookup_tests.out`, and related SQL tests)**: Added new expected outputs for lookup queries incorporating aggregates to verify the fix across different test suites. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4114368, #4696693 Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
96cf82521d |
Merged PR 1809973: [RUM] Make more code sync between internal and documentdb_rum
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Perf/Refactoring ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description 1) Delete the old vacuum posting tree path (it's been unused for a year) 2) Port in more changes to minimize delta between internal and documentdb_rum - enable pruning empty entries in the entry tree. ---- #### AI description (iteration 1) #### PR Classification This pull request performs a code cleanup and synchronization of vacuum routines for RUM indexes across internal and documentdb_rum modules. #### PR Summary This update refactors the vacuum functionality by removing legacy functions and unifying the cleanup logic for empty entries and posting trees, while also enhancing logging and configuration consistency. Key changes include: - **`/oss/internal/pg_documentdb_extended_rum/src/rumvacuum.c`**: Removed obsolete functions (e.g., rumVacuumPostingTreeLeaves and rumVacuumPostingTree) and refactored vacuum routines to use the new rumCleanupEmptyEntries mechanism. - **`/pgmongo_rum/src/rumvacuum.c`**: Made similar removals and simplifications in vacuum scanning, streamlining bulk delete processing and ensuring consistent handling of empty entries. - **Logging Enhancements**: Updated bulk delete routines to report detailed metrics (empty pages, empty entries, pruned pages/entries, etc.) using unredacted logging. - **GUC and Header Updates**: Modified custom variable definitions in `rumutil.c` and adjusted header files (`pg_documentdb_rum.h` and `rum.h`) to remove deprecated flags and introduce a new vacuum cleanup entries setting. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
f46a64e6dd |
Merged PR 1809207: [RUM] Port more changes from internal -> documentdb_rum
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactoring ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Reduce deltas between private and documentdb_rum - port more changes between internal and documentdb_rum. ---- #### AI description (iteration 1) #### PR Classification This pull request ports internal changes to the documentdb_rum extension, introducing new unredacted logging hook integration and enhanced scan logic. #### PR Summary The changes integrate a new hook setter for unredacted logging, add support for pre-consistent scanning using function pointers, and update TS function exports and minor comments for clarity. - **`pgmongo_rum/src/rumutil.c` & `pgmongo_rum/src/rum.h`**: Removed the legacy unredacted logging function and added a new exported hook setter (`SetRumUnredactedLogEmitHook`) along with macros for unredacted logging. - **`oss/internal/pg_documentdb_extended_rum/src/rumget.c` & related files**: Enhanced scan routines by incorporating pre-consistent function pointers and bounds checking to determine the appropriate scan type. - **`oss/pg_documentdb/src/index_am/rum.c`**: Modified the initialization routine to load and invoke the new unredacted logging hook from the external library. - **TS utility files (`rum_ts_utils.c` and others)**: Added PGDLLEXPORT declarations for TS functions and made minor comment updates for improved clarity. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
a4f7922959 |
Merged PR 1807745: [PG18] More changes to build & test with PG18
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Enable more builds and tests with PG18 Enable backend tests for documentdb with pg18 ---- #### AI description (iteration 7) #### PR Classification This pull request implements PG18 support by updating APIs, tests, and build pipelines. #### PR Summary The changes introduce PG18‑specific logic across regression tests, build scripts, and core GIN indexing code to enhance compatibility and performance with PostgreSQL 18. - **`oss/pg_documentdb/src/test/regress/expected/`**: New PG18 output files (e.g. `collection_management_pg18.out` and inverse match explain tests) were added while obsolete PG17/PG15 files were removed or moved. - **`/.pipelines/templates/extension_build_flex_core.yml`**: Build pipelines were updated with new parameters (`buildExtensionPackage`, `buildDiskANN`) and conditional steps to support PG18 builds. - **`helio_gin` sources (e.g. `ginget.c`, `ginutil.c`, `ginordered.c`)**: Core GIN code was refactored using `#if PG_VERSION_NUM >= 180000` conditionals to utilize new PG18 functions and adapt to API changes. - **Test scheduling and mutation scripts**: Updated to include PG18 tokens and adjust SQL file references for proper test execution on PostgreSQL 18. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
95d7a4b6a6 |
Merged PR 1806432: [OSS] Remove explicit RUM dependency on the documentdb extension
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactoring ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Make DocumentDB not explicitly depend on the RUM extension (it's done implicitly) on the binary. Instead pull in the SQL dependency into documentdb and reference the C functions internally. This allows us to swap between rum.so and extended_rum. For Pg18 (where there isn't yet official RUM support) default to using extended_rum for the .so file and remove all dependencies on the "rum.so" ---- #### AI description (iteration 2) #### PR Classification This PR is an API and dependency update that removes the explicit dependency on the external RUM extension by integrating its functionality internally. #### PR Summary The update integrates the RUM text-search capabilities directly into DocumentDB and cleans up the external linkage. - `oss/internal/pg_documentdb_extended_rum/src/rum_ts_utils.c` is newly added with core text-search functions. - `oss/pg_documentdb/src/index_am/rum.c` is modified to load internal wrappers that call the new RUM functions. - SQL UDF adapter files and operator class definitions are updated to reference the internal functions instead of public RUM functions. - Build scripts, metadata, and control files are adjusted to remove the explicit dependency on the external RUM extension. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
4bcd774983 |
Merged PR 1804966: [Update performance] Improve update performance in a few codepaths
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Perf ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description 1) Use prepared statements with UpdateAllMatchingDocuments 2) Don't pass shard_key_value by default if using the `@@` operator 3) Do something similar for SELECT update candidate. ---- #### AI description (iteration 1) #### PR Classification Performance optimization and bug fix for update query execution. #### PR Summary This PR enhances update performance by refining shard key filter handling and optimizing SPI plan selection, while also introducing new query plan cache keys and a configuration flag for legacy behavior. - `oss/pg_documentdb/src/commands/update.c`: Refactors update candidate selection by conditionally adding shard key filters, adjusting argument indexing, and using the full collection object instead of separate identifiers. - `oss/pg_documentdb/include/infrastructure/documentdb_plan_cache.h`: Introduces new constants for query plan caching to support optimized update queries. - `oss/pg_documentdb/src/configs/feature_flag_configs.c`: Adds the `UseLegacyShardKeyFilterOnUpdate` flag to offer backward compatibility with the previous shard key filtering behavior. - Test files in the distributed module: Update expected SQL query strings to match the revamped query construction logic. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
a0c2c59c97 |
Merged PR 1804958: [Code Cleanup] Remove composite feature flags that are default
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactoring ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Remove feature flags that are default on with the v106: - EnableNewCompositeIndexOpclass - EnableDescendingCompositeIndex - EnableCompositeUniqueIndex ---- #### AI description (iteration 1) #### PR Classification Code cleanup aimed at removing redundant composite feature flags that are now always default. #### PR Summary This PR simplifies the codebase by eliminating conditional logic, configuration variables, and test settings related to composite feature flags such as descending composite index, composite unique index, and the new composite index opclass. The changes remove obsolete definitions and related conditional checks while updating tests and config files accordingly. - In `configs/feature_flag_configs.c`, removed custom variable definitions and configuration settings for composite flags and introduced new default flags like `EnableIndexOnlyScan` and `EnableRangeOptimizationForComposite`. - In files such as `commands/create_indexes.c`, `index_am/rum.c`, and others, removed conditional logic and external declarations dependent on composite feature flags. - Updated numerous test scripts and configuration files to remove legacy settings (e.g., removing `enableNewCompositeIndexOpClass` and `enableDescendingCompositeIndex`) and adjust expected outputs. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
080ea66e2b |
Merged PR 1803246: [Hidden Index] Support for hidden indexes in DocumentDB
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add support for hidden indexes with collMod This will mark the index as not ready in the postgres catalog. Handles distributed setups as well. However, this is currently restricted to single node (sharded/unsharded). A subsequent change can help multi-node set ups. ---- #### AI description (iteration 1) #### PR Classification This pull request implements a new feature by adding support for hidden indexes in DocumentDB. #### PR Summary The pull request extends the collMod command and associated index management logic to recognize and update a new “hidden” flag in index options, while enforcing constraints (e.g. disallowing hidden modifications for primary or unique indexes) in both single-node and distributed environments. - **`coll_mod.c`**: Introduces helper functions (e.g. GetHiddenFlagFromOptions, UpdateHiddenInIndexOptions, UpdatePostgresIndex) and SPI query updates to process the hidden flag properly. - **Test Files**: Adds and updates multiple test cases (e.g. in `commands_collmod_tests_hidden.out` and SQL test scripts) to verify hidden index creation, query planning, and error handling. - **Distributed Hooks and API**: Updates in files like `distributed_hooks.c` and `api_hooks.c` (with corresponding header changes) enable proper handling of hidden index updates across shard indexes in distributed environments. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Deepak Bisht
|
0a41ca6948 |
Merged PR 1801242: Add point read and lookup extern param SQL Tests
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Testing ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) NA ### Description Add point read and lookup extern param SQL Tests ---- #### AI description (iteration 2) #### PR Classification This PR adds new SQL tests to validate the fix for crashes in lookup queries that involve point reads with external parameters. #### PR Summary The pull request introduces new test cases across multiple backend files to ensure that lookup inner join operations with point reads—where the inner plan depends on an external parameter—are executed correctly without crashing. - `oss/internal/pg_documentdb_distributed/src/test/regress/expected/bson_aggregation_tests_lookup_inner_join.out` and `oss/internal/pg_documentdb_distributed/src/test/regress/expected/bson_aggregation_pg17_tests_lookup_inner_join.out`: Added tests for lookup inner join scenarios with point reads. - `pgmongo/src/test/docdb_compat/expected/bson_aggregation_tests_lookup_inner_join.out` and `pgmongo/src/test/docdb_compat/expected/bson_aggregation_pg17_tests_lookup_inner_join.out`: Introduced equivalent tests to ensure compatibility across backends. - `oss/internal/pg_documentdb_distributed/src/test/regress/sql/bson_aggregation_tests_lookup_inner_join_core.sql`: Updated SQL test script to insert test data, run ANALYZE, and execute aggregation pipelines verifying the crash fix. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Deepak Bisht
|
0f390b2cbb |
Merged PR 1798239: Fix Segmentation faults in ordered aggregate
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) BugFix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description - For aggregates such as `$last` or `$first` where the total number of aggregate values is 1 and existing aggregate state is more than the space required we try to use the same memory space while also updating it on the go. This will overwrite some later referred variable from the state. - The changes also fixes a memory leak in ordered aggregates where the new transition state was allocate in top level aggregate context and PG makes a copy and discards the original value (without freeing it), now we allocate the new transition state only in the CurrentContext which is the per tuple execution context So always create a new memory space for transition in ordered aggregates ---- #### AI description (iteration 1) #### PR Classification This pull request fixes a bug causing segmentation faults in ordered aggregate operations involving the `$last` operator used with the `$setWindowFields` stage. #### PR Summary This update resolves memory issues by removing unsafe byte array reuse in the state serialization for ordered aggregates and enhances regression tests to verify correct behavior. - `oss/pg_documentdb/src/aggregation/bson_sorted_accumulator.c`: Modified `SerializeOrderState` to always allocate new memory and removed the old byte array reuse logic. - `oss/pg_documentdb/include/aggregation/bson_sorted_accumulator.h`: Updated the function signature of `SerializeOrderState` to reflect the new implementation. - Test files in `oss/internal/pg_documentdb_distributed/src/test/regress` and `pgmongo/src/test/docdb_compat`: Added new tests for `$last` to check ordered aggregate functionality with `$setWindowFields`. - Documentation files (`docs/release_notes/pgmongo_1.108.md` and `oss/CHANGELOG.md`): Updated release notes and changelog to document the segmentation fault fix. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4687821 Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
32b11ab18f |
Merged PR 1799781: [Reindex] Support internally handling Reindex with background index processor
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add infrastructure to handle REINDEX CONCURRENTLY using hte background index queue. This doesn't expose any new public API at this time, and is meant as a first pass to enable administrators to enqueue reindex concurrently in the background. Progress can be tracked by currentOp ---- #### AI description (iteration 2) #### PR Classification This PR introduces a new feature that adds asynchronous reindex operations using the background index processor. #### PR Summary This update extends the background index processor to support reindexing by implementing new C functions and updating index queue management, SQL UDF definitions, and tests. Key changes include: - `src/commands/create_indexes_background.c`: Added `command_reindex_index_background` and `command_reindex_index_background_internal` functions and refactored common command core logic. - `src/metadata/index.c`: Modified index queue functions and metadata handling to support the new REINDEX command type. - `sql/udfs/index_mgmt/create_index_background--0.108-0.sql`: Added new SQL UDFs for reindexing indexes. - Test files under `src/test/regress` and `/pgmongo/src/test/regress`: Updated to validate the background reindexing functionality. - Minor updates in logging and distributed hook functions to accommodate reindex operations in the background processor. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
c02bc5ac46 |
Merged PR 1798510: [DocumentDB] Add basic validation for pg18
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactor ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add runs for documentdb_core tests, and documentdb check minimal to validate extension startup with pg18. Fix the issue with pgvector upgrade. ---- #### AI description (iteration 3) #### PR Classification This pull request implements a new feature by adding basic validation and compatibility enhancements for pg18 support in DocumentDB. #### PR Summary The changes update test validations, pipeline configurations, and code conventions to ensure proper pg18 behavior and compatibility. Highlights include: - **`pgmongo/src/test/regress/sql/extension_upgrade_vector.sql` & expected output**: Replace `\dx vector` commands with SQL queries that verify the pgvector extension version. - **`oss/internal/pg_documentdb_extended_rum/src/rumutil.c` and `pgmongo_rum/src/rumutil.c`**: Refactor rum options parsing by introducing a constant (`offsetIfDefault`) instead of hardcoded `-1` values. - **Pipelines (`.pipelines/templates/oss_documentdb_build.yml`, `.pipelines/templates/common_variables.yml`, `.pipelines/pgmongo_backend_test.yml`)**: Add a new parameter (`runMinimalTests`) and update container image references to support minimal OSS backend tests for pg18. - **`pgmongo/src/test/regress/Makefile`**: Modify dummy pgvector version file generation to dynamically select the latest available version. - **`oss/pg_documentdb/src/background_worker/background_worker.c`**: Remove the `WL_POSTMASTER_DEATH` flag from WaitLatch calls to refine background worker dependency handling. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Vinod Sridharan
|
fc4fc65237 |
Merged PR 1797043: [DocumentDB] Part 1: Enable builds of DocumentDB for pg18
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactor ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? no ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Support building documentdb for pg18 This sets up DocumentDB for PG18 builds. Note that execution and tests are not validated yet (just compilation). Also bumps up extension dependencies to support the builds of pg18. ---- #### AI description (iteration 7) #### PR Classification This pull request adds PG18 support by implementing the necessary compatibility and feature changes in DocumentDB. #### PR Summary The update enables DocumentDB builds for PG18 by adapting ICU collation handling, updating version and pipeline configurations, and incorporating compatibility macros and API adjustments across the codebase. Key changes include: - **`oss/pg_documentdb_core/src/collation/collation.c`**: Introduced new ICU converter functions and updated collation conversion logic. - **`oss/scripts/setup_versions.sh` and pipelines**: Adjusted dependency versions and container build configurations to support PG18. - **`oss/pg_documentdb_core/include/utils/documentdb_pg_compatibility.h`**: Added a new compatibility header with macros to bridge PG18 API differences. - **Extended RUM modules (`oss/internal/pg_documentdb_extended_rum` and `pgmongo_rum`)**: Updated type definitions (e.g., replacing `PostingItem` with `RumPostingItem`) and switched from `GinStatsData` to `RumStatsData`. - **Various DocumentDB source files**: Modified executor calls, background worker logic, and tuple attribute access to comply with PG18’s API changes. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
 Jia Hu
|
2a8a93311a |
Merged PR 1554321: [json schema] support keyword required
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md)) ### Description support keyword `required` in `$jsonSchema` ---- #### AI description (iteration 1) #### PR Classification New feature #### PR Summary This pull request adds support for the `required` keyword in JSON schema validation. - Added `ParseRequired` function to handle the `required` keyword in `bson_json_schema_tree.c`. - Updated `BuildSchemaTreeCoreOnNode` to call `ParseRequired` when the `required` keyword is encountered. - Modified `ValidateBsonValueObject` in `bson_json_schema_validator.c` to validate required fields. - Added test cases for the `required` keyword in multiple test files (`bson_dollar_ops_json_schema_query_tests.out`, `bson_dollar_ops_json_schema_build_tree_tests.out`). - Updated `ValidationsObject` structure in `bson_json_schema_tree.h` to include a `required` field. Related work items: #4638803 Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
 Cornelius Boateng
|
ce0cb575be |
Merged PR 1794050: [engine][collation] Support for $sortArray
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Support for the `$sortArray` aggregation operator. ---- #### AI description (iteration 1) #### PR Classification This pull request implements collation support for the $sortArray operator as a new feature. #### PR Summary This change integrates collation awareness into the $sortArray operator so that array sorting properly respects collation rules. The modifications update both the underlying sorting logic and the tests to reflect these new capabilities. - `pg_documentdb/src/operators/bson_expression_array_operators.c`: Updated parsing of the $sortArray operator to copy the collation string into the sort context. - `pg_documentdb/src/utils/sort_utils.c` and `pg_documentdb/include/utils/sort_utils.h`: Switched to collation-aware comparison functions by replacing calls to CompareBsonValueAndType with CompareBsonValueAndTypeWithCollation. - Test expectation files under `/oss/internal/pg_documentdb_distributed/` and `/pgmongo/` were updated to include new $sortArray outputs with different collation settings. - Integration CSV files were modified to mark the previously failing `$sortArray` tests as passing. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Signed-off-by: Shuai Tian <shuaitian@microsoft.com> |
||
|
Deepak Bisht
|
95db7f3de7 |
Merged PR 1788321: Add a unit test for limit with large docs
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Testing ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description - Add a limit unit test for cursors to make sure we never break the correctness of $limit stage when batchSize is more than the limit size. We need persisted cursors and this can't be optimized out by running as a signle batch query because the document size if huge will be returned in batches ---- #### AI description (iteration 1) #### PR Classification This pull request adds a new unit test to verify that the system enforces persisted cursors when using $limit on large documents. #### PR Summary This PR introduces tests that insert large (~16MB) documents to ensure that even if the batch size can accommodate all limited documents, the response size still triggers persisted cursor behavior. Key changes include: - `oss/internal/pg_documentdb_distributed/src/test/regress/expected/bson_aggregation_cursor_tests.out`: added a test block for validating persisted cursors with the $limit operator on large docs. - `pgmongo/src/test/docdb_compat/expected/bson_aggregation_cursor_tests.out`: introduced a similar test for docdb compatibility using the helio API. - `oss/internal/pg_documentdb_distributed/src/test/regress/sql/bson_aggregation_cursor_tests.sql`: appended SQL statements to execute and verify the $limit test scenario. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Deepak Bisht
|
2044bce7a2 |
Merged PR 1786924: Fetch the TTL index entries with desc sort order behind a GUC
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Refactoring ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes, `documentdb.enableTTLDescSort` fetches the TTL documents on descending sort order ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description - Fetch the TTL index entries with desc sort order behind a GUC - Adds an ORDER BY expression in the SELECT query for potential documents to be deleted by TTL ---- #### AI description (iteration 1) #### PR Classification This pull request introduces a new feature that enables fetching TTL index entries in descending sort order controlled by a new GUC. #### PR Summary The changes add support for descending TTL sort ordering and update related regression tests to verify the behavior. - `oss/pg_documentdb/src/ttl/ttl_index.c`: Adds logic to append a descending ORDER BY clause when `EnableTTLDescSort` is true and the index is ordered. - `oss/pg_documentdb/src/configs/testing_configs.c`: Defines and registers the new GUC variable for enabling TTL descending sort. - Test output files in both `oss/internal/pg_documentdb_distributed/src/test/regress/expected/commands_create_ttl_indexes.out` and `/pgmongo/src/test/docdb_compat/expected/commands_create_ttl_indexes.out`: Updated to include commands and expected outputs for the descending TTL ordering. - `oss/internal/pg_documentdb_distributed/src/test/regress/sql/commands_create_ttl_indexes.sql`: Modifies SQL regression tests to incorporate the descending TTL ordering scenario. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4644360 |
||
|
Cornelius Boateng
|
7492f14676 |
Merged PR 1754062: [engine] Enable EnableNowSystemVariable & EnableVariablesSupportForWriteCommands GUCs
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Automatically support `$$NOW` for find and aggregate queries and all variables for delete, update, findAndModify queries. ---- #### AI description (iteration 1) #### PR Classification This pull request deprecates and removes the `EnableNowSystemVariable` GUC to simplify time variable handling in the engine. #### PR Summary The changes eliminate conditionals based on `EnableNowSystemVariable` and update related function signatures to always process time system variables uniformly. - **`oss/pg_documentdb/src/operators/bson_expression.c`**: Removed the `isWriteCommand` parameter and flag-based checks, so that the current time value is always appended without referencing `EnableNowSystemVariable`. - **`oss/pg_documentdb/src/commands/*` and `oss/pg_documentdb/src/aggregation/*`**: Replaced conditional logic dependent on `EnableNowSystemVariable` with direct checks for valid time system variable values. - **`oss/pg_documentdb/src/configs/feature_flag_configs.c` and header files**: Deleted the definition, default value, and extern declarations for `EnableNowSystemVariable`. - **Configuration scripts and test files**: Removed the now redundant `enableNowSystemVariable` settings from various configuration and test files. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> Related work items: #4287488 |
||
|
Cornelius Boateng
|
f9e51fce34 |
Merged PR 1783524: [fix][collation] Simple Locale
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Simple locale must use binary comparison and not necessarily case-insensitive comparison. ---- #### AI description (iteration 1) #### PR Classification This pull request is a bug fix to correctly handle the simple collation locale. #### PR Summary The changes ensure that when the collation locale is specified as `simple`, all additional collation options are ignored and binary comparison is applied. - `oss/pg_documentdb_core/src/collation/collation.c`: Added logic to bypass extra collation options when the locale is `simple`. - `oss/pg_documentdb_core/src/query/bson_compare.c`: Updated string comparison to use binary comparison for the `simple` collation. - Test files under `oss/internal/pg_documentdb_distributed/src/test` and `pgmongo/src/test` were updated to add regression checks for simple collation behavior. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Cornelius Boateng
|
d90748a718 |
Merged PR 1784757: [fix] Fix Crash Due to NULL variableSpec and Valid Collation
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Bugfix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Internal functions (`*WithLetAndCollation`) that accept both variableSpec and collationString must pass a `null const` value in place of variableSpec if the variableSpec is NULL, with a valid collation string. ---- #### AI description (iteration 1) #### PR Classification Bug fix to prevent a crash when an empty variableSpec is provided together with a valid collation. #### PR Summary This pull request fixes a crash by properly handling NULL/empty variableSpec in conjunction with collation in both the core aggregation functions and nested pipelines. Highly important changes include: - **`pg_documentdb/src/aggregation/bson_aggregation_pipeline.c`**: Updated the logic to create and pass collation and variableSpec constants safely, avoiding null dereferences. - **`pg_documentdb/src/aggregation/bson_aggregation_nested_pipeline.c`** and **`pg_documentdb/src/query/query_operator.c`**: Adjusted handling of variableSpec and collation parameters in nested pipeline and query operator functions. - **Test files (e.g., in `/oss/internal/pg_documentdb_distributed/src/test/regress/expected/bson_dollar_ops_collation_tests_runtime.out` and `/pgmongo/src/test/docdb_compat/expected/bson_dollar_ops_collation_tests_runtime.out`)**: Added session settings and BEGIN/ROLLBACK blocks to disable auto $$NOW generation and to validate collation-dependent queries. - **Documentation and changelog**: Updated release notes and the CHANGELOG to document the bug fix regarding the empty variableSpec with collation. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Santiago Fernandez Madero
|
820d5b4d0c |
Merged PR 1785241: Fix one rum compiler warning propertly
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Fix one rum compiler warning propertly ---- #### AI description (iteration 1) #### PR Classification Bug fix: The changes eliminate redundant initializations in the rum modules to resolve a compiler warning. #### PR Summary This pull request refactors the `rumReadTuple` function in two modules by relocating the block number initialization into the conditional branch and removing the extra pointer initialization in the else branch. - In `oss/internal/pg_documentdb_extended_rum/src/rumentrypage.c`, the call to `InitBlockNumberIncr` is removed from the pre-conditional code and is instead applied only within the `RumUseNewItemPtrDecoding` branch, while the redundant `ItemPointerSetMin` call in the else block is removed. - In `pgmongo_rum/src/rumentrypage.c`, similar adjustments are made by moving the block number initialization into the if block and eliminating the duplicate pointer setup in the else branch. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Santiago Fernandez Madero
|
fa15686ec1 |
Merged PR 1784970: Fix some rum compile warnings and a correctness issue whe using the new ptr decoding
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Fix some rum compile warnings and a correctness issue whe using the new ptr decoding ---- #### AI description (iteration 1) #### PR Classification Bug fix addressing compile warnings and a correctness issue with new pointer decoding. #### PR Summary This pull request fixes initialization order and parameter issues in the new pointer decoding logic for rum modules to resolve compile warnings and correctness problems. - `rumentrypage.c` in both `/oss/internal/pg_documentdb_extended_rum/src` and `/pgmongo_rum/src` now call `RumItemSetMin` and `InitBlockNumberIncr` before checking `RumUseNewItemPtrDecoding`. - `pg_documentdb_rum.h` adds an else-if branch to use `rumDataPageLeafReadItemPointerNew` when new decoding is enabled. - `rum.h` in `/pgmongo_rum/src` updates the function call to `rumDataPageLeafReadItemPointerNew` by passing the complete item instead of its pointer. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Jinrui Zhang
|
3ccd656663 |
Merged PR 1777067: [Refector] Update to match coding standards - part 0
### Description 1. Fixing issues identified by similarity checker 2. Fix the corresponding sql unit tests 3. Fix JSTests failure due to above changes 4. Fix corresponding C# test failure ---- #### AI description (iteration 1) #### PR Classification This pull request is a code cleanup that standardizes and improves error message wording across the codebase. #### PR Summary The changes update error messages throughout the repository to provide clearer, more consistent, and more detailed feedback when operations fail. - Updates in geospatial operator files (e.g. in `$geoNear`, `$near`, `$nearSphere`) replace informal phrasing with precise messages. - Numerous error messages in aggregation and projection test files have been reworded for consistent phrasing <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Siwen Tang
|
be11798bef |
Merged PR 1778507: [Refector] Update to match coding standards - part 5
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description draft update error msg ---- #### AI description (iteration 1) #### PR Classification This is a code cleanup and bug‑fix pull request that standardizes and clarifies error messages and log details across multiple modules. #### PR Summary The changes improve the consistency, clarity, and accuracy of user‑facing messages and logging throughout the code base. Key changes include: - In files under directories such as `oss/internal/pg_documentdb_distributed/src/test/regress/expected/` and `pgmongo/src/test/docdb_compat/expected/`, error messages for geospatial query operators (e.g. geoNear, $center, $within) and aggregation stage operators have been rewritten for clarity (e.g. replacing vague “Too many geoNear expressions” with “Excessive number of geoNear query expressions” and clarifying numeric parameter validations). - Several operators’ messages (e.g. `$cond`, `$not`, `$abs`, `$floor`, `$divide`, `$log`, etc.) have been updated to state exactly how many arguments are required, using <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Xing Fan
|
3b6061e6ea |
Merged PR 1777085: [Refector] Update to match coding standards - part 6
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description draft ---- #### AI description (iteration 1) #### PR Classification Bug fix: This PR revises error messages and inline comments to improve clarity and consistency across the code and test outputs. #### PR Summary The changes refactor and update a large number of error messages across various modules—ranging from aggregation and update operators to geospatial, schema validation, and command handling—to provide more precise diagnostic feedback. - **Tests (e.g. in `tests/expected/... <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Duan Pei
|
6a129514a9 |
Merged PR 1777887: [Refector] Update to match coding standards - part 1
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description update errmsg ---- #### AI description (iteration 1) #### PR Classification This pull request focuses on updating and standardizing error messages across various operators, aggregation stages, and test expectation files. #### PR Summary The changes reword error messages throughout the codebase and test outputs to provide clearer, more consistent, and more informative feedback for users. Key updates include: - **`/oss/internal/pg_documentdb_distributed/src/test/regress/expected/*.out`** – Revised expected error outputs for aggregation, update, and geospatial tests to use clearer phrasing (e.g., for `$merge`, `$pop`, `$indexOfArray`, and `$geoNear`). - **Core operator files (e.g., `bson_aggregation_output_pipeline.c`, `bson_expression_arithmetic_operators.c`, `bson_expression_conditional_operators.c`)** – Updated error message texts to explicitly state operator requirements and types. - **Geospatial and vector modules (e.g., files under `bson_geojson_utils.c`, `bson_geospatial_*`, and vector search/index configuration files)** – Standardized validations and error reporting for GeoJSON, CRS, and vector search parameters. - **Additional modules (e.g., collection view creation and user management files)** – Improved error handling messages to better guide users on input misconfigurations. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Jia Hu
|
7db51ff43b |
Merged PR 1777927: [Refector] Update to match coding standards - part 2
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) similarity issue ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description ---- #### AI description (iteration 1) #### PR Classification This pull request is a code cleanup aimed at standardizing and clarifying error and diagnostic messages throughout the codebase. #### PR Summary The changes update error messages and logging across many modules (aggregation, query, update, index creation, collation, etc.) to use clearer, more precise, and consistent language. - **Aggregation Operators:** Many files in the aggregation code (e.g. in `bson_aggregation_*` and related operator implementations) now report errors such as division by zero, invalid inputs, and argument count in a formal and uniform manner. - **Index & Projection Validation <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Yang Qiao
|
7f0c061b83 |
Merged PR 1778033: [Refactor] Update to match coding standards - part 4
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Testing ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Update to match coding standards ---- |
||
|
Vinod Sridharan
|
739bdfb1aa |
Merged PR 1780627: A few API changes & RUM changes for performance
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Features ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description 1) Port a few more changes into the public documentdb_rum to reduce diff to internal RUM 2) Add more inlining of computation of RumItem from the pages (behind a GUC defaulted on) 3) Add option for bson_build_document and fix bson_get_value_text to not quote strings unnecessarily to be compatible with `::text` casts ---- #### AI description (iteration 1) #### PR Classification This PR implements API updates and performance improvements by introducing a new item pointer decoding mechanism in RUM indexes and extending BSON UDF functionalities. #### PR Summary The pull request refactors core RUM index scan routines and enhances BSON API functions to improve performance and API consistency. Key changes include: - **`oss/internal/pg_documentdb_extended_rum` and `pgmongo_rum`**: Introduced new macros and inline functions (e.g., `decode_varbyte_blocknumber`, `rumDataPageLeafReadWithBlockNumberIncr`) along with the boolean flag `RumUseNewItemPtrDecoding` to optimize item pointer decoding. - **`rumget.c`, `rumentrypage.c`, and `rumdatapage.c`**: Updated the posting tree scan logic to conditionally apply the new decoding approach, streamlining data page processing. - **`pg_documentdb_core/sql/udfs/bson_io`**: Added new BSON UDFs (such as `bson_build_document`) and overloads (e.g., a three-argument version of `bson_get_value_text`) to expand API capabilities. - **`pgbson.c` and `bson_io.c`**: Enhanced BSON logging functions with optional quoting support for JSON conversion, improving logging clarity and consistency. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Yang Qiao
|
f9cbecf598 |
Merged PR 1777990: [Refector] Update to match coding standards - part 3
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Testing ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Fix issues identified by similarity checker ---- #### AI description (iteration 1) #### PR Classification This pull request is a bug fix and code cleanup that standardizes and clarifies error messages across aggregation, query, and index creation operators. #### PR Summary The changes update error text throughout the codebase to be more explicit and consistent regarding type validations and operator requirements. Notable updates include: - Updates to error messages in aggregation operator files (e.g. in `bson_expression_bitwise_operators.c`, `bson_expression_conditional_operators.c`, and similar files) to clearly specify expected input types and constant requirements. - Rewording of messages for operators such as `$regexFindAll`, `$merge`, `$switch`, `$ifNull`, and trigonometric functions to use precise language (e.g. “requires” or “must be provided” instead of “only supports”). - Improved clarity in messages for view cycle detection (e.g. in `create_indexes.c` and view creation tests) and hashed index validations. - Consistent rephrasing for messages in densify and percentile operators to better indicate numeric or date constraints and limits. - Refinements in error reporting for operators in type conversion and array handling (e.g. in `$arrayElemAt`, `$slice`, `$getField`, and `$unsetField`), ensuring that expected value types (such as string, numeric, or object) are explicitly stated. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 Parag Jain
|
fa57f40720 |
Merged PR 1779662: [Observability] advanced feature usage for write commands
### Does this PR have any customer impact? ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) ### Are you introducing any new config? If yes, do you have tests with and without them being set? ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description [Insert] Feature Usage Insert ---- #### AI description (iteration 1) #### PR Classification New feature addition that implements feature usage tracking for insert and update commands. #### PR Summary This pull request adds inline functions to report feature usage based on batch sizes for insert and update operations and updates the associated feature enum definitions. - `/oss/pg_documentdb/src/commands/insert.c`: Added `ReportInsertFeatureUsage` to report usage metrics based on batch size and integrated its call during batch insertion processing. - `/oss/pg_documentdb/src/commands/update.c`: Introduced `ReportUpdateFeatureUsage` with similar threshold-based reporting, integrated into the batch update process. - `/oss/pg_documentdb/include/utils/feature_counter.h`: Extended the feature counter enums with new constants for detailed insert and update reporting, and increased `MAX_FEATURE_COUNT`. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
c7613604d7 |
Merged PR 1777606: [Bugfix] Fixes for point read planner and lookup
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Bugfix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description For $lookup we were ignoring the joinquals for point read pushdown - this happened to work before if the planner chose a point read plan before. But now that we're more explicit in looking for this we need to handle this explicitly. Also ensure point read plan generation considers outer_relids (this wasn't done before). ---- #### AI description (iteration 1) #### PR Classification Bugfix: This PR corrects issues in the point read planner and lookup functionality. #### PR Summary The changes fix bugs related to index selection and query planning for multi-layer lookups by improving how join and restriction information is handled, and by extending test coverage for point read scenarios. - **`oss/pg_documentdb/src/opclass/index_support.c`**: Updated handling of outerRelids and join restrictions to ensure proper index path identification. - **`oss/pg_documentdb/src/planner/documents_planner.c` and `oss/pg_documentdb/include/opclass/bson_index_support.h`**: Modified calls to pass join info into restriction path processing for accurate planning. - Test files under **`oss/internal/pg_documentdb_distributed`** and **`pgmongo`**: Added new multi-layer lookup queries and expected outputs to verify that lookups consistently pick the _id index. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
 CosmosDB Build Service (msdata)
|
72fb39d9a9 |
Merged PR 1775074: [tagv.1.108] Update extension version for new release 1.108
This PR updates the extension version for the new release 1.108. Changes included: - Control files updated to version 1.107-0 - SQL files generated for version upgrades - Release notes file created for version 1.108 - TOC.yml updated with new release notes entry - CHANGELOG.md updated with new version entry |
||
|
Vinod Sridharan
|
0c81ad7665 |
Merged PR 1775707: Order by fixes for backward scans
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Bugfix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Fix two bugs found in backward scan. ---- #### AI description (iteration 1) #### PR Classification Bug fix enhancing ordering behavior during backward scans. #### PR Summary This pull request refines the logic for backward scans to ensure correct order-by results and updates related pipelines for schema version management. - `/.pipelines/templates/single_node_tests.yml`: Introduced the `useNextSchemaVersionOverride` parameter and conditional assignment for `CurrentSchemaVersion`. - `/oss/pg_documentdb/src/opclass/bson_gin_composite_entrypoint.c`: Modified comparator logic to adjust return values for backward scans, ensuring proper boundary handling. - `/pgmongo_rum/src/rumget.c`: Adjusted order scan offset for backward scanning in the order-by process. - `/.pipelines/pgmongo_backend_test.yml`: Enabled the schema version override by setting `useNextSchemaVersionOverride` to true. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
a7b1076f31 |
Merged PR 1773571: Improve IndexScan and IndexOnlyScan Performance
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Improve performance of indexscans and indexonlyscans IndexTerm: >1) BsonIndexTerm is used in several places - reduce the size of this struct by removing isTruncated bool from the struct. >2) Add options to check metadata state without parsing the index term at all. >3) InitializeBsonIndexTerm - do not create a wrapper bson_value_t - just initialize the term directly. OrderedScans: >1) For every page we visit, we free the prior buffer and call PageGetTempPageCopy. Instead, keep one buffer per scan and just copy the buffer into the static buffer created per scan >2) When doing indexonlyscan, don't recreate an indextuple very time - instead reuse the memory and just fill the tuple buffer with what was previously allocated PgbsonWriter: >1) Add support for reusing the same writer across invocations by calling reset on the buffer >2) Add options to retrieve the value from a pgbsonwriter Composite: >1) Use a case dispatch for strategies rather than multiple if/else >2) Work on the serialized bson index terms as much as possible and lazy load the index terms >3) Skip Compare on bounds calls if there's no bounds declared ---- #### AI description (iteration 1) #### PR Classification This pull request delivers performance improvements and code refactoring for index scan operations. #### PR Summary The changes optimize index scans by refactoring tuple construction and buffer page handling for RUM indexes, while also unifying and streamlining BSON composite index term processing. - `pgmongo_rum/src/rumget.c`: Introduced `IndexBuildTupleDynamic` to build tuples without compression and reuse memory, and refactored page copy logic using `CopyPageContents` and an added `isPageValid` flag. - `oss/pg_documentdb/src/opclass/bson_gin_composite_entrypoint.c` (and related files): Replaced direct truncation checks with function calls (e.g., `IsIndexTermTruncated`), updated serialized composite index term initialization, and adjusted ordering transform logic. - `oss/pg_documentdb/src/opclass/bson_gin_index_term.c` (and accompanying headers): Added functions for handling serialized composite index terms and metadata, consolidating truncation flag logic. - `oss/pg_documentdb_core/src/io/pgbson.c`/`pgbson_writer.h`: Added new writer functions (e.g., reset and value retrieval) to improve memory reuse and reduce allocation overhead. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
10b395a75b |
Merged PR 1773514: [Orderby] Improve order by pushdown behavior & selectivity
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Pushing down order by and filters to a composite index should prefer the smallest index with the highest number of order by columns matching. To do this we need to participate in the core add_path set up as opposed to after. Consequently, we mutate the indexPath when its created in create_index_path as opposed to modifying it in the relpathlist_hook. We also reorder the indexes by paths so that for filters, we prefer indexes with fewer paths over many that can match the same paths. ---- #### AI description (iteration 1) #### PR Classification This pull request refines the ORDER BY pushdown mechanism in index scans by adding new processing logic and improving index selectivity while introducing a feature flag for the legacy behavior. #### PR Summary The changes enhance query performance by introducing distinct pushdown paths for ORDER BY clauses, refining composite index handling, and updating test expectations to reflect the new "orderByScan" operator. The following key modifications were made: - **`oss/pg_documentdb/src/opclass/index_support.c`**: Added new functions (e.g., `ConsiderIndexOrderByPushdownNew`, `ProcessOrderByStatements`) to split new versus legacy ORDER BY pushdown logic. - **Test Files** (under `/oss/internal/pg_documentdb_distributed/src/test/regress/expected/` and `/pgmongo/src/test/docdb_compat/expected/`): Updated expected query plans to use the new "orderByScan" predicate instead of "fullScan". - **Index and Planner Modules** (e.g., `oss/pg_documentdb/src/index_am/rum.c` and `planner/documents_planner.c`): Enhanced composite index ranking to factor in the number of index paths. - **Configuration Files** (e.g., `oss/pg_documentdb/src/configs/feature_flag_configs.c`): Introduced the `EnableIndexOrderbyPushdownLegacy` flag for toggling the legacy behavior. - **Supporting Modules** (in aggregation and query operators): Adjusted functions in files like `bson_query_common.c` and related headers to handle new order-by scan directions consistently. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
4226c3a069 |
Merged PR 1774203: Fix up crash in the entryFindItem path
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Bugfix ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Fix up crash in the entryFindItem path - it would be the same fix as what was observed in the entryGetNextItem path. ---- #### AI description (iteration 1) #### PR Classification Bug fix: This PR resolves a crash in the entryFindItem path caused by page splits. #### PR Summary This change adds logic in the entryFindItem function to repeatedly locate the leftmost page when a page split occurs, ensuring proper buffer locking and preventing crashes. - `oss/internal/pg_documentdb_extended_rum/src/rumget.c`: Introduced a while loop to refind the leftmost page by checking for non-leaf pages and re-locking the new buffer during entryFindItem. - `pgmongo_rum/src/rumget.c`: Made similar modifications by adding the `Page page;` declaration and corresponding loop to handle page splits effectively. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
df1e6f4e61 |
Merged PR 1773287: [Explain] A few explain scan improvements based on recent investigations
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description A few explain improvements: 1) Push index usage under each index IXSCAN node (rather than at the top level aggregated) 2) Add a bool for indexonlyscan and add number of heap fetches under indexonlyscan 3) Only mark a scan as _id_ and COLLSCAN if it's an unsharded lookup with no object_id filter 4) For BitmapAnd, don't add an unnecessary FETCH stage before the bitmapindexscan IXSCAN 5) Don't print the index name in the parent FETCH stage of an ixscan 6) For aggregation scans, push the indexdetails to the $cursor stage (as opposed to next level stages). 7) In the backend - handle BitmapAnd for explain scan. Other bugs found in this: 1) Don't skip writing WAL for multikey status after index build (since the whole page image is already marked for WAL before that point) 2) Don't call ginUpdateStats (call rumUpdateStats) in parallel index build. ---- #### AI description (iteration 1) #### PR Classification This PR enhances explain scan functionality by improving index usage reporting and scan diagnostics across multiple components. #### PR Summary The changes refine the explain plan output for various scan types by better handling index conditions, propagating index details, and introducing new diagnostic fields. - `PostgresExplainHelper.cs`: Refactored index condition processing, improved index usage reporting, and added flags like `isIndexOnlyScan` and handling for heap fetches. - `mod.rs` (in pg_documentdb_gw explain): Updated query planning and execution stats logic to support Bitmap Index Scan and propagate index details into inner plans. - C source files (`index_am/rum_adapter.c` and `custom_explain_scan.c`): Simplified WAL logging, improved multi-key index status updating, and added handling for BitmapAndState in scan explanations. - `utf8.strings.csv`: Added new keys (`isIndexOnlyScan` and `totalDocsAnalyzed`) to support the enhanced explain output. - `CHANGELOG.md`: Documented the explain scan improvements as a new feature. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
338287cd46 |
Merged PR 1772730: [Logging] Add support for adding unredacted server logs in the extension
### Does this PR have any customer impact? No ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) DevOps ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? No ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description Add a hook that enables the service to dump unredacted logs to the server. Add a macro that can be called in this case. The default implementation does nothing, but the custom impl can provide a way to ship `safe` unredacted logs to a telemetry sink. ---- #### AI description (iteration 1) #### PR Classification This pull request is an API change that introduces unredacted logging support in the extension. #### PR Summary The changes add new logging hooks and functions to output unredacted server logs by sanitizing log messages and replacing standard logging calls in critical workflows. - **`pgmongo/src/api_hook_impl.c`**: Added `PgmongoUnredactedLogEmitHook` to sanitize messages (by replacing `%` with `p`) and register it via a new logging hook. - **`pg_documentdb/src/commands/create_indexes_background.c`**: Replaced multiple `ereport(LOG, ...)` calls with the `elog_unredacted` macro to log unredacted messages during index operations. - **`pg_documentdb/src/utils/error_utils.c` & `pg_documentdb_core/include/utils/error_utils.h`**: Introduced `errmsg_unredacted` and defined the `elog_unredacted` macro to support the new logging mechanism. - **Miscellaneous**: Adjusted logging verbosity in files like `distribution/distributed_hooks.c` and `create_collection_core.c` by adding contextual flags for hiding statements and contexts. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |
||
|
Vinod Sridharan
|
0e1f7c952d |
Merged PR 1769332: [CollStats] Improve behavior of collstats for small/large collections
### Does this PR have any customer impact? Yes ### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature ### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No ### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes ### ChangeLog (Refer [Template](../oss/CHANGELOG.md)) ### Description CollStats was predicated on the ability to run ANALYZE very frequently. However, as called out in https://www.postgresql.org/message-id/1935.1554869432%40sss.pgh.pa.us Pg stats has a live tuples counter which is kept up to date and refreshed at every vacuum/analyze to match reltuples. So we should use this to capture a more up to date live tuples where possible. As long as there's not too many pg_stats_reset() we should be okay with this. Secondly, for getting avg stats, we were doing a runtime AVERAGE on the column width which can get expensive if we're wrong about the count and the collection is indeed large. Instead use TABLESAMPLE and get a statistical average of the table for collstats reporting to make it behave fast. ---- #### AI description (iteration 1) #### PR Classification This PR improves the accuracy and performance of the collStats command by refining how document counts and average object sizes are computed for both small and large collections. #### PR Summary The changes enhance collStats functionality by incorporating live tuple statistics and runtime sampling to compute counts and average document sizes, with the new "analyzedCount" field added to the output. This update improves precision especially for small collections and adjusts related debug messaging and configuration flags. Key changes include: - **`commands_coll_stats.out`**: Expected outputs now include an `"analyzedCount"` field in collStats responses. - **`coll_stats.c`**: Modified to use live tuple counts (via `pg_stat_all_tables`) and added functions (`GetPostgresLiveDocumentStats` and `GetAverageColumnWidthSampled`) for optimized sampling. - **`feature_flag_configs.c`**: Introduced and initialized the `UsePgStatsLiveTuplesForCount` flag for toggling the live stats mode. - **Planner and Test Files**: Updated in files such as `documents_planner.c` and various test scripts to accommodate the new counting logic and ensure correct behavior in index policies and recovery scenarios. <!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot --> |