upstream/documentdb
2
0 Fork 0
mirror of https://github.com/FerretDB/documentdb.git synced 2026-11-07 11:32:47 -05:00
Code Releases 1 Activity
1,134 Commits 7 Branches 21 Tags
main
Commit Graph
40 Commits
Author SHA1 Message Date
Vinod Sridharan 84b999ae88 Merged PR 1815453: [Composite] Support optimized elemMatch pushdown 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

Improve pushdown of $elemMatch for composite indexes.

When there's an elemMatch of the form `{ "a": { "$elemMatch": { "$gt": 10, "$lt": 15 }}`
This can scan the index only from 10 -> 15

However given the multi-key index requirement we were doing 2 scans:
- One from 10 to infinity and the other from -infinity to 15.

Optimize this path for $elemMatch.

----
#### AI description  (iteration 1)
#### PR Classification
This PR introduces a new feature that optimizes the pushdown of `$elemMatch` queries for composite indexes.

#### PR Summary
The pull request implements optimized handling of `$elemMatch` predicates on composite indexes by converting them into specialized index pushdown expressions and updating the corresponding test expectations.
- **`oss/pg_documentdb/src/opclass/bson_gin_composite_core.c` & `index_support.c`**: Added logic to detect `$elemMatch` in range queries and convert them into optimized pushdown expressions (via functions like `GetElemMatchIndexPushdownOperator`).
- **Configuration Changes**: Introduced a new GUC (`documentdb.useNewElemMatchIndexOperatorOnPushdown`) in `pg_documentdb/src/configs/feature_flag_configs.c` to control the new behavior.
- **Test Updates**: Modified expected explain outputs in both `/oss/internal/pg_documentdb_distributed/src/test/regress/` and `/pgmongo/src/test/docdb_compat/expected/` to reflect the revised index condition format using `elemMatchIndexOp`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
 2025-10-12 21:30:52 -07:00
Jinrui Zhang 3ccd656663 Merged PR 1777067: [Refector] Update to match coding standards - part 0 
### Description
1. Fixing issues identified by similarity checker
2. Fix the corresponding sql unit tests
3. Fix JSTests failure due to above changes
4. Fix corresponding C# test failure

----
#### AI description  (iteration 1)
#### PR Classification
This pull request is a code cleanup that standardizes and improves error message wording across the codebase.

#### PR Summary
The changes update error messages throughout the repository to provide clearer, more consistent, and more detailed feedback when operations fail.
- Updates in geospatial operator files (e.g. in `$geoNear`, `$near`, `$nearSphere`) replace informal phrasing with precise messages.
- Numerous error messages in aggregation and projection test files have been reworded for consistent phrasing
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-08-25 11:39:17 +08:00
Vinod Sridharan c7613604d7 Merged PR 1777606: [Bugfix] Fixes for point read planner and lookup 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

For $lookup we were ignoring the joinquals for point read pushdown - this happened to work before if the planner chose a point read plan before. But now that we're more explicit in looking for this we need to handle this explicitly.

Also ensure point read plan generation considers outer_relids (this wasn't done before).

----
#### AI description  (iteration 1)
#### PR Classification
Bugfix: This PR corrects issues in the point read planner and lookup functionality.

#### PR Summary
The changes fix bugs related to index selection and query planning for multi-layer lookups by improving how join and restriction information is handled, and by extending test coverage for point read scenarios.
- **`oss/pg_documentdb/src/opclass/index_support.c`**: Updated handling of outerRelids and join restrictions to ensure proper index path identification.
- **`oss/pg_documentdb/src/planner/documents_planner.c` and `oss/pg_documentdb/include/opclass/bson_index_support.h`**: Modified calls to pass join info into restriction path processing for accurate planning.
- Test files under **`oss/internal/pg_documentdb_distributed`** and **`pgmongo`**: Added new multi-layer lookup queries and expected outputs to verify that lookups consistently pick the _id index.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-08-21 20:14:39 +08:00
Vinod Sridharan a7b1076f31 Merged PR 1773571: Improve IndexScan and IndexOnlyScan Performance 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

Improve performance of indexscans and indexonlyscans

IndexTerm:
>1) BsonIndexTerm is used in several places - reduce the size of this struct by removing isTruncated bool from the struct.
>2)  Add options to check metadata state without parsing the index term at all.
>3) InitializeBsonIndexTerm - do not create a wrapper bson_value_t - just initialize the term directly.

OrderedScans:
>1) For every page we visit, we free the prior buffer and call PageGetTempPageCopy. Instead, keep one buffer per scan and just copy the buffer into the static buffer created per scan
>2) When doing indexonlyscan, don't recreate an indextuple very time - instead reuse the memory and just fill the tuple buffer with what was previously allocated

PgbsonWriter:
>1) Add support for reusing the same writer across invocations by calling reset on the buffer
>2) Add options to retrieve the value from a pgbsonwriter

Composite:
>1) Use a case dispatch for strategies rather than multiple if/else
>2) Work on the serialized bson index terms as much as possible and lazy load the index terms
>3) Skip Compare on bounds calls if there's no bounds declared

----
#### AI description  (iteration 1)
#### PR Classification
This pull request delivers performance improvements and code refactoring for index scan operations.

#### PR Summary
The changes optimize index scans by refactoring tuple construction and buffer page handling for RUM indexes, while also unifying and streamlining BSON composite index term processing.
- `pgmongo_rum/src/rumget.c`: Introduced `IndexBuildTupleDynamic` to build tuples without compression and reuse memory, and refactored page copy logic using `CopyPageContents` and an added `isPageValid` flag.
- `oss/pg_documentdb/src/opclass/bson_gin_composite_entrypoint.c` (and related files): Replaced direct truncation checks with function calls (e.g., `IsIndexTermTruncated`), updated serialized composite index term initialization, and adjusted ordering transform logic.
- `oss/pg_documentdb/src/opclass/bson_gin_index_term.c` (and accompanying headers): Added functions for handling serialized composite index terms and metadata, consolidating truncation flag logic.
- `oss/pg_documentdb_core/src/io/pgbson.c`/`pgbson_writer.h`: Added new writer functions (e.g., reset and value retrieval) to improve memory reuse and reduce allocation overhead.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-08-21 20:14:39 +08:00
Vinod Sridharan 10b395a75b Merged PR 1773514: [Orderby] Improve order by pushdown behavior & selectivity 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

Pushing down order by and filters to a composite index should prefer the smallest index with the highest number of order by columns matching.

To do this we need to participate in the core add_path set up as opposed to after.
Consequently, we mutate the indexPath when its created in create_index_path as opposed to modifying it in the relpathlist_hook. We also reorder the indexes by paths so that for filters, we prefer indexes with fewer paths over many that can match the same paths.

----
#### AI description  (iteration 1)
#### PR Classification
This pull request refines the ORDER BY pushdown mechanism in index scans by adding new processing logic and improving index selectivity while introducing a feature flag for the legacy behavior.

#### PR Summary
The changes enhance query performance by introducing distinct pushdown paths for ORDER BY clauses, refining composite index handling, and updating test expectations to reflect the new "orderByScan" operator. The following key modifications were made:
- **`oss/pg_documentdb/src/opclass/index_support.c`**: Added new functions (e.g., `ConsiderIndexOrderByPushdownNew`, `ProcessOrderByStatements`) to split new versus legacy ORDER BY pushdown logic.
- **Test Files** (under `/oss/internal/pg_documentdb_distributed/src/test/regress/expected/` and `/pgmongo/src/test/docdb_compat/expected/`): Updated expected query plans to use the new "orderByScan" predicate instead of "fullScan".
- **Index and Planner Modules** (e.g., `oss/pg_documentdb/src/index_am/rum.c` and `planner/documents_planner.c`): Enhanced composite index ranking to factor in the number of index paths.
- **Configuration Files** (e.g., `oss/pg_documentdb/src/configs/feature_flag_configs.c`): Introduced the `EnableIndexOrderbyPushdownLegacy` flag for toggling the legacy behavior.
- **Supporting Modules** (in aggregation and query operators): Adjusted functions in files like `bson_query_common.c` and related headers to handle new order-by scan directions consistently.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-08-21 20:14:39 +08:00
Vinod Sridharan df1e6f4e61 Merged PR 1773287: [Explain] A few explain scan improvements based on recent investigations 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

A few explain improvements:
1) Push index usage under each index IXSCAN node (rather than at the top level aggregated)
2) Add a bool for indexonlyscan and add number of heap fetches under indexonlyscan
3) Only mark a scan as _id_ and COLLSCAN if it's an unsharded lookup with no object_id filter
4) For BitmapAnd, don't add an unnecessary FETCH stage before the bitmapindexscan IXSCAN
5) Don't print the index name in the parent FETCH stage of an ixscan
6) For aggregation scans, push the indexdetails to the $cursor stage (as opposed to next level stages).
7) In the backend - handle BitmapAnd for explain scan.

Other bugs found in this:
1) Don't skip writing WAL for multikey status after index build (since the whole page image is already marked for WAL before that point)
2) Don't call ginUpdateStats (call rumUpdateStats) in parallel index build.

----
#### AI description  (iteration 1)
#### PR Classification
This PR enhances explain scan functionality by improving index usage reporting and scan diagnostics across multiple components.

#### PR Summary
The changes refine the explain plan output for various scan types by better handling index conditions, propagating index details, and introducing new diagnostic fields.
- `PostgresExplainHelper.cs`: Refactored index condition processing, improved index usage reporting, and added flags like `isIndexOnlyScan` and handling for heap fetches.
- `mod.rs` (in pg_documentdb_gw explain): Updated query planning and execution stats logic to support Bitmap Index Scan and propagate index details into inner plans.
- C source files (`index_am/rum_adapter.c` and `custom_explain_scan.c`): Simplified WAL logging, improved multi-key index status updating, and added handling for BitmapAndState in scan explanations.
- `utf8.strings.csv`: Added new keys (`isIndexOnlyScan` and `totalDocsAnalyzed`) to support the enhanced explain output.
- `CHANGELOG.md`: Documented the explain scan improvements as a new feature.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-08-21 20:14:39 +08:00
Santiago Fernandez Madero dbdaee55ea Merged PR 1764519: [Index Only Scan] Add support for index only scans at the planner and rum layer 
### Does this PR have any customer impact?

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)

### Are you introducing any new config? If yes, do you have tests with and without them being set?

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

This adds index only scan support for simple aggregate queries that don't project anything. i.e: a count query.

```
db.index_only_scan.explain("allPlansExecution").aggregate([
  { $match: { country: { $gte: "Brazil" } } },
  { $group: { _id: "1", n: { $sum: 1 } } }
]);
```

Next steps:

1. Support on sharded collections
2. Support with index hints
3. Support when projection matches index
4. Check null equality support
5. Add explain entry for truncated terms

----
#### AI description  (iteration 1)
#### PR Classification
This pull request implements a new feature to support index only scans across the planner and rum layers.

#### PR Summary
The changes enable queries to be satisfied solely by the index, reducing table heap access and potentially improving performance.
- **Test Files:** New composite index only scan tests were added in files under `oss/internal/pg_documentdb_distributed/src/test/regress` and `pgmongo/src/test/docdb_compat`, along with new SQL test scripts.
- **Index Support Code:** Modifications in `oss/pg_documentdb/src/opclass/index_support.c` (and related files) add functions such as `ConsiderIndexOnlyScan` and update projection logic to assess and enable index only scan pathways.
- **Rum Layer Adjustments:** Changes in files like `rumget.c` and `rumscan.c` (including their pgmongo counterparts) implement support for projecting index tuple results and adjust cost computations for index only scans.
- **Configuration Updates:** New GUC settings and feature flag changes (e.g. in `/oss/pg_documentdb/src/configs/feature_flag_configs.c`) introduce and control the `EnableIndexOnlyScan` parameter.
- **Planner Integration:** Updates in `documents_planner.c` now invoke the index only scan consideration function during path list construction to integrate the new feature into query planning.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-08-13 15:53:16 +08:00
Vinod Sridharan 90b70c07f2 Merged PR 1760321: [Sort] Support descending sort on an ascending index 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

Today, sorts require the index and sort direction to match exactly. However, RUM indexes can support backward scan as well, so we can push descending sorts to ascending indexes.

Enable this capability. However, this required a few changes to the ordered scan to support moving backward in the tree in RUM
This also required us to use a new order by descending operator since Postgres by default only supports ascending sorts on orderbyop.

----
#### AI description  (iteration 1)
#### PR Classification
New feature enhancement to add descending sort capabilities on ascending indexes.

#### PR Summary
This pull request introduces comprehensive support for descending sort—integrating reverse flags into comparator functions, scan direction adjustments, and new operator metadata—to enable proper pushdown and execution of reverse order queries.
- `oss/pg_documentdb/src/query/bson_dollar_operators.c`: Refactored sort input and comparator functions to incorporate reverse flags (`isReverse`) and adjust value comparisons for descending order.
- `oss/pg_documentdb/src/opclass/bson_gin_composite_entrypoint.c` and related index modules: Added logic for backward scans, including new functions to determine scan direction and reverse comparison adjustments.
- Test SQL files (e.g. `bson_composite_order_by_index_tests.sql`): Extended the test suite with descending order queries and validate explain plans for reverse scan ordering.
- Metadata and operator definitions: Introduced new reverse order operator functions (e.g. `BsonOrderByReverseIndexOperatorId`) and updated index support utilities to correctly process backward scans.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-08-07 11:14:08 +08:00
Vinod Sridharan 0c509dc131 Merged PR 1754381: [Sort] Support sort by id pushdown to the index 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Support pushdown of sort of _id to the primary key btree index if available.

----
#### AI description  (iteration 1)
#### PR Classification
This pull request implements a new feature to push down sort on the _id field into the index, optimizing query ordering for primary key lookups.

#### PR Summary
The changes introduce new logic to detect and push down sort-by-_id operations by leveraging primary key indexes, update sort expression creation to use collation-aware functions, and remove outdated configuration flags.
- `src/opclass/index_support.c`: Added and refactored code to identify primary key indexes and create new index paths when sorting by _id.
- `src/aggregation/bson_aggregation_pipeline.c`: Updated sort function logic to apply collation and proper pushdown criteria, including full-scan clause insertion when necessary.
- Test and expected output files: Adjusted explain plan outputs and renamed feature counters (switching from `sort_by_id_pushdownable` to `sort_by_id`) along with modifying configuration flags.
- `include/opclass/bson_index_support.h` and config files: Introduced a new context structure (`PlannerQueryOrderByData`) and removed obsolete variables related to sort-by-_id pushdown.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-08-07 11:14:08 +08:00
Santiago Fernandez Madero 3245167027 Merged PR 1749636: [Composite] Prefer composite indexes over non composite on the planer layer 
### Does this PR have any customer impact?

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)

### Are you introducing any new config? If yes, do you have tests with and without them being set?

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

[Composite] Prefer composite indexes over non composite on the planer layer

This implements the get_rel_info_hook and reorders the rel indexlist based on the following ordering:

1. primary key index
2. composite index
3. regular indexes
4. wildcard indexes
5. Any other index

This also renames enableCompositeTerm to enableOrderedIndex option on the metadata options we return to the user

----
#### AI description  (iteration 1)
#### PR Classification
Feature enhancement: this PR updates the planner to preferentially use composite indexes over non-composite ones.

#### PR Summary
The pull request implements index ordering logic in the planner to favor composite indexes by introducing new sorting and logging functions, replacing the legacy “enableCompositeTerm” with “enableOrderedIndex”, and updating the associated feature flags and tests.
- **`src/planner/documents_planner.c`**: Added functions (e.g., `GetIndexOptInfoSortOrder`, `CompareIndexOptionsFunc`, and `ExtensionGetRelationInfoHook*`) to prioritize indexes so that composite indexes rank higher.
- **`src/configs/feature_flag_configs.c` and `src/configs/testing_configs.c`**: Introduced new boolean flags (`enableIndexPriorityOrdering` and `logRelationIndexesOrder`) to control the index ordering and logging behavior.
- **`src/metadata/index.c` and `src/commands/create_indexes.c`**: Updated index creation and comparison logic to replace `enableCompositeTerm` with `enableOrderedIndex` for consistent composite index handling.
- **Test files (under `/oss/pg_documentdb/src/test/regress/` and `/pgmongo/src/test/`)**: Revised expected outputs and SQL tests to validate that composite indexes are now preferred in the query planner.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-08-07 11:14:08 +08:00
Vinod Sridharan 623ef26f85 Merged PR 1742378: [Hints] Add support for index hints in DocumentDB 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

Index hints are a mongo feature that allows customers to specify which index they would like to target for a query. Typically this is used in diagnostic scenarios when testing query performance.
With the new composite index op class coming in, we’re introducing side by side support for composite which makes it so we would want customers to be able to test which index is better for planning.

In this spirit, index hints is introduced which follows the same infra as the force pushdown for geonear  and text and adds a new option for index hints.

As part of this the code is also refactored to cleanup some of the logic in the relpathlist and fix correctness issues around text index force pushdown (which was needed to handle index hints)

----
#### AI description  (iteration 1)
#### PR Classification
This pull request introduces a new feature to support index hints in DocumentDB.

#### PR Summary
The changes integrate index hint support into query planning and execution, allowing users to force a specific index during query processing. Key updates include:
- **`src/opclass/index_support.c`**: Added new functions and a force index operator type (`ForceIndexOpType_IndexHint`) for filtering available indexes and generating custom operator expressions based on index hints.
- **`src/planner/documents_planner.c`**: Modified base table query planning logic to detect index hints and adjust index pushdown, including updates to path and restriction handling.
- New UDFs (e.g., `bson_dollar_index_hint`) and configuration variables (`EnableIndexHintSupport`, `UseLegacyForcePushdownBehavior`) were added to support the hint mechanism.
- Test files and SQL scripts (e.g., in `/oss/internal/pg_documentdb_distributed/src/test/regress/`) were introduced or updated to validate index hint pushdown behavior.
- Documentation and release notes (e.g., `docs/pgmongo/design_docs/data_plane/indexing/index_hints.md`, `pgmongo_1.106.md`, and `CHANGELOG.md`) were updated to describe the index hint feature.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-07-23 15:56:08 +08:00
Vinod Sridharan 7688125032 Merged PR 1737756: [RUM] Support ordered scans without explicit order by 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

1) Move forceRumOrderedScan from the documentdb rum layer to the rum index layer
2) Support having ordered scans even without there being an explicit order by in the query
3) Push this order by information to the op class using the GIN_SEARCH_MODE

This is part one of many to support cleaning up the rum.c in the documentdb layer and push the extensibility needed into the index am directly.

----
#### AI description  (iteration 1)
#### PR Classification
This pull request implements a new feature to support ordered scans in RUM without requiring an explicit "order by" clause.

#### PR Summary
The changes refactor and enhance the RUM scan logic to support ordered scans by introducing a new ordering mechanism and related configuration. This improves the flexibility of index scanning operations.
- **`/oss/internal/pg_documentdb_extended_rum/src/rumget.c` & `/pgmongo_rum/src/rumget.c`**: Refactored scan functions to use a dedicated `orderByEntry` (and updated recheck flags) instead of relying on a key from the keys array.
- **`/oss/internal/pg_documentdb_extended_rum/src/rumscan.c` & `/pgmongo_rum/src/rumscan.c`**: Updated functions such as `MoveScanForward` and `startScanEntryOrderedCore` to handle ordered scanning through the newly defined `orderByEntry` and centralized `orderStack`, simplifying signature parameters.
- **Configuration and Initialization Files**: Introduced the GUC variable `RumForceOrderedIndexScan` and modified scan key initialization in `rumutil.c` and related header files to support forced ordered index scans.
- **Test and Opclass Updates**: Adjusted test configurations and expected outputs, and updated composite index-related code (e.g., removal of the `BuildCompositeOrderByScanKeyArgument` function) to reflect the new order scan behavior.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-07-23 15:56:08 +08:00
Vinod Sridharan a597ec62c2 Merged PR 1733513: [Composite] Enable pushdown of Group by to the composite index 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

Allow pushing down of the Group By _id field to the index if it's a "path" and the index is not multi-key.

----
#### AI description  (iteration 1)
#### PR Classification
This pull request implements a new feature to enable pushdown of GROUP BY operations to composite indexes for improved query performance.

#### PR Summary
The changes optimize query execution by modifying query planning and index support to allow GROUP BY clauses to be pushed down to composite indexes, with careful handling for multi-key indexes.
- `pg_documentdb/src/aggregation/bson_aggregation_pipeline.c`: Added logic to validate GROUP BY expressions and conditionally inject full scan clauses for index pushdown.
- `pg_documentdb/src/opclass/index_support.c` (and related index files): Enhanced order by and group by handling for composite indexes, including adjustments for non-multi-key versus multi-key cases.
- Test files under `/oss/internal/pg_documentdb_distributed/src/test/` and `/pgmongo/src/test/docdb_compat/expected/`: Updated SQL tests and expected output to validate the new pushdown behavior.
- `pg_documentdb/src/commands/create_indexes.c` and configuration files: Updated index creation options to support ordered composite indexes.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-07-23 15:56:08 +08:00
Vinod Sridharan 7efbfff5f3 Merged PR 1732117: [Composite] Force requirement that the first path be specified on cost estimate 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
In composite index, require that the first column be specified.

----
#### AI description  (iteration 1)
#### PR Classification
Bug fix to ensure composite index cost estimation requires the first column to be specified.

#### PR Summary
This PR enforces that composite indexes include a filter on the first column during cost estimation, preventing improper usage of composite indexes.
- In `src/opclass/bson_gin_composite_entrypoint.c`, a new function `CompositePathHasFirstColumnSpecified` is added to check that the first column is specified with a constant value.
- In `src/index_am/rum.c`, the cost estimation logic is modified to return an infinite cost if the composite index does not have the first column specified.
- In `include/opclass/bson_gin_composite_scan.h`, the interface is updated to declare the new function.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-07-23 15:56:08 +08:00
Vinod Sridharan 2c2306036d Merged PR 1728407: [Composite] Support for unique indexes with composite op-class 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Add support for composite unique indexes
1) Support for create index with unique composite
2) Support op-class for unique exclusion constraint check
3) Support for querying the unique index as usual - to do this we "append" the shard key value check instead of pre-pending so that all the prefix on query just works off the gate.

----
#### AI description  (iteration 1)
#### PR Classification
This pull request introduces a new feature by enabling unique indexes to use the composite op‐class and updates the expected outputs used by regression tests.

#### PR Summary
The changes add new expected output files reflecting the behavior of composite unique indexes in various query plans and EXPLAIN outputs. These outputs cover a range of query operators (such as $eq, $ne, $gt, $in, etc.) and complex logical constructs to validate the new composite op‐class support across both local and distributed execution modes.
- Added the expected output file ``bson_query_operator_tests_pg17_explain_index_comp_unique.out`` under ``pgmongo/src/test/docdb_compat/expected/``.
- Added corresponding expected output files under ``oss/internal/pg_documentdb_distributed/src/test/regress/expected/`` to cover distributed query planning.
- Updated test outputs to verify the behavior of composite unique indexes with various query operators and nested expressions.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-07-23 15:56:08 +08:00
Vinod Sridharan 4c190c5f02 Merged PR 1713276: [POC: Order by] Support order by descending pushdown to the index if available 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
1) Support order by descending as a feature under a feature flag if the underlying index can support it
2) Fix up memory performance issues causing OOMs with composite index with large numbers of bounds
3) On doing this noticed that row_get_bson was busted for int arrays, fix this behavior to not crash
 2025-07-09 18:56:11 +08:00
Vinod Sridharan 71e7d304b1 Merged PR 1695552: [Composite|Order by] Enable JSTest validation for these features 
### Does this PR have any customer impact?

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)

### Are you introducing any new config? If yes, do you have tests with and without them being set?

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

----
#### AI description  (iteration 1)
#### PR Classification
This pull request implements a new feature that enhances explain plan output and composite index functionality by adding order‑by pushdown and related index option validations.

#### PR Summary
The changes add new metrics in explain plans and strengthen composite index handling for order-by pushdown across the system. Key changes include:
- **`/gateway/managed/Mongo.Service.Postgres/Helpers/PostgresExplainHelper.cs` & `/oss/pg_documentdb_gw/src/explain/mod.rs`** – Added fields like `sortKeysCount`, `sortKey`, `presortedKey(s)`, and `hasOrderBy` to enhance explain output.
- **`/oss/pg_documentdb/src/metadata/index.c` and related index management files** – Implemented improved composite index option checks and support functions such as `CompositeIndexSupportsOrderByPushdown` and equality-range predicate validation.
- **UDF and operator files (e.g., `bson_dollar_comparison--*.sql`, `/oss/pg_documentdb/src/query/bson_dollar_operators.c`)** – Introduced new functions like `bson_dollar_fullscan` to support full scan pushdowns in combination with composite indexes.
- **`/pgmongo_rum/src/rumget.c`** – Enhanced ordered scan handling and duplicate management for multi-key indexes.
- **Pipeline and configuration files (e.g., `.pipelines/templates/single_node_tests.yml`)** – Updated test scripts and experimental configuration parameters to enable the new composite opclass and order‑by pushdown features.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-06-13 15:13:47 +08:00
Vinod Sridharan c6bc2bcb73 Merged PR 1694458: [POC|Order by] Order by Index & Pushdown improvements 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
More general improvements to order by and composite:
Create Indexes
1) Support creating a composite index side by side with a regular index
2) Block creation of descending or wildcard and unsupported set ups for composite indexes

Planner:
1) Support ordering by the Nth path of an index if the prefixes are equality.
2) Validate that if it's not the first column, prefixes are provided
3) Support incremental sort with index pushdown.

----
#### AI description  (iteration 1)
#### PR Classification
This pull request delivers a new feature and API enhancement to enable order-by index pushdown and refined composite index support.

#### PR Summary
The update streamlines composite index processing by introducing stricter equivalency checks, new UDF functions for full-scan order-by pushdown, and enhanced validations during index creation while updating explain outputs and tests.
- `src/metadata/index.c` refines index options equivalency logic to correctly compare composite term settings.
- `src/opclass/bson_gin_composite_entrypoint.c` and `src/index_am/rum.c` add functions to support order-by pushdown and manage duplicate entries efficiently.
- `sql/udfs/query/bson_dollar_comparison--0.105-0.sql` (and related operator files) introduce new UDFs including `bson_dollar_fullscan` to enable full scan operations.
- `src/commands/create_indexes.c` enforces new composite index validations (restricting use to ascending, non-wildcard regular indexes) with precise error reporting.
- Test scripts and explain plan outputs in the `regress` and `pgmongo` directories are updated to confirm the new composite order-by behavior.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-06-13 15:13:47 +08:00
Vinod Sridharan 06a96c8975 Merged PR 1691723: Fix up correctness of composite and orderby 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix/Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
1) Orderby on null/undefined documents was incorrect - fix up the ordering
2) Equality on null/undefined was also incorrect fix this up
3) When traversing the document, we were doing an extra traversal into arrays when it was not needed. fix this up
4) Order by on composite index didn't handle nulls correctly - fix this too.
5) Support pushdown of $elemMatch into composite indexes without array terms.

----
#### AI description  (iteration 1)
#### PR Classification
This pull request fixes bugs in the composite index and orderby functionality.

#### PR Summary
The changes correct the orderby comparison logic and improve composite index term generation in BSON processing. The pull request also adds new functions to handle intermediate array paths and introduces legacy behavior flags, while supplementing the fixes with comprehensive integration tests.
- **`src/query/bson_dollar_operators.c`**: Modified `OrderByVisitArrayField` to pass the correct BSON value and added new functions for array path handling.
- **`src/opclass/gin_index_term.c` and related opclass files**: Introduced `GenerateValueMaybeUndefinedTerm` and updated index term comparison to distinguish between undefined and partially undefined values.
- **`pgbson_utils.c`**: Adjusted array traversal logic with a new callback for missing intermediate array paths and a flag to control array traverse optimization.
- **Test files under `/oss/pg_documentdb/src/test/regress/expected/` and `/pgmongo/src/test/regress/expected/`**: Added extensive composite ordering and filtering tests to validate the corrected behavior.
- **`configs/feature_flag_configs.c`**: Added custom GUCs for `useLegacyOrderByBehavior` and `useLegacyNullEqualityBehavior` to manage legacy ordering and null equality rules.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-06-05 10:31:32 +08:00
Vinod Sridharan 0ddb09436c Merged PR 1690832: [Composite] Remove array term generation for composite indexes 
### Does this PR have any customer impact?
Not yet

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature/POC

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

This adds more changes for orderby support in the index
1) Fixes the correctness issue arising from indexing arrays for order by - we remove the top level array term from indexing. To handle this, we move the query operators to support querying the index correctly when the query is on an array
2) For `$elemMatch` this temporarily makes elemMatch just a full scan for now (a subsequent change wil tackle updating the logic)
3) Adds orderby to the top level schema for the internal extension
4) For exists false/true, we were pushing a lot more terms to the runtime recheck. Ensure that we're hitting the right index terms and evaluate correctly against the index.

----
#### AI description  (iteration 1)
#### PR Classification
This pull request is a code cleanup and enhancement that removes array term generation for composite indexes and updates related index term handling.

#### PR Summary
The changes streamline composite index generation by eliminating dedicated array terms, and they adjust boundary, metadata, and recheck logic to ensure consistent behavior. Key updates include:
- **`oss/pg_documentdb/src/opclass/bson_gin_composite_core.c`**: Removed redundant array term generation and introduced functions like `SetArrayEqualityBound` to properly handle equality bounds for arrays.
- **`oss/pg_documentdb/src/opclass/gin_index_term.c`**: Updated index term serialization and metadata management (including handling of undefined values via `GenerateValueUndefinedTerm`) to align with the new composite index behavior.
- **Test and expected output files**: Revised expected composite index term outputs and SQL tests to reflect the removal of array-specific terms.
- **Documentation and configuration**: Updated design docs and removed the configuration flag for skipping array term generation, consolidating the new composite index strategy.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-06-05 10:31:32 +08:00
Vinod Sridharan c8fdba1168 Merged PR 1684751: [Composite|Order] Supporting order by on composite index 
### Does this PR have any customer impact?
No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

----
#### AI description  (iteration 1)
#### PR Classification
This pull request introduces a new feature to support “order by” queries on composite indexes and refines how composite query operators are handled.

#### PR Summary
The changes enable ordering by composite indexes by expanding the $all operator into an AND of expressions, adjusting index pushdown and term generation logic, and enforcing index-only paths when sequential scans are disabled. Key changes include:
- **`src/query/query_operator.c`**: Added the `ExpandExprForDollarAll` function to decompose $all into composite AND expressions, and integrated its use controlled by a new feature flag.
- **`src/planner/documents_planner.c`**: Introduced helper functions (e.g. `TrimPathListForSeqTypeScans` and `ForceExcludeNonIndexPaths`) to eliminate sequential scan paths when `ForceDisableSeqScan` is enabled, ensuring better index-based execution.
- **`src/configs/feature_flag_configs.c` & `src/configs/testing_configs.c`**: Added new configuration flags like `ExpandDollarAllInQueryOperator`, `SkipGeneratingArrayTermForCompositeIndex`, and `ForceDisableSeqScan` to control composite index behavior.
- **Composite Index Code (e.g. in `bson_gin_composite_entrypoint.c` and related files)**: Updated index strategy handling and term generation to support composite ordering and correct treatment of array terms.
- **Test Files**: Numerous expected output and SQL regression tests (both in the distributed and pgmongo modules) were updated to reflect the new composite index ordering and query operator changes.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-06-05 10:31:32 +08:00
Vinod Sridharan efa2b802f6 Merged PR 1659003: [Composite OpClass] Enable Runtime Functions and add test coverage 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature/POC
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Fix up runtime index recheck function support to reduce loading documents.
This should reduce the number of runtime rechecks that are needed for documents and trust the index more as we will evaluate these clauses directly on the index first.

Also add option to trigger jstest coverage for the new opclass - and set them to run on the MX jstests - this is still disabled right now since there are some failing tests but will be enabled separately.

----
#### AI description  (iteration 1)
#### PR Classification
New feature: This pull request introduces runtime functions for composite index operations and enhances test coverage.

#### PR Summary
This pull request enables runtime functions for composite index operations, improving the flexibility and efficiency of index handling. It also adds test coverage to ensure the reliability of these new features.
- `bson_gin_composite_entrypoint.c`: Refactored comparison logic to support runtime functions for index rechecks.
- `bson_gin_composite_core.c`: Implemented `IsValidForIndexValue` function to handle various BSON index strategies with runtime rechecks.
- `create_indexes.c`: Modified index creation logic to conditionally enable new composite index opclass based on configuration.
- `bson_gin_composite_private.h`: Added `IndexRecheckArgs` structure to support runtime recheck strategies.
- `testing_configs.c`: Introduced configuration options for enabling the new composite index opclass by default.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-05-09 14:08:53 +08:00
Vinod Sridharan 71f084dca8 Merged PR 1644618: [Index] Improve wildcard index performance 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Improve index build and insert performance (especially for wildcard indexes).
The current wildcard indexes are heavily bloated - for 50 GB of data that is very array/document heavy, we could end up producing a 300 GB index in some cases.

To improve this, there's considerations of parallel index builds (introduced in PG17+), but also there's something to be said about reducing index bloat. This PR does that in two ways:

1) Given that index truncation happens at ~2500 bytes (or less), the possibility of compression for index terms is low. Therefore, even if the primary document is compressable, the index isn't. This introduces compression for index entries with a custom GUC that manages that. Especially for larger arrays and documents this yields a non-trivial saving.

2) There's a realization that with wildcard indexes and arrays, there's term duplication: An path `"a": [ 1, 2, 3 ]` generates
```
{ "a": [ 1, 2, 3 ] }
{ "a": 1 }
{ "a": 2 }
{ "a": 3 }
{ "a.0": 1 }
{ "a.1": 2 }
{ "a.2": 3 }
```
where  the last 3 terms can be made redundant with the first 3. this attempts to remove those extra terms by now only generating:
```
{ "a": [ 1, 2, 3 ] }
{ "a": 1 }
{ "a": 2 }
{ "a": 3 }
```
Query plans that query a.0 will not be pushed to this index currently.
However, in the future this can be removed by querying a.0 as 'a' and running a runtime recheck.

Given that this only impacts queries that explicitly query by integer keys, and on wildcard indexes, this is deemed acceptable for the improvement in size reduction on the index.

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request introduces support for wildcard terms in the indexing system, enhancing query capabilities and performance.
- Refactored `SerializeBsonIndexTermCore` in `gin_index_term.c` to use a new helper function `BuildSerializedIndexTerm`.
- Added `SerializeBsonIndexTermWithCompression` function in `gin_index_term.c` for handling compressed index terms.
- Implemented `QueryPathHasDigits` and `FailIfQueryPathHasDigitsForWildcard` functions in `bson_gin_entrypoint.c` to validate query paths for wildcard indexes.
- Updated various query extraction functions in `bson_gin_core.c` to use `filterElement` from `BsonExtractQueryArgs` instead of directly accessing the query.
- Enhanced `GenerateIndexExprStr` in `create_indexes.c` to include options for reduced wildcard terms and generating not-found terms.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-23 12:04:26 +08:00
Vinod Sridharan c3baecd540 Merged PR 1642833: [Wildcard Index] Improve term generation for wildcard index 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
1) InitializeBsonIndexTerm called BsonValueToPgbsonElement that was using the bson_iter_next a lot. This is a lot of overhead when we know it's a single element. Use an unsafe variant of it to generate a pgbsonelement

2) Introduce a MatchAndRecurse option for index paths that allows the term generation logic to know whether it's an exact match or if the child paths also match. This is currently unused, but will be used in a subsequent change.

----
#### AI description  (iteration 1)
#### PR Classification
Code improvement for term generation in wildcard index.

#### PR Summary
This pull request enhances the term generation process for wildcard indexes by introducing an unsafe transformation method and refining index traversal options.
- `pgbsonelement.c`: Added `FillPgbsonElementUnsafe` and `BsonValueToPgbsonElementUnsafe` functions for unsafe BSON element transformation.
- `bson_gin_entrypoint.c`: Introduced `IndexTraverse_MatchAndRecurse` option to improve path matching and recursion.
- `feature_flag_configs.c`: Added feature flag `indexTermUseUnsafeTransform` to enable unsafe transformation.
- `gin_index_term.c`: Utilized the unsafe transformation method based on the new feature flag.
- `bson_gin_core.c`: Updated term path generation to handle the new traversal option.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-17 18:40:16 +08:00
Vinod Sridharan 73db6c6000 Merged PR 1639906: [Order by POC] Push down order by to the index under a flag 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
PoC

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

Implement ordered scan in the index layer - leverage GIN extensibility points to do so.
Allow order by pushdown to the index.

Still doesn't work:
- Truncation
- Array terms and order by

Still needs improvement:
- Performance when there's disjoint index trees to be scanned.

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request introduces the ability to push down the `ORDER BY` clause to the index under a feature flag.
- `pgmongo_rum/src/rumget.c`: Added functions `MoveScanForward`, `scanGetItemOrdered`, and `startOrderedScanEntries` to handle ordered scans.
- `pgmongo/src/test/regress/expected/internal_bson_composite_order_by_index_tests.out`: Added tests to validate the new ordered index scan feature, including correctness and edge cases.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-17 18:40:16 +08:00
Vinod Sridharan a72e6563d6 Merged PR 1634773: [Orderby POC] Add plumbing in planner for orderby pushdown 
### Does this PR have any customer impact?
No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
POC Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Not yet

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

Add plumbing in planner for orderby pushdown to the index. This is guarded behind a GUC and doesn't work yet - but at least introduce this for later.

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request adds support for orderby pushdown in the planner.
- `index_support.c`: Introduced `ConsiderIndexOrderByPushdown` function to handle orderby pushdown logic.
- `rum.c`: Added support for orderby pushdown in RUM index scan and validation.
- `metadata_cache.c`: Added `BsonOrderByIndexOperatorId` function.
- `testing_configs.c`: Defined new configuration variable `EnableIndexOrderbyPushdown`.
- New test files `bson_composite_order_by_index_tests.sql` and `bson_composite_order_by_index_tests.out` to validate the feature.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-10 10:29:08 -06:00
Vinod Sridharan 6349b5adca Merged PR 1633710: [Composite POC] Improve performance on multi-key path scenarios 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
For multi-key paths, track them on insert/update.

When doing a query, check the index for multi-key paths - if there are any then rewrite the query differently.

----
#### AI description  (iteration 1)
#### PR Classification
Performance improvement for multi-key path scenarios in composite indexes.

#### PR Summary
This pull request enhances the performance of multi-key path scenarios by introducing optimizations and refactoring the handling of composite indexes.
- `oss/pg_documentdb/src/index_am/rum.c`: Introduced `IndexMultiKeyStatus` enum and `CheckIndexHasArrays` function to optimize index scans.
- `oss/pg_documentdb/src/opclass/bson_gin_composite_core.c` and `oss/pg_documentdb/src/opclass/bson_gin_composite_entrypoint.c`: Added functions to merge single variable bounds and parse composite query specifications.
- `oss/pg_documentdb/include/index_am/documentdb_rum.h`: Updated function signatures to use `IndexAmRoutine` pointers.
- `pgmongo/src/helio_gin_am/helio_gin_utils.c`: Adjusted function calls to use `IndexAmRoutine` pointers.
- `pgmongo/src/metadata/oid_cache.c`: Changed `missingOk` parameter to `true` in `RunIndexOpAsUserProcedureId`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-10 10:29:08 -06:00
Vinod Sridharan b657d1057e Merged PR 1633206: [Composite POC] Fix correctness in composite op class 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature/POC
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Fix up diff and correctness issues with the composite op class
- Fix $range correctness with arrays
- Fix type bracketing for $range
- Fix term generation for numbers.

----
#### AI description  (iteration 1)
#### PR Classification
Bug fix

#### PR Summary
This pull request addresses correctness issues in the composite operation class.
- `bson_gin_composite_entrypoint.c`: Improved handling of truncation and scan keys in `gin_bson_composite_path_consistent` and `gin_bson_composite_path_extract_query`.
- `bson_gin_composite_core.c`: Enhanced `UpdateRunDataForVariableBounds` to correctly handle term mapping and bounds.
- `bson_query_operator_tests_index_composite.out`: Updated expected test results to reflect the corrected behavior.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-10 10:29:08 -06:00
Vinod Sridharan 77f9dc3b75 Merged PR 1631702: [Composite POC] Add Basic tests for Composite Index OpClass 
### Does this PR have any customer impact?
No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Testing

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Enable query_operator_tests for composite index op-class
Found 2 minor bugs that's fixed

Found issues in the existing tests that didn't use index in certain cases. Fix that too.

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request adds basic tests for Composite Index OpClass.
- Changes in `src/opclass/bson_gin_composite_core.c` include the addition of multiple functions to handle various BSON strategies (`$all`, `$in`, `$nin`, bitwise operators, `$type`, etc.) and error handling improvements.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-10 10:29:08 -06:00
Vinod Sridharan 1a7e1b65b1 Merged PR 1630725: [Composite POC] Support multi-range operators 
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Support operators such as `$in`, `$nin`, `$type`, `$ngte` etc. These are multi-range queries scanning multiple parts of the index.

Things still remaining
1) Testing the operators
2) Runtime evaluation of operators against index terms.

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request introduces support for multi-range operators in composite indexes.
- Added functions in `bson_gin_composite_core.c` to handle multi-boundary conditions for operators like `$in`, `$nin`, bitwise operators, and `$type`.
- Implemented `ParseOperatorStrategy` to parse and apply different operator strategies.
- Enhanced existing functions to support new multi-range operator logic and error handling.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-10 10:29:08 -06:00
Vinod Sridharan 271537908f Merged PR 1630327: [Composite POC] Cleanup: Move core operator logic from entrypoint.c to core.c 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Move core composite operator logic from entrypoint.c to core.c

----
#### AI description  (iteration 1)
#### PR Classification
Code cleanup

#### PR Summary
This pull request moves core operator logic from `entrypoint.c` to a new file `core.c` for better organization and maintainability.
- Added `bson_gin_composite_core.c` with core operator logic for BSON composite index.
- Implemented functions for processing index bounds and query strategies in `bson_gin_composite_core.c`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-10 10:29:08 -06:00
Vinod Sridharan 5ad889feeb Merged PR 1629302: [Composite POC] Fix up operators and type bracketing 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
1) Fix up the remaining operators - implementations are sub-optimal
2) Add type bracketing options for all types and use them in operators
3) Add support for truncation of terms

TODO:
- Support recheck on the index for operators than just bounds
- Support N boundaries for index terms.

----
#### AI description  (iteration 1)
#### PR Classification
Bug fix

#### PR Summary
This pull request addresses issues with operators and type bracketing in BSON composite indexes.
- Added `bson_gin_index_types_core.c` to manage BSON type bounds for indexes.
- Updated `bson_gin_composite_entrypoint.c` to fix operator handling and apply type bracketing for various BSON index strategies.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-10 10:29:08 -06:00
Vinod Sridharan b31d2b2751 Merged PR 1628723: [Composite POC] Planner integration and basic plumbing 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
- This is the second part of the plumbing for composite indexes.
- Add the planner and query integration to push $eq filters in the composite index

Still not done:
- Correctness of query
- inequality and other operators (`$in/$regex/etc`)
- Type bracketing
- Truncation

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request integrates a planner and adds basic plumbing for composite index operations.
- `bson_gin_composite_entrypoint.c`: Added functions for handling composite index paths, parsing bounds, and modifying scan keys.
- `bson_composite_index_tests_insert_terms.out`: Added tests to validate composite index functionality and queries.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-07 11:39:49 +08:00
Vinod Sridharan db6a3de7fd Merged PR 1627378: [Composite POC] Create single composite operator class 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
As part of testing out the composite operator classes, introduce a new composite operator class for Bson. This will be behind a test only GUC and will be for testing composite index pushdown.

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request introduces a new composite operator class for BSON GIN indexes in PostgreSQL.
- Added `bson_gin_composite_entrypoint.c` to implement GIN operator functions for BSON composite indexes.
- Created test cases in `bson_composite_index_tests_insert_terms.out` to validate term generation for composite paths.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-07 11:39:49 +08:00
Vinod Sridharan 338e1dc6ef Merged PR 1622609: [Query Perf] Improve performance of index pushed down large $in queries 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Improve the consistent function performance for $in in the index.

----
#### AI description  (iteration 1)
#### PR Classification
Performance improvement

#### PR Summary
This pull request improves the performance of index pushed down large `$in` queries by introducing a fast path for certain conditions.
- `bson_gin_core.c`: Added a fast path for `$in` queries without truncation, nulls, or regex.
- `feature_flag_configs.c`: Introduced a new feature flag `EnableRumInOperatorFastPath`.
- Updated regression tests in `regex.sql` and `regex.out` to include scenarios for the new fast path feature.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-07 11:39:49 +08:00
Vinod Sridharan 2856f55e44 Merged PR 1576168: Fix vector search for OSS & add support for force pushdown of vector index. 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
- Vector search pushdown to vector index was broken in OSS.
- Add option to force push vector search to vector index.

----
#### AI description  (iteration 2)
#### PR Classification
Bug fix and new feature.

#### PR Summary
This pull request fixes vector search for OSS and adds support for force pushdown of vector index.
- Added new test files `bson_aggregation_pipeline_tests_vector_hnsw_planner.out` and `bson_aggregation_pipeline_tests_vector_hnsw_planner.sql` for vector search and HNSW index.
- Updated `index_support.c` to include support for vector index pushdown, including new functions `UpdateIndexListForVector`, `MatchIndexPathForVector`, and `ThrowNoVectorIndexFound`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-02-13 17:09:16 -08:00
Vinod Sridharan f505385088 Merged PR 1550572: [Cleanup] Remove dead feature flags in Pgmongo code 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?

### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description
- Refactor to remove dead features or features already enabled by default
- Remove any checks of cluster version < V21 that are not dependent on persistent storage.

----
#### AI description  (iteration 1)
#### PR Classification
Code cleanup

#### PR Summary
This pull request removes dead feature flags and related code from the Pgmongo codebase.
- Removed the `OptimizeIndexExpressionsForDollarIn` function and related comments from `src/opclass/index_support.c`.
- Simplified the `ProcessLookupCoreWithLet` function in `src/aggregation/bson_aggregation_nested_pipeline.c` by removing conditional logic based on cluster version.
- Updated test cases in `src/test/regress/expected/conflicting_indexes.out` to reflect the removal of extended index filters and unsupported expressions in partial indexes.
 2025-01-15 07:48:41 +00:00
Zinan Ren 0ad54c20b8 Merged PR 1540839: [OSS] Remove helio references from sources of documentdb and documentdb_distributed 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description
Remove helio references from documentdb
Remove helio references from documentdb_distributed

----
#### AI description  (iteration 1)
#### PR Classification
Code cleanup

#### PR Summary
This pull request removes references to "helio" and replaces them with "documentdb" in the sources of `documentdb` and `documentdb_distributed`.
- `metadata_cache.h` and `metadata_cache.c`: Replaced "Helio" with "DocumentDB" in variable names and function names.
- `pgmongo.c`: Updated include statements and function calls to use "documentdb" instead of "helio".
- `planner.c`: Updated planner function and related calls to use "DocumentDBApiPlanner".
- Moved several header files from `helio_*` to `documentdb_*` naming convention without code changes.
- `commands_common.c` and `create_indexes.c`: Updated function calls and variable names to use "DocumentDB" instead of "Helio".
 2025-01-15 07:48:40 +00:00
Zinan Ren 8e7bd960bf Merged PR 1540262: [OSS] Remove helio references from sources of documentdb_core 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description
Remove helio from file name
Remove helio references from source content except schema reference
Start local documentdb server with the documentdb extension
Regression tests using the documentdb  extension

----
#### AI description  (iteration 1)
#### PR Classification
Code cleanup

#### PR Summary
This pull request removes references to "helio" from the `documentdb_core` source code.
- `scripts/utils.sh`: Renamed variables from `preloadExtensions` to `preloadLibraries`.
- `pg_documentdb_core/include/utils/error_utils.h`: Updated function and macro names to replace "Helio" with "DocumentDB".
- `pg_documentdb_core/include/io/helio_bson_core.h` and `pg_documentdb_core/include/query/helio_bson_compare.h`: Moved to `pg_documentdb_core/include/io/bson_core.h` and `pg_documentdb_core/include/query/bson_compare.h`.
- `pg_documentdb_core/src/collation/collation.c` and `pg_documentdb_core/src/io/pgbson.c`: Replaced error codes and messages to use "DocumentDB" instead of "Helio".
- `scripts/validate_helio_extension_state.sh`: Added exceptions for certain files to skip validation for "helio" references.
 2025-01-15 07:48:40 +00:00
Vinod Sridharan 3f9a507562 Merged PR 1539950: [OSS] Move helio_api and helio_distributed sources under documentdb 
### Does this PR have any customer impact?
No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description
Move source files under new directory.

----
#### AI description  (iteration 1)
#### PR Classification
Code reorganization

#### PR Summary
This pull request moves the `helio_api` and `helio_distributed` sources under the `documentdb` directory, updating references and initialization logic accordingly.
- Added `pg_documentdb.c` for initializing the `pg_documentdb` extension.
- Added `documentdb_distributed.c` for initializing the `pg_documentdb_distributed` extension.
- Updated `validate_helio_extension_state.sh` to include new checks for `documentdb`.
- Moved `helio_api_extension_version.c` to `documentdb_extension_version.c` without code changes.
- Updated various `Makefile` configurations to reflect the new directory structure and dependencies.
 2025-01-15 07:48:39 +00:00