upstream/documentdb
2
0 Fork 0
mirror of https://github.com/FerretDB/documentdb.git synced 2026-11-07 11:32:47 -05:00
Code Releases 1 Activity
1,134 Commits 7 Branches 21 Tags
main
Commit Graph
54 Commits
Author SHA1 Message Date
Vinod Sridharan 2108af32e6 Merged PR 1812986: [Hidden Index] Support Hidden Indexes with Distribution 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Add support for hidden indexes with distributed tables.

- Add function that supports per node distribution with shards: Select all shards for the table and pick one shard per node.
- Add a planner worker function that executes on every shard
- On the shard skip execution if it's not the selected shard
- Run the local function on the worker function.

- Plumb in hooks to execute the index functions remotely.

----
#### AI description  (iteration 1)
#### PR Classification
This pull request implements a new feature to support hidden indexes in a distributed environment.

#### PR Summary
The changes integrate hidden index support into distributed index update operations by introducing new per-node functions, updating API hooks, and modifying planner and UDF logic for proper shard routing and logging.
- **`node_distribution_operations.c`**: New file that adds functions (e.g., `UpdateDistributedPostgresIndex` and its worker) to execute index update commands per node in a distributed setting.
- **`diagnostic_commands_common.c`**: Introduces `command_node_worker` to ensure that update commands target the appropriate shard based on table identity.
- **`coll_mod.c` and `coll_mod.h`**: Updated to use `UpdatePostgresIndexWithOverride` and correctly incorporate hidden index modifications with distributed operations.
- **API hooks and metadata files** (e.g., `api_hooks.c`, `api_hooks_def.h`, `api_hooks.h`, and `metadata_cache.c`): Enhanced to support new function pointers and caching mechanisms for hidden index updates via the new command worker.
- **SQL UDF scripts**: New UDF definitions for `command_node_worker` and `update_postgres_index_worker` enable proper registration and execution of hidden index operations in the distributed framework.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
 2025-09-26 10:26:08 -07:00
Vinod Sridharan 95d7a4b6a6 Merged PR 1806432: [OSS] Remove explicit RUM dependency on the documentdb extension 
### Does this PR have any customer impact?
No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

Make DocumentDB not explicitly depend on the RUM extension (it's done implicitly) on the binary.
Instead pull in the SQL dependency into documentdb and reference the C functions internally. This allows us to swap between rum.so and extended_rum.

For Pg18 (where there isn't yet official RUM support) default to using extended_rum for the .so file and remove all dependencies on the "rum.so"

----
#### AI description  (iteration 2)
#### PR Classification
This PR is an API and dependency update that removes the explicit dependency on the external RUM extension by integrating its functionality internally.

#### PR Summary
The update integrates the RUM text-search capabilities directly into DocumentDB and cleans up the external linkage.
- `oss/internal/pg_documentdb_extended_rum/src/rum_ts_utils.c` is newly added with core text-search functions.
- `oss/pg_documentdb/src/index_am/rum.c` is modified to load internal wrappers that call the new RUM functions.
- SQL UDF adapter files and operator class definitions are updated to reference the internal functions instead of public RUM functions.
- Build scripts, metadata, and control files are adjusted to remove the explicit dependency on the external RUM extension.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
 2025-09-18 08:10:29 -07:00
William Wang 190c5f7710 Merged PR 1750825: [Full RBAC] Adding Support for RolesInfo Command in Engine (2/2, RolesInfo Impl.) 
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
1.108
### Description
We are adding support for the RolesInfo command in Engine

----
#### AI description  (iteration 1)
#### PR Classification
New feature enhancement: introducing the `rolesInfo` command to retrieve detailed role information including inherited roles and privileges.

#### PR Summary
This pull request adds comprehensive support for the `rolesInfo` command in the engine, implementing new parsing, processing, and BSON output functions while updating tests and auxiliary query strings.
- `oss/pg_documentdb/src/commands/roles.c`: Added functions (e.g., `ParseRolesInfoSpec`, `WriteRoleDocumentWithInheritedRoles`, `WriteInheritedRolesToArray`) to parse rolesInfo parameters and construct role documents with inherited roles and optional privileges.
- `oss/pg_documentdb/src/test/regress/expected/role_crud_commands.out` and `pgmongo/src/test/regress/expected/role_crud_commands.out`: Expanded test suites with numerous rolesInfo scenarios, error conditions, and inheritance verification tests.
- `oss/pg_documentdb/include/roles.h`: Introduced new types and function prototypes to support rolesInfo and role CRUD operations.
- `pgmongo/src/pgmongo.c`: Updated query string definitions to handle role name substitutions for OSS compatibility.
- `oss/pg_documentdb/src/commands/users.c`: Modified role checking and privilege extraction utilities to align with the new rolesInfo functionality.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

Related work items: #4552244

Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
 2025-09-18 08:10:29 -07:00
Vinod Sridharan 32b11ab18f Merged PR 1799781: [Reindex] Support internally handling Reindex with background index processor 
### Does this PR have any customer impact?
No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Add infrastructure to handle REINDEX CONCURRENTLY using hte background index queue. This doesn't expose any new public API at this time, and is meant as a first pass to enable administrators to enqueue reindex concurrently in the background.

Progress can be tracked by currentOp

----
#### AI description  (iteration 2)
#### PR Classification
This PR introduces a new feature that adds asynchronous reindex operations using the background index processor.

#### PR Summary
This update extends the background index processor to support reindexing by implementing new C functions and updating index queue management, SQL UDF definitions, and tests. Key changes include:
- `src/commands/create_indexes_background.c`: Added `command_reindex_index_background` and `command_reindex_index_background_internal` functions and refactored common command core logic.
- `src/metadata/index.c`: Modified index queue functions and metadata handling to support the new REINDEX command type.
- `sql/udfs/index_mgmt/create_index_background--0.108-0.sql`: Added new SQL UDFs for reindexing indexes.
- Test files under `src/test/regress` and `/pgmongo/src/test/regress`: Updated to validate the background reindexing functionality.
- Minor updates in logging and distributed hook functions to accommodate reindex operations in the background processor.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

Signed-off-by: Shuai Tian <shuaitian@microsoft.com>
 2025-09-18 08:10:29 -07:00
William Wang 5a37471d7d Merged PR 1784924: Utility File and Query Refactoring for Role/User CRUD APIs 
### Does this PR have any customer impact?

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)

### Are you introducing any new config? If yes, do you have tests with and without them being set?

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

----
#### AI description  (iteration 1)
#### PR Classification
This PR refactors and centralizes role/user CRUD utility functions.

#### PR Summary
The changes migrate and consolidate the logic for user and role management into a new utility module and header file, streamlining the codebase and removing redundant implementations from command files.
- `src/utils/role_utils.c` adds core functions for password hashing, privilege consolidation, and user/role document writing.
- `include/utils/role_utils.h` provides declarations for the newly centralized role/user utilities.
- `src/commands/users.c` and `src/commands/roles.c` are cleaned up by removing duplicated role/user functionality now handled in the new module.
- `pgmongo.c` and related metadata files are updated to use the new role replacement variable and adjust related constants.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

Related work items: #4639026, #4639027
 2025-09-09 11:03:42 +08:00
Jinrui Zhang 3ccd656663 Merged PR 1777067: [Refector] Update to match coding standards - part 0 
### Description
1. Fixing issues identified by similarity checker
2. Fix the corresponding sql unit tests
3. Fix JSTests failure due to above changes
4. Fix corresponding C# test failure

----
#### AI description  (iteration 1)
#### PR Classification
This pull request is a code cleanup that standardizes and improves error message wording across the codebase.

#### PR Summary
The changes update error messages throughout the repository to provide clearer, more consistent, and more detailed feedback when operations fail.
- Updates in geospatial operator files (e.g. in `$geoNear`, `$near`, `$nearSphere`) replace informal phrasing with precise messages.
- Numerous error messages in aggregation and projection test files have been reworded for consistent phrasing
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-08-25 11:39:17 +08:00
Duan Pei 6a129514a9 Merged PR 1777887: [Refector] Update to match coding standards - part 1 
### Does this PR have any customer impact?

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)

### Are you introducing any new config? If yes, do you have tests with and without them being set?

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

update errmsg

----
#### AI description  (iteration 1)
#### PR Classification
This pull request focuses on updating and standardizing error messages across various operators, aggregation stages, and test expectation files.

#### PR Summary
The changes reword error messages throughout the codebase and test outputs to provide clearer, more consistent, and more informative feedback for users. Key updates include:
- **`/oss/internal/pg_documentdb_distributed/src/test/regress/expected/*.out`** – Revised expected error outputs for aggregation, update, and geospatial tests to use clearer phrasing (e.g., for `$merge`, `$pop`, `$indexOfArray`, and `$geoNear`).
- **Core operator files (e.g., `bson_aggregation_output_pipeline.c`, `bson_expression_arithmetic_operators.c`, `bson_expression_conditional_operators.c`)** – Updated error message texts to explicitly state operator requirements and types.
- **Geospatial and vector modules (e.g., files under `bson_geojson_utils.c`, `bson_geospatial_*`, and vector search/index configuration files)** – Standardized validations and error reporting for GeoJSON, CRS, and vector search parameters.
- **Additional modules (e.g., collection view creation and user management files)** – Improved error handling messages to better guide users on input misconfigurations.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-08-25 11:39:17 +08:00
Yang Qiao 7f0c061b83 Merged PR 1778033: [Refactor] Update to match coding standards - part 4 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Testing
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Update to match coding standards

----
 2025-08-25 11:39:17 +08:00
Vinod Sridharan c01cd9bf6c Merged PR 1768176: [Index Policies] Add a few more metadata lines in index policy logging 
### Does this PR have any customer impact?
No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Logging

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
1) Track whether or not the index is TTL
2) Track numPaths different from numKeys (needed for composite)
3) Track whether it's a composite index or not

Also add a check that checks that the PG index is valid as part of updating the stats before marking the index as valid in the documentdb Catalog

----
#### AI description  (iteration 1)
#### PR Classification
Enhancement: The pull request updates index policy logging by adding new metadata fields and refining index property validations.

#### PR Summary
This pull request enriches index policy logging by incorporating new utility functions and metadata details to better capture index characteristics and ensure index integrity.
- **`src/metadata/index.c`**: Added `IndexSpecIsWildcardIndex` and `IndexSpecIsOrderedIndex` functions to determine index types.
- **`src/utils/index_policies_utils.c`**: Enhanced `BuildIndexMetricLog` by logging additional fields such as `numPaths`, `hasTtl`, `hasPfe`, and computing the wildcard and ordered index flags.
- **`include/metadata/index.h`**: Introduced new inline functions for boolean conversion with default behaviors.
- **Commands modules** (`create_indexes.c`, `coll_mod.c`, and `drop_indexes.c`): Updated validations (e.g., checking index liveliness) and adjusted boolean handling to align with the new metadata scheme.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-08-13 15:53:16 +08:00
Vinod Sridharan 90b70c07f2 Merged PR 1760321: [Sort] Support descending sort on an ascending index 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

Today, sorts require the index and sort direction to match exactly. However, RUM indexes can support backward scan as well, so we can push descending sorts to ascending indexes.

Enable this capability. However, this required a few changes to the ordered scan to support moving backward in the tree in RUM
This also required us to use a new order by descending operator since Postgres by default only supports ascending sorts on orderbyop.

----
#### AI description  (iteration 1)
#### PR Classification
New feature enhancement to add descending sort capabilities on ascending indexes.

#### PR Summary
This pull request introduces comprehensive support for descending sort—integrating reverse flags into comparator functions, scan direction adjustments, and new operator metadata—to enable proper pushdown and execution of reverse order queries.
- `oss/pg_documentdb/src/query/bson_dollar_operators.c`: Refactored sort input and comparator functions to incorporate reverse flags (`isReverse`) and adjust value comparisons for descending order.
- `oss/pg_documentdb/src/opclass/bson_gin_composite_entrypoint.c` and related index modules: Added logic for backward scans, including new functions to determine scan direction and reverse comparison adjustments.
- Test SQL files (e.g. `bson_composite_order_by_index_tests.sql`): Extended the test suite with descending order queries and validate explain plans for reverse scan ordering.
- Metadata and operator definitions: Introduced new reverse order operator functions (e.g. `BsonOrderByReverseIndexOperatorId`) and updated index support utilities to correctly process backward scans.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-08-07 11:14:08 +08:00
Vinod Sridharan 623ef26f85 Merged PR 1742378: [Hints] Add support for index hints in DocumentDB 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

Index hints are a mongo feature that allows customers to specify which index they would like to target for a query. Typically this is used in diagnostic scenarios when testing query performance.
With the new composite index op class coming in, we’re introducing side by side support for composite which makes it so we would want customers to be able to test which index is better for planning.

In this spirit, index hints is introduced which follows the same infra as the force pushdown for geonear  and text and adds a new option for index hints.

As part of this the code is also refactored to cleanup some of the logic in the relpathlist and fix correctness issues around text index force pushdown (which was needed to handle index hints)

----
#### AI description  (iteration 1)
#### PR Classification
This pull request introduces a new feature to support index hints in DocumentDB.

#### PR Summary
The changes integrate index hint support into query planning and execution, allowing users to force a specific index during query processing. Key updates include:
- **`src/opclass/index_support.c`**: Added new functions and a force index operator type (`ForceIndexOpType_IndexHint`) for filtering available indexes and generating custom operator expressions based on index hints.
- **`src/planner/documents_planner.c`**: Modified base table query planning logic to detect index hints and adjust index pushdown, including updates to path and restriction handling.
- New UDFs (e.g., `bson_dollar_index_hint`) and configuration variables (`EnableIndexHintSupport`, `UseLegacyForcePushdownBehavior`) were added to support the hint mechanism.
- Test files and SQL scripts (e.g., in `/oss/internal/pg_documentdb_distributed/src/test/regress/`) were introduced or updated to validate index hint pushdown behavior.
- Documentation and release notes (e.g., `docs/pgmongo/design_docs/data_plane/indexing/index_hints.md`, `pgmongo_1.106.md`, and `CHANGELOG.md`) were updated to describe the index hint feature.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-07-23 15:56:08 +08:00
Lucas Borges Fernandes 4800068a67 Merged PR 1741680: [OSS] Improving index categorization api 
### Does this PR have any customer impact?
No.

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring.

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No.

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No.

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
N/A

### Description
The purpose of this change is to add infrastructure to improve categorization and parsing of index specifications.

This PR adds a function named `GetIndexTypeFromKeyDocument` to extract the index type from a pgbson key document. The index types can be any of regular, hashed, 2d, text, 2dsphere and cosmosSearch. There are no schema changes and no new configs.

----
#### AI description  (iteration 1)
#### PR Classification
This pull request improves the index categorization API by refactoring utility functions and consolidating index type definitions.

#### PR Summary
The update adds a common helper to compare index types and a new function to determine an index’s type from a key document, while unifying index type definitions across headers and source files.
- `src/metadata/index.c`: Introduced `IsIndexOfType` and `GetIndexTypeFromKeyDocument` to centralize index type comparisons.
- `include/metadata/index.h`: Added the `MongoIndexKind` enum and `MongoIndexSupport` struct, with declarations for global index type arrays.
- `include/commands/create_indexes.h` and `src/commands/create_indexes.c`: Removed duplicate index type definitions and updated the index support list for consistency.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-07-23 15:56:08 +08:00
William Wang 331d692a63 Merged PR 1686552: Adding Support for All Role Querying in UsersInfo/ConnectionStatus and User Query Optimization 
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No.
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No.
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))
1.106
### Description
This PR introduces the following changes:
- UsersInfo and ConnectionStatus commands to return all supported roles, instead of only ReadOnly and Admin roles; specifically, this PR adds the supported roles of ReadWrite, UserAdmin, and Root roles, along with privilege sets dbAdmin, userAdmin, clusterMonitor, clusterManager, and hostManager.
- Modifies get user logic to use a hash table to record users and their roles, with role filtering.
- Optimizes and beautifies queries within users.c.

Related work items: #4158026, #4315160
 2025-07-23 15:56:08 +08:00
Lucas Borges Fernandes 1cf6873426 Merged PR 1661393: Add OSS background_worker initial jobs framework 
### Does this PR have any customer impact?
No.

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature.

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No.

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes.

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

Add OSS background_worker initial jobs framework. For this initial version, only schedulable jobs are allowed (i.e. jobs that run on timed intervals). The background worker loop has a latch timeout with a predefined duration of 10 seconds that is configured through a GUC. Every time this timeout occurs, it iterates on the JobRegistry array.

1. Checks if the job is running and if it timed out (i.e. running longer than timeoutInSeconds). If yes, cancels the query and finishes libpq connection.
2. Checks if the job finished running (i.e. libpq connection isn't busy). If yes, finishes the libpq connection.
3. Runs the job if its scheduled interval has reached (i.e. currentTime - lastStartTime >= scheduledInterval). Each job execution has its own libpq connection to the database.

In order to add a new job callers must pass a BackgroundWorkerJob structure to RegisterBackgroundWorkerJob.

----
#### AI description  (iteration 1)
#### PR Classification
This pull request introduces a new feature by adding an OSS background worker jobs framework.

#### PR Summary
The changes implement a framework for scheduling, executing, and managing background jobs within the background worker process. This provides structured job processing with timeout, cancellation, and configuration support.

- **`src/background_worker/background_worker.c`**: Added job management functions (e.g., ManageJobs, ExecuteJob, CanExecuteJob, CheckJobCompletion, CancelJobIfTimeIsUp, ValidateJob) and integrated job scheduling in the main loop.
- **`include/background_worker/background_worker_job.h`**: Introduced new structures for defining background worker jobs and their execution objects.
- **`src/api_hook_impl.c`**: Added a hook function to retrieve background worker jobs for registration.
- **`src/configs/background_job_configs.c`**: Defined new configuration variables (GUCs) to enable background worker jobs.
- **API Hooks and Connection Management Files**: Updated declarations and functions (e.g., in `include/api_hooks_def.h` and connection management files) to support connection handling necessary for job execution.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

Related work items: #4225917
 2025-07-09 18:56:11 +08:00
Vinod Sridharan c5b5c71596 Merged PR 1712147: [Docs] Fix up docs and such to match contributing guide 
### Does this PR have any customer impact?
No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Fix up some more documentation and comments around the code.

----
#### AI description  (iteration 1)
#### PR Classification
This PR performs a code and documentation cleanup by refactoring naming conventions and removing deprecated staging functions to align with the contributing guide.

#### PR Summary
The pull request removes obsolete C wrappers for staging collection operations and standardizes internal API naming from "MongoCatalog" to "ApiCatalog" while refining inline comments for clarity and consistency.
- **`src/metadata/collection.c` & `include/metadata/collection.h`**: Removed unused functions for handling staging collections during $out operations.
- **`src/metadata/metadata_cache.c` & `include/metadata/metadata_cache.h`**: Renamed functions and variables from "MongoCatalog" to "ApiCatalog" to standardize internal identifiers.
- **`src/aggregation/bson_aggregation_config_queries.c` & `src/aggregation/bson_aggregation_metadata_queries.c`**: Updated BSON aggregation queries to reference the new API naming.
- **`src/infrastructure/cursor_store.c`**: Enhanced documentation by updating comments and adding a shared memory structure for cursor tracking.
- **`src/metadata/index.c` & `src/index_am/rum.c`**: Applied minor comment corrections and removed outdated TODO notes for clarity.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-07-09 18:56:11 +08:00
Parag Jain 2dc64d2081 Merged PR 1711000: [oss] Cleanup as per new change guideline, part -1 
### Does this PR have any customer impact?

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)

### Are you introducing any new config? If yes, do you have tests with and without them being set?

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

3 c files

----
#### AI description  (iteration 1)
#### PR Classification
This pull request focuses on code cleanup and documentation updates.

#### PR Summary
The changes standardize and simplify documentation comments and examples across several operator implementations, removing obsolete references and unnecessary details.
- `oss/pg_documentdb/src/aggregation/bson_aggregation_output_pipeline.c`: Updated doc examples for `$merge` and `$out` to use generic placeholders and removed temporary test-related comments.
- `oss/pg_documentdb/src/query/bson_dollar_operators.c`: Cleaned up comments for multiple BSON dollar operators by eliminating "Mongo's" references and redundant test notes.
- `oss/pg_documentdb_core/src/types/pcre_regex.c`: Refined regex compile option comments and removed legacy test reference lines.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-07-09 18:56:11 +08:00
Vinod Sridharan a547931808 Merged PR 1706921: [Background Index] Improve create indexes performance 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Performance

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Currently, each round of the sproc that does create indexes does exactly 1 index build and bails. This means if a single client request queues 4 indexes and the job runs every 2 seconds, it'll take 8 seconds to build all 4 indexes. This is wasteful since we know all 4 indexes up front.
Add a loop to the index build process so that multiple indexes can be built without waiting the 2 seconds.
As part of this, there were a few places allocating in memory contexts that are unstable across executions. Fix this.
Additionally there was a loop that could cause index builds to starve for ~20 mins if it picked the wrong index. Fix this loop too.

Finally, Noticed that the version cache does not cache properly if the Major version is 0. Fix this (which can cause perf regressions in DocumentDB).
 2025-06-19 14:10:04 +08:00
Vinod Sridharan c6bc2bcb73 Merged PR 1694458: [POC|Order by] Order by Index & Pushdown improvements 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
More general improvements to order by and composite:
Create Indexes
1) Support creating a composite index side by side with a regular index
2) Block creation of descending or wildcard and unsupported set ups for composite indexes

Planner:
1) Support ordering by the Nth path of an index if the prefixes are equality.
2) Validate that if it's not the first column, prefixes are provided
3) Support incremental sort with index pushdown.

----
#### AI description  (iteration 1)
#### PR Classification
This pull request delivers a new feature and API enhancement to enable order-by index pushdown and refined composite index support.

#### PR Summary
The update streamlines composite index processing by introducing stricter equivalency checks, new UDF functions for full-scan order-by pushdown, and enhanced validations during index creation while updating explain outputs and tests.
- `src/metadata/index.c` refines index options equivalency logic to correctly compare composite term settings.
- `src/opclass/bson_gin_composite_entrypoint.c` and `src/index_am/rum.c` add functions to support order-by pushdown and manage duplicate entries efficiently.
- `sql/udfs/query/bson_dollar_comparison--0.105-0.sql` (and related operator files) introduce new UDFs including `bson_dollar_fullscan` to enable full scan operations.
- `src/commands/create_indexes.c` enforces new composite index validations (restricting use to ascending, non-wildcard regular indexes) with precise error reporting.
- Test scripts and explain plan outputs in the `regress` and `pgmongo` directories are updated to confirm the new composite order-by behavior.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-06-13 15:13:47 +08:00
Shuai Tian 17d4306a28 Merged PR 1422337: [pgmongo] Aggregate stage $bucketAuto implementation 
### Does this PR have any customer impact?

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)

### Are you introducing any new config? If yes, do you have tests with and without them being set?

### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description

This pull request adds documentation for the `$bucketAuto` aggregation stage in PostgreSQL, detailing its design, usage, and implementation.
- Adds `bucketAuto.md` with details on the `$bucketAuto` aggregation stage
- code implementation
- support all granularity types: POWERSOF2, 1-2-5, Renard numbers, E series

Related work items: #3234185
 2025-06-05 10:31:32 +08:00
Cornelius Boateng b204b58b82 Merged PR 1619783: [engine][collation] Support for sort in runtime 
### Does this PR have any customer impact?  No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes, yes

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Support sorting with collation.
We define new custom sort operators which compare operands using applicable collation string.

`<<<` is for ascending order.
`>>>` is for descending order.

Example:
`SELECT * from <collection> ORDER BY <value> USING <<<`

This will further be used for partitioning, grouping, and related comparisons.

----
#### AI description  (iteration 1)
#### PR Classification
New feature: Support for collation in sort operations.

#### PR Summary
This pull request adds support for collation in sort operations within the database engine.
- `src/test/regress/expected/bson_dollar_ops_collation_tests_runtime.out`: Added multiple test cases to verify collation behavior with various sort and order by operations, including case sensitivity and numeric ordering.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

Related work items: #4014123
 2025-05-29 12:12:31 +08:00
Amit Prabhudesai 84ba0c99d8 Merged PR 1572685: [OSS] Fix background index builds for documentdb 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Get background index builds to work for single-node documentdb.

This changeset fixes background index builds for OSS by:

1. Scheduling the background index build tasks
2. Moving the distributed functionality (which had not being refactored out of the indexing code during the move to OSS) into distributed hooks, with the base implementation for single-node.
3. Refactoring SQL tests to get even with the above changes.

----
#### AI description  (iteration 1)
#### PR Classification
Bug fix for background index builds in DocumentDB.

#### PR Summary
This pull request addresses issues with background index builds in DocumentDB by adding comprehensive tests and handling various edge cases.
- `commands_create_indexes_background.out`: Added extensive test cases to validate the behavior of `create_indexes_background` function, including handling of invalid inputs, unique constraints, and various index specifications.
- Introduced a new schema and function to manage index job statuses.
- Enhanced error handling and validation for index creation requests, ensuring proper error messages and statuses are returned for invalid operations.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-05-09 14:08:53 +08:00
Zinan Ren ac6f028c86 Merged PR 1653287: [Vector Search] Enable iterative index scans for HNSW and IVF in filtering searches 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes

### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description
With the release of [pgvector 0.8.0](https://github.com/pgvector/pgvector/tree/5bc7937715c67add21c8bcc0d4284162c7a0174f?tab=readme-ov-file#iterative-index-scans), which supports iterative scan for HNSW and IVF, we enable it in filtering search to support pre-filtering(currently it is post-filtering)
- Pass the new parameter "iterativeScan" through the function bson_search_param from the coordinator to the worker.
Example : { "efSearch" : 10, "iterativeScan" : "relaxed_order"}
- Enable iterative scan for hnsw/ivf index on workers:
SET LOCAL  hnsw.iterative_scan = relaxed_order;
SET LOCAL  ivfflat.iterative_scan = relaxed_order;
relaxed_order by default, which allows results to be slightly out of order by distance, but provides better recall.
After iterative search, re-ordering will be applied to the results.
- Set iterative scan for diskann index to relaxed_order
SET LOCAL  diskann.iterative_search = relaxed_order;
- Check the availability of iterative scan in pgvector
The pgvector iterative scan is available from version 0.8.0. Currently, if the pgvector extension has not been upgraded to 0.8.0, the iterative scan will be disabled in those clusters until pgvector is upgraded to 0.8.0.

###Related GUCs
**documentdb.vectorPreFilterIterativeScanMode**: enable iterative index scans, which will automatically scan more of the index until enough results are found (relaxed_order by default)
- off, disable iterative scan
- relaxed_order,  allows results to be slightly out of order by distance, but provides better recall
- strict_order, ensures results are in the exact order by distance

####GUCs from pgvector
**hnsw.max_scan_tuples**: Specify the max number of tuples to visit (20,000 by default)
**hnsw.scan_mem_multiplier**: Specify the max amount of memory to use, as a multiple of work_mem (1 by default)
**ivfflat.max_probes**: Specify the max number of probes(32768 by default)

###Mongo Command(no change in this PR)
```
db.testing_hnsw_vector_collection.aggregate(
[
  {
    "$search": {
	  "cosmosSearch": {
        "vector": [ 3.0, 4.9, 1.0 ],
        "path": "vect",
        "filter": {"meta.a": [ { "b" : 3 } ]},
        "k": 4,
        "efSearch": 100
      },
    }
  },
  {
    "$project": {
      "rank": {
        "$floor": [{"$multiply": [{ "$meta": "searchScore"},100000]}]
      }
    }
  }
])
```

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request introduces iterative index scans for HNSW and IVF in filtering searches, enhancing the pre-filtering capabilities of vector searches...
 2025-05-09 14:08:53 +08:00
Zinan Ren e05e91ba51 Merged PR 1650017: [Vector Search] Add support for PQ compression in the Diskann index 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Adding support Product Quantization (PQ) compression in vector index, which is a popular method for dramatically compressing high-dimensional vectors to use less memory, and for making nearest-neighbor search faster.
Note: currently, this PQ compression is only supported by the Diskann index.

### Related GUCs
**documentdb.enableVectorCompressionPQ**: Enables vector PQ compression (enabled by default).

### Create index spec:
- **compression**(optional):
  - "half": half-precision vector, which using 16 bits floats.
  - "pq": product quantization compression.

- **pqCompressedDims**(optional): indicates the dimensions after compression. If not specified, it will be automatically calculated based on the original dimensions.
  - Minimum: 1
  - Maximum: 8000

- **pqSampleSize**(optional):Indicates the number of samples PQ uses to generate the initial centroids. The default value is 1000.
  - Minimum: 1000
  - Maximum: 100000

**Note**: It is recommended to create the index after the collection has some data.. If the collection is empty, random vectors within the range [-1, 1] are used to generate the initial centroids and create the index. If the number of documents is less than pqSampleSize, the training data is padded with random data within the range [min, max] from the sampled vector data.

### Command example
```
db.runCommand(
{ "createIndexes": "testing_vector_collection", "indexes":
[{
     "key": { "v": "cosmosSearch" },
     "name": "hsnw_index",
     "cosmosSearchOptions":
     {
         "kind": "vector-diskann",
         "similarity": "L2",
         "dimensions": 1536,
         "compression": "pq",
         "pqCompressedDims": 96,
         "pqSampleSize": 2000
     }
}]
} )
```
### Search spec:
No changes

----
#### AI description  (iteration 1)
#### PR Classification
New feature: This pull request adds support for Product Quantization (PQ) compression in the Diskann index.

#### PR Summary
This pull request introduces support for PQ compression in the Diskann index, enhancing vector search capabilities. It includes error handling for unsupported compression types and validation for PQ parameters.
- Updated `/pgmongo/src/test/regress/expected/bson_aggregation_pipeline_tests_vector_diskann.out` to include tests for PQ compression, error cases, and validation of PQ parameters such as `pqCompressedDims` and `pqSampleSize`.
- Added tests for vector search with PQ compression, including L2, cosine, and IP similarity measures.
- Implemented checks for maximum dimensions and resource limits during PQ index creation.
<!-- GitOpsUserAgent=GitOps.Apps.S...
 2025-04-30 14:57:09 +08:00
Cornelius Boateng bec1aa3562 Merged PR 1638286: [engine][collation][enhancement] Extend bson_expression_get for collation support 
### Does this PR have any customer impact? No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set? No

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Add a new bson_expression_get that appends the collationString to the got value, if a valid collationString is passed to it.
This will be used as a helper function to operations that rely on bson_expression_get to pass down collation such as bson_lookup_join_filter.

If no collation string is passed as an arg, we simply return the got value.
----
#### AI description  (iteration 1)
#### PR Classification
New feature to support `$graphLookup` in the engine with collation.

#### PR Summary
This pull request adds support for `$graphLookup` with collation in the engine.
- Added new functions in `oss/pg_documentdb/sql/udfs/aggregation/bson_lookup_functions--0.103-0.sql` for `$graphLookup` operations.
- Modified `oss/pg_documentdb/src/aggregation/bson_aggregation_nested_pipeline.c` to handle collation in `$graphLookup` join filters.
- Updated `oss/pg_documentdb/src/operators/bson_expression.c` to remove collation handling from `bson_expression_get`.
- Added `BsonDollarLookupJoinFilterWithCollationFunctionOid` in `oss/pg_documentdb/src/metadata/metadata_cache.c`.
- Adjusted collation handling in `oss/pg_documentdb/src/query/bson_dollar_operators.c` for `$graphLookup`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

Related work items: #4104899
 2025-04-30 14:57:09 +08:00
Zinan Ren 591a38b86d Merged PR 1606660: [Vector Search] Support half-precision vector indexing 
### Does this PR have any customer impact?
No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
With the release of pgvector 0.7.0, which supports [Half-Precision Indexing](https://github.com/pgvector/pgvector?tab=readme-ov-file#half-precision-indexing), vectors can have up to 4,000 dimensions

###Related GUCs
**documentdb.enableVectorCompressionHalf**: enable vector compression(enabled by default)
- off, disable vector compression
- on, enable vector compression

###New parameters
- Create index spec: "compression": string
"half": half-precision vector, which using 16 bits float

- Search spec: "oversampling": double
Oversampling factor: Oversampling retrieval from half index to compensate for the loss of precision.
Minimum value is 1. This parameter is only permitted when a compression method is used on the vector index.
 For example, if oversampling=1.5 and k=10, then oversampling*k=15 vectors from a compressed index. Then, the full 32-bits vector will be used for the subsequent ordering and scoring,

###Mongo Command
```
db.runCommand(
{ "createIndexes": "testing_vector_collection", "indexes":
[{
     "key": { "v": "cosmosSearch" },
     "name": "hsnw_index",
     "cosmosSearchOptions":
     {
         "kind": "vector-hnsw",
         "similarity": "L2",
         "dimensions": 3,
         "compression": "half"
     }
}]
} )
```
```
db.testing_hnsw_vector_collection.aggregate(
[
  {
    "$search": {
      "cosmosSearch": {
        "vector": [ 3.0, 4.9, 1.0 ],
        "path": "vect",
        "filter": {"meta.a": [ { "b" : 3 } ]},
        "k": 4,
        "efSearch": 100,
        "oversampling": 2.0
      },
    }
  },
  {
    "$project": {
      "rank": {
        "$floor": [{"$multiply": [{ "$meta": "searchScore"},100000]}]
      }
    }
  }
])
```

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request adds support for half-precision vector indexing in the vector search functionality.
- `metadata_cache.c`: Added functions for half-precision vector similarity operators and functions.
- `SearchTests.cs`: Added integration tests to validate half-precision vector search, including error cases and index creation scenarios.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-23 12:04:26 +08:00
Vinod Sridharan a72e6563d6 Merged PR 1634773: [Orderby POC] Add plumbing in planner for orderby pushdown 
### Does this PR have any customer impact?
No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
POC Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Not yet

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

Add plumbing in planner for orderby pushdown to the index. This is guarded behind a GUC and doesn't work yet - but at least introduce this for later.

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request adds support for orderby pushdown in the planner.
- `index_support.c`: Introduced `ConsiderIndexOrderByPushdown` function to handle orderby pushdown logic.
- `rum.c`: Added support for orderby pushdown in RUM index scan and validation.
- `metadata_cache.c`: Added `BsonOrderByIndexOperatorId` function.
- `testing_configs.c`: Defined new configuration variable `EnableIndexOrderbyPushdown`.
- New test files `bson_composite_order_by_index_tests.sql` and `bson_composite_order_by_index_tests.out` to validate the feature.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-10 10:29:08 -06:00
Andrew Khoma 6e970adba8 Merged PR 1600265: [Gateway][SinglePool] Implement index DDL procedures 
### Does this PR have any customer impact?
yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
yes, new UDF introduced
### Are you introducing any new config? If yes, do you have tests with and without them being set?
no, goes under the ddl flag
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

- implement the last 2 procedures that are related to indexes
- improve error handling when updating the metadata column for ro user
- test coverage through the isolation + regular sql tests

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request implements index DDL procedures in the C# gateway for a global PostgreSQL connection pool.
- `pgmongo/src/metadata/oid_cache.c`: Added functions to handle OID caching for new index operations and helper methods for type parsing.
- `pgmongo/src/infrastructure/command_funcs.c`: Introduced `ExecuteIndexOpCore` and `command_index_op_as_user` functions to execute index operations.
- `oss/pg_documentdb/src/metadata/index.c`: Enhanced `MarkIndexRequestStatus` to handle failures using subtransactions.
- `pgmongo/src/test/regress/expected/isolation-currentop-singlepool.out` and `pgmongo/src/test/regress/expected/user_dispatch_auth_tests.out`: Updated test cases to validate new index operations and user dispatch authentication.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

Related work items: #3920602
 2025-04-10 10:29:08 -06:00
Vinod Sridharan b31d2b2751 Merged PR 1628723: [Composite POC] Planner integration and basic plumbing 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
- This is the second part of the plumbing for composite indexes.
- Add the planner and query integration to push $eq filters in the composite index

Still not done:
- Correctness of query
- inequality and other operators (`$in/$regex/etc`)
- Type bracketing
- Truncation

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request integrates a planner and adds basic plumbing for composite index operations.
- `bson_gin_composite_entrypoint.c`: Added functions for handling composite index paths, parsing bounds, and modifying scan keys.
- `bson_composite_index_tests_insert_terms.out`: Added tests to validate composite index functionality and queries.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-07 11:39:49 +08:00
Cornelius Boateng d4f127106a Merged PR 1574869: [engine][compat] BinData Conversion to/from String 
### Does this PR have any customer impact? No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) No

### Are you introducing any new config? If yes, do you have tests with and without them being set? No

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Support conversion to binData from string and vice versa.
$convert to BinData
```
"$convert": {
   "input": <string>,
   "to": {
      "type": "binData",
      "subtype": <bson_subtype_t>
   },
   "format": <"base64"/"string"/"hex"/"uuid"/"base64url">
}
```

OR
```
--- subtype will default to 0
"$convert": {
   "input": <string>,
   "to": "binData",
   "format": <"base64"/"string"/"hex"/"uuid"/"base64url">
}

```

BinData $toString
```
{ $toString: <binData expression> }
```

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request introduces support for converting data to `BinData` using the `$convert` operator.
- `src/operators/bson_expression_type_operators.c`: Added new functions and modified existing ones to handle `BinData` conversion, including `ProcessDollarToBinData`, `GetToTypeAndSubTypeForBinDataConversion`, and `ValidateAndGetBinDataConversionSubTypeAndFormat`.
- Refactored `ParseDollarConvert` and `ApplyDollarConvert` to accommodate the new `BinData` conversion logic.
- Introduced a new struct `ConvertToTypeArguments` to manage conversion arguments.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-07 11:39:49 +08:00
Vinod Sridharan 86165f7774 Merged PR 1622478: [Cursors] Enable support for primary key based cursor scan 
### Does this PR have any customer impact?
No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Enable support for running streaming cursors based on the primary key index. Do this optionally behind a feature flag.

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request enables support for primary key-based cursor scans in the system.
- Added new test cases in `bson_aggregation_cursor_tests_txn.sql` and `bson_aggregation_cursor_tests_txn.out` to validate primary key cursor scan functionality.
- Updated `custom_scan_continuation.c` to handle primary key cursor scans and build primary key index clauses.
- Modified `metadata_cache.c` to include functions for retrieving OIDs for integer and BSON B-tree operation families.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-04-07 11:39:49 +08:00
Vinod Sridharan f31e0b6cad Merged PR 1615832: Support for pushdown of in to PFE indexes 
### Does this PR have any customer impact?

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)

### Are you introducing any new config? If yes, do you have tests with and without them being set?

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

Support for pushdown of  to PFE indexes

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request adds support for pushdown operations to PFE indexes, enhancing query performance and index utilization.
- Added new functions in `sql/udfs/index_mgmt/bson_indexbounds_functions--0.102-0.sql` for BSON index bounds operations.
- Introduced new operators in `sql/operators/bson_btree_pfe_operators--0.102-0.sql` for BSON and BSON query comparisons.
- Updated `src/query/query_operator.c` to handle new index bounds operators and expressions.
- Added new type definitions in `sql/types/bsonindexbounds--0.102-0.sql` for BSON index bounds.
- Modified `src/metadata/metadata_cache.c` to include OIDs and functions for BSON index bounds types and operators.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-03-26 12:19:42 +08:00
Xing Fan 399e01d954 Merged PR 1539417: [Operator] $toUUID in Mongo 8.0 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description
support $toUUID operator

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request introduces the `$toUUID` operator for MongoDB 8.0, enabling conversion of strings to UUIDs.
- Added `ProcessDollarToUUID` function in `bson_expression_type_operators.c` to handle the conversion logic.
- Updated `bson_expression_operators.h` to declare new parsing and handling functions for `$toUUID`.
- Included new test cases in `bson_aggregation_type_operators_tests.sql` and `bson_aggregation_type_operators_tests.out` to validate the `$toUUID` functionality.
- Modified `feature_counter.h` and `feature_counter.c` to include the new `$toUUID` feature in the feature count and mapping.
 2025-03-26 12:19:42 +08:00
Cornelius Boateng 98a5603d64 Merged PR 1594069: [engine][collation] Support for find 
### Does this PR have any customer impact? No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set? No

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Support collation with `find` queries.

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request adds support for the `project find` functionality with collation in the engine.
- Added new SQL functions in `sql/udfs/projection/bson_projection--0.102-0.sql` for handling BSON projection with collation.
- Updated `metadata_cache.c` to include OID caching for the new `bson_dollar_project_find` function with collation.
- Modified `bson_project.c` to handle additional arguments for collation in the `bson_dollar_project_find` function.
- Updated `bson_projection--latest.sql` to include the new function definition.
- Added function declaration in `metadata_cache.h` for the new `bson_dollar_project_find` function with collation.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

Related work items: #3924476
 2025-03-18 21:57:32 -07:00
Cornelius Boateng 70887c4a1a Merged PR 1601128: [engine][collation] Support for replaceRoot 
### Does this PR have any customer impact? No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set? No

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Support collation for `replaceRoot` aggregation stage

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request adds support for the `$replaceRoot` operation with collation in the BSON aggregation pipeline.
- `bson_projection--0.102-0.sql` and `bson_projection--latest.sql`: Added new function `bson_dollar_replace_root` with collation support.
- `metadata_cache.c` and `metadata_cache.h`: Added OID cache and retrieval function for the new `bson_dollar_replace_root` with collation.
- `bson_project.c`: Updated `bson_dollar_replace_root` to handle collation.
- `bson_aggregation_pipeline.c`: Integrated collation handling in `HandleReplaceRoot`.
- Updated test files to include new test cases for `$replaceRoot` with collation.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-03-18 21:57:32 -07:00
Vinod Sridharan 7d31ca16ff Merged PR 1605959: [Index Changes] A few fixes for background indexing for blocking & pruning old indexes 
### Does this PR have any customer impact?
Yes

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfixes

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
1) Support blocking background index which will do a non-concurrent index create
2) Delete dead code in the gW for background blocking index
3) Show queued index builds in currentOp
4) Prioritize pruning old skippable indexes in the queue.

----
#### AI description  (iteration 1)
#### PR Classification
Bug fix for background indexing.

#### PR Summary
This pull request addresses several issues related to background indexing, including handling skippable index requests and improving logging.
- `index.c`: Added function `GetSkippableRequestFromIndexQueue` to fetch skippable index requests.
- `create_indexes_background.c`: Implemented `PruneSkippableIndexes` to remove stale index requests and updated `command_build_index_concurrently` to prioritize pruning.
- `PostgresMongoClient.cs`: Removed temporary fix for blocking createIndexes requests.
- `current_op.c`: Enhanced `AddFailedIndexBuilds` to include queued index builds.
- Updated test files to reflect changes in index queue handling and logging.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-03-18 21:57:32 -07:00
Cornelius Boateng d1b099258b Merged PR 1591622: [engine][collation] Support for addFields & set 
### Does this PR have any customer impact? No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set? No

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Support collation for `$addFields` and `$set` aggregation stages

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request adds support for the `addFields` operation with collation in the aggregation pipeline.
- `bson_project.c`: Added collation support to `bson_dollar_add_fields` and related functions.
- `bson_expression_comparison_operators.c`: Updated comparison operators to handle collation.
- `bson_dollar_operators.c`: Enhanced query state population to include collation.
- Test files `bson_dollar_ops_collation_tests_runtime.out` and `bson_dollar_ops_collation_tests_runtime.sql`: Added tests for collation with `$expr` and `$addFields`.
- `bson_projection--latest.sql`: Added new function definition for `bson_dollar_add_fields` with collation.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

Related work items: #3919349, #3919367
 2025-03-18 21:57:32 -07:00
Cornelius Boateng 90c92c44ec Merged PR 1591616: [engine][collation] Support for project 
### Does this PR have any customer impact?

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)

### Are you introducing any new config? If yes, do you have tests with and without them being set?

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

more

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request adds support for collation in BSON projection functions.
- Added new SQL functions in `sql/udfs/projection/bson_projection--0.102-0.sql` to handle collation.
- Updated `src/aggregation/bson_project.c` to include collation string handling in BSON projection.
- Modified `src/aggregation/bson_aggregation_pipeline.c` to support collation in projection stages.
- Enhanced `src/query/bson_dollar_operators.c` to populate collation string in query state.
- Added collation-related tests in `sql/bson_dollar_ops_collation_tests_runtime.sql`.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

Related work items: #3918941
 2025-03-18 21:57:32 -07:00
Cornelius Boateng 8bf9489e33 Merged PR 1592819: [engine][collation] Support for redact 
### Does this PR have any customer impact?

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)

### Are you introducing any new config? If yes, do you have tests with and without them being set?

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

more

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request adds support for collation in the redact function.
- `bson_project.c`: Added handling for an additional collation string argument in the `bson_dollar_redact` function.
- `bson_aggregation_redact--0.24-0.sql` and `bson_aggregation_redact--latest.sql`: Updated to include the new collation string parameter in the `bson_dollar_redact` function definition.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

Related work items: #3919400
 2025-03-18 21:57:32 -07:00
Cornelius Boateng 6e39f125d0 Merged PR 1591206: [engine][collation] Collation Support for Query Match Unsharded 
### Does this PR have any customer impact?

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)

### Are you introducing any new config? If yes, do you have tests with and without them being set?

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description
Change signature of `bson_query_match` with `let` and `collation` support to match other internal functions.

Fix schema name for pgmongo.

----
#### AI description  (iteration 1)
#### PR Classification
New feature: Collation support for query match in unsharded collections.

#### PR Summary
This pull request adds collation support for query match operations in unsharded collections, ensuring proper collation enforcement.
- Modified `bson_query_match` function calls in `/pgmongo/src/test/docdb_compat/expected/bson_dollar_ops_collation_tests_runtime.out` to adjust collation parameters.
- Updated collation parameters in `bson_query_match` function calls in `/oss/internal/pg_documentdb_distributed/src/test/regress/expected/bson_dollar_ops_collation_tests_runtime.out`.
- Adjusted SQL test cases in `/oss/internal/pg_documentdb_distributed/src/test/regress/sql/bson_dollar_ops_collation_tests_runtime.sql` to reflect new collation support.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->

Related work items: #3914004
 2025-03-18 21:57:32 -07:00
Cornelius Boateng e5ede8df79 Merged PR 1586629: [engine][collation] Collation Support for Query Match Function / Operator 
### Does this PR have any customer impact? No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc) Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes) Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set? Yes; yes

### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

Support `collation` with query operator using an internal UDF:
```
bson_query_match(document bson, query bson, collationString text, variableSpec bson)
```

Enable feature with `EnableCollationAndLetForQueryMatch` GUC.

Collation on sharded collections with collation-aware shard keys not supported yet.

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request adds collation support for query operators in the engine.
- `query_operator.c`: Added logic to handle collation strings in BSON query operators and expanded function to support collation and variable specifications.
- `metadata_cache.c`: Introduced a new function to get the OID of `bson_query_match` with collation and let arguments.
- `feature_flag_configs.c`: Added a new feature flag to enable collation and let for write commands.
- `bson_query_match--0.102.sql` and `bson_query_match--latest.sql`: Added new SQL functions to support collation in BSON queries.
- `documents_planner.c`: Updated planner to recognize and handle the new collation and let functionality in queries.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-02-25 22:09:25 -08:00
Santiago Fernandez Madero 322fba0605 Merged PR 1566855: [OSS] Fix TTL indexes in documentdb 
### Does this PR have any customer impact?

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)

### Are you introducing any new config? If yes, do you have tests with and without them being set?

### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description

[OSS] Fix TTL indexes in documentdb

This moves the function to C and instead of querying the indexes and then getting the collection shards from pg_dist_shard table, it uses the C hook to get the table shards where for non-distributed it returns the base table and then uses that information to drop the expired records.

----
#### AI description  (iteration 1)
#### PR Classification
Bug fix

#### PR Summary
This pull request addresses issues with TTL indexes in DocumentDB by refactoring and optimizing the deletion of expired rows.
- `ttl_index.c`: Refactored the function to delete expired rows, added a new function `delete_expired_rows`, and optimized the deletion process.
- `ttl_support_functions--latest.sql` and `delete_expired_rows--latest.sql`: Replaced PL/pgSQL procedures with C language implementations for better performance.
- `collection.c`: Moved the function `GetMongoCollectionShardOidsAndNames` from `coll_stats.c` to `collection.c` for better organization.
 2025-02-19 17:00:35 +08:00
Zinan Ren 1d3b47c814 Merged PR 1573602: [Vector] Add "exact" search parameter to enable enn search 
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature, Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### ChangeLog (Refer [Template](../oss/CHANGELOG.md))

### Description

### Introduce new field "**exact**" in vector search spec to specify whether to run ENN or ANN search.
[exact field spec](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-stage/#fields)

### Value can be one of the following, defaults to false.
- false - to run ANN search
- true - to run ENN search

### Example
`{ "k": 1, "path": "vect", "filter": {"attr": {"$eq":1}}, "exact": true, "efSearch": 40 }`
----
#### AI description  (iteration 1)
#### PR Classification
Code enhancement for filter selectivity in vector search queries.

#### PR Summary
This pull request enhances the filter selectivity mechanism in vector search queries by adding new functions and improving existing ones.
- `bson_aggregation_vector_search.c`: Added functions to calculate filter selectivity and handle vector search options, including `AddEmptySearchParamFunctionToQuery` and `GetFilterSelectivity`.
- `index_support.c`: Added logic to identify vector search queries in the planner.
- `system_configs.c`: Introduced new configuration variables for maximum filter selectivity and document count.
- `documents_planner.c`: Enhanced planner to handle vector search queries based on selectivity.
- `vector_common.h`: Declared new configuration variables for vector filter runtime sort selectivity and document count.
<!-- GitOpsUserAgent=GitOps.Apps.Server.pullrequestcopilot -->
 2025-02-19 17:00:35 +08:00
Rajesh Kumar Thandapani 762d962035 Merged PR 1539952: allChangesForCluster option for change streams 
### Does this PR have any customer impact?
Yes. This PR provides the option of "allChangesForCluster" to be able to track all the changes happening in the cluster with a change stream cursor.

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No

### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description

first version of allChangesForCluster implementation

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
Implemented the first version of the `allChangesForCluster` feature in the change stream.
- `change_stream_wal_utils.c`: Added logic to track changes for the entire cluster and refactored relation ID retrieval.
- `change_stream_aggregation_stage.c`: Updated context initialization to handle `allChangesForCluster` and added error handling for missing collections.
- `change_stream_decoding_utils.c`: Enhanced event building to support cluster-wide change tracking and added a function to get collection by relation ID.
- `change_stream.h`: Extended `ChangeStreamEvent` and `ChangeStreamSpec` structures to include cluster-wide tracking.
- Updated tests in `pgmongo_change_stream_crud_tests.sql` and `pgmongo_change_stream_crud_tests.out` to reflect the new feature.
 2025-02-10 14:08:55 -08:00
Zinan Ren dcf3e2f721 Merged PR 1548926: [OSS] Migrate tests to documentdb and clean helio references from sources 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description

Migrate tests to documentdb and clean helio references from source

----
#### AI description  (iteration 1)
#### PR Classification
Code cleanup and migration of tests.

#### PR Summary
This pull request migrates tests from `helio` to `documentdb` and removes all references to `helio` from the source code.
- `oss/pg_documentdb/src/test/regress/expected/commands_update.out`: Replaced all `helio` references with `documentdb` and updated test cases accordingly.
 2025-01-15 07:48:41 +00:00
Xing Fan 722ef60784 Merged PR 1479858: $percentile & $median for $group accumulator 
### Does this PR have any customer impact?

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)

### Are you introducing any new config? If yes, do you have tests with and without them being set?

### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description

draft

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request introduces new aggregation functions and support for t-digest in the project.
- Added `bson_tdigest.c` to implement t-digest helper functions.
- Created new SQL aggregate functions in `group_aggregates--1.23-0.sql` and `group_aggregates_support--1.23-0.sql`.
- Updated `bson_aggregation_pipeline.c` to handle the new `$percentile` accumulator.
- Modified `metadata_cache.c` and `metadata_cache.h` to include OID for the new `BsonPercentileAggregateFunction`.
- Updated migration scripts `pg_helio_api--1.22-0--1.23-0.sql` and `pgmongo--1.22-0--1.23-0.sql` to include new SQL files.
 2025-01-15 07:48:41 +00:00
Shuai Tian e3dc6c889e Merged PR 1404676: stage implementation: redact 
### Does this PR have any customer impact?
yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes, UDF
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description
Implement for stage redact

Related work items: #1798586
 2025-01-15 07:48:41 +00:00
Deepak Bisht 7a674be896 Merged PR 1542122: [Part 1] Improve $lookup and $unwind aggregation stage combination when the lookup options are simple 
### Does this PR have any customer impact?
No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No

### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
Aggregation performance pipeline optimization for $lookup and $unwind stage combination on same fields.

### Description
1. Optimize $lookup and $unwind stage combination on same field and lookup  options are simple `localField` and `foreignField`
2. Delete dead code for $lookup

----
#### AI description  (iteration 1)
#### PR Classification
Code cleanup and optimization.

#### PR Summary
This pull request improves the combination of `$lookup` and `$unwind` aggregation stages by removing redundant and complex code, simplifying the logic, and optimizing performance.
- Removed the `ProcessLookupCore` function and its associated logic from `bson_aggregation_nested_pipeline.c`.
- Removed the `UnwindWalkerContext` struct and related functions for optimizing `$unwind` with `ARRAY_AGG` from `bson_aggregation_nested_pipeline.c`.
 2025-01-15 07:48:41 +00:00
Jia Hu 55f6fa3840 Merged PR 1485956: [schema validation] support schema rule included top level operator 
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description
To enable top level query operators working in schema validation.

----
#### AI description  (iteration 1)
#### PR Classification
New feature

#### PR Summary
This pull request introduces support for the `$jsonSchema` query operator for schema validation.
- Added handling for `QUERY_OPERATOR_JSONSCHEMA` in `query_operator.c` and `mongo_query_operator.c`.
- Implemented `GetExpressionEvalStateForBsonInput` in `bson_expr_eval.c`.
- Added `BsonJsonSchemaFunctionId` and related OID cache in `metadata_cache.c`.
- Modified `PrepareForSchemaValidation` in `schema_validation.c` to use the new evaluation state function.
 2025-01-15 07:48:41 +00:00
Zinan Ren 47ec4231ba Merged PR 1543189: [OSS] Porting type/operator/schema into documentdb - part 1 
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description

Start porting schemas for documentdb api.
Introducing API_SCHEMA_NAME_V2, API_SCHEMA_INTERNAL_NAME_V2, API_CATALOG_SCHEMA_NAME_V2

----
#### AI description  (iteration 1)
#### PR Classification
Porting type/operator/schema into documentdb.

#### PR Summary
This pull request ports various types, operators, and schemas into documentdb as part of an ongoing effort.
- `pg_documentdb/sql/documentdb--0.24-0.sql`: Created multiple schemas and included various SQL files for query functions, operators, and metadata.
- `pg_documentdb/sql/operators/bson_dollar_operators--0.16-0.sql`: Updated operator definitions to use `__API_SCHEMA_INTERNAL_V2__`.
- `pgmongo/sql/operators/bson_dollar_negation_operators--1.15-1.sql` and `pg_documentdb/sql/operators/bson_dollar_negation_operators--0.16-0.sql`: Changed operator definitions to use `__API_SCHEMA_INTERNAL_V2__`.
- `Makefile.cflags`: Added checks for new schema variables `API_SCHEMA_NAME_V2`, `API_SCHEMA_INTERNAL_NAME_V2`, and `API_CATALOG_SCHEMA_NAME_V2`.
- `pg_documentdb/sql/schema/unique_shard_path_operator_class--0.24-0.sql`: Updated operator class definitions to use `__API_SCHEMA_INTERNAL_V2__`.
 2025-01-15 07:48:40 +00:00
Vinod Sridharan f0722edc50 Merged PR 1542411: [OSS] Start porting Schemas into documentdb_core 
### Does this PR have any customer impact?
No

### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor

### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes

### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))

### Description
Start porting schemas for SQL in documentdb_core.
Maintain back-compat by introducing CORE_SCHEMA_v2 to handle cases where the existing logic already depends on there being 2 core schemas

----
#### AI description  (iteration 1)
#### PR Classification
Code refactoring to port schemas into `documentdb_core`.

#### PR Summary
This pull request refactors the code to port BSON type definitions and related functions into `documentdb_core`.
- `pg_helio_core/sql/types/bson--1.10-0.sql`: Replaced BSON type definitions with an include directive.
- Added `pg_documentdb_core/sql/types/bson--0.24-0.sql` and `pg_documentdb_core/sql/types/bson_typanalyze--0.24-0.sql` for BSON type and analysis functions.
- `pg_documentdb_core/sql/documentdb_core--0.24-0.sql`: Created `documentdb_core` schema and included BSON type definitions.
- Updated Makefiles (`Makefile.cflags`, `pg_helio_api/Makefile`, `pg_helio_core/Makefile`, `internal/pg_helio_distributed/Makefile`) to include new schema definitions.
- Updated `postgresql.conf` files to preload `pg_documentdb_core`.
 2025-01-15 07:48:40 +00:00