### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Our Collstats response is not line with the native mongo response.
Native mongo behaviour :
1) When we run collstats the response size and all fields are expected to be Int32
2) If they overflow the range for int32 in response the output is of type double
With this pr we change the PgbsonWriter to reflect the same functionality while sending final response
----
#### AI description (iteration 1)
#### PR Classification
Bug fix for failing `coll_stats` JavaScript test.
#### PR Summary
This pull request addresses a bug in the `coll_stats` function by ensuring the correct data type is used for various statistics. It modifies the test output and the core implementation to handle integer and double values appropriately.
- `/pgmongo/src/test/regress/expected/commands_coll_stats.out`: Updated expected output to reflect correct data types (`$numberInt` and `$numberLong`).
- `/oss/pg_helio_api/src/commands/coll_stats.c`: Introduced `WriteStatsAsIntorDouble` helper function to decide whether to write stats as `int32` or `double`. Updated `WriteCoreStorageStats` and `BuildEmptyResponseMessage` to use this helper function.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
NA
### Description
Ignore apiVersion in command requests
----
#### AI description (iteration 1)
#### PR Classification
Bug fix to ignore the `apiVersion` field in command requests if present.
#### PR Summary
This pull request ensures that the `apiVersion` field is ignored in various command requests, adding it to the list of common spec fields to be ignored.
- Added `apiVersion` to `IgnoredCommonSpecFields` in `commands_common.c`.
- Created new test cases in `commands_crud_ignore_common_spec_fields.sql` and `commands_crud_ignore_common_spec_fields.out` to verify the changes.
- Updated existing test cases in `commands_crud_ignore_common_spec_fields.sql` and `commands_crud_ignore_common_spec_fields.out` to include `apiVersion`.
- Modified `basic_schedule` to include the new test suite.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
BugFix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
This PR has following fix :
1. $bitwise query operator fixes: Validate If document is double it must should be under INT_64 Range.
2. corrected double to INT_64 conversion underflow case to Match with Native mongo behavior, Before change underflow was `-9223372036854775295` but after change underflow `-9223372036854776832`.
with the fix the all operators has improved correctness: $skip, $convert, $toDate, $toLong and all 4 bitwise query operator .
Related work items: #3100851, #3100975
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Test
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Making more compat tests pass
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Fix handling of skip in count command/aggregate
----
#### AI description (iteration 1)
#### PR Classification
Bug fix for handling invalid values in the `$skip` stage of BSON aggregation.
#### PR Summary
This pull request addresses the handling of invalid values in the `$skip` stage of BSON aggregation by improving error messages and type checks.
- `src/aggregation/bson_aggregation_pipeline.c`: Improved error messages for invalid `$skip` values and refined type checks to ensure values are numbers and non-negative.
- `Tests.Integration.Shell.Postgres/custom_js_tests.csv`: Added a new test case for FerretDB issue 401.
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
Add aggregation operator: $sampleRate
### Description
Implementation for $sampleRate operator
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
UDF
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Implemented covariancePop & covarianceSamp for setWindowFields
----
#### AI description (iteration 1)
#### PR Classification
Documentation
#### PR Summary
This pull request adds design documentation for the `$covariancePop` and `$covarianceSamp` aggregation operators.
- Added `covariancePop.md` detailing the specification, behavior, examples, implementation plan, and test plan for the `$covariancePop` operator.
- Added `covarianceSamp.md` detailing the specification, behavior, examples, implementation plan, and test plan for the `$covarianceSamp` operator.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
rename pgmongo.next_collection_id to helio_api.next_collection_id globally
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Update complete_upgrade to work even if cluster is readonly for full disk
If the control plane sets the cluster into readonly, we prohibit write operations at the planner level. However complete_upgrade should be successful to be able to move the clusters forward when they're on that state.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix to ensure the `complete_upgrade` function works even if the cluster is in a read-only state due to full disk.
#### PR Summary
This pull request updates the `complete_upgrade` function to handle scenarios where the cluster is read-only due to a full disk, ensuring the upgrade process can still proceed.
- `cluster_operations.c`: Modified `command_complete_upgrade` to temporarily disable the read-only setting during the upgrade.
- `extension_upgrade_diskfull_readonly.sql` and `extension_upgrade_diskfull_readonly.out`: Added new regression tests to verify the upgrade process in read-only mode.
- `upgrade_schedule`: Included the new test case for disk full read-only upgrade scenario.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Run helio_compat tests in pgmongo
----
#### AI description (iteration 1)
#### PR Classification
Failing test setup for `helio_compat` tests in `pgmongo`.
#### PR Summary
This pull request sets up a baseline for running `helio_compat` tests in the `pgmongo` project, although the tests are still failing.
- Added expected output for various BSON expression evaluations in `pgmongo/src/test/helio_compat/expected/bson_expr_eval_tests.out`.
### Does this PR have any customer impact?
No.
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix.
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No.
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No.
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
This PR fixes a bad input error when an empty document expression was handled by an addToSet transition.
```
[direct: mongos] test> db.coll1.aggregate([ { $group: { _id: "$_id", items: { $addToSet: { $getField: { field: "a", input: { b: 1 } } } } } } ])
MongoServerError: [ActivityId=03940874-08ea-4df5-ad67-9e1f87499861] Bad input format for addToSet transition.
```
The new implementation will ignore the empty document and continue the aggregation execution.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix to handle empty documents in the `addToSet` transition.
#### PR Summary
This pull request fixes the handling of empty documents in the `addToSet` transition within the BSON aggregation pipeline.
- `oss/pg_helio_api/src/aggregation/bson_aggregates.c`: Added a check to ensure `addToSet` transition skips empty documents.
- `oss/pg_helio_api/src/test/regress/sql/bson_aggregation_pipeline_tests_add_to_set_group.sql` and `oss/pg_helio_api/src/test/regress/expected/bson_aggregation_pipeline_tests_add_to_set_group.out`: Added new test cases to verify the correct handling of empty documents in the `addToSet` transition.
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes (helio_api.enableCollation). Yes.
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
Done
### Description
Runtime implementation of collation support in Mongo Find query.
We parse the Mongo collation spec and covert it to an ICU standard collation string. An example string is "und-u-ks-level1-kc-true This string uniquely identify collation-based string comparison logic by Postgres. See: https://www.postgresql.org/docs/current/collation.html
This document tracks the current support matrix for query operators
| Operator | Runtime | Index Pushdown | Comments
| --------- | ---- | ----- | ----
| $eq | Yes | No |
| $gt | Yes | No |
| $gte | Yes | No |
| $lt | Yes | No |
| $lte | Yes | No |
| $ne | Yes | No |
| $in | No* | No | See below
| $nin | No* | No | Do
| $and | Yes | No |
| $not | Yes | No |
| $nor | Yes | No |
| $or | Yes | No |
| $all | Yes | No |
| $elemMatch | Yes | No |
| $positional | N/A | N/A | positional query operators are only used in projections.
| $regex | N/A | N/A | $regex ignores collations in Native Mongo
| $exists | N/A | N/A |
| $size | N/A | N/A |
| $type | N/A | N/A |
| $bitsAllClear | N/A | N/A | Not relevant for Utf8
| $bitsAnyClear | N/A | N/A | Do
| $bitsAllSet | N/A | N/A | Do
| $bitsAnySet | N/A | N/A | Do
| $mod | N/A | N/A | Do
| $range | N/A | N/A | Only applies to Number
| $text | N/A | N/A | Seprate Index, has its own collation spec
| $geowithin | N/A | N/A | Not relevant for Utf8
| $geointersects | N/A | N/A | Do
* $in/$nin is implemented with HashMap that does not support collation friendly Hashkeys
* $regex : collations are ignored with
* $text: $text spec has collation specific parameters
* $rage: For numbers only
This document tracks the current support matrix for update operators/commands
| Operator | Runtime | Index Pushdown |
| --------- | ---- | ----- |
| find_and_modify | No | No |
| updateOne | No | No |
| updateMany | No | No |
| createIndex | No | No |
Tests:
1. All Gated Tests pass when Collation is enabled by default
TODO:
1. Index support [already implemented - but breaking down for easy review]
2. Persist collation info while collection created [already implemented - but breaking down for easy review]
3. $cond, $let
Related work items: #3412359
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Seg fault fix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
- Fix geospatial segmentation fault for $geonear, $near, $nearSpher when no valid geometry found in runtime
### Description
Fix geospatial segmentation fault for $geonear, $near, $nearSpher when no valid geometry found in runtime
----
#### AI description (iteration 1)
#### PR Classification
Bug fix for segmentation fault in `ST_Distance` function when calculating distance for null geometry.
#### PR Summary
This pull request addresses a core dump issue caused by a null pointer reference in the `ST_Distance` function during `$near` queries.
- `oss/pg_helio_api/src/geospatial/bson_geospatial_geonear.c`: Added checks to handle empty geo values and prevent null pointer dereference.
- `pgmongo/src/test/regress/sql/bson_aggregation_pipeline_tests_geonear.sql` and `pgmongo/src/test/regress/expected/bson_aggregation_pipeline_tests_geonear.out`: Added test cases to validate the fix and ensure proper error handling for invalid geo values.
Related work items: #3380924
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
DevOps
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
N/A
### Description
Updated libbson build to remove libicu as a dependency.
----
#### AI description (iteration 1)
#### PR Classification
Build configuration update to remove `libicu` as a dependency.
#### PR Summary
This pull request updates the build configuration for the `libbson` library to remove `libicu` as a dependency.
- `oss/scripts/install_setup_libbson.sh`: Updated build script to disable ICU support by adding `-DMONGOC_ENABLE_ICU=OFF -DENABLE_ICU=OFF` flags.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Testing
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Port bson basic compare tests to the OSS distributed extension
----
#### AI description (iteration 1)
#### PR Classification
Porting tests to the open-source distributed extension.
#### PR Summary
This pull request ports the BSON basic compare tests to the open-source distributed extension.
- Removed test cases from `pgmongo/src/test/regress/expected/bson_dollar_ops_basic_compare_tests_explain_index_no_bitmap.out`.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
New feature: Adds support for `$addToSet` in window aggregation stages.
#### PR Summary
This pull request introduces the `$addToSet` operator for window aggregation stages, enhancing the aggregation capabilities.
- `bson_aggregates.c`: Added logic to handle `$addToSet` for both regular and window aggregations, including state management and result finalization.
- `bson_aggregation_window_operators.c`: Implemented the `HandleDollarAddToSetWindowOperator` function to support the new operator in window contexts.
- Updated the `WindowOperatorDefinitions` to include the `$addToSet` operator.
Related work items: #2151565
### Does this PR have any customer impact?
Cx can use $merge stage
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature Enable
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Enabling Merge GUC to True By Default
----
#### AI description (iteration 1)
#### PR Classification
Configuration change to enable a feature by default.
#### PR Summary
This pull request enables the `$merge` stage in the pipeline by default and updates the error message for unsupported target collection creation.
- Modified `oss/pg_helio_api/src/helio_api_init.c` to set `DEFAULT_ENABLE_MERGE_STAGE` to `true`.
- Updated error and hint messages in `pgmongo/src/test/regress/expected/bson_aggregation_stage_merge_tests.out` to reflect the new behavior for `$merge` target collection creation.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Port 4 tests to helio_distributed and clean up the code to make them pass.
----
#### AI description (iteration 1)
#### PR Classification
Porting initial tests to a distributed environment.
#### PR Summary
This pull request ports the first four tests to a distributed environment, ensuring that the query evaluation functions work correctly in this setup.
- Added distributed test cases for various BSON expression evaluations in `src/test/regress/expected/bson_expr_eval_tests.out`.
### Does this PR have any customer impact?
It exposes a new API customers can use
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Adding code for CreateUser, UpdateUser and DropUser
Related work items: #3264072, #3278392, #3292323, #3292361, #3292364, #3315917
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Prep for porting backend distributed tests
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup and preparation for porting backend distributed tests.
#### PR Summary
This pull request removes obsolete code and prepares the codebase for porting backend distributed tests.
- Deleted `cluster_operations.c` and `version_utils.c` from `pgmongo/src`.
- Added new functions and triggers in `helio_distributed_test_helpers.sql` and `helio_distributed_test_helpers.out` for testing.
- Updated `cluster_operations.c` in `pg_helio_distributed` to include new cluster management functions.
- Added new SQL files for cluster operations and version utilities in `pg_helio_distributed`.
- Modified `pgmongo.c` to set up global variables for distributed functions and index build queue table creation.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Implement the arithmetic and boolean operators after the new aggregation framework pattern.
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Improve Performance of find queries by targeting the shard if available
----
#### AI description (iteration 1)
#### PR Classification
Performance improvement.
#### PR Summary
This pull request aims to enhance the performance of `find` queries by targeting the shard if available.
- `bson_aggregation_pipeline.c`: Added logic to use shard table if `allowShardBaseTable` is enabled and shard is available.
- `adhoc-perf-profiling.md`: Updated YCSB commands for loading data and running performance benchmarks with more detailed parameters.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
[let] Enable let by default on top level find, aggregate and lookup
This adds the missing validation and parsing of let spec to be able to enable it by default, and also changes the let GUCs to be true by default.
----
#### AI description (iteration 1)
#### PR Classification
New feature: Enable `let` by default on top-level `find`, `aggregate`, and `lookup` commands.
#### PR Summary
This pull request enables `let` by default for top-level `find`, `aggregate`, and `lookup` commands, and adds validation to disallow certain expressions in top-level `let` specifications.
- `bson_aggregation_pipeline.c`: Added functions to parse and validate top-level `let` expressions, disallowing path expressions, `CURRENT`, and `ROOT`.
- `bson_aggregation_nested_pipeline.c`: Updated validation for `let` expressions in nested pipelines to ensure no variables are used to define other variables.
- `bson_aggregation_pipeline_tests_let.out` and `bson_aggregation_pipeline_tests_let.sql`: Added tests for new `let` functionality and validation rules.
- `helio_api_init.c`: Enabled `let` and `lookup let` support by default.
Related work items: #1806751
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Updated distributed_hooks.c
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup to remove unnecessary include statements.
#### PR Summary
This pull request removes an unused include statement from the `distributed_hooks.c` file to improve code maintainability.
- `distributed_hooks.c`: Removed the unnecessary `#include <distributed/backend_data.h>` statement.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactor
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes(ish)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Introduce pg_helio_distributed for distribution hooks and make pgmongo depend on that.
Also makes the rum_join_scan loading of private rum bits optional
----
#### AI description (iteration 1)
#### PR Classification
New feature: Introduce Helio Distributed Extension
#### PR Summary
This pull request introduces the Helio Distributed Extension, which adds distributed execution capabilities to the Helio system.
- Removed `shard_colocation.c` from `/pgmongo/src/distribution/` and added `distributed_hooks.c` in `/oss/private/pg_helio_distributed/src/distribution/`.
- Updated `api_hook_impl.c` to use new distributed hooks.
- Added new Makefile and build scripts for the Helio Distributed Extension in `/oss/private/pg_helio_distributed/`.
- Introduced new test validation scripts and configurations in `/oss/private/pg_helio_distributed/src/test/regress/`.
- Updated `pgmongo` Makefile to include the new distributed extension.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Chmod +x all scripts
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
TSG
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
This change introduces a section in the SOP that includes a script enabling simultaneous updates across multiple accounts under the same subscription.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Write directly to shard if available
----
#### AI description (iteration 1)
#### PR Classification
Code cleanup and optimization.
#### PR Summary
This pull request refactors the shard handling logic to directly write to a shard if available, removing redundant code and simplifying the shard query process.
- `src/api_hook_impl.c`: Simplified shard query logic by removing the `GetShardIdOptimistically` function and optimizing the shard ID retrieval process.
- `src/metadata/collection.c`: Removed the `allowLocalShard` parameter and related logic, streamlining the collection retrieval process.
- `src/aggregation/bson_aggregation_output_pipeline.c` and `src/aggregation/bson_aggregation_pipeline.c`: Updated functions to use the simplified collection retrieval method.
- `include/metadata/collection.h`: Removed the declaration for `GetMongoCollectionOrViewByNameDatumWithLocalShard`.
- `src/helio_api_init.c`: Changed the default value of `UseLocalExecutionShardQueries` to `true`.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
adding support for passing extra window value
This window value is required in individual setWindowField operators like $rank, $denseRank, $locF,
As for each of them we need a way to check if the windowValue is empty or specifically content
Hence, raising a common pr
----
#### AI description (iteration 1)
#### PR Classification
New feature: adding support for passing window value in context.
#### PR Summary
This pull request introduces the ability to pass a window value within the context for window aggregation operations.
- `src/aggregation/bson_aggregation_window_operators.c`: Added `windowValue` to the context and updated the order of function calls to include `UpdateWindowAggregationOperator` with the new context.
- `src/aggregation/bson_aggregation_window_operators.c`: Introduced a new field `windowValue` in the `WindowOperatorContext` struct.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
https://www.mongodb.com/docs/manual/reference/operator/aggregation/unsetField/
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Refactoring
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
pgmongo.enableVectorDiskANNIndex: default is off
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Refactoring vector search to support different index types in oss helio and pgmongo
### 1. Index type
- helio: vector-ivf, vector-hnsw
- pgmongo: vector-ivf, vector-hnsw, vector-diskann
### 2. Vector index creation
- ParseIndexCreationSpecFunc
- ValidateIndexCreationSpecFunc
- GenerateIndexCreationCmdFunc
### 3. Vector search
- ParseIndexSearchSpecFunc
- ValidateIndexSearchSpecFunc
### 4. Planner and custom scan
- GetIndexAccessMethodOidFunc
- GetSimilarityOpOidByFamilyOidFunc
- SetSearchParametersToGUCFunc
- GetDefaultSearchParamBsonFunc
- CalculateSearchParamBsonFunc
### 5. Test cases
- bson_aggregation_pipeline_tests_vector_diskann
- bson_aggregation_pipeline_tests_vector_hnsw
- bson_aggregation_pipeline_tests_vector_ivf
- bson_aggregation_pipeline_tests_vector_mongo
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Dleete unsharded
----
#### AI description (iteration 1)
#### PR Classification
Performance improvement
#### PR Summary
This pull request enhances the performance of unsharded delete operations by introducing new serialization and deserialization methods and optimizing batch deletion processing.
- `src/commands/delete.c`: Added `SerializeUnshardedDeletionSpec` and `DeserializeDeleteWorkerSpecForUnsharded` functions to handle unsharded deletions.
- `src/commands/delete.c`: Modified `command_delete_worker` to support unsharded delete operations and improved batch deletion handling.
- `src/helio_api_init.c`: Introduced a new configuration flag `EnableUnshardedBatchDelete`.
- `src/commands/find_and_modify.c`: Updated `CallDeleteOne` to include a `forceInlineWrites` parameter.
- `include/commands/delete.h`: Updated function signature for `CallDeleteOne` to include `forceInlineWrites`.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Enable support for same source and target : Merge stage
----
#### AI description (iteration 1)
#### PR Classification
New feature: Enable support for using the same source and target in the merge stage of BSON aggregation.
#### PR Summary
This pull request introduces support for using the same collection as both the source and target in the merge stage of BSON aggregation pipelines.
- Modified `bson_aggregation_stage_merge_tests.sql` and `bson_aggregation_stage_merge_tests.out` to add tests for the new feature.
- Updated `bson_aggregation_output_pipeline.c` to remove the error check preventing the same source and target collections and added a TODO for future optimization.
### Does this PR have any customer impact?
Bugfixes for Background indexes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfixes
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
1) On index failure due to unique index violation, return DuplicateKey with the same error as insert
2) If an index build is in progress and a concurrent build is requested, wait on the concurrent build (rather than returning "build with the same options exist" - this means that if the first index build fails both calls fail and the second call doesn't assume the index exists
3) Don't query index builds during write commands (only query valid indexes for `$merge`, `$indexStats` etc
4) When an index build fails, don't immediately remove it from the index queue. Instead mark it as skippable. Once all the index builds complete after 20 mins, the index will be removed from the queue. This solves the issue of "index build failed but index was not found in main table"
5) Push unique index build to the background - running it in the foreground did not produce a better experience from private preview feedback
6) If the index build fails with a Mongo error, immediately mark it as skippable (not as failed)
----
#### AI description (iteration 1)
#### PR Classification
Bug fix
#### PR Summary
This pull request addresses issues with creating indexes in the background by improving error handling and index conflict detection.
- `commands_common.c`: Added `TryGetErrorMessageAndCode` function to improve error message retrieval and handling.
- `create_indexes.c`: Enhanced conflict detection for index specifications and added handling for indexes in progress.
- `create_indexes_background.c`: Improved error handling and logging for background index creation.
- `index.c`: Updated SQL queries to include index build progress status.
- `cluster_operations.c`: Adjusted setup logic to handle index build tasks more effectively.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Testing
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
N/A
### Description
Fixing locale of the ubuntu Test container, Mariner already sets this the way we want
----
#### AI description (iteration 1)
#### PR Classification
Bug fix to ensure consistent collation settings during PostgreSQL initialization.
#### PR Summary
This pull request updates the PostgreSQL initialization scripts to enforce consistent collation settings.
- Updated `oss/scripts/utils.sh` to include locale and encoding settings in `initdb` command.
- Modified `scripts/emulator_entrypoint.sh` to set locale and encoding during PostgreSQL initialization.
- Adjusted `scripts/pipeline_setup_jstests.sh` to apply locale and encoding settings in multiple `initdb` commands.
- Added collation settings to `scripts/utils.sh` for database initialization.
- Included schema listing commands in `helio_test_helpers.sql` and `pgmongo_test_helpers.sql` for better test visibility.
Flows a parse aggregation expression context while parsing the expression so that we can track if there were variables, operators, path expressions and do top level validation if needed for things like $let, etc.
----
#### AI description (iteration 1)
#### PR Classification
API change to add a parse aggregation expression context while parsing aggregation expressions.
#### PR Summary
This pull request introduces a `ParseAggregationExpressionContext` to track the types of expressions encountered during the parsing of aggregation expressions.
- `include/operators/bson_expression_operators.h`: Added `ParseAggregationExpressionContext` parameter to various `ParseDollar*` functions.
- `src/operators/bson_expression_bitwise_operators.c`: Updated bitwise operator parsing functions to include the new context parameter.
- `src/operators/bson_expression_date_operators.c`: Modified date operator parsing functions to use the new context parameter.
- `src/operators/bson_expression_string_operators.c`: Updated string operator parsing functions to include the context parameter.
- `include/operators/bson_expression.h`: Defined the `ParseAggregationExpressionContext` struct.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
adding a defensive check before
Generally all the drivers or clients have the check to only allow valid ranges of datetime/timestamp as input.
But , on a specific case if cx tries to convert to date a number or long using $todate function and then tries to convert it to string from date object then we were giving internal error.
As of now , have added a defensive check to fail fast on error before calling pg function
----
#### AI description (iteration 1)
#### PR Classification
Bug fix to add a defensive check for epoch conversion to PostgreSQL timestamp.
#### PR Summary
This pull request introduces a defensive check to prevent errors during the conversion of Unix epoch milliseconds to PostgreSQL timestamps.
- `oss/pg_helio_core/include/utils/date_utils.h`: Added a range check to validate the seconds before conversion to `timestamptz`.
- `pgmongo/src/test/regress/sql/bson_aggregation_date_operators_tests.sql`: Added test cases to verify the defensive check.
- `pgmongo/src/test/regress/expected/bson_aggregation_date_operators_tests.out`: Updated expected output to reflect the new error messages for invalid conversions.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
correcting error code
----
#### AI description (iteration 1)
#### PR Classification
Bug fix to change error codes so that errors are not shown as internal errors.
#### PR Summary
This pull request updates error codes to prevent certain errors from being classified as internal errors.
- `pcre_regex.c`: Changed error codes from `MongoInternalError` to `MongoOverflow` for PCRE2 context creation failures.
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Use the AOT C# Gateway as default. Run the JSTests against an out-of-proc AOT gateway.
Delete some dead code and support tests setting JSTest config.
----
#### AI description (iteration 1)
#### PR Classification
New feature
#### PR Summary
This pull request introduces a new AOT (Ahead-Of-Time) Gateway for the project, including configuration, logging, and pipeline setup.
- Added `SetupConfiguration.JsTests.json` for configuring the AOT Gateway.
- Updated `.pipelines/templates/managed_test.yml` to include steps for setting up and starting the managed gateway.
- Enhanced logging capabilities in `LoggingOptions.cs` and `PostgresMongoServiceHost.cs` to support file-based logging.
- Modified `pipeline_build_gateway_layout.sh` to always use the NativeAOT build path.
- Added `Serilog.Extensions.Logging.File` package in `Mongo.Host.Postgres.Linux.csproj` for file logging support.
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Perf
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Enable Colocation for existing clusters and turn it on by default.
Skip Colocating tables if they are part of an existing database that has non-colocation enabled tables. This will maintain back-compat for older clusters.
----
#### AI description (iteration 1)
#### PR Classification
Enable colocation by default.
#### PR Summary
This pull request enables colocation by default and updates the related test cases to reflect this change.
- `native_colocation_tests.sql`: Updated SQL queries to use specific database names (`dbcolo` and `dbcolo2`) for colocation operations.
- `native_colocation_tests.out`: Adjusted expected output to match the new colocation settings and database names.
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
----
#### AI description (iteration 1)
#### PR Classification
New feature: Adds support for index pushdown in join conditions for the `$merge` stage in aggregation pipelines.
#### PR Summary
This pull request introduces index pushdown capabilities for join conditions in the `$merge` stage, enhancing query performance.
- `src/opclass/index_support.c`: Added support functions for index pushdown in `$merge` join filters.
- `sql/udfs/aggregation/bson_merge_functions--1.20-0.sql`: Created new SQL functions to support index pushdown and handle merge operations.
- `src/aggregation/bson_aggregation_output_pipeline.c`: Implemented functions to create join expressions and extract merge filters.
- `Tests.Postgres.CSharp/Integration/Aggregation/MergeTests.cs`: Added integration tests to verify index pushdown functionality.
- `src/metadata/metadata_cache.c`: Added function OIDs for new merge-related functions.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Added support for $count in $setWindowFields Stage
----
#### AI description (iteration 1)
#### PR Classification
New feature: Adds support for `$count` accumulator in `setWindowFields` aggregation stage.
#### PR Summary
This pull request introduces the `$count` accumulator to the `setWindowFields` aggregation stage, along with corresponding tests and implementation.
- `oss/pg_helio_api/src/aggregation/bson_aggregation_window_operators.c`: Added `HandleDollarCountWindowOperator` function to handle `$count` window aggregation operator.
- `pgmongo/src/test/regress/sql/bson_aggregation_pipeline_stage_setWindowFields.sql`: Added multiple test cases for `$count` accumulator with various window specifications.
- `gateway/managed/Tests.Integration.Shell.Postgres/60_postgres_10260.csv` and `70_postgres_10260.csv`: Updated test results to reflect the support for `$count` in `setWindowFields`.
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Version update to start pgmongo 1.20 release
----
#### AI description (iteration 1)
#### PR Classification
Version update
#### PR Summary
This pull request updates the version numbers and adds release notes for the pgmongo 1.20 release.
- Added release notes in `docs/release_notes/pgmongo_1.21.md`
- Updated version numbers in `pg_helio_api.control`, `pg_helio_core.control`, and `pgmongo.control`
- Added SQL migration scripts for version updates in `pg_helio_api`, `pg_helio_core`, and `pgmongo`
- Updated table of contents in `docs/release_notes/toc.yml`
- Updated SHA1 checksums for SQL scripts in `pgmongo/sql/sha1`
### Does this PR have any customer impact?
No
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Feature
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
Yes
### Are you introducing any new config? If yes, do you have tests with and without them being set?
No
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
- Add support for $avg, $push in $setWindowFields stage.
- Add inverse transition functions for $sum, $avg, $push
### Description
Add the inverse transition function for $sum, $avg and $push for optimal window aggregation.
Also for `$push` it is not currently usable in windowing because of its specific state requirement, need to add another state for window operation for `$push`
### Does this PR have any customer impact?
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
### Are you introducing any new config? If yes, do you have tests with and without them being set?
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
Implementation of $stdDevPop and $stdDevSamp accumulators.
Reference:
[stdDevPop](https://www.mongodb.com/docs/manual/reference/operator/aggregation/stdDevPop/)
[stdDevSamp](https://www.mongodb.com/docs/manual/reference/operator/aggregation/stdDevSamp/)
Related work items: #3277074
### Does this PR have any customer impact?
Yes
### Type (Feature, Refactoring, Bugfix, DevOps, Testing, Perf, etc)
Bugfix
### Does it involve schema level changes? (Table, Column, Index, UDF, etc level changes)
No
### Are you introducing any new config? If yes, do you have tests with and without them being set?
Yes
### Release Note (Refer [Template](../docs/release_notes/pgmongo_template.md))
### Description
- Implement READ for some scan types to avoid internal errors
- Move feature flag for local shard use to the local shard use feature flag.
----
#### AI description (iteration 1)
#### PR Classification
Bug fix and feature flag correction.
#### PR Summary
This pull request addresses an internal error and corrects the usage of a feature flag.
- `custom_scan_continuation.c`: Added functions for reading and writing custom scan nodes, replacing unsupported functions.
- `rum_custom_join_scan.c`: Added functions for reading and writing custom join scan nodes, replacing unsupported functions.
- `api_hook_impl.c`: Corrected feature flag usage from `EnableNativeColocation` to `UseLocalExecutionShardQueries`.
- `custom_query_scan.c`: Updated error messages for unsupported node types to be more specific.
Murat Tuncer
Gledis Zeneli
Ayush Saluja
Deepak Bisht
Parag Jain
Vinod Sridharan
Amit Prabhudesai
Lusen Dong
Murat Tuncer
Xing Fan ⛳
Santiago Fernandez Madero
Lucas Borges Fernandes
Jayanta Mondal
John Pao
Cornelius Boateng
Deepak Pemmaraju
Aditya Kotalwar
Zinan Ren 🐾
Manish Gaur
Xiaochun Liang