Previously, only FuzzDocument was reaching the marshal+unmarshal
steps due to the document type assertion. If it wasn't a documentType,
the rest of the fuzzing function was skipped.
Since data validation is available only via `types.Document.ValidateData`,
a helper `isValidDocumentData` function is introduced. It may go
away in the future. But for now, it allows all sjson types to pass
the validation.
The other change is related to the way we fuzz sjson.
Instead of generating both values and schemes at the same time,
focus on one thing at the time: a fixed scheme against the generated
and fixed documents against the generated schemes.
This approach can be further improved by the increased number
of relevant corpus seed entries, but that's a different issue (#3067).
To answer the question is it any better: yes, it is.
The old approach skipped any cases where either of those two
was invalid; it was a very rare occasion for a fuzz runner
to generate a valid JSON for both document and scheme.
It's even more rare to get both of them even remotely compatible.
Closes#1273
Remove double decoding in unmarshalSingleValue()
The callers of unmarshalSingleValue() always unmarshal
the inputs of this function into `json.RawMessage`.
These `json.RawMessage` contain valid JSON data
that can be parsed directly. Doing an Unmarshal of
`json.RawMessage` into `json.RawMessage` looks confusing
and not very useful.
This change is mostly about readability and code clarity.
As a side note, it also improves `sjson` unmarshalling performance.
name old time/op new time/op delta
Array/array_all/UnmarshalJSON-8 34.6µs ± 1% 21.7µs ± 1% -37.36% (p=0.000 n=10+9)
Document/all/UnmarshalJSON-8 120µs ± 0% 77µs ± 1% -36.04% (p=0.000 n=9+10)
[Geo mean] 64.5µs 40.8µs -36.70%
name old speed new speed delta
Array/array_all/UnmarshalJSON-8 1.62MB/s ± 1% 2.58MB/s ± 1% ~ (p=0.000 n=10+9)
Document/all/UnmarshalJSON-8 2.15MB/s ± 1% 3.37MB/s ± 1% +56.37% (p=0.000 n=9+10)
[Geo mean] 1.87MB/s 2.95MB/s +57.97%
name old alloc/op new alloc/op delta
Array/array_all/UnmarshalJSON-8 41.4kB ± 0% 22.0kB ± 0% -46.92% (p=0.000 n=10+10)
Document/all/UnmarshalJSON-8 117kB ± 0% 62kB ± 0% -46.89% (p=0.000 n=10+10)
[Geo mean] 69.6kB 37.0kB -46.91%
name old allocs/op new allocs/op delta
Array/array_all/UnmarshalJSON-8 184 ± 0% 119 ± 0% -35.33% (p=0.000 n=10+10)
Document/all/UnmarshalJSON-8 580 ± 0% 374 ± 0% -35.52% (p=0.000 n=10+10)
[Geo mean] 327 211 -35.42%
# Description
Closes#2157.
<!--
Write a short description to explain changes that are not mentioned in
the initial issue.
What were the reasons for those changes?
Which decisions did you make and why?
What else should reviewers know about your changes?
-->
## Readiness checklist
<!--
If you want your changes to be merged quickly,
please follow CONTRIBUTING.md.
-->
* [ ] I added/updated unit tests.
* [ ] I added/updated integration/compatibility tests.
* [ ] I added/updated comments and checked rendering.
* [ ] I made spot refactorings.
* [ ] I updated user documentation.
* [ ] I ran `task all`, and it passed.
* [x] I ensured that PR title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
---------
Co-authored-by: Chi Fujii <chi.fujii@ferretdb.io>
# Description
<!--
Write a short description to explain changes that are not mentioned in
the initial issue.
What were the reasons for those changes?
Which decisions did you make and why?
What else should reviewers know about your changes?
-->
## Readiness checklist
<!--
If you want your changes to be merged quickly,
please follow CONTRIBUTING.md.
-->
* [ ] I added/updated unit tests.
* [ ] I added/updated integration/compatibility tests.
* [ ] I added/updated comments and checked rendering.
* [ ] I made spot refactorings.
* [x] I updated user documentation.
* [ ] I ran `task all`, and it passed.
* [x] I ensured that PR title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
# Description
Closes#2157.
<!--
Write a short description to explain changes that are not mentioned in
the initial issue.
What were the reasons for those changes?
Which decisions did you make and why?
What else should reviewers know about your changes?
-->
## Readiness checklist
<!--
If you want your changes to be merged quickly,
please follow CONTRIBUTING.md.
-->
* [ ] I added/updated unit tests.
* [ ] I added/updated integration/compatibility tests.
* [ ] I added/updated comments and checked rendering.
* [ ] I made spot refactorings.
* [x] I updated user documentation.
* [ ] I ran `task all`, and it passed.
* [x] I ensured that PR title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
# Description
I stumbled upon this whilst testing Studio 3T.
<!--
Write a short description to explain changes that are not mentioned in
the initial issue.
What were the reasons for those changes?
Which decisions did you make and why?
What else should reviewers know about your changes?
-->
## Readiness checklist
<!--
If you want your changes to be merged quickly,
please follow CONTRIBUTING.md.
-->
* [ ] I added/updated unit tests.
* [ ] I added/updated integration/compatibility tests.
* [ ] I added/updated comments and checked rendering.
* [ ] I made spot refactorings.
* [ ] I updated user documentation.
* [ ] I ran `task all`, and it passed.
* [x] I ensured that PR title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
# Description
Closes#1996.
<!--
Write a short description to explain changes that are not mentioned in
the initial issue.
What were the reasons for those changes?
Which decisions did you make and why?
What else should reviewers know about your changes?
-->
## Readiness checklist
<!--
If you want your changes to be merged quickly,
please follow CONTRIBUTING.md.
-->
* [ ] I added/updated unit tests.
* [ ] I added/updated integration/compatibility tests.
* [ ] I added/updated comments and checked rendering.
* [ ] I made spot refactorings.
* [ ] I updated user documentation.
* [ ] I ran `task all`, and it passed.
* [x] I ensured that PR title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
# Description
<!--
Write a short description to explain changes that are not mentioned in
the initial issue.
What were the reasons for those changes?
Which decisions did you make and why?
What else should reviewers know about your changes?
-->
## Readiness checklist
<!--
If you want your changes to be merged quickly,
please follow CONTRIBUTING.md.
-->
* [ ] I added/updated unit tests.
* [ ] I added/updated integration/compatibility tests.
* [ ] I added/updated comments and checked rendering.
* [ ] I made spot refactorings.
* [ ] I updated user documentation.
* [ ] I ran `task all`, and it passed.
* [x] I ensured that PR title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
# Description
Closes#2015.
<!--
Write a short description to explain changes that are not mentioned in
the initial issue.
What were the reasons for those changes?
Which decisions did you make and why?
What else should reviewers know about your changes?
-->
## Readiness checklist
<!--
If you want your changes to be merged quickly,
please follow CONTRIBUTING.md.
-->
* [ ] I added/updated unit tests.
* [ ] I added/updated integration/compatibility tests.
* [ ] I added/updated comments and checked rendering.
* [ ] I made spot refactorings.
* [ ] I updated user documentation.
* [ ] I ran `task all`, and it passed.
* [x] I ensured that PR title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
# Description
Closes#1868.
<!--
Write a short description to explain changes that are not mentioned in
the initial issue.
What were the reasons for those changes?
Which decisions did you make and why?
What else should reviewers know about your changes?
-->
## Readiness checklist
<!--
If you want your changes to be merged quickly,
please follow CONTRIBUTING.md.
-->
* [ ] I added/updated unit tests.
* [ ] I added/updated integration/compatibility tests.
* [ ] I added/updated comments and checked rendering.
* [ ] I made spot refactorings.
* [x] I updated user documentation.
* [ ] I ran `task all`, and it passed.
* [x] I ensured that PR title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
# Description
Closes#1869.
<!--
Write a short description to explain changes that are not mentioned in
the initial issue.
What were the reasons for those changes?
Which decisions did you make and why?
What else should reviewers know about your changes?
-->
## Readiness checklist
<!--
If you want your changes to be merged quickly,
please follow CONTRIBUTING.md.
-->
* [ ] I added/updated unit tests.
* [ ] I added/updated integration/compatibility tests.
* [ ] I added/updated comments and checked rendering.
* [ ] I made spot refactorings.
* [ ] I updated user documentation.
* [ ] I ran `task all`, and it passed.
* [x] I ensured that PR title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
# Description
ClosesFerretDB/engineering#32.
<!--
Write a short description to explain changes that are not mentioned in
the initial issue.
What were the reasons for those changes?
Which decisions did you make and why?
What else should reviewers know about your changes?
-->
## Readiness checklist
<!--
If you want your changes to be merged quickly,
please follow CONTRIBUTING.md.
-->
* [ ] I added/updated unit tests.
* [ ] I added/updated integration/compatibility tests.
* [ ] I added/updated comments and checked rendering.
* [ ] I made spot refactorings.
* [ ] I updated user documentation.
* [ ] I ran `task all`, and it passed.
* [x] I ensured that PR title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
# Description
ClosesFerretDB/engineering#45
<!--
Write a short description to explain changes that are not mentioned in
the initial issue.
What were the reasons for those changes?
Which decisions did you make and why?
What else should reviewers know about your changes?
-->
## Readiness checklist
<!--
If you want your changes to be merged quickly,
please follow CONTRIBUTING.md.
-->
* [ ] I added/updated unit tests.
* [ ] I added/updated integration/compatibility tests.
* [ ] I added/updated comments and checked rendering.
* [ ] I made spot refactorings.
* [ ] I updated user documentation.
* [ ] I ran `task all`, and it passed.
* [x] I ensured that PR title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
# Description
ClosesFerretDB/engineering#16.
<!--
Write a short description to explain changes that are not mentioned in
the initial issue.
What were the reasons for those changes?
Which decisions did you make and why?
What else should reviewers know about your changes?
-->
## Readiness checklist
<!--
If you want your changes to be merged quickly,
please follow CONTRIBUTING.md.
-->
* [ ] I added/updated unit tests.
* [ ] I added/updated integration/compatibility tests.
* [ ] I added/updated comments and checked rendering.
* [ ] I made spot refactorings.
* [ ] I updated user documentation.
* [ ] I ran `task all`, and it passed.
* [x] I ensured that PR title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
# Description
<!--
Write a short description to explain changes that are not mentioned in
the initial issue.
What were the reasons for those changes?
Which decisions did you make and why?
What else should reviewers know about your changes?
-->
## Readiness checklist
<!--
If you want your changes to be merged quickly,
please follow CONTRIBUTING.md.
-->
* [ ] I added/updated unit tests.
* [ ] I added/updated integration/compatibility tests.
* [ ] I added/updated comments and checked rendering.
* [ ] I made spot refactorings.
* [ ] I updated user documentation.
* [ ] I ran `task all`, and it passed.
* [x] I ensured that PR title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
# Description
Closes#1825.
<!--
Write a short description to explain changes that are not mentioned in
the initial issue.
What were the reasons for those changes?
Which decisions did you make and why?
What else should reviewers know about your changes?
-->
## Readiness checklist
<!--
If you want your changes to be merged quickly,
please follow CONTRIBUTING.md.
-->
* [ ] I added/updated unit tests.
* [ ] I added/updated integration/compatibility tests.
* [ ] I added/updated comments and checked rendering.
* [ ] I made spot refactorings.
* [x] I updated user documentation.
* [ ] I ran `task all`, and it passed.
* [x] I ensured that PR title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
## Readiness checklist
* [x] I added/updated unit tests.
* [ ] I added/updated integration/compatibility tests.
* [x] I added/updated comments and checked rendering.
* [ ] I made spot refactorings.
* [ ] I updated user documentation.
* [x] I ran `task all`, and it passed.
* [x] I ensured that PR title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
# Description
ClosesFerretDB/engineering#17
<!--
Write a short description to explain changes that are not mentioned in
the initial issue.
What were the reasons for those changes?
Which decisions did you make and why?
What else should reviewers know about your changes?
-->
## Readiness checklist
<!--
If you want your changes to be merged quickly,
please follow CONTRIBUTING.md.
-->
* [ ] I added/updated unit tests.
* [ ] I added/updated integration/compatibility tests.
* [ ] I added/updated comments and checked rendering.
* [ ] I made spot refactorings.
* [ ] I updated user documentation.
* [ ] I ran `task all`, and it passed.
* [x] I ensured that PR title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
Co-authored-by: Patryk Kwiatek <patryk@kwiatek.xyz>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
# Description
Closes#1824.
<!--
Write a short description to explain changes that are not mentioned in
the initial issue.
What were the reasons for those changes?
Which decisions did you make and why?
What else should reviewers know about your changes?
-->
## Readiness checklist
<!--
If you want your changes to be merged quickly,
please follow CONTRIBUTING.md.
-->
* [ ] I added unit tests for new functionality or bug fixes.
* [ ] I added integration tests for new functionality or bug fixes.
* [ ] I added compatibility tests for new functionality or bug fixes.
* [ ] I made spot refactorings.
* [x] I updated user documentation.
* [ ] I ran `task all`, and it passed.
* [ ] I added/updated comments for both exported and unexported
top-level declarations (functions, types, etc).
* [ ] I checked comments rendering with `task godocs`.
* [x] I ensured that the title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
Bumps ferretdb/docusaurus-docs from 2.2.0-2 to 2.2.0-3.
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alexey Palazhchenko <alexey.palazhchenko@ferretdb.io>
# Description
<!--
Write a short description to explain changes that are not mentioned in
the initial issue.
What were the reasons for those changes?
Which decisions did you make and why?
What else should reviewers know about your changes?
-->
## Readiness checklist
<!--
If you want your changes to be merged quickly,
please follow CONTRIBUTING.md.
-->
* [ ] I added unit tests for new functionality or bug fixes.
* [ ] I added integration tests for new functionality or bug fixes.
* [ ] I added compatibility tests for new functionality or bug fixes.
* [ ] I made spot refactorings.
* [ ] I updated user documentation.
* [ ] I ran `task all`, and it passed.
* [ ] I added/updated comments for both exported and unexported
top-level declarations (functions, types, etc).
* [ ] I checked comments rendering with `task godocs`.
* [x] I ensured that the title is good enough for the changelog.
* [x] (for maintainers only) I set Reviewers
([`@FerretDB/core`](https://github.com/orgs/FerretDB/teams/core)),
Assignee, Labels, Project and project's Sprint fields.
* [x] I marked all done items in this checklist.
Closes#1593.
We cannot enable authorization to run both FerretDB and MongoDB with `PLAIN` authorization mechanism because it is not available for MongoDB. Implemented stub was tested locally with FerretDB.
For example, `find` should not fail when `returnKey` is explicitly set to `false`, not omitted.
That's useful for some drivers.
Some type checking, particularly around numerical parameters, looks a bit strange, but refactoring of that will be done later.
We run a lot of tests in parallel, so the number of collections is in the range between 1 (this test) and many, where "many" is 100 for now.
Uptime can also be as small as 0 seconds on a fast machine but can also be large if the environment wasn't restarted for a long time.
This PR provides additional examples for tests' shared data composites.
These examples will help to cover more test cases.
For example, they are helpful for #724.
Also, `array-empty-nested` reveals a bug described in #732.
Explicitly install Go on CI before invoking any actions. Our actions will soon require Go 1.18, but the default version of Go on GitHub-hosted runners is 1.17. The installation step is still fast as 1.18 is also present, just not enabled by default, and `actions/setup-go@v3` is smart enough to use it.
Additionally, we skip the Go test cache entirely as it is not smart enough to notice changes in the environment. Our tests are fast, so that's not a problem. We might re-enable cache for local runs later, but we will not use it on CI.
This adds the default Task target that should be run before submitting PRs. It can be invoked as `bin/task all` or simply as `bin/task`.
This also adds a very basic PR template that will be improved in the future.
This PR adds an empty Tigris handler (copied from the dummy handler with minimal edits) and dependencies. We need it to make future PRs smaller and simpler.
# Description
This PR closes#590. This change aims to improve the visibility of errors in logs.
The change proposes a new function (`logResponse`) to handle the logic of log levels according to the requirements described in #590. The function is suitable for both "normal" and "proxy" responses.
In addition to that, in case the response returns a warning/error, diffs are also logged with warning/error level accordingly.
# Testing
There are no special tests for this logging format change. One can run a FerretDB instance locally and send various queries from the client to see what the log levels are.
`.golangci.yml` configuration contains linters that should pass for all code.
`.golangci-new.yml` configuration (currently empty) contains new or experimental linters that should pass (or be reconfigured) for new code.
@@ -4,24 +4,74 @@ This file contains some process notes for FerretDB Inc. engineers and long-term
If you are a community contributor, feel free to ignore all this!
We will do all that for you so you can focus on what is really interesting for you.
We encourage diversity between us as different points of view expand our vision and contribute to the project's success.
This guide tries to be short and does not mention things that are completely automated by our tooling.
Keep guides short: we don't create rules; we automate.
## Showing progress
We want our work to be visible and transparent.
This way we can help each other and make sure that we are on the right track.
To uphold these principles, we agree to make results of our daily work available for others through various means
such as pushed commits, updated pull requests, code reviews, created issues, or any other relevant method.
### Community
We expect issues picked up by the community to be done in about two weeks.
After about a week of inactivity, we should ping contributors.
## Pull requests
1. Pull request **title** should be accurate and descriptive as it is used in the generated changelog.
It should not end with a dot.
It should not mention the issue number but describe the fixed issue instead.
2. Pull request **description** typically just mentions the issue number like "Refs #123." or "Closes #456.".
Do not use `https://github.com/org/repo/issue/123` even if it is rendered the same on GitHub.
End sentences with a dot.
3. Commit messages (both titles and bodies) are not important and can be "WIP" or anything else.
All commits are always squashed on merge by GitHub, so there is no need to squash them manually, amend them, and/or do force pushes.
Autogenerated GitHub's squash commit's title and body should be manually replaced by PR's title and description; that includes both manual merges and auto-merges.
Do not leave original commit messages in the squashed commit body.
For FerretDB Inc. engineers and long-term contributors,
besides the guides in [CONTRIBUTING.md](../CONTRIBUTING.md),
please follow these notes below when you working with pull request:
## Calls "to show and/or to explain"
1. Send pull requests from forks; do not make personal branches in the main repository.
This way, we are similar to community members and could notice similar problems that we could fix for everyone,
not just for us.
2. Pull request title should be accurate and descriptive
as it is used in the generated [changelog](../CHANGELOG.md) as is.
It generally should start with imperative verb ("Fix …", not "Fixing …", "Fixed …" or other forms).
It should not have a prefix ("Fix …", not "Fix: …") (we use labels for categorization).
It should not mention the issue number.
3. It is fine to send several sequential pull requests for one issue to make them easier to review.
In that case, please still use the `Closes` word as described in [CONTRIBUTING.md](../CONTRIBUTING.md)
(because [words like "refs" do not link PR to the issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword)),
but don't forget to reopen the issue once PR is merged,
but the issue as a whole is not done.
4. Please create a draft pull request as soon as you start working on an issue; that is needed for our process tooling.
Once it is ready for review, please mark it as such.
After that, there is no need to switch between draft and non-draft states.
5. Pull requests should be merged by an auto-merge that the author should enable.
If the pull request is not ready to be merged for some reason (but ready to be reviewed as usual),
the description should explain why, and the `do not merge` label should be applied.
Even in that case, auto-merge should still be enabled by the author.
6. Autogenerated GitHub's squash commit's body should be manually replaced by `Closes #<issue_number>.`
with optional [trailers for co-authors](https://docs.github.com/en/pull-requests/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors).
7. Ultimately, the author is responsible for doing everything to ensure that the pull request is merged and done.
Please multiply the knowledge shared on calls and 1-1 messages by writing a separate detailed blog-post article (not in the project's source code).
1-1s are good to check what you might have missed along the way - what seemed to you obvious and to others is not.
One can see all pull requests that await review [there](https://github.com/pulls/review-requested?q=user%3AFerretDB+is%3Aopen).
We also have [#devlog channel](https://ferretdb.slack.com/archives/C02P0MR7VJS)
in our community Slack with periodic reminders.
To make them useful,
talk with [@GitHub bot in direct messages](https://ferretdb.slack.com/archives/D02P4EJPFGV)
to link your Slack and GitHub identities,
and then configure notification only for your name.
### Test scripts
Pull requests that add temporary `test.js` test scripts have slightly different rules:
1. The title should start with "Add test script for".
2. The descriptions of both PR and issue should link to each other.
3. Commits must change only the `test.js` file.
Once PR with a test script is sent and verified by someone else, it is closed.
Once the linked issue is resolved, the PR should be reopened and merged with `main` to confirm that the test script pass now.
Then it should be closed again.
## Creating issues
1. Please use `Chore` type instead of `Enhancement` or `Feature` (but not `Bug`)
because its template is much more useful
(but may be too daunting for regular users, that's why templates for enhancements and features are much simpler).
After creating an issue, please change its type (label) manually.
2. Do not change other automatically applied labels, assignees, or projects.
name:Integration# visible at https://github.com/FerretDB/FerretDB/actions
on:
workflow_call:
inputs:
task:
required:true
type:string
shard_index:
required:true
type:number
shard_total:
required:true
type:number
test_timeout:
required:false
type:string
default:20m
no_xfail:
required:false
type:boolean
default:false
skip_flaky_check:
required:false
type:boolean
default:false
env:
GOPATH:/home/runner/go
GOCACHE:/home/runner/go/cache
GOLANGCI_LINT_CACHE:/home/runner/go/cache/lint
GOMODCACHE:/home/runner/go/mod
GOPROXY:https://proxy.golang.org
GOTOOLCHAIN:local
jobs:
run:
# make it short to fit in GitHub UI; all parameters are already in the caller's name
name:Run
runs-on:ubuntu-24.04
timeout-minutes:25
steps:
# TODO https://github.com/FerretDB/github-actions/issues/211
- name:Checkout code
uses:actions/checkout@v5
with:
fetch-depth:0# for `git describe` to work
lfs:false# LFS is used only by website
# for branch.txt on pull_request; the correct branch is already checked out on push / schedule
- name:Name branch
if:github.event_name == 'pull_request'
env:
BRANCH:${{ github.head_ref }}# see https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable
# TODO https://github.com/FerretDB/github-actions/issues/211
- name:Checkout code
uses:actions/checkout@v3
uses:actions/checkout@v5
with:
fetch-depth:0
fetch-depth:0# for `git describe` to work
lfs:false# LFS is used only by website
# for branch.txt on pull_request; the correct branch is already checked out on push / schedule
- name:Name branch
if:github.event_name == 'pull_request'
env:
BRANCH:${{ github.head_ref }}# see https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable
run:git checkout -b $BRANCH
- name:Setup Go
uses:FerretDB/github-actions/setup-go@main
with:
cache-key:short-test
- name:Install Task
run:go generate -x
working-directory:tools
- name:Run init
run:bin/task init
- name:Run short unit tests
run:bin/task test-unit-short TEST_TIMEOUT=10m
env:
GOFLAGS:${{ runner.debug == '1' && '-v' || '' }}
GITHUB_TOKEN:invalid# to test that -short is handled correctly
# we don't want them on CI
- name:Clean test and fuzz caches
if:always()
run:go clean -testcache -fuzzcache
- name:Check dirty
if:always()
run:|
git status --untracked-files --ignored
git status
git diff --exit-code
test:
name:Test
runs-on:ubuntu-24.04
timeout-minutes:15
# Do not run this job in parallel for any PR change or branch push.
# TODO https://github.com/FerretDB/github-actions/issues/211
- name:Checkout code
uses:actions/checkout@v5
with:
fetch-depth:0# for `git describe` to work
lfs:false# LFS is used only by website
# for branch.txt on pull_request; the correct branch is already checked out on push / schedule
- name:Name branch
if:github.event_name == 'pull_request'
env:
BRANCH:${{ github.head_ref }}# see https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable
run:git checkout -b $BRANCH
- name:Setup Go
uses:FerretDB/github-actions/setup-go@main
@@ -38,148 +123,256 @@ jobs:
run:go generate -x
working-directory:tools
# install.sh starts ollama service, stop it since we want to run with `bin/task ollama` with environment variables
- name:Setup Ollama
run:|
curl -fsSL https://ollama.com/install.sh | sudo -E sh
sudo systemctl stop ollama
- name:Start Ollama in the background
run:|
bin/task ollama &
- name:Login to GitHub Container Registry
uses:docker/login-action@v3
with:
registry:ghcr.io
username:${{ github.actor }}
password:${{ secrets.GITHUB_TOKEN }}
- name:Start environment
run:bin/task env-up-detach
- name:Pull and load Ollama model
run:|
bin/task ollama-model
- name:Run init
run:bin/task init
- name:Regenerate and reformat
run:bin/task gen fmt docs-fmt
- name:Wait for and setup environment
run:bin/task env-setup
- name:Run unit tests
run:bin/task test-unit
- name:Regenerate and reformat
run:bin/task gen
env:
# must be a SUPERUSER to ensure parameter_default is set
# TODO https://github.com/FerretDB/github-actions/issues/211
- name:Checkout code
uses:actions/checkout@v5
with:
name:linux-logs
path:/tmp/logs
fetch-depth:0# for `golangci-lint run --new` to work
lfs:false# LFS is used only by website
# To prevent Setup Go's cache to include it.
# See https://github.com/golang/go/issues/48901.
- name:Clear generated corpus
if:always()
run:go clean -fuzzcache
# for branch.txt on pull_request; the correct branch is already checked out on push / schedule
- name:Name branch
if:github.event_name == 'pull_request'
env:
BRANCH:${{ github.head_ref }}# see https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable
# TODO https://github.com/FerretDB/github-actions/issues/211
- name:Checkout code
if:github.event_name != 'pull_request_target'
uses:actions/checkout@v5
with:
fetch-depth:0# for `git describe` to work
lfs:false# LFS is used only by website
# TODO https://github.com/FerretDB/github-actions/issues/211
- name:Checkout pull request code
if:github.event_name == 'pull_request_target'
uses:actions/checkout@v5
with:
fetch-depth:0# for `git describe` to work
lfs:false# LFS is used only by website
ref:${{ github.event.pull_request.head.sha }}
# for version.txt on push tags; see https://github.com/actions/checkout/issues/290
- name:Fetch annotated tags
run:|
git fetch --tags --force
git status
# for branch.txt on pull_request_target; the correct branch is already checked out on push / schedule
- name:Name branch
if:github.event_name == 'pull_request_target'
env:
BRANCH:${{ github.head_ref }}# see https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable
run:git checkout -b $BRANCH
- name:Setup Go
uses:FerretDB/github-actions/setup-go@main
# no cache-key on server
- name:Install Task
run:go generate -x
working-directory:tools
- name:Run init
run:bin/task init
- name:Setup QEMU
uses:docker/setup-qemu-action@v3
- name:Initialize Docker Buildx builder
run:bin/task docker-init
# mostly to check that this command still works
- name:Build local Docker image
run:bin/task docker-local FILE=production
- name:Define Docker image names
id:tag
run:bin/definedockertag
- name:Login to Docker Hub
uses:docker/login-action@v3
with:
username:ferretdbbot
password:${{ secrets.DOCKER_HUB_TOKEN }}
- name:Login to GitHub Container Registry
uses:docker/login-action@v3
with:
registry:ghcr.io
username:${{ github.actor }}
password:${{ secrets.GITHUB_TOKEN }}
- name:Login to Quay.io
uses:docker/login-action@v3
with:
registry:quay.io
username:ferretdbbot
password:${{ secrets.QUAY_TOKEN }}
- name:Build and push evaluation development Docker images ${{ steps.tag.outputs.eval_dev_images }}
# TODO https://github.com/FerretDB/github-actions/issues/211
- name:Checkout code
uses:actions/checkout@v5
with:
fetch-depth:0# for wrangler to work
lfs:true
ref:${{ github.event.pull_request.head.sha }}
# for Wrangler / Cloudflare Pages on pull_request_target
- name:Name branch
env:
BRANCH:${{ github.head_ref }}# see https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable
run:git checkout -b $BRANCH
# no ppremk/lfs-warning due to https://github.com/ppremk/lfs-warning/issues/150
- name:Setup Go
uses:FerretDB/github-actions/setup-go@main
with:
cache-key:website-preview
- name:Install Task
run:go generate -x
working-directory:tools
- name:Build tools
run:bin/task env-pull
- name:Build blog
run:bin/task blog
env:
GITHUB_TOKEN:${{ secrets.GITHUB_TOKEN }}# for checkdocs
- name:Check dirty
run:|
git status
git diff --exit-code
- name:Deploy to Cloudflare Pages
run:bin/task blog-cloudflare-preview | tee deploy.txt
# That also allows us to pass branch name, commit hash, etc from git to wrangler.
# TODO https://github.com/FerretDB/github-actions/issues/211
- name:Checkout code
uses:actions/checkout@v5
with:
fetch-depth:0# for wrangler to work
lfs:true
ref:${{ github.event.pull_request.head.sha }}# TODO https://github.com/FerretDB/github-actions/issues/211
# for Wrangler / Cloudflare Pages on pull_request_target
- name:Name branch
env:
BRANCH:${{ github.head_ref }}# see https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable
run:git checkout -b $BRANCH
# no ppremk/lfs-warning due to https://github.com/ppremk/lfs-warning/issues/150
- name:Setup Go
uses:FerretDB/github-actions/setup-go@main
with:
cache-key:website-preview
- name:Install Task
run:go generate -x
working-directory:tools
- name:Build tools
run:bin/task env-pull
- name:Create directory
run:mkdir -p tmp/githubcache
- name:Build docs
run:bin/task docs
env:
GITHUB_TOKEN:${{ secrets.GITHUB_TOKEN }}# for checkdocs
- name:Check dirty
run:|
git status
git diff --exit-code
- name:Deploy to Cloudflare Pages
run:bin/task docs-cloudflare-preview | tee deploy.txt
We plan to promote most of them to stable in the next release.
### What's Changed
In this release, we made a big change in the way FerretDB fetches data from PostgreSQL.
#### RPM packages for DocumentDB
Previously, we generated a single SQL query that extensively used json/jsonb PostgreSQL functions for each incoming MongoDB request, then converted fetched data.
All the filtering was performed by PostgreSQL.
Unfortunately, the semantics of those functions do not match MongoDB behavior in edge cases like comparison or sorting of different types.
That resulted in a difference in behavior between FerretDB and MongoDB, and that is a problem we wanted to fix.
DocumentDB `.rpm` packages for Red Hat Enterprise Linux are
So starting from this release we fetch more data from PostgreSQL and perform filtering on the FerretDB side.
This allows us to match MongoDB behavior in all cases.
Of course, that also greatly reduces performance.
We plan to address it in future releases by pushing down parts of filtering queries that can be made fully compatible with MongoDB.
For example, a simple query like `db.collection.find({_id: 'some-id-value'})` can be converted to SQL `WHERE` condition relatively easy and be compatible even with weird values like IEEE 754 NaNs, infinities, etc.
### Documentation 📄
In short, we want FerretDB to be compatible with MongoDB first and fast second, and we are still working towards the first goal.
- Enable full visibility for images on zoom by @Fashander in https://github.com/FerretDB/FerretDB/pull/5346
- Add Vault blog post by @Fashander in https://github.com/FerretDB/FerretDB/pull/5368
- Add WeKan blog post by @Fashander in https://github.com/FerretDB/FerretDB/pull/5381
### Other Changes 🤖
- Add YugabyteDB integration tests by @chilagrow in https://github.com/FerretDB/FerretDB/pull/5274
- Run all YugabyteDB integration tests by @chilagrow in https://github.com/FerretDB/FerretDB/pull/5283
- Refactor normal and error responses by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5326
- Update DocumentDB by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5353
- Use Codecov for flaky integration tests detection by @noisersup in https://github.com/FerretDB/FerretDB/pull/5361
- Make accessing YugabyteDB logs easier by @chilagrow in https://github.com/FerretDB/FerretDB/pull/5362
- Add components in codecov by @noisersup in https://github.com/FerretDB/FerretDB/pull/5364
- Skip YugabyteDB setup for `arm64` by @chilagrow in https://github.com/FerretDB/FerretDB/pull/5372
- Tweak Ollama and MCPHost setup by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5375
- Simplify flags handling by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5378
- Update DocumentDB by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5380
- Extract wiring into a separate package by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5383
- Small cleanup by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5388
- Encapsulate PostgreSQL pool in the Handler by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5389
- Bump DocumentDB version by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5392
- Test flaky tests detection by @noisersup in https://github.com/FerretDB/FerretDB/pull/5393
- Small tweaks by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5394
- Make a resource.Untrack thread-safe by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5396
- Update DocumentDB by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5398
- Enable MongoDB test commands by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5400
- Update Prometheus metrics by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5403
- Update DocumentDB by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5411
[All closed issues and pull requests](https://github.com/FerretDB/FerretDB/milestone/79?closed=1).
* Implement `$bitsAllClear` by @w84thesun in https://github.com/FerretDB/FerretDB/pull/394
* Support `$elemMatch` projection query operator by @seeforschauer in https://github.com/FerretDB/FerretDB/pull/383
* Support all bitwise query operators by @w84thesun in https://github.com/FerretDB/FerretDB/pull/400
* Support `$eq` comparision operator by @ribaraka in https://github.com/FerretDB/FerretDB/pull/309
### Fixed Bugs 🐛
* Fix a few issues found by fuzzing by @AlekSi in https://github.com/FerretDB/FerretDB/pull/345
* More fixes for bugs found by fuzzing by @AlekSi in https://github.com/FerretDB/FerretDB/pull/346
* Commands are case-sensitive by @AlekSi in https://github.com/FerretDB/FerretDB/pull/369
* Make updates work by @AlekSi in https://github.com/FerretDB/FerretDB/pull/385
* Handle any number type for `limit` by @AlekSi in https://github.com/FerretDB/FerretDB/pull/399
* Fix numbers comparision by @ribaraka in https://github.com/FerretDB/FerretDB/pull/356
* Fix finding zero documents with `FindOne` by @AlekSi in https://github.com/FerretDB/FerretDB/pull/409
* Fix `sort` for arrays and documents by @AlekSi in https://github.com/FerretDB/FerretDB/pull/424
* Fix pgdb helpers by @AlekSi in https://github.com/FerretDB/FerretDB/pull/425
- Allow `clusterAdmin` users to perform user management commands by @chilagrow in https://github.com/FerretDB/FerretDB/pull/5314
### Enhancements 🛠
* Update `insert` command's help by @narqo in https://github.com/FerretDB/FerretDB/pull/321
* Return correct error codes for projections by @seeforschauer in https://github.com/FerretDB/FerretDB/pull/384
* Add SortDocuments by @w84thesun in https://github.com/FerretDB/FerretDB/pull/378
- Use debug-level log messages for continuations by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5266
### Documentation 📄
* Add Docker badge by @AlekSi in https://github.com/FerretDB/FerretDB/pull/305
* Tweak Markdown linter a bit by @AlekSi in https://github.com/FerretDB/FerretDB/pull/393
- Add blog post on setting up LibreChat with FerretDB by @Fashander in https://github.com/FerretDB/FerretDB/pull/4999
- Add blog post for Payload CMS by @Fashander in https://github.com/FerretDB/FerretDB/pull/5257
- Add blog post on using Compass GUI by @Fashander in https://github.com/FerretDB/FerretDB/pull/5263
- Fix social links by @Fashander in https://github.com/FerretDB/FerretDB/pull/5264
- Add PayloadCMS to compatible apps by @Fashander in https://github.com/FerretDB/FerretDB/pull/5270
- Add Compass to compatible apps by @Fashander in https://github.com/FerretDB/FerretDB/pull/5275
- Wrap long lines in a blog post by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5280
- Add NodeBB compatibility blog post by @Fashander in https://github.com/FerretDB/FerretDB/pull/5285
- Add Librechat to compatibility section by @Fashander in https://github.com/FerretDB/FerretDB/pull/5286
- Add NodeBB to compatible apps by @Fashander in https://github.com/FerretDB/FerretDB/pull/5293
- Cleanup NodeBB blog post by @chilagrow in https://github.com/FerretDB/FerretDB/pull/5294
- Add Growthbook blogpost by @Fashander in https://github.com/FerretDB/FerretDB/pull/5304
- Add Heyform compatibility blog post by @Fashander in https://github.com/FerretDB/FerretDB/pull/5316
- Add GrowthBook to compatible apps by @Fashander in https://github.com/FerretDB/FerretDB/pull/5319
- Add DBeaver compatibility blogpost by @Fashander in https://github.com/FerretDB/FerretDB/pull/5321
- Add HyperDX blog post by @Fashander in https://github.com/FerretDB/FerretDB/pull/5334
- Add compatible applications by @Fashander in https://github.com/FerretDB/FerretDB/pull/5335
- Prepare v2.4.0 release by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5344
### Other Changes 🤖
* Remove Docker volumes on `make env-down` by @AlekSi in https://github.com/FerretDB/FerretDB/pull/315
* Update deps by @AlekSi in https://github.com/FerretDB/FerretDB/pull/320
* Build static binaries by @AlekSi in https://github.com/FerretDB/FerretDB/pull/322
* Integrate with dance PRs by @AlekSi in https://github.com/FerretDB/FerretDB/pull/324
* Bump Docker images by @AlekSi in https://github.com/FerretDB/FerretDB/pull/325
* Bump github.com/golangci/golangci-lint from 1.44.0 to 1.44.2 in /tools by @dependabot in https://github.com/FerretDB/FerretDB/pull/327
* Bump mvdan.cc/gofumpt from 0.2.1 to 0.3.0 in /tools by @dependabot in https://github.com/FerretDB/FerretDB/pull/329
* Bump actions/checkout from 2 to 3 by @dependabot in https://github.com/FerretDB/FerretDB/pull/333
* Various small cleanups by @AlekSi in https://github.com/FerretDB/FerretDB/pull/334
* Rewrite `generate.sh` in Go by @w84thesun in https://github.com/FerretDB/FerretDB/pull/338
* Add helper for getting required parameters by @AlekSi in https://github.com/FerretDB/FerretDB/pull/339
* Use safe type assertions for inputs by @AlekSi in https://github.com/FerretDB/FerretDB/pull/341
* Fix seed fuzz corpus collection by @AlekSi in https://github.com/FerretDB/FerretDB/pull/340
* Add fuzzing tests for handlers by @AlekSi in https://github.com/FerretDB/FerretDB/pull/328
* Add `bin/task` to tools by @AlekSi in https://github.com/FerretDB/FerretDB/pull/349
* Add more checks for Go versions by @AlekSi in https://github.com/FerretDB/FerretDB/pull/350
* Improve Windows tooling by @w84thesun in https://github.com/FerretDB/FerretDB/pull/348
* Add assertions for BSON values comparision by @AlekSi in https://github.com/FerretDB/FerretDB/pull/352
* Replace `Makefile` with `Taskfile` by @w84thesun in https://github.com/FerretDB/FerretDB/pull/358
* Fix Taskfile by @w84thesun in https://github.com/FerretDB/FerretDB/pull/365
* Remove OS-specific Taskfiles, cleanup by @AlekSi in https://github.com/FerretDB/FerretDB/pull/366
* Remove SQL storage by @w84thesun in https://github.com/FerretDB/FerretDB/pull/367
* Use square brackets for nicer logs by @AlekSi in https://github.com/FerretDB/FerretDB/pull/373
* Fix build tags by @AlekSi in https://github.com/FerretDB/FerretDB/pull/374
* Add converter from types.Regex to regexp.Regexp by @AlekSi in https://github.com/FerretDB/FerretDB/pull/375
* Log test failures for updates and deletes by @AlekSi in https://github.com/FerretDB/FerretDB/pull/376
* Filter documents using Go code by @AlekSi in https://github.com/FerretDB/FerretDB/pull/370
* Projection: `<field>: <1 or true>` and `<field>: <0 or false>` by @seeforschauer in https://github.com/FerretDB/FerretDB/pull/377
* Fix small issues after rewrite by @AlekSi in https://github.com/FerretDB/FerretDB/pull/380
* Projection: `<field>: <1 or true>` and `<field>: <0 or false>`: error messages formatting by @seeforschauer in https://github.com/FerretDB/FerretDB/pull/382
* Bump dependecnies by @AlekSi in https://github.com/FerretDB/FerretDB/pull/387
* Fix some fluky tests by @AlekSi in https://github.com/FerretDB/FerretDB/pull/351
* Minor CI and build tweaks by @AlekSi in https://github.com/FerretDB/FerretDB/pull/390
* Add Markdown linter by @fenogentov in https://github.com/FerretDB/FerretDB/pull/386
* Do not cache modules by @AlekSi in https://github.com/FerretDB/FerretDB/pull/392
* Fix more fluky tests by @AlekSi in https://github.com/FerretDB/FerretDB/pull/391
* Fix the last fluky test by @AlekSi in https://github.com/FerretDB/FerretDB/pull/395
* Allow access to actual listener's address by @AlekSi in https://github.com/FerretDB/FerretDB/pull/397
* Add a new way to write integration tests by @AlekSi in https://github.com/FerretDB/FerretDB/pull/389
* Move `internal/pg` to `internal/handlers/pg/pgdb` by @AlekSi in https://github.com/FerretDB/FerretDB/pull/401
* Move `handlers/jsonb1` to `handlers/pg` by @AlekSi in https://github.com/FerretDB/FerretDB/pull/402
* Move handler to `pg` by @AlekSi in https://github.com/FerretDB/FerretDB/pull/403
* Move tests back for now by @AlekSi in https://github.com/FerretDB/FerretDB/pull/404
* Use `testutil.AssertEqual` helper by @AlekSi in https://github.com/FerretDB/FerretDB/pull/407
* Move `$size` tests to integration tests by @AlekSi in https://github.com/FerretDB/FerretDB/pull/410
* Improve logging in integration tests by @AlekSi in https://github.com/FerretDB/FerretDB/pull/412
* Tweak `hello`/`ismaster`/`isMaster` responses by @AlekSi in https://github.com/FerretDB/FerretDB/pull/418
* Fix named loggers by @AlekSi in https://github.com/FerretDB/FerretDB/pull/427
* Add tests for `getLog` by @AlekSi in https://github.com/FerretDB/FerretDB/pull/421
* Bump deps by @AlekSi in https://github.com/FerretDB/FerretDB/pull/430
## New Contributors
*@narqo made their first contribution in https://github.com/FerretDB/FerretDB/pull/321
*@w84thesun made their first contribution in https://github.com/FerretDB/FerretDB/pull/338
*@seeforschauer made their first contribution in https://github.com/FerretDB/FerretDB/pull/377
*@fenogentov made their first contribution in https://github.com/FerretDB/FerretDB/pull/386
*@ribaraka made their first contribution in https://github.com/FerretDB/FerretDB/pull/356
- Add YugabyteDB to local setup by @chilagrow in https://github.com/FerretDB/FerretDB/pull/5258
- Update DocumentDB by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5261
- Remove skipping of `bypassEmptyTsReplacement` by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5267
- Update DocumentDB by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5268
- Bring back Codecov and Coveralls integration by @chilagrow in https://github.com/FerretDB/FerretDB/pull/5276
- Update DocumentDB by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5282
- Add `mcphost` as tool by @chilagrow in https://github.com/FerretDB/FerretDB/pull/5287
- Refactor requests handling by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5292
- Add Dependabot configuration for mcphost by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5296
- Extend middleware requests and responses by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5301
- Minor refactoring of Data API by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5302
- Bump deps by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5307
- Refactor responses handling by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5310
- Add authentication test for client reconnecting by @chilagrow in https://github.com/FerretDB/FerretDB/pull/5311
- Do not use deprecated method by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5312
- Update TODO URLs by @chilagrow in https://github.com/FerretDB/FerretDB/pull/5317
- Remove `must.NotFail(wire[bson].NewXXX(` pattern by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5318
- Update TODO URL for `dropUsers` test by @chilagrow in https://github.com/FerretDB/FerretDB/pull/5324
- Avoid extra marshaling by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5327
- Split command handling and handlers by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5328
- Update DocumentDB by @AlekSi in https://github.com/FerretDB/FerretDB/pull/5342
[All closed issues and pull requests](https://github.com/FerretDB/FerretDB/milestone/12?closed=1).
* A new name! ([see here](https://github.com/FerretDB/FerretDB/discussions/100))
* Added support for databases sizes in `listDatabases` command ([#61](https://github.com/FerretDB/FerretDB/issues/61), thanks to [Leigh](https://github.com/OpenSauce)).
- Fix version detection for embeddable package by @AlekSi in https://github.com/FerretDB/FerretDB/pull/4936
* Added support for comparison query operators: `$eq`,`$gt`,`$gte`,`$in`,`$lt`,`$lte`,`$ne`,`$nin`. ([#26](https://github.com/FerretDB/FerretDB/issues/26))
* Added support for logical query operators: `$and`, `$not`, `$nor`, `$or`. ([#27](https://github.com/FerretDB/FerretDB/issues/27))
### Enhancements 🛠
[All closed issues and pull requests](https://github.com/FerretDB/FerretDB/milestone/1?closed=1).
Thank you for your interest in making FerretDB better!
## Contributing code
## Finding something to work on
The supported way of developing FerretDB is to modify and run it on the host (Linux, macOS, or Windows with [WSL2](https://docs.microsoft.com/en-us/windows/wsl/)), with PostgreSQL and other dependencies running inside Docker Compose.
We are interested in all contributions, big or small, in code or documentation.
But unless you are fixing a tiny issue like a typo,
we kindly ask you first to [create an issue](https://github.com/FerretDB/FerretDB/issues/new/choose)
or leave a comment on an existing issue if you want to work on it.
This way, we could assign the issue to you, marking it as being worked on,
preventing others from wasting efforts working on it at the same time.
Additionally, you can [join our Slack chat](./README.md#community) and leave a message for us in the `#dev` channel.
You will need Go 1.18 as FerretDB extensively uses ([fuzzing](https://go.dev/doc/tutorial/fuzz)) and [generics](https://go.dev/doc/tutorial/generics)).
If your package manager does not provide it yet, please install it from [go.dev](https://go.dev/dl/).
You can find a list of good first issues for contributors [there](https://github.com/FerretDB/FerretDB/contribute).
Once you have some experience with contributing to FerretDB,
feel free to pick any issue
[that is not assigned to anyone and doesn't have `not ready` label](https://github.com/FerretDB/FerretDB/issues?q=is%3Aissue+is%3Aopen+no%3Aassignee+-label%3A%22not+ready%22).
Still, please leave a comment on it as described above.
### Cloning the Repository
## Setting up the environment
After forking FerretDB on GitHub, you can clone the repository:
### Requirements
The supported way of contributing to FerretDB is to modify and run it on the host (Linux, macOS, or Windows)
with PostgreSQL and other dependencies running inside Docker containers via Docker Compose.
On Linux, `docker` (with `docker compose` subcommand a.k.a. [Compose V2](https://docs.docker.com/compose/compose-v2/),
not old `docker-compose` tool) should be installed on the host.
On macOS and Windows, [Docker Desktop](https://www.docker.com/products/docker-desktop/) should be used.
On Windows, it should be [configured to use WSL 2](https://docs.docker.com/desktop/windows/wsl/) without any distro;
all commands should be run on the host.
You will need Go 1.25 or later on the host.
If your package manager doesn't provide it yet,
please install it from [go.dev](https://go.dev/dl/).
You will also need `git` installed; the version provided by your package manager should do.
On Windows, the simplest way to install it might be https://gitforwindows.org.
Finally, you will also need [git-lfs](https://git-lfs.github.com) installed and configured (`git lfs install`).
### Making a working copy
Fork the [FerretDB repository on GitHub](https://github.com/FerretDB/FerretDB/fork).
After that, you can clone the repository and add the upstream repository as a remote:
The last command would fetch all Git tags from our repository that GitHub won't copy to fork by default.
`git describe` command uses them to determine the FerretDB version during the build process.
To run development commands, you should first install the [`task`](https://taskfile.dev/) tool.
You can do this by changing the directory to `tools` (`cd tools`) and running `go generate -x`.
That will install `task` into the `bin` directory (`bin/task` on Linux and macOS, `bin\task.exe` on Windows).
You can then add `./bin` to `$PATH` either manually (`export PATH=./bin:$PATH` in `bash`) or using something like (`direnv` (`.envrc` files)[https://direnv.net], or replace every invocation of `task` with explicit `bin/task`.
You can also [install `task` globally](https://taskfile.dev/#/installation), but that might lead to the version skew.
You can then add `./bin` to `$PATH` either manually (`export PATH=./bin:$PATH` in `bash`)
or using something like [`direnv` (`.envrc` files)](https://direnv.net),
or replace every invocation of `task` with explicit `bin/task`.
You can also [install `task` globally](https://taskfile.dev/#/installation),
but that might lead to the version skew.
With `task` installed, you may do the following:
With `task` installed,
you can see all available tasks by running `task -l` in the root of the repository
(not in the `tools` directory).
1. Install development tools with `task init`.
2. Download required Docker images with `task env-pull`.
3. Start the development environment with `task env-up`.
This will start PostgreSQL and MongoDB containers, filling them with identical sets of test data.
4. Run all tests in another terminal window with `task test`.
5. Start FerretDB with `task run`.
This will start it in a development mode where all requests are handled by FerretDB, but also routed to MongoDB.
The differences in response are then logged and the FerretDB response is sent back to the client.
6. Run `mongosh` with `task mongosh`.
This allows you to run commands against FerretDB.
7. Change the code.
8. Run `task fmt` to format source code.
9. Run `task lint` to run linter checks.
After that, you should install development tools with `task init`
and download required Docker images with `task env-pull`.
You can see all available `task` tasks with `task -l`.
If something does not work correctly,
you can reset the environment with `task env-reset`.
### Code Overview
### Building a production release binaries
Package `cmd` provides commands implementation. `ferretdb` is the main FerretDB binary; others are tools for development.
To build a production release binaries, run `task build-production`.
The results will be saved `tmp/bin`.
Package `tools` uses "tools.go" approach to fix tools versions.
### Setting a GITHUB_TOKEN
Some of our development tools require access to public information on GitHub
at a rate higher than allowed for unauthenticated requests.
Those tools will report a problem in this case.
It could be solved by creating a new classic or fine-grained personal access token
[there](https://github.com/settings/tokens).
No scopes are needed for classic tokens, not even `public_repo`.
For fine-grained tokens, only read-only access to public repositories is needed without any additional permissions.
After generating a token, set the `GITHUB_TOKEN` environment variable:
```sh
exportGITHUB_TOKEN=ghp_XXX
```
or
```sh
exportGITHUB_TOKEN=github_pat_XXX
```
## Reporting a bug
We appreciate reporting a bug to us.
To help us accurately identify the cause, we encourage you to include a pull request with test script.
Please write the test script in [build/legacy-mongo-shell/test.js](build/legacy-mongo-shell/test.js).
You can find an overview of the available assertions [here](build/legacy-mongo-shell/README.md).
Use these assertions to validate your test's assumptions and invariants.
With `task` installed (see above), you may test your script using following steps:
1. Start the development environment with `task env-up`.
2. Start FerretDB with `task run`.
3. Run the test script with `task testjs`.
Please create a pull request and include the link of the pull request in the bug issue.
## Contributing code
### Commands for contributing code
With `task` installed (see above), you may do the following:
1. Start the development environment with `task env-up`.
2. Run all tests with `task test` (see [below](#running-tests)).
3. Start FerretDB with `task run`.
The development environment uses `diff-normal` mode as the default mode.
Set preferred mode such as `task run MODE=diff-proxy`.
(See [Operation modes](https://docs.ferretdb.io/configuration/operation-modes/) page in our documentation.)
4. Fill collections in the `test` database with data for experiments with `task env-data`.
5. Run `mongosh` with `task mongosh`.
See what collections were created by the previous command with `show collections`.
#### Commands for authenticated connection
1. Start the development environment with `task env-up`.
2. Start FerretDB with `task run-secure MODE=normal` or `task run-secure MODE=proxy`.
3. Run `task mongosh-secure` to establish an authenticated connection.
### Code overview
The directory `cmd` provides commands implementation.
Its subdirectory `ferretdb` is the main FerretDB binary; others are tools for development.
The package `tools` uses ["tools.go" approach](https://github.com/golang/go/issues/25922#issuecomment-402918061) to fix tools versions.
They are installed into `bin/` by `cd tools; go generate -x`.
`internal` subpackages contain most of the FerretDB code:
The `internal` subpackages contain most of the FerretDB code:
*`types` package provides Go types matching BSON types that don't have built-in Go equivalents:
we use `int32` for BSON's int32, but `types.ObjectID` for BSON's ObjectId.
*`fjson` provides converters from/to FJSON for built-in and `types` types.
FJSON adds some extensions to JSON for keeping object keys in order, preserving BSON type information, etc.
FJSON is used by `jsonb1` handler/storage.
*`bson` package provides converters from/to BSON for built-in and `types` types.
-`mongoerrors` provides MongoDB-compatible error types and codes.
They are used by`handler` and `documentdb`.
Those packages are tested by "unit" tests that are placed inside thosepackages.
Some of them are truly hermetic and test only the package that contains them; you can run those "short" tests with `go test -short` or `task test-unit-short`, but that's typically not required.
Other unit tests use real databases; you can run those with `task test-unit` after starting the environment as described above.
The wire protocol implementation and BSON handling code were extracted into a separate repository:
https://github.com/FerretDB/wire
We also have a set of "integration" tests in `integration` Go module that uses the Go MongoDB driver and tests either a running MongoDB-compatible database (such as FerretDB or MongoDB itself) or in-process FerretDB.
They allow us to ensure compatibility between FerretDB and MongoDB.
You can run them with `task test-integration-ferretdb` for in-process FerretDB (meaning that integration tests start and stop FerretDB themselves), `task test-integration-mongodb` for MongoDB running on port 37017 (as in our development environment), or `task test-integration` to run both in parallel.
#### Running tests
Finally, you may run all tests in parallel with `task test`.
`internal` packages are tested by "unit" tests that are placed inside those packages.
Some of them are truly hermetic and test only the package that contains them;
you can run those "short" tests with `go test -short` or `task test-unit-short`,
but that's typically not required.
Other unit tests use real databases;
you can run those with `task test-unit` after starting the environment as described above.
We also have a set of "integration" tests in the `integration` directory.
They use the Go MongoDB driver like a regular user application.
They could test any MongoDB-compatible database (such as FerretDB or MongoDB itself) via a regular TCP or TLS port
or Unix domain socket.
They also could test in-process FerretDB instances
(meaning that integration tests start and stop them themselves).
Finally, some integration tests (so-called compatibility or "compat" tests) connect to two systems
("target" for FerretDB and "compat" for MongoDB) at the same time,
send the same queries to both, and compare results.
You can run them with:
-`task test-integration-postgresql` for in-process FerretDB and MongoDB;
-`task test-integration-mongodb` for MongoDB only, skipping compat tests;
- or `task test-integration` to run all in parallel.
You may run all tests in parallel with `task test`.
If tests fail and the output is too confusing, try running them sequentially by using the commands above.
In general, we prefer integration tests over unit tests, tests using real databases over short tests and real objects over mocks.
You can also run `task -C 1` to limit the number of concurrent tasks, which is useful for debugging.
(You might disagree with our terminology for "unit" and "integration" tests; let's not fight over it.)
To run a subset of integration tests and test cases, you may use Task variable `TEST_RUN`.
For example, to run all tests for the `getMore` command with in-process FerretDB
you may use `task test-integration-postgresql TEST_RUN='(?i)GetMore'`.
We have an additional integration testing system in another repository: <https://github.com/FerretDB/dance>.
Finally, since all tests just run `go test` with various arguments and flags under the hood
(for example, `TEST_RUN` just provides the value for the [`-run` flag](https://pkg.go.dev/cmd/go#hdr-Testing_flags)),
you may also use all standard `go` tool facilities,
including [`GOFLAGS` environment variable](https://pkg.go.dev/cmd/go#hdr-Environment_variables).
For example:
- to run all tests for the `getMore` command sequentially,
you may use `env GOFLAGS='-parallel=1 -p=1' task test-integration-postgresql TEST_RUN='(?i)GetMore'`;
- to run all tests with MongoDB with [Go execution tracer](https://pkg.go.dev/runtime/trace) enabled
you may use `env GOFLAGS='-trace=trace.out' task test-integration-mongodb`.
<!-- textlint-disable one-sentence-per-line -->
> [!NOTE]
> It is not recommended to set `GOFLAGS` and other Go environment variables with `export GOFLAGS=...`
> or `go env -w GOFLAGS=...` because they are invisible and easy to forget about, leading to confusion.
<!-- textlint-enable one-sentence-per-line -->
In general, we prefer integration tests over unit tests,
tests using real databases over short tests
and real objects over mocks.
(You might disagree with our terminology for "unit" and "integration" tests;
let's not fight over it.)
We have an additional integration testing system in another repository: https://github.com/FerretDB/dance.
#### Observability in tests
Integration tests start a debug handler with pprof profiles and execution traces on a random port
(to allow running multiple test configurations in parallel).
They also send telemetry traces to the local Jaeger instance that can be accessed at http://127.0.0.1:16686/.
### Code style and conventions
Above everything else, we value consistency in the source code.
If you see some code that doesn't follow some best practice but is consistent,
please keep it that way;
but please also tell us about it, so we can improve all of it.
If, on the other hand, you see code that is inconsistent without apparent reason (or comment),
please improve it as you work on it.
Our code follows most of the standard Go conventions,
documented on [CodeReviewComments wiki page](https://go.dev/wiki/CodeReviewComments)
and some other pages such as [Spelling](https://go.dev/wiki/Spelling).
Some of our idiosyncrasies are documented below.
1. We use type switches over BSON types in many places in our code.
The order of `case`s follows this order: https://pkg.go.dev/github.com/FerretDB/wire/wirebson#hdr-Types
It may seem random, but it is only pseudo-random and follows BSON spec: https://bsonspec.org/spec.html
2. We generally pass and return `struct`s by pointers.
There are some exceptions like `time.Time` that have value semantics, but when in doubt – use pointers.
#### Comments conventions
1. All top-level declarations, even unexported, should have documentation comments.
2. In documentation comments do not describe the name in terms of the name itself (`// Registry is a registry of …`).
Use other words instead; often, they could add additional information and make reading more pleasant (`// Registry stores …`).
3. In code comments, in general, do not describe _what_ the code does: it should be clear from the code itself
(and when it doesn't and the code is tricky, simplify it instead).
Instead, describe _why_ the code does that if it is not clear from the surrounding context, names, etc.
There is no need to add comments just because there are no comments if everything is already clear without them.
4. For code comments, write either
sentence fragments (do not start it with a capital letter, do not end it with a dot, use the simplified grammar) for short notes
or full sentences (do start them with capital letters, do end them with dots, do check their grammar) when a longer (2+ sentences)
explanation is needed (and the code could not be simplified).
#### Logging conventions
(See also our user documentation for notes about logging levels, logging sensitive information, etc.)
1. Log messages should contain a single sentence; use a semicolon if you must.
2. Log messages should not have leading/trailing spaces or end with punctuation.
3. Log field names should use `snake_case`.
4. Whatever sensitive information can be logged should be checked by calling `.Enabled(slog.LevelDebug)` on the appropriate logger,
not by directly comparing levels with `<` / `>` operators.
The same check should be applied if additional sensitive fields should be added to the log message on a different level.
#### Integration tests conventions
We prefer our integration tests to be straightforward and
branchless (with a few, if any, `if` and `switch` statements).
Ideally, the same test should work for both FerretDB and MongoDB.
If that's impossible without some branching, use helpers exported from the `setup` package,
such us `FailsForFerretDB`, `SkipForMongoDB`, etc.
Please use expected failure rather than skipping the test, whenever it's possible.
It makes tracking development progress much easier.
The bar for using other ways of branching, such as checking error codes and messages, is very high.
Writing separate tests might be much better than making a single test that checks error text.
Also, we should use driver methods as much as possible instead of testing commands directly via `RunCommand`.
In most cases, helpers such as `AssertEqualDocuments` are used to compare `bson.D` documents.
The differences in field order, types, missing fields and others should be reported.
If differences are expected such as `version` field, only the field key can be used for `bson.D` document comparison.
Use of (deprecated) `bson.D.Map()` or converting `bson.D` to other types using helpers should be avoided when possible.
The bar for adding new helpers is very high.
Please check all existing ones.
If there's a need to use any large number to test corner cases,
we create constants for them with explanation what do they represent, and refer to them.
For example:
```go
constdoubleMaxPrec=float64(1<<53-1)// 9007199254740991.0: largest double values that could be represented as integer exactly
```
Avoid writing tests that cannot run in parallel with other tests.
The `integration` directory contains multiple test packages,
and tests from other packages may run at the same time.
*`postgres` container runs PostgreSQL 14 that would store data.
*`postgres_setup` container creates a PostgreSQL schema `test` that would act like a FerretDB database of the same name.
*`ferretdb` runs FerretDB.
2. Start services with `docker-compose up -d`.
3. If you have `mongosh` installed, just run it to connect to FerretDB database `test`.
If not, run the following command to run `mongosh` inside the temporary MongoDB container, attaching to the same Docker network:
Run this command to start FerretDB with PostgreSQL, make sure to update `<username>` and `<password>`:
```sh
docker run --rm -it --network=ferretdb --entrypoint=mongosh mongo:5 mongodb://ferretdb/
docker run -d --rm --name ferretdb -p 27017:27017 \
-e POSTGRES_USER=<username> \
-e POSTGRES_PASSWORD=<password> \
ghcr.io/ferretdb/ferretdb-eval:2
```
<!-- markdownlint-enable MD029 -->
This command will start a container with FerretDB, pre-packaged PostgreSQL with DocumentDB extension, and MongoDB Shell for quick testing and experiments.
However, it is unsuitable for production use cases because it keeps all data inside and loses it on shutdown.
See our [installation guides](https://docs.ferretdb.io/installation/) for instructions
that don't have those problems.
With that container running, you can:
- Connect to it with any MongoDB client application using the MongoDB URI `mongodb://<username>:<password>@127.0.0.1:27017/`.
- Connect to it using the MongoDB Shell by just running `mongosh`.
If you don't have it installed locally, you can run `docker exec -it ferretdb mongosh`.
- For PostgreSQL, connect to it by running `docker exec -it ferretdb psql -U <username> postgres`.
You can stop the container with `docker stop ferretdb`.
We also provide binaries and packages for various Linux distributions.
as well as [Go library package](https://pkg.go.dev/github.com/FerretDB/FerretDB/v2/ferretdb)
that embeds FerretDB into your application.
See [our documentation](https://docs.ferretdb.io/installation/) for more details.
## Building and packaging
<!-- textlint-disable one-sentence-per-line -->
> [!NOTE]
> We advise users not to build FerretDB themselves.
> Instead, use binaries, Docker images, or packages provided by us.
<!-- textlint-enable one-sentence-per-line -->
FerretDB could be built as any other Go program,
but a few generated files and build tags could affect it.
See [there](https://pkg.go.dev/github.com/FerretDB/FerretDB/v2/build/version) for more details.
* [Slack chat](https://join.slack.com/t/ferretdb/shared_invite/zt-zqe9hj8g-ZcMG3~5Cs5u9uuOPnZB8~A) for quick questions.
* [GitHub Discussions](https://github.com/FerretDB/FerretDB/discussions) for longer topics.
* [GitHub Issues](https://github.com/FerretDB/FerretDB/issues) for bugs and missing features.
* [Open Office House meeting](https://calendar.google.com/event?action=TEMPLATE&tmeid=NjNkdTkyN3VoNW5zdHRiaHZybXFtb2l1OWtfMjAyMTEyMTNUMTgwMDAwWiBjX24zN3RxdW9yZWlsOWIwMm0wNzQwMDA3MjQ0QGc&tmsrc=c_n37tquoreil9b02m0740007244%40group.calendar.google.com&scp=ALL)
every Monday at 18:00 UTC at [Google Meet](https://meet.google.com/mcb-arhw-qbq).
- Website: https://www.ferretdb.com/.
- Blog: https://blog.ferretdb.io/.
- Documentation: https://docs.ferretdb.io/.
- X (Twitter): [@ferret_db](https://x.com/ferret_db).
Loaded 100 of 1497 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.