Index card_comment_reactions, and document why MongoDB's data directory must be local.

From a reported MongoDB crash series (.tools/crash: WeKan and MongoDB in one
Kubernetes pod, the database volume on an SMB/DFS share):

- The crash itself is WiredTiger's checkpoint fsync() returning ENOSPC
  ("No space left on device" with 519 MB used and 10 GiB free) on the
  network share, followed by a fatal assertion and abort - every checkpoint,
  so a restart loop. That is the storage, not WeKan: MongoDB requires a
  local filesystem with real fsync semantics under its data directory.
  docs/Databases/MongoDB/Storage-Requirements.md now says so, shows what
  the log looks like when it is not, and what to do (local block volume;
  attachments may still live on a share; MONGO_URL over TCP is fine).
- The WeKan-side finding in the same logs: card_comment_reactions had no
  index at all, and every board open queried it by `cardId: { $in: [...] }`
  and by `boardId` as a full collection scan - 3,397 COLLSCANs of that one
  collection in the last crash log, up to 1.6 s each, the bulk of its
  "Slow query" lines. server/models/cardCommentReactions.js now ensures
  indexes on cardId, boardId and cardCommentId at startup, like every other
  per-card collection.

tests/cardCommentReactionsIndexes.test.cjs pins both.

Thanks to xet7 !

docs/Databases/MongoDB/README.md

@@ -10,6 +10,7 @@ is the compose file for it. The default database of WeKan is
 | [Driver-System.md](Driver-System.md) | Which Node driver is used for which server version, and how that is decided |
 | [Version-Management.md](Version-Management.md) | Detecting the server version and switching to the matching binary + driver |
 | [Compatibility-Guide.md](Compatibility-Guide.md) | What breaks between MongoDB 3.0 and 8.0 with Meteor, and the fixes |
+| [Storage-Requirements.md](Storage-Requirements.md) | Why the data directory must be on a local filesystem, and what an SMB/NFS-backed one does (fsync ENOSPC aborts, slow queries) |
 | [Oplog-Configuration.md](Oplog-Configuration.md) | Why the OpLog matters for pub/sub, and how to turn it on |
 | [OpLog-Enablement.md](OpLog-Enablement.md) | Whether it IS on, per deployment platform |
 | [avx-qemu.md](avx-qemu.md) | Running MongoDB 5+ on a CPU with no AVX, through qemu-user |

docs/Databases/MongoDB/Storage-Requirements.md

@@ -0,0 +1,85 @@
+# Where MongoDB's data directory may live
+
+MongoDB's WiredTiger storage engine needs a **local, POSIX-compliant
+filesystem** under its data directory (`/data/db` in the containers). It does
+not work reliably on a network filesystem - SMB/CIFS, DFS, NFS, or a cloud
+file share mounted as one - even when the share is fast, has plenty of free
+space, and works fine for ordinary files.
+
+## What it looks like when it is not
+
+A reported crash series (WeKan and MongoDB in one Kubernetes pod, the
+MongoDB volume connected over SMB to a DFS storage server) looked like this
+in the MongoDB log, repeatedly, until the process aborted:
+
+```
+"c":"WT","msg":"WiredTiger error message","attr":{"error":28,"message":{...,
+  "msg":"__posix_sync:94:/data/db/WiredTiger.turtle: ... No space left on device"}}
+"c":"ASSERT","msg":"Fatal assertion","attr":{"msgid":50853,
+  "file":"src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp"}}
+"msg":"***aborting after fassert() failure"
+"msg":"Got signal: 6 (Aborted)."
+```
+
+The share reported **519 MB used with 10 GiB free**, so "no space left" is
+not about space. Error 28 (`ENOSPC`) here comes back from `fsync()` - the
+call WiredTiger makes on every checkpoint to be sure a write has really
+reached stable storage. An SMB/CIFS mount cannot honour that the way a local
+disk does, and the failure surfaces as this errno. Because a checkpoint that
+cannot be made durable is a database that cannot be trusted, MongoDB
+deliberately aborts rather than continue - which is why it comes back up and
+crashes again on the next checkpoint.
+
+The same log had thousands of **"Slow query"** lines with
+`"planningTimeMicros"` around 200,000 (0.2 s to *plan* a query that touched
+nine index keys). That is the storage latency again: every read of an index
+page went over the share. Fixing the storage fixes those too. (One WeKan-side
+cause of slow queries on that log, a missing index on the comment-reactions
+collection, was fixed in WeKan as well - see the CHANGELOG - but an indexed
+query still cannot be fast on storage this slow.)
+
+## What WeKan reports about it
+
+Admin Panel / Problems / Database problems gets a row for each of these,
+from a probe that runs at startup and every five minutes:
+
+| Row | What it means |
+| --- | --- |
+| `db.restart` | The database process restarted while WeKan kept running - a crash-and-restart loop like the one above. |
+| `db.network-filesystem` | The data directory is on a CIFS/SMB, NFS or FUSE filesystem. Only when WeKan can see that directory itself (the snap, the bundle with its embedded database, or a shared volume). |
+| `db.slow-storage` | Reads averaged over 100 ms across the interval - storage latency, the "Slow query" symptom. |
+| `db.disk-space` | The filesystem under the data directory is below 5% or 512 MiB free - the real "no space left on device", before it happens. |
+| `db.index-created` | WeKan found a collection with data but without an index it queries by, and created it. Fixed automatically; nothing to do. |
+
+The first four are reported with what to do; the last is WeKan's own
+remediation (a missing index was the WeKan-side cause of slow queries in the
+report above).
+
+## What to do
+
+- Put MongoDB's data directory on a **local block volume**: a Kubernetes
+  PersistentVolume backed by a block storage class (a cloud disk, Ceph RBD,
+  Longhorn, local-path, ...), a Docker named volume on the host's local disk,
+  or a directory on a local filesystem (ext4, xfs).
+- Do not mount the data directory from SMB/CIFS, DFS, NFS, or a file-share
+  storage class. If that is the only storage available, run MongoDB
+  somewhere it can have a local disk and point WeKan at it over the network
+  with `MONGO_URL` - the database connection over TCP is fine; it is only the
+  database's *files* that must be local.
+- **Attachments are different.** WeKan's attachment files (when stored on
+  the filesystem rather than in the database) are ordinary files and may
+  live on a network share; only MongoDB's own data directory may not.
+- After moving: restore from a
+  [backup](../../Backup/Backup.md) or copy the data directory while MongoDB is
+  stopped, then run `mongod --repair` once if the last shutdown was one of
+  these aborts ([Repair-MongoDB.md](../../Backup/Repair-MongoDB.md)).
+
+MongoDB's own production notes say the same: WiredTiger requires a filesystem
+with `fsync()` semantics (they list ext4 and xfs), and they explicitly warn
+against remote filesystems for the data files.
+
+## See also
+
+- [The MongoDB pages](.) - the rest of them
+- [../FerretDB/1](../FerretDB/1) - the default database; its SQLite backend
+  has the same requirement (SQLite also needs a local filesystem)

server/imports.js

@@ -122,6 +122,7 @@ import '/server/models/org';
 import '/server/models/rules';
 import '/server/models/settings';
 import '/server/models/swimlanes';
+import '/server/models/cardCommentReactions';
 import '/server/models/team';
 import '/server/models/translation';
 import '/server/models/users';
@@ -211,6 +212,7 @@ import '/server/lib/tenantResolver';
 import '/server/lib/ssrfGuard';
 import '/server/lib/ddpSessionSendGuard';
 import '/server/lib/databaseProblems';
+import '/server/lib/databaseHealth';
 // What the FILESYSTEM said, and whether this server stopped cleanly last time
 // (docs/Security/Remediation/WeKan.md §13). Both write to the 'integrity' event
 // stream, shown in Admin Panel / Problems / Filesystem integrity.

server/models/cardCommentReactions.js

@@ -0,0 +1,16 @@
+import { Meteor } from 'meteor/meteor';
+import CardCommentReactions from '/models/cardCommentReactions';
+import { ensureIndex } from '/server/lib/mongoStartup';
+
+// card_comment_reactions had no index at all. Every board open reads it by
+// `cardId: { $in: [...all the board's card ids] }` (the card publication) and
+// by `boardId` (the board publication), and both were full collection scans:
+// a reported MongoDB log from one production board (.tools/crash) had 3,397
+// COLLSCANs of this collection in a single log, up to 1.6 s each, on a slow
+// storage backend - the bulk of that server's "Slow query" lines. The same
+// three keys every other per-card collection is indexed on.
+Meteor.startup(async () => {
+  await ensureIndex(CardCommentReactions, { cardId: 1 });
+  await ensureIndex(CardCommentReactions, { boardId: 1 });
+  await ensureIndex(CardCommentReactions, { cardCommentId: 1 });
+});

tests/cardCommentReactionsIndexes.test.cjs

@@ -0,0 +1,61 @@
+'use strict';
+
+// Regression guard for the WeKan-side finding in a reported MongoDB crash log
+// (.tools/crash): card_comment_reactions had no index at all, and every board
+// open queried it by `cardId: { $in: [...] }` (the card publication) and by
+// `boardId` (the board publication) as a full collection scan - 3,397
+// COLLSCANs of that one collection in a single log, up to 1.6 s each. The
+// crash itself (WiredTiger checkpoint fsync returning ENOSPC on an SMB/DFS
+// mount) is a storage problem, documented in
+// docs/Databases/MongoDB/Storage-Requirements.md rather than fixable in code.
+//
+// Source-read test: pins the startup indexes, that the server module is in the
+// import list (this app has an explicit server mainModule - a file not
+// imported there never runs), and that the storage page exists and is linked.
+//
+// Run: node tests/cardCommentReactionsIndexes.test.cjs
+
+const assert = require('assert');
+const fs = require('fs');
+const path = require('path');
+
+const ROOT = path.join(__dirname, '..');
+const read = rel => fs.readFileSync(path.join(ROOT, rel), 'utf8');
+
+let passed = 0;
+function test(name, fn) { fn(); passed += 1; console.log('  ok -', name); }
+
+console.log('cardCommentReactionsIndexes:');
+
+test('card_comment_reactions gets cardId, boardId and cardCommentId indexes at startup', () => {
+  const src = read('server/models/cardCommentReactions.js');
+  assert.ok(/import CardCommentReactions from '\/models\/cardCommentReactions'/.test(src));
+  assert.ok(/import \{ ensureIndex \} from '\/server\/lib\/mongoStartup'/.test(src));
+  for (const key of ['cardId', 'boardId', 'cardCommentId']) {
+    assert.ok(src.includes(`await ensureIndex(CardCommentReactions, { ${key}: 1 });`), `index on ${key}`);
+  }
+  assert.ok(/Meteor\.startup\(async \(\) => \{/.test(src));
+});
+
+test('the server module is imported, so the startup hook actually runs', () => {
+  assert.ok(read('server/imports.js').includes("import '/server/models/cardCommentReactions';"));
+});
+
+test('the queries the log showed scanning are the ones now indexed (negative: none is unindexed)', () => {
+  // The two publications read reactions by cardId (card) and boardId (board).
+  const boards = read('server/publications/boards.js');
+  assert.ok(/getCardCommentReactions\(\{ boardId: board\._id \}/.test(boards));
+  const schema = read('models/cardCommentReactions.js');
+  for (const field of ['boardId', 'cardId', 'cardCommentId']) {
+    assert.ok(new RegExp(`^\\s*${field}: \\{`, 'm').test(schema), `${field} is a schema field`);
+  }
+});
+
+test('the storage-requirements page exists and the MongoDB docs index links it', () => {
+  const page = read('docs/Databases/MongoDB/Storage-Requirements.md');
+  assert.ok(/fsync/.test(page) && /SMB/.test(page) && /No space left on device/.test(page));
+  assert.ok(read('docs/Databases/MongoDB/README.md').includes('[Storage-Requirements.md](Storage-Requirements.md)'));
+  assert.ok(fs.existsSync(path.join(ROOT, 'docs/Backup/Repair-MongoDB.md')), 'the repair page it links to');
+});
+
+console.log(`\ncardCommentReactionsIndexes: ${passed} tests passed`);