0 Fork 1

mirror of https://github.com/wekan/wekan.git synced 2026-11-07 02:52:33 -05:00

Files

T

Lauri Ojansivu 8e4d57c21d Show a recovery maintenance page on all FerretDB v1 platforms

During a #6492 data recovery, users should never see a bare connection error.
Two layers now cover the whole window on every FerretDB v1 launch path (snap,
bundled release, Docker):

- In-app spinner (already present): the server clears the RECOVERY_IN_PROGRESS
  marker only after a real health probe, so the full-screen overlay behaves
  identically everywhere.
- Static bridge page (new, cross-platform): for the brief window while a
  just-restored FerretDB comes back up and before Meteor can serve the client,
  the launch scripts serve a tiny standalone "recovering your data" page (HTTP
  503) on the web port. The snap reuses wekan-maintenance-page.mjs with a
  recovery wording; release/Docker serve the portable recovery-bridge.mjs. The
  bridge is time-bounded (WEKAN_RECOVERY_BRIDGE_SECONDS, default 20s) so it can
  never block WeKan from starting, hands straight over to the in-app spinner, and
  is skipped if its page file or the marker is absent.

The launch scripts do NOT delete the marker; only the server does, after
verifying the database serves reads. Tests enforce the bridge is bounded, is
guarded by the page file, and that no script deletes the marker.

Thanks to bluetopaz1204, mueschel and xet7 !

Fixes #6492,

2026-07-21 10:06:41 +03:00

6.7 KiB

Raw Blame History

Design: SQLite corruption/bloat safety, automatic recovery, and Admin Panel → Problems → Recovery

Status: Implemented · Owner: xet7 · Related (#6492): models/lib/recoveryPlan.js, models/lib/recoveryEventsJsonl.js, models/recoveryEvents.js, server/recovery.js, server/publications/recoveryReport.js, client/components/settings/adminReports.*, snap-src/bin/ferretdb-control, releases/ferretdb/*, and the FerretDB fork (internal/backends/sqlite/metadata/pool/opendb.go, internal/handler/handler.go).

When WeKan stores its data in FerretDB v1 (SQLite), the text data lives in wekan.sqlite. Attachments and avatars live on the filesystem, not in the database. This subsystem keeps that text data safe: it prevents the database from bloating, detects corruption, keeps a ready-to-use backup, can restore or re-migrate automatically, and shows the whole remediation history in Admin Panel → Problems → Recovery.

What each layer does

FerretDB (the database engine)

Automatic corruption detection. Every time a database file is opened, FerretDB runs SQLite's fast PRAGMA quick_check; anything but ok is logged prominently. Corruption cannot be repaired in place, so this only reports it — recovery is done by WeKan (restore/re-migrate).
Automatic bloat repair. On open, FerretDB VACUUMs a file whose free pages dominate it (≥ ~1 MiB and ≥ ¼ free), rebuilding it compactly. Content-preserving.
Bounded OpLog. local.oplog.rs is capped small (16 MiB) so the transient local.sqlite cannot bloat and drive CPU high.
Toggle: FERRETDB_SQLITE_AUTO_REPAIR=false.

WeKan startup (the launch scripts, files at rest)

Before FerretDB opens the files, every launch path (snap ferretdb-control, the bundled release start-wekan.sh, the Docker wekan-entrypoint.sh):

Backs up the text data. Copies wekan.sqlite* into a backup/ subfolder of the same data dir, keeping the previous generation under backup/prev. It only ever copies the live database — never moves or deletes it — and never copies attachments/avatars. Toggle: WEKAN_SQLITE_BACKUP=false.
Restores on request. If a restore is requested — the WEKAN_FORCE_RESTORE env or a RESTORE_REQUESTED marker file containing backup, prev or remigrate — it copies the chosen known-good backup into the live database (dropping the stale WAL side-files so the copy is used cleanly) and records the action. The backup copies are never deleted; the main wekan.sqlite is only ever overwritten, never removed. remigrate requests a re-migration of the text data from MongoDB (handled by migration-control); attachments/avatars on the filesystem are untouched.
Resets the transient OpLog (local.sqlite) so a bloated/corrupt OpLog cannot persist across a restart. Toggle: WEKAN_FERRETDB_RESET_OPLOG=false.

The recovery decision

models/lib/recoveryPlan.js (decideRecovery) is the pure, unit-tested brain: given the integrity result and what backups / MongoDB are available, it chooses the least-invasive recovery — latest good backup → previous backup → re-migrate → (else) manual. It never chooses anything destructive unless the database is known corrupt.

What users see during a recovery

Recovery must never look like a broken site. Two layers cover the whole window, on every FerretDB v1 platform (snap, bundled release, Docker):

In-app maintenance spinner (all platforms). When a recovery is in progress the server publishes a public status document — everyone, including logged-out users on the sign-in page, sees it — that drives a full-screen overlay with a spinner in both the app and sign-in layouts. The launch scripts write a RECOVERY_IN_PROGRESS marker when they restore/re-migrate; the server keeps the spinner up until it has health-probed the database (a real read), then clears the marker and hides the spinner — or, if it still cannot read, keeps the spinner and records that manual recovery is required. Admins can also toggle it for a server-initiated re-migration. Because the server owns clearing the marker, the spinner behaves identically on all platforms.
Static bridge page (before the app is up). For the brief window while a just-restored FerretDB comes back up and before Meteor can serve the client, the launch scripts serve a tiny standalone "recovering your data" page (HTTP 503) on the web port so users never hit a bare connection error: the snap reuses wekan-maintenance-page.mjs (with a recovery wording), and the release/Docker paths serve the portable releases/ferretdb/recovery-bridge.mjs. The bridge is time-bounded (WEKAN_RECOVERY_BRIDGE_SECONDS, default 20s) and is only a visual bridge — it can never block WeKan from starting, and it hands straight over to the in-app spinner above. It is skipped cleanly if its page file or the marker is absent.

Admin Panel → Problems → Recovery

The startup scripts append one JSON line per action to recovery-events.jsonl in the data dir. On startup the WeKan server imports the new lines into the recoveryEvents collection (server/recovery.js), and the Recovery report (server/publications/recoveryReport.js, admin-only) lists them newest-first with the same search + pagination as the other admin reports. Admins can also record a manual event with the recordRecoveryEvent method.

Event types include backup-created, corruption-detected, restore-backup, restore-prev, remigrate, bloat-repaired, integrity-ok, manual-required, each with a severity (info / warning / error).

Manual recovery

To force a restore on the next start, set WEKAN_FORCE_RESTORE=backup (or prev, or remigrate) in the environment, or create a RESTORE_REQUESTED file containing that word in the SQLite data dir, then restart WeKan. The action is recorded in the Recovery report.

Tests

tests/recoveryPlan.test.cjs — the decision logic (positive + negatives: never act on a healthy/unknown database; never restore a known-bad backup; nothing to restore → manual, non-destructive).
tests/recoveryEventsJsonl.test.cjs — the JSONL parser (skips junk, normalizes severity, bounds line size; never throws).
tests/recoveryReportQuery.test.cjs — the report search selector (escapes regex metacharacters).
tests/recoveryReportWiring.test.cjs — the Recovery report is wired and the publication/count/method are admin-gated.
tests/ferretdbTextDataBackup.test.cjs — the backup/restore scripts (critical negatives: never delete the live text data or a backup copy, never copy attachments/avatars).
FerretDB: opendb_test.go (corruption check + bloat VACUUM) and msg_replset_test.go (OpLog cap).

6.7 KiB Raw Blame History