Admin Panel / Problems / CPU usage could stay empty while FerretDB
pegged the machine: both WeKan and FerretDB measured only system-wide
CPU, and FerretDB at 170-300% (2-3 cores) is only ~40-75% system-wide on
a many-core host, below the high threshold.
Read FerretDB's new processCpuPercent from the throttle response and
surface it: high-CPU rows include it, and a new watch (a pure status
read that never slows FerretDB down, so it cannot make a slow board load
worse) records a start/end episode when FerretDB alone monopolises cores
even though system-wide CPU stays moderate. Tunable via
WEKAN_FERRETDB_WATCH_PERCENT / WEKAN_FERRETDB_PROC_HIGH_PERCENT.
Complete the high-CPU → FerretDB-slow-down → measure → report loop, and make it
robust when WeKan itself is CPU-starved or FerretDB is unresponsive.
- Adaptive feedback (server/lib/cpuMonitor.js): while CPU stays high WeKan now
INCREASES (doubles) the delay FerretDB adds between operations, up to a cap,
until system CPU drops below a headroom target (WEKAN_CPU_TARGET_PERCENT,
default 70) — enough CPU for other processes — then holds. Each escalation, the
headroom result, and the "even max delay not enough (load is elsewhere)" case
are logged.
- FerretDB self-regulates too: WeKan reads the throttle response's autoSlowDownMs,
hostCpuPercent and operationsSummary (what FerretDB has been doing) and logs
them — so the feature works even if WeKan is too starved to drive the loop.
- No flooding / no hammering (server/lib/ferretdbGovernor.js): every call has a
timeout (WEKAN_FERRETDB_TIMEOUT_MS); WeKan talks to FerretDB at most once per
WEKAN_FERRETDB_ASK_INTERVAL_MS (30s); a non-answer starts a cooldown
(WEKAN_FERRETDB_COOLDOWN_MS) instead of retrying. WeKan records WHEN FerretDB
became unresponsive and, on recovery, logs the full start -> end span plus
FerretDB's status and operations summary. "command not found" (plain MongoDB)
is still treated as permanently unsupported.
- i18n: cpuReportTitle ("CPU usage") translated to all 154 languages.
Guard tests (tests/cpuMonitorWiring.test.cjs) extended for escalation, rate-limit,
timeout/cooldown, outage span, and FerretDB self-regulation/summary. Docs
(docs/Features/CPU-usage/CPU-usage.md) and CHANGELOG updated.
FerretDB de-branded its custom command to the general name `throttle` (it is a
general-purpose MongoDB replacement, usable by any MongoDB-wire client, not only
WeKan). Update WeKan's caller to match: server/lib/ferretdbGovernor.js now sends
{ throttle: 1, ... } and the comments/guard test refer to the `throttle` command.
No behaviour change. WeKan-side names (ferretdbGovernor, slowDownFerretDb) are
WeKan's own and unchanged.
FerretDB is the other big CPU user on the same host. When WeKan's CPU monitor
detects a sustained high-CPU period it now asks FerretDB (via the custom
wekanThrottle command added to the bundled wekan/FerretDB v1 fork) what it is
doing and to slow down, and writes it to Admin Panel / Problems / CPU usage.
- server/lib/ferretdbGovernor.js: runs the wekanThrottle command over the raw
Mongo driver. slowDownFerretDb(slowDownMs, durationMs) asks FerretDB to pause a
few ms before each command for a short, self-expiring window and returns its
activity (commandsProcessed); resumeFerretDb() turns it off. Best-effort — on
plain MongoDB or an older FerretDB the command fails once and is never retried.
- server/lib/cpuMonitor.js: on 'start' asks FerretDB to slow down and logs a row
with FerretDB's reported activity and what it was asked; renews the
self-expiring throttle each sample while CPU stays high; on 'end' asks FerretDB
to resume and notes it. Tunable via WEKAN_FERRETDB_SLOWDOWN_MS.
Guard tests (cpuMonitorWiring) extended for the FerretDB governor. CHANGELOG +
docs/Features/CPU-usage/CPU-usage.md updated.
When CPU is high, the report now also records what automatic mitigation was
taken and whether it noticeably lowered CPU:
- Once the governor actually pauses a governed operation during a high-CPU
period, write ONE 'rate-limited' row: slowing down "<activity>" — pausing
200ms between steps to yield the CPU (also lowers FerretDB query load), with
the CPU% at slow-down start.
- Track the lowest CPU% seen after slowing down; the 'end' row reports the
effect: "slowed down <activity> (paused N times, Xs total). After slowing
down, CPU went A% -> B% — noticeably lower (about -Z points)" (or "not
noticeably lower"). When nothing was running to slow down, it says so.
Still at most three rows per episode (start, mitigation, end). The effect is a
correlation over the sampling window, reported as observed.
Also rename the design doc to docs/Features/CPU-usage/CPU-usage.md (CHANGELOG
link updated). Guard test extended for the mitigation logging.
WeKan and FerretDB on one machine can drive CPU high enough to starve other
software. Add a monitor, a governor, and a report. Design: docs/Features/CPU/CPU.md.
Monitor (server/lib/cpuMonitor.js): samples system-wide CPU% from os.cpus()
idle/total deltas on an interval (so it includes WeKan, FerretDB and every other
process). A pure, hysteresis state machine (models/lib/cpuHighTracker.js) turns
the samples into ONLY the START and END of each sustained high-CPU period — two
rows per episode, never a flood. Enters high after 3 consecutive samples >= 85%,
leaves after 3 consecutive < 70%. Each row carries duration, peak, system load,
and what WeKan was doing (setActivity label). Recorded to a new 'cpu' event
stream (server/lib/cpuLog.js).
Governor: cpuMonitor exports isHighCpu() and pauseIfBusy(), so long batch
operations await a short pause and yield the CPU only while the system is busy
(no cost when idle). Wired into the existing-file extension corrector. Tunable
via WEKAN_CPU_* env vars.
Report: 'cpu' added to EVENT_STREAMS; reuses the generic eventStreamReport
(stream="cpu") with search + pagination; a "CPU usage" item is added to the
Admin Panel / Problems side menu (cpuReportTitle i18n).
Tests: cpuHighTracker unit (start/end, duration/peak, no-flap hysteresis;
positive + negative) and cpuMonitorWiring source guards. CHANGELOG + design doc.
Lauri Ojansivu