wekan/docs/Features/Admin-Panel/Problems/CPU-usage.md

# Design: CPU-usage monitor, governor, and Admin Panel → Problems report

Status: **Implemented** · Owner: xet7 · Related: `server/lib/cpuMonitor.js`,
`server/lib/cpuLog.js`, `models/lib/cpuHighTracker.js`, `models/eventLog.js`,
Admin Panel → Problems.

WeKan and FerretDB running on the same machine can drive CPU high enough to starve
other software. This subsystem (1) **watches** system-wide CPU and reports sustained
high-CPU periods, and (2) lets long operations **slow down** and yield when the
machine is already busy.

## 1. Requirements

1. While WeKan (and FerretDB alongside it) run, watch that CPU usage does not go too
   high; if it does, **slow down** — put pauses between operations so other software
   can still run.
2. Add an **Admin Panel → Problems → CPU usage** report listing high-CPU periods and
   what WeKan/FerretDB were doing at the time.
3. Record **only the START and END** of each high-CPU period, so the report never
   floods with rows.
4. When CPU is high, also write **what automatic mitigation was taken** (e.g. slowing
   down the current WeKan/FerretDB operation) and **whether it helped noticeably** —
   i.e. when WeKan/FerretDB pauses, does CPU usage actually go lower.

## 2. Design

### Measurement — system-wide, includes FerretDB

`server/lib/cpuMonitor.js` samples `os.cpus()` idle/total jiffies on an interval
(`WEKAN_CPU_SAMPLE_INTERVAL_MS`, default 5s) and computes the system CPU% since the
last sample. Because it is measured across all cores at the OS level, it includes
WeKan, FerretDB, and every other process — the true "is the machine overloaded"
signal. `os.loadavg()` and the core count are recorded alongside.

### Start/end only — `models/lib/cpuHighTracker.js`

A pure, unit-tested state machine turns the sample stream into just two events per
episode. **Hysteresis** prevents flapping: it enters the high state only after
`WEKAN_CPU_HIGH_SAMPLES` (3) consecutive samples at/above `WEKAN_CPU_HIGH_PERCENT`
(85%), and leaves it only after `WEKAN_CPU_LOW_SAMPLES` (3) consecutive samples below
`WEKAN_CPU_LOW_PERCENT` (70%). On enter it records one `start` row; on leave, one
`end` row with the **duration** and **peak**.

### What WeKan was doing

The `end`/`start` detail carries `system CPU X%, load a/b/c, N cores` and, when set,
a `WeKan: <activity>` label. Long operations call `cpuMonitor.setActivity('…')`
around their work (e.g. the existing-file extension corrector sets "correcting file
extensions"), so the report shows what was running during the spike.

### Governor — slow down when busy

`cpuMonitor` exports `isHighCpu()` and `pauseIfBusy(ms)`. `pauseIfBusy()` awaits a
short delay (`WEKAN_CPU_PAUSE_MS`, default 200ms) **only** while the system is in a
high-CPU period, and resolves immediately otherwise. Batch loops `await
pauseIfBusy()` between items so they yield the CPU under load without slowing the
common (idle) case. Wired into the file-extension corrector; the same helper can be
added to other batch jobs (migrations, bulk moves).

### Mitigation logging + effectiveness

During a high-CPU period, once the governor has actually paused a governed operation,
the monitor writes **one** `rate-limited` row saying what mitigation was taken —
`slowing down "<activity>" — pausing 200ms between steps to yield the CPU (this also
lowers FerretDB query load)` — and remembers the CPU% at that moment. It then tracks
the lowest CPU% seen afterwards. The `end` row reports the **effect**: `slowed down
"<activity>" (paused N times, Xs total). After slowing down, CPU went A% → B% —
noticeably lower (about -Z points)` (or `not noticeably lower`). When no governed
operation was running to slow down, the `end` row says so honestly. This keeps each
episode to at most three rows: **start**, **mitigation taken**, **end (with effect)**.

The effect is a correlation over the sampling window (a short pause on a 5s window is
a coarse signal), reported as observed — not a controlled measurement.

### Asking FerretDB to slow down (cross-process)

FerretDB is the other big CPU user on the same host, and WeKan cannot pause
FerretDB's internal work from the outside — so the **wekan/FerretDB v1 fork adds a
general `throttle` command** (`FerretDB/internal/handler/`). When WeKan detects
high CPU it calls it (`server/lib/ferretdbGovernor.js`), which:

1. **asks what FerretDB is doing** — the response includes `commandsProcessed`, a
   running count of commands handled (higher = busier), logged as FerretDB's activity;
2. **asks FerretDB to slow down** — for `durationMs`, FerretDB pauses `slowDownMs`
   before every command in its dispatch path, lowering its CPU use and yielding to
   other software.

**Adaptive feedback loop (increasing delays until there is headroom).** Because
only WeKan measures the host CPU, WeKan drives the escalation and FerretDB applies
it. The first request uses a small delay (`WEKAN_FERRETDB_SLOWDOWN_MS`, default 5ms).
Then, **each sample while the system CPU is still at/above the headroom target**
(`WEKAN_CPU_TARGET_PERCENT`, default = the leave-high threshold 70%), WeKan
**doubles** the delay FerretDB adds between operations — 5 → 10 → 20 → … up to
`WEKAN_FERRETDB_SLOWDOWN_MAX_MS` (default 200ms). As soon as CPU **drops below the
target** (enough CPU is free for other processes), WeKan stops escalating and just
**holds** the current delay. Every step is written to the CPU usage log:

- each increase — `CPU still X% (target < 70%): increased FerretDB slow-down A → B ms`;
- success — `CPU dropped to X% (< target 70%) after slowing FerretDB to B ms/op —
  enough CPU is now free for other processes`;
- the honest failure case — if even the maximum delay does not free enough CPU, one
  `slowing FerretDB did not free enough CPU — the load is (partly) elsewhere` line, so
  the admin knows FerretDB was not the culprit.

The throttle **self-expires** on the FerretDB side (max 5 min), so a WeKan crash can
never leave FerretDB permanently slow. On plain MongoDB or an older FerretDB without
the command, the call fails once and is never retried (best-effort, no effect).

**FerretDB self-regulates too (so it works even if WeKan is starved).** WeKan can
only measure the host CPU and send requests if it has CPU time itself — which it may
not, under load. So FerretDB **also** samples the host CPU (`/proc/stat`) on its own
and, when it is too high, adds an increasing delay before each command until CPU
recovers, independent of WeKan. The delay FerretDB applies is `max(WeKan-requested,
FerretDB-self-regulated)`. WeKan reads FerretDB's `autoSlowDownMs` and `hostCpuPercent`
from the throttle response and includes them in the log, and FerretDB also returns an
`operationsSummary` (its busiest commands, e.g. `find=12000, update=340, …`) so the
log shows what FerretDB was doing.

**WeKan does not flood the log or hammer FerretDB.** WeKan still samples CPU locally
every `WEKAN_CPU_SAMPLE_INTERVAL_MS`, but it only **talks to FerretDB at most once per
`WEKAN_FERRETDB_ASK_INTERVAL_MS`** (default 30 s) — FerretDB self-regulates in
between. Every call has a **timeout** (`WEKAN_FERRETDB_TIMEOUT_MS`, default 2 s); if
FerretDB does not answer, WeKan records the moment it **became unresponsive**, then
**backs off** for `WEKAN_FERRETDB_COOLDOWN_MS` (default 60 s) instead of retrying, so
it lets FerretDB recover. When FerretDB answers again, WeKan writes one recovery row
with the full **unresponsive span (start → end)**, the current CPU, and FerretDB's
status + operations summary.

### Report

The `cpu` value is added to `EVENT_STREAMS` (`models/eventLog.js`); the report reuses
the generic `eventStreamReport` (`stream="cpu"`) and its `eventLogPage` method, so it
gets search + pagination like Security/Speed/Tests. A "CPU usage" item is added to
the Admin Panel → Problems side menu.

## 3. Tuning (environment variables)

| Variable | Default | Meaning |
| --- | --- | --- |
| `WEKAN_CPU_MONITOR` | `true` | set `false` to disable the monitor entirely |
| `WEKAN_CPU_SAMPLE_INTERVAL_MS` | `5000` | sampling interval |
| `WEKAN_CPU_HIGH_PERCENT` | `85` | enter-high threshold |
| `WEKAN_CPU_LOW_PERCENT` | `70` | leave-high threshold (hysteresis) |
| `WEKAN_CPU_HIGH_SAMPLES` | `3` | consecutive high samples before a `start` |
| `WEKAN_CPU_LOW_SAMPLES` | `3` | consecutive low samples before an `end` |
| `WEKAN_CPU_PAUSE_MS` | `200` | governor pause length while busy |
| `WEKAN_CPU_TARGET_PERCENT` | `70` | headroom target — escalate FerretDB's delay until CPU drops below this |
| `WEKAN_FERRETDB_SLOWDOWN_MS` | `5` | first FerretDB per-operation delay |
| `WEKAN_FERRETDB_SLOWDOWN_MAX_MS` | `200` | cap on the escalated FerretDB delay |
| `WEKAN_FERRETDB_ASK_INTERVAL_MS` | `30000` | min gap between calls to FerretDB (no log flooding) |
| `WEKAN_FERRETDB_TIMEOUT_MS` | `2000` | per-call timeout when FerretDB is unresponsive |
| `WEKAN_FERRETDB_COOLDOWN_MS` | `60000` | back-off after a failed/timed-out call, letting FerretDB recover |

FerretDB's own self-regulation is tuned on the FerretDB side with
`FERRETDB_CPU_SELF_REGULATE` (default on), `FERRETDB_CPU_HIGH_PERCENT` (85),
`FERRETDB_CPU_TARGET_PERCENT` (70), `FERRETDB_CPU_SLOWDOWN_MS` (5),
`FERRETDB_CPU_SLOWDOWN_MAX_MS` (200) and `FERRETDB_CPU_INTERVAL_MS` (5000).

## 4. Tests

- `tests/cpuHighTracker.test.cjs` — start/end thresholds, duration + peak, and the
  no-flap hysteresis (positive + negative).
- `tests/cpuMonitorWiring.test.cjs` — source guards: system-CPU sampling, start/end
  only, the `cpu` stream, the governor (`isHighCpu`/`pauseIfBusy` + batch usage), and
  the Admin Panel report wiring.