mirror of
https://github.com/wekan/wekan.git
synced 2026-11-20 21:45:01 -05:00
A feature's guide now sits under the menu it is used from, read from the templates rather than from where somebody once put it: Header-Bar, All-Boards, Pages, Board-Header (search and every Board View), Board-Sidebar (Board Settings, Filter, Members, Labels, Archive), Rules, Swimlanes, Lists, Cards, Member-Settings, Admin-Panel and Login-Page. Topics used from several menus (editor, history, theme, translations, email, troubleshooting) are in General. - Each area has a README mapping its menu entries to their guides and to the template or server module that implements them. Board-Header/README.md lists all 40 Board View entries with the template boardBody.jade renders; five views (Table, Calendar Mode, Time, Timeline, Map) have no guide of their own yet and say so. - Board-View.md listed 25 views while BOARD_VIEWS has 40; it now lists all of them with their English labels. - Pages keep their file names. Where a released changelog links an old path, a "# Moved" page stays there and points to the new one; test-menu.sh skips folders holding only those. - Undo was filed under Login; it is General/Undo now. - Every link in docs, code comments, scripts and tests follows the move; docsLinksResolve checks them all. Tests that read a doc follow it to its new place. boardViewSettings now fails when Board-View.md or Board-Header/README.md differs from BOARD_VIEWS, or names a template boardBody.jade does not render. Thanks to xet7 !
184 lines
11 KiB
Markdown
184 lines
11 KiB
Markdown
# Design: CPU-usage monitor, governor, and Admin Panel → Problems report
|
|
|
|
> **This page uses the shared [Table Page](../../General/Table.md) design.**
|
|
> The layout, search, pagination, column spec and per-page data loading are defined
|
|
> there and are not repeated here.
|
|
|
|
Status: **Implemented** · Owner: xet7 · Related: `server/lib/cpuMonitor.js`,
|
|
`server/lib/cpuLog.js`, `models/lib/cpuHighTracker.js`, `models/eventLog.js`,
|
|
Admin Panel → Problems.
|
|
|
|
WeKan and FerretDB running on the same machine can drive CPU high enough to starve
|
|
other software. This subsystem (1) **watches** system-wide CPU and reports sustained
|
|
high-CPU periods, and (2) lets long operations **slow down** and yield when the
|
|
machine is already busy.
|
|
|
|
The low-load maintenance scheduler and rolling statistics shown in Problems →
|
|
Speed are specified in
|
|
[Verified FerretDB SQLite recovery](../../../Databases/FerretDB/1/Verified-Recovery.md#low-load-scheduling-and-cpu-statistics).
|
|
CPU-intensive snapshots, checksums, migration comparisons and history audits must
|
|
use that scheduler; corruption recovery needed to make startup safe is the urgent
|
|
exception.
|
|
|
|
## 1. Requirements
|
|
|
|
1. While WeKan (and FerretDB alongside it) run, watch that CPU usage does not go too
|
|
high; if it does, **slow down** — put pauses between operations so other software
|
|
can still run.
|
|
2. Add an **Admin Panel → Problems → CPU usage** report listing high-CPU periods and
|
|
what WeKan/FerretDB were doing at the time.
|
|
3. Record **only the START and END** of each high-CPU period, so the report never
|
|
floods with rows.
|
|
4. When CPU is high, also write **what automatic mitigation was taken** (e.g. slowing
|
|
down the current WeKan/FerretDB operation) and **whether it helped noticeably** —
|
|
i.e. when WeKan/FerretDB pauses, does CPU usage actually go lower.
|
|
|
|
## 2. Design
|
|
|
|
### Measurement — system-wide, includes FerretDB
|
|
|
|
`server/lib/cpuMonitor.js` samples `os.cpus()` idle/total jiffies on an interval
|
|
(`WEKAN_CPU_SAMPLE_INTERVAL_MS`, default 5s) and computes the system CPU% since the
|
|
last sample. Because it is measured across all cores at the OS level, it includes
|
|
WeKan, FerretDB, and every other process — the true "is the machine overloaded"
|
|
signal. `os.loadavg()` and the core count are recorded alongside.
|
|
|
|
### Start/end only — `models/lib/cpuHighTracker.js`
|
|
|
|
A pure, unit-tested state machine turns the sample stream into just two events per
|
|
episode. **Hysteresis** prevents flapping: it enters the high state only after
|
|
`WEKAN_CPU_HIGH_SAMPLES` (3) consecutive samples at/above `WEKAN_CPU_HIGH_PERCENT`
|
|
(85%), and leaves it only after `WEKAN_CPU_LOW_SAMPLES` (3) consecutive samples below
|
|
`WEKAN_CPU_LOW_PERCENT` (70%). On enter it records one `start` row; on leave, one
|
|
`end` row with the **duration** and **peak**.
|
|
|
|
### What WeKan was doing
|
|
|
|
The `end`/`start` detail carries `system CPU X%, load a/b/c, N cores` and, when set,
|
|
a `WeKan: <activity>` label. Long operations call `cpuMonitor.setActivity('…')`
|
|
around their work (e.g. the existing-file extension corrector sets "correcting file
|
|
extensions"), so the report shows what was running during the spike.
|
|
|
|
### Governor — slow down when busy
|
|
|
|
`cpuMonitor` exports `isHighCpu()` and `pauseIfBusy(ms)`. `pauseIfBusy()` awaits a
|
|
short delay (`WEKAN_CPU_PAUSE_MS`, default 200ms) **only** while the system is in a
|
|
high-CPU period, and resolves immediately otherwise. Batch loops `await
|
|
pauseIfBusy()` between items so they yield the CPU under load without slowing the
|
|
common (idle) case. Wired into the file-extension corrector; the same helper can be
|
|
added to other batch jobs (migrations, bulk moves).
|
|
|
|
### Mitigation logging + effectiveness
|
|
|
|
During a high-CPU period, once the governor has actually paused a governed operation,
|
|
the monitor writes **one** `rate-limited` row saying what mitigation was taken —
|
|
`slowing down "<activity>" — pausing 200ms between steps to yield the CPU (this also
|
|
lowers FerretDB query load)` — and remembers the CPU% at that moment. It then tracks
|
|
the lowest CPU% seen afterwards. The `end` row reports the **effect**: `slowed down
|
|
"<activity>" (paused N times, Xs total). After slowing down, CPU went A% → B% —
|
|
noticeably lower (about -Z points)` (or `not noticeably lower`). When no governed
|
|
operation was running to slow down, the `end` row says so honestly. This keeps each
|
|
episode to at most three rows: **start**, **mitigation taken**, **end (with effect)**.
|
|
|
|
The effect is a correlation over the sampling window (a short pause on a 5s window is
|
|
a coarse signal), reported as observed — not a controlled measurement.
|
|
|
|
### Asking FerretDB to slow down (cross-process)
|
|
|
|
FerretDB is the other big CPU user on the same host, and WeKan cannot pause
|
|
FerretDB's internal work from the outside — so the **wekan/FerretDB v1 fork adds a
|
|
general `throttle` command** (`FerretDB/internal/handler/`). When WeKan detects
|
|
high CPU it calls it (`server/lib/ferretdbGovernor.js`), which:
|
|
|
|
1. **asks what FerretDB is doing** — the response includes `commandsProcessed`, a
|
|
running count of commands handled (higher = busier), logged as FerretDB's activity;
|
|
2. **asks FerretDB to slow down** — for `durationMs`, FerretDB pauses `slowDownMs`
|
|
before every command in its dispatch path, lowering its CPU use and yielding to
|
|
other software.
|
|
|
|
**Adaptive feedback loop (increasing delays until there is headroom).** Because
|
|
only WeKan measures the host CPU, WeKan drives the escalation and FerretDB applies
|
|
it. The first request uses a small delay (`WEKAN_FERRETDB_SLOWDOWN_MS`, default 5ms).
|
|
Then, **each sample while the system CPU is still at/above the headroom target**
|
|
(`WEKAN_CPU_TARGET_PERCENT`, default = the leave-high threshold 70%), WeKan
|
|
**doubles** the delay FerretDB adds between operations — 5 → 10 → 20 → … up to
|
|
`WEKAN_FERRETDB_SLOWDOWN_MAX_MS` (default 200ms). As soon as CPU **drops below the
|
|
target** (enough CPU is free for other processes), WeKan stops escalating and just
|
|
**holds** the current delay. Every step is written to the CPU usage log:
|
|
|
|
- each increase — `CPU still X% (target < 70%): increased FerretDB slow-down A → B ms`;
|
|
- success — `CPU dropped to X% (< target 70%) after slowing FerretDB to B ms/op —
|
|
enough CPU is now free for other processes`;
|
|
- the honest failure case — if even the maximum delay does not free enough CPU, one
|
|
`slowing FerretDB did not free enough CPU — the load is (partly) elsewhere` line, so
|
|
the admin knows FerretDB was not the culprit.
|
|
|
|
The throttle **self-expires** on the FerretDB side (max 5 min), so a WeKan crash can
|
|
never leave FerretDB permanently slow. On plain MongoDB or an older FerretDB without
|
|
the command, the call fails once and is never retried (best-effort, no effect).
|
|
|
|
**FerretDB self-regulates too (so it works even if WeKan is starved).** WeKan can
|
|
only measure the host CPU and send requests if it has CPU time itself — which it may
|
|
not, under load. So FerretDB **also** samples the host CPU (`/proc/stat`) on its own
|
|
and, when it is too high, adds an increasing delay before each command until CPU
|
|
recovers, independent of WeKan. The delay FerretDB applies is `max(WeKan-requested,
|
|
FerretDB-self-regulated)`. WeKan reads FerretDB's `autoSlowDownMs` and `hostCpuPercent`
|
|
from the throttle response and includes them in the log, and FerretDB also returns an
|
|
`operationsSummary` (its busiest commands, e.g. `find=12000, update=340, …`) so the
|
|
log shows what FerretDB was doing.
|
|
|
|
**WeKan does not flood the log or hammer FerretDB.** WeKan still samples CPU locally
|
|
every `WEKAN_CPU_SAMPLE_INTERVAL_MS`, but it only **talks to FerretDB at most once per
|
|
`WEKAN_FERRETDB_ASK_INTERVAL_MS`** (default 30 s) — FerretDB self-regulates in
|
|
between. Every call has a **timeout** (`WEKAN_FERRETDB_TIMEOUT_MS`, default 2 s); if
|
|
FerretDB does not answer, WeKan records the moment it **became unresponsive**, then
|
|
**backs off** for `WEKAN_FERRETDB_COOLDOWN_MS` (default 60 s) instead of retrying, so
|
|
it lets FerretDB recover. When FerretDB answers again, WeKan writes one recovery row
|
|
with the full **unresponsive span (start → end)**, the current CPU, and FerretDB's
|
|
status + operations summary.
|
|
|
|
### Report
|
|
|
|
The `cpu` value is added to `EVENT_STREAMS` (`models/eventLog.js`); the report reuses
|
|
the generic `eventStreamReport` (`stream="cpu"`) and its `eventLogPage` method, so it
|
|
gets search + pagination like Security/Speed/Tests. A "CPU usage" item is added to
|
|
the Admin Panel → Problems side menu.
|
|
|
|
The monitor also retains a bounded rolling sample window (720 samples by default)
|
|
and exposes minimum, average, maximum, sample count and the time of the lowest sample
|
|
in both Problems → Speed and Problems → CPU usage. `runWhenCpuLow()` grants one heavy
|
|
maintenance lease only after three consecutive below-threshold samples; another job
|
|
waits, and a job whose window never arrives is deferred. The current activity label
|
|
and existing governor remain active throughout the lease.
|
|
|
|
## 3. Tuning (environment variables)
|
|
|
|
| Variable | Default | Meaning |
|
|
| --- | --- | --- |
|
|
| `WEKAN_CPU_MONITOR` | `true` | set `false` to disable the monitor entirely |
|
|
| `WEKAN_CPU_SAMPLE_INTERVAL_MS` | `5000` | sampling interval |
|
|
| `WEKAN_CPU_HIGH_PERCENT` | `85` | enter-high threshold |
|
|
| `WEKAN_CPU_LOW_PERCENT` | `70` | leave-high threshold (hysteresis) |
|
|
| `WEKAN_CPU_HIGH_SAMPLES` | `3` | consecutive high samples before a `start` |
|
|
| `WEKAN_CPU_LOW_SAMPLES` | `3` | consecutive low samples before an `end` |
|
|
| `WEKAN_CPU_PAUSE_MS` | `200` | governor pause length while busy |
|
|
| `WEKAN_CPU_TARGET_PERCENT` | `70` | headroom target — escalate FerretDB's delay until CPU drops below this |
|
|
| `WEKAN_FERRETDB_SLOWDOWN_MS` | `5` | first FerretDB per-operation delay |
|
|
| `WEKAN_FERRETDB_SLOWDOWN_MAX_MS` | `200` | cap on the escalated FerretDB delay |
|
|
| `WEKAN_FERRETDB_ASK_INTERVAL_MS` | `30000` | min gap between calls to FerretDB (no log flooding) |
|
|
| `WEKAN_FERRETDB_TIMEOUT_MS` | `2000` | per-call timeout when FerretDB is unresponsive |
|
|
| `WEKAN_FERRETDB_COOLDOWN_MS` | `60000` | back-off after a failed/timed-out call, letting FerretDB recover |
|
|
|
|
FerretDB's own self-regulation is tuned on the FerretDB side with
|
|
`FERRETDB_CPU_SELF_REGULATE` (default on), `FERRETDB_CPU_HIGH_PERCENT` (85),
|
|
`FERRETDB_CPU_TARGET_PERCENT` (70), `FERRETDB_CPU_SLOWDOWN_MS` (5),
|
|
`FERRETDB_CPU_SLOWDOWN_MAX_MS` (200) and `FERRETDB_CPU_INTERVAL_MS` (5000).
|
|
|
|
## 4. Tests
|
|
|
|
- `tests/cpuHighTracker.test.cjs` — start/end thresholds, duration + peak, and the
|
|
no-flap hysteresis (positive + negative).
|
|
- `tests/cpuMonitorWiring.test.cjs` — source guards: system-CPU sampling, start/end
|
|
only, the `cpu` stream, the governor (`isHighCpu`/`pauseIfBusy` + batch usage), and
|
|
the Admin Panel report wiring.
|