Files
wekan/docs/Features/Admin-Panel/Problems/RAM-usage.md
T
Lauri Ojansivu cfd9a62466 Swap alone is not high RAM usage, and resource monitors are not named like vulnerabilities.
A developer laptop with many programs open showed in Admin Panel /
Problems / Memory usage: "RamBleed, high, high RAM usage started
(>= 90%): RAM 16.1/30.9 GB (52%), swap 18.7/30.9 GB (60%)". Swap at or
above 25% started a high-memory period by itself, made it high severity,
and the text still said ">= 90%". Idle pages in swap while RAM is free
are not pressure.

- Swap now counts only while RAM is also at or above the low threshold
  (80%), and a period ends when RAM is back under it.
- The start row says what started it: (RAM >= 90%) or (swap >= 25% with
  RAM >= 80%). Only RAM at 95% or more is high severity.
- The RAM, CPU, disk and speed monitors named their rows like Hall of
  Fame entries (RamBleed, CpuBleed, DiskBleed, SlowBleed) with CWE-400.
  They are HighRamUsage, HighCpuUsage, HighDiskUsage and SlowRequest now,
  without a CWE, and stored rows are renamed once at startup.

tests/ramHighTracker.test.cjs replays the laptop's numbers as a
negative case; tests/resourceStreamNames.test.cjs pins the names.

Thanks to xet7 !
2026-11-19 00:21:33 +02:00

119 lines
6.3 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# RAM-usage monitor and Admin Panel → Problems → Memory Usage
Status: **Implemented** (2026-10-17) · Owner: xet7 · Files:
`server/lib/ramMonitor.js`, `server/lib/ramLog.js` (through the shared
`server/lib/resourceLog.js`), `models/lib/ramHighTracker.js`, the `ram` stream in
`models/eventLog.js`, Admin Panel → Problems → **Memory Usage**
(`/admin/problems/ram`). Tests: `tests/ramHighTracker.test.cjs`,
`tests/resourceMonitorsWiring.test.cjs`.
Sibling of the [CPU-usage](CPU-usage.md) monitor. WeKan and FerretDB on the same
machine can use enough RAM (and push the machine into swap) to starve or crash other
software — and swapping makes everything slow. This subsystem **watches** system-wide
memory + swap usage and records only the **start and end** of each sustained
high-usage period to an **Admin Panel → Problems → RAM usage** report, so operators
can see when memory pressure began, how bad it got, and when it cleared — without a
flood of rows.
## 1. Requirements
1. While WeKan (and FerretDB alongside it) run, watch how much **RAM and swap** are
used.
2. Add an **Admin Panel → Problems → RAM usage** report.
3. Log **when high usage starts** and **when high usage ends** — only those two rows
per episode, so the report never floods.
4. Each row records how much **RAM** and **swap** were used (used / total and
percentages), the peak during the episode, its duration, and what WeKan/FerretDB
were doing at the time (the same coarse activity label the CPU monitor uses).
5. Best-effort and never throws; a no-op where the numbers are unavailable (e.g. a
sandbox with no `/proc/meminfo`).
## 2. Design
Mirrors the CPU-usage subsystem so the three Problems monitors (CPU, RAM, Disk) share
one shape, one event-stream mechanism, and one report template.
### Measurement — system-wide, includes FerretDB
`server/lib/ramMonitor.js` samples on an interval (`WEKAN_RAM_SAMPLE_INTERVAL_MS`,
default 5s):
- **RAM**: on Linux, `MemTotal` and `MemAvailable` from `/proc/meminfo`; used RAM% =
`(total − available) / total × 100`. `MemAvailable` counts the page cache the
kernel can drop as available - `os.freemem()` (MemFree) does not, and would show a
healthy server with a full cache as out of memory. Without `/proc/meminfo` (not
Linux, or an old kernel) the monitor uses `os.totalmem()` and `os.freemem()`. Measured at the OS level, so it includes
WeKan, FerretDB and every other process — the true "is the machine low on memory"
signal.
- **Swap**: on Linux, read `SwapTotal` and `SwapFree` from `/proc/meminfo`; used
swap% = `(SwapTotal − SwapFree) / SwapTotal × 100`. When `/proc/meminfo` is
unreadable or there is no swap, swap is reported as unavailable and only RAM drives
the episode. (Node's `process.memoryUsage()` — WeKan's own RSS — is recorded in the
detail for context but is **not** the trigger; the trigger is system-wide.)
### Start/end only — `models/lib/ramHighTracker.js`
A pure, unit-tested state machine (identical shape to `cpuHighTracker.js`) turns the
sample stream into just two events per episode, with **hysteresis** to prevent
flapping:
- enter the high state after `WEKAN_RAM_HIGH_SAMPLES` (3) consecutive samples at/above
`WEKAN_RAM_HIGH_PERCENT` (default 90% RAM used), **or** with swap at/above
`WEKAN_SWAP_HIGH_PERCENT` (default 25% swap used) **while RAM is also** at/above
`WEKAN_RAM_LOW_PERCENT` (default 80%);
- leave it after `WEKAN_RAM_LOW_SAMPLES` (3) consecutive samples with RAM below
`WEKAN_RAM_LOW_PERCENT`.
Swap alone is not memory pressure. A machine running many programs keeps their
idle pages in swap while plenty of RAM is free; a developer laptop at RAM 52% and
swap 60% used to be reported as "high RAM usage started (>= 90%)", high severity.
Swap now counts only when RAM is tight as well, and the start row says which of
the two started it: `(RAM >= 90%)` or `(swap >= 25% with RAM >= 80%)`.
- it tracks the peak RAM% and peak swap% seen during the episode for the end row.
### Report — Admin Panel → Problems → Memory Usage
Episodes are written to the `eventLog` collection on the **`ram`** stream (a new
`securityCategories`-style stream alongside `cpu`), and shown by the existing
`eventStreamReport` template (paginated, searchable, read-only). Two rows per episode:
- **start** (`detected`): `high RAM usage started (RAM >= 90%): RAM 7.4/8.0 GB (92%),
swap 0.6/2.0 GB (30%), load …, WeKan: <activity>`
- **end** (`remediated`): `high RAM usage ended after 42s (peak RAM 96%, peak swap
41%, back under 80%): RAM 5.1/8.0 GB (64%), swap 0.1/2.0 GB (5%)`
Severity: `high` when RAM is at ≥ 95% when the period starts, else `medium`. The
rows are named `HighRamUsage` (they were `RamBleed`, as if they were a
vulnerability; the old rows are renamed at startup) and carry no CWE: this is how
busy the machine is, not an attack. The end row is `info`, so it does not count as a new
problem.
Above the rows, a live line shows the current RAM use (percentage and used / total
GB) and swap when the machine has any, refreshed every 5 seconds. The menu entry and
the page title use the existing `memory-usage` translation, so every language
already has them.
## 3. Environment variables
| Variable | Default | Meaning |
|---|---|---|
| `WEKAN_RAM_MONITOR` | `true` | Master on/off. |
| `WEKAN_RAM_SAMPLE_INTERVAL_MS` | `5000` | Sampling interval. |
| `WEKAN_RAM_HIGH_PERCENT` | `90` | Enter high state at/above this RAM used %. |
| `WEKAN_RAM_LOW_PERCENT` | `80` | Leave high state below this RAM used %. |
| `WEKAN_SWAP_HIGH_PERCENT` | `25` | Also enter high state at/above this swap used %. |
| `WEKAN_RAM_HIGH_SAMPLES` | `3` | Consecutive high samples to enter. |
| `WEKAN_RAM_LOW_SAMPLES` | `3` | Consecutive low samples to leave. |
## 4. Notes
- The tracker and the `/proc/meminfo` parser are **pure functions** unit-tested
without Meteor, exactly like `cpuHighTracker.js` and its tests, so the enter/leave
hysteresis and the meminfo parsing (including missing/zero swap) are covered by
positive and negative tests.
- This is intentionally **observe-and-report only**: unlike CPU, WeKan cannot safely
"free" RAM on demand, so there is no governor here — the value is visibility (a
self-describing start→end record) so an operator can size the host or cap
concurrency. See [CPU-usage](CPU-usage.md) for the governed sibling and
[Disk-usage](Disk-usage.md) for the disk sibling.