mirror of
https://github.com/wekan/wekan.git
synced 2026-11-20 09:45:04 -05:00
A developer laptop with many programs open showed in Admin Panel / Problems / Memory usage: "RamBleed, high, high RAM usage started (>= 90%): RAM 16.1/30.9 GB (52%), swap 18.7/30.9 GB (60%)". Swap at or above 25% started a high-memory period by itself, made it high severity, and the text still said ">= 90%". Idle pages in swap while RAM is free are not pressure. - Swap now counts only while RAM is also at or above the low threshold (80%), and a period ends when RAM is back under it. - The start row says what started it: (RAM >= 90%) or (swap >= 25% with RAM >= 80%). Only RAM at 95% or more is high severity. - The RAM, CPU, disk and speed monitors named their rows like Hall of Fame entries (RamBleed, CpuBleed, DiskBleed, SlowBleed) with CWE-400. They are HighRamUsage, HighCpuUsage, HighDiskUsage and SlowRequest now, without a CWE, and stored rows are renamed once at startup. tests/ramHighTracker.test.cjs replays the laptop's numbers as a negative case; tests/resourceStreamNames.test.cjs pins the names. Thanks to xet7 !
119 lines
6.3 KiB
Markdown
119 lines
6.3 KiB
Markdown
# RAM-usage monitor and Admin Panel → Problems → Memory Usage
|
||
|
||
Status: **Implemented** (2026-10-17) · Owner: xet7 · Files:
|
||
`server/lib/ramMonitor.js`, `server/lib/ramLog.js` (through the shared
|
||
`server/lib/resourceLog.js`), `models/lib/ramHighTracker.js`, the `ram` stream in
|
||
`models/eventLog.js`, Admin Panel → Problems → **Memory Usage**
|
||
(`/admin/problems/ram`). Tests: `tests/ramHighTracker.test.cjs`,
|
||
`tests/resourceMonitorsWiring.test.cjs`.
|
||
|
||
Sibling of the [CPU-usage](CPU-usage.md) monitor. WeKan and FerretDB on the same
|
||
machine can use enough RAM (and push the machine into swap) to starve or crash other
|
||
software — and swapping makes everything slow. This subsystem **watches** system-wide
|
||
memory + swap usage and records only the **start and end** of each sustained
|
||
high-usage period to an **Admin Panel → Problems → RAM usage** report, so operators
|
||
can see when memory pressure began, how bad it got, and when it cleared — without a
|
||
flood of rows.
|
||
|
||
## 1. Requirements
|
||
|
||
1. While WeKan (and FerretDB alongside it) run, watch how much **RAM and swap** are
|
||
used.
|
||
2. Add an **Admin Panel → Problems → RAM usage** report.
|
||
3. Log **when high usage starts** and **when high usage ends** — only those two rows
|
||
per episode, so the report never floods.
|
||
4. Each row records how much **RAM** and **swap** were used (used / total and
|
||
percentages), the peak during the episode, its duration, and what WeKan/FerretDB
|
||
were doing at the time (the same coarse activity label the CPU monitor uses).
|
||
5. Best-effort and never throws; a no-op where the numbers are unavailable (e.g. a
|
||
sandbox with no `/proc/meminfo`).
|
||
|
||
## 2. Design
|
||
|
||
Mirrors the CPU-usage subsystem so the three Problems monitors (CPU, RAM, Disk) share
|
||
one shape, one event-stream mechanism, and one report template.
|
||
|
||
### Measurement — system-wide, includes FerretDB
|
||
|
||
`server/lib/ramMonitor.js` samples on an interval (`WEKAN_RAM_SAMPLE_INTERVAL_MS`,
|
||
default 5s):
|
||
|
||
- **RAM**: on Linux, `MemTotal` and `MemAvailable` from `/proc/meminfo`; used RAM% =
|
||
`(total − available) / total × 100`. `MemAvailable` counts the page cache the
|
||
kernel can drop as available - `os.freemem()` (MemFree) does not, and would show a
|
||
healthy server with a full cache as out of memory. Without `/proc/meminfo` (not
|
||
Linux, or an old kernel) the monitor uses `os.totalmem()` and `os.freemem()`. Measured at the OS level, so it includes
|
||
WeKan, FerretDB and every other process — the true "is the machine low on memory"
|
||
signal.
|
||
- **Swap**: on Linux, read `SwapTotal` and `SwapFree` from `/proc/meminfo`; used
|
||
swap% = `(SwapTotal − SwapFree) / SwapTotal × 100`. When `/proc/meminfo` is
|
||
unreadable or there is no swap, swap is reported as unavailable and only RAM drives
|
||
the episode. (Node's `process.memoryUsage()` — WeKan's own RSS — is recorded in the
|
||
detail for context but is **not** the trigger; the trigger is system-wide.)
|
||
|
||
### Start/end only — `models/lib/ramHighTracker.js`
|
||
|
||
A pure, unit-tested state machine (identical shape to `cpuHighTracker.js`) turns the
|
||
sample stream into just two events per episode, with **hysteresis** to prevent
|
||
flapping:
|
||
|
||
- enter the high state after `WEKAN_RAM_HIGH_SAMPLES` (3) consecutive samples at/above
|
||
`WEKAN_RAM_HIGH_PERCENT` (default 90% RAM used), **or** with swap at/above
|
||
`WEKAN_SWAP_HIGH_PERCENT` (default 25% swap used) **while RAM is also** at/above
|
||
`WEKAN_RAM_LOW_PERCENT` (default 80%);
|
||
- leave it after `WEKAN_RAM_LOW_SAMPLES` (3) consecutive samples with RAM below
|
||
`WEKAN_RAM_LOW_PERCENT`.
|
||
|
||
Swap alone is not memory pressure. A machine running many programs keeps their
|
||
idle pages in swap while plenty of RAM is free; a developer laptop at RAM 52% and
|
||
swap 60% used to be reported as "high RAM usage started (>= 90%)", high severity.
|
||
Swap now counts only when RAM is tight as well, and the start row says which of
|
||
the two started it: `(RAM >= 90%)` or `(swap >= 25% with RAM >= 80%)`.
|
||
- it tracks the peak RAM% and peak swap% seen during the episode for the end row.
|
||
|
||
### Report — Admin Panel → Problems → Memory Usage
|
||
|
||
Episodes are written to the `eventLog` collection on the **`ram`** stream (a new
|
||
`securityCategories`-style stream alongside `cpu`), and shown by the existing
|
||
`eventStreamReport` template (paginated, searchable, read-only). Two rows per episode:
|
||
|
||
- **start** (`detected`): `high RAM usage started (RAM >= 90%): RAM 7.4/8.0 GB (92%),
|
||
swap 0.6/2.0 GB (30%), load …, WeKan: <activity>`
|
||
- **end** (`remediated`): `high RAM usage ended after 42s (peak RAM 96%, peak swap
|
||
41%, back under 80%): RAM 5.1/8.0 GB (64%), swap 0.1/2.0 GB (5%)`
|
||
|
||
Severity: `high` when RAM is at ≥ 95% when the period starts, else `medium`. The
|
||
rows are named `HighRamUsage` (they were `RamBleed`, as if they were a
|
||
vulnerability; the old rows are renamed at startup) and carry no CWE: this is how
|
||
busy the machine is, not an attack. The end row is `info`, so it does not count as a new
|
||
problem.
|
||
|
||
Above the rows, a live line shows the current RAM use (percentage and used / total
|
||
GB) and swap when the machine has any, refreshed every 5 seconds. The menu entry and
|
||
the page title use the existing `memory-usage` translation, so every language
|
||
already has them.
|
||
|
||
## 3. Environment variables
|
||
|
||
| Variable | Default | Meaning |
|
||
|---|---|---|
|
||
| `WEKAN_RAM_MONITOR` | `true` | Master on/off. |
|
||
| `WEKAN_RAM_SAMPLE_INTERVAL_MS` | `5000` | Sampling interval. |
|
||
| `WEKAN_RAM_HIGH_PERCENT` | `90` | Enter high state at/above this RAM used %. |
|
||
| `WEKAN_RAM_LOW_PERCENT` | `80` | Leave high state below this RAM used %. |
|
||
| `WEKAN_SWAP_HIGH_PERCENT` | `25` | Also enter high state at/above this swap used %. |
|
||
| `WEKAN_RAM_HIGH_SAMPLES` | `3` | Consecutive high samples to enter. |
|
||
| `WEKAN_RAM_LOW_SAMPLES` | `3` | Consecutive low samples to leave. |
|
||
|
||
## 4. Notes
|
||
|
||
- The tracker and the `/proc/meminfo` parser are **pure functions** unit-tested
|
||
without Meteor, exactly like `cpuHighTracker.js` and its tests, so the enter/leave
|
||
hysteresis and the meminfo parsing (including missing/zero swap) are covered by
|
||
positive and negative tests.
|
||
- This is intentionally **observe-and-report only**: unlike CPU, WeKan cannot safely
|
||
"free" RAM on demand, so there is no governor here — the value is visibility (a
|
||
self-describing start→end record) so an operator can size the host or cap
|
||
concurrency. See [CPU-usage](CPU-usage.md) for the governed sibling and
|
||
[Disk-usage](Disk-usage.md) for the disk sibling.
|