The reported crash series (.tools/crash) was invisible from inside WeKan:
MongoDB aborting on every checkpoint, restarting, aborting again, with
every query in between slow - and the admin found it in the container log.
Admin Panel / Problems / Database problems now gets a row for each thing
WeKan can measure from its side of the socket, from a probe that runs 30 s
after startup and every five minutes (server/lib/databaseHealth.js, the
pure decisions in models/lib/databaseHealth.js):
- db.restart: serverStatus.uptime went down while WeKan kept running - the
database process restarted; points at the fatal-assertion/fsync cause and
the storage page.
- db.network-filesystem: getCmdLineOpts' dbPath, when it is a directory
this process can see (snap, embedded database, shared volume), is on a
CIFS/SMB, NFS or FUSE filesystem (fs.statfs f_type).
- db.slow-storage: serverStatus.opLatencies reads averaged over 100 ms
across the interval, on a real number of reads - the storage latency
behind the "Slow query" lines.
- db.disk-space: dbStats fsUsedSize/fsTotalSize below 5% or 512 MiB free -
the real "no space left on device", before it happens.
- db.index-created: the one remediation WeKan can do itself. ensureIndex
(server/lib/mongoStartup.js) now reports an index it created on a
collection that already held documents - until then every query on it
was a collection scan - as found and fixed; a new, empty collection
getting its indexes at first start stays silent.
All rows go through the existing `database` stream (recordDatabaseHealth in
server/lib/databaseProblems.js: same shape, same per-minute quieting per
type), so they land on the same page as what the database itself says.
FerretDB answers only some of these commands; each step is independent and
what it cannot answer is skipped. Nothing here can throw into startup or
sits on a request path.
tests/databaseHealth.test.cjs drives every decision (including the
reported case: 519 MB used, 10 GiB free is NOT a space problem) and pins
the wiring.
Thanks to xet7 !
The security event log said what a guard DID - a request blocked, a filename
sanitized - which answers "is WeKan defending itself" and not the question an
admin asks next: who did that, and from where. It also could not tell a browser
that got confused from somebody working through the Hall of Fame one entry at a
time.
A CANARY is a tripwire at a point that only a permission-override attempt
reaches. Ordinary use never gets there - the code path runs when a request asks
for something the rules forbid - so a trip is not noise, and it is worth an
admin's attention with the actor attached. Design:
docs/Security/Remediation/WeKan.md §12.
Three properties, each enforced by a test:
SILENT. Tripping one changes NOTHING the caller can observe. tripCanary()
always returns false and tripCanaryDeny() always returns true, so a call site
reads as the refusal it replaces:
if (forbidden) return false; // before
if (forbidden) return tripCanary('card.vote-field', { userId });
The return value is decided before any reporting is attempted and the reporting
is wrapped, so a canary can never throw into, delay or disturb the request it
watches. The REST handler re-throws the ORIGINAL error object. This is not
politeness: a canary that announces itself is a map of which paths are watched,
and a probe would simply avoid them.
BOUNDED. A canary sits where an attacker can loop, so one row per attempt would
be a denial of service they get for free. models/lib/canaryTokens.js - pure, no
Meteor, no clock of its own - records the first trip of a (canary, actor) pair
at once, COUNTS the rest of the window, flushes one summary carrying the total
when the window closes, and stops writing after 60 summaries for a pair that
never gives up. The tracked-pair map is capped at 5000 and evicts the LEAST
RECENTLY seen, so a long-running attacker is not pushed out by a passing one and
a botnet varying its address cannot grow it. Idle pairs are swept on a timer. A
thousand attempts in a minute cost ONE row, and a suppressed trip costs one map
lookup. The username lookup is cached and capped, and neither it nor the insert
is awaited on the caller's path.
ATTRIBUTED. Every event carries userId, username, ip and count. The username is
denormalised at write time on purpose: it is what the account was CALLED when it
tried, so a rename does not rewrite history and a deletion does not erase it.
The address uses the same spoofing-safe resolver as the login throttle -
X-Forwarded-For only as far as HTTP_FORWARDED_COUNT says to trust it - or an
attacker could write a colleague's address into the security log by sending a
header. `detail` says what was attempted in the words of the feature, never the
payload.
Seventeen canaries are wired at the permission checks that refuse the attempts
behind BoardBleed, ParentBleed, ChecklistBleed, PathBleed and CommentBleed, plus
the direct vote/poker field writes and a stored path that points outside the
storage root. All server-side: no Playwright, no browser, nothing to install.
Admin Panel / Problems / Security gains Username, IP address and Attempts
columns, and both new columns are searchable - the thing an admin does with one
security event is pivot on it.
tests/canaryTokens.test.cjs (25 checks) drives the limiter through bursts, an
endless probe, eviction order and junk input; tests/canaryCoverage.test.cjs (18)
pins every canary to its call site and asserts none of them changes what the
caller gets back.
Thanks to xet7 !
A database problem arrived in Admin Panel / Problems / Database problems
with an empty Category, Name and Action, a severity of warning, and a
detail that said "The database returned an error WeKan has no rule for.
Read the message below" - with no message anywhere below, or above, or on
the page at all. So the row said that something happened and nothing
about what.
Nothing was wrong with the classifier or the recorder. EventLog has a
SimpleSchema attached, and collection2 cleans every insert against it
with filter:true, so a field the schema does not declare is dropped
silently on the way to the database. The four fields the database stream
uses - type (the rule id), db (which database said it), kind and message
- were never added to that schema, so they were the four things thrown
away, and they are exactly the four the page shows in Category, Name,
Action and the message it told the admin to read. The other four streams
were unaffected because they only ever write schema fields.
The schema declares them now, and a test pins the general rule instead of
the one instance: for each of the five event loggers, every key of the
document it inserts must be declared in the EventLog schema, so the next
field somebody adds to a logger cannot vanish the same way.
The message is shown too - the Detail cell carries WeKan's reading of the
error and then the database's own sentence, which is the one thing an
admin can search for or paste into an issue - and the stream search looks
at message, db, kind and type, so searching for "postgresql" or for a
phrase out of the message finds the row that displays it. The
unclassified advice no longer promises a message "below": the message
follows in the same cell.
Because that message is now stored and displayed, it loses its
credentials first. A database that refuses a login or cannot be reached
quotes the connection URL back, password and all - "failed to connect to
mongodb://wekan:s3cret@mongo:27017" - so classifyDatabaseError redacts
the userinfo of any URL in the message before returning it, keeping the
host, the port and the database name, which are what makes the message
useful. It is done there rather than in the recorder so no future caller
can forget it. The message is also sanitized like every other stream's
detail: one line, control characters out, capped.
Thanks to xet7 !
WeKan runs on MongoDB or on FerretDB v1 over SQLite, PostgreSQL, MySQL, MariaDB
or SAP HANA, and each answers a broken query, a full disk or a refused login in
its own words. Those words reached an admin as an opaque string: "Error 1064
(42000): You have an error in your SQL syntax" with no way to know which
database said it, that it is FerretDB's bug rather than their data, and that
upgrading FerretDB is the fix.
models/lib/databaseErrors.js is the translation - pure, so it is unit-tested
without a database: a message in, and out comes which database said it, what
kind of problem it is, what it means, what the admin should do, and whether
WeKan can do anything itself. Sixteen rules, every one of them a message a real
database produced in the conformance runs or the documented wording of the
condition: injection refused by FerretDB's SQL guard, the syntax errors of all
four dialects, the two permission failures that stopped MySQL and MariaDB dead
this week, authentication, disk full, connection limits, deadlocks (retried),
lost connections (reconnected), timeouts, duplicate keys (which is an index
doing its job, not a problem), unimplemented operations, and a handler the
binary was not built with.
server/lib/databaseProblems.js records them into a new `database` event stream,
one per problem per minute - a database that is down answers every query - and
watches Meteor._debug, which is where a database error surfaces without wrapping
every call site in the app. Admin Panel / Problems has the pane, and its first
column is the database that said it.
Thanks to xet7 !
Lauri Ojansivu