A feature's guide now sits under the menu it is used from, read from the templates rather than from where somebody once put it: Header-Bar, All-Boards, Pages, Board-Header (search and every Board View), Board-Sidebar (Board Settings, Filter, Members, Labels, Archive), Rules, Swimlanes, Lists, Cards, Member-Settings, Admin-Panel and Login-Page. Topics used from several menus (editor, history, theme, translations, email, troubleshooting) are in General. - Each area has a README mapping its menu entries to their guides and to the template or server module that implements them. Board-Header/README.md lists all 40 Board View entries with the template boardBody.jade renders; five views (Table, Calendar Mode, Time, Timeline, Map) have no guide of their own yet and say so. - Board-View.md listed 25 views while BOARD_VIEWS has 40; it now lists all of them with their English labels. - Pages keep their file names. Where a released changelog links an old path, a "# Moved" page stays there and points to the new one; test-menu.sh skips folders holding only those. - Undo was filed under Login; it is General/Undo now. - Every link in docs, code comments, scripts and tests follows the move; docsLinksResolve checks them all. Tests that read a doc follow it to its new place. boardViewSettings now fails when Board-View.md or Board-Header/README.md differs from BOARD_VIEWS, or names a template boardBody.jade does not render. Thanks to xet7 !
36 KiB
Design: restart-safe background and external operations
Status: Implementation in progress · Owner: xet7
This design applies whenever WeKan starts work that can outlive one request or one server process: imports, attachment moves, database migrations, backups, integrity scans, scheduled rules, webhooks, mail and other external services. The recovery rule is not "start it again from the beginning". WeKan persists a checkpoint after each idempotent unit and continues after an unclean stop.
Durable job contract
Every background operation has a database record containing its type, owner and
tenant, sanitized input reference, state, current checkpoint, attempt counters,
nextAttemptAt, bounded error history, timestamps, and a renewable lease. Secrets
are referenced through server-side configuration or an encrypted credential
record; plaintext tokens are never copied into a job.
Workers claim a due job with one atomic conditional update. The lease has an owner ID and expiry, is renewed during work, and is released on a clean pause or completion. A replacement process may reclaim an expired lease. Multiple WeKan replicas therefore cannot execute the same unit concurrently.
Each unit uses a stable idempotency key and follows this order:
- Claim or reclaim the job and read its persisted checkpoint.
- Check whether the unit's result already exists under its idempotency key.
- Perform one bounded unit of work.
- Verify the result, then atomically advance the checkpoint.
- Renew the lease, yield, and claim the next unit.
A crash before step 4 repeats the same unit, not the whole job. Repeating must be
safe: database copies upsert by _id; imported boards retain their source/job
identity; file transfers verify destination bytes before removing a source; and
external requests carry an idempotency key when the provider supports one.
External-service policy
Every outbound operation has connect and total timeouts and a bounded response. HTTP 408, 425, 429 and transient 5xx responses, DNS/connect resets and timeouts are retryable. Authentication, authorization, validation and SSRF refusals are not. Retries use exponential backoff with jitter and a configured maximum.
Retry-After is authoritative when it is a valid delta or HTTP date. Provider
rate-limit reset headers may extend, but never shorten, that delay. The next
attempt time is persisted before sleeping, so restarting does not reset a rate
limit or cause a retry storm. Per-provider concurrency and minimum-spacing gates
apply across all jobs in this WeKan instance.
After the automatic-attempt limit, the job stays persisted as paused or
failed; it is never discarded. Problems → Recovery records the operation,
checkpoint, attempt, next retry, bounded failure reason and whether restart
recovery reclaimed it. Security-sensitive refusals remain in Problems → Security.
Operation-specific checkpoints
| Operation | Durable unit and completion evidence |
|---|---|
| Trello and other board imports | One source board; source ID maps to exactly one imported board before the queue index advances |
| ZIP/JSON/CSV/Jira/Kanboard/ICS imports | Parsed source plus one board/card batch; created records carry the job/source key |
| Board imports and copies (implemented) | An importRuns record written before the first write names the board id the import will create; the board carries the run id. A stopped run is flagged once and kept or discarded in Recovery; the source is not kept, so the rest is completed by importing again (details) |
| Attachment/avatar moves | One file version; destination size/checksum and metadata agree before source removal |
| Text database migration | One collection batch ordered by _id; target upserts and evidence cover the checkpoint |
| Backup/restore | One collection/file entry; staged archive/object and checksum manifest are published last |
| Integrity/recovery scans | One bounded inventory batch; the last stable object key is persisted |
| Scheduled rules | Trigger occurrence ID; an occurrence already recorded is not applied twice |
| Webhooks/mail | One delivery record per event and destination; provider idempotency key or an explicit at-least-once status |
Request/response exports do not continue an HTTP socket after restart. They use a snapshot read and fail visibly; downloadable asynchronous exports, when added, must use this contract. A client may safely retry a synchronous idempotent read.
Startup and shutdown
Startup first marks expired running leases as reclaimable, records a
job-reclaimed-after-restart Recovery event, then starts due jobs gradually with
jitter. Jobs whose required credential or storage is unavailable become paused
with an actionable reason. They are not falsely marked failed or completed.
On SIGTERM WeKan stops claiming work, checkpoints the current safe boundary and releases leases within the shutdown grace period. SIGKILL, power loss and process crashes rely on lease expiry and idempotent replay.
Tests
Each durable operation needs positive completion, crash before checkpoint, crash
after side effect, expired-lease reclaim, concurrent-worker exclusion, pause and
cancel tests. External operations additionally test timeout, network reset, each
retryable status, non-retryable 4xx, valid/invalid Retry-After, jitter bounds,
attempt exhaustion and restart during backoff. Negative tests prove secrets and
unbounded response bodies are not persisted in jobs or reports.
Sync write-plan engine checkpoint
server/lib/syncOperationJournal.js now implements the private write-plan and
checkpoint engine for the next Sync adapter. It is not yet wired into manual
or scheduled Sync. Current production Sync still uses reconciliation and
retained diagnostic reports; it does not automatically resume this journal.
The engine requires a caller-held renewable list lease and a fixed scope (list, board, list lifetime, configuration revision and source identity). A unique pending document owns the operation; separate indexed step records hold only allowed Sync card fields before and after each unit. No credential or raw provider-response objects belong in the plan. Snapshots are private content, not values safe for publication, export or the diagnostic report endpoint. The integration must provide private collections without TTL expiry.
Preparation stores all steps before enabling application writes. An interrupted preparation may be rebuilt because it cannot have applied a unit. Once applying, the original stored plan is authoritative. Resuming validates its per-step checksums and whole-plan checksum before calling any application adapter. Plans are limited to 10,000 unique cards and 1 MiB per step; exceeding a limit fails before application writes. BSON dates and missing/null distinctions are retained. A scope change or damaged plan preserves evidence and stops the run.
Mapped-estimate plans also retain the complete typed custom-field array and source baseline (including zero or explicit null), with the canonical local field/provider field/unit identity. Other field values and their order must remain unchanged. Dates, booleans, strings, finite numbers, string arrays and empty assignments survive storage; unknown properties, duplicate IDs, invalid values and missing or malformed mappings are rejected. Custom-field arrays are limited to 10,000 entries within the existing 1 MiB step budget. A mapping's unit may be a configured custom unit, as in ordinary Jira estimate import/Sync.
Real MongoDB tests cover lost acknowledgements for estimate updates and clears, unchanged typed local fields, mapping-checksum damage and local edits before replay. They use a test application adapter; this does not enable production Sync recovery. The production adapter must still validate current field ownership/mapping under the list guard and coordinate card writes with durable History/activity effects before acknowledging a unit.
server/lib/syncOperationMutation.js now prepares the conditional predicates
and field-only $set/$unset mutation from a validated stored step. It compares
the union of before/after fields, so additions require absence and removals
remain absent in the expected result. It never replaces the whole card or
changes fields outside the plan. Creation returns a separate insert document;
unchanged updates return an empty modifier for the caller to recognize as a
no-op. Scope changes and explicit undefined values are refused, BSON dates
survive cloning, and input/driver-side mutations cannot alter result predicates.
The same literal-value selector helper is used by production Sync. Object and array values are wrapped as equality operands, while null includes a presence check. The journal integration fixture uses the shared mutation planner and verifies lost-acknowledgement replay without replacing unrelated card metadata. The planner does not write cards or establish completion of History/activities; those obligations still belong to the unfinished production adapter.
server/lib/syncOperationApply.js now supplies the internal application step.
It runs the ownership guard around conditional writes, reads the expected card
state back even after successful replies, and accepts lost acknowledgements
only when that state is readable. An occupied creation ID or a changed local
value cannot be overwritten. Empty modifiers still require confirmed state.
Matching card state never completes the unit by itself. Every unacknowledged
attempt calls completeEffects with a stable identifier derived from the
operation and step index, including retries whose card already matches. That
callback must durably finish or reuse History, activities and downstream
effects before returning the exact identifier. Missing/wrong acknowledgements
or a lost lease prevent journal advancement. The real MongoDB test interrupts
effects after saving a card, then verifies recovery before checkpoint cleanup.
This adapter is not yet connected to manual or scheduled Sync. Its card interface must preserve application validation/hooks, and the durable effects implementation, production scope checks and lifecycle remain unfinished. The MongoDB test uses a separate effect receipt as a stand-in; it does not prove delivery of actual rules, notifications or webhooks. Guards and readbacks also do not fence in-flight writes or provide cross-collection atomicity.
server/lib/syncHistoryBatch.js prepares and persists the History component
for saved Sync update/archive steps. The journal can now save the effect plan
in the same unit document as the card plan before mutation. The optional
prepareEffects and validateEffects adapters must both be supplied; the
operation marker retains this mode and refuses a downgrade during recovery.
The checksum covers both plans, and the entire persisted plan is validated
before any unit is applied. Each unit, including its BSON envelope, is limited
to 15 MiB. History validation binds event contents and identity to the exact
card step. Missing/corrupted effects cannot be replaced by an empty plan or a
freshly generated one while applying. Both plans share checkpoint retention
and verified cleanup, with no separate effect-plan collection to orphan.
Journal plan builders receive the stable operation/intent IDs,
scope and ownership guard, and syncOperationEffectId shares the same effect
identity with application/retry. Field rows use the ordinary History content
format, fixed timestamps and deterministic IDs, sharing one batch and a fixed
integrity chain. Estimate arrays retain nested dates. Baseline-only changes
produce no History rows and do not invalidate redo.
The plan snapshots the exact hashed redo candidates and undo timestamps before the change. Persistence conditionally supersedes only those candidates and reads the flags back; it never queries a fresh redo set on retry. Missing or damaged predecessors, changed redo snapshots, failed confirmations and wrong event contents prevent acknowledgement. Partially inserted rows are reused without rewriting timestamps, chain links or later undo flags. Plans are bounded to 16 event rows, 10,000 redo candidates and 15 MiB; legacy unhashed redo rows are refused rather than silently skipped.
Real MongoDB coverage saves a card, interrupts between History rows, retains the journal checkpoint, then resumes the saved plan without rebuilding or duplicate events. A different row undone during interruption remains available for redo, and the resulting chain verifies. This is an internal History component, not production Sync integration or acknowledgement of activity/rule/notification delivery. Creation's initial-position effects, production collection/job lifecycle, safe suppression of ordinary duplicate hooks, legacy redo handling and coordination with independent board History writers remain unfinished. In particular, a fixed chain does not serialize concurrent appenders or make redo/event writes atomic with the card mutation.
The adapter must compare exact current/before/after states, perform a conditional
write only from the before state, verify its result, and return applied or
already-applied. A committed write with a lost acknowledgement must return
already-applied on retry, without repeating hooks or side effects. The engine
advances only the matching operation owner's current checkpoint. Completed
operations retain their marker until plan cleanup finishes, so interrupted
cleanup never repeats card writes.
Cleanup now verifies the absence of operation-specific plan rows before removing their checkpoint, then verifies that the list's operation slot is empty. Zero/partial deletions and failed reads do not report success. A lost delete acknowledgement is accepted only after absence is read back. Ownership is rechecked between the two stages, and a successor operation is never removed or treated as this operation's successful cleanup. A retained cleaning marker resumes cleanup without applying card units again.
The engine now requires a caller-persisted UUID intentId and a private raw
MongoDB completions collection. Reuse the same intent for every retry of one
request, including after the operation marker is gone; generate a new intent
only for a new request. Do not derive retry intent from the pending
marker, which is deliberately removed at completion. A different intent cannot
take over that marker while it remains pending.
Before cleanup, an immutable completion record captures the intent, operation ID, exact scope, total, plan checksum and application-completion timestamp. The engine reads it back before deleting any recovery evidence. Failed writes, missing rows and failed verification retain the applied plan. Lost insertion acknowledgements are accepted only when the exact saved proof is readable. Conflicting, malformed and foreign-scope records cannot acknowledge the work. These records contain no card values or credentials, but remain private.
If the final cleanup read fails after deleting the marker, the current attempt still reports an unknown outcome. Retrying the SAME intent reads its durable proof, finishes operation-specific cleanup and returns its original result without rebuilding or applying card units. A newer operation on that list stays untouched. Completion records have no automatic deletion or TTL here: removing a record while its intent can still be retried would lose this guarantee.
Production integration must persist intent before invoking the engine and wire private completion storage, retention and job/outcome lifecycle. The proof means all adapter units were acknowledged; it cannot independently prove unfinished History/activity effects. These readbacks do not fence already in-flight writes or form a transaction across collections.
Real MongoDB tests inject interruptions before application, after a side effect, after checkpoint progress and during cleanup. They exercise lost ownership, changed local state, changed configuration scope, BSON dates, updates/archives, damaged plans and unverified adapter results. The adapter in those tests is a raw-collection fixture using the shared planner, not the Meteor card writer.
The shared Sync card selector now distinguishes an absent field from explicit null, so deleting a nullable field cannot satisfy an older write snapshot. Remaining integration includes normal card hooks and History, source/permission rechecks, archive dependencies, private collection lifecycle, pause/cancel/review controls, retained outcome reports and scheduler/startup recovery. Journal ownership protects checkpoint acknowledgements; it does not fence an already in-flight card write. That cross-collection boundary and atomic multi-card reconciliation remain open. No automatic retry scheduler or application UI is enabled by this engine alone.
Manual and scheduled Sync use the engine (2026-09-30)
Manual "Sync now" and the 15-minute scheduled Sync now write through this
engine when it can do everything (server/lib/listSyncSteps.js):
- the board enabled Sync effects, and for a scheduled run the instance enabled cron effects;
- the list has a versioned scope, which a list gets when its settings are saved;
- every rule action on the board has a durable adapter. Only sending email has one so far.
Otherwise the run uses the direct writes it always used, so no rule stops
running. server/lib/listSyncApplication.js turns the reconcile plan into saved
steps. The steps are built from the stored cards, and a local edit since the
fetch stops the run. It plans History and activities with the shared effect
planner and applies each step through the hooked card and activity adapters and
durable delivery.
A new run first finishes the list's unfinished operation. The
wekan-list-sync-replay cron job resumes interrupted operations every minute,
under the list lease, with the actor and trigger stored in the intent (intent
version 2). An operation still preparing has written nothing and is discarded,
since a replay has no source data to rebuild it from.
An operation whose scope or access went stale - the list was removed, recreated
or reconfigured, its actor lost full-list write access, or its intent has no
trigger - can never be replayed and would block the list's Sync for ever. It is
marked once with one list-sync-operation-stuck Recovery event
(server/lib/listSyncStuck.js), and an administrator discards it in
Problems → Recovery:
an immutable decision record first, then removal of its steps and marker under
the list lease, with no further card writes. The discard is refused while a live
check says the operation can still be replayed, and is idempotent.
The in-flight-write and atomicity limits above still apply. Rule actions other than email need their own durable adapters before those boards can use this path.
Card activity and History boundary found during adapter integration
The installed collection-hooks update wrapper calls after.update even when
MongoDB reports zero affected documents. Title/description activity hooks also
used to run before the card update, so a later failed conditional write could
already have emitted an activity and triggered rules/webhooks.
Title and description activities now run after successful writes, comparing the
previous snapshot with the requested text. Description clearing handles the
schema's $unset form. Archive activity ignores zero-match and unchanged-state
writes. Universal entity History and rule/trigger/action History use the same
successful-write guard before recording. This protects current Sync and other
ordinary write paths while the durable adapter is being built.
It does not close the crash window between the card mutation and its separate
activity/History writes. A durable adapter must retain and replay those effects
under stable identities before acknowledging a unit; simply seeing the new card
values cannot prove all hooks completed. Aggregate multi-update counts also do
not identify individual successful rows. Full replay integration remains unfinished
at this boundary, not declared finished by moving hooks to after.update.
Scrum History restoration acknowledgements
Scrum restore/undo/redo keeps its pending journal until each affected author's restoration event is acknowledged. These events now use a deterministic row ID from the board, journal operation and author. A retry validates the stored event's immutable contents and integrity hash before accepting it, including legacy random-ID events from an already pending operation. It retains the original creation timestamp and chain link. A lost insert acknowledgement or concurrent retry is accepted only after reading back the exact intended event. Successful insert replies require the same readback: a hook refusing the insert, an altered stored event or an unavailable confirmation read cannot release the journal. A retry after a failed confirmation reuses the persisted event without changing its timestamp or integrity chain.
After those events are acknowledged, finalization verifies the original History row's integrity, author and superseded state. Undo/redo changes use a conditional write followed by a readback of the persisted flag and timestamp. An already completed undo retains its timestamp on retry. Missing rows, zero-match writes, invalid final states and lost write acknowledgements retain recovery evidence. Checkpoint deletion matches the board, source row, direction, author and operation ID, so an older worker cannot delete a replacement operation's checkpoint. These checks do not make verification and deletion a cross-document transaction; in-flight writers still need coordinated fencing for that stronger guarantee.
A failed write or mismatched/damaged event leaves the journal pending. Ordinary History writes retain their best-effort contract; this strict writer applies only to Scrum restoration. It does not make the original Scrum mutation and History atomic, serialize independent writers to the board's integrity chain, or provide startup replay. Those remain separate durability requirements.
Email outbox and restart recovery
Activity notifications now enter a private notificationEmailJobs collection.
Each event/recipient pair has a deterministic ID, rendered subject/body,
language, board/card references, due time and attempt counter. Insertion is read
back before enqueue succeeds. Duplicate events retain the original rendered
copy, including after completion. Notifications without an activity ID receive
a new UUID and cannot deduplicate a replay across separate enqueue calls.
The worker scans due jobs after startup and once a second after each completed
pass. Each pass selects up to 100 distinct due recipients, ordered by their
earliest due time and recipient ID, so a recipient with many queued messages
cannot hide other recipients behind a message-count limit. Four workers per
application process handle recipients concurrently; concurrent callers share
the same pass. All processes also compete for four renewable reservations in
notificationEmailSendSlots, acquired before an attempt is recorded. Full
capacity leaves the message pending without spending retries. A slow recipient
does not occupy the other three reservations. Slots renew every 15 seconds and
expire after 60; a crashed owner's slot can then be reclaimed. An independent
local expiry timer cancels SMTP even when the renewal database call hangs.
Ownership loss cancels only that sender's connection, and old-owner cleanup
cannot remove a replacement's slot. This is a bound on live queue reservations;
SMTP cannot fence a stale connection from a paused host. Application clocks
must be synchronized. Direct mail outside the outbox is not counted.
A pass waits for its active transports before polling again. SMTP phase, idle
and total deadlines close unresponsive or continuously active connections.
It reserves a recipient in notificationEmailLeases using the shared
renewable lease primitive, in a separate collection from list Sync. A normal
live reservation prevents another process from sending that recipient's digest.
Expired reservations can be reclaimed. It streams at most 100 rows per digest,
combining up to 4 MiB of body text; a larger single event travels alone. Individual
jobs must fit within a 15 MiB BSON budget. Temporary failures retain rendered
content and retry after 5 seconds, then 10, 20 and so on, with 0–25% positive
jitter. The base caps at 48 minutes, giving a final spread of 48–60 minutes.
Backoff survives restart. Each job gets at most twelve attempts per cycle,
reserved and read back before delivery preparation so a crash consumes its
attempt. Older records use their stored failure count as the initial budget.
Numeric SMTP 4xx responses retry; 5xx responses stop. Authentication, envelope,
message and TLS configuration failures without a temporary response also stop.
Console-only or hook-suppressed sends without confirmed acceptance require
operator review. Other failures retry within the same limit. Storage failures
after SMTP acceptance are classified separately from SMTP rejection.
Stopped rows retain their payloads and a fixed reason category; raw transport
errors and credentials are never stored in the queue report. Deleted users and
recipients who lost their active board membership or disabled email are cancelled without sending. Missing
addresses and disabled accounts retain work for retry within the same limit.
Only a transport result listing the recipient as accepted acknowledges delivery. Meteor's development console output and suppressed send hooks are not delivery proof. Accepted/cancelled rows shed subject, body, language and reply target; small identity/state receipts remain without a TTL to suppress event replay. The current account address, From setting and reply signing configuration are resolved at delivery. A multi-card digest replies to its last card reference.
Legacy profile.emailBuffer lines are copied into stable jobs and read back
before $pullAll removes those exact lines. Their original subject and board/
card identity were never stored, so they use the neutral subject WeKan and no
Reply-To. Their board membership cannot be revalidated retrospectively. A failed
migration retains the old buffer and does not stop delivery for other users.
Delivery is explicitly at least once. SMTP acceptance followed by a crash,
a lost acknowledgement or lease expiry can cause duplicate mail. Leases cannot
retract remote acceptance, although a lost reservation closes the sender's
live connection. Original activity creation and notification
enqueue are still separate operations, so this does not yet prove completion of
a durable Sync effect. Recipient summaries and pause/resume/cancel/retry controls
are available in Problems → Recovery; see the operator guide.
Control requests use the recipient lease, persisted holds/cancellation cutoffs
and stable request receipts with generation checks. A paused recipient stays
listed with no pending jobs, so the hold can always be removed. Report totals
scan sorted recipient summaries with bounded memory; concurrent writes can
change counts between reads, and Refresh obtains a new result.
SMTP DNS, connection, greeting and idle limits are configured in the
mail troubleshooting guide.
Terminal receipt metadata is compacted after 30 days by default, with bounded
background batches and permanent minimal replay identities. Atomic replacement
in the original collection prevents gaps in duplicate suppression. Pending or
failed work and control state are retained. See the
receipt policy.
Atomic activity-to-queue insertion remains pending.
Cross-process slot contention, crashed-owner reclaim, hung renewal and live
SMTP cancellation are covered by tests/integration/emailSendSlots.test.cjs.
Set WEKAN_SYNC_TEST_MONGO_URL and WEKAN_SMTP_TEST=1 to run its database and
local SMTP fixtures. No external mail provider or FerretDB was exercised by
these local MongoDB/SMTP tests.
Run tests/integration/emailOutbox.test.cjs with
WEKAN_SYNC_TEST_MONGO_URL pointing to local disposable MongoDB. For a real app
restart, stop the test app, set WEKAN_EMAIL_STARTUP_TEST_MONGO_URL to its test
database and WEKAN_EMAIL_STARTUP_READY_FILE to a new file under .tools/tmp,
then run tests/integration/emailOutboxStartup.test.cjs. Start the app when the
ready file appears, using that database and MAIL_URL=smtp://127.0.0.1:4102.
The suite owns that local capture port, verifies acceptance and a scrubbed
receipt, and removes its own rows. Optional WEKAN_EMAIL_STARTUP_TEST_APP_URL
and WEKAN_EMAIL_STARTUP_TEST_SMTP_PORT override the loopback defaults.
Activity notification write-ahead intent and hook capture
server/lib/activityNotificationIntent.js provides the storage boundary for
activity notification capture. It confirms a private immutable intent before
allowing an activity insert. Its separate persistence helper also reads back
the exact stored activity. Activity IDs and timestamps must already be final.
The intent binds the entire BSON-preserving snapshot and carries a unique
writer identity. Lost intent or activity acknowledgements require matching
readback; conflicting reuse of an activity ID is refused.
The persistence helper allows only the creating call to insert a missing activity. A later call cannot distinguish an interrupted first insertion from subsequent intentional deletion, and must not recreate it. Recovery reads require the exact retained activity; missing or changed activities leave an unresolved intent. Explicit orphan resolution is still needed. A returned intent does not acknowledge tray delivery, email enqueue, rule actions or webhooks.
Ordinary server activity insertion now captures an intent in the private
activityNotificationIntents collection after timestamps and before the
activity write. A failed intent write stops the activity insert. Disabled
notifications create no intent. Scoped deferred Sync inserts skip capture,
leaving delivery to their own persisted effect plans. Raw database writes and
imports that bypass hooks do not get this guarantee.
Ordinary delivery remains asynchronous. Before the first local delivery,
activityNotificationPlans stores an immutable, checksummed plan containing
recipient IDs, tray choices and already rendered email jobs. The original
dispatch actor and exact activity snapshot bind the plan. Changed templates,
languages or watchers cannot rewrite a saved plan on retry. Candidates whose
services are already disabled or inaccessible are excluded at preparation;
a later access or preference change stops replay and retains pending evidence.
Delivery rechecks the current account, board membership, watch/mute scope, assigned-only card scope and channel preferences. It uses the real tray receipt and email job identities, requiring their exact acknowledgements. A crash between these services can replay without recreating a dismissed tray notification or duplicating an already queued event. Only confirmed local deliveries compact the intent to a permanent small receipt; its activity snapshot is removed. The dispatch actor remains bound to the receipt. Subscriber or completion-write failure retains pending evidence. The collection has no client publication, rejects member/admin DDP writes and has no TTL. Webhooks remain independent and nonblocking; completion is not SMTP acceptance.
Each process now scans pending intents after startup and every second after
the previous pass finishes. Set ACTIVITY_NOTIFICATION_RECOVERY_INTERVAL_MS
to an integer from 1000 to 60000 to change the interval. A pass selects at
most 100 IDs, then loads one private payload at a time. Keyset pagination
advances past failed, busy or orphaned records; overlapping local scans share
one pass. Failed storage scans retry on the next timer.
Immediate delivery and recovery acquire the same per-intent reservation in
activityNotificationLeases. Reservations renew every 15 seconds and expire
after 60 seconds, permitting another process to reclaim a crashed owner's
work. Guards check ownership during preparation, before service writes and
before completion. Former owners cannot delete successor reservations. These
are renewable reservations, not cross-collection transactions; stable service
identities still make uncertain writes safe to retry. Keep server clocks
synchronized.
Recovery uses an existing plan without rebuilding recipients or content. If a crash happened after activity persistence but before planning, it prepares the first plan from that exact retained activity and current permitted context. It never recreates missing activities or reruns activity rule and webhook hooks. Missing/changed activities, invalid plans and revoked access remain pending for later review. Completed intents are skipped.
After the intent acknowledges local delivery, its plan is immediately compacted in place. The permanent plan receipt keeps only its unique ID, activity hash, plan checksum and compaction version. Rendered subject/body, recipient IDs, language and channel choices are removed. Keeping the unique plan row prevents a delayed former writer from recreating those payloads. The SMTP outbox retains its own queued content until delivery or cancellation; plan compaction does not delete an unsent email job.
A separate cleanup sweep handles interruption after acknowledgement but
before compaction. It examines up to 100 plan IDs per pass, loading one plan
under the same per-intent reservation as delivery. The default interval is
60 seconds; ACTIVITY_NOTIFICATION_PLAN_CLEANUP_INTERVAL_MS accepts integer
values from 1000 to 86400000. Failed/busy/malformed rows do not hide later rows.
Only an exact completed intent with the same activity hash and dispatch actor
permits compaction; the plan checksum must validate too. Conditional atomic
replacement and readback reconcile uncertain replies. Activity deletion after
completion does not prevent removing old plan content.
Pending plans, orphans, inconsistent completion evidence and damaged payloads remain intact. Operator/orphan controls and unresolved-payload retention are still unfinished. Permanent receipts have no TTL and their total count is not capped. Plans and reservations remain private and reject member/admin DDP writes. Limit each plan to 10,000 recipients and 14 MiB, checking size while preparing recipients; oversized preparation leaves the intent pending without delivering a prefix. This recovers local tray/email enqueue only; full card/History/activity/Sync effect coordination remains separate work.
tests/integration/activityNotificationIntent.test.cjs uses a real MongoDB
with WEKAN_SYNC_TEST_MONGO_URL. It covers write ordering, uncertain replies,
concurrent writers, cancelled insertion, deleted/changed activities, corrupted
intents, identity reuse, guard failure and oversized/invalid input. These tests
also cover completion compaction, uncertain completion replies and dispatch
actor mismatch. Full-app Meteor tests exercise real before/after hooks,
failed storage, pending subscribers, disabled notifications and deferred Sync.
Chromium tests verify actual SMTP delivery and private-collection denial.
FerretDB has not been exercised.
tests/integration/activityNotificationPlan.test.cjs covers saved-plan reuse,
uncertain plan writes, corrupted checksums, access revocation, false service
receipts, duplicate/invalid recipients and size limits. It interrupts between
real tray storage and real email enqueue, then retries the same plan. The
full-app intent suite verifies that disabled recipients stop replay and that
resume does not invoke rendering or recipient selection again. Chromium
checks persisted plans against actual delivered mail and denies client writes.
tests/integration/activityNotificationRecovery.test.cjs verifies bounded
scan progress past failures, local coalescing, competing reservations, expired
owner reclaim and successor preservation. The full-app recovery suite uses
the real private collections and email enqueue adapter. Chromium tests seed
persisted intents with and without saved plans, then observe scheduled SMTP
delivery without invoking a notification hook; a missing activity remains an
orphan and is never recreated. These local tests do not run FerretDB.
tests/integration/activityNotificationPlanRetention.test.cjs verifies payload
removal after completion, permanent replay keys, pending/missing/mismatched
receipt preservation, uncertain or false acknowledgements, changed payloads,
ownership loss, malformed-row progress and local scan coalescing. Chromium
checks immediate compaction after actual SMTP delivery and scheduled cleanup
of a previously completed plan while an unfinished orphan retains its body.