Snap: name an AppArmor-refused database connection at once, and stop the snapctl get errors.

The logs in #6746 show the real cause of "write: permission denied" from
database-restore and of WeKan waiting forever for FerretDB: 1300 kernel
lines of apparmor="DENIED" operation="file_perm" class="net" for
snap.wekan.wekan on 127.0.0.1:27019, with the same denial for snap-store
and standard-notes on that computer. FerretDB was running and accepting
connections; the kernel refused the first write. That is snapd 2.77.1 on a
kernel with an AppArmor socket mediation bug (Launchpad snapd bug 2169038),
fixed by a newer kernel and worked around by `snap revert snapd`.

WeKan cannot lift that denial, so it now says what it is. A new
bin/socket-denied recognises the refusal and prints the denial to look for,
the upstream bug and both ways out. wekan-control's FerretDB and MongoDB
wait loops keep each attempt's error and print it on the first refused
attempt instead of "not ready yet" for two minutes; database-backup and
database-restore stop at once instead of starting an already running
service and waiting 90 seconds.

The same log began with "error: snapctl: get which option?" four times:
SAML_IDP_PROFILE, SAML_WANT_RESPONSE_SIGNED, SAML_WANT_ASSERTIONS_SIGNED
and SAML_LOGIN_FLOW were listed in bin/config without a KEY_ name, so
wekan-read-settings ran a bare `snapctl get` for them and `snap set` could
never reach them. They now have their snap keys, the CORS_EXPOSE_HEADERS
description has its right name, and a key without a snap name falls back
to its default instead of calling snapctl.

Tests: snapSocketDenied, snapSettingsKeys, and extended snapDatabaseRestore
and ferretdbWaitTimeout.

Related to #6746: the reporter still has to update the kernel or revert
snapd, so the issue stays open until that is confirmed.

Thanks to fabiosalles and xet7 !

snap-src/bin/config

@@ -283,7 +283,7 @@ DESCRIPTION_CORS_ALLOW_HEADERS="Set Access-Control-Allow-Headers header.  \"Auth
 DEFAULT_CORS_ALLOW_HEADERS=""
 KEY_CORS_ALLOW_HEADERS="cors-allow-headers"
 
-DESCRIPTION_EXPOSE_HEADERS="Set Access-Control-Expose-Headers header.  This is not needed for typical CORS situations. Example: *"
+DESCRIPTION_CORS_EXPOSE_HEADERS="Set Access-Control-Expose-Headers header.  This is not needed for typical CORS situations. Example: *"
 DEFAULT_CORS_EXPOSE_HEADERS=""
 KEY_CORS_EXPOSE_HEADERS="cors-expose-headers"
 
@@ -973,16 +973,20 @@ KEY_S3_SECRET_FILE="s3-secret-file"
 
 DESCRIPTION_SAML_MERGE_EXISTING_USERS="Allow explicit SAML account linking. Default: false"
 DEFAULT_SAML_MERGE_EXISTING_USERS="false"
+KEY_SAML_MERGE_EXISTING_USERS="saml-merge-existing-users"
 
 DESCRIPTION_SAML_IDP_PROFILE="Identity provider profile: standard (signed Response, popup) or signed-assertion-redirect (signed Assertion, full-page redirect). Default: standard"
 DEFAULT_SAML_IDP_PROFILE="standard"
+KEY_SAML_IDP_PROFILE="saml-idp-profile"
 
 DESCRIPTION_SAML_WANT_RESPONSE_SIGNED="Require a signed SAML Response (true/false). Default: from SAML_IDP_PROFILE"
 DEFAULT_SAML_WANT_RESPONSE_SIGNED=""
+KEY_SAML_WANT_RESPONSE_SIGNED="saml-want-response-signed"
 
 DESCRIPTION_SAML_WANT_ASSERTIONS_SIGNED="Require a signed SAML Assertion (true/false). Default: from SAML_IDP_PROFILE"
 DEFAULT_SAML_WANT_ASSERTIONS_SIGNED=""
+KEY_SAML_WANT_ASSERTIONS_SIGNED="saml-want-assertions-signed"
 
 DESCRIPTION_SAML_LOGIN_FLOW="SAML login flow: popup or redirect. Default: from SAML_IDP_PROFILE"
 DEFAULT_SAML_LOGIN_FLOW=""
-KEY_SAML_MERGE_EXISTING_USERS="saml-merge-existing-users"
+KEY_SAML_LOGIN_FLOW="saml-login-flow"

snap-src/bin/database-ready

@@ -22,7 +22,19 @@ database_ready() {
   # Older layouts and the tests' stand-in snap have no db-eval: leave the
   # connection to the tool itself, as before.
   [ -e "$SNAP/bin/db-eval" ] || return 0
-  bash "$SNAP/bin/db-eval" ping "$url" >/dev/null 2>&1 && return 0
+  [ -f "$SNAP/bin/socket-denied" ] && source "$SNAP/bin/socket-denied"
+  err="$(bash "$SNAP/bin/db-eval" ping "$url" 2>&1 >/dev/null)" && return 0
+
+  # #6746: the kernel refused the connection, not the database. Starting the
+  # service changes nothing (it is already running - the connect succeeded)
+  # and waiting only delays the one message that helps, so say it now.
+  if declare -F socket_denied >/dev/null && socket_denied "$err"; then
+    echo "" >&2
+    echo "Cannot talk to the database on ${host}:${port}:" >&2
+    echo "  ${err}" >&2
+    socket_denied_hint >&2
+    return 1
+  fi
 
   role="$(bash "$SNAP/bin/database-role" 2>/dev/null)"
   case "$role" in mongodb|ferretdb) ;; *) role=ferretdb ;; esac
@@ -39,12 +51,9 @@ database_ready() {
       echo "" >&2
       echo "The database (${svc}) does not answer on ${host}:${port} after ${timeout} seconds." >&2
       [ -n "$err" ] && echo "  ${err}" >&2
-      case "$err" in
-        *"permission denied"*|*EACCES*|*"operation not permitted"*|*EPERM*)
-          echo "  The connection itself was refused by this system's security policy, not" >&2
-          echo "  by the database. 'sudo journalctl -k | grep -i denied' names the rule." >&2
-          ;;
-      esac
+      if declare -F socket_denied >/dev/null && socket_denied "$err"; then
+        socket_denied_hint >&2
+      fi
       echo "The reason is in its log:" >&2
       echo "  sudo snap logs -n 100 ${svc}" >&2
       echo "  sudo snap services ${SNAP_INSTANCE_NAME:-${SNAP_NAME:-wekan}}" >&2

snap-src/bin/socket-denied

@@ -0,0 +1,52 @@
+#!/bin/bash
+# socket-denied — recognise, and explain, a database connection that the
+# KERNEL refused rather than the database.
+#
+# #6746: on the reporter's computer every WeKan command that talks to its own
+# database failed with
+#   write tcp 127.0.0.1:53014->127.0.0.1:27019: write: permission denied
+# and `journalctl -k` was full of
+#   apparmor="DENIED" operation="file_perm" class="net" profile="snap.wekan.wekan"
+#   ... faddr=127.0.0.1 fport=27019 ... requested="send" denied="send"
+# The database was running: the TCP connection was ACCEPTED, and only the first
+# write on it was refused. The same log shows the same denial for other snaps on
+# that computer (snap-store, standard-notes), so it is not something in WeKan's
+# profile - it is snapd 2.77.1 on a kernel with an AppArmor socket mediation
+# bug (https://bugs.launchpad.net/snapd/+bug/2169038). A newer kernel fixes it,
+# and `snap revert snapd` (to 2.76.x) works around it until then.
+#
+# Without this, WeKan said only "FerretDB not ready yet" for two minutes and
+# then pointed at FerretDB's log - which is clean, because FerretDB never got a
+# byte - and database-restore started a service that was already running and
+# waited 90 seconds for it. Nothing WeKan can do from inside the snap lifts an
+# AppArmor denial, so the useful thing is to say at once what it is and how the
+# owner of the computer fixes it.
+#
+# Source it, then:
+#   socket_denied "$err"        returns 0 when $err is that kernel refusal
+#   socket_denied_hint [svc]    prints the explanation (to stdout)
+
+socket_denied() {
+  case "$1" in
+    *"permission denied"*|*"Permission denied"*|*EACCES*|*"operation not permitted"*|*"Operation not permitted"*|*EPERM*)
+      return 0 ;;
+  esac
+  return 1
+}
+
+socket_denied_hint() {
+  local snap="${SNAP_INSTANCE_NAME:-${SNAP_NAME:-wekan}}"
+  echo "  The database IS running: this computer's security policy (AppArmor)"
+  echo "  refused WeKan's own connection to it, before the database saw anything."
+  echo "  Its log will therefore look normal. Confirm with:"
+  echo "    sudo journalctl -k | grep -i denied"
+  echo "  Lines like  apparmor=\"DENIED\" operation=\"file_perm\" class=\"net\""
+  echo "  profile=\"snap.${snap}....\"  requested=\"send\"  - often for OTHER snaps"
+  echo "  too - are a known problem of snapd 2.77.1 on some Linux kernels, not of"
+  echo "  WeKan: https://bugs.launchpad.net/snapd/+bug/2169038"
+  echo "  Fix it by updating the kernel and rebooting:"
+  echo "    sudo apt update && sudo apt full-upgrade && sudo reboot"
+  echo "  or, until the kernel can be updated, go back to the previous snapd:"
+  echo "    sudo snap revert snapd"
+  echo "    sudo snap restart ${snap}"
+}

snap-src/bin/wekan-control

@@ -474,6 +474,26 @@ fi
 # and nothing phoning home. db-eval speaks the wire protocol to both FerretDB v1
 # and MongoDB 6/7.
 DB_EVAL="$SNAP/bin/db-eval"
+# #6746: socket_denied / socket_denied_hint - tell a connection the KERNEL refused
+# (AppArmor, snapd 2.77.1 on some kernels) from a database that is not up yet.
+# shellcheck source=/dev/null
+[ -f "$SNAP/bin/socket-denied" ] && source "$SNAP/bin/socket-denied"
+# Prints the AppArmor explanation once per wait loop, as soon as the first
+# refused connection shows it - not after the two-minute timeout, and not in
+# a log that otherwise only says "not ready yet".
+show_socket_denied() {
+    declare -F socket_denied >/dev/null || return 1
+    socket_denied "$2" || return 1
+    echo ""
+    echo "========================================================================"
+    echo "WeKan: cannot talk to ${1} on ${MONGO_HOST}:${MONGO_PORT}:"
+    echo "  $2"
+    echo ""
+    socket_denied_hint
+    echo "========================================================================"
+    echo ""
+    return 0
+}
 MONGO_PORT="${MONGODB_PORT:-27019}"
 MONGO_HOST="${MONGODB_BIND_IP:-127.0.0.1}"
 # Bracketed for the same reason as MONGO_URL above (#6550): the readiness probe
@@ -522,7 +542,13 @@ if [ "ferretdb" = "${DATABASE}" ]; then
     WEKAN_DB_WAIT_TIMEOUT=${WEKAN_DB_WAIT_TIMEOUT:-120}
     db_waited=0
     db_hint_shown=false
-    until "$DB_EVAL" ping "$DB_URL" >/dev/null 2>&1; do
+    db_denied_shown=false
+    # #6746: keep the driver's error from every attempt, so a refusal by the
+    # kernel is recognised on the first try instead of after the timeout.
+    until _ping_err="$("$DB_EVAL" ping "$DB_URL" 2>&1 >/dev/null)"; do
+        if [ "$db_denied_shown" != "true" ] && show_socket_denied FerretDB "$_ping_err"; then
+            db_denied_shown=true
+        fi
         if [ "$db_hint_shown" != "true" ] && [ "$db_waited" -ge "$WEKAN_DB_WAIT_TIMEOUT" ]; then
             echo ""
             echo "========================================================================"
@@ -544,12 +570,11 @@ if [ "ferretdb" = "${DATABASE}" ]; then
             echo "  - FerretDB left disabled/stopped (a failed migration hand-off):"
             echo "      snap start --enable ${WEKAN_SNAP_SVC}.ferretdb"
             echo ""
-            # #6746: the ping above discards its error, so the log said only
-            # "not ready" - "permission denied" and "connection refused" looked
-            # the same. Ask once more and show what the driver says. (The
-            # `database` setting this used to suggest is gone: the configure
-            # hook unsets it, so that advice changed nothing.)
-            _ping_err="$("$DB_EVAL" ping "$DB_URL" 2>&1 >/dev/null)"
+            # #6746: the log used to say only "not ready" - "permission denied"
+            # and "connection refused" looked the same. Show what the driver
+            # said on the last attempt. (The `database` setting this used to
+            # suggest is gone: the configure hook unsets it, so that advice
+            # changed nothing.)
             if [ -n "$_ping_err" ]; then
                 echo "The connection attempt says:"
                 echo "  $_ping_err"
@@ -592,7 +617,11 @@ elif [ "true" != "${DISABLE_MONGODB}" ]; then
     WEKAN_DB_WAIT_TIMEOUT=${WEKAN_DB_WAIT_TIMEOUT:-120}
     db_waited=0
     db_hint_shown=false
-    until "$DB_EVAL" primary "$DB_URL" >/dev/null 2>&1; do
+    db_denied_shown=false
+    until _primary_err="$("$DB_EVAL" primary "$DB_URL" 2>&1 >/dev/null)"; do
+        if [ "$db_denied_shown" != "true" ] && show_socket_denied MongoDB "$_primary_err"; then
+            db_denied_shown=true
+        fi
         if [ "$db_hint_shown" != "true" ] && [ "$db_waited" -ge "$WEKAN_DB_WAIT_TIMEOUT" ]; then
             echo ""
             echo "========================================================================"

snap-src/bin/wekan-read-settings

@@ -13,7 +13,15 @@ do
     default_value="DEFAULT_$key"
     description="DESCRIPTION_$key"
     snappy_key="KEY_$key"
-    value=$(snapctl get ${!snappy_key})
+    # #6746: a key listed in $keys without its KEY_ name made this run a bare
+    # `snapctl get`, which prints "error: snapctl: get which option?" into every
+    # service's log (four of them, for the SAML_* settings) and could never
+    # read the setting. Such a key falls back to its default instead; the test
+    # snapSettingsKeys keeps the list and the names in step.
+    value=""
+    if [ -n "${!snappy_key}" ]; then
+        value=$(snapctl get "${!snappy_key}")
+    fi
     if [ "x$value" == "x" ]; then
         echo -e "$key=${!default_value} (default value)"
         export $key="${!default_value}"

tests/ferretdbWaitTimeout.test.cjs

@@ -61,10 +61,14 @@ test('the hint points to the FerretDB service log and the SQLite dir', () => {
     'must not suggest the removed database setting');
   assert.ok(/_ping_err="\$\("\$DB_EVAL" ping "\$DB_URL" 2>&1 >\/dev\/null\)"/.test(ferretBranch),
     'must show why the connection failed, not only that it did');
+  assert.ok(/echo "  \$_ping_err"/.test(ferretBranch),
+    'the hint must print the error of the last attempt');
 });
 
 test('it still keeps retrying after the hint (does not give up / crash)', () => {
-  assert.ok(/until "\$DB_EVAL" ping "\$DB_URL"/.test(ferretBranch),
+  // #6746: the loop keeps each attempt's error so a kernel refusal is named
+  // on the first attempt (see tests/snapSocketDenied.test.cjs).
+  assert.ok(/until _ping_err="\$\("\$DB_EVAL" ping "\$DB_URL" 2>&1 >\/dev\/null\)"; do/.test(ferretBranch),
     'must keep polling FerretDB readiness');
   assert.ok(/FerretDB not ready yet, retrying in 5 seconds/.test(ferretBranch));
   assert.ok(/\n\s*sleep 5\n/.test(ferretBranch));
@@ -77,7 +81,7 @@ test('it still keeps retrying after the hint (does not give up / crash)', () =>
 // no timeout bookkeeping).
 test('NEGATIVE: the FerretDB wait is not a silent infinite loop', () => {
   const loopBody = ferretBranch.slice(
-    ferretBranch.indexOf('until "$DB_EVAL" ping'),
+    ferretBranch.indexOf('until _ping_err="$("$DB_EVAL" ping'),
     ferretBranch.indexOf('echo "FerretDB is ready."'),
   );
   assert.ok(loopBody.includes('WEKAN_DB_WAIT_TIMEOUT'),

tests/snapDatabaseRestore.test.cjs

@@ -19,7 +19,8 @@ function test(name, fn) {
 
 // db: undefined = no db-eval in the snap (older layout); 'up' = answers at
 // once; 'after-start' = answers once snapctl started a service; 'never' =
-// never answers, failing the way #6746's reporter saw.
+// nothing listens and nothing ever will; 'denied' = the kernel refuses the
+// write on an accepted connection, exactly as #6746's reporter saw.
 function fixture({ db } = {}) {
   const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'wekan-restore-test-'));
   const snap = path.join(dir, 'snap fixture');
@@ -30,7 +31,7 @@ function fixture({ db } = {}) {
   fs.mkdirSync(fakeBin, { recursive: true });
   fs.mkdirSync(common, { recursive: true });
   fs.mkdirSync(hostBin, { recursive: true });
-  for (const name of ['database-ready', 'database-role']) {
+  for (const name of ['database-ready', 'database-role', 'socket-denied']) {
     fs.copyFileSync(path.join(root, 'snap-src', 'bin', name), path.join(snap, 'bin', name));
   }
   fs.writeFileSync(
@@ -48,12 +49,17 @@ function fixture({ db } = {}) {
     );
   }
   if (db) {
-    const answers = { up: 'true', 'after-start': '[ -f "$DB_STARTED" ]', never: 'false' }[db];
+    const answers = {
+      up: 'true', 'after-start': '[ -f "$DB_STARTED" ]', never: 'false', denied: 'false',
+    }[db];
+    const error = db === 'denied'
+      ? 'db-eval ping: connection(127.0.0.9:27099[-62]) unable to write wire message to network: write tcp 127.0.0.1:53014->127.0.0.9:27099: write: permission denied'
+      : 'db-eval ping: connect ECONNREFUSED 127.0.0.9:27099';
     fs.writeFileSync(
       path.join(snap, 'bin', 'db-eval'),
       '#!/bin/bash\n' +
       `if ${answers}; then exit 0; fi\n` +
-      'echo "db-eval ping: connection(127.0.0.9:27099[-62]) unable to write wire message to network: write tcp 127.0.0.1:53014->127.0.0.9:27099: write: permission denied" >&2\n' +
+      `echo "${error}" >&2\n` +
       'exit 1\n',
       { mode: 0o755 },
     );
@@ -192,9 +198,37 @@ test('negative #6746: a database that never answers stops restore with the reaso
     assert.strictEqual(result.status, 1);
     assert.strictEqual(fs.existsSync(f.capture), false, 'mongorestore must not run');
     assert.match(result.stderr, /wekan\.ferretdb\) does not answer on 127\.0\.0\.9:27099/);
-    assert.match(result.stderr, /write: permission denied/, 'the driver error must be shown');
-    assert.match(result.stderr, /security policy/, 'permission denied must be explained');
+    assert.match(result.stderr, /ECONNREFUSED/, 'the driver error must be shown');
     assert.match(result.stderr, /sudo snap logs -n 100 wekan\.ferretdb/);
+    // A database that is simply not up is not an AppArmor problem: the
+    // snapd/kernel advice must not be given for it.
+    assert.doesNotMatch(result.stderr, /AppArmor|snap revert snapd/);
+  });
+});
+
+test('#6746 a connection the kernel refuses is explained at once, without starting anything', () => {
+  withFixture({ db: 'denied' }, (f) => {
+    const archive = path.join(f.dir, 'wekan.backup');
+    fs.writeFileSync(archive, 'fixture');
+    const result = run(f, [archive]);
+    assert.strictEqual(result.status, 1);
+    assert.strictEqual(fs.existsSync(f.capture), false, 'mongorestore must not run');
+    // The connect succeeded, so the database is running: starting it again
+    // and waiting for it is the 90 seconds this replaces.
+    assert.deepStrictEqual(snapctlCalls(f), []);
+    assert.doesNotMatch(result.stderr, /does not answer .* after/);
+    assert.match(result.stderr, /write: permission denied/, 'the driver error must be shown');
+    assert.match(result.stderr, /AppArmor/);
+    assert.match(result.stderr, /operation="file_perm" class="net"/);
+    assert.match(result.stderr, /bugs\.launchpad\.net\/snapd\/\+bug\/2169038/);
+    assert.match(result.stderr, /sudo snap revert snapd/);
+    assert.match(result.stderr, /sudo apt full-upgrade/);
+  });
+  withFixture({ db: 'denied' }, (f) => {
+    const result = run(f, [path.join(f.dir, 'x.backup')], backup);
+    assert.strictEqual(result.status, 1);
+    assert.strictEqual(fs.existsSync(f.capture), false, 'mongodump must not run');
+    assert.match(result.stderr, /sudo snap revert snapd/);
   });
 });
 

tests/snapSettingsKeys.test.cjs

@@ -0,0 +1,114 @@
+'use strict';
+
+// wekan/wekan#6746: every WeKan snap service log began with
+//   error: snapctl: get which option?
+// four times. snap-src/bin/config lists the settings in $keys and maps each one
+// to its `snap set` name in KEY_<name>; four SAML settings (SAML_IDP_PROFILE,
+// SAML_WANT_RESPONSE_SIGNED, SAML_WANT_ASSERTIONS_SIGNED, SAML_LOGIN_FLOW) were
+// added to $keys without a KEY_, so wekan-read-settings ran a bare
+// `snapctl get` for them - an error in every log, and settings that
+// `snap set wekan saml-login-flow=redirect` could never reach.
+//
+// Run: node tests/snapSettingsKeys.test.cjs
+
+const assert = require('assert');
+const fs = require('fs');
+const os = require('os');
+const path = require('path');
+const { spawnSync } = require('child_process');
+
+const root = path.join(__dirname, '..');
+const bin = path.join(root, 'snap-src', 'bin');
+
+let passed = 0;
+function test(name, fn) {
+  fn();
+  passed += 1;
+  console.log('  ok -', name);
+}
+
+function readConfig(configPath) {
+  const r = spawnSync('bash', ['-c', `
+    source "${configPath}"
+    for k in $keys; do
+      kv="KEY_$k"; dv="DEFAULT_$k"; sv="DESCRIPTION_$k"
+      printf '%s\\t%s\\t%s\\t%s\\n' "$k" "\${!kv}" "\${!dv+set}" "\${!sv+set}"
+    done`], { encoding: 'utf8' });
+  assert.strictEqual(r.status, 0, r.stderr);
+  return r.stdout.trim().split('\n').map((line) => {
+    const [name, key, hasDefault, hasDescription] = line.split('\t');
+    return { name, key, hasDefault, hasDescription };
+  });
+}
+
+test('every listed setting has a snap key, a default and a description', () => {
+  const rows = readConfig(path.join(bin, 'config'));
+  assert.ok(rows.length > 200, 'the settings list must have been read');
+  const missing = rows.filter((r) => !r.key).map((r) => `KEY_${r.name}`)
+    .concat(rows.filter((r) => !r.hasDefault).map((r) => `DEFAULT_${r.name}`))
+    .concat(rows.filter((r) => !r.hasDescription).map((r) => `DESCRIPTION_${r.name}`));
+  assert.deepStrictEqual(missing, []);
+});
+
+test('the SAML settings from #6746 are reachable with snap set', () => {
+  const byName = Object.fromEntries(readConfig(path.join(bin, 'config')).map((r) => [r.name, r.key]));
+  assert.strictEqual(byName.SAML_IDP_PROFILE, 'saml-idp-profile');
+  assert.strictEqual(byName.SAML_WANT_RESPONSE_SIGNED, 'saml-want-response-signed');
+  assert.strictEqual(byName.SAML_WANT_ASSERTIONS_SIGNED, 'saml-want-assertions-signed');
+  assert.strictEqual(byName.SAML_LOGIN_FLOW, 'saml-login-flow');
+});
+
+test('negative: no two settings share a snap key', () => {
+  const seen = new Map();
+  for (const { name, key } of readConfig(path.join(bin, 'config'))) {
+    assert.ok(!seen.has(key), `${name} and ${seen.get(key)} both use ${key}`);
+    seen.set(key, name);
+  }
+});
+
+// Run the real wekan-read-settings against a stand-in snapctl that fails the
+// way the real one does when called without a name.
+function readSettings(configText) {
+  const dir = fs.mkdtempSync(path.join(process.env.TMPDIR || os.tmpdir(), 'wekan-settings-'));
+  try {
+    const snap = path.join(dir, 'snap');
+    const hostBin = path.join(dir, 'host-bin');
+    fs.mkdirSync(path.join(snap, 'bin'), { recursive: true });
+    fs.mkdirSync(hostBin);
+    fs.copyFileSync(path.join(bin, 'wekan-read-settings'), path.join(snap, 'bin', 'wekan-read-settings'));
+    fs.writeFileSync(path.join(snap, 'bin', 'config'), configText);
+    fs.writeFileSync(path.join(hostBin, 'snapctl'),
+      '#!/bin/bash\n' +
+      'if [ "$1" = get ] && [ -z "$2" ]; then echo "error: snapctl: get which option?" >&2; exit 1; fi\n' +
+      'if [ "$1" = get ] && [ "$2" = saml-login-flow ]; then echo redirect; fi\n',
+      { mode: 0o755 });
+    return spawnSync('bash', ['-c', `source "$SNAP/bin/wekan-read-settings"; echo "RESULT=$SAML_LOGIN_FLOW|$UNMAPPED"`], {
+      encoding: 'utf8',
+      env: { ...process.env, SNAP: snap, SNAP_INSTANCE_NAME: 'wekan', PATH: `${hostBin}:${process.env.PATH}` },
+    });
+  } finally {
+    fs.rmSync(dir, { recursive: true, force: true });
+  }
+}
+
+test('the real config reads without a single snapctl error, and a set value wins', () => {
+  const r = readSettings(fs.readFileSync(path.join(bin, 'config'), 'utf8'));
+  assert.strictEqual(r.status, 0, r.stderr);
+  assert.doesNotMatch(r.stderr, /get which option/);
+  assert.match(r.stdout, /^SAML_LOGIN_FLOW=redirect$/m);
+  assert.match(r.stdout, /RESULT=redirect\|$/m);
+});
+
+test('negative: a setting listed without a KEY_ falls back to its default, silently', () => {
+  const r = readSettings(
+    'keys="SAML_LOGIN_FLOW UNMAPPED"\n' +
+    'DEFAULT_SAML_LOGIN_FLOW=""\nKEY_SAML_LOGIN_FLOW="saml-login-flow"\nDESCRIPTION_SAML_LOGIN_FLOW="x"\n' +
+    'DEFAULT_UNMAPPED="fallback"\nDESCRIPTION_UNMAPPED="x"\n',
+  );
+  assert.strictEqual(r.status, 0, r.stderr);
+  assert.doesNotMatch(r.stderr, /get which option/);
+  assert.match(r.stdout, /^UNMAPPED=fallback \(default value\)$/m);
+  assert.match(r.stdout, /RESULT=redirect\|fallback$/m);
+});
+
+console.log(`\nsnapSettingsKeys: all ${passed} tests passed`);

tests/snapSocketDenied.test.cjs

@@ -0,0 +1,144 @@
+'use strict';
+
+// wekan/wekan#6746: on the reporter's computer every WeKan snap command that
+// talked to its own database failed with
+//   write tcp 127.0.0.1:53014->127.0.0.1:27019: write: permission denied
+// and `journalctl -k` held 1300 lines of
+//   apparmor="DENIED" operation="file_perm" class="net" profile="snap.wekan.wekan"
+//   ... fport=27019 ... requested="send" denied="send"
+// - with the SAME denial for snap-store and standard-notes. FerretDB was
+// running (the connection was accepted); the kernel refused the first write.
+// That is snapd 2.77.1 on a kernel with an AppArmor socket mediation bug
+// (https://bugs.launchpad.net/snapd/+bug/2169038), fixed by a newer kernel and
+// worked around by `snap revert snapd`. WeKan cannot lift the denial; what it
+// can do is say so at once instead of "FerretDB not ready yet" for two minutes
+// followed by advice to read FerretDB's (clean) log.
+//
+// Run: node tests/snapSocketDenied.test.cjs
+
+const assert = require('assert');
+const fs = require('fs');
+const path = require('path');
+const { spawnSync } = require('child_process');
+
+const root = path.join(__dirname, '..');
+const bin = path.join(root, 'snap-src', 'bin');
+const helper = path.join(bin, 'socket-denied');
+const control = fs.readFileSync(path.join(bin, 'wekan-control'), 'utf8');
+
+let passed = 0;
+function test(name, fn) {
+  fn();
+  passed += 1;
+  console.log('  ok -', name);
+}
+
+function bash(script, env = {}) {
+  return spawnSync('bash', ['-c', script], {
+    encoding: 'utf8',
+    env: { ...process.env, SNAP_INSTANCE_NAME: 'wekan', ...env },
+  });
+}
+
+function denied(err) {
+  return bash(`source "${helper}"; socket_denied "$ERR"`, { ERR: err }).status === 0;
+}
+
+// The exact errors the reporter posted: mongorestore (Go driver) and db-eval
+// (Node.js driver) word the same refusal differently.
+const REPORTED = [
+  'error connecting to host: failed to connect to mongodb://127.0.0.1:27019/: server selection error: context deadline exceeded, current topology: { Type: Single, Servers: [{ Addr: 127.0.0.1:27019, Type: Unknown, Last error:  connection(127.0.0.1:27019[-62]) unable to write wire message to network: write tcp 127.0.0.1:53014->127.0.0.1:27019: write: permission denied }, ] }',
+  'db-eval ping: connect EACCES 127.0.0.1:27019',
+  'db-eval ping: write EPERM',
+];
+
+test('the refusals from the report are recognised', () => {
+  for (const err of REPORTED) assert.ok(denied(err), err);
+});
+
+test('negative: a database that is merely not up is not called an AppArmor problem', () => {
+  for (const err of [
+    '',
+    'db-eval ping: connect ECONNREFUSED 127.0.0.1:27019',
+    'db-eval ping: Server selection timed out after 5000 ms',
+    'db-eval ping: Authentication failed.',
+    'db-eval ping: connection <monitor> to 127.0.0.1:27019 closed',
+  ]) {
+    assert.ok(!denied(err), `must not match: ${JSON.stringify(err)}`);
+  }
+});
+
+test('the explanation names the denial, the upstream bug, and both ways out', () => {
+  const r = bash(`source "${helper}"; socket_denied_hint`);
+  assert.strictEqual(r.status, 0, r.stderr);
+  assert.match(r.stdout, /journalctl -k \| grep -i denied/);
+  assert.match(r.stdout, /operation="file_perm" class="net"/);
+  assert.match(r.stdout, /profile="snap\.wekan\./);
+  assert.match(r.stdout, /snapd 2\.77\.1/);
+  assert.match(r.stdout, /https:\/\/bugs\.launchpad\.net\/snapd\/\+bug\/2169038/);
+  assert.match(r.stdout, /sudo apt update && sudo apt full-upgrade && sudo reboot/);
+  assert.match(r.stdout, /sudo snap revert snapd/);
+  assert.match(r.stdout, /sudo snap restart wekan/);
+  // A parallel install (wekan_2) is told to restart ITSELF.
+  const r2 = bash(`source "${helper}"; socket_denied_hint`, { SNAP_INSTANCE_NAME: 'wekan_2' });
+  assert.match(r2.stdout, /sudo snap restart wekan_2/);
+});
+
+// wekan-control: both readiness loops keep each attempt's error and report a
+// kernel refusal on the FIRST attempt, once.
+function loop(head) {
+  const at = control.indexOf(head);
+  assert.ok(at > 0, `missing loop: ${head}`);
+  return control.slice(at, control.indexOf('\n    done\n', at));
+}
+
+test('wekan-control sources the helper and both wait loops use it', () => {
+  assert.match(control, /\[ -f "\$SNAP\/bin\/socket-denied" \] && source "\$SNAP\/bin\/socket-denied"/);
+  const ferret = loop('until _ping_err="$("$DB_EVAL" ping "$DB_URL" 2>&1 >/dev/null)"; do');
+  assert.match(ferret, /show_socket_denied FerretDB "\$_ping_err"/);
+  const mongo = loop('until _primary_err="$("$DB_EVAL" primary "$DB_URL" 2>&1 >/dev/null)"; do');
+  assert.match(mongo, /show_socket_denied MongoDB "\$_primary_err"/);
+  for (const body of [ferret, mongo]) {
+    assert.match(body, /\[ "\$db_denied_shown" != "true" \] && show_socket_denied/);
+    assert.match(body, /db_denied_shown=true/);
+    // Not behind the two-minute timeout: that is the delay being removed.
+    const call = body.indexOf('show_socket_denied');
+    const timeout = body.indexOf('-ge "$WEKAN_DB_WAIT_TIMEOUT"');
+    assert.ok(call > 0 && timeout > call, 'the refusal must be named before the timeout hint');
+  }
+});
+
+test('show_socket_denied prints only for a refusal, and says so in the log', () => {
+  const start = control.indexOf('show_socket_denied() {');
+  const fn = control.slice(start, control.indexOf('\n}\n', start) + 3);
+  const run = (err) => bash(
+    `source "${helper}"; MONGO_HOST=127.0.0.1; MONGO_PORT=27019\n${fn}\nshow_socket_denied FerretDB "$ERR"`,
+    { ERR: err },
+  );
+  const yes = run(REPORTED[0]);
+  assert.strictEqual(yes.status, 0);
+  assert.match(yes.stdout, /WeKan: cannot talk to FerretDB on 127\.0\.0\.1:27019:/);
+  assert.match(yes.stdout, /write: permission denied/);
+  assert.match(yes.stdout, /sudo snap revert snapd/);
+  const no = run('db-eval ping: connect ECONNREFUSED 127.0.0.1:27019');
+  assert.strictEqual(no.status, 1);
+  assert.strictEqual(no.stdout, '');
+});
+
+test('negative: no snap script explains "permission denied" on its own any more', () => {
+  // One explanation, in one file: a second, vaguer copy is how the
+  // "refused by this system's security policy" line came to say nothing about
+  // what to do. Every script that talks to the database goes through
+  // socket-denied for it.
+  for (const name of fs.readdirSync(bin)) {
+    if (name === 'socket-denied') continue;
+    const text = fs.readFileSync(path.join(bin, name), 'utf8');
+    assert.ok(!/\*"permission denied"\*/.test(text),
+      `${name} must use socket_denied, not its own "permission denied" match`);
+    if (/socket_denied/.test(text)) {
+      assert.match(text, /source "\$SNAP\/bin\/socket-denied"/, `${name} must source the helper`);
+    }
+  }
+});
+
+console.log(`\nsnapSocketDenied: all ${passed} tests passed`);