Snap Launchpad: retry a stopped build with a new build, and name a build-farm proxy failure.

The v12.21 s390x snap failed. Attempt 1's Launchpad build stopped
because the build farm's own package proxy answered "501 Gateway error"
for libyaml-cpp0.8_0.8.0+dfsg-6build1_s390x.deb while every neighbouring
package downloaded. Attempt 2 submitted a new build that also stopped,
and snapcraft crashed fetching its log (MissingSchema "Invalid URL
'None'"). Attempt 3 then ran `remote-build --recover`, which reconnects to
that SAME stopped build, and was over in eight seconds: a finished build
cannot be recovered into a snap. Each attempt also reprinted every
earlier attempt's log, so attempt 1's failure appeared again under
attempt 2.

Now an attempt whose build ended "Stopped:"/"Failed:"/"Cancelled:" for
the architecture makes the next attempt submit a new build; recovery is
still what reconnects after a TLS/polling error or an auth failure, as
before. Each attempt prints only the build logs it downloaded, and says
so when it got none. A 5xx from the build farm's package proxy is named
as Launchpad infrastructure with the package that failed, so the next
step is a re-run rather than an edit of snapcraft.yaml.

tests/releaseSnapRecovery.test.cjs runs the real step with a stand-in
snapcraft: a stopped build is retried without --recover and then builds;
negatively, three stopped builds with the proxy error fail with the
infrastructure message, print attempt 1's log once, and are not called a
deterministic stage-package failure. Both new tests fail on the previous
workflow. Not run on GitHub Actions or Launchpad here.

Thanks to xet7 !

.github/workflows/release-all.yml

@@ -4785,19 +4785,41 @@ jobs:
           }
 
           attempts=3
+          # Set once an attempt ends with Launchpad reporting the build as
+          # finished without a snap ("Stopped:" / "Failed:" for this arch).
+          # Recovery reconnects to that SAME build, so it can only download the
+          # same failure again: v12.21 s390x attempt 3 "recovered" attempt 2's
+          # stopped build and was over in eight seconds. A finished build is
+          # retried by submitting a fresh one.
+          fresh_next=0
+          attempt_marker="$PWD/.tools/tmp/remote-build-attempt.start"
           for i in $(seq 1 "$attempts"); do
             # A partial file from a previous attempt must not be mistaken for
             # this attempt's artifact: snapcraft does not re-download a name it
             # already has, so the truncated one would be uploaded again.
             rm -f wekan_${VERSION}_${{ matrix.arch }}*.snap *_${{ matrix.arch }}.snap
             echo "::group::Remote build ${{ matrix.arch }} attempt $i/$attempts"
-            # Always try recovery first, including a re-run of this matrix job.
-            # start_builds replaces the recipe; it does NOT resume polling after
-            # an SSL EOF. Only an explicitly absent recipe/repository permits a
-            # new submission. Authentication/network failures are not absence.
-            run_remote_build --recover && rc=0 || rc=$?
-            if [ "$rc" -ne 0 ] && grep -qE 'Could not find (snap recipe|repository at path)' "$remote_log"; then
+            # Build logs this attempt downloads are newer than this marker; the
+            # earlier attempts' logs stay in the directory for the diagnostics
+            # below, but must not be printed again as if they were this one's.
+            touch "$attempt_marker"
+            if [ "$fresh_next" = 1 ]; then
+              echo "The previous attempt's Launchpad build finished without a snap; submitting a new build instead of recovering it."
               run_remote_build && rc=0 || rc=$?
+            else
+              # Otherwise try recovery first, including a re-run of this matrix
+              # job. start_builds replaces the recipe; it does NOT resume polling
+              # after an SSL EOF. Only an explicitly absent recipe/repository
+              # permits a new submission. Authentication/network failures are not
+              # absence.
+              run_remote_build --recover && rc=0 || rc=$?
+              if [ "$rc" -ne 0 ] && grep -qE 'Could not find (snap recipe|repository at path)' "$remote_log"; then
+                run_remote_build && rc=0 || rc=$?
+              fi
+            fi
+            fresh_next=0
+            if grep -qE "^(Stopped|Failed|Cancelled): ${{ matrix.arch }}\$" "$remote_log"; then
+              fresh_next=1
             fi
             echo "::endgroup::"
             # A real glob, so nullglob can empty it, plus any name Launchpad chose -
@@ -4835,7 +4857,10 @@ jobs:
             # and it is downloaded here. Print it whenever there is no snap - not only
             # when snapcraft's exit code is non-zero, which is how v10.48 fetched it
             # and threw it away.
+            new_logs=0
             for log in snapcraft-wekan-*.txt; do
+              [ -e "$log" ] && [ "$log" -nt "$attempt_marker" ] || continue
+              new_logs=1
               echo "::group::Launchpad build log: $log (what failed, then the last 150 lines)"
               # The tail alone is not where the cause is. snapcraft's own error is
               # followed by lpbuildd's Python traceback, the proxy-token revocation
@@ -4848,6 +4873,12 @@ jobs:
               tail -n 150 "$log"
               echo "::endgroup::"
             done
+            if [ "$new_logs" = 0 ] && ls snapcraft-wekan-*.txt >/dev/null 2>&1; then
+              # v12.21 s390x: snapcraft crashed fetching attempt 2's log
+              # (MissingSchema "Invalid URL 'None'"), and attempt 1's log was
+              # printed in its place, reading as the same failure twice.
+              echo "Attempt $i downloaded no Launchpad build log of its own; the logs printed earlier belong to earlier attempts."
+            fi
             # ...and there is no Launchpad build log at all when the build never
             # STARTED. v10.55 riscv64 failed three times on
             #   "Git operation failed with: Could not push 'HEAD' to
@@ -4881,6 +4912,16 @@ jobs:
           # stage-package as a "Stopped" build with no artifact, which looks
           # identical to a cancelled one from the outside; the difference is one
           # line in the log, so say it here rather than leave it to be found.
+          # The build farm's own package proxy failing is infrastructure, not
+          # WeKan: v12.21 s390x stopped on
+          #   GET http://ftpmaster.internal/.../libyaml-cpp0.8_..._s390x.deb ... 501
+          #   Failed to fetch package: ... 501  Gateway error [IP: 10.10.10.1 8222]
+          # while every neighbouring package downloaded. Name it, so the next
+          # step is waiting and re-running rather than editing snapcraft.yaml.
+          if grep -qsE 'could not be fetched: 5[0-9][0-9] |Gateway error \[IP: ' snapcraft-wekan-*.txt; then
+            proxypkg="$(grep -hs -oE "could not be fetched: 5[0-9][0-9] [^[]*|/download/[^ ']*\.deb" snapcraft-wekan-*.txt | sed 's#^/download/##' | sort -u | tr '\n' ' ')"
+            echo "::error::Launchpad's build-farm package proxy failed (HTTP 5xx) while the ${{ matrix.arch }} build downloaded its stage-packages: ${proxypkg}. That is Launchpad infrastructure, not WeKan, and snapcraft.yaml needs no change; re-run this job (or Release All Missing) later. Builder status: https://launchpad.net/builders" >&2
+          fi
           if grep -qs 'Stage package not found' snapcraft-wekan-*.txt; then
             missingpkg="$(grep -hs -o "Stage package not found in part '[^']*': [^ ]*" snapcraft-wekan-*.txt | sort -u | tr '\n' ' ')"
             echo "::error::This is NOT a transient failure and re-running will not fix it: ${missingpkg}. A stage-package that resolves on the 64-bit architectures can be missing on ${{ matrix.arch }} - Ubuntu 24.04's 64-bit time_t transition renamed libraries to a t64 suffix, and the renamed package carries 'Provides: <old name>' on amd64/arm64 but NOT on armhf, where the ABI really changed. Name the real package in snapcraft.yaml (libssl3 -> libssl3t64, libcurl4 -> libcurl4t64, libgoogle-perftools4 -> libgoogle-perftools4t64). Check a name against the archive: curl -fsSL http://ports.ubuntu.com/ubuntu-ports/dists/noble/main/binary-armhf/Packages.gz | gunzip -c | grep '^Package: <name>$'" >&2

tests/releaseSnapRecovery.test.cjs

@@ -36,6 +36,10 @@ snapcraft() {
   pending:*|budget:2) echo 'Pending: '$ARCH; return 124 ;;
   denied:*) echo 'Recovering build hash'; echo 'HTTP 401 Unauthorized'; return 1 ;;
   bad:*) echo 'Monitoring build'; printf 'oops' > "wekan_11.85_\${ARCH}.snap"; return 0 ;;
+  stopped:1|proxy:1) echo 'Starting new build.'; echo 'Monitoring build'; echo "Stopped: $ARCH";
+    printf 'Downloading package: libyaml-cpp0.8\nFailed to fetch package: The item %s could not be fetched: 501  Gateway error [IP: 10.10.10.1 8222].\nBuild failed\n' \
+      "'/root/.cache/snapcraft/download/libyaml-cpp0.8_0.8.0+dfsg-6build1_\${ARCH}.deb'" > "snapcraft-wekan-hash_\${ARCH}_1.txt"; return 0 ;;
+  proxy:*) echo 'Starting new build.'; echo "Stopped: $ARCH"; echo "snapcraft internal error: MissingSchema(\"Invalid URL 'None'\")"; return 70 ;;
  esac
  truncate -s 52428800 "wekan_11.85_\${ARCH}.snap"
  printf hsqs | dd of="wekan_11.85_\${ARCH}.snap" conv=notrunc status=none
@@ -72,6 +76,29 @@ test('only an explicitly absent recipe permits submitting a new build', () => {
   assert.ok(denied.calls.every(call => call.endsWith('--recover')));
   assert.doesNotMatch(denied.output, /built=true|pending=true/);
 });
+test('#v12.21 s390x: a build that ended Stopped is retried with a NEW build, not recovered', () => {
+  const result = remote('stopped', 's390x');
+  assert.equal(result.status, 0, result.stderr);
+  assert.equal(result.calls.length, 2, result.calls.join('\n'));
+  assert.match(result.calls[0], /--recover$/);
+  assert.doesNotMatch(result.calls[1], /--recover/, 'a finished build cannot be recovered into a snap');
+  assert.match(result.stdout, /submitting a new build instead of recovering it/);
+  assert.match(result.output, /built=true/);
+});
+test('negative: a Launchpad proxy 5xx is named as infrastructure, and old logs are not reprinted', () => {
+  const result = remote('proxy', 's390x');
+  assert.equal(result.status, 1);
+  assert.equal(result.calls.length, 3);
+  assert.match(result.calls[0], /--recover$/);
+  for (const call of result.calls.slice(1)) assert.doesNotMatch(call, /--recover/);
+  assert.match(result.stderr, /build-farm package proxy failed \(HTTP 5xx\).*libyaml-cpp0\.8_0\.8\.0\+dfsg-6build1_s390x\.deb/);
+  assert.match(result.stderr, /snapcraft\.yaml needs no change/);
+  // Attempt 1's log is printed once; attempts 2 and 3 fetched none of their own.
+  assert.equal((result.stdout.match(/::group::Launchpad build log: /g) || []).length, 1, result.stdout);
+  assert.equal((result.stdout.match(/downloaded no Launchpad build log of its own/g) || []).length, 2);
+  assert.doesNotMatch(result.stderr, /This is NOT a transient failure/);
+  assert.doesNotMatch(result.output, /built=true|pending=true/);
+});
 test('all Launchpad architectures stop waiting cleanly without claiming a build', () => {
   for (const arch of ['armhf', 'ppc64el', 's390x', 'riscv64']) {
     const result = remote('pending', arch);