wal-restore fails with "Cannot create /var/tmp/walrestore directory: Read-only file system"

[leonardoce]

The plugin-barman-cloud native sidecar (initContainer: plugin-barman-cloud)
fails some WAL restore of an existing object with:

barman-cloud-wal-restore: ERROR: Cannot create /var/tmp/walrestore directory: Errno 30 Read-only file system: '/var/tmp/walrestore'
connectivity failure while executing barman-cloud-wal-restore, retrying: connectivity failure

PostgreSQL's restore_command therefore does not succeeds.

Environment

• plugin-barman-cloud: v0.15.0 (stock manifest)
• CNPG (operator + API): 1.30.0
• Kubernetes: v1.37.0 on Kind (v0.33.0), single-node CI cluster
• Object store: MinIO (S3-compatible), one bucket per CNPG cluster,
  referenced via ObjectStore + serverName
• Topology: 3 single-instance CNPG replica clusters recovering purely
  through the plugin (no TCP streaming). No backup/schedule involved in
  the failure — this is steady-state WAL restore on a running replica.

Excerpt (one of 33 identical cycles; ...007 fails, the alternating
00000002.history request returns a clean "does not exist"):

{"msg":"Restoring WAL file","objectStore":"e2e-switchover-archiving-dc-a","serverName":"cluster-dc-a","walName":"000000010000000000000007"}
{"logger":"barman-cloud-wal-restore","msg":"... ERROR: Cannot create /var/tmp/walrestore directory: Errno 30 Read-only file system: '/var/tmp/walrestore'"}
{"msg":"WAL file not found in the recovery object store",...,"error":"connectivity failure while executing barman-cloud-wal-restore, retrying: connectivity failure"}

[mnencia]

Without --parallel, barman-cloud-wal-restore lists the bucket using the requested WAL's full path as prefix. That also matches <wal>.partial, since it starts with the same string. _get_wals_to_download() accepts both as "the requested WAL" (is_requested_wal matches wal_name or wal_name + ".partial"), and this check skips the --parallel count gate entirely.

During a switchover, CNPG's replicaclusterswitch reconciler archives the demoted primary's current segment as <wal>.partial. That's most likely the sibling in dc-a's bucket next to the finished 000000010000000000000007.

The bucket listing is sorted on the full object path, so which file comes first depends on compression. Uncompressed, .gz, .bz2, .lz4: the complete WAL sorts first, download_wal() restores that one, and the .partial is left over. .xz, .snappy, .zst: the .partial sorts first instead, and gets restored as if it were the requested file, immediately, with the complete WAL left over.

Either way, the leftover file is queued as an extra WAL to prefetch. That alone triggers _ensure_spool_dir_exists(). The spool dir defaults to /var/tmp/walrestore. That's read-only in our sidecar. mkdir fails with EROFS, and barman exits before it even attempts the real restore, in the uncompressed/gz/bz2/lz4 case. --no-partial isn't a workaround here. We need the .partial to be restorable during a DC switchover, that's the reason we archive it.

This looks like something to address in barman. Two things stand out: the spool dir gets created even when --parallel isn't requested, and when both the full WAL and its .partial exist, which one gets served as "the requested WAL" depends on compression rather than being deterministic. The second point matters beyond the crash: with a writable spool, the leftover file gets saved under the bare WAL name, so a later request for that WAL could return the wrong content, and with .xz/.snappy/.zst the partial can be served on the very first request, no spool involved at all.

Exit code 2 here doesn't mean a real connectivity failure. Our sidecar already treats exit 2 and exit 4 the same way, so this doesn't change anything for us today, but it's still worth fixing: an operator reading the log sees "connectivity failure" for a bug that has nothing to do with connectivity, and that would mislead anyone debugging this directly with barman.

[mnencia]

Filed the full picture upstream: EnterpriseDB/barman#1224