The database nobody wanted backed up took the other 41 with it

My nightly Postgres backup fans a pg_dump -Fc of every database out to four destinations — object storage, a NAS, and two cloud drives. The idea is that losing the house doesn't lose the data.

One morning the monitor said the success marker was stale. The log said this:

=== pg-backup-fanout start (20260911T073002Z) dry_run=false ===
dumping claude_memory_sqlascii_20260911 -> .../claude_memory_sqlascii_20260911.dump
pg_dump: error: connection to server failed: FATAL:  too many connections
                for database "claude_memory_sqlascii_20260911"
FATAL: pg_dump failed for claude_memory_sqlascii_20260911

Four lines. The run died on the first database, alphabetically, and stopped. Forty-one healthy databases and the globals dump never happened. Nothing reached any of the four destinations. A full night of backups, gone.

The error is lying to you

"Too many connections for database X" reads like a connection-pool problem. It isn't. Postgres emits that exact message when a database has datconnlimit = 0:

SELECT datname, datconnlimit FROM pg_database WHERE datname LIKE '%sqlascii%';

             datname             | datconnlimit
---------------------------------+--------------
 claude_memory_sqlascii_20260911 |            0
 sentinel_sqlascii_20260911      |            0

A limit of zero means no connections, ever. Which is exactly what I wanted, because the day before I'd converted that database from SQL_ASCII to UTF8 and kept a frozen pre-migration copy as the rollback path. Locking it against connections was the correct call. I just never thought about the one job on the box whose entire purpose is to connect to every database it can find.

The enumeration query had a filter, and it was almost right:

SELECT datname FROM pg_database
WHERE NOT datistemplate AND datallowconn;

datallowconn = false is the other way to make a database unreachable, and I'd handled it. datconnlimit = 0 is a separate column with the same practical effect, and I hadn't. So the frozen snapshot passed the filter, failed the dump, and the script's || die did the rest.

The Postgres part of the fix is one clause

SELECT datname FROM pg_database
WHERE NOT datistemplate AND datallowconn AND datconnlimit <> 0;

I also log every database it skips, because a backup job that silently declines to back something up is its own future incident:

SKIP claude_memory_sqlascii_20260911 (not connectable: datallowconn=false or datconnlimit=0)
SKIP sentinel_sqlascii_20260911 (not connectable: datallowconn=false or datconnlimit=0)

The part that actually mattered

The one-clause fix closes this bug. It does nothing about the next one.

The real defect is that a job iterating over a set I don't control treated one bad member as grounds to abandon the whole set. Any CREATE DATABASE by anyone, any permissions change, any half-dropped database in a weird state — each of them is one alphabetically-unlucky name away from costing me a night of backups across every destination.

What's darkly funny is that the script already knew this. Its own header comment said a single failed destination must not cost the other three copies, and that logic was there and worked. I'd applied the principle to destinations and not to databases, in the same file, and never noticed the asymmetry.

So a failed dump is now recorded, not fatal:

if ! pg_dump -Fc --no-owner --no-privileges "$db" -f "$out"; then
  log "ERROR: pg_dump failed for $db — continuing with the remaining databases"
  FAILED_DBS+=("$db")
  rm -f "$out"
  continue
fi

The run keeps going, dumps everything else, and ships it. Then it refuses to call itself a success:

[[ "${#ARTIFACTS[@]}" -gt 0 ]] || die "no databases dumped at all — nothing to ship"

and at the end, the success marker is withheld, the healthcheck isn't pinged, the failed names go in the summary line, and it exits 3. Degraded, not dead — and still loud. That distinction is the whole point. "Keep going on error" is only safe if the job is equally committed to telling you it was degraded.

What I'd do differently

Two things.

The frozen snapshot database shouldn't have been a surprise to a job that enumerates everything. If you lock a database against connections, go tell the things that iterate over databases. I've added an explicit exclude-glob config so the next migration snapshot gets named out of scope deliberately instead of being caught by a heuristic.

And the monitor caught this only because the success marker went stale — the job's own exit code and logs were happily reporting a clean failure to nobody. Assert on the artifact the job is supposed to produce, never on the fact that the job ran. That has now bitten me enough times that I'm starting to think it's the only monitoring rule worth remembering.