Your backup is a zero byte file and nobody noticed

Something goes wrong with the data, a migration drops a column or a job deletes rows it should not have, and somebody opens the backup directory. The files are there, one per night, named correctly, going back months. The newest one is twenty bytes. So is the one before it. The last real backup is from five weeks ago, which is when somebody rebuilt the container image.

Why backups fail without saying anything

A backup job has more ways to half succeed than to fail cleanly. These are the ones I keep running into.

1. The tool is not in the image. The application container carries the database client library, not the command line dump tool, and a rebuilt image quietly loses whatever was installed by hand. The job now fails with a command not found, into a log nobody reads.
2. The pipeline swallows the exit code. pg_dump ... | gzip > file returns the status of the compressor. The compressor happily compresses an empty stream and returns zero, so the script sees success and writes a twenty byte file.
3. The user lacks permission. A dump run by an application user that cannot read one table stops partway. Depending on the flags, you get a partial file that looks plausible by size and is missing the table you will need.
4. The disk filled during the write. The file exists, it is truncated, and it will fail on restore rather than on backup. This is the same class of problem as a full disk taking the service down, except here the damage is invisible until you need the file.
5. The output path is inside the container. Everything works, the file is written, and it disappears with the next restart.

Every one of these produces a file, a timestamp and a green line in whatever dashboard you built. The only honest signal is the content of the file.

How to see it in thirty seconds

The size series over time tells you almost everything. A healthy backup grows slowly and predictably:

stat -c '%n %s' /backups/*.dump | tail -7

/backups/app_20260501.dump 184238336
/backups/app_20260502.dump 184501760
/backups/app_20260503.dump 184703488
/backups/app_20260504.dump        20
/backups/app_20260505.dump        20
/backups/app_20260506.dump        20
/backups/app_20260507.dump        20

Three numbers in a row that are identical and tiny mean the job has been failing since the fourth. Then prove the newest file is readable without restoring it:

pg_restore -l /backups/app_20260507.dump | head -5
# pg_restore: error: did not find magic string in file header

That message is worth more than a month of green dashboard lines.

The fix: a job that refuses to lie

The script has to check its own work. Four checks, in order, each one failing loudly:

set -euo pipefail

DB=app
OUT="/backups/${DB}_$(date -u +%Y%m%d).dump"
FLOOR=$((150 * 1024 * 1024))     # smaller than this is not a real dump

fail() { logger -t backup "FAILED ${DB}: $*"; notify "backup ${DB} failed: $*"; exit 1; }

command -v pg_dump >/dev/null || fail "pg_dump is not installed in this image"

pg_dump -Fc -Z 6 -O -x -d "$DB" -f "$OUT" || fail "pg_dump exited $?"

size=$(stat -c %s "$OUT")
[ "$size" -ge "$FLOOR" ] || fail "dump is ${size} bytes, floor is ${FLOOR}"

pg_restore -l "$OUT" >/dev/null || fail "dump header unreadable"

rows=$(psql -At -d "$DB" -c "SELECT count(*) FROM orders")
prev=$(tail -1 /backups/manifest.tsv | cut -f3 || echo 0)
[ "$rows" -ge $(( prev * 95 / 100 )) ] || fail "orders dropped ${prev} to ${rows}"

printf '%s\t%s\t%s\n' "$(date -u +%F)" "$size" "$rows" >> /backups/manifest.tsv

Writing to a file format instead of piping into a compressor removes the pipeline problem entirely. Where a pipeline is unavoidable, set -o pipefail and a check on PIPESTATUS are the minimum, and the habit of reading exit codes before moving on is the same one I argued for in checking the build exit code before you restart.

The row count check is the part people skip and the part that catches the interesting failures. A dump that is the right size but has lost a table is the one that ruins a recovery, and comparing counts against the previous run finds it the next morning.

Then get a copy off the machine. Three copies, two kinds of storage, one somewhere else is the old rule and it still holds. The important detail is verifying the arrival rather than the upload:

scp -q "$OUT" "$OFFSITE:/vault/" || fail "offsite copy failed"

remote_size=$(ssh "$OFFSITE" "stat -c %s /vault/$(basename "$OUT")")
[ "$remote_size" = "$size" ] || fail "offsite size ${remote_size} differs from ${size}"

How to check it worked: restore on a schedule

A backup is a claim. A restore is evidence. Once a week, into a throwaway database, timed:

set -euo pipefail
LATEST=$(ls -t /backups/*.dump | head -1)
SCRATCH="drill_$(date -u +%s)"

createdb "$SCRATCH"
trap 'dropdb "$SCRATCH" || true' EXIT

start=$(date +%s)
pg_restore -j 4 -O -x -d "$SCRATCH" "$LATEST"
echo "restore took $(( $(date +%s) - start )) seconds"

psql -At -d "$SCRATCH" -c "
  SELECT 'orders', count(*) FROM orders
  UNION ALL SELECT 'users', count(*) FROM users
  UNION ALL SELECT 'newest', max(created_at)::text FROM orders;"

restore took 214 seconds
orders|1284401
users|48219
newest|2026-05-06 23:58:11+00

Those four lines are the whole point. The file restores, the tables are populated, the newest record is from last night rather than from five weeks ago, and recovery takes three and a half minutes rather than the unknown number that was in the plan. Record the duration each week, because that number is what you will promise during an incident.

The last piece costs an hour and is never done: write the procedure down. Exact commands with real flags, where the credentials live, how long it takes, how to verify completeness, and who to inform while it runs. A runbook that says to ask a particular person is not a runbook, because that person is exactly who might be unreachable on the day.

What to watch out for

• A floor on size stops working after the data grows. Compare against the previous run with a tolerance, not against a constant you set two years ago.
• Encrypted backups need their key tested too. A drill that skips decryption proves nothing about the copy you would actually use.
• Restoring into the same instance as production competes for the same disk and memory, and a drill that slows the live system will be switched off within a month. Use a separate machine or a quiet window.
• A silent job needs a heartbeat. If the script fails before it can notify, nothing is emitted at all, so also alert when the expected success signal does not arrive. That alert needs the same discipline as everything else in logs worth reading: a reason code, not a sentence.

The pattern behind all of this is that a backup system reports on itself, and self reporting is the weakest evidence available. Size, row counts and a timed restore are external facts that do not depend on the job's opinion of how it went. Take the hour, make the script fail loudly, and run the drill on a calendar rather than on a memory. The day you need it, the only question that matters is how recent the last verified restore was.