The rsync flag that breaks certificate renewal

A deploy mirrors the build directory onto the server. It is fast, it is repeatable, and the state on the server is always exactly what came out of the build. It runs like that for two months. Then the site goes down on a Saturday with a certificate warning, and the deploy that caused it went out in September. The flag that did the damage is the one that makes mirroring mirroring.

What actually happens

rsync -a --delete source/ host:/var/www/site/ means make the destination identical to the source. Files on the server that are not in the build are removed, which is the behaviour you asked for and the reason you chose it: without it, every renamed asset stays behind forever and the directory fills with files nobody can account for.

The trouble is that a web root usually holds more than the build. Three kinds of thing live there that no build produces:

• Files another process writes, such as user uploads, generated sitemaps or a cached export.
• Files a platform or a panel wrote once and expects to stay, such as a rewrite config or an ownership verification file.
• Files a certificate authority reads during validation, under the well known path.

The last one is the interesting failure because of its timing. The HTTP validation method works like this: the renewal client writes a token file into a directory, the authority requests it over plain HTTP, and if the contents match, the certificate is issued. A common hardening step is to keep that directory out of the application entirely, either by symlinking it from the web root to a shared directory, or by pointing the client at a path and letting the server config map the URL to it. The symlink version is the one that breaks, because a symlink in the web root is a file, and a mirroring deploy deletes files.

From there, the sequence is slow enough that nobody connects the two events:

1. A deploy removes the symlink. The site is unaffected, every page still works, and nothing is logged as an error.
2. The certificate on the server is good for another seventy days, so no renewal is attempted.
3. Around day sixty, the renewal timer starts trying. The authority requests the token URL and gets a 404, or gets the application's catch all page with a status of 200 and the wrong body.
4. The attempt fails, writes a line to a log, and the timer tries again twice a day with the same result.
5. Ten days later the certificate expires and every visitor sees a warning, including the ones who cannot get past it.

There is a second version of this with the same shape and a different cause. If the deploy replaces a rewrite config so that the application router now owns every path, the challenge URL returns the front page with a status of 200. The authority reports an invalid response rather than a missing file, which sends you looking at permissions instead of routing.

How to see it

The first command to run is the deploy itself, in dry run mode, showing only what it would remove:

rsync -avn --delete build/ deploy@host:/var/www/site/ | grep '^deleting'
# deleting .well-known/acme-challenge
# deleting uploads/2025/09/invoice-template.pdf
# deleting sitemap-products.xml

Those three lines are worth more than an hour of reading the script. -n performs no changes, and the deleting prefix is exactly the list of things the server has and the build does not.

Then ask the certificate itself how much time is left, rather than trusting a calendar entry:

echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -enddate
# notAfter=Dec 2 08:14:31 2025 GMT

And ask whether validation would work right now, using a file you place yourself:

printf 'probe\n' | sudo tee /var/lib/acme/.well-known/acme-challenge/probe >/dev/null
curl -sS -o /dev/null -w '%{http_code}\n' http://example.com/.well-known/acme-challenge/probe
# 404

It has to be plain HTTP, because that is how the authority starts. A redirect to HTTPS is fine and is followed, but a firewall that closes port 80 is not. A 200 with a page of HTML instead of the word probe is the routing version of the bug, so check the body and not only the status.

Finally, look at where the renewal client thinks it is writing, and what happened last time it tried:

grep -h webroot_path /etc/letsencrypt/renewal/*.conf
# webroot_path = /var/www/site,

journalctl -u certbot.service --since '60 days ago' | grep -i 'error\|failed' | tail -3

The fix

Move the challenge out of the web root and map it in the server config, so no deploy can reach it:

# beats the application catch all, because ^~ wins over regex locations
location ^~ /.well-known/acme-challenge/ {
    root          /var/lib/acme;
    default_type  text/plain;
    try_files     $uri =404;
}

With root rather than alias, the request path is appended to the directory, so /.well-known/acme-challenge/token is read from /var/lib/acme/.well-known/acme-challenge/token. That is exactly where the client writes when its webroot path is /var/lib/acme, so point it there once and the two agree forever:

sudo mkdir -p /var/lib/acme/.well-known/acme-challenge
sudo certbot certonly --webroot --webroot-path /var/lib/acme -d example.com

If the challenge has to stay inside the web root, protect it explicitly with a filter file that lives next to the deploy script and is read by every deploy:

# deploy-filter.txt
P /.well-known/
P /uploads/
P /storage/
P /sitemap-*.xml
P /.env

rsync -a --delete --filter="merge deploy-filter.txt" build/ deploy@host:/var/www/site/

A P rule protects a path from the deletion pass while leaving transfers alone, which says what you mean more clearly than an exclude does. An exclude protects too, as a side effect, and that side effect is undone the moment somebody adds the flag that deletes excluded files on the destination.

The list of things to protect is worth writing down once, because it is the same list on most projects: uploads, generated files, the challenge path, verification files for search consoles and mobile apps, a maintenance page, and any config a panel maintains. The cost of this fix is a file that has to be kept up to date. The cost of skipping it is a certificate outage on a weekend.

A deploy that swaps a whole directory into place instead of mirroring into the live one avoids the deletion pass entirely, which is one more reason to build into an inactive directory and switch. It does not remove the need to keep state outside the release, it just makes the boundary obvious.

How to check it worked

Prove the challenge survives a deploy, then prove renewal actually works:

printf 'probe\n' | sudo tee /var/lib/acme/.well-known/acme-challenge/probe >/dev/null
curl -sS http://example.com/.well-known/acme-challenge/probe
# probe

./deploy.sh

curl -sS http://example.com/.well-known/acme-challenge/probe
# probe

sudo certbot renew --dry-run
# Congratulations, all simulated renewals succeeded

The dry run is the real test. It performs the full validation against a staging environment and fails in the same way a genuine renewal would, so it catches a routing change, a permission change and a deleted path alike. Run it after any change to the deploy script, the server config or the web root layout.

Then stop relying on the expiry email. A check that alerts at twenty one days left gives you three working weeks:

days=$(( ( $(date -d "$(echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -enddate | cut -d= -f2)" +%s) - $(date +%s) ) / 86400 ))
[ "$days" -lt 21 ] && echo "certificate expires in $days days"

What to watch out for

• The flag that deletes excluded files on the destination undoes every protection in your filter file in one word. Grep for it in the deploy script and never add it to fix a stale file.
• A trailing slash on the source changes what gets mirrored. build/ copies the contents, build copies the directory itself, and the second form deletes almost everything on the destination on the first run.
• Renewal failures go wherever the client logs, which is usually a file nobody has opened. If mail from the authority goes to an address that no longer has a reader, the browser warning is your first notification, which is the same pattern as a backup that turns out to be a zero byte file.
• Do not solve a 404 on the challenge path by loosening the rules for dotted paths across the whole site. The well known path is the one dotted path that is meant to be public, and everything else under a leading dot should stay invisible to the internet.
• DNS validation avoids the web root completely, which is useful for a wildcard, but it puts credentials for your DNS account on the server. That is a different risk, not a smaller one.

The general shape of this is that a deploy has an opinion about what belongs on the server, and it is always narrower than the truth. Anything written by a process other than the build is invisible to it and therefore deletable. Writing down which paths are not the build's business takes ten minutes, and the dry run that shows the deletion list takes ten seconds. Both of them are cheaper than finding out on a Saturday that a machine stopped being able to prove who it was.