The reboot that took the site down: process managers and boot persistence

A server reboots without being asked: a host maintenance window, a kernel update, a power event. It comes back in under a minute and the site returns 502 on every request. The web server is running and answering, the certificate is valid, the config is unchanged, and nothing is listening on the port behind the proxy. The process manager that has been restarting the application reliably for months did not start, and on closer inspection it would not have known what to start if it had.

What actually happens

A process manager is a long running process like any other. It watches your applications, restarts them when they crash and keeps their logs, and none of that has anything to do with what happens when the machine is off. For applications to come back at boot, three separate things have to be true:

1. Something starts the process manager. On a systemd machine that means a service unit, installed and enabled, so the symlink exists in the target that is reached during boot.
2. The manager knows what to start. That is a saved process list, written to the home directory of a specific user, and read back when the unit starts.
3. The saved list matches reality. It is a snapshot taken at the moment someone ran the save command, not a live mirror of what is running.

Nearly every broken setup I have looked at satisfies one of these and not the others. The common shape is a manager that was started by hand during the first deploy, kept running ever since, and never wrote a list or installed a unit. Everything about that machine works, for as long as nobody turns it off.

The second common shape is subtler. The unit exists and is enabled, but it runs as one user while the process list was saved as another. The unit starts, looks for a list in that user's home directory, finds nothing, restores nothing and exits successfully. Every status check reports green.

The third shape is drift. The unit and the list are both in place, but the list was saved eleven months ago, before two more applications were added. Those two do not come back, and because the main application does, the machine looks like it recovered.

How to see it

Start from the symptom and work down. The 502 means the proxy could not connect, so the first question is whether anything holds the port:

curl -o /dev/null -s -w '%{http_code}\n' https://example.com/
# 502

ss -ltnp | grep 3000
# (no output)

Then ask the two questions separately. Is the unit installed and enabled, and does it run as the user you think:

systemctl is-enabled pm2-deploy.service
# disabled

systemctl cat pm2-deploy.service | grep -i '^User\|^Environment'
# User=deploy
# Environment=PM2_HOME=/home/deploy/.pm2

And does a saved list exist for that user, with the applications you expect in it:

ls -l /home/deploy/.pm2/dump.pm2
# ls: cannot access '/home/deploy/.pm2/dump.pm2': No such file or directory

grep -o '"name":"[^"]*"' /home/deploy/.pm2/dump.pm2 | sort -u

Two answers, two independent failures. A missing unit means nothing will start the manager. A missing or stale dump means the manager starts and restores nothing. The reason this bites so often is that the everyday commands, listing processes and restarting one of them, exercise neither of these paths.

The fix

Three commands, run as the user that owns the applications, and then one line added to the deploy script.

pm2 list
# confirm this is exactly what should be running after a reboot

pm2 save
# saves the current list to the home directory of this user

pm2 startup
# prints a command that installs and enables the service unit for this user
# run the printed command as root, exactly as printed

The last one is the step people skip, because pm2 startup only prints a command rather than doing the work itself. Run what it prints, then confirm:

systemctl is-enabled pm2-deploy.service
# enabled

Then close the drift. The saved list is a snapshot, so the save belongs at the end of the deploy, after the applications are running and healthy:

# in the deploy script, after the restart and the health check
pm2 save

If the deploy script already checks the build exit code before it restarts anything, this is the same discipline one step later. A deploy that restarts a broken build and then saves it into the boot list turns one bad release into a permanent one, which is the argument for checking the exit code before you restart.

Boot ordering is the part the three commands do not solve. At boot, your application may start before the database is accepting connections, or before a mounted volume is available. A process that exits immediately gets restarted, and a manager that sees several failures in a few seconds will stop trying and leave the application stopped. The fix is on the application side: retry the first connection with a short backoff instead of exiting, and let the health check decide when the instance is ready.

How to check it worked

Do not trust the configuration, exercise it. Stopping the manager completely and starting the unit again runs the same code path a boot would:

pm2 kill
ss -ltnp | grep 3000
# (no output, as expected)

systemctl start pm2-deploy.service
pm2 list
# the applications are back

curl -o /dev/null -s -w '%{http_code}\n' https://example.com/
# 200

That test takes fifteen seconds and covers the unit, the saved list and the user. What it does not cover is ordering against other services on the machine, so schedule one real reboot in a quiet window and check the same three things afterwards:

uptime
# up 2 minutes

systemctl is-active pm2-deploy.service
# active

curl -o /dev/null -s -w '%{http_code}\n' https://example.com/
# 200

Doing it on purpose once is much cheaper than discovering it during someone else's maintenance window. It is the same argument as restoring a backup on purpose rather than believing the backup job's exit code.

What to watch out for

• Restarting an application proves nothing about boot. pm2 restart and systemctl restart both use paths that are already warm. The only meaningful test starts from nothing.
• Check which user everything belongs to. A list saved as root and a unit running as a deploy user is the most common mismatch, and every individual piece looks correct.
• Enabled is not the same as running, and running is not the same as enabled. A service started by hand today will not be there tomorrow, and a service enabled today is not running until something starts it.
• A full disk stops services from starting at boot in ways that look like configuration errors, because logs cannot be written and sockets cannot be created. Retention and rotation are part of keeping a machine bootable, not only part of keeping it tidy.
• Containers have the same trap with different words. A container without a restart policy is gone after a host reboot, and a restart policy only covers containers that existed when the host went down.
• If you run applications under a user scoped service rather than a system one, the session ends at logout and takes the applications with it unless lingering is enabled for that user.

Boot is a code path like any other, and it is the one path that never runs during normal operation, which is why it is the one that is broken. The check is not "is the process manager installed", it is "what starts it, what does it read, and when was that written". Every deploy should leave the answer to the third question current, and once a year the machine should be restarted on purpose to prove the first two. A reboot you chose is an experiment, and a reboot you did not choose is an incident.