Why does my deploy script continue after the build fails?

Because that is what a shell does unless you tell it otherwise. Each command runs and its exit code is discarded unless something reads it, so a failing build followed by a restart is two independent statements. Add set -e so the script stops on the first failure, and set -o pipefail so a failure inside a pipeline is not hidden by the exit code of the last command in it.

Does set -e catch every failure?

No, and treating it as a guarantee is how scripts get trusted too much. It is suppressed for any command whose status is being tested, which covers if conditions, while conditions, anything on the left of && or ||, and anything negated with an exclamation mark. For those you read the exit code yourself and decide, which is clearer anyway at the points that matter.

What makes the switch between releases atomic?

Replacing a symlink with rename, which the kernel performs as a single operation, so a request either sees the old target or the new one and never an empty path. Creating the new link beside the current one and moving it over the top with mv -T does this, while removing the old link and creating a new one leaves a gap of a few milliseconds. On systems without mv -T, a directory holding the symlink can be swapped instead.

How many old releases should I keep?

Enough to roll back past the release that broke, which in practice means three to five. Each one costs the size of your build output plus its dependencies, so prune on every deploy rather than when the disk fills. Keep them by timestamp so the ordering is obvious in a directory listing at three in the morning.

Operations Practice

Blue green deploys: check the build exit code before you restart

A deploy script that restarts after a failed build serves a half written directory. Exit codes, an atomic switch, a smoke test and a rollback that works.

By Ömer YanbaşPublished October 27, 20255 min read

A deploy runs, the log scrolls past, and the last line says the site is live. It is not. Somewhere in the middle the build threw a type error, the script carried on regardless, the process restarted, and the running application is now reading a directory that contains half of yesterday's output and half of a build that never finished. The page either errors or loads without its styles, and the only thing that looked healthy was the deploy script's own final message.

What actually happens

Three separate mistakes stack up, and each one is harmless on its own.

The first is building in place. If the build writes into the directory the server reads from, then for the duration of the build the site is a mixture. A successful build makes that window short. A failed build makes it permanent.

The second is that a shell script does not stop when something fails. Each line runs and its exit code is thrown away unless someone looks at it. A script that ends like this has no opinion at all about whether the build worked:

cd /srv/app
git pull
npm run build
systemctl restart app

The third is pipelines. Adding | tee to keep a log quietly changes what the script knows, because the exit code of a pipeline is the exit code of its last command. npm run build | tee build.log reports whatever tee thinks, and tee almost always succeeds:

false | tee /dev/null; echo "exit=$?"
# exit=0
set -o pipefail
false | tee /dev/null; echo "exit=$?"
# exit=1

Put together, a build failure becomes a restart into a directory nobody validated. The process manager does exactly what it was told, comes up, and reports itself healthy, because a process that starts is not the same thing as an application that works.

How to see it

Before changing anything, find out what your current script actually knows. Run the deploy with the build forced to fail and watch whether the restart happens:

bash -x deploy.sh 2>&1 | tail -20

The -x trace prints each command as it runs, so a restart appearing after a build error is on screen in plain text. Then check the two things a script usually gets wrong:

grep -n 'set -' deploy.sh
# (no output at all is the common answer)

grep -n '|' deploy.sh
# 14: npm run build | tee /var/log/deploy-build.log

If a script has no set line and pipes its build through anything, it cannot tell a good deploy from a bad one. You can confirm what the live process is serving at any time by asking the filesystem rather than the log:

ls -l /srv/app/current
# current -> /srv/app/releases/20251024T081140Z

If that path is the same directory the build writes into, there is nothing to switch and nothing to roll back to.

The fix

Build somewhere new, prove it works, then switch. The whole pattern fits in a page of shell:

#!/usr/bin/env bash
set -Eeuo pipefail

APP=/srv/app
REL="$APP/releases"
REF="${1:-origin/main}"
TARGET="$REL/$(date -u +%Y%m%dT%H%M%SZ)"
trap 'echo "deploy failed on line $LINENO, live site untouched" >&2' ERR

mkdir -p "$TARGET"
git -C "$APP/repo" fetch --quiet origin
git -C "$APP/repo" archive "$REF" | tar -x -C "$TARGET"

cd "$TARGET"
npm ci --omit=dev --no-audit --no-fund
npm run build                       # set -e stops the script here on failure
[ -s dist/index.html ]              # a build can succeed and produce nothing

# smoke test the new release on a spare port, before anything is switched
PORT=8081 node server.js & NEW=$!
trap 'kill "$NEW" 2>/dev/null || true' EXIT
for i in $(seq 30); do
  curl -fsS -o /dev/null "http://127.0.0.1:8081/healthz" && break
  [ "$i" -eq 30 ] && { echo "new release never became healthy" >&2; exit 1; }
  sleep 1
done
curl -fsS "http://127.0.0.1:8081/" | grep -q '</html>' || { echo "home page is empty" >&2; exit 1; }
kill "$NEW"; wait "$NEW" 2>/dev/null || true; trap - EXIT

# atomic switch, then reload
ln -sfn "$TARGET" "$APP/next"
mv -T "$APP/next" "$APP/current"
systemctl restart app
curl -fsS -o /dev/null "https://app.example.com/healthz"

ls -1dt "$REL"/*/ | tail -n +6 | xargs -r rm -rf
echo "live: $(readlink -f "$APP/current")"

Five things in there are doing the work:

set -Eeuo pipefail stops on the first failure, treats an unset variable as an error, and refuses to let a pipeline hide a bad exit code. The E makes the ERR trap apply inside functions too.
The build writes into a directory nothing is serving, so a failure costs a wasted directory and nothing else.
[ -s dist/index.html ] checks the output, not just the exit code. A build tool that exits zero with an empty output directory is a real failure mode, usually after a config change.
The smoke test runs the new release as a process on another port and asks it for a page. This is the only step that catches a build which compiles perfectly and cannot start, for instance because a dependency was pruned or an environment variable is missing.
ln -sfn beside the live link, then mv -T over the top, is a rename, which the kernel does in one step. Deleting the old symlink and creating a new one gives you a gap where the path does not exist.

A release directory should hold everything needed to run and nothing that has to outlive a release. Configuration, uploads and anything a process writes at runtime live outside it and are reached by absolute path, because they must survive the switch in both directions. The cost is disk: each release carries its build output and its dependencies, so five releases of a modest application is a few hundred megabytes. Prune on every deploy rather than when the disk fills, because a full disk turns a routine deploy into an incident.

Rollback is the same switch pointed at the previous directory:

PREV=$(ls -1dt /srv/app/releases/*/ | sed -n 2p)
ln -sfn "$PREV" /srv/app/next && mv -T /srv/app/next /srv/app/current
systemctl restart app

What this does not solve: database migrations. A symlink swap moves code back in seconds and cannot move a dropped column back at all, so migrations need to be backwards compatible for at least one release. That is a separate discipline, and it is the same one as long jobs done in small reversible steps.

How to check it worked

The check that matters is not the good path. Break the build on purpose and prove the site never moved:

echo 'syntax error here' >> src/app.js
./deploy.sh; echo "deploy exit=$?"
# deploy failed on line 17, live site untouched
# deploy exit=1

readlink -f /srv/app/current
# /srv/app/releases/20251024T081140Z   (unchanged)

curl -fsS -o /dev/null -w '%{http_code}\n' https://app.example.com/
# 200

A deploy script that returns a non zero exit code on a bad build is also the thing that lets an automation stop instead of carrying on to the next step. Then run a good deploy and check that current moved and that at most five directories remain under releases.

Keep the broken build around as a fixture. A deploy script is code, it gets edited under pressure, and the only thing that keeps it honest is running the failure case on purpose every time somebody changes it.

What to watch out for

set -e does not fire for a command whose exit status is being tested. Anything inside an if, a while, the left side of && or ||, or after an exclamation mark, runs without stopping the script. Read $? explicitly at those points.
A process manager that resolved the real path at boot keeps serving the old release after the symlink moves. Restart or reload it as part of the switch, and make sure it comes back after a reboot as well, which is its own class of outage.
mv -T is a GNU coreutils option and is missing on some systems. Check for it once in the script rather than discovering it mid deploy.
Build scratch space belongs inside the release directory, not in a shared temp path, for the same reason that a generic name in /tmp is a trap on a shared server.
Pruning with ls | xargs rm -rf is fine while the directory names are timestamps you generated, and dangerous the day someone creates a directory with a space in the name. Keep the naming mechanical.

The useful part of blue green is not the colours, it is that the decision to go live becomes a single reversible operation that happens after everything else has already passed. Everything before the switch is allowed to fail, because failing there costs a directory. Everything after it is expensive, so there should be as little after it as possible. A deploy script earns trust by what it refuses to do, and the way to find out whether yours refuses is to break the build and watch.

Questions and answers

Why does my deploy script continue after the build fails?: Because that is what a shell does unless you tell it otherwise. Each command runs and its exit code is discarded unless something reads it, so a failing build followed by a restart is two independent statements. Add set -e so the script stops on the first failure, and set -o pipefail so a failure inside a pipeline is not hidden by the exit code of the last command in it.
Does set -e catch every failure?: No, and treating it as a guarantee is how scripts get trusted too much. It is suppressed for any command whose status is being tested, which covers if conditions, while conditions, anything on the left of && or ||, and anything negated with an exclamation mark. For those you read the exit code yourself and decide, which is clearer anyway at the points that matter.
What makes the switch between releases atomic?: Replacing a symlink with rename, which the kernel performs as a single operation, so a request either sees the old target or the new one and never an empty path. Creating the new link beside the current one and moving it over the top with mv -T does this, while removing the old link and creating a new one leaves a gap of a few milliseconds. On systems without mv -T, a directory holding the symlink can be swapped instead.
How many old releases should I keep?: Enough to roll back past the release that broke, which in practice means three to five. Each one costs the size of your build output plus its dependencies, so prune on every deploy rather than when the disk fills. Keep them by timestamp so the ordering is obvious in a directory listing at three in the morning.