omeryanbas.com

Ömer Yanbaş

General Manager, Ticofab Yazılım

OperationsPractice

Never write to /tmp with a generic name on a shared server

On a shared machine a generic temp filename may already exist and belong to someone else. The write fails, the read succeeds, you ship the wrong file.

A deployment script on a shared machine wrote a commit message to a temporary file, and the commit went out carrying text that nobody on the project had written. The file already existed, owned by a different user, left over from a job with no connection to ours. The copy was refused, the refusal went into a redirect nobody read, and the next command read the file that was already sitting there. Everything in the pipeline reported success.

What actually happens

/tmp is world writable with the sticky bit, mode 1777. That combination is deliberate: any user can create a file there, and only the owner of an existing file can overwrite or remove it. On a machine with one user this is invisible. On a shared machine, with several deploy accounts, several projects and a handful of cron jobs, it becomes a collision waiting to happen, because everybody has the same taste in filenames. msg.txt, out.json, list.txt, data.csv, backup.sql, payload.txt.

The sequence is short:

  1. Some earlier job, possibly months ago, wrote /tmp/msg.txt as another user.
  2. Your script runs cp ./message.txt /tmp/msg.txt.
  3. The copy is refused with permission denied, because you do not own the target.
  4. The refusal is lost, because stderr was redirected, or the commands were chained with a semicolon, or the whole thing ran inside an ssh command string whose status nobody inspected.
  5. The next step reads /tmp/msg.txt. The file exists and is readable, so the read succeeds and returns the other user's content.

The shape of the bug is a write that fails and a read that succeeds. A script that never asks for the exit status cannot tell those apart, and neither can the log.

How to see it

Look at the file, not at your script. Ownership and modification time usually end the investigation in ten seconds:

ls -l /tmp/msg.txt
stat -c '%U %G %a %y %s' /tmp/msg.txt
id -un
# otheruser otheruser 644 2025-03-14 09:12:41 +0300 1187
# appuser

A modification time from months ago is the giveaway. Your script did not write that file today, whatever it reported.

Reproducing it is one command plus the status you were not printing:

cp ./message.txt /tmp/msg.txt 2>/dev/null
echo "exit=$?"
# exit=1
head -2 /tmp/msg.txt
# someone else's content

To see how much of this is waiting to happen on the machine, list what is in the shared directory that you do not own:

find /tmp -maxdepth 1 -not -user "$(id -un)" -printf '%u %10s %p\n' 2>/dev/null | sort | head -20

If that list contains names your scripts also use, you have found the next incident before it happened. Full disks produce the same class of silent write failure, which is one reason disk pressure and retention is worth handling before it reaches you.

The fix

Four rules, and they fit in one script.

  • Unique paths. Either mktemp, or a directory you own with a name nobody else would pick, such as a project specific directory under your own home.
  • Never suppress the exit status of a copy or a move. If you must silence output, capture the status first.
  • Verify after writing. Check that the file exists and is not empty, rather than assuming the previous line worked.
  • Clean up with a trap, so failure and success both leave the machine tidy.
#!/usr/bin/env bash
set -euo pipefail

work="$(mktemp -d "${TMPDIR:-/tmp}/release.XXXXXXXX")"
trap 'rm -rf "$work"' EXIT

msg="$work/message.txt"
printf '%s\n' "$COMMIT_MESSAGE" > "$msg"

if [ ! -s "$msg" ]; then
  printf 'message file missing or empty: %s\n' "$msg" >&2
  exit 1
fi

git commit -F "$msg"

mktemp -d creates a directory with a random suffix, owned by you, mode 0700. There is no name to collide with, no other user can read what is inside, and the trap removes it whether the script finishes or dies.

The remote case is where this mistake is easiest to make, because the copy crosses a boundary and the status has to come back across it:

host="$1"
remote="/root/release-$(date +%Y%m%d-%H%M%S)-$$"

if ! ssh "$host" "mkdir -m 700 -p '$remote'"; then
  printf 'could not create remote directory\n' >&2
  exit 1
fi

if ! scp ./build.tar.gz "$host:$remote/build.tar.gz"; then
  printf 'upload failed\n' >&2
  exit 1
fi

ssh "$host" "set -eu; cd '$remote'; tar xzf build.tar.gz; test -f public/index.html"
printf 'remote unpack exit: %s\n' "$?"

Two details carry most of the weight. $(date ...) plus $$, the process id, produces a path that a second job on the same machine will not choose. And set -eu inside the remote command string means that a failure in the middle of the chain is reported, rather than only the status of the last command in it.

How to check it worked

Run the script with tracing and look at the path it actually used:

bash -x ./release.sh 2>&1 | grep -i mktemp
# ++ mktemp -d /tmp/release.XXXXXXXX
stat -c '%a %U %n' /tmp/release.*
# 700 appuser /tmp/release.kQ9dT3ax

Mode 700 and your own username are the two things to confirm. Then prove that the verification step actually stops the run, by feeding it nothing:

COMMIT_MESSAGE="" ./release.sh; echo "exit=$?"
# message file missing or empty: /tmp/release.kQ9dT3ax/message.txt
# exit=1

A script that stops on an empty file will also stop on an unwritable one. That is the whole point of the check: it does not care why the file is wrong, only that it is.

What to watch out for

  • TMPDIR may not be /tmp. Respect it in scripts, and remember that a systemd unit with PrivateTmp has its own namespace, so the same path from your shell and from the service are two different files. Two views of one path that disagree is the same confusion as a control panel and an nginx file that disagree.
  • /tmp is cleaned on a timer or at boot on most distributions. Anything that has to survive a restart does not belong there, whatever its name.
  • The sticky bit prevents deletion, not reading. A mode 644 file in /tmp with a token inside is readable by every account on the machine, including any service account that gets compromised. Credentials do not go to /tmp even with a unique name.
  • A predictable path is a target. Another user can place a symlink there in advance and turn your write into a write somewhere else entirely. An unpredictable name from mktemp closes that door.
  • Redirecting stderr to /dev/null is how most of these bugs stay alive for years. If a command is noisy, send its output to a file you can read later rather than discarding it.

The habit worth keeping is smaller than the bug: treat every write as something that can fail and every read as something that can succeed for the wrong reason. A unique path, a checked exit status and one line of verification cost nothing at write time and save you from shipping a stranger's content with your name on it. The same discipline catches the class of problem where the difference between right and wrong is invisible on screen, such as the invisible suffix that split every message in two. On a machine you share with other people and other jobs, assume the generic name is already taken, because eventually it is.

Questions and answers

Why did my copy to /tmp fail with permission denied?
/tmp has the sticky bit set, mode 1777. Every user can create files there, but only the owner of an existing file can overwrite or delete it. If another user or another service already created a file with the name you chose, your copy is refused even though the directory is world writable.
Is mktemp enough to make a temporary file safe?
It solves the two problems that matter most: the name is unpredictable so it cannot already exist or be pre placed as a symlink, and with mktemp -d the directory is created mode 0700 so other users cannot read it. It does not make the file secret from the superuser, and it does not survive a reboot, so secrets and anything you need later still belong elsewhere.
Why does the file look different from my shell than from the service?
Systemd units with PrivateTmp enabled get their own private /tmp namespace. The path is identical but the filesystem behind it is not, so the file you wrote interactively is invisible to the service and the other way round. Check the unit for PrivateTmp before concluding that a write failed.
How do I get the exit status of a command run over ssh?
ssh returns the remote command's exit status as its own, so a plain if statement around the ssh call works. What breaks it is redirecting output to /dev/null and continuing with a semicolon, or wrapping several remote commands in one string without set -e, in which case only the last command's status is reported.