When your control panel and your nginx file disagree

A change that was made weeks ago is gone. The security headers that were added by hand to a vhost file are not in the response any more, the file on disk does not contain them, and nobody deployed anything that day. The modification time on the file is recent, though, more recent than the last time a person logged in. The file was rewritten by the hosting panel, and it did that because it considers the file its own.

What actually happens

A hosting panel does not store your site's configuration in the config file. It stores it in its own database: the domain, the document root, the certificate paths, the PHP version, the extra directives someone typed into a text box in the interface. The file under the nginx config directory is the rendered output of a template plus that database row. It is an artifact, in the same sense as a compiled binary, and editing it by hand is editing the output of a build.

The render runs more often than people expect. In practice the triggers are:

1. Certificate issue and certificate renewal. The panel writes new paths into the vhost and re-renders the whole file to do it.
2. Any save in the interface for that site, even a change with nothing to do with the web server, such as switching a runtime version or adding a mail alias.
3. Adding or removing a domain alias, which changes server_name and therefore the file.
4. A panel upgrade, which can ship a new template and re-render every site on the machine.

Renewal is the important one, because it runs on a timer. The edit happens on a Tuesday, the site behaves correctly for six or eight weeks, and then a scheduled job that nobody was watching rewrites the file at four in the morning. By the time somebody notices the missing behaviour, the edit is far enough in the past that nobody connects the two events. It looks like the change never worked, or like something else broke it.

The render can also do more than drop your lines. A template that emits includes in a fixed order will reorder anything you moved, and directives the template does not know about are simply not in its output. There is no merge, no conflict and no log line saying what was removed.

How to see it

Read the top of the generated file first. Panels almost always leave a banner:

head -5 /etc/nginx/sites-enabled/example.com.conf
# # This file is generated. Changes will be lost on the next update.

Then compare timestamps against your own history. If the file is newer than your last login, something else wrote it:

stat -c '%y %n' /etc/nginx/sites-enabled/*.conf
last -n 5

To find the template, take a literal string from the rendered file that is distinctive enough to be in the source, and search the panel's installation directory for it. Keep the search away from the nginx directory so the output is only sources:

grep -rl 'fastcgi_read_timeout' /opt /usr/local /usr/share 2>/dev/null | grep -v '/etc/nginx/'
# /usr/local/panel/templates/vhost_nginx.tpl

A file with placeholders in it rather than real values is the template. Next to it, or in the panel's data directory, you will usually find where the per site custom directives are stored, which is the thing you actually want.

The last confirmation is a deliberate one. Save the site in the panel interface without changing anything, and watch the file:

sha256sum /etc/nginx/sites-enabled/example.com.conf
# press save in the interface
sha256sum /etc/nginx/sites-enabled/example.com.conf

Two different hashes means the file is rendered on every save, and the question is no longer whether your edit will be lost but when.

The fix

Decide which layer owns the file and stop fighting it. There are three places to put a change, in order of preference.

The first is the custom directives field that most panels offer per site. It is stored in the panel database and rendered into the file on every pass, so it survives by design. It is also the one place the next person will look.

The second is an include directory that the template already emits. If the rendered file contains a line like include /etc/nginx/custom/example.com.d/*.conf;, you can drop a file in there and the panel will keep emitting the include:

# /etc/nginx/custom/example.com.d/headers.conf
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;

Watch the level that include lands in. If the panel emits it inside a location block rather than at server level, you have just met the rule where one add_header drops every other header for that path.

The third is the template itself, for changes that have to apply to every site on the machine. Edit the template, then force a render so that the file and the template agree immediately instead of at some unpredictable moment months later. Keep your own copy of the template in version control, because a panel upgrade will replace it and you want a diff, not a memory.

Whichever layer you use, apply the change through a script that can undo itself:

#!/bin/sh
set -e

target=/etc/nginx/custom/example.com.d/headers.conf
backup=/root/vhost-rollback-$(date +%Y%m%d%H%M%S).conf

[ -f "$target" ] && cp "$target" "$backup"
cp ./headers.conf "$target"

if nginx -t; then
  systemctl reload nginx
  echo "applied"
else
  echo "config test failed, rolling back"
  if [ -f "$backup" ]; then cp "$backup" "$target"; else rm -f "$target"; fi
  nginx -t && systemctl reload nginx
  exit 1
fi

The backup goes to a directory of your own with a timestamped name. On a machine where more than one person or more than one project has a shell, a generic name in a shared directory is its own accident waiting to happen.

How to check it worked

The change working today proves nothing. What you want to know is whether it works after a render. Force one, then check both the file and the behaviour:

sha256sum /etc/nginx/sites-enabled/example.com.conf > /root/vhost.sha
# save the site in the panel, or renew the certificate
sha256sum -c /root/vhost.sha
# /etc/nginx/sites-enabled/example.com.conf: FAILED

grep -c 'custom/example.com.d' /etc/nginx/sites-enabled/example.com.conf
# 1

curl -sI https://example.com/ | grep -ci 'strict-transport-security'
# 1

The hash changing is expected, since the panel rewrote the file. What matters is that the include is still there and the header is still on the response. If both hold after a renewal, the change is in the right layer.

Then make it a check rather than a memory. A small scheduled job that requests the site and alerts when a required header is missing turns a silent regression into a message, and it costs one request a day.

What to watch out for

• Certificate renewal is the trigger that catches people, because it is on a timer and it also touches other things. A renewal that half succeeds can leave a site serving an expired certificate or a file that does not reference the new one. The related trap is a deploy step that deletes the directory the renewal needs, which is how an rsync flag breaks certificate renewal.
• A passing nginx -t does not mean your change survived. The test checks syntax, and a file with your directives removed is perfectly valid syntax.
• Order matters for some directives, and the template decides the order. If the panel emits its own add_header or its own location after your include, yours is not the one that wins.
• After a panel upgrade, diff the template against your copy before assuming anything still applies. This is the render that changes every site at once.
• Write down which layer owns the file, next to the change itself, in whatever place your team actually reads. The next person to hand edit that file will be someone who never saw the banner at the top.

Anything generated by a tool is output, and output belongs to the tool. The habit that prevents this class of problem is asking one question before editing any file on a server: what wrote this, and when will it write again. If the answer is a template and a timer, the edit goes in the template, and the check goes in a scheduled job rather than in your head. The cost of finding the right layer is twenty minutes once, and the cost of skipping it is a regression that arrives two months later with no obvious cause.