Operations posts
Everything in the journal filed under operations.
2026
Where your team writes down its decisions
Decisions live in chat threads and in people's heads until they vanish. A one paragraph record next to the code keeps the reasons where the work is.
The scheduled job that never runs at midnight
A schedule written in one zone on a machine running another, plus daylight saving twice a year, makes a nightly job run late, twice or not at all.
Your backup is a zero byte file and nobody noticed
Backups fail quietly: a missing tool in the container, a permission, a full disk. Check the size and the row counts, and restore on a schedule.
Three fields that make logs worth reading
Most logs cannot answer the one question an incident asks. A correlation id, the subject being worked on and an outcome with a reason fix that.
What your AI feature does when the credit runs out
A model provider can fail for billing, quota or an outage. Without a fallback ladder every user sees the same generic error. How to degrade on purpose.
2025
Invisible dot underscore files travelling from macOS to your server
Extended attributes become AppleDouble files inside archives made on a Mac, land in your web root and get served. How to spot, stop and clean them up.
The rsync flag that breaks certificate renewal
A mirroring deploy removes everything the build did not create, including the path a certificate authority reads. How to catch it weeks before the expiry.
Blue green deploys: check the build exit code before you restart
A deploy script that restarts after a failed build serves a half written directory. Exit codes, an atomic switch, a smoke test and a rollback that works.
Proxy tunnels that die after ten idle seconds
A tunnel that closes on idle turns one long session into thousands of reconnects. On metered bandwidth the handshake traffic quietly becomes the bill.
Never write to /tmp with a generic name on a shared server
On a shared machine a generic temp filename may already exist and belong to someone else. The write fails, the read succeeds, you ship the wrong file.
Log rotation, retention and the silent cost of a full disk
A disk creeping towards full is a slow outage: writes get refused, builds fail, backups truncate. How to find what grew and stop it growing back.
The campaign that never sent: time zones in the database
A job scheduled for 09:00 sits in the queue with no error at all. The application wrote local time, the worker reads UTC, and the offset does the rest.
2024
The reboot that took the site down: process managers and boot persistence
A process manager only brings apps back after a reboot if its service unit is enabled and the process list was saved. How to check both and prove it.
When your control panel and your nginx file disagree
Hosting panels re-render the vhost file from their own template, so hand edits vanish at the next certificate renewal. Where to put the change instead.