The scheduled job that never runs at midnight

A nightly job is set for midnight and the report lands at three in the morning, every day, while the crontab line looks exactly right. Twice a year it gets stranger: on one Sunday the job runs twice and the same rows are counted again, and on one Sunday in spring it does not run at all. The expression is not wrong. The clock it is measured against belongs to the machine, and that is not the clock the person who wrote it had in mind.

What actually happens

A cron expression is a wall clock time with no zone attached. 0 0 * * * means midnight according to whatever the running process believes local time to be, and that belief comes from somewhere else entirely: the host configuration, the container image, or the environment a service manager hands over. Base images ship as UTC, because for a machine that is the only defensible default. The person writing the schedule is sitting in a zone a few hours away from it. Nothing in the crontab records the disagreement, so nothing can report it.

The constant offset is the easy half. It is wrong every single day, so somebody notices within a week and adjusts the number until the report arrives at a sensible hour. That adjustment is the real damage, because the schedule now encodes an offset instead of an intent, and it will be wrong again the moment the machine moves or the rule changes.

The other half arrives twice a year, and only in zones that still observe daylight saving. When the clock springs forward, an hour of wall clock time does not exist: in the zones that change at 02:00 local, the minute after 01:59 is 03:00. A job set for 02:30 has no instant to run at. Most cron implementations skip it. Some run it right after the jump, which is a different behaviour with the same configuration, and nobody on your team has read the manual page that says which.

When the clock falls back, that hour happens twice. 02:30 arrives, and an hour later 02:30 arrives again, and cron cannot tell them apart. The job may fire once or twice. If the job sums a day of rows into a total, the total is now wrong and nothing errored. This is the expensive case, because it looks like a data problem for weeks before anybody connects it to a date in October.

There is a third clock that belongs to the same family: the one in the database. A timestamp stored without a zone, written by a process in UTC and read by a report in local time, gives an answer that is off by the offset and looks exactly like a scheduling bug. Time zones in the database is the same mistake approached from the other end.

How to see it

Ask the machine what time it thinks it is, and ask it again in the zone the schedule was written in:

date +"%F %T %Z %z"
# 2026-05-24 21:12:04 UTC +0000

TZ=Europe/Istanbul date +"%F %T %Z %z"
# 2026-05-25 00:12:04 +03 +0300

If those two lines disagree, every schedule on the box is offset by that difference. Note that cron does not read your shell profile, so a TZ you exported in a login file is not the TZ the job gets.

For the daylight saving half, test whether the minute you chose actually exists. This walks a day one minute at a time and counts how many times the target wall clock time appears in a given zone:

// how many times does 02:30 happen on this date in this zone?
const zone = 'Europe/Berlin';
const fmt = new Intl.DateTimeFormat('en-GB', {
  timeZone: zone, hour: '2-digit', minute: '2-digit', hour12: false,
});

for (const day of ['2026-03-29', '2026-10-25']) {
  const start = new Date(`${day}T00:00:00Z`).getTime();
  const hits = [];
  for (let m = 0; m < 24 * 60; m++) {
    const t = start + m * 60000;
    if (fmt.format(t) === '02:30') hits.push(new Date(t).toISOString());
  }
  console.log(day, hits.length, hits);
}
// 2026-03-29 0 []
// 2026-10-25 2 [ '2026-10-25T00:30:00.000Z', '2026-10-25T01:30:00.000Z' ]

Zero on one date and two on the other is the whole bug in two lines of output. Run it for every zone your schedules claim to be in, not just your own.

The fix

Decide, per schedule, which of two things the business actually means. Either it is a fixed interval, in which case UTC is the truth and local time is only a display, or it is a local wall clock promise such as a report on somebody's desk at nine, in which case you have to store the zone and recompute the instant each time. Most schedules are the first kind and get written as the second by accident.

Pin the process zone so the machine stops having an opinion:

CRON_TZ=UTC
30 23 * * * /usr/bin/flock -n /var/lock/nightly.lock /opt/app/bin/nightly

The systemd equivalent puts the zone in the calendar expression itself, which is clearer because it survives a copied unit file:

[Timer]
OnCalendar=*-*-* 23:30:00 UTC
Persistent=true
AccuracySec=1s

Then make the schedule explain itself at the moment somebody saves it. This is the single change that catches the most mistakes, because it turns an abstract expression into five dates a human can read:

// printed in the form the moment a schedule is saved
// wanted: 02:30 local, zone Europe/Berlin
// local             utc                 note
// 2026-03-27 02:30  2026-03-27T01:30Z
// 2026-03-28 02:30  2026-03-28T01:30Z
// 2026-03-29 03:30  2026-03-29T01:30Z   02:30 does not exist, runs an hour late
// 2026-03-30 02:30  2026-03-30T00:30Z
// 2026-03-31 02:30  2026-03-31T00:30Z

Nobody reads a cron expression and sees the March row. Everybody reads the March row.

Finally, make a double run harmless. Give each run a slot, which is the scheduled instant in UTC, and claim it before doing any work:

create table job_run (
  job      text        not null,
  slot     timestamptz not null,
  started  timestamptz not null default now(),
  finished timestamptz,
  ok       boolean,
  primary key (job, slot)
);

/* first statement the job runs; zero rows means somebody already has this slot */
insert into job_run (job, slot) values ('nightly', $1)
on conflict do nothing
returning slot;

A file lock stops two copies on one host. The unique key stops two copies on two hosts, and it also stops a retry an hour later from redoing the work. That is the same property described in doing the work exactly once, and a schedule is one of the few places where you get the identifier for free.

How to check it worked

The timer list shows both clocks, which is what you want to see side by side:

systemctl list-timers nightly.timer
# NEXT                        LEFT     LAST                        PASSED  UNIT
# Sun 2026-05-24 23:30:00 UTC 2h 17min Sat 2026-05-23 23:30:00 UTC 21h ago nightly.timer

Then stop watching the crontab and watch the runs. One query answers the only question that matters, which is whether each job has succeeded recently enough:

select job,
       max(finished) filter (where ok)            as last_ok,
       now() - max(finished) filter (where ok)    as age
from job_run
group by job
order by age desc nulls first;

Alert when age passes the interval plus a margin. A job that disappears from the schedule entirely will trip this alert, and a job that runs and fails will trip it too, which is the behaviour you want from both.

What to watch out for

• Catch up replays cut both ways. A timer with persistence will run the job it missed while the machine was off, which is correct for a report and wrong for anything that sends a message. Decide per job, and write the decision down next to the schedule.
• Zone rules change when a government changes them, sometimes with a few weeks of notice. Keep the zone database updated with the rest of the operating system and restart long lived processes afterwards, because many runtimes read the zone once at startup and never again.
• One schedule on two machines is two schedules. A failover pair, a scaled service or a second instance someone started for testing will all fire. Put the lock where both can see it, not on local disk.
• A cron expression cannot say "the last working day of the month". Faking it with a wide day range plus an early exit is fine, as long as the early exit writes a line saying it skipped. A silent early exit is indistinguishable from a job that never ran.
• Half hour and forty five minute offsets exist. Code that assumes whole hour offsets will be wrong for real users, and the bug will look like an unrelated rounding error.

The general lesson is that a schedule is a piece of data with a zone, not a string, and it should be stored, validated and displayed like one. The cheapest version of this discipline costs an afternoon: pin every machine to UTC, keep the intended zone as its own field, print the next few runs when a human saves a schedule, and claim a slot before doing any work. After that, monitor the thing you actually care about, which is that the work happened, not that a line exists in a file. The same way of thinking makes logs worth reading, because both come down to recording what a run meant, not just that it started.