Idempotent jobs: doing the work exactly once

A queue worker sends the same notification twice. Nothing in the code sends twice, the logs show one job accepted, and the job clearly ran once. What happened is ordinary: the worker finished the work, the process died before the acknowledgement reached the broker, and the broker did what it promises to do. It delivered the message again.

What actually happens

Queues, webhook senders and background schedulers almost all guarantee at least once delivery. The sender keeps a message until it is acknowledged, and if the acknowledgement does not come back in time it sends the message again. That is not a defect. Losing a payment notification is worse than sending it twice, so every serious system is built to prefer the duplicate.

The duplicate arrives in more situations than most people assume:

1. The worker completed the side effect and was killed before it could acknowledge.
2. The acknowledgement was sent and lost on the way back.
3. The job took longer than the visibility timeout, so the broker decided it was lost and gave it to a second worker while the first one was still running.
4. Someone redelivered a batch by hand after an incident, which is the most common one of all.
5. The sender retried because your endpoint answered slowly, even though it answered correctly.

Case three is the one that surprises people, because it means two copies of the same job can run at the same time rather than one after the other. Any defence that is a check followed by a write will fail there: both workers look, both see nothing, both proceed.

How to see it

You usually find out from the data rather than the logs. Group by whatever should be unique and count:

SELECT external_event_id, count(*) AS runs
FROM notifications
WHERE created_at > now() - interval '7 days'
GROUP BY external_event_id
HAVING count(*) > 1
ORDER BY runs DESC
LIMIT 20;

If that returns rows, you are already running the work more than once and only noticing now. Two more places confirm it. The broker exposes a delivery count or a redelivered flag per message, and logging it costs nothing:

logger.info('job received', {
  jobId: msg.id,
  attempt: msg.deliveryCount,
  eventId: msg.body.eventId,
});

Then search for a single event id in the logs. A job that appears with attempt: 1 and attempt: 2 a minute apart, both reaching the same success line, is the whole bug in two lines of output. This is also why an event id belongs in every log line for the job, the same way a delivery report needs a record to attach itself to before it means anything.

The fix

Make the database decide who is first. A unique constraint is the only arbiter that works when two workers race, so the key goes into a table of its own with the result beside it:

CREATE TABLE job_runs (
  key          text PRIMARY KEY,
  status       text NOT NULL,
  result       jsonb,
  attempts     int NOT NULL DEFAULT 1,
  created_at   timestamptz NOT NULL DEFAULT now(),
  completed_at timestamptz
);

The handler then claims the key before it does anything else. The conflict clause is what turns a race into a decision:

async function handle(event) {
  const key = `notify:${event.id}`;

  const claimed = await db.query(
    `INSERT INTO job_runs (key, status)
     VALUES ($1, 'running')
     ON CONFLICT (key) DO NOTHING
     RETURNING key`,
    [key],
  );

  if (claimed.rowCount === 0) {
    const prior = await db.one(
      `SELECT status, result FROM job_runs WHERE key = $1`, [key],
    );
    if (prior.status === 'done') return prior.result;
    throw new RetryLater('a run is already in progress');
  }

  const result = await doTheWork(event);

  await db.query(
    `UPDATE job_runs
     SET status = 'done', result = $2, completed_at = now()
     WHERE key = $1`,
    [key, result],
  );

  return result;
}

Three properties matter here. The insert either succeeds or reports a conflict in one atomic step, so exactly one worker gets to run. A repeat delivery after success returns the stored result, which means the caller sees the same answer rather than a new one. A repeat delivery while the first run is still going is told to come back later rather than being allowed to run in parallel.

The last case needs a time limit. If a worker dies between claiming and completing, the key stays in running and nothing will ever retry it. A sweeper that resets rows stuck in running for longer than the longest reasonable job duration fixes that, and the reset should bump attempts so a job that keeps dying becomes visible instead of looping quietly.

Side effects you cannot take back

Sending money and sending a message do not roll back with the transaction. For those, claim the key before the call, not after, and give the other system the same key:

await claim(key);
await provider.charge({
  amount: order.total,
  idempotencyKey: key,
});
await markDone(key);

If the process dies between the call and markDone, the key is stuck in running and the sweeper will retry. The retry sends the same idempotency key, the provider recognises it and returns the original charge instead of making a second one. Most payment and messaging APIs accept a key like this, and the ones that do not will at least have a lookup by your own reference, which you can check before calling again. When the answer is a rate limit rather than an error, hold the key and retry later instead of failing the job, for the reasons in treating a rate limit as a permanent failure.

For inserts of your own data, the simpler form is enough. An upsert with a natural key does the same job without a separate table:

INSERT INTO recipients (list_id, phone, name)
VALUES ($1, $2, $3)
ON CONFLICT (list_id, phone) DO UPDATE SET name = excluded.name;

That is what makes a large import safe to rerun after a failure, which matters when the import itself has been moved into a background queue and can be restarted from any point.

How to check it worked

Call the handler twice with the same event and assert that the work happened once:

const a = await handle({ id: 'evt_1', to: '900000000', body: 'test' });
const b = await handle({ id: 'evt_1', to: '900000000', body: 'test' });

assert.deepEqual(a, b);
assert.equal(await count('SELECT count(*) FROM notifications'), 1);

Then run the same thing concurrently, because the sequential test passes even with a check and write implementation:

const results = await Promise.allSettled(
  Array.from({ length: 5 }, () => handle(event)),
);

One should succeed, four should either return the stored result or ask to retry, and the table should still hold one row. In production, keep the duplicate query from the start of this post as a scheduled check. A handler that used to produce duplicates and now produces none is the only proof that counts.

What to watch out for

• A key built from the current time, a random value or the attempt number is not a key. Every retry generates a new one and the constraint never fires.
• A unique index on a nullable column does not stop duplicate nulls in most databases. If part of the key can be absent, use a generated column or a partial index with a condition that makes the intent explicit.
• Idempotent is not the same as ordered. Two different events can still arrive in the wrong order, so a handler that sets state should also check that it is not applying an older state over a newer one.
• The insert and the external call cannot be in one transaction. Accept that there is a window where the key exists and the call has not happened, and design the sweeper for it, rather than pretending the window is not there.
• The table grows forever unless someone deletes from it. Set a retention window on day one and check that the job actually runs.

Every system that retries will eventually hand you the same work twice, usually at the least convenient moment. The cheapest insurance is a key the sender repeats, a unique constraint that decides who goes first, and a stored result that lets the second run answer without doing anything. It is a small amount of code, it fits in one function, and it turns a class of incident that is painful to investigate into a row that already exists.