Payment webhooks: signatures, replays and idempotency keys

A callback endpoint is the one place in an application where a stranger gets to say the words that move money. It receives a POST, and the body claims that order 4711 was paid. Most implementations I am asked to review parse that body, find the order, mark it paid, and return 200. Three small additions separate that from an endpoint you can leave exposed to the internet.

What actually happens

The provider signs the request. It takes the exact bytes of the body, usually combined with a timestamp, and computes an HMAC with a secret only the two of you share. The header carries that digest and the timestamp beside it. You recompute the same value and compare.

Four things go wrong in practice, and they go wrong in roughly this order.

1. The web framework parses the body into an object before your code sees it. Verifying against a re-serialised copy fails, because key order and spacing changed. Developers then work around it by verifying nothing.
2. The comparison is a normal equality check, which returns as soon as two bytes differ. Repeated requests turn that timing difference into a way of guessing the digest one byte at a time. It is a slow attack and it is a real one.
3. A signature never expires. Anyone who gets hold of a valid past request, from a proxy log, an error report or a monitoring trace, can send it again. It verifies, and the order gets paid twice.
4. The amount in the body is taken at face value. If the field is trusted without being checked against your own record, a manipulated or mismatched payment is accepted as full payment.

How to see it

Two curl calls tell you where you stand. The first sends a body with no signature at all:

curl -s -o /dev/null -w "%{http_code}\n" \
  -X POST https://app.example.com/webhooks/payments \
  -H "content-type: application/json" \
  -d '{"event_id":"evt_1","order_id":4711,"status":"paid","amount":100}'

If that prints 200, the endpoint is not verifying anything and the order is now paid. The second takes a genuine past request out of your logs and sends it again, unchanged. A 200 there and a second credit on the account means there is no replay protection and no idempotency.

While you are in the logs, check what they contain. A raw body stored in full for a payment callback is a copy of customer data you did not intend to keep, and it is the same class of exposure as serving a dotfile because of a misconfigured root. Log the event id and the signature verdict, not the payload.

The fix

Capture the raw body on that route only, verify, then parse. In an Express style application that is one line of middleware and about twenty lines of handler:

import crypto from 'node:crypto';

app.post('/webhooks/payments',
  express.raw({ type: 'application/json' }),
  async (req, res) => {
    const signature = req.get('x-signature') || '';
    const timestamp = req.get('x-timestamp') || '';

    const age = Math.abs(Date.now() / 1000 - Number(timestamp));
    if (!timestamp || Number.isNaN(age) || age > 300) {
      return res.status(400).send('stale');
    }

    const expected = crypto
      .createHmac('sha256', process.env.WEBHOOK_SECRET)
      .update(`${timestamp}.`)
      .update(req.body)
      .digest('hex');

    const a = Buffer.from(expected, 'utf8');
    const b = Buffer.from(signature, 'utf8');
    if (a.length !== b.length || !crypto.timingSafeEqual(a, b)) {
      return res.status(401).send('bad signature');
    }

    const event = JSON.parse(req.body.toString('utf8'));

    const stored = await db.query(
      `INSERT INTO payment_events (id, type, payload, received_at)
       VALUES ($1, $2, $3, now())
       ON CONFLICT (id) DO NOTHING`,
      [event.id, event.type, event],
    );

    if (stored.rowCount === 1) await queue.add('payment-event', { id: event.id });

    return res.status(200).send('ok');
  });

Read it as four decisions. The timestamp is checked first and cheaply, so a flood of replayed requests never reaches the HMAC. The digest covers the timestamp and the raw bytes together, which is what stops someone reusing a signature with a new timestamp. timingSafeEqual compares in constant time, after a length check, because it throws on mismatched lengths. And the insert with ON CONFLICT DO NOTHING is what makes the whole endpoint idempotent: a repeated delivery finds the row already there, enqueues nothing, and still answers 200 so the provider stops retrying.

The handler ends there. Everything else happens in the worker, which has time to be careful:

async function processPaymentEvent({ id }) {
  const { payload } = await db.one(`SELECT payload FROM payment_events WHERE id = $1`, [id]);
  const order = await db.one(`SELECT id, total, currency, status FROM orders WHERE id = $1`,
    [payload.order_id]);

  if (order.status === 'paid') return;
  if (order.total !== payload.amount || order.currency !== payload.currency) {
    await flagForReview(order.id, 'amount mismatch');
    return;
  }

  await markPaid(order.id, id);
}

The order was created by you, before the payment started, with a total you calculated. The callback is a claim about that order and the worker checks it. A mismatch is not an error to throw, it is a case for a human to look at, and it should be rare enough that someone actually looks.

The retry behaviour of the worker matters as much as the endpoint. Since the provider delivers at least once and your own queue does the same, the job needs the properties described in doing the work exactly once. The event id is already a natural idempotency key, so use it as one.

How to check it worked

Four requests, four different answers:

BODY='{"id":"evt_test_1","type":"payment.succeeded","order_id":4711,"amount":100}'
URL=https://app.example.com/webhooks/payments

send() {
  TS=$1
  SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -r | cut -d' ' -f1)
  curl -s -o /dev/null -w "$2: %{http_code}\n" -X POST "$URL" \
    -H "content-type: application/json" \
    -H "x-timestamp: $TS" -H "x-signature: ${3:-$SIG}" -d "$BODY"
}

send "$(date +%s)" valid            # 200
send "$(date +%s)" tampered  deadbeef  # 401
send "$(( $(date +%s) - 7200 ))" old   # 400
send "$(date +%s)" repeat           # 200

The fourth one is the important check, and the status code is not what proves it. Count the rows afterwards: payment_events has one row for that id, the order has one payment, and the queue received one job.

Then build the safety net. Once a day, pull the provider's list of settled payments for the previous day and compare it with your own:

SELECT o.id, o.total, o.status, p.id AS event_id
FROM orders o
LEFT JOIN payment_events p ON p.payload->>'order_id' = o.id::text
WHERE o.created_at::date = current_date - 1
  AND (p.id IS NULL OR o.status <> 'paid');

Reconciliation is the only check that catches what the endpoint missed: a delivery that never arrived, a webhook you rejected during an outage, a refund processed on the provider side that your database knows nothing about. A signature verifies one message. Reconciliation verifies the day.

What to watch out for

• Do not return 500 for a business problem. A non-2xx answer tells the provider to retry, so an unknown order or an amount mismatch should be recorded and answered with 200, while a genuine failure to store the event should fail loudly and be retried.
• An IP allowlist is a useful second lock and a poor first one. Addresses change without notice and a request from the right address is still unauthenticated.
• Support both the current and the previous secret during a rotation, and accept a request if either verifies. Without that, rotation means a window of rejected payments.
• Test mode and live mode events can reach the same endpoint. Check the environment flag in the payload and refuse the ones that do not belong, or a test payment will mark a real order paid.
• Events arrive out of order. A refund can reach you before the charge it refunds, so let the worker handle an event whose parent it has not seen yet, the same way it handles a delivery report that arrives first.

A payment callback is an untrusted request that happens to be signed. Verify the bytes, reject anything old, write the event id down once, and treat the numbers inside as something to confirm against a record you created yourself. None of it is more than an hour of work, and it turns the most attractive endpoint in the application into one of the least interesting.