Treating a rate limit as a permanent failure is expensive

A queue drained faster than usual and a large share of the items came back marked failed. There was nothing wrong with any of them. The remote side had answered with the equivalent of too many requests, and the client wrote that down in the same bucket as a malformed payload. Work that had already been paid for was gone, and the dashboard showed a quality problem when the actual problem was that we were sending faster than the agreement allowed.

What actually happens

Two signals mean the same thing in different protocols.

Over HTTP it is status 429, often with a Retry-After header naming a number of seconds or a date. Over SMPP it is a submit response with a throttling status, and a neighbouring one that says the remote queue is full. Both are explicitly temporary: the request was fine, the timing was not.

A naive client has two buckets. Anything that is not a success is a failure. So the path looks like this:

1. The item is taken from the queue and submitted.
2. The answer is a throttle.
3. The client writes status REJECTED on the item, which is a terminal state.
4. The retry logic only looks at items in a retryable state, so it never sees this one again.
5. The queue empties sooner and the throughput graph goes up.

That last point is what makes this survive review for so long. Throwing work away looks like getting faster. The per hour count of attempted sends rises, the queue drains, and the only place the loss is visible is a count of failures that nobody can attribute to anything.

There is a second cost that outlives the lost items. A flat error count destroys the information in the signal. Eleven minutes of running above the contracted rate and eleven minutes of a genuinely broken integration produce the same number on the same chart. You cannot tell whether to fix the code, buy more capacity or slow down, because all three look identical from where you are standing.

How to see it

Count by what the remote side said, not by what you stored. If those two are the same column, that is already the bug:

SELECT provider_code,
       internal_status,
       COUNT(*) AS n
FROM send_attempts
WHERE created_at >= UTC_TIMESTAMP() - INTERVAL 1 HOUR
GROUP BY 1, 2
ORDER BY n DESC;

provider_code  internal_status      n
200            SUBMITTED        41230
429            REJECTED          4812
0x58           REJECTED          1907
400            REJECTED            58

Nearly seven thousand items in a terminal state because of timing, and fifty eight because of an actual problem with the request. Then look at the shape over time, one minute at a time:

SELECT DATE_FORMAT(created_at, '%H:%i') AS minute,
       SUM(CASE WHEN provider_code IN ('429', '0x58') THEN 1 ELSE 0 END) AS throttled,
       COUNT(*) AS attempts
FROM send_attempts
WHERE created_at >= UTC_TIMESTAMP() - INTERVAL 30 MINUTE
GROUP BY 1 ORDER BY 1;

A plateau rather than a spike means you are sitting on a ceiling. That is the signature of pacing, and it usually appears the same week somebody raises the size of the worker pool.

The fix

Replace two buckets with three, then let the classification drive the state machine.

Terminal means the request will never succeed as written: an invalid recipient, a blocked destination, a malformed payload, a rejected sender, a failed authentication. Mark it failed, record the reason, stop.

Retryable means the request was fine and the timing was not: throttling, a full remote queue, a 5xx, a connection reset, a timeout with no answer.

Unknown is everything you have not classified yet. Treat it as retryable with a low attempt cap and log it loudly enough that it gets classified properly within a release or two.

const TERMINAL = new Set([400, 401, 403, 404, 422]);

function classify(res) {
  if (res.status === 429) return { kind: 'retry', afterMs: retryAfterMs(res) };
  if (res.status >= 500) return { kind: 'retry', afterMs: null };
  if (TERMINAL.has(res.status)) return { kind: 'terminal', reason: res.body?.code };
  if (res.status === 200) return { kind: 'ok' };
  return { kind: 'unknown' };
}

// protocol level equivalents, same three kinds
const THROTTLE_STATUSES = new Set([0x58, 0x14]);

Then the delay. Honour Retry-After when the remote side sends one, because that is the only authoritative number available. Otherwise use full jitter, which means a random point inside the current backoff window rather than the edge of it:

function nextDelayMs(attempt, afterMs) {
  if (afterMs) return afterMs;
  const window = Math.min(60000, 500 * 2 ** attempt);  // 0.5s, 1s, 2s ... cap 60s
  return Math.floor(Math.random() * window);           // full jitter
}

Without the random part, every item throttled in the same second comes back in the same second and rebuilds the burst. I have watched a system spend twenty minutes oscillating between a stampede and an idle connection for exactly this reason.

The state machine is small enough to write out:

QUEUED     -> IN_FLIGHT
IN_FLIGHT  -> SUBMITTED   accepted by the remote side
IN_FLIGHT  -> PENDING     retryable; attempts += 1, next_attempt_at set
PENDING    -> QUEUED      when next_attempt_at is reached
PENDING    -> DEFERRED    attempts above the cap; waits for a person
IN_FLIGHT  -> FAILED      terminal only, with a reason stored

next_attempt_at is a scheduled instant like any other, so store it in UTC and compare it against a UTC clock, or you will rediscover the time zone bug that leaves rows sitting in a queue in a new place.

One more piece, and it is the one people skip. The throttle has to change the sender, not only the item. If every item backs off politely while the pool keeps submitting at the same rate, you stay at the ceiling and everything queues. Feed the throttle rate into the pacing: when it goes above a small threshold, cut the token rate by half, then raise it in small steps while the rate stays clean. Slow to increase, quick to decrease.

How to check it worked

Run the same hour of traffic and compare three numbers rather than one:

                       before      after
attempts               48007       44113
throttled               6719        1204
terminal failures       6777          61
delivered              41230       44052

Terminal failures fall back to the number of genuinely bad items, and the delivered count rises even though attempts fell, because attempts are no longer being wasted on work that is thrown away. The throttle count does not need to reach zero. It is a control signal, and a small steady number means the pacing is sitting just under the ceiling, which is where it should be.

What to watch out for

• A retry is only safe if the operation is idempotent. Send a stable identifier of your own with every attempt and let the remote side recognise a repeat, otherwise an ambiguous timeout turns into paying twice.
• Do not retry terminal errors. An unreachable destination retried six times is six times the cost and the same answer, and it pushes real work further down the queue.
• Keeping items pending means the queue depth is no longer a progress bar. Expose in flight, waiting and done as separate numbers, or support will report a stuck queue every time backoff is doing its job.
• Cap attempts per item and per time window. An item that retries forever is an outage of a different kind, and it is harder to notice because nothing is marked failed.
• Alert on the throttle rate and on the age of the oldest pending item, not on the total error count. Those two say whether you are pacing correctly and whether anything is actually stuck.

The wider habit is to treat a remote system's answers as a vocabulary rather than a pass or fail. Every integration has at least three kinds of no: not ever, not like this, and not right now. Collapsing them into one loses the only information that tells you what to do next, and in a system where each item costs money it loses the item as well. Write the classification down in one place, keep the retryable ones in the queue, and let the rate at which you are told to slow down become a number you watch on purpose.