From 300 to 19,000 an hour: where sending throughput actually goes

A sending job on one platform we run handled about 300 messages an hour. The machine was bored: a few per cent of one core, almost no network traffic, a database that answered every query in single digit milliseconds. Adding another core did nothing, and a bigger server did nothing. The time was not being spent, it was being waited out, one message at a time.

Where the hour goes

Two structures were stacked on top of each other, and both of them cost the same thing.

The inner one is serial sending. The process submits a message, blocks until the remote side acknowledges it, then starts the next. Throughput in that shape is one divided by the service time. Measured per message the service time was about 1.1 seconds, almost all of it waiting for the acknowledgement, which puts the ceiling near 3,200 an hour even if the process ran without a break.

The outer one is the tick. A scheduled task started once a minute, loaded configuration, opened a connection, resolved the list, sent a capped batch and exited. The cap existed for a sensible reason: a tick must finish before the next one starts, so somebody set the batch to five. Five messages a minute is 300 an hour, and that number has nothing to do with the hardware. It is the batch size multiplied by sixty.

Each tick also paid for setup again: process start, configuration load, DNS, the handshake, the first query. Setup was roughly 300 milliseconds, spread over five messages, which is a fifth of the per message budget thrown away on work that a long lived process pays for once.

None of this shows up as a resource problem. There is nothing to see in the load average, nothing in the query log, nothing in the network graph. The only visible symptom is a campaign that takes three days.

How to see it

Do not measure throughput first. Measure where a single message spends its wall time, by phase, and print the averages:

# send.log: one line per phase, "id phase milliseconds"
awk '{ sum[$2] += $3; n[$2]++ }
     END { for (p in sum) printf "%-12s %6d calls %8.1f ms avg\n", p, n[p], sum[p]/n[p] }' send.log

connect            60 calls    212.4 ms avg
auth               60 calls     88.1 ms avg
submit            300 calls     14.9 ms avg
await_ack         300 calls    861.3 ms avg
db_write          300 calls      7.6 ms avg

That table decides the whole plan. Submitting costs nothing, the database costs nothing, connecting is expensive but only once per tick, and 86 per cent of the message budget is spent waiting for an answer from someone else. Waiting is the one cost that parallelism removes, because waits overlap.

The second measurement is how many messages actually leave per minute, straight from the table that records them:

SELECT DATE_FORMAT(sent_at, '%Y-%m-%d %H:%i') AS minute, COUNT(*) AS sent
FROM messages
WHERE sent_at >= UTC_TIMESTAMP() - INTERVAL 20 MINUTE
GROUP BY 1 ORDER BY 1 DESC;

A flat five per minute is not a performance curve, it is a configured limit. Flat numbers usually mean a cap somewhere, and finding it is faster than tuning anything.

Before any of this, confirm the campaign is running at all. A job that never became due because of a time zone mismatch looks exactly like a slow one on a dashboard that counts sends per hour.

The fix

Four changes, in the order I would make them.

Stop paying setup per message. Replace the per minute task with a long lived worker that keeps its connection open and re-authenticates only when the connection drops. That alone removes the 300 milliseconds of setup and the artificial batch cap.

Batch the database work. One insert with many rows instead of one insert per message, and one status update per batch. At 7.6 milliseconds a row this is not the bottleneck yet, but it becomes one the moment the send rate goes up twenty times.

Run the waits in parallel, with the pool sized from the measurement. Concurrency needed is target rate multiplied by service time. For five and a half messages a second at 1.1 seconds each, that is six in flight, so a pool of eight covers variance without pretending the remote side is infinite:

const RATE_PER_SEC = 25;   // provider ceiling is 30, leave headroom
const POOL = 8;            // 5.5/s target * 1.1 s measured = 6, plus variance

const bucket = tokenBucket(RATE_PER_SEC);
const queue = boundedQueue(POOL * 4);   // back pressure: fill blocks when full

async function worker() {
  for (;;) {
    const item = await queue.take();    // blocks, no unbounded in memory list
    if (!item) return;
    await bucket.take();                // blocks until a token is free
    const res = await submit(item);
    results.push(res);                  // flushed in batches by a writer task
  }
}

await Promise.all(Array.from({ length: POOL }, worker));

The bounded queue matters as much as the pool. A reader that loads a hundred thousand rows into an array to feed eight workers has moved the problem into memory. Fill the queue from a cursor, let it block, and the producer slows down on its own.

Check that the setting you are about to change is read by anything. This is where the hours went on my first attempt. The platform had a configuration key whose name promised parallel sending per campaign. It was in the config file, it was in the documentation, and setting it changed nothing, which made parallelism look like a dead end. The code never read it:

grep -rn "parallel_processes_per_campaign" ./app ./lib ./config
# config/app.ini:118:parallel_processes_per_campaign = 5
# docs/settings.md:240:| parallel_processes_per_campaign | number of ...

Two hits, both of them documentation. The behaviour was controlled by two other keys: whether forking was enabled at all, and how many batches were allowed to run at once. Settings outlive the code that reads them, and a stale key is worse than a missing one, because it answers the question you were about to ask.

How to check it worked

The same query, before and after:

# before
2025-05-05 09:56     5
2025-05-05 09:57     5
2025-05-05 09:58     5

# after
2025-05-05 10:29   318
2025-05-05 10:30   326
2025-05-05 10:31   331

About 326 a minute is 19,560 an hour. Note that per message latency went from 1.1 to roughly 1.4 seconds under load, which is normal and is exactly why the pool is sized with headroom: eight workers at 1.4 seconds each is about 5.7 a second, and the measured 5.4 a second sits just under it. The rate limiter at 25 a second is not the binding constraint here, it is the guard rail for the day the remote side gets faster.

What to watch out for

• Database connections multiply. Eight senders, a writer and a scheduler can hold more connections than the whole application did before. Size the pool explicitly and check the server's maximum, or you will trade a slow campaign for a site that cannot log anyone in.
• File descriptors run out quietly. Each in flight connection is a descriptor, and the default limit per process is low enough to hit within a day of raising concurrency. Raise it deliberately and watch for errors about too many open files.
• The provider breaks before your machine does. Above the contracted rate you start getting throttle answers, and a client that records those as permanent failures throws away work you paid for. Treat them as a signal to slow down and retry, and keep the item in the queue.
• Throughput that ends in a spam folder is not throughput. If you are sending mail, the authentication side has to be right first, because an unaligned From domain makes a faster sender an efficient way to damage a reputation.
• Parallel workers finish out of order. Anything that assumed sequential completion, such as progress counters, per customer fairness or ordered numbering, needs saying explicitly now.

The lesson generalises past sending. When a process is slow and the machine is idle, the time is going into waiting, and waiting is the only cost that concurrency actually removes. Measure one unit of work by phase, work out how many of those waits have to overlap to hit the number you want, and set the pool to that figure plus a margin. Then go and find out what the new bottleneck is, because there is always one, and it is better to meet it on a Tuesday afternoon than during the first large campaign.