The invisible suffix that split every message in two

A campaign of a hundred thousand messages cost twice what the estimate said. The interface had shown one segment per message, the drafts were well inside the limit, and nobody had changed the text. The messages that left the process were ten characters longer than the messages that were measured, because two different features each appended five characters after the measurement had already happened.

What actually happens

Both features were reasonable on their own and both were built months apart.

The first was a duplicate guard. Some routes drop a message that is identical to one they carried a moment ago, so the sender appends a short random string to each body. Five characters, added at submit time, invisible to the recipient on a busy screen.

The second was a rotating tag used to tell one batch from another in the reports, also five characters, also appended at submit time, also by code that had no idea the first feature existed.

The composer did its job correctly at the wrong moment:

/* the order that costs money */
const draft = renderTemplate(template, contact);   /* 155 characters */
const quote = analyse(draft);                      /* GSM7, 155 units, 1 segment */
showToUser(quote);

/* ... later, deep in the sender ... */
const body = draft + " " + nonce() + " " + rotatingTag();  /* 165 characters */
await provider.submit({ to, body });                       /* 2 segments */

Every number the user saw came from draft. Every number the provider billed came from body. The two strings were never compared to each other, so the gap could sit there for as long as nobody happened to add up an invoice.

What makes it hard to notice is that nothing looks wrong at any point. The text on the screen is under the limit. The text on the phone looks normal, because a two segment message is reassembled before it is displayed. The logs contain the composed draft, because that is what was interesting when the logging was written. The only artefact that contains the truth is the bill, and the bill arrives weeks later as a single number.

This is a general shape, not a messaging problem. It appears wherever a value is validated and then modified:

• A compliance layer appends a mandatory opt out line to the body.
• A link rewriter replaces a short URL with a longer tracked one.
• A template renderer expands a placeholder into a value longer than the placeholder.
• A formatter adds a trailing newline, or a normaliser converts a character into two.
• A transliteration step changes the encoding, which changes the segment size from 67 to 153 or the other way round.

Each of those is a mutation applied after the length check, and every one of them can cross a boundary that somebody already told the user they were safely inside.

How to see it

Do not reason about it. Log both strings and compare them over real traffic. Two fields at quote time and two at submit time are enough:

log.info("quote",  { id, len: draft.length, seg: analyse(draft).segments });
log.info("submit", { id, len: body.length,  seg: analyse(body).segments });

Then join them for a day:

grep -h '"quote"\|"submit"' /var/log/app/send.log \
  | node scripts/pair-by-id.js \
  | awk '$2 != $4 { print }' \
  | head -20
# id=01J1... quote_len=155 submit_len=165  quote_seg=1 submit_seg=2
# id=01J1... quote_len=148 submit_len=158  quote_seg=1 submit_seg=1

The first line is the bug. The second line is the same mutation on a shorter body, where it costs nothing and therefore never gets reported. That is why this survives: the mutation is harmless for most of the length range and expensive in a narrow band near the boundary, and the messages that land in that band are exactly the ones written to use the full segment.

If you already store the payload you sent, you can skip the logging and ask the database directly. If you do not store it, the rest of this post is an argument for starting.

The fix

The rule is one sentence: measure the string you are about to send, not the string you are about to modify.

Put every mutation in one place, in order, and give each one a name:

type Step = { name: string; apply(text: string, ctx: Ctx): string };

const PIPELINE: Step[] = [
  renderTemplate,
  transliterate,
  appendOptOut,
  appendNonce,
  appendRotatingTag,
];

export function buildPayload(draft: string, ctx: Ctx) {
  const body = PIPELINE.reduce((t, step) => step.apply(t, ctx), draft);
  return { body, ...analyse(body) };   /* measured after everything */
}

Now buildPayload is the only thing that produces a payload, and it is also the only thing that produces a number. The interface calls it with a representative context so the counter on the screen reflects the real result. The sender calls it and submits body without touching it. The billing estimate calls it and multiplies. Nothing downstream is allowed to append.

Two details make it stick.

The first is that the interface must show the overhead honestly. If the pipeline will add ten characters, the counter says 145 remaining, not 155. People write to the number in front of them, and a number that lies by ten characters will produce messages that sit exactly on the wrong side of the boundary.

The second is to look hard at whether you need both mutations. In this case the duplicate guard and the rotating tag existed for nearly the same reason, and one of them was enough. Removing a feature is a better fix than accounting for it, and it is the only fix that makes messages shorter rather than more expensive. What remains, count it: if the pipeline adds five characters, the segment calculation must include those five characters everywhere, including in the character set and segment maths that decides whether you are working in units of 153 or 67.

What this does not fix is a mutation applied outside your process. If the route itself prefixes something to the body, no amount of internal discipline will show it. For that you need a real delivery, read on a real handset, compared against what you sent.

How to check it worked

The test that keeps this fixed does not test the pipeline. It tests that the number you quoted and the bytes you sent are the same thing, by capturing at the transport boundary:

it("sends exactly the payload it quoted", async () => {
  const sent: string[] = [];
  const provider = { submit: async (m: any) => { sent.push(m.body); return { id: "x" }; } };

  for (const len of [1, 69, 70, 71, 142, 143, 144, 152, 153, 154, 160, 161]) {
    sent.length = 0;
    const draft = "x".repeat(len);
    const quote = buildPayload(draft, ctx);

    await send({ draft, to: "0000", provider, ctx });

    expect(sent).toHaveLength(1);
    expect(sent[0]).toBe(quote.body);                     /* byte for byte */
    expect(analyse(sent[0]).segments).toBe(quote.segments);
  }
});

The lengths in that list are chosen on purpose: each one sits on or next to a boundary in one encoding or the other. A regression that adds one character to the payload fails this test on at least two of them, which is the behaviour you want, because a regression that adds one character is otherwise completely invisible.

After deploying, the observable result is in the data rather than the test suite:

SELECT sum(predicted_segments) AS predicted,
       sum(billed_segments)    AS billed
FROM messages
WHERE created_at > now() - interval '1 day';

Those two numbers should be equal, or differ by the handful of rows where a report never came back. Any systematic gap is another mutation you have not found yet.

What to watch out for

• A mutation that only sometimes applies is worse than one that always applies. An opt out line added only on the first message of the month will pass every test run on a Tuesday.
• Storing the draft in the database instead of the payload makes this bug unprovable after the fact. Store what you sent, or at least its length and a hash.
• The interface counter and the sender must import the same function. A second implementation written for the browser will be wrong within a release, and it will be wrong in the direction of promising fewer segments.
• Encoding changes the boundary, so an added character can also change the encoding. A suffix containing a single accented letter moves a 150 character message from one segment to three, not two.

The habit worth taking from this is to be suspicious of every step that happens between a promise and an action. A validator, a counter, a quote and a confirmation dialogue are all promises made about a value, and each one is only true until the next function touches that value. Move the promise to the last possible moment, keep the transformations in one visible list, and write the test against the thing that actually crosses the process boundary. The same discipline is what makes an asynchronous callback match the record it belongs to: trust the artefact that leaves or enters the process, not the one you were holding when you decided what to say about it.