Why one Turkish character can double your message cost

A message of 130 characters goes out as one billed unit. Somebody edits the text, changes one word, and the same campaign costs twice as much. Nothing else changed: the audience is the same, the length is nearly the same, the provider is the same. The word that changed contained a letter that does not exist in the 7 bit alphabet, and that one letter pushed the entire message into a 16 bit encoding where only 70 characters fit.

What actually happens

A short message is carried as 140 bytes of payload. There are two ways to fill those bytes and they are not close to each other:

The 7 bit alphabet packs eight characters into seven bytes, which is how 160 characters fit in 140 bytes. It contains ASCII, a handful of currency symbols, a few Greek capitals and a specific set of accented letters. Everything else on earth falls into the 16 bit encoding, which stores two bytes per code unit and therefore holds 70.

The rule that costs money is that the encoding is chosen for the whole message. There is no mixing. One character outside the table and every other character in the text is re encoded too.

For Turkish this is sharper than people expect. The basic table contains Ö, ö, Ü, ü and the uppercase Ç. It does not contain the lowercase ç, and it contains neither case of ğ, ı, İ or ş. So ÇAY is a 7 bit word and çay is not, which is the kind of asymmetry that makes a character counter written by hand disagree with the invoice.

The second cost is quieter. Ten characters live in an extension table and are sent as an escape followed by the character, so they take two units each inside the 7 bit alphabet:

^  {  }  \  [  ~  ]  |  €  and the form feed

A message with three pairs of curly braces in a template is six units longer than its length suggests. At 155 characters that difference is the segment boundary.

When a message does not fit, it is split and each part carries a six byte header that says which part it is. That header is taken out of the payload, which is where 153 and 67 come from. A 130 character text is one segment in the 7 bit alphabet. The same text in 16 bit is 130 units against a 67 unit segment, so it is two. Push it to 140 and it is three. The bill follows the segment count, not the character count.

How to see it

Write the counter as a function first and look at the numbers it produces for a real sentence. The alphabet is a literal, not a regular expression:

const BASIC = new Set([...
  "@£$¥èéùìòÇ\nØø\rÅåΔ_ΦΓΛΩΠΨΣΘΞÆæßÉ !\"#¤%&'()*+,-./0123456789:;<=>?¡" +
  "ABCDEFGHIJKLMNOPQRSTUVWXYZÄÖÑܧ¿abcdefghijklmnopqrstuvwxyzäöñüà"]);

const EXT = new Set([..."\f^{}\\[~]|€"]);

export function analyse(text: string) {
  let units = 0;
  for (const ch of text) {
    if (BASIC.has(ch)) units += 1;
    else if (EXT.has(ch)) units += 2;
    else return ucs2(text);
  }
  return { encoding: "GSM7", units, segments: units <= 160 ? 1 : Math.ceil(units / 153) };
}

function ucs2(text: string) {
  const units = text.length;   /* UTF-16 code units, so an emoji is two */
  return { encoding: "UCS2", units, segments: units <= 70 ? 1 : Math.ceil(units / 67) };
}

Then run it against the same sentence twice, once with an accent and once without:

node -e '
const { analyse } = require("./segments");
const t = "Siparisiniz kargoya verildi, takip numaraniz mesajin devaminda yer aliyor. Iyi gunler dileriz.";
console.log(analyse(t));
console.log(analyse(t.replace("Siparisiniz", "Siparişiniz")));
'
# { encoding: 'GSM7', units: 94, segments: 1 }
# { encoding: 'UCS2', units: 94, segments: 2 }

One letter, one extra segment, and the character count on the screen did not move. This is also the moment to check what your interface is telling people, because a counter that shows 94 / 160 while the provider charges for two segments is worse than no counter at all.

The fix

Four changes, and only the first one is really about code.

1. One function, one place. The interface, the API, the background worker and the billing estimate all import the same analyse. Two implementations of this rule will disagree within a month, and the one in the interface is always the optimistic one.
2. Show the real number before the send. The counter should say the encoding, the units used, the segment count and the resulting cost, and it should update while typing. People edit their text when they can see a segment boundary approaching, and nobody edits an invoice.
3. Offer transliteration as a choice, with a preview. Converting ş to s, ğ to g, ı to i, ç to c, ö to o and ü to u turns a two segment message back into one. Show the converted text next to the original and let the sender approve it.
4. Refuse to guess on the characters that matter. Names, addresses, legal notices and any figure the recipient has to read back should keep their letters. A campaign body can be transliterated. A person's surname should not be, and neither should a sentence whose meaning changes when a letter is dropped.

Case conversion is a trap in its own right here. If you fold text to uppercase to squeeze into the basic table, you are running straight into the dotted and dotless i problem, and a locale unaware uppercase turns ı into I in some places and i into I in others. Pick the transliteration table explicitly instead of relying on a normalisation library to do something sensible.

There is a national language shift table in the standard that allows Turkish letters inside a 7 bit message. I do not build on it. Support across routes and handsets is inconsistent, and a message that arrives as a row of question marks costs the same as one that arrives correctly.

How to check it worked

The check is not a unit test, it is a comparison against what you were charged. Store the predicted segment count on every row at send time and compare it to the count that comes back:

SELECT encoding,
       predicted_segments,
       billed_segments,
       count(*) AS rows
FROM messages
WHERE created_at > now() - interval '1 day'
GROUP BY 1, 2, 3
ORDER BY rows DESC
LIMIT 20;

Every row where the two numbers differ is a bug in the counter, and the pattern tells you which one. A difference of one on long 7 bit messages usually means the extension characters are being counted as one unit. A disagreement about the encoding itself usually means an invisible character, a non breaking space or a typographic quote pasted from a word processor, sitting in the text where nobody can see it.

What to watch out for

• The lowercase ç is not in the basic table while the uppercase Ç is. Any counter built by listing "the Turkish letters" will get this one wrong in the direction that costs money.
• A segment split must not fall between an escape and its extension character. If you pack segments with a simple division rather than a greedy loop, a message ending in } at the wrong offset will lose the character on arrival.
• The 16 bit count is in UTF-16 code units. An emoji is two, and a flag or a skin tone sequence is more. Never cut a segment in the middle of a surrogate pair.
• Watch out for text that arrives from a rich editor. Curly quotes, the ellipsis character and the non breaking space all look ordinary on screen and all force the 16 bit encoding.
• A trailing newline is a character. So is the space someone left after the last word. On a message sitting exactly on 160 units, that is a whole extra segment.

The lesson is broader than one alphabet. Any system that bills by a unit you do not display is a system where a harmless edit becomes an unexplained cost, and the person who made the edit has no way to connect the two. Put the billed unit on the screen next to the thing being edited, compute it with the same code that bills it, and the whole class of surprise disappears. Once the numbers are visible, the next question is usually about throughput rather than cost, and that is a different set of limits entirely.