A streamed response that never closes hangs for a hundred seconds

A feature streams its answer token by token. The text appears, the last sentence lands, and then nothing happens. The stop button stays on screen, the request stays pending in the network panel, and about a hundred seconds later the interface flips to a connection error even though the answer was complete and correct. The same code on a laptop looks perfect. The problem is not the model and not the browser: nobody ever closed the response.

What actually happens

A streamed response is an ordinary HTTP response with a content type of text/event-stream, sent in chunks. The client has no idea how much is coming, because there is no content length. The only signal that the answer is over is that the server closes the stream. A data: [DONE] line is a convention inside the body and means nothing to HTTP at all: it is text, and only code that reads the text knows what it implies.

Now look at the handler almost everyone writes first:

const upstream = await fetch(PROVIDER_URL, { method: 'POST', body });

return new Response(
  new ReadableStream({
    async start(controller) {
      for await (const chunk of upstream.body) controller.enqueue(chunk);
      controller.close();
    },
  }),
  { headers: { 'content-type': 'text/event-stream' } },
);

This reads correctly. The loop ends when the upstream body reaches end of file, and then the stream closes. The catch is that the upstream body does not reach end of file when the answer is over. A provider that supports connection reuse writes its final event and then keeps the socket open, waiting to see whether you want to send another request down the same connection. Its own idle timeout might be sixty seconds, might be five minutes. Until then, our loop is parked on a read that will not return, and the client is parked with it.

On a laptop this is invisible. A local test double writes a fixture and closes immediately, which is the one behaviour the real provider does not have. In manual testing the text is on screen, so the tab gets closed and the socket dies with it. Nothing reports a bug.

Through a reverse proxy the same sequence has a different ending, in this order:

1. The handler writes the last event. The browser paints it. To a reader, the feature is finished.
2. The handler stays in its loop, holding the upstream socket, a worker and a connection slot.
3. The proxy waits on the origin. Its read timeout counts from the last byte it received, and nginx defaults to sixty seconds. On one platform I work on it was set to a hundred.
4. At the timeout the proxy gives up on the origin and closes the client connection abruptly. The fetch promise rejects, the reader throws, and the interface runs its error path over a finished answer.

There is a second proxy behaviour worth separating from the first, because the two get confused. With buffering on, which is the default, nginx collects the response into its own buffers and flushes when one fills. The answer then arrives in lumps, or all at once at the end, and the streaming effect disappears even when the stream closes correctly. One symptom is about buffering, the other is about closing, and they need different lines of config.

How to see it

Time to first byte tells you the model started. Time to last byte tells you whether the response ended. Ask for both:

curl -N -sS -o /dev/null \
  -H 'accept: text/event-stream' \
  -H 'content-type: application/json' \
  -d '{"q":"hello"}' \
  -w 'first byte %{time_starttransfer}s, last byte %{time_total}s\n' \
  https://app.example.com/api/chat
# first byte 0.41s, last byte 100.08s

A first byte under half a second with a last byte at a hundred flat is the whole diagnosis. The number does not drift between runs, because it is a timeout and not a slow answer. Run the same command against the origin port, bypassing the proxy, and you get a different constant, usually the provider's idle timeout, which tells you the origin is holding the connection open by itself.

To see where the time goes inside the stream, stamp each line as it arrives:

curl -N -sS -H 'accept: text/event-stream' -d '{"q":"hello"}' \
  https://app.example.com/api/chat \
  | while IFS= read -r line; do printf '%s %s\n' "$(date +%T.%2N)" "$line"; done
# 14:22:07.11 data: {"delta":"Hello"}
# 14:22:09.64 data: [DONE]
# (nothing for ninety seconds, then the shell returns)

If every line lands at the same timestamp instead, buffering is on and you have the other problem. The proxy log names it directly:

tail -2 /var/log/nginx/error.log
# upstream timed out (110: Connection timed out) while reading upstream, request: "POST /api/chat"

The fix

Three changes: stop reading when the answer ends, say so in the response headers, and give the proxy a timeout that matches how streams behave.

In the handler, parse events instead of copying bytes, break on the done marker, and put the cleanup in a finally so every path goes through it:

export async function POST(req) {
  const upstream = await fetch(PROVIDER_URL, { method: 'POST', body: req.body, signal: req.signal });
  const reader = upstream.body.getReader();
  const dec = new TextDecoder();
  const enc = new TextEncoder();
  let buf = '';

  const stream = new ReadableStream({
    async start(controller) {
      const beat = setInterval(() => controller.enqueue(enc.encode(': keep-alive\n\n')), 15000);
      try {
        for (;;) {
          const { value, done } = await reader.read();
          if (done) break;
          buf += dec.decode(value, { stream: true });
          let i;
          while ((i = buf.indexOf('\n\n')) !== -1) {
            const event = buf.slice(0, i + 2);
            buf = buf.slice(i + 2);
            if (event.includes('[DONE]')) {
              controller.enqueue(enc.encode('event: done\ndata: {"ok":true}\n\n'));
              return;
            }
            controller.enqueue(enc.encode(event));
          }
        }
      } finally {
        clearInterval(beat);
        reader.cancel().catch(() => {});
        try { controller.close(); } catch {}
      }
    },
    cancel() { reader.cancel().catch(() => {}); },
  });

  return new Response(stream, {
    headers: {
      'content-type': 'text/event-stream; charset=utf-8',
      'cache-control': 'no-cache, no-transform',
      'connection': 'keep-alive',
      'x-accel-buffering': 'no',
    },
  });
}

Two details carry most of the value. reader.cancel() releases the upstream connection instead of leaving it half read, which matters when the provider counts open streams against your account. The explicit event: done gives the client a way to tell a finished answer from a cut one, because a closed connection alone cannot say which happened.

On the proxy, turn buffering off for that route and set the gap timeout deliberately:

location /api/chat {
    proxy_pass              http://127.0.0.1:3000;
    proxy_http_version      1.1;
    proxy_set_header        Connection "";
    proxy_buffering         off;
    proxy_cache             off;
    proxy_read_timeout      120s;
    proxy_send_timeout      120s;
    chunked_transfer_encoding on;
}

proxy_read_timeout is the silence the proxy tolerates between two reads, not the length of the whole response, so two minutes against a fifteen second heartbeat is generous and still cuts a hung origin loose. Buffering off costs you small writes forwarded one at a time and no compression on that route, which is the right trade for a stream and the wrong one for a page. If you set the buffering header in nginx rather than in the app, check it on the response: a header declared at server level vanishes as soon as a location block declares one of its own, which is the inheritance rule that quietly drops headers.

How to check it worked

The same command, and the number that changes:

curl -N -sS -o /dev/null -H 'accept: text/event-stream' \
  -d '{"q":"hello"}' -w 'last byte %{time_total}s\n' \
  https://app.example.com/api/chat
# last byte 2.71s

Then keep it. A test that asserts content passes on a broken stream, so assert the gap between the last chunk and the close:

const res = await fetch(`${BASE}/api/chat`, { method: 'POST', body, headers });
const reader = res.body.getReader();
let last = Date.now();
for (;;) {
  const { done } = await reader.read();
  if (done) break;
  last = Date.now();
}
const gap = Date.now() - last;
if (gap > 2000) throw new Error(`stream stayed open ${gap}ms after the last chunk`);

Point BASE at the deployed hostname. A run against localhost skips the proxy, and the proxy is half of what you are testing.

What to watch out for

• A heartbeat you forget to clear is the same bug with a friendlier name. The interval keeps enqueueing, the stream stays open, and the symptom is identical. Clear it in the same finally that closes the controller.
• If you do not pass the request abort signal through to the provider call, a reader who closes the tab leaves you reading and paying for tokens nobody will see, which is one of the quiet line items in keeping an AI feature affordable.
• Count every hop with its own idle timeout, not just the one you configured. A load balancer, a tunnel or a CDN in front of nginx has its own opinion, and tunnels that die after ten idle seconds fail in exactly this shape.
• Raising the read timeout to an hour makes the error message go away and nothing else. The request is still open, still holding a connection, and now for an hour.

A streaming endpoint has two endings: the one written into the body and the one on the wire. Readers see the first, and every machine between you and them only sees the second. It is worth checking that a request you believe is finished is actually finished, and the cheapest way is a number rather than a screenshot: time to last byte, measured through the real hostname, compared against the time the last word appeared. When those two numbers match, the feature is done. When they differ by a round number, you have found somebody's timeout.