Proxy tunnels that die after ten idle seconds

An integration that was supposed to hold a session open for hours was opening a new one every few seconds instead. Nothing failed loudly: requests succeeded, the logs looked ordinary, and the only symptom was a bandwidth bill several times larger than the data we were actually moving. The upstream was closing the tunnel after about ten seconds of silence, and our client was reconnecting, politely, thousands of times a day.

What actually happens

A proxy tunnel is set up with a request to connect to a host and port, after which raw bytes flow through it. The proxy keeps that tunnel only while it is being used. If nothing crosses for the configured idle period, it is closed and the resources are reclaimed. Ten seconds is short but not unusual, and it is rarely documented.

What makes this expensive is how clients handle it. A closed socket is not an error condition to most HTTP clients; it is a reason to open a new one. The library reconnects, redoes the handshake and carries on, and the application layer above never learns that anything happened.

Each of those reconnects costs more than it looks like:

• The tunnel request and its response.
• A full TLS handshake to the destination, including the certificate chain, which is a few kilobytes on a typical server and more on one with a long chain.
• Any authentication or session setup the destination does on a new connection, such as a login round trip or a cookie exchange.

None of that is payload. Multiply it by one reconnect per idle window, by however many workers you run, by a day, and the overhead stops being a rounding error. On metered bandwidth you are paying, by the gigabyte, for handshakes.

There is a second effect that costs more than money. If the upstream assigns an exit address per session, every reconnect is a new identity. Anything that assumed continuity starts behaving strangely: a logged in session drops, a paginated cursor resets, a rate limit bucket that was being respected gets hit from a dozen directions at once. The destination sees a swarm where you intended one client, and it responds the way anyone would respond to a swarm, which brings you back to treating a rate limit as temporary rather than terminal.

How to see it

Two numbers tell the whole story: sessions per hour, and bytes per session.

If your client does not log connection lifecycle events, the kernel will tell you. Sample the established connection count once a second and watch whether it churns or holds steady:

for i in $(seq 1 60); do
  printf '%s %s\n' "$(date +%T)" "$(ss -tn state established "dport = :8080" | wc -l)"
  sleep 1
done

A stable count is a pool that is being reused. A count that rises and falls every few seconds is churn, and the period of that oscillation is usually the idle timeout you are looking for.

For the bytes number, log open time, close time and bytes transferred per session, then summarise:

awk -F'\t' '{n++; d+=$2; b+=$3} END {
  printf "sessions=%d avg_seconds=%.1f avg_bytes=%.0f\n", n, d/n, b/n
}' sessions.tsv
# sessions=41780 avg_seconds=9.8 avg_bytes=4120

An average session life that lands just under a round number is the idle timeout confessing. An average payload of a few kilobytes against a handshake that also costs a few kilobytes means roughly half of everything you are paying for is setup. Keep those session logs long enough to compare one week against another, which means rotating them properly rather than letting them become the thing that fills the disk.

You can also measure the timeout directly in under a minute:

import socket, time

s = socket.create_connection(("proxy.host", 8080), timeout=5)
s.sendall(b"CONNECT example.com:443 HTTP/1.1\r\nHost: example.com:443\r\n\r\n")
print(s.recv(200).split(b"\r\n")[0])

s.settimeout(1)
for i in range(1, 90):
    time.sleep(1)
    try:
        if s.recv(1) == b"":
            print("closed after", i, "idle seconds")
            break
    except TimeoutError:
        continue
else:
    print("still open after 90 idle seconds")

Run it twice. Once as written, and once with a single byte sent every three seconds, so you can see the difference a keep alive makes before you build anything around it.

The fix

Four changes, in order of how much they save.

1. Keep the tunnel warm. Send something inside the idle window, using the protocol's own ping if it has one. Pick an interval comfortably under the timeout, three seconds for a ten second window rather than nine, because a scheduling delay at nine costs you the tunnel. Keep alive bytes are trivial next to a handshake. Socket level TCP keepalive usually does not help here, because the proxy counts what crosses the tunnel and the default probe interval is measured in hours.

2. Make the session sticky. If the provider supports a session identifier, pass the same one across reconnects so the tunnel lands on the same exit and the destination sees continuity. This does not reduce the number of handshakes, it reduces the damage each one causes.

3. Pool and reuse. One warm tunnel per worker beats one tunnel per request by a wide margin. Size the pool to the number of concurrent workers rather than to the request rate, and make sure the pool's own idle eviction is shorter than the upstream's timeout, otherwise the pool hands out sockets that are already dead.

4. Back off on reconnect. A reconnect loop with no delay turns a brief upstream problem into a flood you caused, and on metered bandwidth into an invoice.

POOL_SIZE = 24
PING_SECONDS = 3           # upstream closes at about 10
SESSION_TTL_SECONDS = 600  # rotate on our schedule, not by accident

def session_id(worker):
    window = int(time.time() // SESSION_TTL_SECONDS)
    return f"w{worker}s{window}"

def acquire(worker):
    sid = session_id(worker)
    return pool.get(proxy_user=f"{USER}-session-{sid}", ping=PING_SECONDS)

The port that accepts a protocol it does not carry

The trap that cost the most time was not the timeout. The same port accepted a SOCKS5 greeting, returned a success byte for the method negotiation, returned a success byte for the connect request, and then carried nothing at all. Every layer in the client reported a healthy connection. No payload ever arrived, in either direction, and the failure looked like the destination being slow.

Only HTTP CONNECT was actually implemented. The lesson is that a successful handshake is not a working transport, so prove the path with a request that returns known content:

curl -s -x http://user:pass@proxy.host:8080 https://example.com/status \
  -o /dev/null -w 'code=%{http_code} bytes=%{size_download} time=%{time_total}\n'
# code=200 bytes=612 time=0.412

If that returns content and the SOCKS variant hangs on the same port, the port is HTTP CONNECT only, whatever the documentation says.

How to check it worked

Compare the same two numbers before and after, over the same length of time and the same amount of real work. What you want is the session count falling by an order of magnitude while payload bytes stay flat:

before: sessions/hour ~1,700   avg_bytes 4,100    daily transfer ~6 GB
after:  sessions/hour ~45      avg_bytes 160,000  daily transfer ~1 GB

Then run the idle probe again and watch it stay open past ninety seconds. If the tunnel survives idle but the session count has not dropped, the pool is not being reused and that is a separate bug in the client, not in the proxy.

What to watch out for

• A ping interval close to the timeout is a race you will lose eventually. Leave a margin of at least two thirds.
• Warm tunnels cost something while idle. Two dozen tunnels pinging every three seconds is constant traffic, so let the pool shrink when there is no work, or you end up paying to keep nothing alive.
• Sticky sessions have a maximum lifetime on the provider side. Rotate on your own schedule, before theirs, so the change is something you handle rather than something that happens to you.
• Retry logic that treats a closed tunnel as a failed request will duplicate work. Keep the retry at the transport level, or make the operation idempotent before you allow it at the request level.
• Concurrency and cost do not scale together in a straight line. More workers means more warm tunnels and more pings, so measure cost per session before you add capacity, the same way throughput has to be measured where it actually goes rather than assumed.

Any transport with an idle timeout will turn a long lived design into a reconnect loop, and the loop is invisible until someone reads an invoice or a graph of connection counts. The unit to watch is not requests or errors but sessions and bytes per session, because that is where the cost of a silent failure shows up first. A transport that fails quietly deserves the same instrumentation as one that fails loudly, and it usually needs it more.