An SSE or gRPC inference response can stay open for several minutes, and tens of seconds without a new byte may be normal. Network changes, cache refreshes and origin compression—all routine events—behave differently when a connection is long-lived.
IXCDN’s AI gateway, routed.hk, uses the same edge. Over more than two weeks, we investigated concrete failures in timeout handling, HTTP/3, cache refresh, compression and metering. This post covers only cases we reproduced and verified.
Stream idle timeouts by response type
The old logic gave every response the same read timeout, 10 seconds by default. That can be reasonable for an ordinary file response, but it interrupts an inference stream during a normal compute pause.
Now the edge recognizes long-lived streams from the headers the origin actually returns. Once response headers arrive, SSE and gRPC responses get a body idle window of at least 300 seconds, and every read resets it. A longer configured timeout is preserved.
Slow-client protection draws the same line. Judging a client by cumulative rate applies only to finite responses with a declared length. A stream’s timing belongs to the model, so it says nothing about the client’s ability to receive. Rate limits remain independent and still apply.
Port-validation defect in QUIC connection migration
HTTP/3 connection migration allows a mobile device to keep a connection when it moves from Wi-Fi to cellular. One migration test initially looked flaky: successful runs took about 0.3 seconds, while failures always consumed the complete timeout. Repetition showed that a specific port-range transition triggered the behavior consistently.
Packet captures gave the answer. After a NAT rebinding, the server stopped processing packets from the new address entirely. The cause was the QUIC library’s default validator: if the old and new ports crossed between the User (1024–49151) and Dynamic (49152 and above) ranges, it refused the migration and dropped the packets without reporting an error.
Upstream had already tracked this in aws/s2n-quic#3040 and fixed it in #3166, which only rejects moves between System and non-System ports and keeps the blocked-port and IP-scope checks. We upgraded s2n-quic and compared the two versions in production with a real HTTP/3 client, sending 14 requests on one connection and changing the source port after one second.
| Port change | Before | After |
|---|---|---|
| Same range | 14/14 | 14/14 |
| User → Dynamic | 7/14, then all stalled | 14/14 |
| Dynamic → User | 8/14, then all stalled | 14/14 |
After the upgrade, one connection carried everything, and request times barely moved across the change.
Background refresh over HTTP/2 and HTTP/3
When a cached object expires, an edge can serve the old copy while refreshing in the background. This is the standard way to balance speed and freshness.
Our review found that this refresh failed whenever an HTTP/2 or HTTP/3 request triggered it. The subrequest converts the original request into HTTP/1 headers and parses them, but the request line said HTTP/2, which the parser rejects. HTTP/1.1 was unaffected, so existing tests never caught it.
The fix hid a trap. HTTP/2 carries the hostname in :authority, with no Host header. Changing the version alone gets past the parser, but the subrequest cannot identify the domain and the refresh still fails. The conversion now writes both HTTP/1.1 and Host.
We first wrote an end-to-end test that reproduced the failure, confirmed it failed before the fix and passed after. In production, refreshes triggered over both HTTP/2 and HTTP/3 now finish within seconds.
Compression responsibility at origin and edge
If an origin has already compressed a response, the edge should not unpack it and do the work again.
Hosts with compression enabled now declare their supported codings to the origin. When the origin returns compressed content, the edge caches and delivers it as is. It decompresses only for clients that do not accept the coding, and only content-rewriting features ask the origin for plain bytes. In production, MISS and HIT responses both matched the origin byte for byte.
One industry convention is worth knowing. CDNs send the Via header on origin requests, as RFC 9110 requires, and nginx’s gzip_proxied defaults to off, so nginx will not compress those requests. If your origin runs nginx, we recommend:
gzip_proxied any;On one nginx origin serving more than a dozen sites, enabling this reduced origin bytes for 10 home pages from 220,035 to 62,214, about 72%. Origins without it are unaffected, and the edge keeps compressing before delivery.
Lock-contention reduction in the metering hot path
We split per-domain metering across worker threads, let over-limit rejections skip the mutex, moved log encoding to the background, and removed frame-by-frame copying of upstream bodies. These are isolated benchmarks, not production end-to-end throughput:
| Scenario | Before | After |
|---|---|---|
| Single hot domain metering, 8 threads | 12.44 Mops/s | 483.44 Mops/s |
| Over-limit rejection, same bucket, 8 threads | 15.25 Mops/s | 123.64 Mops/s |
| Admission, same bucket, 8 threads | 14.74 Mops/s | 14.46 Mops/s |
| Target copies for 64 MiB of input | 67,108,864 bytes | 0 |
| Time for 64 KiB reads | 22.57 ms | 7.80 ms |
These are isolated benchmarks, not production numbers, and not everything improved. Same-bucket admission did not get faster, and small reads that retain frames for a long time now allocate more.
Benchmark and validation limits
For AI applications, the numbers that matter most are time to first token and stream completion rate. We cannot give you those yet.
A full load test under representative traffic is still ahead of us, and our production windows are too small to prove a latency gain. The per-core thread model is still a design. What we fixed are specific reasons a long stream would break, slow down or cost more.
The evidence supports a narrower conclusion: several specific interruption and cost paths have been removed. It does not yet support a claim that every AI request has a lower time to first token. Future results will continue to separate isolated benchmarks, production observations and representative end-to-end load tests.
Further reading: Infrastructure Constraints of AI Inference Traffic · Configuration-Delivery Redesign: From Red-Team Report to Staged Rollout