AI inference still uses HTTP and TLS, but its connection duration, call pattern, configuration frequency and payload sensitivity differ from ordinary web traffic. Operating data from IXCDN and routed.hk shows that timeout, retry and resource limits designed for short requests cannot be applied unchanged to inference traffic.
Long-lived streaming responses and idle timeouts
An inference stream can remain open for minutes and may be quiet while the model computes. If an edge proxy applies a uniform short read timeout, it can classify a legitimate inference interval as an upstream failure and terminate the connection.
Connection timeout, time to first byte, stream idle timeout and total request duration protect different phases. They should be configured separately rather than collapsed into one request_timeout.
Automated callers, retry storms and idempotency
Agents and service-to-service clients commonly retry automatically. If the client, gateway and upstream all retry, one failure can multiply traffic and consume inference quota repeatedly. The ingress needs explicit rate limits, concurrency limits, retry budgets and idempotency keys. A gateway should not replay a non-idempotent request without application semantics.
Frequent configuration changes and release consistency
Model routes, tenant quotas, credentials and security policy change continuously. Configuration is production data rather than an occasional static-file edit. The delivery path needs version identifiers, artifact signatures, per-node application state, audit records and verifiable rollback so nodes do not run incompatible fragments.
Sensitive payloads and data minimization
Edge locations may process prompts, API credentials, tool arguments and enterprise context. Cache keys, access logs, error responses and debug sampling need explicit data-minimization rules: authenticated or personalized responses are not cached by default, full credentials are not logged, and request-body samples have defined purpose, retention and access controls.
Security review scope and results
On October 13, 2026, an external team reported two chainable configuration-boundary issues. After correcting them, we reviewed the full path from ingress to metering. The review recorded 71 findings; at publication time, 69 are closed and 2 remain under active tracking.
Configuration delivery was rebuilt during the same period. Each node now has an independent stream, artifacts are signed, and nodes verify and prepare a release before serving it. DNS, edge and site delivery have moved to that workflow. The design and rollout failures are documented in Configuration-Delivery Redesign: From Red-Team Report to Staged Rollout.
The eighteen-day period produced more than six hundred commits, failed canaries and rollbacks. Per-node cutover limited each failure to one node while the team established the cause before expanding the release.
Engineering control principles
- Authorization correctness takes priority over latency optimization. Normalization, ownership checks and purpose-specific credentials are required ingress processing and cannot be removed to reduce latency.
- Failure semantics must be explicit. Keep the previous configuration when verification fails, stop a failed rollout and use only validated alternate paths.
- Performance metrics must be separated by processing stage. Average latency does not distinguish first byte, stream idle time, origin fetch, retries and node changes.
These controls also apply to conventional APIs. Long connections, automatic retries and expensive upstream work expose unsafe defaults sooner.
Incomplete work and validation limits
Per-entity incremental delivery is not enabled in production. A thread-per-core execution model remains in design, and end-to-end testing under representative workloads is continuing. Current results therefore should not be generalized to every model, region and stream duration.
routed.hk uses the same edge capabilities as customer traffic. Related changes are first validated against this operated service, then production defaults are adjusted from observed data.
For the engineering details, read Timeouts, Connection Migration and Hot-Path Optimization for AI Streams.