中文版

On October 13, 2026, an external security team reported two chainable configuration-boundary issues to IXCDN. We began containment and remediation that day and confirmed that the pending records had not entered active configuration.

After the direct fixes, the fact that three product surfaces maintained separate domain-ownership logic still indicated duplicated control at a lower layer. The review therefore expanded to the full request and configuration path and ultimately triggered a configuration-delivery redesign.

End-to-end request-path audit

The team selected an ordinary request and traced ingress parsing, authorization, cache, origin selection, delivery, logging and metering. This made it easier to see the same mistake repeated in separate implementations.

The review recorded 71 findings, mostly in six groups:

  1. custom delivery paths skipped shared phases, leaving controls such as rate limits unevenly applied;
  2. raw Host casing, path encoding and repeated headers influenced decisions too early;
  3. credentials served more than one purpose;
  4. some error paths failed open;
  5. queues and task tables lacked explicit bounds;
  6. hot paths contained deep copies and synchronous blocking.

The fixes were placed at shared boundaries where possible. Requests are normalized once at ingress. Origin addresses are checked after DNS resolution and protected against self-loops. Relays use dedicated credentials. Security decisions fail closed when the system cannot establish a safe answer.

At publication time, 69 findings are closed and 2 remain under active tracking: confirming origin-address enforcement across existing nodes, and reducing metering cost further on the hot path. No finding is unassigned.

Verifiable configuration-release state machine

Configuration delivery had the largest potential blast radius. Domains, certificates, access rules and security policy all use it; the routed.hk AI gateway uses the same path. Rather than add more branches to the old workflow, we redefined the release unit and node states.

  • Per-node streams: each node receives only its configuration, limiting the scope of a failed release.
  • Signed artifacts: Ed25519 signatures are verified with a pinned local public key. Control of the transport alone cannot create a trusted artifact.
  • Prepare before serving: a node verifies and prepares a release before it reaches prepared, then switches to serving. Failure at any step leaves the previous valid version in use.
  • Two-stage rollback: a rollback is created and staged first, then published through a separate confirmation. The stream remains paused afterward until the underlying issue is resolved.

DNS, edge and site delivery now use this workflow. The management edge intentionally remains on the old path because it carries the configuration control channel itself. We are not migrating the control entrance and its managed targets at the same time.

Failures identified during staged rollout

Rollout proceeded one node at a time, with every cutover checked against a site baseline and rolled back on error. That choice limited three real failures:

  • The first canary activated before its listeners were ready, so an authorization probe against local ports failed. Activation now waits for listener readiness.
  • One node had a stale configuration file on disk. The new binary rejected it and entered a restart loop. Binary replacement now validates the on-disk configuration instead of treating a running old process as proof that the file is valid.
  • A few paths still referenced old addresses after cutover. Releases now include a fixed end-to-end acceptance check rather than relying only on process and health endpoints.

When a node could not verify a new configuration, it continued serving the previous version. That behavior did not prevent mistakes, but it kept them in the release workflow instead of customer traffic.

Incomplete work

  • Per-entity incremental publishing is implemented, while production nodes still fetch full artifacts.
  • Write-side sharding and migration gates wait on cleanup of stale process records.
  • Validation on weak cross-border networks is continuing.

We thank the external team for the responsible disclosure. The report led to two direct fixes and identified assumptions shared across several paths. The incomplete items listed above remain under tracking.

Further reading: Infrastructure Constraints of AI Inference Traffic · Timeouts, Connection Migration and Hot-Path Optimization for AI Streams