ServerlessIRCd Docs

Docs home

ADR-003: Cloudflare Durable Object sharding strategy

Date: 2025-02-01 (Phase 3) Status: Accepted Supersedes: — Superseded by: —

Context

Cloudflare Durable Objects (DOs) are single-threaded per-instance: all requests to a given DO instance are serialized. This is ideal for enforcing uniqueness invariants (two racing requests land on the same DO, one after the other), but a single DO instance is a throughput ceiling. ServerlessIRCd needs three DO classes:

  1. ConnectionDO — one per WebSocket connection. Keyed by connectionId. No sharding needed: each connection is its own instance.
  2. ChannelDO — one per channel. Keyed by lowercased channel name. No sharding needed: one DO per channel is the natural unit; a hot channel (1k+ members) is bounded by DO fanout latency, not by DO count.
  3. RegistryDO — holds the global nick → connectionId map. If this were a single DO, every nick registration on the entire server would serialize through one instance. That is an unacceptable bottleneck for any non-trivial deployment.

The open question (PLAN §9): "Channel DO sharding granularity (one DO per channel vs. bucketed)?" and "Registry DO: how to shard?"

Decision

ConnectionDO: one instance per connection (idFromName(connectionId)). No sharding.

ChannelDO: one instance per channel (idFromName(lowercasedChannelName)). No bucketing. If a hot channel exceeds DO throughput limits, the mitigation is fanout batching (one stub() call carrying multiple lines) rather than splitting the channel across DOs. A single channel's roster and modes live in one DO so that all mutations are serialized correctly.

RegistryDO: sharded by hash(lowercased(nick)) % N, where N defaults to 32 (DEFAULT_REGISTRY_SHARDS). The hash function is FNV-1a 32-bit, chosen for its uniform distribution, deterministic output (no randomness), and trivial implementation.

shard = fnv1a32(nick.toLowerCase()) % N
key   = `s${shard}`   // e.g. "s7", "s31"
stub  = env.REGISTRY_DO.get(env.REGISTRY_DO.idFromName(key))

Two clients racing on the same nick always land on the same shard (because lowercasing collapses case variants), so the single-shard uniqueness check covers all case variants. Different nicks spread across shards, so registration throughput scales with the fleet size.

Cross-shard nick changes (old nick and new nick hash to different shards) use a two-phase reserve-new-then-release-old. The new-nick shard wins the race; the old-nick shard is released only after the new reservation succeeds.

N is configurable via the REGISTRY_SHARDS environment variable. Changing N in production requires a migration (re-sharding every existing nick entry to its new shard).

Consequences

Positive:

Negative:

Outbound batching scope (TICKET-087)

The fanout batching decided above is implemented per call: ConnectionDO.deliver() joins every line of one ChannelDO broadcast into a single WebSocket.send(), and the actor's send handler does the same for all lines of one frame's response. A speculative cross-call coalescing optimization (queueing lines from independent deliver/send invocations within one microtask and flushing them as one write) was reserved as a commented-out field in ConnectionDO and subsequently dropped: it would add a drain queue and a queueMicrotask scheduler, introduce ordering hazards between independent fan-out calls, and yield no measured benefit — nothing in the TICKET-049 load-test report or the ChannelDO fan-out path shows WS round-trips as a bottleneck. The AWS adapter batches at a different layer (per-invocation outbound: string[] accumulated then posted once), so no shared abstraction is worth extracting. Revisit only if WS round-trips become a demonstrated bottleneck.

References