AWS Deployment Guide

End-to-end guide for deploying ServerlessIRCd to AWS (Phase 4 of the project plan). Covers prerequisites, first-time AWS setup, local development, staging and production deploys, configuration reference, DynamoDB capacity planning, region strategy, cost notes, CI/CD, and troubleshooting.

Cross-reference: the AWS Adapter Architecture doc (../../ServerlessIRCd/docs/AWS-Adapter-Architecture.md) for the diagrams referenced throughout, and the Cloudflare Deployment Guide (../../ServerlessIRCd/docs/Cloudflare-Deployment-Guide.md) for the equivalent flow on the other supported platform. The shared irc-core/irc-server packages and the contract test suite are identical between platforms; only the I/O shell (this stack) differs.

Key in-repo paths cited below:

Acceptance criterion: a new contributor can deploy their own staging instance following only this doc.

1. What gets deployed

A single CDK v2 stack (IrcAwsStack) provisions the entire AWS substrate. Every resource is regional. The IRC runtime itself is stateful compute (Lambda + DynamoDB + API Gateway WebSocket); the optional web client (S3 + CloudFront + OAC) is provisioned only when the webSite/webSiteCustomDomain CDK context is supplied. See §16 for the web-serving path.

   ┌──────────────────────────────────────────────────────────────────┐
   │  CDK stack: IrcAwsStack  (apps/aws-stack/src/aws-stack.ts)       │
   │                                                                  │
   │  ┌─ WebSocketApi (AWS::ApiGatewayV2::Api)  ───────────────────┐  │
   │  │   routes: $connect  $disconnect  $default                  │  │
   │  │   → all three wired to IrcHandler                          │  │
   │  │   stage: prod (auto-deploy)                                │  │
   │  └────────────────────────────────────────────────────────────┘  │
   │                           │                                      │
   │                           ▼  WebSocket event per frame           │
   │  ┌─ Lambda: IrcHandler ───────────────────────────────────────┐  │
   │  │   handler export `handler` (Node 20, esbuild bundle)       │  │
   │  │   entry: packages/aws-adapter/src/handlers/index.ts        │  │
   │  │   dispatches on event.requestContext.routeKey              │  │
   │  └────────────────────────────────────────────────────────────┘  │
   │                                                                  │
   │  ┌─ Lambda: IrcSweeper ───────────────────────────────────────┐  │
   │  │   handler export `sweeperHandler`                          │  │
   │  │   EventBridge schedule: rate(5 minutes)                    │  │
   │  │   Scans Connections for rows past hard TTL; cleans up.     │  │
   │  └────────────────────────────────────────────────────────────┘  │
   │                                                                  │
   │  ┌─ Lambda: IrcPingChecker ───────────────────────────────────┐  │
   │  │   handler export `pingCheckerHandler`                      │  │
   │  │   EventBridge schedule: rate(1 minute)                     │  │
   │  │   Sends PING to idle connections; disconnects no-PONG.     │  │
   │  └────────────────────────────────────────────────────────────┘  │
   │                                                                  │
   │  ┌─ DynamoDB (on-demand) ─────────────────────────────────────┐  │
   │  │   Connections    (PK connectionId,     TTL idleSince)      │  │
   │  │   ChannelMeta    (PK channelName)                          │  │
   │  │   ChannelMembers (PK channelName, SK connectionId)         │  │
   │  │   Nicks          (PK nickLower)                            │  │
   │  │   Accounts       (PK account)                              │  │
   │  └────────────────────────────────────────────────────────────┘  │
   │                                                                  │
   │  CfnOutput: ConnectUrl     = stage.url         (wss://…)         │
   │  CfnOutput: ManagementUrl  = stage.callbackUrl (https://…)       │
   └──────────────────────────────────────────────────────────────────┘

All IRC protocol logic lives in the shared irc-core reducers and the irc-server ConnectionActor. The Lambdas are a thin I/O shell: they forward WebSocket events into the actor and interpret the resulting Effect[] against DynamoDB and the API Gateway Management API. No IRC state is held in the Lambda process between invocations.

The three Lambda functions share a single bundle (esbuild tree-shakes per handler export at synth time). They run on nodejs20.x and exclude @aws-sdk/* from the bundle because the AWS SDK v3 ships with the Lambda Node 20 runtime.

Stack outputs (CfnOutput):

The same CloudFormation template serves both staging and production. The staging/production distinction today is which AWS account and region the stack is deployed into — there is no per-environment stack parameter (see §6 and §7.4 for the implications).

2. Prerequisites

Confirm the local environment:

node --version          # v20.x or newer
pnpm --version          # 9.x
aws --version           # aws-cli/2.x (any 2.x is fine)
pnpm install            # from repo root, installs the whole workspace
pnpm build              # builds irc-core, irc-server, aws-adapter, aws-stack

pnpm build must succeed before cdk deploy: the NodejsFunction construct transpiles the handler entry with esbuild at synth time, and esbuild resolves @serverless-ircd/* workspace packages via their compiled dist/ outputs.

3. First-time AWS setup

These steps are done once per AWS account.

3.1 Pick a region

API Gateway WebSocket endpoints, DynamoDB, EventBridge Scheduler, and Lambda are all regional. WebSocket clients pay round-trip time to the region, so pick the one closest to your user base (see §10 for the full region discussion). Set it as your default:

export AWS_REGION=us-east-1   # or your nearest region

CDK picks the region up via CDK_DEFAULT_REGION (set automatically by the CDK CLI from the SDK chain).

3.2 Configure credentials

Three equivalent options:

# Static access key (simplest):
aws configure
# → prompts for Access Key ID, Secret Key, region, output format.

# Named profile:
export AWS_PROFILE=my-profile

# SSO:
aws sso login --profile my-sso-profile
export AWS_PROFILE=my-sso-profile

Verify:

aws sts get-caller-identity

3.3 Bootstrap CDK

CDK needs an S3 bucket + a few IAM roles per account/region to hold synthesized CloudFormation assets. Bootstrap once:

cd apps/aws-stack
pnpm cdk bootstrap
# → creates a CFN stack named CDKToolkit with the bootstrap bucket.

If you skip this, the first cdk deploy fails with a bucket-not-found error (see §13.1). Bootstrap is idempotent — re-running on an already-bootstrapped account is a no-op.

3.4 Set up OIDC for CI (required)

Mandatory. Long-lived access keys (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY) are NOT accepted by the deploy workflow. The workflow assumes a role via GitHub OIDC web identity and asserts the assumed role matches AWS_DEPLOY_ROLE_ARN before any cdk deploy. The full trust-policy snippet, condition keys, and verification steps are in §17.

To deploy via the GitHub Actions workflow (§12) you MUST configure GitHub OIDC — there is no static-key fallback. Summary of the one-time setup (full snippet and verification in §17):

1. Create an IAM identity provider for token.actions.githubusercontent.com (one per AWS account).
2. Create a role with a trust policy granting sts:AssumeRoleWithWebIdentity to your repo's ref:refs/heads/main claim (the workflow is workflow_dispatch-only on main).
3. Attach a permission policy broad enough to manage the stack (IAM, API Gateway v2, Lambda, DynamoDB, EventBridge, CloudFormation) — ideally the same surface cdk deploy needs.
4. Note the role ARN — it becomes the AWS_DEPLOY_ROLE_ARN secret in CI.

4. Local development

Unlike the Cloudflare adapter (which has wrangler dev against the real workerd runtime), the AWS adapter does not yet ship a hot-reload local Lambda/API Gateway emulator. The local dev loop is:

1. Synth-time assertions (default pnpm test) — fast, no Docker.
2. Unit tests for the handler, runtime, and access patterns under packages/aws-adapter/tests/.
3. Optional localstack validation — gated, requires Docker.

4.1 Synth-time tests

From the repo root:

pnpm --filter @serverless-ircd/aws-stack test
# or, equivalently:
pnpm test     # runs the whole workspace including aws-stack

These execute Template.fromStack(...) in-process — no AWS credentials, no Docker. Every assertion in apps/aws-stack/tests/stack.test.ts corresponds to a fact this guide states: five on-demand tables, three Node 20 Lambdas, the $connect/$disconnect/$default routes, the prod stage, the two rate(...) schedules, the two CfnOutputs, and least-privilege IAM (no Action: "*" or Resource: "*").

4.2 Synthesize the CloudFormation template

pnpm --filter @serverless-ircd/aws-stack cdk:synth
# → writes cdk.out/<stack>.template.json

Inspecting cdk.out/ is the fastest way to see exactly what will be deployed without touching AWS.

4.3 Optional: validate against localstack

A gated test (apps/aws-stack/tests/localstack.test.ts) runs cdklocal synth against a real localstack container and re-asserts the five-table shape. It is skipped unless LOCALSTACK=1 is set so CI never depends on Docker. To run it locally:

# 1. Start localstack (requires Docker):
docker run --rm -d -p 4566:4566 localstack/localstack

# 2. Install the localstack-aware CDK CLI globally:
npm install -g aws-cdk-local

# 3. Point the AWS SDK at localstack (cdklocal reads these):
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_DEFAULT_REGION=us-east-1

# 4. Run the gated test:
LOCALSTACK=1 pnpm --filter @serverless-ircd/aws-stack test

4.4 Running the smoke script locally

The smoke e2e (apps/aws-stack/scripts/smoke.mjs) defaults to ws://localhost:8787, inherited from the Cloudflare adapter's dev port. To smoke-test the AWS adapter locally you would need to stand up an API Gateway WebSocket emulator in front of the Lambda handler (sam-cli + localstack can do this); that wiring is not preconfigured. In practice, local iteration uses the unit + synth tests, and the smoke script is run against a deployed staging URL (§5.2) or against the Cloudflare adapter's local dev server when you're iterating on the shared protocol layer.

5. Deploy

The deploy is workflow_dispatch-only (a maintainer runs it by hand); you can also deploy manually from a clean checkout.

5.1 Deploy from your machine

# From the repo root:
pnpm deploy:aws
# Equivalent to:
#   pnpm --filter @serverless-ircd/aws-stack run deploy
#   → cdk deploy --all --require-approval never

The first deploy of a given account/region will:

1. Upload the esbuild-bundled handler asset to the CDK bootstrap S3 bucket.
2. Create the IrcAwsStack CloudFormation stack.
3. Provision five DynamoDB tables, three Lambda functions, one API Gateway WebSocket API, one prod stage, two EventBridge rules, least-privilege IAM roles, and the two stack outputs.
4. Print the deployed URL on success — look for the ConnectUrl output, e.g. wss://abc123.execute-api.us-east-1.amazonaws.com/prod.

A typical first deploy takes 2–4 minutes (DynamoDB table creation and API Gateway stage auto-deploy dominate).

5.2 Verify with the smoke script

node apps/aws-stack/scripts/smoke.mjs \
  --url wss://abc123.execute-api.us-east-1.amazonaws.com/prod

The script replays CONNECT → NICK/USER → JOIN #smoke → PRIVMSG → QUIT and asserts the server emits 001, 376, 353, 366, and closes the socket. Expected output on success:

{"ts":"...","level":"info","msg":"aws-stack smoke e2e passed","url":"wss://..."}

On failure the script writes a level: "error" JSON line and exits non-zero, which is what the CI workflow gates on.

5.3 Connect a real IRC client

Point any RFC-compliant WebSocket-aware IRC client at the deployed wss:// URL. For WeeChat:

/server add ircd abc123.execute-api.us-east-1.amazonaws.com/443
/set irc.server.ircd.ssl on
/connect ircd
/join #test

For a quick text-only check, the apps/local-cli and tools/tcp-ws-forwarder packages in the parent repo bridge a plain TCP IRC client to a WebSocket endpoint.

6. Production hardening

There is a single deploy target per platform (pnpm deploy:aws). The staging/prod split was collapsed — there is no -staging / -prod stack-name suffix, no environmentName CDK context, and no per-env table-name prefixing. The stack is always named IrcAwsStack (hardcoded in apps/aws-stack/bin/aws.ts), and table names are the bare logical ids (Connections, Nicks, …). The staging/production distinction is which AWS account and region the stack is deployed into, controlled by the credentials and AWS_REGION in the environment.

For a real production deployment, before your first pnpm deploy:aws:

1. Use a dedicated AWS account (or at least a dedicated region) for production. Do not point production credentials at the same stack CI is deploying to on every push.
2. Keep the default table protection on. State-bearing tables (Nicks, Services, ChannelMeta, ChannelMembers) default to RemovalPolicy.RETAIN + deletionProtection: true + PITR-on, so a mistaken cdk destroy or a stray aws dynamodb delete-table cannot wipe them (see §9.4). Only set -c allowTableDeletion=true for a deliberate teardown.
3. Set the server identity (SERVER_NAME, NETWORK_NAME, MOTD) — see §7.1. SERVER_NAME is required at boot; the stack falls back to irc.localhost when the prop is omitted, which is intentionally not production-ready.
4. Pick a region close to your users (see §10).
5. Do NOT re-enable API Gateway DataTraceEnabled. It is OFF by default (safe). Turning it on logs the full body of every IRC frame (PASS, AUTHENTICATE <SASL-PLAIN>, channel keys, all PRIVMSG/NOTICE) to CloudWatch, leaking credentials to anyone with logs:GetLogEvents. The two-step sandbox-only escape hatch is documented in §7.6; never use it against a production account.

There is no CI auto-deploy to production. The deploy workflow (deploy-aws.yml) only triggers on push to main and targets staging. Production deploys are always manual from a clean checkout.

7. Configuration reference

Server identity (SERVER_NAME, NETWORK_NAME, MOTD) and the logical environment name are CDK construct props on IrcAwsStack (apps/aws-stack/src/aws-stack.ts); pass them when instantiating the stack. All other deployment knobs still live in the stack source. Unlike the Cloudflare adapter (where config lives in a declarative wrangler.toml), AWS config is code — non-identity overrides require editing the stack and re-deploying.

7.1 Server identity (construct props)

The three identity values are sourced from IrcStackProps and joined to the env vars the Lambda config loader reads. motdLines is a string[]; the stack joins it with \n so the loader splits it back into one entry per IRC numeric (375 / 372 / 376):

// apps/aws-stack/src/aws-stack.ts (excerpt)
const motd = (props.motdLines ?? DEFAULT_MOTD_LINES).join('\n');
handler.addEnvironment('SERVER_NAME', serverName);
handler.addEnvironment('NETWORK_NAME', networkName);
handler.addEnvironment('MOTD', motd);

These three values are injected into the IrcHandler and IrcPingChecker Lambda environments at synth time. To override them, pass the construct props when creating the stack:

new IrcAwsStack(app, 'IrcAwsStack', {
  serverName: 'irc.my.net',
  networkName: 'MyNet',
  motdLines: ['Welcome to my network.', 'Be excellent to each other.'],
});

MOTD delimiter contract: the config loader (packages/aws-adapter/src/config-loader.ts) splits MOTD on \n to form the multi-line MOTD array. The CDK stack's motdLines construct prop is a string[] joined with \n, so each array entry becomes its own line and round-trips back through the loader into the expected number of motdLines. Do not use a comma (or any other character) as a separator — the loader treats it as literal line content, collapsing the MOTD into a single line.

7.2 Lambda runtime config (injected by CDK, do not edit)

These env vars are wired automatically by the stack at synth time. Do not hand-set them in the console — the next cdk deploy will overwrite them.

The tablesConfigFromEnv helper (packages/aws-adapter/src/handlers/index.ts:226) throws on cold start if any of the five <NAME>_TABLE vars is missing. The smoke timeout in §13.2 traces back to this.

7.3 Optional config-loader env vars (read if present)

The shared config loader (packages/aws-adapter/src/config-loader.ts) also reads these optional env vars and feeds them through the shared Zod ServerConfig schema. The committed CDK stack does not inject any of them — to enable one, add an addEnvironment(...) line in aws-stack.ts and re-deploy.

The last three are the per-IP admission knobs (§7.8): setting only one half of the rate pair falls back to the schema default (5 per 60 000 ms) for the other half.

SASL_ACCOUNTS is the legacy/config fallback (newline-delimited username:password pairs); it is used only when the Accounts table is empty. When the table has rows it is authoritative — see §8.1.

Invalid values cause a readable boot-time error — the Lambda will throw on the first invocation (cold start). Test changes locally with pnpm --filter @serverless-ircd/aws-adapter test.

Note: WEB_ORIGINS is intentionally NOT in this table. It is an adapter-level operational concern (the $connect CSWSH defence), not a ServerConfig knob — the env var is parsed directly by buildDepsFromEnv via parseWebOrigins rather than threaded through the shared Zod schema. See §8.2.

7.4 No per-environment stack parameter

There is no stageName, environmentType, or environmentName parameter on the CDK stack (the former staging/prod split was collapsed into a single target). The stack name (IrcAwsStack), table names (Connections, ChannelMeta, ChannelMembers, Nicks, Accounts, Services), API Gateway stage name (prod), and Lambda construct ids are all hardcoded. To run staging and production in the same AWS account, you would need to fork the stack to parameterize these names; the recommended path is to use separate accounts (or separate regions) for staging vs production. This is the same architectural choice the Cloudflare adapter makes (one Worker namespace; staging vs production isolation is by account).

7.5 Lambda runtime and bundling

7.6 API Gateway stage

⚠️ Re-enabling DataTraceEnabled leaks IRC credentials

API Gateway execution-log body tracing (DataTraceEnabled = true) writes the FULL payload of every WebSocket frame to the APIGW CloudWatch log group (AWS/ApiGateway…, separate from the stack's access-log group). For an IRC endpoint that includes, verbatim:

• PASS <server password>
• AUTHENTICATE <base64-SASL-PLAIN> (decodes to account\0account\0password)
• JOIN #channel <channel-key>
• every PRIVMSG / NOTICE (private messages)

Anyone with logs:GetLogEvents — operators, SREs, a Logpush destination, or compromised CI credentials — recovers server passwords, SASL accounts, channel keys, and private messages. Retention is account-default (unbounded).

For this reason DataTraceEnabled defaults to false and is the single highest-impact logging control on the stack. INFO-level execution logging alone is enough to diagnose whether API Gateway received and routed a frame or dropped it pre-Lambda.

Because the staging/prod split was collapsed (every synth is effectively a prod synth), there is no quiet "non-prod" path to re-enable it. The only way back on is an explicit, two-step opt-in that exists solely for live debugging in a throwaway sandbox:

pnpm --filter @serverless-ircd/aws-stack run cdk:synth -- \
  -c allowDataTrace=true \
  -c iUnderstandThisLeaksCredentials=true

Setting only -c allowDataTrace=true refuses synthesis with a loud error that names the credentials it would leak. The awkward second flag is intentional — it cannot be set by accident. Tear the sandbox stack down immediately after debugging; never deploy this combination to a shared account.

7.7 DynamoDB table-protection context flags

Two CDK context flags control the state-bearing tables' data-loss protection (Nicks, Services, ChannelMeta, ChannelMembers). Connections is always DESTROY with no PITR and is unaffected by either flag. See §9.4 for the full table and the rationale (and the "no snapshot-on-delete" note).

Both arrive as strings on the CLI and are normalised with the shared parseContextBool helper (same path as allowDataTrace), so -c allowTableDeletion=false stays false-y. They map onto the IrcStackProps.allowTableDeletion / IrcStackProps.enablePitr construct props (apps/aws-stack/bin/aws.ts).

7.8 Connect throttling (per-IP admission, stage limits, WAF)

Connection setup is where credential brute-force spends its budget (PASS / AUTHENTICATE per connection), so the stack throttles it in three layers, outermost first:

Notes:

• Layer 1 is opt-in because WAFv2 web ACLs cost ~$5/mo + $1/rule + $0.60/M requests; for a small deployment, layers 2+3 are usually enough. Enable with pnpm --filter @serverless-ircd/aws-stack run cdk:synth -- -c wafConnectRateLimit=2000 (2000 upgrade requests per 5-minute window per client IP before the IP is blocked at the edge).
• Layer 3's per-IP counts are eventually consistent (GSI reads), so a coordinated burst from one IP can slightly overshoot the cap — the same best-effort TOCTOU contract as the global maxClients counter.
• The 429 response carries ERROR :Closing link semantics at the HTTP layer where possible; clients should treat it as back-pressure (retry after the window decays), not a permanent ban.

8. Secrets

The Lambda consumes one optional secret: SERVER_PASSWORD. When set (as a Lambda env var, conventionally sourced from AWS Secrets Manager or SSM Parameter Store), every connection must supply the matching PASS <value> (or SASL-identify) before 001 RPL_WELCOME; when unset/empty the gate is disabled (the default). The adapter's config loader reads process.env.SERVER_PASSWORD at cold start via loadServerConfigFromLambdaEnv and threads it through to ServerConfig.serverPassword.

The typical provisioning pattern:

1. Create the secret in Secrets Manager (or a SecureString parameter in SSM).
2. Grant the Lambda role secretsmanager:GetSecretValue (or ssm:GetParameter) scoped to that one secret ARN.
3. Read it inside buildDepsFromEnv (cold start) — never inside the per-frame hot path — and expose the value to the Lambda env as SERVER_PASSWORD.

aws secretsmanager create-secret \
  --name serverless-ircd/staging/SERVER_PASSWORD \
  --secret-string "$(openssl rand -base64 32)"

SASL short-circuit: a connection that has authenticated an account via SASL (AUTHENTICATE PLAIN / EXTERNAL) is exempt from the server password — the reducer's gate treats state.account !== undefined as already authorised. Oper creds (OPER_USER / OPER_PASSWORD) follow the same secret pattern; SASL account credentials are not secrets — they are persisted in the Accounts DynamoDB table (see §8.1). Never log SERVER_PASSWORD — treat it as a shared deployment secret, distinct from per-account SASL credentials (which go through the hashed Accounts table).

8.1 SASL accounts (Accounts table + seed tooling)

SASL PLAIN credentials live in the Accounts DynamoDB table as HashedAccountCredential rows — { account, algorithm, salt, hash }, all strings, with scrypt hashes encoded as base64 (never plaintext). At Lambda cold start, buildDepsFromEnv → resolveAccountStore scans the table once and pre-loads every row into a DynamoAccountStore; the synchronous verify() called from the SASL reducer works against that in-memory snapshot. The snapshot is rebuilt on every cold start, so account changes take effect within one cold start (seconds to minutes).

Precedence (resolveAccountStore, in account-store.ts):

1. The Accounts table is authoritative when it has ≥1 row — the table wins and the config seed is ignored.
2. When the table is empty, the adapter falls back to the SASL_ACCOUNTS env var (newline-delimited username:password pairs, see §7.3) so existing deployments keep working without migrating.
3. When neither source has accounts, no AccountStore is bound and AUTHENTICATE PLAIN returns 904 ERR_SASLFAIL.

Seed / CRUD tooling. Provision accounts by writing HashedAccountCredential rows produced by hashAccountCredential (scrypt, random salt, base64-encoded). The adapter exports putAccountCredential(docClient, tableName, username, password) for this — a repeat call for the same username overwrites the row (the PK is account). A minimal seeding snippet (Node 20, from the repo root):

node --input-type=module -e '
  import { createDynamoDocumentClient, putAccountCredential } from "@serverless-ircd/aws-adapter";
  const dynamo = createDynamoDocumentClient({ region: "us-east-1" });
  await putAccountCredential(dynamo, "Accounts", "alice", "change-me");
  console.log("seeded alice");
'

A repo-local seed script (tools/seed-aws-accounts.ts) wraps the same calls for batch provisioning:

# Seed against DynamoDB Local:
npx tsx tools/seed-aws-accounts.ts \
  --table Accounts \
  --endpoint http://localhost:8000 \
  --accounts alice:s3cret bob:password2

# Or from a file (newline-delimited username:password):
npx tsx tools/seed-aws-accounts.ts \
  --table Accounts \
  --region us-east-1 \
  --file accounts.txt

For a one-off aws-cli write, hash the password out-of-band and put the row directly:

# Hash with the adapter, then write the row yourself:
aws dynamodb put-item \
  --table-name Accounts \
  --item '{"account":{"S":"alice"},"algorithm":{"S":"scrypt"},"salt":{"S":"<base64-salt>"},"hash":{"S":"<base64-hash>"}}'

The stack's DynamoDB grants are per-function, not shared (see the grant matrix in §9.6): the interactive handler and the sweeper hold read-write on Services (credential verify / data-lifecycle sweep); the ping-checker holds no Services grant at all.

Never put credentials, password hashes, or API tokens in aws-stack.ts, in Lambda env vars, or in committed files. Lambda env vars are visible in the console and in CloudFormation describe output; they are not secret.

8.2 $connect Origin allowlist (WEB_ORIGINS, CSWSH defence)

WebSocket upgrades do not follow the same-origin policy, so a malicious web page can open a WebSocket to the IRC server from the victim's browser and drive the session with the victim's credentials (Cross-Site WebSocket Hijacking, CSWSH). API Gateway WebSocket has no built-in Origin validation — the $connect Lambda receives event.headers.Origin and must enforce the allowlist itself.

Knob: WEB_ORIGINS — a comma-separated list of allowed web origins (e.g. https://app.example.com, https://staging.app.example.com). Mirror of the CF Worker's WEB_ORIGINS (TICKET-137).

Behaviour (packages/aws-adapter/src/origin-allowlist.ts and packages/aws-adapter/src/handlers/connect.ts):

The defence is opt-in. Unset WEB_ORIGINS skips the check entirely — existing AWS deployments without a web frontend are unchanged on upgrade. The check only matters once you ship a web client (TICKET-164) and want to admit the SPA's origin while rejecting hostile ones.

Matching semantics (mirror the CF Worker):

• entries are trimmed of surrounding whitespace and lowercased (https://APP.Example.COM → https://app.example.com);
• the incoming Origin header is normalised the same way before the membership test, so the match is case-insensitive on scheme+host (per RFC 3986);
• origins are otherwise case-sensitive — http://app.example.com is a different origin from https://app.example.com and is denied if only the https:// form is listed (defends against a stripped-TLS attacker on the same hostname).

Deliberate divergence from the CF policy: the CF Worker has a same-origin auto-derive mode (when WEB_ORIGINS is unset, it compares Origin against the request's own Host). On AWS the SPA is served from a different CloudFront origin than the API Gateway WebSocket endpoint (wss://{api-id}.execute-api.…), so the request's own host is never the SPA's origin — auto-derive does not apply. The AWS defence is therefore explicit-allowlist only.

Wiring. Inject WEB_ORIGINS on the $connect Lambda via the CDK stack prop:

new IrcAwsStack(app, 'IrcAwsStack', {
  webOrigins: 'https://app.example.com, https://staging.app.example.com',
  // ...other props
});

…or at deploy time via CDK context (-c webOrigins=…). The stack injects the env var only on the wss $connect handler (the sweeper, ping-checker, and NLB handler never run $connect). The handler parses the value once per cold start via parseWebOrigins and stores the resulting Set<string> on HandlerDeps.webOrigins.

WEB_ORIGINS is not a secret — it is a public allowlist of origin strings. Do not put it in Secrets Manager / SSM; set it as a plain Lambda env var (the CDK stack does this directly via addEnvironment).

The CI deploy uses GitHub Actions secrets plus an optional repo variable (see §12):

Configure all of these under repo Settings → Secrets and variables → Actions.

9. DynamoDB capacity planning

9.1 All five tables run on-demand

Every table is created with BillingMode.PAY_PER_REQUEST (asserted in apps/aws-stack/tests/stack.test.ts:45). On-demand pricing charges per request:

• Write request units (WRU) — one WRU = one write of up to 1 KB (or 2 WRU for a strongly-consistent write of up to 1 KB). A conditional write that fails (e.g. attribute_not_exists rejects a nick collision) still costs 1 WRU.
• Read request units (RRU) — one RRU = one eventually-consistent read of up to 4 KB (or 2 RRU for strongly consistent).

There is no minimum spend and no capacity to plan at deploy time. This is intentional for v1 — it removes a class of ProvisionedThroughputExceededException failure modes. See §11 for the cost trade-off and when to consider provisioned capacity.

9.2 Schema and access patterns

Five tables (defined in packages/aws-adapter/src/cdk-table-defs.ts):

Uniqueness invariants are enforced structurally (not in application code):

• Nicks.PK = nickLower — conditional PutItem with attribute_not_exists(PK) makes nick reservation atomic across concurrent Lambdas. Two clients racing on Alice both get evaluated against the same row; exactly one wins.
• ChannelMembers PK+SK composite — TransactWriteItems for JOIN/PART updates Connections.joined and ChannelMembers in one atomic write (up to 100 items per transaction, the AWS limit).
• Accounts row schema — { account (PK), algorithm, salt, hash }, all strings. algorithm is 'scrypt'; salt and hash are base64-encoded. Never plaintext. SASL PLAIN verifies against this hash server-side. Seed rows with putAccountCredential (see §8.1).

Connections secondary indexes. Besides the per-IP admission GSI (sourceIp-connectedSince, §7.8), the table carries two sparse single-key GSIs for global fanout: wallopsFlag and operFlag. Each index's partition attribute is written only while the corresponding user mode is set (+w / +o), atomically with the userModes map on every persist (both transports), so each index carries exactly its recipient set. WALLOPS and the OPER per-IP lockout notice fan out via a paginated Query against these indexes — cost is bounded by the recipient count, not the connection fleet (this replaced a full-table Scan per fanout). GSI reads are eventually consistent, as the old Scan was; a divergent index entry is additionally re-checked against the row's userModes before delivery. Deploying the change against an existing stack adds both GSIs online — DynamoDB backfills them with no table downtime and no table-capacity consumption — but rows written before the flag attributes shipped stay invisible to the indexes until their next persist, so a +w user set by an old build may miss WALLOPS for one persist cycle after the upgrade. Deleting or TTL-reaping a row removes it from the indexes along with the row, so both self-clean.

9.3 The hottest pattern: channel fanout

Channel PRIVMSG is the dominant cost driver. There is no long-lived per-channel process (contrast with the Cloudflare adapter's ChannelDO). Fanout is computed per message:

1. The reducer emits Broadcast("#chan", [line], except=sender).
2. AwsRuntime.broadcast() issues a Query on ChannelMembers with PK = "#chan" → returns one row per member.
3. For each member, the Lambda calls ApiGatewayManagementApi.postToConnection(member, line).

That's O(members) Management API calls per message. A 100-member channel receiving one PRIVMSG per second produces ~100 postToConnection calls/second and one Query/second. A 1k-member announcement channel at the same rate produces ~1k calls/second. Mitigations (batched postToConnection, per-channel send-list cache, a dedicated fanout Lambda) are deferred to post-v1; the load-test ticket will surface the ceiling.

9.4 Removal policy, deletion protection, and PITR (IMPORTANT)

Every table is protected from accidental data loss by default. The state-bearing tables (Nicks, Services, ChannelMeta, ChannelMembers) get three independent safeguards, each toggleable so a deliberate teardown can still proceed; the one ephemeral table (Connections) is always DESTROY:

• RemovalPolicy.RETAIN — when the stack is deleted, CloudFormation orphans the table instead of deleting it; the data survives.
• deletionProtection: true — DynamoDB refuses a DeleteTable API call while this is on, blocking a stray aws dynamodb delete-table even from an admin or compromised credential.
• pointInTimeRecovery (PITR) — continuous, per-table restore (within the last ~35 days) against accidental writes/deletes that RETAIN + deletion protection cannot stop (e.g. a buggy UpdateItem that overwrites rows).

No snapshot-on-delete. DynamoDB tables do NOT snapshot when a stack is deleted (RemovalPolicy.SNAPSHOT is a no-op here). RETAIN (orphan + keep) + PITR (continuous, in-place restore) is the correct combination; do not rely on a snapshot that does not exist.

Two CDK context flags control these (threaded through bin/aws.ts, see §7.7):

• allowTableDeletion (default false). Set -c allowTableDeletion=true for a deliberate cdk destroy teardown: every state table flips to RemovalPolicy.DESTROY with deletionProtection: false so CloudFormation can delete them. The data is irreversibly lost — flip this off only when you genuinely want the tables gone. Connections is always DESTROY regardless.
• enablePitr (default true). Set -c enablePitr=false to disable PITR on every state table. Independent of allowTableDeletion — a teardown keeps PITR on (harmless, and the data is still recoverable up until the table is actually deleted). Connections never carries PITR.

# Default deploy: state tables RETAIN + deletion-protected + PITR-on.
pnpm --filter @serverless-ircd/aws-stack run cdk:synth

# Deliberate teardown (state tables become DELETABLE):
pnpm --filter @serverless-ircd/aws-stack run cdk:synth -- -c allowTableDeletion=true

# Disable PITR (e.g. cost-sensitive dev deploy):
pnpm --filter @serverless-ircd/aws-stack run cdk:synth -- -c enablePitr=false

The protection knobs are asserted per parameter combination in apps/aws-stack/tests/stack.test.ts (the "DynamoDB table protection" suite).

9.5 TTL on Connections.idleSince

The Connections table carries a DynamoDB TTL attribute (idleSince), asserted at synth time (stack.test.ts:52). The Lambda updates idleSince on every frame; DynamoDB silently deletes rows whose idleSince is older than the TTL window (typically within 48 hours of expiry, no SLA). This is a safety net for rows the sweeper misses.

9.6 The gone-connection sweeper

A second Lambda (IrcSweeper, handler export sweeperHandler) runs on an EventBridge schedule at rate(5 minutes):

new Rule(this, 'SweeperSchedule', {
  schedule: Schedule.rate(Duration.minutes(5)),
  targets: [new LambdaFunction(sweeper)],
});

Each tick Scans the entire Connections table, identifies rows past a hard stale threshold, and runs the same cleanup path as $disconnect (delete memberships, release nicks, delete the row). The sweeper catches connections that vanished without APIGW emitting $disconnect (e.g. TCP RST that APIGW didn't observe).

Cost note: Scan reads the entire table every 5 minutes. At 1 KB per row, scanning 10k connections costs ~3 RRU (10k / 4 KB per RRU) per tick, ~864 ticks/day, ~2.6k RRU/day — negligible. At 100k connections it's ~26k RRU/day — still small but visible. A future optimization is to make the sweeper Query a sparse GSI keyed by idleSince instead of Scan.

DynamoDB grant matrix (per-function least privilege)

Each Lambda's execution role is scoped at synth time to the tables its handler actually touches, derived from the handler call graph (not from assumption) and pinned by the "scheduled-Lambda DynamoDB grant matrix" suite in apps/aws-stack/tests/stack.test.ts:

Why the scheduled functions get less than the interactive handler:

• Sweeper — Nicks/ChannelMembers write-only: its teardown path (cleanupConnection with managementApi: null) only deletes the nick row and the membership rows; the QUIT-fanout Query never runs without a management-API client.
• Sweeper — Services read-write: the same schedule also runs the services data-lifecycle expiry sweep (handleServicesExpirySweep), which flushes and reloads the services store.
• Sweeper — no ChannelMeta: teardown never touches channel metadata.
• Ping-checker — ChannelMembers read-write: it does hold a real management-API client, so the QUIT fanout Queryies channel members before the teardown deletes the rows.
• Ping-checker — no Services / no ChannelMeta: handlePingCheck never binds the services store and never reads channel metadata.
• Interactive handler — all five read-write: the full IRC actor touches every table (Services for NickServ/SASL, ChannelMeta for JOIN/PART/TOPIC/modes).

(dynamodb:DescribeTable rides along with every CDK data grant — the SDK client needs it — and is treated as neutral by the pinning test.)

9.7 When to consider provisioned capacity

Switch a table to PROVISIONED billing (with or without autoscaling) when its traffic is predictable and steady. The clearest signal is when the on-demand bill for a single table exceeds the cost of provisioned capacity for the same throughput — DynamoDB's pricing page breaks this out. In practice:

• ChannelMembers is the first candidate (channel fanout dominates).
• Connections is a poor candidate (spiky, cold-start-driven).
• Accounts/ChannelMeta/Nicks are unlikely to need provisioned capacity in any v1-size deployment.

Switching billing modes is a no-downtime UpdateTable operation; the stack change is billingMode: BillingMode.PROVISIONED plus readCapacity/writeCapacity props.

10. Region strategy

10.1 Everything is regional

API Gateway WebSocket endpoints are regional. DynamoDB is regional. EventBridge Scheduler is regional. Lambda is regional. The stack deploys into exactly one region (the one your credentials resolve to via CDK_DEFAULT_REGION).

WebSocket clients pay round-trip time to that region. A user in Frankfurt connecting to a us-east-1 deployment will see ~90 ms RTT on every PRIVMSG roundtrip; the same user against eu-central-1 will see ~10 ms. Pick the region closest to your user base.

10.2 Recommendation: single-region for v1

For v1, deploy in one region. The IRC runtime is regional stateful compute (Lambda + DynamoDB + APIGW); CloudFront only enters the picture for the optional web client (§16) and is global-edge, not a region picker. Single-region keeps the data model simple: the cross-table TransactWriteItems flows (§9.2) only work within one region.

10.3 Multi-region is out of scope for v1

If you ever need multi-region active-active, the considerations are:

• DynamoDB global tables — replicate all five tables. Eventual consistency across regions complicates the conditional-write invariants (nick uniqueness, membership transactions): a nick reserved in region A may not be visible in region B for a second or two, allowing duplicate reservations. The reducer logic would need to tolerate this.
• Per-region APIGW + Lambda — each region runs the full stack; clients are routed by DNS (Route 53 latency-based routing).
• Session affinity — a client's connectionId is regional. A reconnect after the 2-hour APIGW cap must land on the same region or treat the new connection as a fresh registration (re-NICK, re-JOIN).

This is post-v1 work; the AWS adapter architecture doc calls it out as an open question.

10.4 Region pick shortlist

For a globally distributed user base, the typical picks:

• US-East-1 (N. Virginia) — newest services, cheapest, ~100 ms to EU and ~150 ms to APAC. Default for the CI workflow's AWS_REGION.
• EU-West-1 (Ireland) / EU-Central-1 (Frankfurt) — best for EU users.
• AP-Northeast-1 (Tokyo) / AP-Southeast-1 (Singapore) — best for APAC users.

11. Cost notes

All figures are the public AWS list price as of late 2024 — always verify against the current AWS pricing page before budgeting:

• API Gateway WebSocket: https://aws.amazon.com/apigateway/pricing/
• Lambda: https://aws.amazon.com/lambda/pricing/
• DynamoDB: https://aws.amazon.com/dynamodb/pricing/
• EventBridge: https://aws.amazon.com/eventbridge/pricing/

11.1 Billable services

The Lambda free tier (1M requests + 400k GB-seconds per month) covers a meaningful slice of a small staging deployment.

Table-protection cost. RemovalPolicy.RETAIN and deletionProtection are free — they are CloudFormation/DynamoDB flags, not billable features. Point-in-time recovery (PITR) is billed: continuously at ~$0.20/GB-month per table, charged on the full table size including indexes, for as long as PITR is on. With the default (enablePitr=true) this applies to the four state tables (Nicks, Services, ChannelMeta, ChannelMembers); Connections never carries PITR. For a small deployment these tables are tiny (kilobytes to low megabytes), so the PITR line item is fractions of a cent; it only becomes material if a table grows to many GB. Disable with -c enablePitr=false on a cost-sensitive dev deploy (see §9.4).

11.2 Back-of-envelope: 100 concurrent users

Assume a small staging deployment with 100 concurrent users, average 5 messages/sec across all channels, each connection active for 1 hour, average Lambda invocation 50 ms at 128 MB:

Rough total: ~$80/month for a 100-concurrent-user staging deployment. The dominant cost is APIGW messages (inbound + the per-member fanout). The single biggest lever is channel size — a single 1k-member announcement channel can multiply the fanout cost by 10×.

11.3 Cost-trim levers

• Cap channel size with +l mode (the existing MODE enforcement).
• Consider provisioned capacity for ChannelMembers once traffic is steady (§9.7).
• Shorten the sweeper cadence if you can tolerate slower cleanup — rate(5 minutes) is a balance; rate(15 minutes) cuts sweeper cost 3× at the cost of slower gone-connection reaping.
• Tune Lambda memory — 128 MB is the default and cheapest per invocation, but a higher memory allocation can reduce duration enough to win on GB-seconds.

12. CI/CD

The GitHub Actions workflow at .github/workflows/deploy-aws.yml is workflow_dispatch-only (a maintainer triggers it by hand); there is no automatic push-to-main deploy. It deploys the single AWS target and replays the smoke e2e.

12.1 What the workflow does

Steps performed (in order):

1. Checkout.
2. Install pnpm 9 + Node 20 (cache: pnpm).
3. pnpm install --frozen-lockfile.
4. pnpm build — builds all workspace packages.
5. pnpm typecheck and pnpm test — gate the deploy.
6. Assert AWS_DEPLOY_ROLE_ARN is non-empty (fail-fast before any AWS call).
7. Configure AWS credentials via OIDC (aws-actions/configure-aws-credentials@v4 with role-to-assume only — no static keys).
8. Assert aws sts get-caller-identity returns the configured role (defence-in-depth against silent fallbacks).
9. pnpm deploy:aws → cdk deploy --all --require-approval never.
10. Resolve the smoke URL: repo variable AWS_SMOKE_URL first, else the stack's ConnectUrl CloudFormation output via aws cloudformation describe-stacks --stack-name IrcAwsStack, else skip the smoke step with a warning.
11. pnpm smoke:aws -- --url "$SMOKE_URL" — failure fails the build.

Concurrency is serialized via concurrency.group: aws-deploy, cancel-in-progress: false, so two runs cannot race the same CloudFormation stack.

12.2 Required repo configuration

Under Settings → Secrets and variables → Actions:

• Secret AWS_DEPLOY_ROLE_ARN — required. ARN of the IAM role the workflow assumes via GitHub OIDC. There is no static-key fallback. Set this up once per AWS account (see §17 for the trust policy).
• Secret AWS_REGION — required (e.g. us-east-1).
• Variable AWS_SMOKE_URL — optional but recommended; the workflow can fall back to the CFN output, but setting this variable explicitly avoids the describe-stacks round-trip and the possibility of a stale cached value after a redeploy.

Removed: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are no longer read by the workflow. Long-lived access keys powerful enough to deploy CloudFormation + IAM + Lambda + DynamoDB are effectively account-admin; a single exfil vector yields full AWS account takeover. The H1 security-hardening ticket dropped them. If you have stale AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY secrets in the repo, delete them and invalidate the underlying IAM access key in IAM → Users → Security credentials.

12.3 OIDC is mandatory

The workflow accepts only GitHub OIDC web-identity credentials:

permissions:
  id-token: write
  contents: read

# …

- name: Assert OIDC deploy role ARN is set
  if: env.AWS_DEPLOY_ROLE_ARN == ''
  env:
    AWS_DEPLOY_ROLE_ARN: ${{ secrets.AWS_DEPLOY_ROLE_ARN }}
  run: |
    echo "::error::AWS_DEPLOY_ROLE_ARN secret is empty …"
    exit 1

- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    aws-region: ${{ secrets.AWS_REGION }}
    role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE_ARN }}

- name: Assert assumed role matches AWS_DEPLOY_ROLE_ARN
  # aws sts get-caller-identity + ARN-shape check; aborts before cdk deploy.
  run: …

The id-token: write permission lets Actions mint the OIDC JWT that sts:AssumeRoleWithWebIdentity consumes. After the assume-role step the workflow calls aws sts get-caller-identity and asserts the returned ARN's role-name + account-id match AWS_DEPLOY_ROLE_ARN, so a silent fallback to ambient static credentials cannot ship a deploy. See §17 for the matching IAM trust policy and verification.

12.4 Manual-only deploy trigger

The workflow's on: block is workflow_dispatch-only — there is no automatic push-to-main trigger, and no separate staging/prod target. A maintainer runs the deploy by hand against whichever AWS account and region the secrets point at; staging vs production isolation is driven by account/region, not by a workflow-per-env split.

13. Troubleshooting

13.1 cdk deploy fails with bucket-not-found

BUCKET_NAME_PARAMETER_<id> does not exist or NoSuchBucket: The specified bucket does not exist

The CDK bootstrap stack hasn't been created in this account/region.

cd apps/aws-stack
pnpm cdk bootstrap
pnpm deploy:aws

Bootstrap is idempotent and per-region — re-run it if you change regions.

13.2 Lambda throws on cold start because of a missing env var

Symptom: smoke e2e times out waiting for 001; CloudWatch Logs for IrcHandler show an Error: Missing env var CONNECTIONS_TABLE (table name for Connections) (or one of the other four <NAME>_TABLE vars).

The tablesConfigFromEnv helper throws on the first invocation if any of CONNECTIONS_TABLE, CHANNELMETA_TABLE, CHANNELMEMBERS_TABLE, NICKS_TABLE, ACCOUNTS_TABLE is missing. The CDK stack always injects these — if one is missing in the console, somebody hand-edited the Lambda config. Fix by re-deploying:

pnpm deploy:aws

If the error mentions MANAGEMENT_URL being absent: that env var is optional at the loader level (buildDepsFromEnv tolerates its absence by setting managementApi = null), but the handler's fanout path will fail at the first postToConnection. Verify both IrcHandler and IrcPingChecker carry MANAGEMENT_URL in the console.

13.3 Smoke e2e times out waiting for 001

The connection upgrade succeeded (the WebSocket open fired) but registration didn't complete. Common causes:

• The handler threw on the first frame. Tail the logs:

  aws logs tail /aws/lambda/IrcHandler --follow
  

  and reproduce. Look for the cold-start buildDepsFromEnv throw (§13.2) or an unexpected exception in handleDefault.
• The prod stage didn't deploy. The stage is autoDeploy: true, so this is rare, but verify the URL ends in /prod and that the route is wired:

  aws apigatewayv2 get-routes --api-id <api-id>
  

  You should see three routes: $connect, $disconnect, $default.
• Client connected over plain ws://. APIGW WebSocket stages only accept wss://. Use the URL printed by cdk deploy (the ConnectUrl output).

13.4 aws cloudformation describe-stacks returns nothing in CI

The workflow uses:

aws cloudformation describe-stacks \
  --stack-name IrcAwsStack \
  --query 'Stacks[0].Outputs[?OutputKey==`ConnectUrl`].OutputValue' \
  --output text

to resolve the smoke URL. If this returns nothing, either:

• The stack name is wrong. It's hardcoded as IrcAwsStack in apps/aws-stack/bin/aws.ts. If you forked the stack to a different name, the workflow's --stack-name IrcAwsStack won't find it. Update the workflow or rename back.
• The deploy failed silently. Check the prior pnpm deploy:aws step's exit code. cdk deploy with --require-approval never can still fail on IAM or CloudFormation errors.
• Wrong region. The describe-stacks call uses AWS_REGION from the secrets; it must match the region the stack was deployed into.

The workflow prints a ::warning:: and skips the smoke step rather than failing the build — a silent skip is the failure mode to watch for.

13.5 WebSocket client can't connect

The $connect route is wired to IrcHandler and currently uses no auth (no authorizer). If the client gets a 4xx/5xx on the HTTP upgrade:

• Verify the URL is the ConnectUrl output, not ManagementUrl (the management endpoint rejects WebSocket upgrades).
• Verify the IrcHandler Lambda has execute-api:ManageConnections on this stage (the stack grants this via stage.grantManagementApiAccess(handler)).
• If you add an authorizer or IP allowlist later, the $connect Lambda is the natural place to gate by source IP (read event.requestContext.connectionId and event.requestContext.sourceIp).

13.6 DynamoDB table vanished after cdk destroy

By design. The stack uses RemovalPolicy.DESTROY on every table (§9.4) so staging teardowns are clean. Do not run cdk destroy against a production stack. For production, fork the stack to RemovalPolicy.RETAIN before the first deploy; flagged in §15.

13.7 High DynamoDB cost on Connections

The gone-connection sweeper Scans the entire Connections table every 5 minutes (§9.6). At very large scale this dominates the read bill. Short-term mitigation: increase the sweeper's rate(...) to rate(15 minutes) or rate(30 minutes) in aws-stack.ts. Long-term: add a sparse GSI keyed by idleSince and switch the sweeper to a Query against it (post-v1 work).

13.8 Nick reservation always fails

A client's NICK always returns 433 ERR_NICKNAMEINUSE, even for nicks that should be free. Two deployments sharing the same DynamoDB table name space is the usual cause — e.g. you deployed IrcAwsStack twice into the same account/region with a forked stack name. The Nicks table name is hardcoded as Nicks (physical name), so two stacks in the same region collide on the same table. Use separate regions or fork the table-name prefix.

14. Platform limitations

Architectural constraints of the API Gateway WebSocket + Lambda substrate that operators should be aware of. These are not bugs — they are consequences of the serverless transport choice (see ADR-005 and ADR-009 for the wss-only v1 decision and the dual-transport v1.x path).

14.1 A Lambda $default invocation cannot close its own socket

This is the most important platform limit to understand.

When a client sends an IRC command that should terminate its own connection — QUIT, or a flood-control Disconnect("Excess Flood"), or a fatal protocol error — the ConnectionActor emits a Disconnect effect. On the Cloudflare adapter the bound ConnectionDO closes its WebSocket immediately. On AWS, nothing closes the socket from within the $default Lambda invocation. The AwsRuntimeHandlers.disconnect implementation in packages/aws-adapter/src/handlers/default.ts is deliberately a no-op:

disconnect: () => {
  // No-op: a Lambda `$default` invocation cannot close its own APIGW
  // socket. … See docs/AWS-Deployment.md §14.1.
},

Why. API Gateway WebSocket connections are managed by the service, not by the Lambda compute. A Lambda handling a $default frame has no API to force-close the calling WebSocket synchronously — the ApiGatewayManagementApi client can post to a connection, but there is no DeleteConnection call available to the handling Lambda for its own socket. The connection stays open until APIGW observes a TCP/FIN (the client disconnects) or the 2-hour idle cap elapses.

What actually tears the connection down. The canonical teardown path is the $disconnect route (packages/aws-adapter/src/handlers/disconnect.ts), which APIGW invokes asynchronously after the socket closes. That route runs cleanupConnection — deleting the Connections row, every ChannelMembers row owned by it, the nick reservation, and broadcasting the QUIT line to every peer sharing a channel (see §9.6 for the sweeper that catches connections which vanish without a $disconnect).

So the effect-driven Disconnect is not lost: the actor's state mutations (nick release, roster deltas) are persisted by handleDefault before the invocation returns, and the eventual $disconnect route completes the fanout. The only observable gap is that the socket lingers between the QUIT command and the client (or APIGW) actually closing it. Well-behaved IRC clients close the socket themselves immediately after sending QUIT, so the window is sub-second.

Cross-connection disconnect is also a no-op. AwsRuntime.disconnect(remoteConn, reason) (packages/aws-adapter/src/aws-runtime.ts:183) cannot directly close another connection's WebSocket either. Commands that remove a user from a channel (e.g. KICK) delete membership rows and fan out the line, but do not force-close the target's socket — the kicked client will not be disconnected until it observes its own removal (via the KICK message) and closes, or the ping-checker/sweeper reaps it after the idle window.

14.2 Mitigations already in place

The adapter ships three mechanisms that bound the gap left by the no-op disconnect:

Together these ensure ghost rosters and stale nick reservations are reaped within minutes even when the transport cannot force-close.

14.3 Future enhancement (not implemented)

A possible mitigation for true synchronous force-close from within $default would be to emit an EventBridge event (or an SQS message) from the handler that a sidecar Lambda consumes to call ApiGatewayManagementApi's connection-deletion path against the target connectionId. This would let KICK/flood-disconnect close the target socket without waiting for the idle window. This is out of scope for v1 and is tracked as a future enhancement; file an ADR before implementing it, because it adds a second async hop and a new failure mode (the sidecar falling behind).

15. Quick reference

# One-time account setup (per region)
aws configure                                     # or: export AWS_PROFILE=...
cd apps/aws-stack && pnpm cdk bootstrap

# Local dev
pnpm install
pnpm build
pnpm --filter @serverless-ircd/aws-stack test     # synth-time assertions
pnpm --filter @serverless-ircd/aws-stack cdk:synth # writes cdk.out/

# Optional localstack validation (Docker required)
docker run --rm -d -p 4566:4566 localstack/localstack
npm install -g aws-cdk-local
export AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_DEFAULT_REGION=us-east-1
LOCALSTACK=1 pnpm --filter @serverless-ircd/aws-stack test

# Deploy (single target — staging vs prod is which account/region you point at)
pnpm deploy:aws
node apps/aws-stack/scripts/smoke.mjs \
  --url wss://<id>.execute-api.<region>.amazonaws.com/prod
aws logs tail /aws/lambda/IrcHandler --follow     # live logs

# Web client (opt-in; provision via webSite/webSiteCustomDomain context)
pnpm deploy:aws -- -c webSiteCustomDomain=app.example.com \
                  -c webSiteCertificateArn=arn:aws:acm:us-east-1:... \
                  -c webSiteHostedZoneName=example.com. -c webSiteHostedZoneId=... \
                  -c webOrigins=https://app.example.com   # CSWSH (§8.2, §16.4)
node scripts/deploy-web-aws.mjs                    # bake + s3 sync + invalidate (§16.3)

# Tests / lint
pnpm test                                         # full workspace
pnpm --filter @serverless-ircd/aws-adapter test
pnpm --filter @serverless-ircd/aws-stack test
pnpm lint

Key files:

16. Web client (S3 + CloudFront + OAC)

The CF Worker serves the SPA from the same origin as the WebSocket (one Worker, one domain). AWS cannot do this — API Gateway WebSocket and a static SPA host are different AWS services on different endpoints — so the AWS web client splits in two: the wss IRC endpoint stays on API Gateway, and the SPA is served from an S3 bucket fronted by CloudFront. This § documents the architecture, the build, the deploy pipeline, the CSWSH defence (mandatory once the SPA ships), and the optional custom-domain setup.

Cross-reference: docs/WebClientGuide.md covers the Cloudflare path (one origin for HTTP + WebSocket). The CF-vs-AWS architectural difference is called out in §16.5 below.

16.1 What gets provisioned

The StaticSite construct (apps/aws-stack/src/static-site.ts, provisioned only when IrcStackProps.webSite is supplied) creates:

• a private S3 bucket holding the apps/web/dist build artifact (index.html for the landing page, webclient/index.html for the Kiwi SPA, webclient/static/* for hashed JS/CSS) — the same layout the CF Worker's [assets] binding serves, so one artifact ships on both platforms unchanged;
• a CloudFront distribution with an S3 origin pointing at the bucket via Origin Access Control (OAC) — the modern replacement for the legacy OAI. Direct S3 access (https://<bucket>.s3.…/…) is denied by the bucket policy; only CloudFront URLs resolve;
• defaultRootObject = index.html so requests to / serve the project landing page (the static HTML front door with links to the SPA, the rendered docs, and the source repo);
• a custom error response mapping S3's 403/404 → 200 with response page /webclient/index.html, so client-side routes under /webclient/* resolve to the SPA shell (Kiwi is hash/history-routed);
• three CFN outputs at the parent-stack level (stable logical IDs): WebsiteURL (the *.cloudfront.net default OR the custom domain), WebsiteBucketName, WebsiteDistributionId. The deploy pipeline (§16.3) consumes the latter two.

Why not Amplify / why not S3-only: Amplify Hosting adds a managed build step the project doesn't need (the SPA build is already pnpm --filter web build:prod-aws); S3-only (no CloudFront) forces either a public bucket (anti-pattern) or a signed-URL-per-asset scheme (heavy). CloudFront + OAC is the AWS-recommended static-site shape and gives HTTPS on *.cloudfront.net for free.

16.2 Opt-in: provision the web client

The web client is opt-in. To provision it, pass webSite props on the stack. From CDK context (the easiest path), set webSiteCustomDomain:

pnpm deploy:aws -- \
  -c webSiteCustomDomain=app.example.com \
  -c webSiteCertificateArn=arn:aws:acm:us-east-1:111122223333:certificate/abc-def-... \
  -c webSiteHostedZoneName=example.com. \
  -c webSiteHostedZoneId=Z2KVMGOMGMOU2

The ACM certificate must be in us-east-1 (CloudFront requirement). Hosted-zone name + ID are both required because CDK's fromHostedZoneAttributes needs both; retrieve them once via aws route53 list-hosted-zones-by-name --dns-name <zone>.

Default-*.cloudfront.net only (no custom domain) gap: bin/aws.ts currently keys the webSite prop's existence on webSiteCustomDomain being set, so provisioning the construct without a custom domain requires editing bin/aws.ts to pass webSite: {} directly. Tracked as a follow-up. For v1 deploys, supply a custom domain (prod-style) or patch bin/aws.ts.

16.3 Two-phase deploy (stack → bake → s3 sync → invalidate)

Unlike the CF Worker (where wrangler deploy ships the SPA in the same command as the runtime), the AWS SPA deploy is two-phase and stack-output-driven: the CloudFormation stack must exist before the SPA can be baked, because the baked config.json carries the API Gateway WebSocket URL as a literal. The deploy pipeline (scripts/deploy-web-aws.mjs) runs:

1. aws cloudformation describe-stacks --stack-name IrcAwsStack reads the ConnectUrl, WebsiteBucketName, and WebsiteDistributionId outputs. If any are missing the script exits non-zero ("stack not deployed" or "web client construct not provisioned").

2. pnpm --filter @serverless-ircd/web run build:prod-aws -- --api-url <ConnectUrl> bakes the SPA with the real API Gateway URL. The --api-url flag is overridden in memory after Zod parse — the source apps/web/static/config.prod-aws.json is never mutated, so the templated {{API_ID}}.execute-api.{{REGION}}.amazonaws.com host and /{{STAGE}} path placeholders stay in the repo and no concrete URL is committed. The build rejects any leftover {{...}} placeholders.

   The flag value (wss://<id>.execute-api.<region>.amazonaws.com/<stage>) is split by parseApiUrlForKiwi into Kiwi's two URL fields: server ← <id>.execute-api.<region>.amazonaws.com (hostname only, no scheme) and direct_path ← /<stage>. irc-framework builds the WS URL itself as wss://<server>:<port><direct_path> — it prepends wss:// whenever tls: true, so passing the full URL through to server would double the scheme (wss://wss//<host>...) and Firefox would reject the connection. The split keeps server host-only and routes the API Gateway stage name through direct_path.

3. aws s3 sync apps/web/dist s3://<WebsiteBucketName> --delete uploads the build. The CLI skips unchanged objects via etag comparison, so a re-deploy with no SPA changes uploads nothing. (This is why the deploy uses the AWS CLI rather than the SDK — s3 sync's etag-based unchanged-skip is non-trivial to reproduce on @aws-sdk/client-s3.)

4. aws cloudfront create-invalidation --distribution-id <WebsiteDistributionId> --paths /* busts the edge cache so the next visitor gets the new build.

CI wiring: the deploy-web job in .github/workflows/deploy-aws.yml runs node scripts/deploy-web-aws.mjs with needs: deploy (after the CDK stack applies). Both jobs are workflow_dispatch-only — a maintainer runs the workflow by hand (T170 collapsed the staging/prod split; there is no push trigger and no staging branch).

Why a script instead of CDK BucketDeployment: BucketDeployment re-synths on every deploy, is slower, and — critically — cannot bake a post-deploy stack output (the API Gateway URL) into the SPA config, because at synth time the URL does not exist yet. The two-phase script reads the output, then bakes.

16.4 CSWSH defence (WEB_ORIGINS) — mandatory once the SPA ships

Once you deploy the web client, set WEB_ORIGINS. WebSocket upgrades do not follow the same-origin policy, so a malicious web page can otherwise open a wss to the IRC endpoint from a victim's browser and drive the session with the victim's credentials (Cross-Site WebSocket Hijacking, CSWSH). API Gateway WebSocket has no built-in Origin validation — the $connect Lambda enforces it.

Pass the SPA's origin alongside the webSite props:

pnpm deploy:aws -- \
  -c webSiteCustomDomain=app.example.com \
  -c webSiteCertificateArn=... \
  -c webSiteHostedZoneName=example.com. \
  -c webSiteHostedZoneId=... \
  -c webOrigins=https://app.example.com

The full behaviour matrix, matching semantics, and the deliberate divergence from the CF Worker policy (no same-origin auto-derive on AWS — the SPA and the wss endpoint are on different origins) are in §8.2. Unset WEB_ORIGINS disables the check; this is why the defence is documented as "mandatory-once-the-SPA-ships" rather than "mandatory always" — bare-IRC deployments without a web frontend (curl, WeeChat, tcp-ws-forwarder) never send Origin and are correctly served without the knob.

16.5 CF-vs-AWS: one origin vs two origins

The architectural difference is the reason this § exists:

Two practical consequences:

1. The baked SPA config (config.prod-aws.json) carries a templated host + stage path because the browser cannot infer either from window.location. The CF config (config.prod.json) uses Kiwi's {{hostname}} template resolved at load time and no direct_path (the Worker's WS upgrade is at root). The AWS config templates server = {{API_ID}}.execute-api.{{REGION}}.amazonaws.com (host only — irc-framework prepends wss:// from tls: true) and direct_path = /{{STAGE}} (the API Gateway stage segment). The AWS deploy pipeline splits the stack-output URL into these two fields at bake time (§16.3 step 2).
2. The CF Worker's CSWSH defence has a same-origin auto-derive mode (when WEB_ORIGINS is unset, compare Origin against the request's own Host). AWS cannot use auto-derive — the request's own host is the API Gateway domain, never the SPA's — so the AWS defence is explicit-allowlist only (§8.2).

16.6 Custom domain (ACM + CloudFront alias + Route53)

StaticSite accepts four optional props for the custom domain (see apps/aws-stack/src/static-site.ts):

When all four are supplied (the CDK context wiring in bin/aws.ts handles the optional spread), the construct adds the alias to the CloudFront distribution and creates a Route53 ARecord aliasing the distribution. When customDomain is omitted, the distribution serves on its default *.cloudfront.net URL — but see the gap note in §16.2 (the current bin/aws.ts wiring only provisions the construct at all when webSiteCustomDomain is set).

ACM certificate validation: the certificate must be DNS-validated and in ISSUED status before cdk deploy runs — CloudFront will not deploy an alias to a pending certificate. Create it ahead of time in us-east-1 via the console or:

aws acm request-certificate \
  --domain-name app.example.com \
  --validation-method DNS \
  --region us-east-1
# add the returned CNAME to Route53, wait for ISSUED

16.7 Verification (manual smoke check)

After a deploy-web job completes:

# Stack outputs
aws cloudformation describe-stacks --stack-name IrcAwsStack \
  --query 'Stacks[0].Outputs[?OutputKey==`WebsiteURL`].OutputValue' --output text

# Landing page
curl -sI https://<distribution-or-custom-domain>/ | head -1
# Landing page (the project front door — links to SPA + docs + source repo)
curl -sI https://<distribution-or-custom-domain>/ | head -1
# SPA shell (200, served via the 404→/webclient/index.html error mapping for /webclient/* routes)
curl -sI https://<distribution-or-custom-domain>/webclient/ | head -1
# Baked config — server MUST be the real wss URL, not a {{...}} template
curl -s https://<distribution-or-custom-domain>/webclient/static/config.json | jq .startupOptions.server

Open https://<distribution-or-custom-domain>/webclient/ in a browser, register, join #welcome, send a PRIVMSG, see the echo. (Visiting the root https://<distribution-or-custom-domain>/ serves the landing page, which links to the SPA at /webclient/.) If the SPA loads but never connects, check (1) the baked config.json server matches the stack's ConnectUrl output, (2) WEB_ORIGINS (§16.4) includes the CloudFront origin you're loading the SPA from, and (3) the $connect route is reachable.

17. OIDC deploy role (CI)

The GitHub Actions deploy workflow (§12) accepts only GitHub OIDC web-identity credentials. Long-lived access keys (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY) are NOT supported — a single exfiltration vector against them (rogue PR printenv, compromised third-party Action, poisoned runner) yields full AWS account takeover because the deploy role can manage CloudFormation, IAM, Lambda, and DynamoDB. This section documents the operator-side setup that the AWS_DEPLOY_ROLE_ARN repo secret points at.

17.1 One-time AWS account setup

1. Create the GitHub OIDC identity provider (one per AWS account, idempotent):

   aws iam create-open-id-connect-provider \
     --url https://token.actions.githubusercontent.com \
     --client-id-list sts.amazonaws.com \
     --thumbprint-list 6938fd4d98bab03faadb97b34396831e3780aea1 \
     --tags Key=managed-by,Value=github-actions
   

   The --thumbprint-list above is the well-known root CA thumbprint GitHub's OIDC tokens chain to. AWS now also accepts the trust-root automatically, but specifying it keeps the provider deterministic across accounts.

2. Create the deploy role with the trust policy below (save as trust-policy.json then create-role). The single condition key binds the role to this repository's main branch — the only ref the workflow_dispatch deploy workflow can run from:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/token.actions.githubusercontent.com"
         },
         "Action": "sts:AssumeRoleWithWebIdentity",
         "Condition": {
           "StringEquals": {
             "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
             "token.actions.githubusercontent.com:sub": "repo:<owner>/<repo>:ref:refs/heads/main"
           }
         }
       }
     ]
   }
   

   Replace <ACCOUNT_ID> with the AWS account id and <owner>/<repo> with the GitHub owner/repo (e.g. ServerlessIRCd/ServerlessIRCd). The sub condition is the security boundary: only workflows running on a ref matching refs/heads/main in this exact repo can assume the role. Do NOT widen it to StringLike with a wildcard — that lets forked PRs mint credentials.

   aws iam create-role \
     --role-name IrcDeployRole \
     --assume-role-policy-document file://trust-policy.json \
     --description "GitHub Actions OIDC role for IrcAwsStack deploys"
   
   # Note the returned ARN — this becomes AWS_DEPLOY_ROLE_ARN.
   aws iam get-role --role-name IrcDeployRole --query 'Role.Arn' --output text
   

3. Attach a permission policy. The role needs the same surface cdk deploy uses: IAM (pass-role + role creation), API Gateway v2, Lambda, DynamoDB, EventBridge, CloudFormation, S3 (the CDK bootstrap bucket), and logs. The CDK bootstrap already creates a cdk-hnb659fds-deploy-role-<account>-<region> role with the right shape — for a v1 deployment, the simplest pattern is to let the GitHub-OIDC role sts:AssumeRole the CDK deploy role:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "sts:AssumeRole",
         "Resource": "arn:aws:iam::<ACCOUNT_ID>:role/cdk-hnb659fds-deploy-role-<ACCOUNT_ID>-<REGION>"
       }
     ]
   }
   

   Attach this as an inline policy on IrcDeployRole. (If you would rather grant the deploy permissions directly on IrcDeployRole, mirror the statement list from the CDK deploy role's DefaultDeploymentRole.)

4. Store the role ARN as the AWS_DEPLOY_ROLE_ARN repo secret (Settings → Secrets and variables → Actions → New repository secret). This is the only AWS-shaped secret the workflow reads.

17.2 Condition key reference

The workflow's on: block is workflow_dispatch only, and a maintainer runs it from the main branch in the GitHub UI, so the sub claim always carries :ref:refs/heads/main. If you later add a push trigger on another branch, add a second condition entry rather than loosening this one.

17.3 CI verification (operator-run, out-of-band)

Once the role and secret are configured, verify end-to-end by running the workflow from the Actions UI against a sandbox account. The workflow's Assert assumed role matches AWS_DEPLOY_ROLE_ARN step prints the assumed role-name + account on success:

OIDC role assertion passed: IrcDeployRole in account 111122223333

If that step fails, the trust policy condition keys do not match the workflow's sub claim — re-check repo:<owner>/<repo> and the branch ref. The workflow aborts BEFORE cdk deploy, so a misconfigured trust policy cannot mutate the stack.

Authoritative reference: GitHub's "Configuring OpenID Connect in Amazon Web Services" guide (https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services) and the AWS "Creating OpenID Connect (OIDC) identity providers" guide walk through the IAM-side setup in more detail.

17.4 What the workflow enforces in-repo

The CI-hardening test suite (tools/ci-hardening/tests/deploy-aws-oidc.test.ts) asserts the workflow shape so the security property cannot regress silently:

• No aws-access-key-id / aws-secret-access-key inputs to configure-aws-credentials@v4 anywhere in the file.
• Top-level permissions: block grants exactly { id-token: write, contents: read } (least-privilege).
• Every configure-aws-credentials step has a role-to-assume input.
• The deploy and deploy-web jobs both have an if: env.AWS_DEPLOY_ROLE_ARN == '' guard that runs exit 1 before any AWS call.
• The deploy job runs aws sts get-caller-identity and references AWS_DEPLOY_ROLE_ARN before any cdk deploy (defence-in-depth against silent OIDC fallback).

The deploy-web job (TICKET-167, §16) inherits the same OIDC model: it has its own empty-ARN guard before configure-aws-credentials and assumes the same role via web identity — no static keys, no ambient credential inheritance from the parent deploy job. The two jobs are independent workflow_dispatch runs against the same role ARN.

18. Open questions / future work

These are facts the code does not yet have an answer for; the recommendations above are deployer guidance, not built-in knobs:

1. RemovalPolicy.DESTROY is hardcoded. There is no stack prop to flip it to RETAIN for production. Recommendation: a future change adds a production: boolean (or similar) construct prop that toggles the removal policy across all five tables.
2. SERVER_NAME / NETWORK_NAME / MOTD are hardcoded in aws-stack.ts. Resolved — these are now CDK construct props (serverName, networkName, motdLines) on IrcStackProps (see §7.1).
3. MOTD multiline parsing. Resolved — the stack's motdLines prop is a string[] joined with \n, which the config loader splits back into one line per entry. The delimiter is \n only; commas are treated as literal line content (see §7.1).
4. No per-environment stack parameter. Staging and production cannot coexist in the same account/region without forking the stack to parameterize names. Recommendation: separate accounts for staging vs production for v1.