feat(traffic): one-box traffic shifting — weighted router in front of production #404

Merged

gmackie merged 1 commit from feat/one-box-traffic into main

2026-08-23 22:28:52 +00:00

gmackie commented

2026-08-23 22:01:33 +00:00

Owner

What

Optional per-app one-box: a second Cloudflare Worker on the production stage (<slug>-onebox) sharing production's database, secrets and resources, behind a ForgeGraph-managed router Worker. A production deploy lands on the one-box first, shifts a configurable slice of real traffic to it, bakes while comparing it against production on the same traffic, then promotes the primary and drains to zero — or drains to zero and fails.

Apps that do not enable it are byte-for-byte unchanged.

Phases 1–7c of the plan. Phase 8 (verification suites) follows in a separate PR.

Engineering review

/plan-eng-review ran against this stack and found 6 issues, 2 of them critical gaps (silent failure, no test, no handling). All were fixed before landing — details in the plan's "Engineering review findings" section. The two that matter most:

WebSocket upgrades were broken at every weight, including 0%. withLaneHeader did new Response(res.body, res), which rejects status 101 and drops the webSocket handle. Enabling the router at all would have broken any WS app. The idle path is now a true passthrough returning the upstream response untouched — which is what the plan claimed it already was.
A bake could promote having proved nothing. Below MIN_REQUESTS_FOR_VERDICT (100) requests, every metrics tick returns insufficient-data with ok: true, so a quiet app auto-promoted on health-probe evidence alone and reported a clean bake. It still promotes (a low-traffic app must not hang forever) but is now recorded and surfaced as promoted without traffic evidence.

Known gap

The lane override's query-param form is gone, and forced requests are marked upstream with x-fg-lane-forced. Excluding that traffic from the bake verdict is not done: fg.lane is a resource attribute fixed at deploy, so exclusion needs a per-span attribute in @forgegraph/otel plus a ClickHouse filter, and only takes effect once apps upgrade. Today's verdict still counts header-forced traffic. Filed rather than half-built.

Verification

go build ./... + go test ./... — green
tsc --noEmit — clean for apps/web, packages/api, packages/db
vitest run — 174 web files / 1105 tests, 138 api files / 974 tests, all passing

Migrations renumbered to 0092–0095 around the mise (#399) and worker-budget (#402) migrations that landed first.

🤖 Generated with Claude Code

## What Optional per-app **one-box**: a second Cloudflare Worker on the production stage (`<slug>-onebox`) sharing production's database, secrets and resources, behind a ForgeGraph-managed router Worker. A production deploy lands on the one-box first, shifts a configurable slice of real traffic to it, bakes while comparing it against production on the same traffic, then promotes the primary and drains to zero — or drains to zero and fails. Apps that do not enable it are byte-for-byte unchanged. Phases 1–7c of the [plan](https://kaih35i27lbc.postplan.dev). Phase 8 (verification suites) follows in a separate PR. ## Engineering review `/plan-eng-review` ran against this stack and found 6 issues, 2 of them critical gaps (silent failure, no test, no handling). All were fixed before landing — details in the plan's "Engineering review findings" section. The two that matter most: - **WebSocket upgrades were broken at every weight, including 0%.** `withLaneHeader` did `new Response(res.body, res)`, which rejects status 101 and drops the `webSocket` handle. Enabling the router at all would have broken any WS app. The idle path is now a true passthrough returning the upstream response untouched — which is what the plan claimed it already was. - **A bake could promote having proved nothing.** Below `MIN_REQUESTS_FOR_VERDICT` (100) requests, every metrics tick returns `insufficient-data` with `ok: true`, so a quiet app auto-promoted on health-probe evidence alone and reported a clean bake. It still promotes (a low-traffic app must not hang forever) but is now recorded and surfaced as *promoted without traffic evidence*. ## Known gap The lane override's query-param form is gone, and forced requests are marked upstream with `x-fg-lane-forced`. Excluding that traffic from the bake verdict is **not** done: `fg.lane` is a resource attribute fixed at deploy, so exclusion needs a per-span attribute in `@forgegraph/otel` plus a ClickHouse filter, and only takes effect once apps upgrade. Today's verdict still counts header-forced traffic. Filed rather than half-built. ## Verification - `go build ./...` + `go test ./...` — green - `tsc --noEmit` — clean for `apps/web`, `packages/api`, `packages/db` - `vitest run` — 174 web files / 1105 tests, 138 api files / 974 tests, all passing Migrations renumbered to 0092–0095 around the mise (#399) and worker-budget (#402) migrations that landed first. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gmackie added 1 commit

2026-08-23 22:01:33 +00:00

feat(traffic): one-box traffic shifting — weighted router in front of production

CI / gitleaks (pull_request) Successful in 7s

Details

forgegraph/ci CI passed

CI / ci (pull_request) Successful in 9m10s

Details

7afaa1a47a

Optional per-app "one-box": a second Cloudflare Worker on the production stage
(`<slug>-onebox`) sharing production's database, secrets and resources, behind
a ForgeGraph-managed router Worker. A production deploy lands on the one-box
first, shifts a configurable slice of real traffic to it, bakes for a window
while comparing it against production on the same traffic, then promotes the
primary and drains to zero — or drains to zero and fails, one write either way.

Apps that do not enable it are byte-for-byte unchanged.

Phases 1-7c of docs/plans/2026-08-21-one-box-traffic-shifting.html:

- Router Worker (service-binding and origin-URL variants), `traffic_splits`,
  the `traffic` tRPC router and `fg traffic`
- Deploy lifecycle: bake, probe, promote, abort, and a stuck-state sweeper
- Pipeline flow chart and traffic panel; live, animated promotion UI
- Node-platform lanes; `prod_canary` wired to `traffic_splits`
- OTel-sourced bake metrics (`fg.lane`) and first-class app pipelines
- Beta → production promotion path

Engineering review (E1-E7) applied before landing:

- The router no longer re-wraps WebSocket upgrades. `new Response(res.body,
  res)` rejects status 101 and drops the `webSocket` handle, so every upgrade
  through the router failed — including at 0% weight. The idle path is now a
  true passthrough that returns the upstream response untouched.
- The lane override is header-only. `?fg_lane=` was shareable, linkable and
  cacheable, so one posted link could push a crowd onto the canary and skew the
  bake verdict that drives auto-promote and auto-rollback. Forced requests are
  marked upstream (`x-fg-lane-forced`) so the verdict can exclude them once
  per-span lane attribution exists; today it still counts them.
- Both router variants are composed from one shared prelude instead of 74
  duplicated lines, so a fix lands once rather than four times.
- A bake that never saw `MIN_REQUESTS_FOR_VERDICT` one-box requests still
  promotes (a quiet app must not hang) but is now recorded and reported as
  promoted without traffic evidence, instead of reading as a clean bake.
- State-machine tests for promote, drain (including the KV-write-failure
  retry) and the stuck-state sweeper, which previously had no coverage.
- Regression tests for the `viaOneBox` deploy branch.
- Migrations renumbered to 0092-0095 around the mise and worker-budget
  migrations that landed first; the superseded `blue-green` router and its
  orphaned page deleted; the duplicate no-DB vitest config dropped in favour of
  `vitest.unit.config.ts`; the bake default reconciled to one authority.

Verification: go build + go test green; tsc clean for apps/web, packages/api
and packages/db; 174 web test files (1105 tests) and 138 api test files (974
tests) pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CMpzX1b6swjezEptw3T71f

gmackie referenced this pull request

2026-08-23 22:08:59 +00:00

feat(verification): post-deploy verification suites and a provider-neutral runner #405

gmackie merged commit 798d4da502 into main

2026-08-23 22:28:52 +00:00

gmackie referenced this pull request from a commit

2026-08-23 22:28:54 +00:00

feat(traffic): one-box traffic shifting — weighted router in front of production (#404)

gmackie referenced this pull request from a commit

2026-08-25 03:35:23 +00:00

feat(storybook): stories for one-box traffic, the pipeline graph and CPU budgets

gmackie referenced this pull request