feat(verification): post-deploy verification suites and a provider-neutral runner

gmackie commented

2026-08-23 22:08:59 +00:00

Owner

What

Phase 8 of the one-box plan: post-deploy verification suites and the runner that executes them — the evidence layer the promotion gate reads from.

Suite definitions synced from a repository's forge-ci.toml ([verification.suites.*]) into versioned definitions. Sync runs only from a passed build: a red build's manifest may be mid-edit, and retiring suites on it would silently drop the promotion gate.
Runs, attempts, tests and evidence custody, so a retry creates a linked immutable attempt instead of mutating history.
Provider-neutral runner protocol — capability-matched claim and report, with Playwright and Maestro translating to one common contract rather than ForgeGraph becoming either framework's test runner.
Verification tRPC router, agent claim/report endpoints, and the pipeline graph surfacing verification state.

Stacked on #404

This builds directly on feat/one-box-traffic (#404) and should merge after it. Reviewing the two together is fine; the diff here is phase 8 only.

Notes

Migrations renumbered to 0096 (and the operator catch-up/verify scripts renamed to match) around the mise and worker-budget migrations that landed first.
The manifest runner now ships declared suites alongside the mise toolchain snapshot rather than replacing it — this patch predated the mise merge, so both were reconciled onto the same CI report.

Verification

go build ./... + go test ./... — green
tsc --noEmit — clean for apps/web, packages/api, packages/db
vitest run — 174 web files / 1105 tests, 140 api files / 985 tests, all passing

🤖 Generated with Claude Code

## What Phase 8 of the [one-box plan](https://kaih35i27lbc.postplan.dev): post-deploy verification suites and the runner that executes them — the evidence layer the promotion gate reads from. - **Suite definitions** synced from a repository's `forge-ci.toml` (`[verification.suites.*]`) into versioned definitions. Sync runs **only from a passed build**: a red build's manifest may be mid-edit, and retiring suites on it would silently drop the promotion gate. - **Runs, attempts, tests and evidence custody**, so a retry creates a linked immutable attempt instead of mutating history. - **Provider-neutral runner protocol** — capability-matched claim and report, with Playwright and Maestro translating to one common contract rather than ForgeGraph becoming either framework's test runner. - Verification tRPC router, agent claim/report endpoints, and the pipeline graph surfacing verification state. ## Stacked on #404 This builds directly on `feat/one-box-traffic` (#404) and should merge after it. Reviewing the two together is fine; the diff here is phase 8 only. ## Notes - Migrations renumbered to `0096` (and the operator catch-up/verify scripts renamed to match) around the mise and worker-budget migrations that landed first. - The manifest runner now ships declared suites **alongside** the mise toolchain snapshot rather than replacing it — this patch predated the mise merge, so both were reconciled onto the same CI report. ## Verification - `go build ./...` + `go test ./...` — green - `tsc --noEmit` — clean for `apps/web`, `packages/api`, `packages/db` - `vitest run` — 174 web files / 1105 tests, 140 api files / 985 tests, all passing 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gmackie added 2 commits

2026-08-23 22:08:59 +00:00

feat(traffic): one-box traffic shifting — weighted router in front of production

CI / gitleaks (pull_request) Successful in 7s

Details

forgegraph/ci CI passed

CI / ci (pull_request) Successful in 9m10s

Details

7afaa1a47a

Optional per-app "one-box": a second Cloudflare Worker on the production stage
(`<slug>-onebox`) sharing production's database, secrets and resources, behind
a ForgeGraph-managed router Worker. A production deploy lands on the one-box
first, shifts a configurable slice of real traffic to it, bakes for a window
while comparing it against production on the same traffic, then promotes the
primary and drains to zero — or drains to zero and fails, one write either way.

Apps that do not enable it are byte-for-byte unchanged.

Phases 1-7c of docs/plans/2026-08-21-one-box-traffic-shifting.html:

- Router Worker (service-binding and origin-URL variants), `traffic_splits`,
  the `traffic` tRPC router and `fg traffic`
- Deploy lifecycle: bake, probe, promote, abort, and a stuck-state sweeper
- Pipeline flow chart and traffic panel; live, animated promotion UI
- Node-platform lanes; `prod_canary` wired to `traffic_splits`
- OTel-sourced bake metrics (`fg.lane`) and first-class app pipelines
- Beta → production promotion path

Engineering review (E1-E7) applied before landing:

- The router no longer re-wraps WebSocket upgrades. `new Response(res.body,
  res)` rejects status 101 and drops the `webSocket` handle, so every upgrade
  through the router failed — including at 0% weight. The idle path is now a
  true passthrough that returns the upstream response untouched.
- The lane override is header-only. `?fg_lane=` was shareable, linkable and
  cacheable, so one posted link could push a crowd onto the canary and skew the
  bake verdict that drives auto-promote and auto-rollback. Forced requests are
  marked upstream (`x-fg-lane-forced`) so the verdict can exclude them once
  per-span lane attribution exists; today it still counts them.
- Both router variants are composed from one shared prelude instead of 74
  duplicated lines, so a fix lands once rather than four times.
- A bake that never saw `MIN_REQUESTS_FOR_VERDICT` one-box requests still
  promotes (a quiet app must not hang) but is now recorded and reported as
  promoted without traffic evidence, instead of reading as a clean bake.
- State-machine tests for promote, drain (including the KV-write-failure
  retry) and the stuck-state sweeper, which previously had no coverage.
- Regression tests for the `viaOneBox` deploy branch.
- Migrations renumbered to 0092-0095 around the mise and worker-budget
  migrations that landed first; the superseded `blue-green` router and its
  orphaned page deleted; the duplicate no-DB vitest config dropped in favour of
  `vitest.unit.config.ts`; the bake default reconciled to one authority.

Verification: go build + go test green; tsc clean for apps/web, packages/api
and packages/db; 174 web test files (1105 tests) and 138 api test files (974
tests) pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CMpzX1b6swjezEptw3T71f

feat(verification): post-deploy verification suites and a provider-neutral runner

CI / gitleaks (pull_request) Has been cancelled

Details

CI / ci (pull_request) Has been cancelled

Details

07637afbf0

Phase 8 of docs/plans/2026-08-21-one-box-traffic-shifting.html: the evidence
layer the promotion gate reads from.

- Suite definitions synced from a repository's `forge-ci.toml`
  (`[verification.suites.*]`), materialized as versioned definitions. Sync runs
  only from a passed build: a red build's manifest may be mid-edit, and
  retiring suites on it would silently drop the promotion gate.
- Runs, attempts, tests and evidence custody, so a retry creates a linked
  immutable attempt rather than mutating history.
- A provider-neutral runner protocol: capability-matched claim and report, with
  Playwright and Maestro translating to one common contract instead of
  ForgeGraph becoming either framework's test runner.
- Verification tRPC router, agent claim/report endpoints, and the pipeline
  graph surfacing verification state.

Migrations renumbered to 0096 (and the operator catch-up/verify scripts with
them) around the migrations that landed ahead of this stack.

The manifest runner now ships declared suites alongside the mise toolchain
snapshot rather than replacing it — both ride the same CI report.

Verification: go build + go test green; tsc clean for apps/web, packages/api
and packages/db; 174 web test files (1105 tests) and 140 api test files (985
tests) pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CMpzX1b6swjezEptw3T71f

gmackie added 1 commit

2026-08-23 22:10:33 +00:00

docs(plan): record the engineering review outcome and the residual bake-verdict gap

CI / gitleaks (pull_request) Successful in 6s

Details

CI / ci (pull_request) Has been cancelled

Details

d99bb637f5

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CMpzX1b6swjezEptw3T71f

gmackie added 1 commit

2026-08-23 22:11:28 +00:00

docs(todos): correct the migration-collision note

CI / gitleaks (pull_request) Successful in 6s

Details

forgegraph/ci CI passed

CI / ci (pull_request) Successful in 9m28s

Details

7f24154ae6

The a/b suffix files (0029a, 0030a, 0050a, 0062a, 0062b) are the deliberate
convention from #134, not violations. One genuine collision remains: two bare
0081_* migrations.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CMpzX1b6swjezEptw3T71f

gmackie force-pushed feat/release-verification from 7f24154ae6

CI / gitleaks (pull_request) Successful in 6s

Details

forgegraph/ci CI passed

CI / ci (pull_request) Successful in 9m28s

Details

to 11ff87d556

CI / gitleaks (pull_request) Successful in 7s

Details

forgegraph/ci CI passed

CI / ci (pull_request) Successful in 9m18s

Details