#3 - feat(ci): CI & runner monitoring — capacity view, live Actions CI, thin alerting - gmackie/ForgeGraph

gmackie commented

2026-06-05 05:03:05 +00:00

Owner

Implements docs/plans/2026-06-05-ci-runner-monitoring. Phase 1 runner capacity view, Phase 2 live Forgejo Actions CI, Phase 3 thin alerting + truthful deploy status. Full pnpm typecheck (10/10) + 94 web tests + go tests green.

gmackie added 15 commits

2026-06-05 05:03:06 +00:00

fix(webhooks): case-insensitive repo lookup so the ForgeGraph repo's own Forgejo webhooks stop dropping as unknown_repo 5d170902f3

docs(plans): CI & runner monitoring design + implementation plan (2026-06-05) 25c0845f98

feat(ci): runner capacity verdict helper 2b0c504bb1

feat(ci): ci_runners schema + agent runner-report ingest endpoint 391cd9422d

feat(ci): runnerCapacity tRPC query + pure combine helper 3c0247e2b1

feat(ci): agent reports Forgejo runner status from local forgejo DB c09c364195

Reads action_runner via psql (self-gating on non-forgejo nodes), derives
online/offline from last_online staleness, POSTs to the runner-report
endpoint. Wired into the agent run cycle on a 60s cadence.
Cross-compile + deploy to hetzner-fg is a separate ops step.

feat(ci): rework runner page — group by host node with CPU/RAM + capacity verdict 01495f2903

feat(ci): agent streams live Forgejo Actions job-status + logs 5b7bb4b0fd

Polls action_run/action_run_job and tails action_task .log files every 2.5s,
pushing per-job status + new log lines. NOTE: verified against a real log on
hetzner-fg — the format is plain newline-delimited '<RFC3339Nano> <content>'
text, NOT the length-prefixed format the plan assumed. Confirmed the live
Forgejo DB is 'forgejo'. Agent POSTs progress to the server (relay in 2.2).
Cross-compile + deploy to hetzner-fg is a separate ops step.

feat(ci): relay live Actions progress to the pipeline stream e802b22847

New POST /api/fg/ci/actions/progress maps the agent's commitSha -> changeset ->
app and broadcasts an ephemeral pipeline_event (no DB row) onto the app's hub
channel, reusing use-pipeline-stream. Maps via commit (not build) because the
workflow_run webhook only lands a build at completion.

feat(ci): live Forgejo Actions panel in the app pipeline tab a3a7c2283a

Subscribes via use-pipeline-stream; renders per-job status chips
(pending->running->green) and an auto-tailing log pane fed by the relayed
ci-actions progress events. Live runs surface here (no build row exists until
the workflow_run webhook lands at completion).

feat(alerts): add runner_offline, webhook_delivery_failed, ci_failed alert types 1eb233cea4

feat(alerts): fire/resolve helper with dedupe + dedupe_key column 1e6812e0c3

fireAlert dedupes on a firing dedupe_key (one row + one notify per transition);
resolveAlert flips firing rows resolved. Adds alerts.dedupe_key column. Extends
the alerts-client AlertType union + label map for the new types.

feat(alerts): monitoring evaluate sweep + webhook-status reporting 95a10d3a3b

Adds ci_webhook_status table; agent now reports Forgejo webhook last_status
alongside runners. POST /api/fg/ci/monitoring/evaluate evaluates runner_offline
/ webhook_delivery_failed / ci_failed into fire/resolve via a pure tested
evaluator. Scheduler: documented systemd timer on hetzner-master (CF cron
fallback noted) per the plan.

feat(alerts): notify fan-out — mobile push + Resend email on fire 77610ac281

notifyAlertFired now resolves the alert's node workspace, sends push, and emails
workspace members via a new lib/email.ts (no-ops with a warn when RESEND_API_KEY
is unset — it currently is). Respects alert_rules: a type whose rules are all
disabled is suppressed. Fires once per transition (dedupe in fireAlert).

fix(deploy): truthful deploy status — secret-sync no longer masks a healthy deploy

AI Code Review / review (pull_request) Failing after 9s

Details

CI / ci (pull_request) Failing after 6m18s

Details

d09659d6f8

deploy.yml: secret-sync is now continue-on-error; an explicit success signal is
POSTed to /api/fg/deploys/complete after wrangler deploy + health check, and a
failure signal only when the deploy/health steps (not secret-sync) fail. New
endpoint resolves/fires the deploy_failed alert accordingly.
Note: requires an FG_API_TOKEN Forgejo secret (curl is non-blocking if unset).

gmackie merged commit 0f988d8009 into main

2026-06-05 05:15:12 +00:00

gmackie referenced this pull request from a commit

2026-06-05 05:15:12 +00:00

feat(ci): CI & runner monitoring — capacity view, live Actions CI, thin alerting (#3)

gmackie referenced this pull request

2026-06-05 05:47:17 +00:00

fix(alerts): scope ci_failed to active changesets #4

Rows
Columns

feat(ci): CI & runner monitoring — capacity view, live Actions CI, thin alerting #3