fix(alerts): clear ci_failed once its changeset settles #502

Merged

gmackie merged 1 commit from fix/settled-changeset-alerts into main

2026-08-27 21:16:28 +00:00

gmackie commented

2026-08-27 20:51:39 +00:00

Owner

Closes the loop the changeset reconciler opened.

A ci_failed alert is keyed ci_failed:<changesetId>. When the changeset merges or closes, the alert is describing a branch that no longer exists — but nothing retires it. The only code that resolves these keys is POST /api/fg/monitoring/evaluate, and nothing calls that endpoint: its only references are its own tests. Its own comment records the result — "that is how 222 ci_failed alerts accumulated on the live instance, 118 of them for changesets already merged or closed."

#489/#490 made this sharper: they retired 372 changesets (698 → 326 active), and every one of those is a candidate for an alert with nothing left to clear it. The reconcile would have traded a stale changeset for a stale alert.

Resolving them is a narrow question — is this alert's changeset still active — so this answers exactly that on each poll, beside the reconciler that creates the condition, rather than running the whole evaluation on the agent hot path.

Deliberately conservative: it only ever resolves, never fires, and only for changesets that have left draft/open/changes_requested/approved. Checked against the live instance, 8 of the 11 firing ci_failed alerts are on branches genuinely 2–8 commits ahead of main with genuinely red CI — those must keep alerting, and this leaves them alone. The other 3 point at deleted branches and clear once the reconciler reaches them.

Bounded at 50 per pass so a backlog drains over polls. 5 tests; turbo run typecheck clean.

🤖 Generated with Claude Code

Closes the loop the changeset reconciler opened. A `ci_failed` alert is keyed `ci_failed:<changesetId>`. When the changeset merges or closes, the alert is describing a branch that no longer exists — but **nothing retires it**. The only code that resolves these keys is `POST /api/fg/monitoring/evaluate`, and nothing calls that endpoint: its only references are its own tests. Its own comment records the result — *"that is how 222 ci_failed alerts accumulated on the live instance, 118 of them for changesets already merged or closed."* #489/#490 made this sharper: they retired **372 changesets** (698 → 326 active), and every one of those is a candidate for an alert with nothing left to clear it. The reconcile would have traded a stale changeset for a stale alert. Resolving them is a narrow question — *is this alert's changeset still active* — so this answers exactly that on each poll, beside the reconciler that creates the condition, rather than running the whole evaluation on the agent hot path. Deliberately conservative: it only ever **resolves**, never fires, and only for changesets that have left `draft/open/changes_requested/approved`. Checked against the live instance, 8 of the 11 firing `ci_failed` alerts are on branches genuinely 2–8 commits ahead of main with genuinely red CI — those must keep alerting, and this leaves them alone. The other 3 point at deleted branches and clear once the reconciler reaches them. Bounded at 50 per pass so a backlog drains over polls. 5 tests; `turbo run typecheck` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gmackie added 1 commit

2026-08-27 20:51:39 +00:00

fix(alerts): clear ci_failed once its changeset settles

CI / gitleaks (pull_request) Successful in 8s

Details

CI / storybook (pull_request) Successful in 1m38s

Details

forgegraph/ci CI passed

CI / ci (pull_request) Successful in 11m51s

Details

f9cdc51db6

The reconciler retires a changeset whose branch has landed, but the ci_failed
alert keyed on that changeset keeps firing: the only thing that resolves it is
POST /api/fg/monitoring/evaluate, and nothing calls that endpoint -- its only
references are its own tests. So closing a changeset silently left its alert
behind, which is how 118 alerts for already-merged changesets accumulated
before, and it would have happened again to every record the reconciler just
drained.

Resolving them is a narrow question -- is this alert's changeset still active
-- so this answers exactly that on each poll, next to the reconciler that
creates the condition, rather than running the whole evaluation.

gmackie commented

2026-08-27 21:00:41 +00:00

Author

Owner

Preview environment is live: https://pr-502-forgegraph.forgegraf.com

Deployed f9cdc51d with the beta stage's environment. It redeploys on every push and is destroyed when this PR closes.

Preview environment is live: https://pr-502-forgegraph.forgegraf.com Deployed `f9cdc51d` with the beta stage's environment. It redeploys on every push and is destroyed when this PR closes.

gmackie merged commit 9a2e9dd5e9 into main

2026-08-27 21:16:28 +00:00

gmackie referenced this pull request from a commit

2026-08-27 21:16:28 +00:00

Merge pull request 'fix(alerts): clear ci_failed once its changeset settles' (#502) from fix/settled-changeset-alerts into main

gmackie referenced this pull request

2026-08-27 21:31:54 +00:00

perf(alerts): look changesets up by id instead of hashing the whole table #506