Tag push to Forgejo rewound main and deployed an old release (gmackie/forge, 2026-10-05) #636

New issue

Open

opened 2026-10-05 21:40:32 +00:00 by gmackie · 1 comment

gmackie commented

2026-10-05 21:40:32 +00:00

Owner

On 2026-10-05, pushing two existing release tags to gmackie/forge on Forgejo moved its main branch backwards and redeployed production (forge-console → forge.gmac.io) from an old release.

Timeline (Forgejo activity feed, UTC)

time	actor	event
21:16:21	gmackie	push `main` → `0d1d851d` (GitHub Actions mirror, non-force)
21:16:22–23	gmackie	push tags `v0.3.0` (`45cfc3df`), `v0.4.0` (`f586db59`)
21:16:43	(no user)	`commit_repo refs/heads/main` head=`45cfc3df`; `push_tag v0.3.0`
21:17:04	(no user)	`commit_repo refs/heads/main` head=`f586db59`

Then ForgeGraph:

21:16:23 deploy of 0d1d851 → failed (git checkout 0d1d851…: reference is not a tree on node bf11e7b1).
21:17:06 deploy of f586db5 (= v0.4.0, 2026-09-24) → active. Production ran the September release.
forge deploy create production --ref main then resolved main to f586db5; --ref <0d1d851 sha> failed with the same "not a tree".
forge deploy rollback 4d6d9ddf… failed with cloudflare_worker_deploy_failed.

The next non-force push to main (f586db5..59bcc38, 21:35:58) proves main really pointed at f586db5. That push deployed normally and restored production.

Expected

A tag push never moves a branch, and never deploys an older commit over the tracked branch.

Notes

The no-user actor suggests a server-side process. agent/internal/cmdexec/git_bridge_sync.go pushes ref:ref without force, so it should not be able to rewind main. Whatever wrote the two refs/heads/main events at 21:16:43 and 21:17:04 did force it.
Mitigation on the forge side: the GitHub mirror no longer pushes tags (gmackie/forgec#215).
The rollback failure (cloudflare_worker_deploy_failed) may be a separate problem.

Filed by Claude Code on behalf of @gmackie.

On 2026-10-05, pushing two existing release tags to `gmackie/forge` on Forgejo moved its `main` branch **backwards** and redeployed production (`forge-console` → forge.gmac.io) from an old release. ## Timeline (Forgejo activity feed, UTC) | time | actor | event | |---|---|---| | 21:16:21 | gmackie | push `main` → `0d1d851d` (GitHub Actions mirror, non-force) | | 21:16:22–23 | gmackie | push tags `v0.3.0` (`45cfc3df`), `v0.4.0` (`f586db59`) | | 21:16:43 | **(no user)** | `commit_repo refs/heads/main` head=`45cfc3df`; `push_tag v0.3.0` | | 21:17:04 | **(no user)** | `commit_repo refs/heads/main` head=`f586db59` | Then ForgeGraph: - 21:16:23 deploy of `0d1d851` → failed (`git checkout 0d1d851…: reference is not a tree` on node bf11e7b1). - 21:17:06 deploy of `f586db5` (= v0.4.0, 2026-09-24) → **active**. Production ran the September release. - `forge deploy create production --ref main` then resolved `main` to `f586db5`; `--ref <0d1d851 sha>` failed with the same "not a tree". - `forge deploy rollback 4d6d9ddf…` failed with `cloudflare_worker_deploy_failed`. The next non-force push to `main` (`f586db5..59bcc38`, 21:35:58) proves `main` really pointed at `f586db5`. That push deployed normally and restored production. ## Expected A tag push never moves a branch, and never deploys an older commit over the tracked branch. ## Notes - The no-user actor suggests a server-side process. `agent/internal/cmdexec/git_bridge_sync.go` pushes `ref:ref` without force, so it should not be able to rewind `main`. Whatever wrote the two `refs/heads/main` events at 21:16:43 and 21:17:04 did force it. - Mitigation on the forge side: the GitHub mirror no longer pushes tags (gmackie/forgec#215). - The rollback failure (`cloudflare_worker_deploy_failed`) may be a separate problem. Filed by Claude Code on behalf of @gmackie.

gmackie commented

2026-10-06 00:16:48 +00:00

Author

Owner

Narrowing, from reading the code (not yet the cause):

Ruled out as the rewind:

agent/internal/cmdexec/git_bridge_sync.go fetches +ref:ref into a scratch bare repo and pushes only ref:ref without force. A tag sync cannot move refs/heads/main, and a backward move of main would be rejected as non-fast-forward.
apps/web/src/app/api/webhooks/github/route.ts (GitHub→Forgejo) dispatches that same job.
gmackie/forge has no Forgejo push mirrors. Its only webhook is https://forgegraf.com/api/fg/webhooks/forgejo (push, pull_request events).

Separate bug found on the way: apps/web/src/app/api/fg/webhooks/forgejo/handlers/push.ts:165 computes branch = payload.ref.replace(/^refs\/heads\//, ""), so a tag push is handled as a "branch" named refs/tags/v0.4.0: changeset upsert, and the default-branch/auto-deploy logic below it. That is plausibly how the tag pushes produced the 21:16:43 and 21:17:04 deploy activity. Tag pushes should probably skip changeset and auto-deploy handling entirely, apart from the GitHub bridge.

Still unexplained: what wrote refs/heads/main at 21:16:43 (to 45cfc3df) and 21:17:04 (to f586db59) with no acting user. Next places to look: the deploy agent's handling of checkout/update-ref for a tag ref on node bf11e7b1, and anything that pushes a node's working clone back to Forgejo. Forgejo's server log for those two timestamps would name the pusher.

Narrowing, from reading the code (not yet the cause): **Ruled out as the rewind:** - `agent/internal/cmdexec/git_bridge_sync.go` fetches `+ref:ref` into a scratch bare repo and pushes only `ref:ref` without force. A tag sync cannot move `refs/heads/main`, and a backward move of `main` would be rejected as non-fast-forward. - `apps/web/src/app/api/webhooks/github/route.ts` (GitHub→Forgejo) dispatches that same job. - `gmackie/forge` has no Forgejo push mirrors. Its only webhook is `https://forgegraf.com/api/fg/webhooks/forgejo` (push, pull_request events). **Separate bug found on the way:** `apps/web/src/app/api/fg/webhooks/forgejo/handlers/push.ts:165` computes `branch = payload.ref.replace(/^refs\/heads\//, "")`, so a tag push is handled as a "branch" named `refs/tags/v0.4.0`: changeset upsert, and the default-branch/auto-deploy logic below it. That is plausibly how the tag pushes produced the 21:16:43 and 21:17:04 deploy activity. Tag pushes should probably skip changeset and auto-deploy handling entirely, apart from the GitHub bridge. **Still unexplained:** what wrote `refs/heads/main` at 21:16:43 (to `45cfc3df`) and 21:17:04 (to `f586db59`) with no acting user. Next places to look: the deploy agent's handling of `checkout`/`update-ref` for a tag ref on node `bf11e7b1`, and anything that pushes a node's working clone back to Forgejo. Forgejo's server log for those two timestamps would name the pusher.