gmackie/ForgeGraph
1
0
Fork 0
Code Issues 8 Pull requests 10 Projects Releases Packages Wiki Activity Actions

fix(agent): bound local snapshot retention and never fill the disk #604

Open
gmackie wants to merge 1 commit from fix/snapshot-local-retention into main
pull from: fix/snapshot-local-retention
merge into: gmackie:main
Conversation 2 Commits 1 Files changed 5 +343 -16
gmackie commented 2026-09-27 22:46:06 +00:00
Owner
Copy link

fix(agent): bound local snapshot retention and never fill the disk

hetzner-master kept the full tiered snapshot retention (7 daily + weekly +
monthly) on its 75 GB root disk at ~11 GB/night. On 2026-09-21 the disk
filled mid-dump, wedged the ForgeGraph Postgres for days, and left a
truncated truecomps dump that the already-dumped-today check then refused
to redo.

  • snapshot.local_keep_days caps local copies; S3 keeps the tiers.
  • Prune local before dumping, not after.
  • Skip a dump when free space < 1.5x the previous dump (floor 1 GiB).
  • Write to .partial and rename on success; clean abandoned partials.

Rollout

After the next agent release, set "local_keep_days": 2 in hetzner-master /etc/forgegraph/agent.json under snapshot, then remove the stopgap /etc/cron.d/fg-snapshot-local-trim + /usr/local/bin/fg-snapshot-local-trim.sh.

Verification

  • go test ./internal/snapshot/ ./internal/agentcfg/ ./internal/cmdexec/: pass
  • Mutation control: disabling keep-days makes TestPruneLocalKeepDaysKeepsOnlyRecentDailies fail

🤖 Generated with Claude Code

fix(agent): bound local snapshot retention and never fill the disk hetzner-master kept the full tiered snapshot retention (7 daily + weekly + monthly) on its 75 GB root disk at ~11 GB/night. On 2026-09-21 the disk filled mid-dump, wedged the ForgeGraph Postgres for days, and left a truncated truecomps dump that the already-dumped-today check then refused to redo. - snapshot.local_keep_days caps local copies; S3 keeps the tiers. - Prune local before dumping, not after. - Skip a dump when free space < 1.5x the previous dump (floor 1 GiB). - Write to .partial and rename on success; clean abandoned partials. ## Rollout After the next agent release, set `"local_keep_days": 2` in hetzner-master `/etc/forgegraph/agent.json` under `snapshot`, then remove the stopgap `/etc/cron.d/fg-snapshot-local-trim` + `/usr/local/bin/fg-snapshot-local-trim.sh`. ## Verification - `go test ./internal/snapshot/ ./internal/agentcfg/ ./internal/cmdexec/`: pass - Mutation control: disabling keep-days makes `TestPruneLocalKeepDaysKeepsOnlyRecentDailies` fail 🤖 Generated with [Claude Code](https://claude.com/claude-code)
fix(agent): bound local snapshot retention and never fill the disk
All checks were successful
CI / gitleaks (pull_request) Successful in 7s
Details
CI / storybook (pull_request) Successful in 1m22s
Details
CI / web-build (pull_request) Successful in 3m27s
Details
forgegraph/ci CI passed
CI / ci (pull_request) Successful in 10m18s
Details
8e6b1088b8 
hetzner-master kept the full tiered snapshot retention (7 daily + weekly +
monthly) on its 75 GB root disk at ~11 GB/night. On 2026-09-21 the disk
filled mid-dump, wedged the ForgeGraph Postgres for days, and left a
truncated truecomps dump that the already-dumped-today check then refused
to redo.

- snapshot.local_keep_days caps local copies; S3 keeps the tiers.
- Prune local before dumping, not after.
- Skip a dump when free space < 1.5x the previous dump (floor 1 GiB).
- Write to .partial and rename on success; clean abandoned partials.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
gmackie commented 2026-09-27 22:53:08 +00:00
Author
Owner
Copy link

Preview environment is live: https://pr-604-forgegraph.forgegraf.com

Deployed 8e6b1088 with the beta stage's environment. It redeploys on every push and is destroyed when this PR closes.

Preview environment is live: https://pr-604-forgegraph.forgegraf.com Deployed `8e6b1088` with the beta stage's environment. It redeploys on every push and is destroyed when this PR closes.
gmackie commented 2026-09-27 22:53:09 +00:00
Author
Owner
Copy link

Preview environment is live: https://pr-604-forgegraph.forgegraf.com

Deployed 8e6b1088 with the beta stage's environment. It redeploys on every push and is destroyed when this PR closes.

Preview environment is live: https://pr-604-forgegraph.forgegraf.com Deployed `8e6b1088` with the beta stage's environment. It redeploys on every push and is destroyed when this PR closes.
All checks were successful
CI / gitleaks (pull_request) Successful in 7s
Required
Details
CI / storybook (pull_request) Successful in 1m22s
Details
CI / web-build (pull_request) Successful in 3m27s
Details
forgegraph/ci CI passed
CI / ci (pull_request) Successful in 10m18s
Required
Details
This pull request has changes conflicting with the target branch.
  • CHANGELOG.md
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/snapshot-local-retention:fix/snapshot-local-retention
git switch fix/snapshot-local-retention
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
community
Community process, governance, contribution UX

  
docs
Documentation improvements

  
forgegraph:public-update
Publish approved PR lifecycle updates to the ForgeGraph public feed.

  
good first issue
Suitable for first-time contributors

  
help wanted
Extra attention is needed; maintainers welcome a PR

No labels
community
docs
forgegraph:public-update
good first issue
help wanted
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
gmackie/ForgeGraph!604
No description provided.