gmackie/ForgeGraph
1
0
Fork 0
Code Issues 8 Pull requests 10 Projects Releases Packages Wiki Activity Actions

infra(signoz): move to Foundry, upgrade to v0.141.1; ship node-app journald logs #579

Open
gmackie wants to merge 1 commit from infra/signoz-foundry-upgrade into main
pull from: infra/signoz-foundry-upgrade
merge into: gmackie:main
Conversation 0 Commits 1 Files changed 15 +384 -334
gmackie commented 2026-09-15 20:51:59 +00:00
Owner
Copy link

What

  • infra/signoz/: replaces the hand-written compose stack (SigNoz 0.55, ClickHouse 24.11) with a Foundry casting pinned to SigNoz v0.141.1 / collector v0.144.10 / ClickHouse 25.12.5. install.sh now forges and applies; README.md is the runbook. Already live on hetzner-fg since 2026-09-15 20:40 UTC — this PR makes the repo match the host.
  • Agent: optional journald receiver in the node collector template (journaldUnits param on install-otel-collector / configure-otel-collector), plus the web route passes it through. configure-otel-collector now starts from DefaultConfig instead of rendering 0.0.0.0:0 ports.

Why logs never worked

A signoz-schema-migrator:latest run moved the logs tables ahead of collector 0.111.34 (expected 5 arguments, got 3 on every batch with attributes), and migration 1011 needed ClickHouse 25.12 and had been failing since 09-10.

Deviations from Foundry defaults (documented in the casting)

  • Ingester runs on its file config, not OpAMP: managed mode starts with a no-op pipeline until the API server pushes one, and the server refuses until first-run setup. Ingestion must not depend on the API server.
  • memory_limiter + mem_limit: 2g kept on the ingester (2026-08 OOM incident).

Verification

  • go test ./internal/otelcollector/ ./internal/cmdexec/ green; rendered template validates on otelcol-contrib 0.102.1.
  • Post-cutover: traces from all services, metrics, journald logs from hetzner-master/hetzner-bob, and a POST to https://otlp.forgegraf.com/v1/logs all land in ClickHouse 25.12.5.

🤖 Generated with Claude Code

## What - `infra/signoz/`: replaces the hand-written compose stack (SigNoz 0.55, ClickHouse 24.11) with a Foundry casting pinned to SigNoz v0.141.1 / collector v0.144.10 / ClickHouse 25.12.5. `install.sh` now forges and applies; `README.md` is the runbook. **Already live on hetzner-fg since 2026-09-15 20:40 UTC** — this PR makes the repo match the host. - Agent: optional `journald` receiver in the node collector template (`journaldUnits` param on `install-otel-collector` / `configure-otel-collector`), plus the web route passes it through. `configure-otel-collector` now starts from `DefaultConfig` instead of rendering `0.0.0.0:0` ports. ## Why logs never worked A `signoz-schema-migrator:latest` run moved the logs tables ahead of collector 0.111.34 (`expected 5 arguments, got 3` on every batch with attributes), and migration 1011 needed ClickHouse 25.12 and had been failing since 09-10. ## Deviations from Foundry defaults (documented in the casting) - Ingester runs on its file config, not OpAMP: managed mode starts with a no-op pipeline until the API server pushes one, and the server refuses until first-run setup. Ingestion must not depend on the API server. - `memory_limiter` + `mem_limit: 2g` kept on the ingester (2026-08 OOM incident). ## Verification - `go test ./internal/otelcollector/ ./internal/cmdexec/` green; rendered template validates on otelcol-contrib 0.102.1. - Post-cutover: traces from all services, metrics, journald logs from hetzner-master/hetzner-bob, and a POST to `https://otlp.forgegraf.com/v1/logs` all land in ClickHouse 25.12.5. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
infra(signoz): move to Foundry, upgrade to v0.141.1; ship node-app journald logs
All checks were successful
CI / gitleaks (pull_request) Successful in 6s
Details
CI / storybook (pull_request) Successful in 1m21s
Details
CI / web-build (pull_request) Successful in 3m3s
Details
forgegraph/ci CI passed
CI / ci (pull_request) Successful in 10m57s
Details
7d53baaf0f 
SigNoz on hetzner-fg was 0.55.0 (2024) on hand-written compose files that
upstream no longer distributes. It is now generated by Foundry from
infra/signoz/casting.yaml: SigNoz v0.141.1, collector v0.144.10,
ClickHouse 25.12.5, keeper split out, SQLite metastore.

Why logs never worked: a `signoz-schema-migrator:latest` run had moved the
logs tables ahead of collector 0.111.34, so every batch with attributes
failed ("expected 5 arguments, got 3"), and a later migration (1011) needed
ClickHouse 25.12 and had been failing since 09-10.

Two deliberate deviations from Foundry defaults, both documented in the
casting: the ingester runs on its file config instead of OpAMP (managed
mode starts with a no-op pipeline until the API server, which refuses
before first-run setup, pushes one), and memory_limiter + mem_limit stay
on the ingester (2026-08 OOM incident).

The 0.55 SQLite metastore cannot be migrated directly, so the metastore
starts fresh; the old file and a ClickHouse volume snapshot are kept
(infra/signoz/README.md).

Node side: the agent's collector template gains an optional journald
receiver (units from the `journaldUnits` param of install/configure) that
ships unit stdout as logs with service.name = unit, severity from PRIORITY,
and @forgegraph/log JSON parsed for level/traceId/spanId. configure-otel-
collector also now starts from DefaultConfig instead of rendering ":0"
ports. hetzner-master and hetzner-bob were hand-configured the same way
today; a dispatch with journaldUnits reproduces it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
All checks were successful
CI / gitleaks (pull_request) Successful in 6s
Required
Details
CI / storybook (pull_request) Successful in 1m21s
Details
CI / web-build (pull_request) Successful in 3m3s
Details
forgegraph/ci CI passed
CI / ci (pull_request) Successful in 10m57s
Required
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin infra/signoz-foundry-upgrade:infra/signoz-foundry-upgrade
git switch infra/signoz-foundry-upgrade
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
community
Community process, governance, contribution UX

  
docs
Documentation improvements

  
forgegraph:public-update
Publish approved PR lifecycle updates to the ForgeGraph public feed.

  
good first issue
Suitable for first-time contributors

  
help wanted
Extra attention is needed; maintainers welcome a PR

No labels
community
docs
forgegraph:public-update
good first issue
help wanted
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
gmackie/ForgeGraph!579
No description provided.