Add SafeApply lifecycle, health tracking, convergence loop, and startup checks

SafeApply pattern (validate → backup → write → reload → verify → rollback):
- HAProxy: SafeApply wraps existing validate+write+reload with backup and
  post-reload health check; rolls back to .bak on failure
- dnsmasq: SafeApply validates via dnsmasq --test, backs up, atomic writes,
  restarts, verifies daemon is active; rolls back on failure
- nftables: SafeApply validates via nft -c, backs up, atomic writes, applies
  to kernel, verifies table loaded; rolls back on failure

Health tracking:
- Add SubsystemHealth and Health structs to reconciler
- Track per-subsystem status (ok/degraded/error) after each reconcile
- Detect recovery: previous error → current ok broadcasts recovery event
- GET /api/v1/health/reconcile endpoint exposes health state
- HAProxy tracks excluded domains as "degraded" state

SSE error events:
- Broadcast haproxy:error, dnsmasq:error on SafeApply failure
- Broadcast haproxy:recovered, dnsmasq:recovered on recovery from error

Convergence loop:
- 5-minute periodic reconcile drives system toward desired state
- Catches config drift, daemon crashes, transient failures
- Serialized by reconcile mutex — no race with event-driven reconciles

Startup:
- CheckPrerequisites verifies required (dnsmasq, haproxy) and optional
  (wg, authelia, cscli, cloudflared, nft) binaries before first reconcile
This commit is contained in:
2026-07-14 11:44:44 +00:00
parent 88acd437a1
commit a0ab8faa1c
7 changed files with 276 additions and 20 deletions

View File

@@ -160,9 +160,14 @@ func main() {
// Tell the API what port it's running on, register Central as a domain
// (if a domain is configured), and reconcile all networking.
api.SetPort(port)
api.CheckPrerequisites()
api.EnsureCentralDomain()
api.Reconcile()
// Start periodic convergence loop — continuously drives system toward
// desired state, recovering from daemon crashes and config drift.
api.StartConvergenceLoop(ctx, 5*time.Minute)
addr := fmt.Sprintf("%s:%d", host, port)
slog.Info("wild-central started", "addr", addr, "version", Version)