The auto-provisioner was checking for individual cert files (e.g.,
git.civilsociety.dev.pem) without checking if a wildcard cert already
covers the domain (*.civilsociety.dev.pem). This caused redundant
individual certs to be provisioned for domains already covered by
a wildcard.
Now uses hasCertForDomain() which checks both individual AND wildcard
cert coverage before deciding a cert is missing.
Replace the passive "log warning about missing certs" approach with active
auto-provisioning:
- When Cloudflare token and operator email are configured, automatically
provision missing certs via certbot DNS-01 during reconciliation
- When credentials aren't available, log actionable guidance (no longer
references a non-existent "Certificates page")
- Track TLS health in reconciler Health struct (ok/degraded with missing
cert list)
- Broadcast tls:recovered SSE event when all certs become available
- Add CertManager interface and GetCloudflareToken callback to reconciler
The convergence loop (5 min) continuously retries failed provisions,
so transient DNS-01 failures self-heal on the next tick.
SafeApply pattern (validate → backup → write → reload → verify → rollback):
- HAProxy: SafeApply wraps existing validate+write+reload with backup and
post-reload health check; rolls back to .bak on failure
- dnsmasq: SafeApply validates via dnsmasq --test, backs up, atomic writes,
restarts, verifies daemon is active; rolls back on failure
- nftables: SafeApply validates via nft -c, backs up, atomic writes, applies
to kernel, verifies table loaded; rolls back on failure
Health tracking:
- Add SubsystemHealth and Health structs to reconciler
- Track per-subsystem status (ok/degraded/error) after each reconcile
- Detect recovery: previous error → current ok broadcasts recovery event
- GET /api/v1/health/reconcile endpoint exposes health state
- HAProxy tracks excluded domains as "degraded" state
SSE error events:
- Broadcast haproxy:error, dnsmasq:error on SafeApply failure
- Broadcast haproxy:recovered, dnsmasq:recovered on recovery from error
Convergence loop:
- 5-minute periodic reconcile drives system toward desired state
- Catches config drift, daemon crashes, transient failures
- Serialized by reconcile mutex — no race with event-driven reconciles
Startup:
- CheckPrerequisites verifies required (dnsmasq, haproxy) and optional
(wg, authelia, cscli, cloudflared, nft) binaries before first reconcile
- Add storage.WriteFileAtomic (temp + rename) and storage.CopyFile helpers
- Convert all 8 production config writers to atomic writes: config/state.yaml,
dnsmasq, nftables, domains, wireguard (config + secrets + peers + wg0.conf),
tunnel/cloudflared
- Add sync.Mutex to Reconciler to serialize concurrent Reconcile() calls
triggered by domain registration goroutines
- Add state.yaml backup (.bak) before every write; LoadState falls back to
backup if primary is corrupted
- Reconciler refuses to use empty config on corruption (only on first run
when no state file exists yet)
- JSON tags: fix snake_case to camelCase in dnsmasq (configFile, domainsConfigured,
lastRestart), crowdsec Machine (lastPush, lastHeartbeat), network (primaryIP,
primaryInterface). cscli raw parsing structs keep snake_case to match CLI output.
- Error wrapping: fix %v to %w in enableAuthelia for proper error chain preservation
- Naming: rename dnsmasq.ConfigGenerator to dnsmasq.Manager (matches all other packages),
rename ServiceStatus to Status in dnsmasq and haproxy (matches authelia, crowdsec, etc.)
- Logging: standardize all slog calls to use "component" key instead of message prefixes.
Affects reconcile, dnsfilter, ddns — now consistent with dnsmasq, haproxy, nftables, sse.
