SafeApply pattern (validate → backup → write → reload → verify → rollback):
- HAProxy: SafeApply wraps existing validate+write+reload with backup and
post-reload health check; rolls back to .bak on failure
- dnsmasq: SafeApply validates via dnsmasq --test, backs up, atomic writes,
restarts, verifies daemon is active; rolls back on failure
- nftables: SafeApply validates via nft -c, backs up, atomic writes, applies
to kernel, verifies table loaded; rolls back on failure
Health tracking:
- Add SubsystemHealth and Health structs to reconciler
- Track per-subsystem status (ok/degraded/error) after each reconcile
- Detect recovery: previous error → current ok broadcasts recovery event
- GET /api/v1/health/reconcile endpoint exposes health state
- HAProxy tracks excluded domains as "degraded" state
SSE error events:
- Broadcast haproxy:error, dnsmasq:error on SafeApply failure
- Broadcast haproxy:recovered, dnsmasq:recovered on recovery from error
Convergence loop:
- 5-minute periodic reconcile drives system toward desired state
- Catches config drift, daemon crashes, transient failures
- Serialized by reconcile mutex — no race with event-driven reconciles
Startup:
- CheckPrerequisites verifies required (dnsmasq, haproxy) and optional
(wg, authelia, cscli, cloudflared, nft) binaries before first reconcile
- Add storage.WriteFileAtomic (temp + rename) and storage.CopyFile helpers
- Convert all 8 production config writers to atomic writes: config/state.yaml,
dnsmasq, nftables, domains, wireguard (config + secrets + peers + wg0.conf),
tunnel/cloudflared
- Add sync.Mutex to Reconciler to serialize concurrent Reconcile() calls
triggered by domain registration goroutines
- Add state.yaml backup (.bak) before every write; LoadState falls back to
backup if primary is corrupted
- Reconciler refuses to use empty config on corruption (only on first run
when no state file exists yet)
- JSON tags: fix snake_case to camelCase in dnsmasq (configFile, domainsConfigured,
lastRestart), crowdsec Machine (lastPush, lastHeartbeat), network (primaryIP,
primaryInterface). cscli raw parsing structs keep snake_case to match CLI output.
- Error wrapping: fix %v to %w in enableAuthelia for proper error chain preservation
- Naming: rename dnsmasq.ConfigGenerator to dnsmasq.Manager (matches all other packages),
rename ServiceStatus to Status in dnsmasq and haproxy (matches authelia, crowdsec, etc.)
- Logging: standardize all slog calls to use "component" key instead of message prefixes.
Affects reconcile, dnsfilter, ddns — now consistent with dnsmasq, haproxy, nftables, sse.
Adds Pi-hole-style DNS filtering using dnsmasq's native address=/ directives,
which block domains and all subdomains. Users subscribe to blocklists by URL
or upload files, with suggested lists from Hagezi, Steven Black, and OISD.
Background runner refreshes lists on a configurable interval (default 24h).
SMTP: Add host, port, username, sender, and password fields to the
Authelia configuration. Uses modern address format (submission:// for
STARTTLS on 587, smtps:// for implicit TLS on 465) with explicit TLS
server name. Falls back to filesystem notifier when SMTP is not
configured.
Fixes: Config changes now persist before attempting service restart,
so a restart failure (e.g. bad SMTP credentials) no longer prevents
saving. The specific Authelia error is extracted from the journal and
shown in the UI.
Frontend: SMTP fields added to the Authentication config card. Form
no longer continuously resets from server state while user is editing.
OIDC client edit UI added (pencil icon).
Authelia runs as a managed native service on Wild Central, providing
two integration patterns for network services:
- Forward-auth via HAProxy for apps without native SSO (Lua auth-request
script intercepts requests, redirects unauthenticated users to login portal)
- OIDC provider for apps with native support (Gitea, Grafana, etc.)
Backend: new internal/authelia/ package with service manager, config
generation, file-based user management (argon2id), and OIDC client
management. API endpoints for status, config, users CRUD, and OIDC
clients CRUD.
HAProxy: config generator extended with lua-load, auth-request
directives, and Authelia backend. Auth directives scoped to session
cookie domain — only subdomains of the auth portal's parent are
eligible for forward-auth.
Frontend: Authentication page with enable/disable, user management,
OIDC client management (add/edit/delete), and protected domains
toggles. Advanced subsystem page for raw config view. Dashboard
service card and sidebar entries.
InstanceConfig was a ~50-field struct designed for Wild Cloud k8s instances,
but only 3 fields were ever read by dnsmasq (the sole consumer). The
reconciliation bridge constructed fake InstanceConfig objects from registered
domains just to satisfy this interface.
Replace with DNSEntry{Domain, IP} — a 2-field struct purpose-built for
dnsmasq. All domains get local=/ directives to prevent AAAA queries from
leaking to upstream DNS (Happy Eyeballs / RFC 8305 latency on LAN).
Removed dead code: InstanceConfig, NodeConfig, LoadCloudConfig,
SaveCloudConfig, DeepMerge, LoadMergedInstanceConfig, EnsureInstanceConfig,
instance path helpers, instanceLoadBalancerIP, GenerateInstanceConfig,
and all modular per-instance dnsmasq methods.
Split /api/v1/state into dedicated resource endpoints:
- /api/v1/operator, /api/v1/central-domain
- /api/v1/ddns/config, /api/v1/dns/settings
- /api/v1/dhcp/config (moved from /dnsmasq/dhcp/)
- /api/v1/nftables/config, /api/v1/haproxy/routes
Each page now talks to its own resource instead of deep-merging
a blob. Removes useConfig hook, CentralState type, and legacy
config methods.
When haproxy -c fails, parse ALERT line numbers, map to service domains
via # service: comments in generated config, exclude broken services,
and regenerate. One retry — prevents a single bad registration from
blocking all config updates.
- Add Routes model to services UI (paths, headers, IP whitelisting per route)
- Show TLS cert info per service with inline provision/renew actions
- Remove TLS Certificates section from dashboard (now on services page)
- Make gateway router port list dynamic from config + VPN state
- Add TODO for header validation in HAProxy config generation
Linter-contributed improvements:
- EnsureCentralService() method: registers Central's own domain as a
service from config, with cleanup on domain change
- Tests for EnsureCentralService: registration, idempotency, domain
change cleanup, no-domain-is-noop
- Improved reconciliation tests with Central as a registered service
- Certbot handler cleanup
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
dnsmasq (6 new, 30.8% → 32%):
- Empty instances, multiple instances, nil config, configured IP
- No leaked empty entries (address=//, local=//)
- Empty fields don't produce broken entries
Also fixed a bug: empty Domain/InternalDomain fields in commented-out
entries (no LB IP case) now guarded against producing # local=// and
# address=// directives.
haproxy (7 new):
- No HTTPS frontend with no routes
- HTTP-only and L4-only route scenarios
- CertsDir customization and default
- Custom TCP route ports
- Health check directive in L7 backends
services (8 new, 72.7% → 75.8%):
- Not-found errors for Get, Deregister, Update
- DeregisterBySource partial match (source AND backend)
- Overwrite preserves file count
- Empty dir and non-YAML file handling
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The reconciliation test for Central domain now verifies the HAProxy
backend uses port 15055 (the configured port) and NOT port 5055
(the default). This catches the bug where Reconcile() was called
before SetPort().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Tests added (13 new):
- HAProxy ACL ordering: L7 before L4 wildcard, L4 exact before L4 wildcard
- HAProxy subdomains: false=exact only, true=wildcard+exact
- Reconciliation integration: HAProxy config from services, DNS config
from services, Central config domain injection, correct DNS target IPs
- dnsmasq: reach:internal has local=/, reach:public does not
- Services: idempotent registration, no duplicates in list
Bug fix:
- Run initial reconciliation on startup so Central's own domain (from
config) gets HAProxy + DNS routes even with zero registered services.
Previously, Central only got routes after the first service was
registered, causing 503 on fresh restart.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The reconciliation was building InstanceRoutes without setting the
Subdomains field, so all services were exact-match only — breaking
wildcard subdomain routing for *.cloud.payne.io and *.cloud2.payne.io.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Central no longer self-registers as a service. Its own domain comes
from config (cloud.central.domain) and is injected directly into
reconciliation as an HTTPRoute. This is cleaner — Central's own
services are config-driven, not registration-driven.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rewrite HAProxy config generation with correct ACL ordering:
1. L7 HTTP exact matches first (central, wild-cloud app)
2. L4 exact matches (payne.io, civilsociety.dev)
3. L4 wildcard matches (*.cloud.payne.io) — only when subdomains:true
4. Default → L7 termination
Key changes:
- InstanceRoute: removed ExtraDomains, added Subdomains bool
- GenerateOpts: removed CentralDomain (Central is now just another
HTTPRoute injected by reconciliation), renamed WildcardCert→CertsDir
- Central's domain comes from config, injected as HTTPRoute with
the actual running port (no more hardcoded 5055)
- Added API.SetPort() so reconciliation knows the running port
- subdomains:false → exact match only (no *.domain shadowing)
New tests: TestGenerate_WithSubdomains, TestGenerate_ACLOrdering
(verifies L7 exact matches come before L4 wildcards).
Fixes: BUG 1 (central routing to wrong port), BUG 2 (wild-cloud
getting traefik cert), BUG 3 (*.payne.io too broad).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rewrite the service registration model per the architecture plan:
Model changes:
- Domain is the unique key (removed Name field)
- Removed ExtraDomains — each domain is its own registration
- Removed SourceNode, BackendStatic, ReachOff
- Added Subdomains bool for wildcard matching control
- Files stored as <domain_with_underscores>.yaml
- API routes: /services/{domain:.+} (regex for dots in path)
- Added DeregisterBySource for cleanup before re-registration
Reconciliation changes:
- No ExtraDomains iteration — each service IS one domain
- reach:internal → sets InternalDomain (generates local=/)
- reach:public → sets Domain (no local=/)
- tcp-passthrough → DNS points to backend IP
- http → DNS points to Central IP
All 22 tests pass (9 manager + 8 handler + 5 config/cert).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The auto-detect picks the default route interface (wlan0 at
192.168.8.152) but dnsmasq is bound to eth0 (192.168.8.151).
Now uses cloud.dnsmasq.ip from config as the Central IP for DNS
entries, falling back to auto-detect only if not configured.
Fixes DNS resolution failures when dnsmasq config points services
to the wrong interface IP.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use systemctl reload-or-restart instead of restart. dnsmasq supports
SIGHUP to reload config without dropping DNS service, preventing
transient DNS resolution failures during reconciliation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix dnsmasq reconciliation to use the correct DNS target IP:
- tcp-passthrough (k8s instances): DNS → k8s LB IP (LAN direct)
- http/static (Central apps): DNS → Central IP (HAProxy terminates)
Also handle ExtraDomains properly:
- Internal domains (internal.cloud.payne.io) get local=/ entries
to prevent upstream DNS forwarding
- Extra domains get their own dnsmasq address= entries
- HAProxy already propagated ExtraDomains for SNI ACLs
Verified: cloud.payne.io → 192.168.8.240 (direct to k8s),
cloud2.payne.io → 192.168.8.80, central.payne.io → 192.168.8.151.
SNI routing through HAProxy works for both instances.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove automatic wildcard cert assumption. Each service gets its own
cert by default. Wildcards are user-initiated via the Certificates page.
Backend:
- HAProxy L7 frontend uses cert directory (/etc/haproxy/certs/)
instead of single wildcard file — loads all PEMs, serves by SNI
- Cert status API shows every registered service individually with
coveredBy field when a wildcard cert covers the domain
- Reconciliation filters L7 routes to services that have a cert
- No auto-provisioning in reconciliation (just warnings)
Frontend:
- Certificates page shows per-service cert status
- "Provision" button for individual certs
- "Add Wildcard" form for opt-in wildcard provisioning
- Fixed CloudflareComponent type errors from cert API changes
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New dedicated Certificates page in Wild Central UI showing:
- All tracked certs (central, wildcard, per-service)
- Status (valid with days remaining, or missing)
- Per-domain Provision button
- Renew All button
Cert provisioning is now user-initiated only — reconciliation logs
warnings about missing certs but does NOT auto-provision. This
prevents unexpected wildcard cert requests on startup.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
API changes:
- /api/v1/cert/status now returns all relevant certs (central,
wildcard, per-service) with existence status and expiry
- /api/v1/cert/provision accepts ?domain= param for per-domain
provisioning
- Removed stale syncHAProxy references, uses reconcileNetworking
UI changes:
- Added TLS Certificates card to Cloudflare page showing cert status
for each domain with provision/renew actions
- Updated useCert hook and cert API types for new response format
Next: extract into dedicated Certificates page.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
HAProxy validation fails if the config references cert files that
don't exist (wildcard.pem for L7 routes, domain.pem for central TLS).
Now reconciliation checks for cert existence before including:
- L7 HTTP routes: only generated when wildcard cert exists
- Central domain TLS frontend: only generated when domain cert exists
This ensures HAProxy never enters a failure state due to missing
certs. Once certs are provisioned (via certbot), the next
reconciliation picks them up automatically.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
If the NATS server fails to start (stale lock files, corrupt store
from a crash), automatically clean the data directory and retry once.
This prevents the frustrating "NATS server failed to start" error
after unclean shutdowns.
The NATS KV data (registered services) is also stored as YAML files
in the services directory, so cleaning the NATS store loses nothing
— services are re-registered on the next startup or API call.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Services registered via the service API have a domain but no internal
domain (that's a Wild Cloud instance concept). The dnsmasq generator
was producing broken entries like `local=//` and `address=//`.
Fix: skip internal domain entries when InternalDomain is empty, in
both config.go and config_modular.go. Also fix DNS entries to point
to Central's IP (where HAProxy listens) rather than the backend
address — all traffic flows through Central.
Added 3 tests covering: service-only, mixed services+instances.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When HTTP services are registered (where Central terminates TLS),
reconciliation now checks for and provisions TLS certificates:
- Derives gateway domain from central domain (e.g., payne.io)
- Checks for existing wildcard cert (*.payne.io)
- If no wildcard exists and Cloudflare token + operator email are
configured, provisions one via certbot DNS-01
- For services outside the gateway domain, provisions individual certs
- Services with TLS passthrough (k8s) are skipped (backend handles TLS)
- Graceful: skips silently if no Cloudflare token or email configured
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
On startup, Wild Central registers its own domain (from config
cloud.central.domain) as an HTTP service with itself. This creates
the DNS entry, proxy route, and TLS cert needed to access the
Central UI via its domain name.
Registration is idempotent — safe on every restart. Skips silently
if no domain is configured. Logs a warning if registration fails.
Verified: after startup, /api/v1/services shows wild-central
registered with domain central.payne.io.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix config endpoint to return wrapped { configured, config } response
that the web UI expects. Fix services handler to return stored version
with defaults applied.
New tests:
- GetGlobalConfig: wrapped response, empty/not-configured case
- GetGlobalSecrets: leaf redaction preserves structure, raw mode
- UpdateGlobalConfig: write and verify
- CloudflareVerify: no-token case
- Services: register, list, get, update, deregister, TCP passthrough
defaults, validation errors
15 new test cases, all passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The secrets endpoint was flattening nested maps (e.g., cloudflare.apiToken)
to a single "********" string. The UI checks for nested keys to determine
if credentials are configured, so the structure must be preserved.
Now redactSecrets() recursively walks the map and only redacts leaf values.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add /api/v1/cloudflare/verify, /api/v1/cloudflare/zone endpoints
so the Central UI's Cloudflare page can verify tokens and manage
zone configuration. This was a missing endpoint from the extraction.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a service is registered, updated, or deregistered, Central now
automatically regenerates HAProxy routes and dnsmasq DNS entries to
match the current set of registered services.
- L4 tcp-passthrough services (Wild Cloud) → HAProxy SNI routes
- L7 http services (Wild Works) → HAProxy Host-based reverse proxy
- All services with reach != off → dnsmasq DNS entries
- SSE events broadcast on config changes
- Reconciliation runs asynchronously (non-blocking)
This closes the loop: service registration is no longer inert — it
actually updates the networking stack.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The default paths for dnsmasq and nftables config files should match
the existing production layout (wild-cloud.conf, wild-cloud.nft) so
Wild Central works as a drop-in when reading from the production data
directory. These can be overridden via env vars for fresh installs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Port Castle's tunnel config generator to Go. Central can now manage
cloudflared ingress configuration for public service exposure.
Services with reach=public get projected at <subdomain>.<publicDomain>
with Host/SNI rewriting to the internal domain so the gateway's wildcard
cert validates and existing routing works unchanged.
- Locally-managed config.yml (not remotely-managed token tunnel)
- Public surface is exactly the set of reach=public services
- Auto-removes config when tunnel is disabled or no public services
- 7 tests covering basic generation, subdomain override, edge cases
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extend HAProxy config generation to support L7 HTTP routes alongside
existing L4 SNI passthrough routes. This enables Wild Works services
to be reverse-proxied with TLS termination by Central while Wild Cloud
k8s instances continue using L4 SNI passthrough.
- Add HTTPRoute type: domain, backend address, optional health path
- Add GenerateWithOpts for full config options (HTTP routes, wildcard cert)
- L4 SNI frontend routes known Cloud domains to k8s LBs; unknown SNIs
fall through to L7 TLS-termination frontend for Host-based routing
- L7 frontend terminates TLS with wildcard cert, routes by Host header
- Health check support via HAProxy httpchk
- Backward compatible: existing Generate() works unchanged
- 3 new tests (L7-only, mixed L4+L7, backward compat)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Central now starts an embedded NATS JetStream server as its core
coordination bus. This provides:
- wild-services KV bucket: service registrations from Cloud/Works
- wild-presence KV bucket: node liveness with TTL-based keys
- wild-events stream: events from all sources (24h retention)
NATS is started in main.go before the API and shut down on SIGTERM.
Cloud and Works will connect to Central's NATS (port 4222) to register
services and maintain presence.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add the service registration abstraction — the key contract between
Central and its consumers (Wild Cloud, Wild Works). Services register
with Central to get DNS, gateway routing, TLS, and public exposure.
- internal/services/manager.go: CRUD for service registrations stored
as YAML files, with reach model (off/internal/public), backend types
(tcp-passthrough/http/static), and TLS modes
- internal/api/v1/handlers_services.go: HTTP endpoints
POST/GET/PATCH/DELETE /api/v1/services
- Full test coverage for registration, update, deregister, validation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract the Central networking functionality from wild-cloud/api into
a standalone service. Wild Central manages DNS (dnsmasq), gateway
(HAProxy), firewall (nftables), VPN (WireGuard), TLS (certbot),
security (CrowdSec), and DDNS — all the network appliance concerns.
All Cloud-specific code (instances, clusters, nodes, apps, backups,
operations, kubectl/talosctl tooling) has been removed. The API struct
and route registration contain only Central endpoints. Tests updated
to match the new API signature.
Builds and all tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
