Paul Payne
959a896115
docs/mesh.md says what runs where (castle-nats, castle-openbao, each node's wildpc-api), how to tell it is healthy, what to do when a node goes stale, the vault is sealed or NATS is down, and the routine tasks -- written from the live config, not the plan. The plan is done and goes; its history is in git, and its unfinished items are tasks in Wild Life. AGENTS.md used the services' old names (wildpc-nats, wildpc-openbao) and pointed at the plan for operations; it now names castle-* and points here.
4.7 KiB
Mesh — runbook
How the nodes find each other, share config and secrets, and route to each
other's services — and what to do when that breaks. The design and its history
are in git: git log -p -- docs/fleet-mesh-plan.md.
What runs where
| piece | where | what it does |
|---|---|---|
castle-nats (civil) |
tls://castle-nats.civil.payne.io:4222, token NATS_TOKEN; monitoring :8222 |
NATS JetStream: the mesh transport and its KV buckets |
castle-openbao (civil) |
https://castle-openbao.civil.payne.io:8200, KV-v2 mount castle |
the secret authority behind ${secret:…} |
wildpc-api (every node) |
env WILDPC_API_NATS_ENABLED/URL/TOKEN |
the mesh client: publishes this node, watches the others |
Config and data: /data/castle/castle-nats/config/nats-server.conf,
/data/castle/castle-openbao/{config/openbao.hcl,unseal.sh,data}. Both present
the node's ACME wildcard cert (expose.tcp.tls), so clients verify against the
system CA; plaintext is refused.
Nodes. civil is the authority (role: authority in wildpc.yaml);
every other node is a follower (primer today). There is no consensus: when
civil is down, shared config and secrets are read-only and every node keeps
running its own deployments from local state.
KV buckets on NATS:
| bucket | holds | written by |
|---|---|---|
wildpc-registry |
each node's registry, secrets stripped | each node, for itself |
wildpc-presence |
a key per node, TTL 90 s, renewed on a heartbeat | each node, for itself |
wildpc-config |
shared fleet config | the authority only |
Secrets. wildpc.yaml's secrets: block selects OpenBao (backend: openbao,
mount: castle). There is no file fallback in that mode — only the bootstrap
tier stays as files in ~/.wildpc/secrets/ (OPENBAO_ROOT_TOKEN,
OPENBAO_UNSEAL_KEY). civil reads with the root token; followers with a
least-privilege wildpc-read token, and node_prefix for per-node overrides.
Write with wildpc secret set, never by hand (see AGENTS.md §6).
Is it healthy?
wildpc mesh status # connected? transport URL, peers
wildpc mesh nodes # each node: online | stale | offline
curl -s localhost:8222/healthz # NATS up
curl -s https://castle-openbao.civil.payne.io:8200/v1/sys/seal-status | jq .sealed
- online — the node's presence key is live.
- stale — no registry update from it for over 5 minutes, but not yet gone.
- offline — its presence key expired (90 s) or it left cleanly.
A peer's services are routed only while it is present; when it goes, its routes
drop out and the gateway answers 404 no wildpc route for this host
immediately, rather than hanging.
When something is wrong
A node shows stale or offline. On that node: is wildpc-api running
(wildpc service logs wildpc-api)? Can it reach
castle-nats.civil.payne.io:4222 by that name (TLS checks the name, so an IP
will not do)? Does its WILDPC_API_NATS_TOKEN resolve to the current
NATS_TOKEN?
OpenBao is sealed (after a restart, if unseal.sh failed): every
${secret:…} read fails and wildpc apply refuses to converge. Unseal by hand:
curl -s -X POST https://castle-openbao.civil.payne.io:8200/v1/sys/unseal \
-d "{\"key\":\"$(cat ~/.wildpc/secrets/OPENBAO_UNSEAL_KEY)\"}" | jq .sealed
NATS down. The mesh view freezes and peers go offline; nothing else stops —
each node keeps serving. wildpc restart castle-nats, then
wildpc mesh status on each node.
The certificate renewed and a client fails TLS. wildpc tls reconcile
re-materializes the wildcard onto both services and restarts them.
Routine tasks
Add a node. Install Wild PC on it; on its wildpc-api set
WILDPC_API_NATS_ENABLED=true,
WILDPC_API_NATS_URL=tls://castle-nats.civil.payne.io:4222,
WILDPC_API_NATS_TOKEN=${secret:NATS_TOKEN}; give it a secrets: block with a
wildpc-read token and node_prefix: nodes/<host>; leave role unset
(follower); wildpc apply wildpc-api. It appears in wildpc mesh nodes within a
heartbeat.
Use a service another node provides. requires: - ref: <service> in the
consumer's deployment. If no local deployment satisfies it and a present peer
does, the gateway routes <service>.<domain> to that peer, with a 2 s dial
timeout so a wedged peer fails fast.
Shared config. wildpc mesh config list|get|set — set works on the
authority only.
Rotate the NATS token. wildpc secret set NATS_TOKEN, then
wildpc apply castle-nats and wildpc apply wildpc-api on every node; peers
drop until each is re-applied.
Cluster NATS (not done): add a cluster {} block and routes to
nats-server.conf on each server — configuration only, by design.