Paul Payne
959a896115
docs/mesh.md says what runs where (castle-nats, castle-openbao, each node's wildpc-api), how to tell it is healthy, what to do when a node goes stale, the vault is sealed or NATS is down, and the routine tasks -- written from the live config, not the plan. The plan is done and goes; its history is in git, and its unfinished items are tasks in Wild Life. AGENTS.md used the services' old names (wildpc-nats, wildpc-openbao) and pointed at the plan for operations; it now names castle-* and points here.
104 lines
4.7 KiB
Markdown
104 lines
4.7 KiB
Markdown
# Mesh — runbook
|
|
|
|
How the nodes find each other, share config and secrets, and route to each
|
|
other's services — and what to do when that breaks. The design and its history
|
|
are in git: `git log -p -- docs/fleet-mesh-plan.md`.
|
|
|
|
## What runs where
|
|
|
|
| piece | where | what it does |
|
|
|---|---|---|
|
|
| `castle-nats` (civil) | `tls://castle-nats.civil.payne.io:4222`, token `NATS_TOKEN`; monitoring `:8222` | NATS JetStream: the mesh transport and its KV buckets |
|
|
| `castle-openbao` (civil) | `https://castle-openbao.civil.payne.io:8200`, KV-v2 mount `castle` | the secret authority behind `${secret:…}` |
|
|
| `wildpc-api` (every node) | env `WILDPC_API_NATS_ENABLED/URL/TOKEN` | the mesh client: publishes this node, watches the others |
|
|
|
|
Config and data: `/data/castle/castle-nats/config/nats-server.conf`,
|
|
`/data/castle/castle-openbao/{config/openbao.hcl,unseal.sh,data}`. Both present
|
|
the node's ACME wildcard cert (`expose.tcp.tls`), so clients verify against the
|
|
system CA; plaintext is refused.
|
|
|
|
**Nodes.** `civil` is the **authority** (`role: authority` in `wildpc.yaml`);
|
|
every other node is a follower (`primer` today). There is no consensus: when
|
|
civil is down, shared config and secrets are read-only and every node keeps
|
|
running its own deployments from local state.
|
|
|
|
**KV buckets** on NATS:
|
|
|
|
| bucket | holds | written by |
|
|
|---|---|---|
|
|
| `wildpc-registry` | each node's registry, secrets stripped | each node, for itself |
|
|
| `wildpc-presence` | a key per node, TTL 90 s, renewed on a heartbeat | each node, for itself |
|
|
| `wildpc-config` | shared fleet config | the authority only |
|
|
|
|
**Secrets.** `wildpc.yaml`'s `secrets:` block selects OpenBao (`backend: openbao`,
|
|
`mount: castle`). There is **no file fallback** in that mode — only the bootstrap
|
|
tier stays as files in `~/.wildpc/secrets/` (`OPENBAO_ROOT_TOKEN`,
|
|
`OPENBAO_UNSEAL_KEY`). civil reads with the root token; followers with a
|
|
least-privilege `wildpc-read` token, and `node_prefix` for per-node overrides.
|
|
Write with `wildpc secret set`, never by hand (see AGENTS.md §6).
|
|
|
|
## Is it healthy?
|
|
|
|
```bash
|
|
wildpc mesh status # connected? transport URL, peers
|
|
wildpc mesh nodes # each node: online | stale | offline
|
|
curl -s localhost:8222/healthz # NATS up
|
|
curl -s https://castle-openbao.civil.payne.io:8200/v1/sys/seal-status | jq .sealed
|
|
```
|
|
|
|
- **online** — the node's presence key is live.
|
|
- **stale** — no registry update from it for over 5 minutes, but not yet gone.
|
|
- **offline** — its presence key expired (90 s) or it left cleanly.
|
|
|
|
A peer's services are routed only while it is present; when it goes, its routes
|
|
drop out and the gateway answers `404 no wildpc route for this host`
|
|
immediately, rather than hanging.
|
|
|
|
## When something is wrong
|
|
|
|
**A node shows `stale` or `offline`.** On that node: is `wildpc-api` running
|
|
(`wildpc service logs wildpc-api`)? Can it reach
|
|
`castle-nats.civil.payne.io:4222` by that name (TLS checks the name, so an IP
|
|
will not do)? Does its `WILDPC_API_NATS_TOKEN` resolve to the current
|
|
`NATS_TOKEN`?
|
|
|
|
**OpenBao is sealed** (after a restart, if `unseal.sh` failed): every
|
|
`${secret:…}` read fails and `wildpc apply` refuses to converge. Unseal by hand:
|
|
|
|
```bash
|
|
curl -s -X POST https://castle-openbao.civil.payne.io:8200/v1/sys/unseal \
|
|
-d "{\"key\":\"$(cat ~/.wildpc/secrets/OPENBAO_UNSEAL_KEY)\"}" | jq .sealed
|
|
```
|
|
|
|
**NATS down.** The mesh view freezes and peers go offline; nothing else stops —
|
|
each node keeps serving. `wildpc restart castle-nats`, then
|
|
`wildpc mesh status` on each node.
|
|
|
|
**The certificate renewed and a client fails TLS.** `wildpc tls reconcile`
|
|
re-materializes the wildcard onto both services and restarts them.
|
|
|
|
## Routine tasks
|
|
|
|
**Add a node.** Install Wild PC on it; on its `wildpc-api` set
|
|
`WILDPC_API_NATS_ENABLED=true`,
|
|
`WILDPC_API_NATS_URL=tls://castle-nats.civil.payne.io:4222`,
|
|
`WILDPC_API_NATS_TOKEN=${secret:NATS_TOKEN}`; give it a `secrets:` block with a
|
|
`wildpc-read` token and `node_prefix: nodes/<host>`; leave `role` unset
|
|
(follower); `wildpc apply wildpc-api`. It appears in `wildpc mesh nodes` within a
|
|
heartbeat.
|
|
|
|
**Use a service another node provides.** `requires: - ref: <service>` in the
|
|
consumer's deployment. If no local deployment satisfies it and a present peer
|
|
does, the gateway routes `<service>.<domain>` to that peer, with a 2 s dial
|
|
timeout so a wedged peer fails fast.
|
|
|
|
**Shared config.** `wildpc mesh config list|get|set` — `set` works on the
|
|
authority only.
|
|
|
|
**Rotate the NATS token.** `wildpc secret set NATS_TOKEN`, then
|
|
`wildpc apply castle-nats` and `wildpc apply wildpc-api` on every node; peers
|
|
drop until each is re-applied.
|
|
|
|
**Cluster NATS** (not done): add a `cluster {}` block and routes to
|
|
`nats-server.conf` on each server — configuration only, by design.
|