docs: a mesh runbook in place of the finished plan
docs/mesh.md says what runs where (castle-nats, castle-openbao, each node's wildpc-api), how to tell it is healthy, what to do when a node goes stale, the vault is sealed or NATS is down, and the routine tasks -- written from the live config, not the plan. The plan is done and goes; its history is in git, and its unfinished items are tasks in Wild Life. AGENTS.md used the services' old names (wildpc-nats, wildpc-openbao) and pointed at the plan for operations; it now names castle-* and points here.
This commit is contained in:
103
docs/mesh.md
Normal file
103
docs/mesh.md
Normal file
@@ -0,0 +1,103 @@
|
||||
# Mesh — runbook
|
||||
|
||||
How the nodes find each other, share config and secrets, and route to each
|
||||
other's services — and what to do when that breaks. The design and its history
|
||||
are in git: `git log -p -- docs/fleet-mesh-plan.md`.
|
||||
|
||||
## What runs where
|
||||
|
||||
| piece | where | what it does |
|
||||
|---|---|---|
|
||||
| `castle-nats` (civil) | `tls://castle-nats.civil.payne.io:4222`, token `NATS_TOKEN`; monitoring `:8222` | NATS JetStream: the mesh transport and its KV buckets |
|
||||
| `castle-openbao` (civil) | `https://castle-openbao.civil.payne.io:8200`, KV-v2 mount `castle` | the secret authority behind `${secret:…}` |
|
||||
| `wildpc-api` (every node) | env `WILDPC_API_NATS_ENABLED/URL/TOKEN` | the mesh client: publishes this node, watches the others |
|
||||
|
||||
Config and data: `/data/castle/castle-nats/config/nats-server.conf`,
|
||||
`/data/castle/castle-openbao/{config/openbao.hcl,unseal.sh,data}`. Both present
|
||||
the node's ACME wildcard cert (`expose.tcp.tls`), so clients verify against the
|
||||
system CA; plaintext is refused.
|
||||
|
||||
**Nodes.** `civil` is the **authority** (`role: authority` in `wildpc.yaml`);
|
||||
every other node is a follower (`primer` today). There is no consensus: when
|
||||
civil is down, shared config and secrets are read-only and every node keeps
|
||||
running its own deployments from local state.
|
||||
|
||||
**KV buckets** on NATS:
|
||||
|
||||
| bucket | holds | written by |
|
||||
|---|---|---|
|
||||
| `wildpc-registry` | each node's registry, secrets stripped | each node, for itself |
|
||||
| `wildpc-presence` | a key per node, TTL 90 s, renewed on a heartbeat | each node, for itself |
|
||||
| `wildpc-config` | shared fleet config | the authority only |
|
||||
|
||||
**Secrets.** `wildpc.yaml`'s `secrets:` block selects OpenBao (`backend: openbao`,
|
||||
`mount: castle`). There is **no file fallback** in that mode — only the bootstrap
|
||||
tier stays as files in `~/.wildpc/secrets/` (`OPENBAO_ROOT_TOKEN`,
|
||||
`OPENBAO_UNSEAL_KEY`). civil reads with the root token; followers with a
|
||||
least-privilege `wildpc-read` token, and `node_prefix` for per-node overrides.
|
||||
Write with `wildpc secret set`, never by hand (see AGENTS.md §6).
|
||||
|
||||
## Is it healthy?
|
||||
|
||||
```bash
|
||||
wildpc mesh status # connected? transport URL, peers
|
||||
wildpc mesh nodes # each node: online | stale | offline
|
||||
curl -s localhost:8222/healthz # NATS up
|
||||
curl -s https://castle-openbao.civil.payne.io:8200/v1/sys/seal-status | jq .sealed
|
||||
```
|
||||
|
||||
- **online** — the node's presence key is live.
|
||||
- **stale** — no registry update from it for over 5 minutes, but not yet gone.
|
||||
- **offline** — its presence key expired (90 s) or it left cleanly.
|
||||
|
||||
A peer's services are routed only while it is present; when it goes, its routes
|
||||
drop out and the gateway answers `404 no wildpc route for this host`
|
||||
immediately, rather than hanging.
|
||||
|
||||
## When something is wrong
|
||||
|
||||
**A node shows `stale` or `offline`.** On that node: is `wildpc-api` running
|
||||
(`wildpc service logs wildpc-api`)? Can it reach
|
||||
`castle-nats.civil.payne.io:4222` by that name (TLS checks the name, so an IP
|
||||
will not do)? Does its `WILDPC_API_NATS_TOKEN` resolve to the current
|
||||
`NATS_TOKEN`?
|
||||
|
||||
**OpenBao is sealed** (after a restart, if `unseal.sh` failed): every
|
||||
`${secret:…}` read fails and `wildpc apply` refuses to converge. Unseal by hand:
|
||||
|
||||
```bash
|
||||
curl -s -X POST https://castle-openbao.civil.payne.io:8200/v1/sys/unseal \
|
||||
-d "{\"key\":\"$(cat ~/.wildpc/secrets/OPENBAO_UNSEAL_KEY)\"}" | jq .sealed
|
||||
```
|
||||
|
||||
**NATS down.** The mesh view freezes and peers go offline; nothing else stops —
|
||||
each node keeps serving. `wildpc restart castle-nats`, then
|
||||
`wildpc mesh status` on each node.
|
||||
|
||||
**The certificate renewed and a client fails TLS.** `wildpc tls reconcile`
|
||||
re-materializes the wildcard onto both services and restarts them.
|
||||
|
||||
## Routine tasks
|
||||
|
||||
**Add a node.** Install Wild PC on it; on its `wildpc-api` set
|
||||
`WILDPC_API_NATS_ENABLED=true`,
|
||||
`WILDPC_API_NATS_URL=tls://castle-nats.civil.payne.io:4222`,
|
||||
`WILDPC_API_NATS_TOKEN=${secret:NATS_TOKEN}`; give it a `secrets:` block with a
|
||||
`wildpc-read` token and `node_prefix: nodes/<host>`; leave `role` unset
|
||||
(follower); `wildpc apply wildpc-api`. It appears in `wildpc mesh nodes` within a
|
||||
heartbeat.
|
||||
|
||||
**Use a service another node provides.** `requires: - ref: <service>` in the
|
||||
consumer's deployment. If no local deployment satisfies it and a present peer
|
||||
does, the gateway routes `<service>.<domain>` to that peer, with a 2 s dial
|
||||
timeout so a wedged peer fails fast.
|
||||
|
||||
**Shared config.** `wildpc mesh config list|get|set` — `set` works on the
|
||||
authority only.
|
||||
|
||||
**Rotate the NATS token.** `wildpc secret set NATS_TOKEN`, then
|
||||
`wildpc apply castle-nats` and `wildpc apply wildpc-api` on every node; peers
|
||||
drop until each is re-applied.
|
||||
|
||||
**Cluster NATS** (not done): add a `cluster {}` block and routes to
|
||||
`nats-server.conf` on each server — configuration only, by design.
|
||||
Reference in New Issue
Block a user