# Mesh — runbook

How the nodes find each other, share config and secrets, and route to each
other's services — and what to do when that breaks. The design and its history
are in git: `git log -p -- docs/fleet-mesh-plan.md`.

## What runs where

| piece | where | what it does |
|---|---|---|
| `castle-nats` (civil) | `tls://castle-nats.civil.payne.io:4222`, token `NATS_TOKEN`; monitoring `:8222` | NATS JetStream: the mesh transport and its KV buckets |
| `castle-openbao` (civil) | `https://castle-openbao.civil.payne.io:8200`, KV-v2 mount `castle` | the secret authority behind `${secret:…}` |
| `wildpc-api` (every node) | env `WILDPC_API_NATS_ENABLED/URL/TOKEN` | the mesh client: publishes this node, watches the others |

Config and data: `/data/castle/castle-nats/config/nats-server.conf`,
`/data/castle/castle-openbao/{config/openbao.hcl,unseal.sh,data}`. Both present
the node's ACME wildcard cert (`expose.tcp.tls`), so clients verify against the
system CA; plaintext is refused.

**Nodes.** `civil` is the **authority** (`role: authority` in `wildpc.yaml`);
every other node is a follower (`primer` today). There is no consensus: when
civil is down, shared config and secrets are read-only and every node keeps
running its own deployments from local state.

**KV buckets** on NATS:

| bucket | holds | written by |
|---|---|---|
| `wildpc-registry` | each node's registry, secrets stripped | each node, for itself |
| `wildpc-presence` | a key per node, TTL 90 s, renewed on a heartbeat | each node, for itself |
| `wildpc-config` | shared fleet config | the authority only |

**Secrets.** `wildpc.yaml`'s `secrets:` block selects OpenBao (`backend: openbao`,
`mount: castle`). There is **no file fallback** in that mode — only the bootstrap
tier stays as files in `~/.wildpc/secrets/` (`OPENBAO_ROOT_TOKEN`,
`OPENBAO_UNSEAL_KEY`). civil reads with the root token; followers with a
least-privilege `wildpc-read` token, and `node_prefix` for per-node overrides.
Write with `wildpc secret set`, never by hand (see AGENTS.md §6).

## Is it healthy?

```bash
wildpc mesh status            # connected? transport URL, peers
wildpc mesh nodes             # each node: online | stale | offline
curl -s localhost:8222/healthz                       # NATS up
curl -s https://castle-openbao.civil.payne.io:8200/v1/sys/seal-status | jq .sealed
```

- **online** — the node's presence key is live.
- **stale** — no registry update from it for over 5 minutes, but not yet gone.
- **offline** — its presence key expired (90 s) or it left cleanly.

A peer's services are routed only while it is present; when it goes, its routes
drop out and the gateway answers `404 no wildpc route for this host`
immediately, rather than hanging.

## When something is wrong

**A node shows `stale` or `offline`.** On that node: is `wildpc-api` running
(`wildpc service logs wildpc-api`)? Can it reach
`castle-nats.civil.payne.io:4222` by that name (TLS checks the name, so an IP
will not do)? Does its `WILDPC_API_NATS_TOKEN` resolve to the current
`NATS_TOKEN`?

**OpenBao is sealed** (after a restart, if `unseal.sh` failed): every
`${secret:…}` read fails and `wildpc apply` refuses to converge. Unseal by hand:

```bash
curl -s -X POST https://castle-openbao.civil.payne.io:8200/v1/sys/unseal \
  -d "{\"key\":\"$(cat ~/.wildpc/secrets/OPENBAO_UNSEAL_KEY)\"}" | jq .sealed
```

**NATS down.** The mesh view freezes and peers go offline; nothing else stops —
each node keeps serving. `wildpc restart castle-nats`, then
`wildpc mesh status` on each node.

**The certificate renewed and a client fails TLS.** `wildpc tls reconcile`
re-materializes the wildcard onto both services and restarts them.

## Routine tasks

**Add a node.** Install Wild PC on it; on its `wildpc-api` set
`WILDPC_API_NATS_ENABLED=true`,
`WILDPC_API_NATS_URL=tls://castle-nats.civil.payne.io:4222`,
`WILDPC_API_NATS_TOKEN=${secret:NATS_TOKEN}`; give it a `secrets:` block with a
`wildpc-read` token and `node_prefix: nodes/<host>`; leave `role` unset
(follower); `wildpc apply wildpc-api`. It appears in `wildpc mesh nodes` within a
heartbeat.

**Use a service another node provides.** `requires: - ref: <service>` in the
consumer's deployment. If no local deployment satisfies it and a present peer
does, the gateway routes `<service>.<domain>` to that peer, with a 2 s dial
timeout so a wedged peer fails fast.

**Shared config.** `wildpc mesh config list|get|set` — `set` works on the
authority only.

**Rotate the NATS token.** `wildpc secret set NATS_TOKEN`, then
`wildpc apply castle-nats` and `wildpc apply wildpc-api` on every node; peers
drop until each is re-applied.

**Cluster NATS** (not done): add a `cluster {}` block and routes to
`nats-server.conf` on each server — configuration only, by design.