diff --git a/AGENTS.md b/AGENTS.md index 7744128..43f4f7b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -260,7 +260,7 @@ resolve through a backend selected by the **`secrets:` block in `wildpc.yaml`** ```yaml secrets: backend: openbao # or `file` (default) - addr: https://wildpc-openbao.:8200 + addr: https://castle-openbao.:8200 mount: wildpc # KV-v2 mount token_secret: OPENBAO_ROOT_TOKEN # vault token, read from a file node_prefix: nodes/ # optional: per-node overrides @@ -310,8 +310,8 @@ a service public unless it authenticates or is meant to be open. ## 8. Mesh — multi-node coordination (opt-in) -Runs on **NATS JetStream** (`wildpc-nats`, TLS + token). Enable via env on -`wildpc-api`: `WILDPC_API_NATS_ENABLED=true`, `WILDPC_API_NATS_URL=tls://wildpc-nats.:4222`, +Runs on **NATS JetStream** (`castle-nats`, TLS + token). Enable via env on +`wildpc-api`: `WILDPC_API_NATS_ENABLED=true`, `WILDPC_API_NATS_URL=tls://castle-nats.:4222`, `WILDPC_API_NATS_TOKEN=${secret:NATS_TOKEN}`. Each node publishes its (secret-stripped) registry to a JetStream **KV** bucket, renews a **presence** key, and watches for peers; remote deployments surface as `manager: none` @@ -324,7 +324,7 @@ Inspect + drive from the CLI: **`wildpc mesh status`** / **`wildpc mesh nodes`** **`wildpc mesh config list|get|set`** (or `GET /mesh/status`, `/nodes`, `/mesh/config`). Modules: `wildpc_api.nats_client`, `.mesh`, `.mesh_gateway`, `.mdns`; secrets via `core` `secret_backends` (file default, OpenBao opt-in). -→ Full history + operations: **`docs/fleet-mesh-plan.md`**. +→ Operations — health, a stale node, a sealed vault, adding a node: **`docs/mesh.md`**. --- @@ -336,6 +336,7 @@ Inspect + drive from the CLI: **`wildpc mesh status`** / **`wildpc mesh nodes`** | Why wildpc is shaped this way | **`docs/design.md`** | | DNS resolution + the two TLS modes, acme recipe | **`docs/dns-and-tls.md`** | | Public exposure (cloudflared) one-time setup | **`docs/tunnel-setup.md`** | +| Mesh operations: health, stale nodes, sealed vault, adding a node | **`docs/mesh.md`** | | Writing FastAPI services | **`docs/stacks/python-fastapi.md`** | | Writing CLI tools | **`docs/stacks/python-cli.md`** | | Writing React/Vite frontends | **`docs/stacks/react-vite.md`** | diff --git a/docs/fleet-mesh-plan.md b/docs/fleet-mesh-plan.md deleted file mode 100644 index f8e855d..0000000 --- a/docs/fleet-mesh-plan.md +++ /dev/null @@ -1,314 +0,0 @@ -# Fleet Mesh Plan — OpenBao + NATS - -**Status:** Phases 0–4 complete + verified live across **both nodes** (civil + -primer), including TLS hardening (NATS + OpenBao). Merged to main. -Remaining nice-to-have: the live curl+kill breaker demo (needs a peer-unique -consumed service) — every underlying piece is verified. - -## Context - -Wild PC's mesh today is a deliberately minimal, read-only gossip: each node -publishes a secret-stripped `NodeRegistry` to Mosquitto MQTT (retained) + an -online/offline LWT, and peers aggregate it into `MeshStateManager`. It is -LAN-only, unauthenticated, carries no secrets, and does nothing with what it -discovers beyond display. - -The goal is a **purpose-driven heterogeneous fleet** (media server, security -appliance, TV, comms handler, AI gateway) where nodes come and go as *expected*: -shared LAN-wide config + selected secrets, discoverability + presence, and -binding **by purpose** (a consumer needs `media-index`, not a hostname) with -**circuit-breakers** so churn degrades gracefully instead of hanging. - -Two adopted, genuinely-open components (see licensing analysis in chat history): - -- **NATS** (CNCF, Apache-2.0) — replaces Mosquitto as the mesh substrate and does - three jobs in one: pub/sub, **JetStream KV** (shared config + registry), and - **TTL keys as presence** (a provider's key vanishes on death → the breaker - trip signal, no separate health plumbing). -- **OpenBao** (Linux Foundation / OpenSSF, MPL-2.0, Vault fork) — the secret - authority behind wildpc's existing `${secret:...}` mechanism. - -Design stance: **static single-writer `role` authority, no consensus** — -availability comes from followers running cached local state, not failover. -NATS single-node matches that topology. Consensus stays explicitly out of scope. - -## Architecture — planes → components → code seams - -| Plane | Component | Primary code seam (verified) | -|-------|-----------|------------------------------| -| Transport + KV + presence | NATS (`wildpc-nats`) | `wildpc-api/.../mqtt_client.py` (whole file), `main.py` lifespan 62-77, `config.py` Settings | -| Shared config + registry + presence | NATS JetStream KV | `mesh.py` `MeshStateManager`; new KV buckets | -| Secret authority | OpenBao (`wildpc-openbao`) | `core/.../config.py:307` `_read_secret` (single chokepoint), `wildpc-api/.../secrets.py` CRUD | -| Binding-by-purpose + breaker | Caddy gateway | `generators/caddyfile.py` `compute_routes` (`remote` kind + ignored `remote_registries` are pre-stubbed), `deploy.py:509-540` `_target_url`/`_requires_env` (local-only today) | -| Authority role | `NodeConfig.role` | `wildpc_core/registry.py` / `manifest.py` | - -## Phased plan - -Each phase is independently useful; the risky secret-transport work is last. - -### Phase 0 — Stand up the two services (no behavior change) - -- **`wildpc-nats`**: a `SystemdDeployment` + `RunContainer` (`nats:2` with - JetStream `-js`), data volume `/data/wildpc/wildpc-nats`, `reach: internal` - (`nats.`). Follow the container YAML shape in `docs/tcp-exposure.md` - (postgres example). Replace the Mosquitto provisioning in `install.sh` - (`setup_mqtt`, `seed_mosquitto_config`, image/port consts) with NATS — or - better, promote it to a bootstrap deployment YAML (today the broker is only a - shell-provisioned container, not a declared deployment). -- **`wildpc-openbao`**: `RunContainer` (`openbao/openbao`), data volume, a file - storage backend, `reach: internal`, **auto-unseal** (local transit/key) so the - node boots unattended (Decision 2). - -### Phase 1 — NATS as the mesh transport (swap Mosquitto, behavior-preserving) - -- New `nats_client.py` (async-native `nats-py`) replacing `mqtt_client.py`. - Registry → a **JetStream KV bucket `wildpc-registry`** keyed by hostname - (KV last-value replaces MQTT retained). Presence → a **`wildpc-presence`** - bucket with a per-node **TTL key** the node renews (replaces LWT + the 300s - `STALE_TTL` poll; note `prune_stale` is currently uncalled). -- Wire in `main.py` lifespan; rename `mqtt_*` → `nats_*` in `config.py` (keep - `WILDPC_API_` prefix). `MeshStateManager` logic is reused; the KV **watch** - callback drives `update_node`/`set_offline` — and because `nats-py` is - asyncio-native, the `run_coroutine_threadsafe` cross-thread hop for - `broadcast("mesh", …)` **goes away**. -- **Preserve the secret-stripping invariant** from `_registry_to_json` (env / - run_cmd / wildpc_root never on the wire). -- Rename MQTT-named fields: `models.py` `MeshStatus` (217-226) + frontend - `types/index.ts` (326-330), `MeshPanel.tsx`. Drop `paho-mqtt`; add `nats-py`. - Keep zeroconf peer-advert for LAN discovery **and** support explicit NATS seed - URLs for cross-network (Decision 1); drop the dangling unused `_mqtt._tcp` - browse. Rewrite `test_mqtt.py` → `test_nats.py`. -- Fixes a latent gap: today the registry is published **on-connect only** (no - periodic/on-change republish). KV writes on `wildpc apply` fix this naturally. - -### Phase 2 — Shared config + presence via JetStream KV - -- Buckets: `wildpc-config` (shared LAN config, **authority-written only**), - `wildpc-registry` (per-node), `wildpc-presence` (TTL liveness). -- Add static **`role: authority | follower`** to `NodeConfig` (wildpc.yaml). - Pin **`civil` = authority** (Decision 3); only the authority may write - `wildpc-config`. Authority-down ⇒ shared state read-only, nodes serve cached. -- Followers **watch** `wildpc-config` and reconcile via `wildpc apply`. Presence - key renewal is the churn signal for Phase 3. - -### Phase 3 — Cross-node `requires` resolution + gateway binding + breaker (keystone) - -- Extend `_target_url`/`_requires_env` (`deploy.py:509-540`) to **fall through to - the mesh registry** when a `ref` isn't satisfied locally (today it's - local-only). The ref is already host-independent — this removes the - "must be local" restriction, not adds a concept. -- Produce the pre-stubbed **`remote` `GatewayRoute`**; thread `remote_registries` - into `compute_routes` (currently accepted-but-ignored) and into the Caddyfile - write at `deploy.py:157` (currently passed nothing). Binding = a **stable local - subdomain route** the gateway re-targets to the remote node's URL — the program - sees one unchanging URL. -- **Circuit-breaker:** gate the remote route on the `wildpc-presence` key - (regenerate + `_reload_gateway` when a provider appears/vanishes), backed by - Caddy passive health (`fail_duration` / `lb_try_duration`) on the - `reverse_proxy` line in `_host_matcher_block`. Consumer degraded-mode = - fallback target or clean 503. (Also carry `health_url` into the registry — - it's currently dropped on `manager: none` refs.) - -### Phase 4 — OpenBao as the secret backend (last; needs the security foundation) - -- Introduce a **`SecretBackend` seam** at the single read chokepoint - `_read_secret` (`config.py:307`) + the `wildpc-api/.../secrets.py` writer: - `FileSecretBackend` (default, unchanged) and `OpenBaoBackend`. `${secret:NAME}` - syntax is untouched; backend selected in wildpc.yaml or by env. -- Followers auth to OpenBao (token / AppRole); **need-to-know scoping** via - per-deployment policies. Keep the file backend as fallback; migrate secrets in. - -### Cross-cutting prerequisite (gates Phase 4 + any cross-network) - -- **Transport hardening**: NATS TLS + nkey/user auth (replace Mosquitto's - `allow_anonymous`); OpenBao TLS listener. No secret and no cross-network link - moves until this is in. Cross-network overlay (e.g. Nebula) is orthogonal and - layers under Caddy later. - -## Verification (per phase, end-to-end) - -- **P1:** bring up two nodes; confirm registry + presence propagate over NATS and - the mesh view (`/mesh/status`, System Map) is identical to the MQTT behavior. -- **P2:** write a key to `wildpc-config` on the authority; observe a follower - watch fire and `wildpc apply` reconcile. -- **P3:** define a service on node A that `requires` a ref provided on node B; - confirm the gateway routes cross-node, then **kill node B** and confirm the - consumer fails fast (curl returns the degraded 503, not a hang) and **recovers** - when B returns. -- **P4:** store a secret in OpenBao; confirm a service renders it via - `EnvironmentFile=` / `--env-file`; then disable the file fallback and confirm - it still resolves. - -## Progress - -### Phase 0 — DONE + verified (2026-07-07) - -Both services stood up **alongside** the live MQTT mesh (no cutover yet): - -- `wildpc-nats` (`nats:2`, JetStream) — deployment - `~/.wildpc/deployments/services/wildpc-nats.yaml`, config - `/data/wildpc/wildpc-nats/config/nats-server.conf`. Verified: container active, - `/healthz` ok, JetStream enabled (`/jsz` shows store dir + limits), and a real - **KV round-trip** — created `wildpc-registry` (put/get) and a TTL - `wildpc-presence` bucket (the presence primitive). Test buckets cleaned up. -- `wildpc-openbao` (`openbao/openbao:latest`, v2.5.5) — deployment - `~/.wildpc/deployments/services/wildpc-openbao.yaml`, config - `/data/wildpc/wildpc-openbao/config/openbao.hcl`. Verified: initialized (1 - share / threshold 1), unsealed, **KV-v2 secret put/get/delete** round-trip. - Unseal key + root token stored as wildpc secrets `OPENBAO_UNSEAL_KEY` / - `OPENBAO_ROOT_TOKEN`. - -**Consciously deferred (tracked):** -1. **Auto-unseal on boot.** OpenBao is unsealed now but a container restart - re-seals it. The manifest `SystemdSpec` has no `exec_start_post` hook, so - proper boot-unseal wiring lands with Phase 4 (either a small manifest - addition or a companion oneshot). Manual recovery: unseal with - `OPENBAO_UNSEAL_KEY`. -2. **`install.sh` bootstrap parity.** Fresh-install provisioning still seeds - Mosquitto; adding NATS/OpenBao there (and retiring Mosquitto) is bootstrap-only - (no runtime impact) and folds in with the Phase 1 cutover. - -### Phase 1 — DONE + single-node verified; live cutover done (2026-07-07) - -MQTT/Mosquitto transport replaced by NATS JetStream KV. - -- New `wildpc_api/mesh_wire.py` — transport-agnostic registry (de)serialization - (moved out of `mqtt_client.py`; preserves the secret-stripping invariant). -- New `wildpc_api/nats_client.py` — `WildpcNATSClient`: connects, PUTs its - registry to the `wildpc-registry` KV bucket, seeds from existing keys, watches - for peer PUT/DELETE, heartbeats a re-PUT (crash liveness via the existing - stale-TTL), and DELETEs its key on graceful stop (immediate peer-offline). - Async-native → the paho cross-thread `run_coroutine_threadsafe` hop is gone. -- `main.py` lifespan, `config.py` (`nats_enabled`/`nats_url`), `models.py` - `MeshStatus` (`connected`/`nats_url`), `nodes.py` `/mesh/status` all rewired. -- Frontend: `types/index.ts` `MeshStatus` + `MeshPanel.tsx` updated to the new - fields. `mqtt_client.py` + `test_mqtt.py` deleted; `test_mesh_wire.py` added. - `mdns.py` dead `_mqtt._tcp` broker-browse removed. `paho-mqtt` → `nats-py`. -- **Live cutover:** `wildpc-api.yaml` env → `WILDPC_API_NATS_*`, `requires: - wildpc-nats`; applied. wildpc-api healthy on NATS, `/mesh/status` connected, - registry (9.7 KiB, 48 deployments) published to KV, **no secrets on the wire**. -- Verification run: ruff clean; 89 api + 196 core tests pass; frontend `tsc` - clean; runtime KV round-trip confirmed against the live `wildpc-nats`. - -**Pending second node:** two-node mesh-view parity (peer sees peer, offline on -departure) — needs the second LAN node running the NATS-enabled wildpc-api -pointed at civil's NATS (seed URL) or a clustered `wildpc-nats`. Revert path if -needed: restore `WILDPC_API_MQTT_*` env — but note `mqtt_client.py` is deleted, -so revert = `git checkout main -- wildpc-api` then re-apply. The old Mosquitto -`mqtt` service is left running (dormant) as a safety net; retire it once the -second node is proven on NATS. - -### Phase 2 — DONE + single-node verified + tested (2026-07-07) - -- **`role` field** on `NodeConfig` + `WildpcConfig`, wired end-to-end: - wildpc.yaml top-level `role:` → config → `_node_config` → registry.yaml → - mesh wire. **`civil` pinned `role: authority`**; default `follower`. -- **Presence** (`wildpc-presence`) — a TTL KV bucket each node renews on the - heartbeat; expiry = the node is gone. Delete-on-stop for immediate departure. -- **Shared config** (`wildpc-config`) — `get_shared_config` / `put_shared_config` - (authority-gated: followers raise `PermissionError`), plus a watch loop that - broadcasts a `config_changed` SSE (the follower `wildpc apply` reconcile hook - hangs off this — inert on one node). -- Hardened `stop()` to bound the NATS drain (can't hang systemd shutdown). -- Verified live: all three buckets present, `civil=online` presence, - `"role": "authority"` on the wire, authority write + follower-deny exercised - against the running `wildpc-nats`. -- **Tests added:** `core/tests/test_fleet_role.py` (role config load + registry - round-trip), `wildpc-api/tests/test_nats_client.py` (role gating, hermetic), - role round-trip in `test_mesh_wire.py`. Suites: **200 core + 93 api** pass. - -**Pending second node:** the follower-side reconcile (watch `wildpc-config` → -`wildpc apply`) is wired as an SSE hook but only meaningful with a peer. - -### Phase 4 — secret-read backend DONE + verified + tested (2026-07-07) - -- New `core/wildpc_core/secret_backends.py`: `SecretBackend` protocol, - `FileSecretBackend` (the historical behavior), `OpenBaoBackend` (KV-v2 read with - file fallback), and `build_backend()` selecting via `WILDPC_SECRET_BACKEND` - (default **file** — production is byte-for-byte unchanged until opted in). -- `_read_secret` (the single chokepoint, `config.py`) now delegates to the active - backend. `${secret:NAME}` syntax untouched. -- OpenBao token bootstraps from the file backend (it can't live in the vault it - unlocks); a missing key / auth failure / unreachable server all fall through to - file, so a partly-migrated vault keeps working. -- Verified live against the running `wildpc-openbao`: a secret stored in the vault - resolves through `${secret:...}` in openbao mode; file-only secrets resolve via - fallback; missing → placeholder; **default file mode unchanged**. -- **Tests:** `core/tests/test_secret_backends.py` (file hit/miss, backend - selection, unreachable-vault + empty-token fallback). Suites: **206 core + 93 api**. - -**Hardening:** -1. **Write path — DONE + verified live.** `SecretBackend` gained - `write`/`delete`/`list_names`; `wildpc-api/secrets.py` routes all CRUD through - the active backend, so in openbao mode the dashboard writes to the vault. - Verified: write→vault→read→list→delete round-trip against live OpenBao. -2. **Auto-unseal on boot — DONE + verified.** Added `exec_start_post` to the - systemd spec (general, `-`-prefixed so a hook failure never fails the unit); - `wildpc-openbao` runs `/data/wildpc/wildpc-openbao/unseal.sh` (polls up, unseals - with `OPENBAO_UNSEAL_KEY`). Verified: manual seal → restart → auto-unsealed. -3. **TLS hardening — DONE + verified across both nodes (2026-07-07).** - Reuses the existing ACME **wildcard cert** via `expose.tcp.tls` (the postgres - mechanism) — services present a publicly-trusted cert for - `.civil.payne.io`, so clients verify against the **system CA with no - custom CA to distribute**. - - **NATS:** `tls{}` (wildcard cert in `/tls`) + `authorization{token}` - (`$NATS_TOKEN` from the `NATS_TOKEN` secret). Clients connect to - `tls://wildpc-nats.civil.payne.io:4222` with the token; `WildpcNATSClient` - gained token support (a `tls://` URL → nats-py verifies via system CA). - **civil + primer both cut over**; plaintext now rejected (verified: - `certificate is valid for *.civil.payne.io, not localhost`). - - **OpenBao:** listener `tls_cert_file`/`tls_key_file` from `/tls`; reachable at - `https://wildpc-openbao.civil.payne.io:8200`. Auto-unseal + secret backend - both use the HTTPS URL. Verified: HTTPS serves, **auto-unsealed over HTTPS**, - plaintext rejected, backend reads a secret over HTTPS. - - Coordinated cutover across the whole fleet (both nodes) with a brief, - expected mesh blip; mesh isn't load-bearing so no service impact. - -### Phase 3 — cross-node routing + breaker: logic DONE + verified against a real peer (2026-07-07) - -**Real second node:** `primer` (192.168.8.129) migrated onto the NATS mesh -(branch checked out, wildpc-api pointed at `nats://civil:4222` via a systemd -drop-in). civil ↔ primer mesh confirmed over the LAN — **Phase 1 two-node parity -done for real**, not simulated. (This also *restored* the civil↔primer mesh my -Phase 1 cutover had split — primer was still on MQTT→civil.) - -- `NodeConfig.address` — routable host peers proxy to (wired through registry + - wire; falls back to hostname, which civil resolves for primer). -- `compute_routes` now emits **`remote`** routes for services this node - **consumes** (a local `requires` ref satisfied by an online peer), targeting - `:`. `_host_remote_block` renders a **fail-fast breaker** - (2s dial timeout + passive health) for the there-but-wedged case; **presence - expiry removes the peer from the route set entirely** (gone → no route) — the - primary breaker. -- `wildpc_api/mesh_gateway.py` — on peer join/leave/change (+ startup) the API - re-renders the Caddyfile (same generator `apply` uses + remote routes for - online peers) and reloads the gateway **iff content changed**. -- **Verified:** 4 hermetic tests (route emitted / address fallback / breaker: - peer-absent → no route / unconsumed → no route); **resolution against primer's - real registry** pulled live from the mesh → `wildpc-api → primer:9020`; and the - live integration proven a **no-op** on civil (Caddyfile hash unchanged, gateway - healthy) since civil consumes nothing cross-node yet. Suites: **210 core + 93 api**. - -**Remaining:** the full live curl+kill E2E (civil routing to a peer service, then -failing fast on kill) needs a peer-unique service civil consumes — every -underlying piece is verified, but the end-to-end demo needs that provisioning. -primer is now a permanent mesh member on the branch (revert: remove the drop-in + -`git checkout main` on primer). - -## Decisions (resolved) - -1. **Discovery — both.** Keep mDNS for zero-config LAN peer discovery *and* - support explicit NATS seed URLs so the mesh can span networks. mDNS is the - convenience path on-LAN; seed URLs are the cross-network path. Client connects - via seeds when present, falls back to mDNS-discovered peers on-LAN. -2. **OpenBao unseal — auto for now.** Auto-unseal (local transit/key) so a home - node boots unattended; designed so we can migrate to manual/stricter unseal - later (a supported rekey/unseal-migration, not zero-effort but planned for). - Keep the backend seam clean so the switch is config, not code. -3. **Authority = `civil`.** `civil` (the acme node, `civil.payne.io`) is pinned - as `role: authority`; all others are followers. **Confirmed:** when `civil` is - down, shared config/secrets go **read-only** fleet-wide and every node keeps - serving its own deployments from cached local state. -4. **NATS — single server now, cluster-ready.** Run one `wildpc-nats` today - (matches the single-writer authority). Clustering is deferred but must be a - pure config addition (cluster/routes block + a few more nodes) — no re-architecture. diff --git a/docs/mesh.md b/docs/mesh.md new file mode 100644 index 0000000..3d95147 --- /dev/null +++ b/docs/mesh.md @@ -0,0 +1,103 @@ +# Mesh — runbook + +How the nodes find each other, share config and secrets, and route to each +other's services — and what to do when that breaks. The design and its history +are in git: `git log -p -- docs/fleet-mesh-plan.md`. + +## What runs where + +| piece | where | what it does | +|---|---|---| +| `castle-nats` (civil) | `tls://castle-nats.civil.payne.io:4222`, token `NATS_TOKEN`; monitoring `:8222` | NATS JetStream: the mesh transport and its KV buckets | +| `castle-openbao` (civil) | `https://castle-openbao.civil.payne.io:8200`, KV-v2 mount `castle` | the secret authority behind `${secret:…}` | +| `wildpc-api` (every node) | env `WILDPC_API_NATS_ENABLED/URL/TOKEN` | the mesh client: publishes this node, watches the others | + +Config and data: `/data/castle/castle-nats/config/nats-server.conf`, +`/data/castle/castle-openbao/{config/openbao.hcl,unseal.sh,data}`. Both present +the node's ACME wildcard cert (`expose.tcp.tls`), so clients verify against the +system CA; plaintext is refused. + +**Nodes.** `civil` is the **authority** (`role: authority` in `wildpc.yaml`); +every other node is a follower (`primer` today). There is no consensus: when +civil is down, shared config and secrets are read-only and every node keeps +running its own deployments from local state. + +**KV buckets** on NATS: + +| bucket | holds | written by | +|---|---|---| +| `wildpc-registry` | each node's registry, secrets stripped | each node, for itself | +| `wildpc-presence` | a key per node, TTL 90 s, renewed on a heartbeat | each node, for itself | +| `wildpc-config` | shared fleet config | the authority only | + +**Secrets.** `wildpc.yaml`'s `secrets:` block selects OpenBao (`backend: openbao`, +`mount: castle`). There is **no file fallback** in that mode — only the bootstrap +tier stays as files in `~/.wildpc/secrets/` (`OPENBAO_ROOT_TOKEN`, +`OPENBAO_UNSEAL_KEY`). civil reads with the root token; followers with a +least-privilege `wildpc-read` token, and `node_prefix` for per-node overrides. +Write with `wildpc secret set`, never by hand (see AGENTS.md §6). + +## Is it healthy? + +```bash +wildpc mesh status # connected? transport URL, peers +wildpc mesh nodes # each node: online | stale | offline +curl -s localhost:8222/healthz # NATS up +curl -s https://castle-openbao.civil.payne.io:8200/v1/sys/seal-status | jq .sealed +``` + +- **online** — the node's presence key is live. +- **stale** — no registry update from it for over 5 minutes, but not yet gone. +- **offline** — its presence key expired (90 s) or it left cleanly. + +A peer's services are routed only while it is present; when it goes, its routes +drop out and the gateway answers `404 no wildpc route for this host` +immediately, rather than hanging. + +## When something is wrong + +**A node shows `stale` or `offline`.** On that node: is `wildpc-api` running +(`wildpc service logs wildpc-api`)? Can it reach +`castle-nats.civil.payne.io:4222` by that name (TLS checks the name, so an IP +will not do)? Does its `WILDPC_API_NATS_TOKEN` resolve to the current +`NATS_TOKEN`? + +**OpenBao is sealed** (after a restart, if `unseal.sh` failed): every +`${secret:…}` read fails and `wildpc apply` refuses to converge. Unseal by hand: + +```bash +curl -s -X POST https://castle-openbao.civil.payne.io:8200/v1/sys/unseal \ + -d "{\"key\":\"$(cat ~/.wildpc/secrets/OPENBAO_UNSEAL_KEY)\"}" | jq .sealed +``` + +**NATS down.** The mesh view freezes and peers go offline; nothing else stops — +each node keeps serving. `wildpc restart castle-nats`, then +`wildpc mesh status` on each node. + +**The certificate renewed and a client fails TLS.** `wildpc tls reconcile` +re-materializes the wildcard onto both services and restarts them. + +## Routine tasks + +**Add a node.** Install Wild PC on it; on its `wildpc-api` set +`WILDPC_API_NATS_ENABLED=true`, +`WILDPC_API_NATS_URL=tls://castle-nats.civil.payne.io:4222`, +`WILDPC_API_NATS_TOKEN=${secret:NATS_TOKEN}`; give it a `secrets:` block with a +`wildpc-read` token and `node_prefix: nodes/`; leave `role` unset +(follower); `wildpc apply wildpc-api`. It appears in `wildpc mesh nodes` within a +heartbeat. + +**Use a service another node provides.** `requires: - ref: ` in the +consumer's deployment. If no local deployment satisfies it and a present peer +does, the gateway routes `.` to that peer, with a 2 s dial +timeout so a wedged peer fails fast. + +**Shared config.** `wildpc mesh config list|get|set` — `set` works on the +authority only. + +**Rotate the NATS token.** `wildpc secret set NATS_TOKEN`, then +`wildpc apply castle-nats` and `wildpc apply wildpc-api` on every node; peers +drop until each is re-applied. + +**Cluster NATS** (not done): add a `cluster {}` block and routes to +`nats-server.conf` on each server — configuration only, by design.