docs: fleet mesh plan (OpenBao + NATS)
Phased plan to evolve the mesh into a purpose-driven fleet: NATS (JetStream KV + presence) replacing Mosquitto, OpenBao as the secret authority, cross-node requires resolution + gateway circuit-breaker, static civil=authority role. Records resolved decisions and Phase 0/1 progress.
This commit is contained in:
215
docs/fleet-mesh-plan.md
Normal file
215
docs/fleet-mesh-plan.md
Normal file
@@ -0,0 +1,215 @@
|
|||||||
|
# Fleet Mesh Plan — OpenBao + NATS
|
||||||
|
|
||||||
|
**Status:** in progress on branch `feat/fleet-mesh-nats-openbao`.
|
||||||
|
Phases 0–1 complete + single-node verified (live cutover done). Phases 2–4 pending.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Castle's mesh today is a deliberately minimal, read-only gossip: each node
|
||||||
|
publishes a secret-stripped `NodeRegistry` to Mosquitto MQTT (retained) + an
|
||||||
|
online/offline LWT, and peers aggregate it into `MeshStateManager`. It is
|
||||||
|
LAN-only, unauthenticated, carries no secrets, and does nothing with what it
|
||||||
|
discovers beyond display.
|
||||||
|
|
||||||
|
The goal is a **purpose-driven heterogeneous fleet** (media server, security
|
||||||
|
appliance, TV, comms handler, AI gateway) where nodes come and go as *expected*:
|
||||||
|
shared LAN-wide config + selected secrets, discoverability + presence, and
|
||||||
|
binding **by purpose** (a consumer needs `media-index`, not a hostname) with
|
||||||
|
**circuit-breakers** so churn degrades gracefully instead of hanging.
|
||||||
|
|
||||||
|
Two adopted, genuinely-open components (see licensing analysis in chat history):
|
||||||
|
|
||||||
|
- **NATS** (CNCF, Apache-2.0) — replaces Mosquitto as the mesh substrate and does
|
||||||
|
three jobs in one: pub/sub, **JetStream KV** (shared config + registry), and
|
||||||
|
**TTL keys as presence** (a provider's key vanishes on death → the breaker
|
||||||
|
trip signal, no separate health plumbing).
|
||||||
|
- **OpenBao** (Linux Foundation / OpenSSF, MPL-2.0, Vault fork) — the secret
|
||||||
|
authority behind castle's existing `${secret:...}` mechanism.
|
||||||
|
|
||||||
|
Design stance: **static single-writer `role` authority, no consensus** —
|
||||||
|
availability comes from followers running cached local state, not failover.
|
||||||
|
NATS single-node matches that topology. Consensus stays explicitly out of scope.
|
||||||
|
|
||||||
|
## Architecture — planes → components → code seams
|
||||||
|
|
||||||
|
| Plane | Component | Primary code seam (verified) |
|
||||||
|
|-------|-----------|------------------------------|
|
||||||
|
| Transport + KV + presence | NATS (`castle-nats`) | `castle-api/.../mqtt_client.py` (whole file), `main.py` lifespan 62-77, `config.py` Settings |
|
||||||
|
| Shared config + registry + presence | NATS JetStream KV | `mesh.py` `MeshStateManager`; new KV buckets |
|
||||||
|
| Secret authority | OpenBao (`castle-openbao`) | `core/.../config.py:307` `_read_secret` (single chokepoint), `castle-api/.../secrets.py` CRUD |
|
||||||
|
| Binding-by-purpose + breaker | Caddy gateway | `generators/caddyfile.py` `compute_routes` (`remote` kind + ignored `remote_registries` are pre-stubbed), `deploy.py:509-540` `_target_url`/`_requires_env` (local-only today) |
|
||||||
|
| Authority role | `NodeConfig.role` | `castle_core/registry.py` / `manifest.py` |
|
||||||
|
|
||||||
|
## Phased plan
|
||||||
|
|
||||||
|
Each phase is independently useful; the risky secret-transport work is last.
|
||||||
|
|
||||||
|
### Phase 0 — Stand up the two services (no behavior change)
|
||||||
|
|
||||||
|
- **`castle-nats`**: a `SystemdDeployment` + `RunContainer` (`nats:2` with
|
||||||
|
JetStream `-js`), data volume `/data/castle/castle-nats`, `reach: internal`
|
||||||
|
(`nats.<domain>`). Follow the container YAML shape in `docs/tcp-exposure.md`
|
||||||
|
(postgres example). Replace the Mosquitto provisioning in `install.sh`
|
||||||
|
(`setup_mqtt`, `seed_mosquitto_config`, image/port consts) with NATS — or
|
||||||
|
better, promote it to a bootstrap deployment YAML (today the broker is only a
|
||||||
|
shell-provisioned container, not a declared deployment).
|
||||||
|
- **`castle-openbao`**: `RunContainer` (`openbao/openbao`), data volume, a file
|
||||||
|
storage backend, `reach: internal`, **auto-unseal** (local transit/key) so the
|
||||||
|
node boots unattended (Decision 2).
|
||||||
|
|
||||||
|
### Phase 1 — NATS as the mesh transport (swap Mosquitto, behavior-preserving)
|
||||||
|
|
||||||
|
- New `nats_client.py` (async-native `nats-py`) replacing `mqtt_client.py`.
|
||||||
|
Registry → a **JetStream KV bucket `castle-registry`** keyed by hostname
|
||||||
|
(KV last-value replaces MQTT retained). Presence → a **`castle-presence`**
|
||||||
|
bucket with a per-node **TTL key** the node renews (replaces LWT + the 300s
|
||||||
|
`STALE_TTL` poll; note `prune_stale` is currently uncalled).
|
||||||
|
- Wire in `main.py` lifespan; rename `mqtt_*` → `nats_*` in `config.py` (keep
|
||||||
|
`CASTLE_API_` prefix). `MeshStateManager` logic is reused; the KV **watch**
|
||||||
|
callback drives `update_node`/`set_offline` — and because `nats-py` is
|
||||||
|
asyncio-native, the `run_coroutine_threadsafe` cross-thread hop for
|
||||||
|
`broadcast("mesh", …)` **goes away**.
|
||||||
|
- **Preserve the secret-stripping invariant** from `_registry_to_json` (env /
|
||||||
|
run_cmd / castle_root never on the wire).
|
||||||
|
- Rename MQTT-named fields: `models.py` `MeshStatus` (217-226) + frontend
|
||||||
|
`types/index.ts` (326-330), `MeshPanel.tsx`. Drop `paho-mqtt`; add `nats-py`.
|
||||||
|
Keep zeroconf peer-advert for LAN discovery **and** support explicit NATS seed
|
||||||
|
URLs for cross-network (Decision 1); drop the dangling unused `_mqtt._tcp`
|
||||||
|
browse. Rewrite `test_mqtt.py` → `test_nats.py`.
|
||||||
|
- Fixes a latent gap: today the registry is published **on-connect only** (no
|
||||||
|
periodic/on-change republish). KV writes on `castle apply` fix this naturally.
|
||||||
|
|
||||||
|
### Phase 2 — Shared config + presence via JetStream KV
|
||||||
|
|
||||||
|
- Buckets: `castle-config` (shared LAN config, **authority-written only**),
|
||||||
|
`castle-registry` (per-node), `castle-presence` (TTL liveness).
|
||||||
|
- Add static **`role: authority | follower`** to `NodeConfig` (castle.yaml).
|
||||||
|
Pin **`civil` = authority** (Decision 3); only the authority may write
|
||||||
|
`castle-config`. Authority-down ⇒ shared state read-only, nodes serve cached.
|
||||||
|
- Followers **watch** `castle-config` and reconcile via `castle apply`. Presence
|
||||||
|
key renewal is the churn signal for Phase 3.
|
||||||
|
|
||||||
|
### Phase 3 — Cross-node `requires` resolution + gateway binding + breaker (keystone)
|
||||||
|
|
||||||
|
- Extend `_target_url`/`_requires_env` (`deploy.py:509-540`) to **fall through to
|
||||||
|
the mesh registry** when a `ref` isn't satisfied locally (today it's
|
||||||
|
local-only). The ref is already host-independent — this removes the
|
||||||
|
"must be local" restriction, not adds a concept.
|
||||||
|
- Produce the pre-stubbed **`remote` `GatewayRoute`**; thread `remote_registries`
|
||||||
|
into `compute_routes` (currently accepted-but-ignored) and into the Caddyfile
|
||||||
|
write at `deploy.py:157` (currently passed nothing). Binding = a **stable local
|
||||||
|
subdomain route** the gateway re-targets to the remote node's URL — the program
|
||||||
|
sees one unchanging URL.
|
||||||
|
- **Circuit-breaker:** gate the remote route on the `castle-presence` key
|
||||||
|
(regenerate + `_reload_gateway` when a provider appears/vanishes), backed by
|
||||||
|
Caddy passive health (`fail_duration` / `lb_try_duration`) on the
|
||||||
|
`reverse_proxy` line in `_host_matcher_block`. Consumer degraded-mode =
|
||||||
|
fallback target or clean 503. (Also carry `health_url` into the registry —
|
||||||
|
it's currently dropped on `manager: none` refs.)
|
||||||
|
|
||||||
|
### Phase 4 — OpenBao as the secret backend (last; needs the security foundation)
|
||||||
|
|
||||||
|
- Introduce a **`SecretBackend` seam** at the single read chokepoint
|
||||||
|
`_read_secret` (`config.py:307`) + the `castle-api/.../secrets.py` writer:
|
||||||
|
`FileSecretBackend` (default, unchanged) and `OpenBaoBackend`. `${secret:NAME}`
|
||||||
|
syntax is untouched; backend selected in castle.yaml or by env.
|
||||||
|
- Followers auth to OpenBao (token / AppRole); **need-to-know scoping** via
|
||||||
|
per-deployment policies. Keep the file backend as fallback; migrate secrets in.
|
||||||
|
|
||||||
|
### Cross-cutting prerequisite (gates Phase 4 + any cross-network)
|
||||||
|
|
||||||
|
- **Transport hardening**: NATS TLS + nkey/user auth (replace Mosquitto's
|
||||||
|
`allow_anonymous`); OpenBao TLS listener. No secret and no cross-network link
|
||||||
|
moves until this is in. Cross-network overlay (e.g. Nebula) is orthogonal and
|
||||||
|
layers under Caddy later.
|
||||||
|
|
||||||
|
## Verification (per phase, end-to-end)
|
||||||
|
|
||||||
|
- **P1:** bring up two nodes; confirm registry + presence propagate over NATS and
|
||||||
|
the mesh view (`/mesh/status`, System Map) is identical to the MQTT behavior.
|
||||||
|
- **P2:** write a key to `castle-config` on the authority; observe a follower
|
||||||
|
watch fire and `castle apply` reconcile.
|
||||||
|
- **P3:** define a service on node A that `requires` a ref provided on node B;
|
||||||
|
confirm the gateway routes cross-node, then **kill node B** and confirm the
|
||||||
|
consumer fails fast (curl returns the degraded 503, not a hang) and **recovers**
|
||||||
|
when B returns.
|
||||||
|
- **P4:** store a secret in OpenBao; confirm a service renders it via
|
||||||
|
`EnvironmentFile=` / `--env-file`; then disable the file fallback and confirm
|
||||||
|
it still resolves.
|
||||||
|
|
||||||
|
## Progress
|
||||||
|
|
||||||
|
### Phase 0 — DONE + verified (2026-07-07)
|
||||||
|
|
||||||
|
Both services stood up **alongside** the live MQTT mesh (no cutover yet):
|
||||||
|
|
||||||
|
- `castle-nats` (`nats:2`, JetStream) — deployment
|
||||||
|
`~/.castle/deployments/services/castle-nats.yaml`, config
|
||||||
|
`/data/castle/castle-nats/config/nats-server.conf`. Verified: container active,
|
||||||
|
`/healthz` ok, JetStream enabled (`/jsz` shows store dir + limits), and a real
|
||||||
|
**KV round-trip** — created `castle-registry` (put/get) and a TTL
|
||||||
|
`castle-presence` bucket (the presence primitive). Test buckets cleaned up.
|
||||||
|
- `castle-openbao` (`openbao/openbao:latest`, v2.5.5) — deployment
|
||||||
|
`~/.castle/deployments/services/castle-openbao.yaml`, config
|
||||||
|
`/data/castle/castle-openbao/config/openbao.hcl`. Verified: initialized (1
|
||||||
|
share / threshold 1), unsealed, **KV-v2 secret put/get/delete** round-trip.
|
||||||
|
Unseal key + root token stored as castle secrets `OPENBAO_UNSEAL_KEY` /
|
||||||
|
`OPENBAO_ROOT_TOKEN`.
|
||||||
|
|
||||||
|
**Consciously deferred (tracked):**
|
||||||
|
1. **Auto-unseal on boot.** OpenBao is unsealed now but a container restart
|
||||||
|
re-seals it. The manifest `SystemdSpec` has no `exec_start_post` hook, so
|
||||||
|
proper boot-unseal wiring lands with Phase 4 (either a small manifest
|
||||||
|
addition or a companion oneshot). Manual recovery: unseal with
|
||||||
|
`OPENBAO_UNSEAL_KEY`.
|
||||||
|
2. **`install.sh` bootstrap parity.** Fresh-install provisioning still seeds
|
||||||
|
Mosquitto; adding NATS/OpenBao there (and retiring Mosquitto) is bootstrap-only
|
||||||
|
(no runtime impact) and folds in with the Phase 1 cutover.
|
||||||
|
|
||||||
|
### Phase 1 — DONE + single-node verified; live cutover done (2026-07-07)
|
||||||
|
|
||||||
|
MQTT/Mosquitto transport replaced by NATS JetStream KV.
|
||||||
|
|
||||||
|
- New `castle_api/mesh_wire.py` — transport-agnostic registry (de)serialization
|
||||||
|
(moved out of `mqtt_client.py`; preserves the secret-stripping invariant).
|
||||||
|
- New `castle_api/nats_client.py` — `CastleNATSClient`: connects, PUTs its
|
||||||
|
registry to the `castle-registry` KV bucket, seeds from existing keys, watches
|
||||||
|
for peer PUT/DELETE, heartbeats a re-PUT (crash liveness via the existing
|
||||||
|
stale-TTL), and DELETEs its key on graceful stop (immediate peer-offline).
|
||||||
|
Async-native → the paho cross-thread `run_coroutine_threadsafe` hop is gone.
|
||||||
|
- `main.py` lifespan, `config.py` (`nats_enabled`/`nats_url`), `models.py`
|
||||||
|
`MeshStatus` (`connected`/`nats_url`), `nodes.py` `/mesh/status` all rewired.
|
||||||
|
- Frontend: `types/index.ts` `MeshStatus` + `MeshPanel.tsx` updated to the new
|
||||||
|
fields. `mqtt_client.py` + `test_mqtt.py` deleted; `test_mesh_wire.py` added.
|
||||||
|
`mdns.py` dead `_mqtt._tcp` broker-browse removed. `paho-mqtt` → `nats-py`.
|
||||||
|
- **Live cutover:** `castle-api.yaml` env → `CASTLE_API_NATS_*`, `requires:
|
||||||
|
castle-nats`; applied. castle-api healthy on NATS, `/mesh/status` connected,
|
||||||
|
registry (9.7 KiB, 48 deployments) published to KV, **no secrets on the wire**.
|
||||||
|
- Verification run: ruff clean; 89 api + 196 core tests pass; frontend `tsc`
|
||||||
|
clean; runtime KV round-trip confirmed against the live `castle-nats`.
|
||||||
|
|
||||||
|
**Pending second node:** two-node mesh-view parity (peer sees peer, offline on
|
||||||
|
departure) — needs the second LAN node running the NATS-enabled castle-api
|
||||||
|
pointed at civil's NATS (seed URL) or a clustered `castle-nats`. Revert path if
|
||||||
|
needed: restore `CASTLE_API_MQTT_*` env — but note `mqtt_client.py` is deleted,
|
||||||
|
so revert = `git checkout main -- castle-api` then re-apply. The old Mosquitto
|
||||||
|
`mqtt` service is left running (dormant) as a safety net; retire it once the
|
||||||
|
second node is proven on NATS.
|
||||||
|
|
||||||
|
## Decisions (resolved)
|
||||||
|
|
||||||
|
1. **Discovery — both.** Keep mDNS for zero-config LAN peer discovery *and*
|
||||||
|
support explicit NATS seed URLs so the mesh can span networks. mDNS is the
|
||||||
|
convenience path on-LAN; seed URLs are the cross-network path. Client connects
|
||||||
|
via seeds when present, falls back to mDNS-discovered peers on-LAN.
|
||||||
|
2. **OpenBao unseal — auto for now.** Auto-unseal (local transit/key) so a home
|
||||||
|
node boots unattended; designed so we can migrate to manual/stricter unseal
|
||||||
|
later (a supported rekey/unseal-migration, not zero-effort but planned for).
|
||||||
|
Keep the backend seam clean so the switch is config, not code.
|
||||||
|
3. **Authority = `civil`.** `civil` (the acme node, `civil.payne.io`) is pinned
|
||||||
|
as `role: authority`; all others are followers. **Confirmed:** when `civil` is
|
||||||
|
down, shared config/secrets go **read-only** fleet-wide and every node keeps
|
||||||
|
serving its own deployments from cached local state.
|
||||||
|
4. **NATS — single server now, cluster-ready.** Run one `castle-nats` today
|
||||||
|
(matches the single-writer authority). Clustering is deferred but must be a
|
||||||
|
pure config addition (cluster/routes block + a few more nodes) — no re-architecture.
|
||||||
Reference in New Issue
Block a user