Compare commits

...

10 Commits

Author SHA1 Message Date
Paul Payne
c6c361e38a core: static sites serve a nested directory's index.html
`try_files {path} {path}/index.html /index.html` for SPA sites and the
dashboard: a URL like /docs/setup/ serves that directory's own index.html
before falling back to the root one, so a multi-page static site works at its
sub-paths while single-page apps still get client-side routing.

Found uncommitted in the checkout, author unknown, and already live: wildpc
runs from this tree, so every Caddyfile generated since used it.
2026-10-23 15:19:33 -07:00
Paul Payne
959a896115 docs: a mesh runbook in place of the finished plan
docs/mesh.md says what runs where (castle-nats, castle-openbao, each node's
wildpc-api), how to tell it is healthy, what to do when a node goes stale, the
vault is sealed or NATS is down, and the routine tasks -- written from the live
config, not the plan. The plan is done and goes; its history is in git, and its
unfinished items are tasks in Wild Life.

AGENTS.md used the services' old names (wildpc-nats, wildpc-openbao) and
pointed at the plan for operations; it now names castle-* and points here.
2026-10-23 15:07:11 -07:00
Paul Payne
01819b2984 core: a login at the gateway, and container ports bound to one address
For services with no authentication of their own -- a trace viewer, an admin
UI, an OTLP receiver -- which must not be open to the LAN just because they run
here.

- `expose.http.basic_auth: {user, password_hash}` has the gateway ask for HTTP
  Basic auth (Caddy's `basic_auth`, bcrypt from `caddy hash-password`) before
  proxying, on the internal name and any public one. It is recorded in the
  registry too, so a Caddyfile generated from the registry alone still asks
  rather than serving the route open. Checked against caddy 2.11: 401 without
  credentials, through with them.
- A container's `ports` may give the host side an address, `4318:
  127.0.0.1:4318` -> `-p 127.0.0.1:4318:4318`. A bare port still publishes on
  every interface, as before, and the docs now say that this means the LAN.
2026-10-22 10:13:44 -07:00
Paul Payne
517471605a core: drop two manifest fields that were never wired up
`SystemdSpec.readiness` (ReadinessHttpGet) and `SystemdSpec.requires` were
parsed and validated but read nowhere — no generator, CLI or API touched
either. Both looked load-bearing and were not:

- readiness duplicated `expose.http.health_path`, which wildpc-api already
  polls. wild-life-api.yaml declared both spellings of the same /health
  probe; only one did anything.
- requires would mean systemd `Requires=`, a lifecycle binding (stop
  postgres, stop every dependent). `manage.systemd.after` already gives the
  ordering callers actually want, and a deployment's own `requires:` is the
  relationship model, not systemd wiring.

Unknown keys parse as ignored, so existing deployment YAML keeps loading.
2026-10-16 10:42:54 -07:00
Paul Payne
0fefa143e4 core: bind a raw-TCP service as <name>.<domain>:<port>
A `{ref, bind}` to a TCP-exposed deployment (e.g. postgres) projected
nothing, so consumers hardcoded the docker bridge IP. Project the address
it's already documented as reachable at.
2026-10-16 03:11:38 -07:00
Paul Payne
55c2f269ec core: GPU containers, bind sub-paths, and stop containers on unit stop
- container launcher: `gpus:` passes through as docker `--gpus`.
- requires: `path:` is appended to the bound URL, for consumers that want a
  sub-path as their base (e.g. an OpenAI-compatible server's /v1).
- container services now get `ExecStop=-docker stop wildpc-<name>`. SIGTERM
  only reached the `docker run` client, so a stop/restart left the container
  running unsupervised and holding its name, and the next start failed with
  a name conflict until systemd gave up.
2026-10-16 03:10:01 -07:00
Paul Payne
8a9843658c core: back off and give up on crash-looping services
Generated service units took systemd's default start limit (5 starts in
10s), which can never trip when each failed attempt outlives the window.
openclaw failed ~5s in and restarted every ~5.5s, so the counter reset on
every cycle: 260k restarts at ~7.4s CPU each, over a core burned
continuously for days, with the unit stuck in `activating (auto-restart)`
rather than anything a status check would flag.

Service units now get exponential restart backoff (RestartSteps=8,
RestartMaxDelaySec=300) and a start limit sized against it (10 starts per
hour, in [Unit] where systemd reads it). A broken service retries with a
growing delay for ~17min — wide enough to ride out a boot-time dependency
race — then lands in `failed` where it's visible and stops consuming CPU.
All four values are SystemdSpec fields, overridable per deployment.

Timer-driven oneshot jobs are unaffected; they never restart. systemd
older than 254 ignores RestartSteps/RestartMaxDelaySec with a warning and
keeps the flat RestartSec, so the start limit still applies there.
2026-08-24 09:24:16 -07:00
Paul Payne
6cf25ad2c7 core: evaluate node semver ranges instead of guessing the leading major
resolve_node_bin() could not parse a range, so it fell back to "pin by the
first number in the string" and took the newest installed version of that
major. For openclaw's engines pin `>=22.19.0 <23 || >=23.11.0` that picked
v22.17.0 — below the range's own floor — and baked it into the service's
unit PATH. The daemon refused to start on it and crash-looped 260k+ times
(~7.4s CPU per attempt, every ~5.5s, since before the journal's retention
window). StartLimitBurst never tripped because five attempts span ~27s,
outside the 10s StartLimitIntervalUSec.

Parse the pin properly: split on `||`, expand each comparator (>=, >, <=,
<, =, ^, ~, hyphen ranges, x/* wildcards, partial versions) into bounds
with node-semver's partial semantics, and pick the newest installed version
satisfying any alternative. openclaw now resolves to v24.18.0 via its
second alternative. A pin this parser cannot read at all still falls back
to newest-installed rather than guessing a version.

When nothing satisfies the pin it now fails with the installed versions
listed, so the next occurrence is diagnosable at apply time rather than
six days later in a restart loop.
2026-08-21 10:37:26 -07:00
Paul Payne
6acd63cf0a api: load wildpc.yaml once per listing, not once per item
/services and /jobs called get_wildpc_root() + load_config() *inside* the
loop over registry.all(), so every item paid a full reparse of wildpc.yaml
and all 77 program/deployment YAMLs (~55ms registry + ~103ms config). With
38 services and 12 jobs on this node that was 6.8s and 3.0s per request,
uncached — the dashboard's /services and /scheduled pages each fire one of
these plus /status, which is where their >10s load came from.

Hoist the load to a single _try_load_config() call per request; the source
backfill and the non-deployed pass both read that one config. Responses are
byte-identical to before; both endpoints now serve in ~0.27s.
2026-08-21 10:21:51 -07:00
Paul Payne
15957677ff gateway: serve the dashboard at wild-pc.<domain>, 404 unrouted subdomains
The dashboard's deployment kept the pre-rename name `wildpc`, so it was
published at wildpc.<domain> while everything else had moved to Wild PC
naming. Rename the deployment and its program to `wild-pc` (the subdomain
is the deployment name) and follow the `_DASHBOARD` constant in both the
Caddyfile generator and doctor — that constant also drives the `:9000`
redirect and doctor's `wildpc program build` hint, which is why the
program is renamed alongside the deployment.

Browsing the old name exposed a second problem: in acme mode every route
lives inside one `*.<domain>` wildcard site, so an unrouted host still
matches the site and completes its TLS handshake. Caddy then ran off the
end of the block and returned a bare 200 with an empty body — a typo'd or
renamed subdomain looked *up* in a browser while rendering nothing. Each
wildcard site now ends with an unmatched `handle` that responds 404.

The API subdomain is untouched: the dashboard derives its API origin by
stripping the first label, so wild-pc.<domain> still resolves to
wildpc-api.<domain>, and the WebSocket origin gate matches on
gateway.domain rather than a hostname.

Also seed the bootstrap dashboard deployment with the `requires:
wildpc-api` block the live deployment already carried, so a fresh install
declares the dependency.
2026-08-02 16:37:04 -07:00
26 changed files with 938 additions and 445 deletions

View File

@@ -260,7 +260,7 @@ resolve through a backend selected by the **`secrets:` block in `wildpc.yaml`**
```yaml
secrets:
backend: openbao # or `file` (default)
addr: https://wildpc-openbao.<domain>:8200
addr: https://castle-openbao.<domain>:8200
mount: wildpc # KV-v2 mount
token_secret: OPENBAO_ROOT_TOKEN # vault token, read from a file
node_prefix: nodes/<host> # optional: per-node overrides
@@ -310,8 +310,8 @@ a service public unless it authenticates or is meant to be open.
## 8. Mesh — multi-node coordination (opt-in)
Runs on **NATS JetStream** (`wildpc-nats`, TLS + token). Enable via env on
`wildpc-api`: `WILDPC_API_NATS_ENABLED=true`, `WILDPC_API_NATS_URL=tls://wildpc-nats.<domain>:4222`,
Runs on **NATS JetStream** (`castle-nats`, TLS + token). Enable via env on
`wildpc-api`: `WILDPC_API_NATS_ENABLED=true`, `WILDPC_API_NATS_URL=tls://castle-nats.<domain>:4222`,
`WILDPC_API_NATS_TOKEN=${secret:NATS_TOKEN}`. Each node publishes its
(secret-stripped) registry to a JetStream **KV** bucket, renews a **presence**
key, and watches for peers; remote deployments surface as `manager: none`
@@ -324,7 +324,7 @@ Inspect + drive from the CLI: **`wildpc mesh status`** / **`wildpc mesh nodes`**
**`wildpc mesh config list|get|set`** (or `GET /mesh/status`, `/nodes`,
`/mesh/config`). Modules: `wildpc_api.nats_client`, `.mesh`, `.mesh_gateway`,
`.mdns`; secrets via `core` `secret_backends` (file default, OpenBao opt-in).
→ Full history + operations: **`docs/fleet-mesh-plan.md`**.
→ Operations — health, a stale node, a sealed vault, adding a node: **`docs/mesh.md`**.
---
@@ -336,6 +336,7 @@ Inspect + drive from the CLI: **`wildpc mesh status`** / **`wildpc mesh nodes`**
| Why wildpc is shaped this way | **`docs/design.md`** |
| DNS resolution + the two TLS modes, acme recipe | **`docs/dns-and-tls.md`** |
| Public exposure (cloudflared) one-time setup | **`docs/tunnel-setup.md`** |
| Mesh operations: health, stale nodes, sealed vault, adding a node | **`docs/mesh.md`** |
| Writing FastAPI services | **`docs/stacks/python-fastapi.md`** |
| Writing CLI tools | **`docs/stacks/python-cli.md`** |
| Writing React/Vite frontends | **`docs/stacks/react-vite.md`** |

View File

@@ -238,7 +238,7 @@ to Wild PC's computed values. Secrets live in `~/.wildpc/secrets/` (never in a r
Every gateway-exposed deployment gets its own subdomain — `<name>.<gateway.domain>`
— routed to it by the Caddy gateway (there are no path-prefix routes). Exposure is a
single checkbox: `proxy: true` on a service, while a static deployment is inherently
served. The dashboard is `wildpc.<domain>` and the API `wildpc-api.<domain>`; on a
served. The dashboard is `wild-pc.<domain>` and the API `wildpc-api.<domain>`; on a
node with no domain, `:9000` serves the dashboard plus a `/api` proxy.
`gateway.tls` is a per-node choice: `off` (plain HTTP on `:9000`) or `acme` (a real
@@ -266,7 +266,7 @@ programs (CLI, API, gateway) know Wild PC internals.
## Dashboard & API
The **dashboard** (`app/`, served at `wildpc.<domain>` or `http://localhost:9000`)
The **dashboard** (`app/`, served at `wild-pc.<domain>` or `http://localhost:9000`)
lists programs and deployments, edits their config, drives lifecycle, shows the
gateway route table and logs, and hosts the agent terminal dock.

View File

@@ -1,5 +1,5 @@
// Resolve the wildpc-api base URL. The gateway serves each service at its own
// subdomain (<name>.<domain>), so when the dashboard runs at wildpc.<domain>
// subdomain (<name>.<domain>), so when the dashboard runs at wild-pc.<domain>
// the API lives at wildpc-api.<domain> — a cross-origin call (wildpc-api allows
// CORS *). When served at a bare host (dev, or the off-mode :9000 gateway), the
// API is reachable same-origin at /api.

View File

@@ -0,0 +1,6 @@
program: wild-pc
description: Wild PC web app
requires:
- kind: deployment
ref: wildpc-api
manager: caddy

View File

@@ -1,3 +0,0 @@
program: wildpc
description: Wild PC web app
manager: caddy

View File

@@ -28,7 +28,7 @@ _ICON = {
_GATEWAY = "wildpc-gateway"
_API = "wildpc-api"
_DASHBOARD = "wildpc"
_DASHBOARD = "wild-pc"
@dataclass

View File

@@ -18,7 +18,6 @@ from wildpc_core.manifest import ( # noqa: F401 — explicit re-exports for typ
PathDeployment,
ProgramSpec,
Reach,
ReadinessHttpGet,
RemoteDeployment,
Requirement,
RestartPolicy,

View File

@@ -682,6 +682,11 @@ def _target_url(config: WildpcConfig, target_name: str) -> str | None:
base = getattr(dep, "base_url", None)
if base:
return base
# A raw-TCP service has no URL scheme — project its `host:port`, reachable
# through wildcard DNS (see docs/tcp-exposure.md).
tcp_port = getattr(dep, "tcp_port", None)
if tcp_port and config.gateway.domain:
return f"{target_name}.{config.gateway.domain}:{tcp_port}"
expose = getattr(dep, "expose", None)
http = getattr(expose, "http", None) if expose else None
tport = http.internal.port if http else None
@@ -697,7 +702,7 @@ def _requires_env(config: WildpcConfig, dep: DeploymentSpec) -> dict[str, str]:
if r.kind == "deployment" and r.bind:
url = _target_url(config, r.ref)
if url:
out[r.bind] = url
out[r.bind] = url.rstrip("/") + r.path if r.path else url
return out
@@ -787,7 +792,7 @@ def _registry_requires(dep: DeploymentSpec) -> list[dict]:
"""A deployment's `requires` as plain dicts for the registry — carried into the
mesh payload so peers can draw cross-node consumption."""
return [
{"kind": r.kind, "ref": r.ref, "bind": r.bind}
{"kind": r.kind, "ref": r.ref, "bind": r.bind, "path": r.path}
for r in (getattr(dep, "requires", None) or [])
]
@@ -952,6 +957,11 @@ def _build_deployed(
subdomain=(name if expose else None),
public=bool(dep.public and expose),
public_host=(dep.public_host if (dep.public and expose) else None),
basic_auth=(
dep.expose.http.basic_auth.model_dump()
if expose and dep.expose and dep.expose.http and dep.expose.http.basic_auth
else None
),
tcp_port=tcp_port,
schedule=getattr(dep, "schedule", None),
managed=managed,
@@ -1092,6 +1102,8 @@ def _build_run_cmd(
# certs/data/secrets readable with no chown). ${uid}/${gid} expand
# to the wildpc process's own ids.
cmd.extend(["--user", _subst(run.user, placeholders)]) # type: ignore[union-attr]
if run.gpus: # type: ignore[union-attr]
cmd.extend(["--gpus", run.gpus]) # type: ignore[union-attr]
for tp in run.tmpfs: # type: ignore[union-attr]
cmd.extend(["--tmpfs", _subst(tp, placeholders)])
for container_port, host_port in run.ports.items(): # type: ignore[union-attr]
@@ -1158,10 +1170,16 @@ def _build_stop_cmd(name: str, run: object, source_dir: Path | None) -> list[str
"""The ExecStop teardown command for a runner, or [] if a plain SIGTERM suffices.
Compose stacks need an explicit ``down`` so networks/anonymous volumes are
reclaimed rather than left dangling when the unit stops.
reclaimed rather than left dangling when the unit stops. A container needs an
explicit ``stop``: SIGTERM only reaches the ``docker run`` client, leaving the
container running unsupervised and holding its name, so the next start fails.
``-`` tells systemd a container that's already gone isn't a failure.
"""
if run.launcher == "compose": # type: ignore[union-attr]
return [*_compose_base(name, run, source_dir), "down"]
if run.launcher == "container": # type: ignore[union-attr]
runtime = shutil.which("docker") or shutil.which("podman") or "docker"
return [f"-{runtime}", "stop", f"wildpc-{name}"]
return []

View File

@@ -46,6 +46,9 @@ class GatewayRoute:
public_host: str | None = None
# static routes only: SPA fallback (True) vs content-site serving (False).
spa: bool = True
# proxy routes only: HTTP Basic auth the gateway demands first,
# {"user", "password_hash"} — see manifest.BasicAuthSpec.
basic_auth: dict[str, str] | None = None
@property
def is_host(self) -> bool:
@@ -72,15 +75,16 @@ def service_proxy_targets(name: str, dep: SystemdDeployment) -> ProxyTargets:
def _local_routes(
config: WildpcConfig | None, registry: NodeRegistry
) -> list[tuple[str, str, str, bool, str | None, bool]]:
"""Each local deployment's route as ``(name, kind, target, public, public_host, spa)``.
) -> list[tuple[str, str, str, bool, str | None, bool, dict[str, str] | None]]:
"""Each local deployment's route as
``(name, kind, target, public, public_host, spa, basic_auth)``.
``kind`` is ``static`` (a caddy deployment — file-serve a built dir) or
``proxy`` (a proxied systemd process). Prefers ``wildpc.yaml``
(``config.deployments``) so a regenerated Caddyfile reflects the current spec;
falls back to the deployed registry snapshot when config isn't available.
"""
out: list[tuple[str, str, str, bool, str | None, bool]] = []
out: list[tuple[str, str, str, bool, str | None, bool, dict[str, str] | None]] = []
if config is not None:
# Only HTTP-exposed kinds route: static (file-serve) and service (proxy).
# jobs/tools/references never do.
@@ -93,21 +97,30 @@ def _local_routes(
src = _program_source(config, dep.program)
if src is not None:
pub_host = dep.public_host if dep.public else None
out.append((name, "static", str(src / dep.root), bool(dep.public), pub_host, dep.spa))
out.append(
(name, "static", str(src / dep.root), bool(dep.public), pub_host, dep.spa, None)
)
elif kind == "service" and isinstance(dep, SystemdDeployment):
expose, port, base_url = service_proxy_targets(name, dep)
if expose and (port or base_url):
pub_host = dep.public_host if dep.public else None
out.append((name, "proxy", base_url or f"localhost:{port}", bool(dep.public), pub_host, True))
auth = dep.expose.http.basic_auth if dep.expose and dep.expose.http else None
out.append((
name, "proxy", base_url or f"localhost:{port}", bool(dep.public), pub_host,
True, auth.model_dump() if auth else None,
))
return out
# No config → route from the deployed registry snapshot.
for _kind, name, d in registry.all():
if not d.enabled:
continue
if d.static_root:
out.append((name, "static", d.static_root, d.public, d.public_host, d.spa))
out.append((name, "static", d.static_root, d.public, d.public_host, d.spa, None))
elif d.subdomain and (d.port or d.base_url):
out.append((name, "proxy", d.base_url or f"localhost:{d.port}", d.public, d.public_host, True))
out.append((
name, "proxy", d.base_url or f"localhost:{d.port}", d.public, d.public_host,
True, d.basic_auth,
))
return out
@@ -148,8 +161,10 @@ def compute_routes(
# built dir; everything else that's exposed reverse-proxies its port/base_url.
# (Static frontends are `runner: static` services now — no separate program
# branch, so routing derives from one place.)
for name, kind, target, is_public, pub_host, spa in _local_routes(config, registry):
routes.append(GatewayRoute(name, kind, target, name, node, is_public, pub_host, spa))
for name, kind, target, is_public, pub_host, spa, auth in _local_routes(config, registry):
routes.append(
GatewayRoute(name, kind, target, name, node, is_public, pub_host, spa, basic_auth=auth)
)
if remote_registries:
routes.extend(_remote_routes(config, registry, remote_registries))
@@ -195,7 +210,20 @@ def _remote_routes(
return out
def _host_matcher_block(label: str, host: str, target: str) -> list[str]:
def _basic_auth_lines(auth: dict[str, str] | None, indent: str) -> list[str]:
"""Caddy's `basic_auth`, ahead of the proxy, when the route asks for a login."""
if not auth:
return []
return [
f"{indent}basic_auth {{",
f"{indent} {auth['user']} {auth['password_hash']}",
f"{indent}}}",
]
def _host_matcher_block(
label: str, host: str, target: str, basic_auth: dict[str, str] | None = None
) -> list[str]:
"""A `@host_X host <host> / handle @host_X { reverse_proxy <target> }` block.
Shared by the off-mode `:<port>` site and the acme wildcard site so the two
@@ -204,6 +232,7 @@ def _host_matcher_block(label: str, host: str, target: str) -> list[str]:
return [
f" {matcher} host {host}",
f" handle {matcher} {{",
*_basic_auth_lines(basic_auth, " "),
f" reverse_proxy {target}",
" }",
"",
@@ -232,12 +261,15 @@ def _host_remote_block(label: str, host: str, target: str) -> list[str]:
def _static_serve_lines(serve_dir: str, spa: bool, indent: str) -> list[str]:
"""The `root`/`file_server` body for a static site. SPA sites get a fallback to
the root `index.html` (client-side routing); content sites (Hugo) get plain
`file_server`, which resolves directory indexes and 404s missing paths."""
"""The `root`/`file_server` body for a static site. SPA sites try a nested
directory index (`{path}/index.html`) before falling back to the root
`index.html` (client-side routing) — so a multi-page static site keeps working
at `/sub/dir/` URLs while single-page apps still resolve deep routes. Content
sites (Hugo) get plain `file_server`, which resolves directory indexes and
404s missing paths."""
lines = [f"{indent}root * {serve_dir}"]
if spa:
lines.append(indent + "try_files {path} /index.html")
lines.append(indent + "try_files {path} {path}/index.html /index.html")
lines.append(f"{indent}file_server")
return lines
@@ -254,7 +286,27 @@ def _host_static_block(label: str, host: str, serve_dir: str, spa: bool) -> list
]
def _public_site_block(host: str, kind: str, target: str, spa: bool = True) -> list[str]:
def _no_route_fallback() -> list[str]:
"""The trailing catch-all for a wildcard site: an explicit 404 for a host that
matched `*.<domain>` but no route.
Without it Caddy runs off the end of the site block and returns a bare
`200 Content-Length: 0` — a typo'd or stale subdomain looks *up* to a browser
(TLS negotiates, the request succeeds) while rendering nothing. A `handle`
with no matcher is the fallback of its handle group, so it only fires when
every `@host_*` matcher above missed."""
return [
" handle {",
' respond "no wildpc route for this host" 404',
" }",
"",
]
def _public_site_block(
host: str, kind: str, target: str, spa: bool = True,
basic_auth: dict[str, str] | None = None,
) -> list[str]:
"""A standalone Caddy site for a custom ``public_host`` (apex or another zone).
Unlike the ``*.<public_domain>`` wildcard site, an apex/foreign host isn't
@@ -274,14 +326,14 @@ def _public_site_block(host: str, kind: str, target: str, spa: bool = True) -> l
" }",
]
else:
body = [f" reverse_proxy {target}"]
body = [*_basic_auth_lines(basic_auth, " "), f" reverse_proxy {target}"]
return [f"{host} {{", *body, "}", ""]
# Wild PC's own control plane: the dashboard frontend and the API it calls. These
# names are the subdomains they're published at in acme mode, and the pair served
# on the :<port> site in off mode (no domain → no subdomains).
_DASHBOARD = "wildpc"
_DASHBOARD = "wild-pc"
_API = "wildpc-api"
@@ -345,7 +397,8 @@ def generate_caddyfile_from_registry(
elif r.kind == "remote":
lines += _host_remote_block(r.name or r.address, host, r.target)
else:
lines += _host_matcher_block(r.name or r.address, host, r.target)
lines += _host_matcher_block(r.name or r.address, host, r.target, r.basic_auth)
lines += _no_route_fallback()
lines.append("}")
lines.append("")
# Public exposure: public deployments are also reachable by their public
@@ -370,12 +423,15 @@ def generate_caddyfile_from_registry(
elif r.kind == "remote":
lines += _host_remote_block(label, host, r.target)
else:
lines += _host_matcher_block(label, host, r.target)
lines += _host_matcher_block(label, host, r.target, r.basic_auth)
lines += _no_route_fallback()
lines.append("}")
lines.append("")
for r in custom_pub:
if r.public_host: # always true (custom_pub filter); narrows the type
lines += _public_site_block(r.public_host, r.kind, r.target, r.spa)
lines += _public_site_block(
r.public_host, r.kind, r.target, r.spa, basic_auth=r.basic_auth
)
# Redirect the bare gateway port to the dashboard subdomain.
lines += [
f":{gw_port} {{",
@@ -403,7 +459,7 @@ def generate_caddyfile_from_registry(
lines += [
" handle {",
f" root * {root}",
" try_files {path} /index.html",
" try_files {path} {path}/index.html /index.html",
" file_server",
" }",
"}",

View File

@@ -161,15 +161,29 @@ ExecStart={exec_start}
else:
restart = (sd.restart if sd else RestartPolicy.ON_FAILURE).value
restart_sec = sd.restart_sec if sd else 5
restart_steps = sd.restart_steps if sd else 8
restart_max_delay = sd.restart_max_delay_sec if sd else 300
burst = sd.start_limit_burst if sd else 10
interval = sd.start_limit_interval_sec if sd else 3600
# A crash-looping service must back off and eventually give up. systemd's
# defaults (5 starts / 10s) never trip when each attempt outlives the
# window — openclaw failed ~5s in, so the counter reset every time and it
# restarted 260k times, burning a core for days. StartLimit* here is sized
# against the backoff below, so the unit lands in `failed` (visible in
# `wildpc status`) instead of looping silently forever.
unit = f"""[Unit]
Description=Wild PC: {description}
After={after}
StartLimitIntervalSec={interval}
StartLimitBurst={burst}
[Service]
Type=simple
ExecStart={exec_start}
{env_lines}Restart={restart}
RestartSec={restart_sec}
RestartSteps={restart_steps}
RestartMaxDelaySec={restart_max_delay}
SuccessExitStatus=143
"""
# Explicit teardown (e.g. compose `down`) so the stack's networks/volumes

View File

@@ -2,10 +2,11 @@
from __future__ import annotations
import re
from enum import Enum
from typing import Annotated, Literal, Union
from pydantic import BaseModel, ConfigDict, Field, model_validator
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
EnvMap = dict[str, str]
@@ -86,7 +87,10 @@ class RunContainer(LaunchBase):
image: str
command: list[str] | None = None
args: list[str] = Field(default_factory=list)
ports: dict[int, int] = Field(default_factory=dict)
# container port → host port. The host side may name the address to publish
# on, "127.0.0.1:4318": a bare port publishes on every interface (docker's
# default), which puts it on the LAN whatever `reach:` says.
ports: dict[int, int | str] = Field(default_factory=dict)
volumes: list[str] = Field(default_factory=list)
env: EnvMap = Field(default_factory=dict)
workdir: str | None = None
@@ -97,6 +101,21 @@ class RunContainer(LaunchBase):
# tmpfs mounts (e.g. ["/var/run/postgresql"]) for image runtime dirs that must
# be writable when the container runs as a non-default uid.
tmpfs: list[str] = Field(default_factory=list)
# GPUs to hand the container (docker `--gpus`), e.g. "all" or "device=0".
# None → no GPU access.
gpus: str | None = None
@field_validator("ports")
@classmethod
def _host_binding(cls, ports: dict[int, int | str]) -> dict[int, int | str]:
for container_port, host in ports.items():
if isinstance(host, str) and not _HOST_BINDING.match(host):
raise ValueError(
f"ports[{container_port}]: {host!r} is not a host port or "
"an <ipv4>:<port> binding (e.g. 127.0.0.1:4318)"
)
return ports
class RunNode(LaunchBase):
@@ -139,24 +158,33 @@ LaunchSpec = Annotated[
# ---------------------
class ReadinessHttpGet(BaseModel):
http_get: str
timeout_seconds: int = 2
interval_seconds: int = 2
success_codes: list[int] = Field(default_factory=lambda: [200])
class SystemdSpec(BaseModel):
enable: bool = True
user: bool = True
description: str | None = None
# Unit ordering (systemd ``After=``). There is deliberately no ``requires:``
# partner emitting ``Requires=``: that binds lifecycles, so stopping postgres
# would stop everything downstream, and ordering is what callers actually
# want. A deployment's own ``requires:`` list is the relationship model (see
# docs/relationships.md), not systemd wiring.
after: list[str] = Field(default_factory=list)
requires: list[str] = Field(default_factory=list)
wanted_by: list[str] = Field(default_factory=lambda: ["default.target"])
restart: RestartPolicy = RestartPolicy.ON_FAILURE
restart_sec: int = 2
# Exponential restart backoff: the delay grows from ``restart_sec`` to
# ``restart_max_delay_sec`` over ``restart_steps`` restarts, so a service that
# can't start stops hammering the box (systemd >= 254; older systemd ignores
# both keys with a warning and keeps the flat ``restart_sec``).
restart_steps: int = 8
restart_max_delay_sec: int = 300
# Give up eventually: ``start_limit_burst`` starts inside
# ``start_limit_interval_sec`` puts the unit in `failed` instead of looping
# forever. Sized against the backoff above — with the defaults, ~10 restarts
# span ~17min, long enough to ride out a boot-time dependency race and short
# enough that a genuinely broken service surfaces as failed the same hour.
start_limit_burst: int = 10
start_limit_interval_sec: int = 3600
no_new_privileges: bool = True
readiness: ReadinessHttpGet | None = None
exec_reload: str | None = None
# Commands run after the main process starts (systemd ``ExecStartPost=``), one
# line each — e.g. an OpenBao auto-unseal step. Failures don't fail the unit.
@@ -172,6 +200,21 @@ class ManageSpec(BaseModel):
# ---------------------
#: "127.0.0.1:4318" — an IPv4 address and a port, as docker's `-p` takes them.
_HOST_BINDING = re.compile(r"^(\d{1,3}\.){3}\d{1,3}:\d{1,5}$")
class BasicAuthSpec(BaseModel):
"""HTTP Basic auth at the gateway, for a service with no login of its own
(a trace viewer, an admin UI). Caddy's ``basic_auth``: one user, and a bcrypt
hash of the password — ``caddy hash-password`` makes one. The hash is not the
password, which is why it may live in the manifest; the gateway is HTTPS, so
the credential never crosses the LAN in the clear."""
user: str = Field(min_length=1, pattern=r"^[^\s:]+$")
password_hash: str = Field(pattern=r"^\$2[aby]\$\d\d\$[./A-Za-z0-9]{53}$")
class HttpInternal(BaseModel):
host: str = "127.0.0.1"
port: int = Field(ge=1, le=65535)
@@ -181,6 +224,9 @@ class HttpInternal(BaseModel):
class HttpExposeSpec(BaseModel):
internal: HttpInternal
health_path: str | None = None
# Require a login at the gateway before proxying. Applies wherever the
# gateway routes this service (internal and public names alike).
basic_auth: BasicAuthSpec | None = None
class TlsMaterial(str, Enum):
@@ -213,9 +259,10 @@ class TcpExposeSpec(BaseModel):
"""A raw-TCP service (postgres, redis, …). It doesn't ride the HTTP gateway:
with ``reach: internal`` it's reachable at ``<name>.<domain>:<port>`` via the
wildcard DNS record + the bound port (no Caddy route). Publishing the port on
the LAN is the deployment's own job (a container's ``run.ports``, or a native
service binding ``0.0.0.0``); wildpc doesn't rebind it, so there's no bind-host
field here to imply otherwise. ``tls`` (optional) has wildpc drop the wildcard
the LAN is the deployment's own job (a container's ``run.ports`` — whose host
side may name an address, ``127.0.0.1:4000`` — or a native service binding
``0.0.0.0``); wildpc doesn't rebind it, so there's no bind-host field here to
imply otherwise. ``tls`` (optional) has wildpc drop the wildcard
cert onto the service so it presents a trusted cert for ``<name>.<domain>``.
"""
@@ -295,6 +342,9 @@ class Requirement(BaseModel):
kind: Literal["system", "deployment", "tool"] = "deployment"
ref: str
bind: str | None = None
# Appended to the target's URL when binding — for a consumer that wants a
# sub-path of it (e.g. an OpenAI-compatible server's "/v1") as its base URL.
path: str | None = None
# ---------------------
@@ -530,7 +580,9 @@ class CaddyDeployment(DeploymentBase):
@model_validator(mode="after")
def _validate_reach(self) -> CaddyDeployment:
if self.reach == Reach.OFF:
raise ValueError("a static (caddy) deployment is always served; reach must be internal|public")
raise ValueError(
"a static (caddy) deployment is always served; reach must be internal|public"
)
_validate_public_host(self.public_host, self.reach)
return self

View File

@@ -86,6 +86,10 @@ class Deployment:
# Optional public hostname override (exact FQDN, apex allowed). When set it is
# the public-facing name instead of the derived <subdomain>.<public_domain>.
public_host: str | None = None
# HTTP Basic auth the gateway demands before proxying: {"user", "password_hash"}.
# Recorded here too, so a Caddyfile generated from the registry alone (no
# config to hand) still asks for it rather than serving the route open.
basic_auth: dict[str, str] | None = None
# Raw-TCP exposure port (postgres, redis, …). Set → reachable at
# <name>.<gateway.domain>:<tcp_port> via bind + wildcard DNS (no Caddy route).
tcp_port: int | None = None
@@ -193,6 +197,7 @@ def load_registry(path: Path | None = None) -> NodeRegistry:
subdomain=comp_data.get("subdomain"),
public=comp_data.get("public", False),
public_host=comp_data.get("public_host"),
basic_auth=comp_data.get("basic_auth"),
tcp_port=comp_data.get("tcp_port"),
static_root=comp_data.get("static_root"),
spa=comp_data.get("spa", True),
@@ -273,6 +278,8 @@ def save_registry(registry: NodeRegistry, path: Path | None = None) -> None:
entry["public"] = comp.public
if comp.public_host:
entry["public_host"] = comp.public_host
if comp.basic_auth:
entry["basic_auth"] = dict(comp.basic_auth)
if comp.tcp_port is not None:
entry["tcp_port"] = comp.tcp_port
if comp.static_root:

View File

@@ -27,12 +27,24 @@ import os
import re
from pathlib import Path
# A pin that is a bare, exact-ish version: "24", "24.14", "24.14.1" (optional "v").
_EXACT_RE = re.compile(r"v?(\d+)(?:\.(\d+))?(?:\.(\d+))?$")
# An installed nvm dir: v24.14.1
_INSTALLED_RE = re.compile(r"v(\d+)\.(\d+)\.(\d+)$")
# Wildcards/aliases that mean "newest installed".
_NEWEST_ALIASES = {"", "*", "x", "node", "latest", "current", "lts", "lts/*"}
# One comparator: an optional operator then a (possibly partial/wildcard) version,
# e.g. ">=22.19.0", "^24.1", "~24", "24.x", "23".
_PART = r"\d+|[xX*]"
_COMPARATOR_RE = re.compile(
rf"^(?P<op>>=|<=|>|<|=|\^|~)?\s*v?(?P<maj>{_PART})"
rf"(?:\.(?P<min>{_PART}))?(?:\.(?P<pat>{_PART}))?"
r"(?:[-+][0-9A-Za-z.\-]+)?$"
)
# A hyphen range: "22.19.0 - 23.11.0".
_HYPHEN_RE = re.compile(r"^(.+?)\s+-\s+(.+)$")
Version = tuple[int, int, int]
# A parsed comparator: (op, version), ANDed within one range alternative.
Bound = tuple[str, Version]
class ToolchainError(Exception):
@@ -83,17 +95,136 @@ def _installed() -> list[tuple[tuple[int, int, int], Path]]:
return out
def _num(tok: str | None) -> int | None:
"""A version part as an int, or None when absent or a wildcard (``x``/``*``)."""
if tok is None or tok in ("x", "X", "*"):
return None
return int(tok)
def _bounds(token: str) -> list[Bound] | None:
"""One comparator as a list of ANDed (op, version) bounds.
Returns ``[]`` for an unconstrained token (``*``/``x``) and ``None`` when the
token isn't a comparator this parser understands. Partial versions follow
node-semver: ``>22`` means ``>=23.0.0``, ``<23`` means ``<23.0.0``, a bare
``24`` means ``>=24.0.0 <25.0.0``."""
t = token.strip()
if not t or t in ("*", "x", "X"):
return []
m = _COMPARATOR_RE.match(t)
if not m:
return None
op = m["op"] or "="
maj, minor, patch = _num(m["maj"]), _num(m["min"]), _num(m["pat"])
if maj is None:
return [] # "x" / "x.y" — unconstrained
# How many parts were given concretely: 24 → 1, 24.14 → 2, 24.14.1 → 3.
precision = 1 if minor is None else (2 if patch is None else 3)
base: Version = (maj, minor or 0, patch or 0)
if op == ">=":
return [(">=", base)]
if op == ">":
if precision == 1:
return [(">=", (maj + 1, 0, 0))]
if precision == 2:
return [(">=", (maj, (minor or 0) + 1, 0))]
return [(">", base)]
if op == "<":
return [("<", base)]
if op == "<=":
if precision == 1:
return [("<", (maj + 1, 0, 0))]
if precision == 2:
return [("<", (maj, (minor or 0) + 1, 0))]
return [("<=", base)]
if op == "^":
# Caret: up to the next non-zero leftmost part.
if maj > 0:
hi: Version = (maj + 1, 0, 0)
elif (minor or 0) > 0:
hi = (0, (minor or 0) + 1, 0)
else:
hi = (0, 0, (patch or 0) + 1)
return [(">=", base), ("<", hi)]
if op == "~":
hi = (maj + 1, 0, 0) if precision == 1 else (maj, (minor or 0) + 1, 0)
return [(">=", base), ("<", hi)]
# "=" or bare: a partial pin is the range it prefixes.
if precision == 1:
return [(">=", base), ("<", (maj + 1, 0, 0))]
if precision == 2:
return [(">=", base), ("<", (maj, (minor or 0) + 1, 0))]
return [("==", base)]
def _parse_range(pin: str) -> list[list[Bound]] | None:
"""A semver range as alternatives (``||``) of ANDed bounds.
Returns ``None`` when no alternative could be parsed — the caller then falls
back to "newest installed" rather than guessing a version."""
alternatives: list[list[Bound]] = []
for alt in pin.split("||"):
alt = alt.strip()
hyphen = _HYPHEN_RE.match(alt)
tokens = (
[f">={hyphen[1]}", f"<={hyphen[2]}"]
if hyphen
else [t for t in re.split(r"[\s,]+", alt) if t]
)
bounds: list[Bound] = []
ok = True
for tok in tokens:
parsed = _bounds(tok)
if parsed is None:
ok = False
break
bounds.extend(parsed)
if ok:
alternatives.append(bounds)
return alternatives or None
def _satisfies(ver: Version, bounds: list[Bound]) -> bool:
"""Does a concrete version meet every bound in one alternative?"""
for op, target in bounds:
if op == ">=" and not ver >= target:
return False
if op == ">" and not ver > target:
return False
if op == "<=" and not ver <= target:
return False
if op == "<" and not ver < target:
return False
if op == "==" and ver != target:
return False
return True
def _install_hint(pin: str) -> str:
"""What to tell the user to `nvm install`.
An exact-ish pin installs verbatim; for a range, name the highest major it
mentions — the newest line the program declares it supports."""
majors = [int(m) for m in re.findall(r"(\d+)", pin)]
if _COMPARATOR_RE.match(pin.strip()) and not re.search(r"[<>=^~|]", pin):
return pin
return str(max(majors)) if majors else "--lts"
def resolve_node_bin(source: Path | None) -> Path | None:
"""Resolve a program's pinned node version to its ``.../bin`` dir on this box.
Returns ``None`` when the program pins nothing — wildpc injects no node, it does
not guess. Raises :class:`ToolchainError` when a version *is* pinned but no
matching version is installed.
installed version satisfies it.
Matching: an exact-ish pin (``24`` / ``24.14`` / ``24.14.1``) matches installed
versions by that precision, newest wins. A range or alias (``>=24``, ``^24.1``,
``lts/*``) is pinned by its leading major (or newest installed if it names none)
— precise pins belong in ``.node-version``."""
Matching is real semver-range evaluation: every ``||`` alternative is parsed
into comparators and the **newest installed version satisfying any alternative**
wins. So ``>=22.19.0 <23 || >=23.11.0`` selects an installed 24.x, and never an
installed 22.17.0 — which sits below the range's own floor. A pin this parser
can't read at all falls back to newest-installed rather than guessing."""
if source is None:
return None
pin = read_node_pin(source)
@@ -106,28 +237,23 @@ def resolve_node_bin(source: Path | None) -> Path | None:
return installed[0][1]
raise ToolchainError(
f"node {pin!r} pinned (in {source}) but no node is installed under "
f"{node_versions_dir()} — run `nvm install {pin}`"
f"{node_versions_dir()} — run `nvm install {_install_hint(pin)}`"
)
low = pin.strip().lower()
if low in _NEWEST_ALIASES:
if pin.strip().lower() in _NEWEST_ALIASES:
return newest_or_raise()
exact = _EXACT_RE.match(low)
if exact:
parts = [int(g) for g in exact.groups() if g is not None]
else:
# A range/alias (>=24, ^24.1, ~24, 24.x): pin by leading major only.
major = re.search(r"\d+", low)
if not major:
return newest_or_raise()
parts = [int(major.group())]
precision = len(parts)
alternatives = _parse_range(pin)
if alternatives is None:
return newest_or_raise()
for ver, bindir in installed: # newest first
if list(ver[:precision]) == parts:
if any(_satisfies(ver, bounds) for bounds in alternatives):
return bindir
have = ", ".join(".".join(str(p) for p in v) for v, _ in installed) or "none"
raise ToolchainError(
f"node {pin!r} pinned (in {source}) but not installed under "
f"{node_versions_dir()} — run `nvm install {pin}`"
f"node {pin!r} pinned (in {source}) but no installed version satisfies it "
f"(installed: {have}) under {node_versions_dir()} — "
f"run `nvm install {_install_hint(pin)}`"
)

View File

@@ -101,7 +101,16 @@ class TestAcmeMode:
def test_port_9000_redirects_to_dashboard(self) -> None:
cf = generate_caddyfile_from_registry(_acme({"api": _dep(9020, expose=True, name="api")}))
assert ":9000 {" in cf
assert "redir https://wildpc.example.com{uri}" in cf
assert "redir https://wild-pc.example.com{uri}" in cf
def test_unmatched_host_404s_instead_of_empty_200(self) -> None:
"""A host that matches `*.<domain>` but no route must 404. Without the
trailing catch-all Caddy falls off the end of the site block and returns
an empty 200, so a typo'd subdomain looks up but renders nothing."""
cf = generate_caddyfile_from_registry(_acme({"api": _dep(9020, expose=True, name="api")}))
wildcard = cf.split("*.example.com {", 1)[1].split("\n}", 1)[0]
assert "handle {" in wildcard
assert 'respond "no wildpc route for this host" 404' in wildcard
def test_no_path_routes(self) -> None:
cf = generate_caddyfile_from_registry(_acme({"api": _dep(9020, expose=True, name="api")}))
@@ -136,7 +145,7 @@ class TestAcmeMode:
cf = generate_caddyfile_from_registry(_acme({}))
assert "@host_wildpc host wildpc.example.com" in cf
assert "root * /data/repos/wildpc/app/dist" in cf
assert "try_files {path} /index.html" in cf
assert "try_files {path} {path}/index.html /index.html" in cf
assert "file_server" in cf

View File

@@ -205,3 +205,24 @@ def test_compose_stop_cmd_is_down(tmp_path: Path) -> None:
def test_non_compose_runner_has_no_stop_cmd(tmp_path: Path) -> None:
run = RunPython(launcher="python", program="svc")
assert _build_stop_cmd("svc", run, tmp_path) == []
def test_container_gpus_flag() -> None:
"""`gpus` passes through as docker `--gpus`; omitted → no GPU access."""
run = RunContainer(launcher="container", image="img:latest", gpus="all")
with patch("wildpc_core.deploy.shutil.which", return_value="/usr/bin/docker"):
cmd = _build_run_cmd("svc", run, {}, [])
assert cmd[cmd.index("--gpus") + 1] == "all"
assert cmd.index("--gpus") < cmd.index("img:latest")
plain = RunContainer(launcher="container", image="img:latest")
with patch("wildpc_core.deploy.shutil.which", return_value="/usr/bin/docker"):
assert "--gpus" not in _build_run_cmd("svc", plain, {}, [])
def test_container_stop_cmd_stops_named_container() -> None:
"""Stopping the unit must stop the container itself, not just the docker client."""
run = RunContainer(launcher="container", image="img:latest")
with patch("wildpc_core.deploy.shutil.which", return_value="/usr/bin/docker"):
stop = _build_stop_cmd("svc", run, None)
assert stop == ["-/usr/bin/docker", "stop", "wildpc-svc"]

View File

@@ -0,0 +1,152 @@
"""A login at the gateway, and container ports that stay off the LAN.
Both exist for the same kind of service: one with no authentication of its own
(a trace viewer, an admin UI, an OTLP receiver) that must not be reachable by
anything on the network just because it runs in a container.
"""
from __future__ import annotations
from pathlib import Path
from unittest.mock import patch
import pytest
from pydantic import ValidationError
from wildpc_core.config import GatewayConfig, WildpcConfig
from wildpc_core.deploy import _build_run_cmd
from wildpc_core.generators.caddyfile import compute_routes, generate_caddyfile_from_registry
from wildpc_core.manifest import (
BasicAuthSpec,
ExposeSpec,
HttpExposeSpec,
HttpInternal,
Reach,
RunContainer,
RunPython,
SystemdDeployment,
)
from wildpc_core.registry import (
Deployment,
NodeConfig,
NodeRegistry,
load_registry,
save_registry,
)
#: Caddy's documented example: `caddy hash-password` output for "hiccup".
HASH = "$2a$14$Zkx19XLiW6VYouLHR5NmfOFU0z2GTNmpkT/5qqR7hx4IjWJPDhjvG"
AUTH = {"user": "paul", "password_hash": HASH}
@pytest.fixture(autouse=True)
def _isolate_config(monkeypatch: pytest.MonkeyPatch) -> None:
import wildpc_core.config as config_mod
def _no_config(*args: object, **kwargs: object) -> object:
raise FileNotFoundError("isolated in tests")
monkeypatch.setattr(config_mod, "load_config", _no_config)
def _registry(deployed: dict[str, Deployment] | None = None, **node: object) -> NodeRegistry:
reg = NodeRegistry(node=NodeConfig(
hostname="test", gateway_port=9000, gateway_tls="acme",
gateway_domain="example.com", acme_email="p@e.com", **node, # type: ignore[arg-type]
))
for name, d in (deployed or {}).items():
d.name = name
reg.put(d)
return reg
def _service(port: int, auth: dict[str, str] | None = None) -> SystemdDeployment:
return SystemdDeployment(
manager="systemd",
run=RunPython(launcher="python", program="svc"),
expose=ExposeSpec(http=HttpExposeSpec(
internal=HttpInternal(port=port),
basic_auth=BasicAuthSpec(**auth) if auth else None,
)),
reach=Reach.INTERNAL,
)
def _config(**deployments: SystemdDeployment) -> WildpcConfig:
return WildpcConfig(
root=None, # type: ignore[arg-type]
gateway=GatewayConfig(port=9000), repo=None, programs={},
deployments=dict(deployments), # type: ignore[arg-type]
)
def _block(caddyfile: str, matcher: str) -> str:
"""The handle block for one host matcher."""
start = caddyfile.index(f"handle {matcher} {{")
return caddyfile[start:caddyfile.index("\n }", start)]
class TestBasicAuth:
def test_a_login_is_asked_for_before_proxying(self, monkeypatch: pytest.MonkeyPatch) -> None:
import wildpc_core.config as config_mod
config = _config(jaeger=_service(16686, AUTH), open=_service(9001))
assert [r.basic_auth for r in compute_routes(_registry(), config) if r.name == "jaeger"] == [AUTH]
monkeypatch.setattr(config_mod, "load_config", lambda *a, **k: config)
cf = generate_caddyfile_from_registry(_registry())
guarded = _block(cf, "@host_jaeger")
assert f"basic_auth {{\n paul {HASH}\n }}" in guarded
assert guarded.index("basic_auth") < guarded.index("reverse_proxy localhost:16686")
assert "basic_auth" not in _block(cf, "@host_open")
def test_the_registry_alone_still_asks(self) -> None:
"""No config to hand → routes come from the deployed registry. A route
that needs a login must not be served open because of where it was read."""
dep = Deployment(manager="systemd", launcher="python", run_cmd=["x"], port=16686,
subdomain="jaeger", basic_auth=AUTH)
cf = generate_caddyfile_from_registry(_registry({"jaeger": dep}))
assert "basic_auth" in _block(cf, "@host_jaeger")
def test_a_public_host_asks_too(self) -> None:
dep = Deployment(manager="systemd", launcher="python", run_cmd=["x"], port=16686,
subdomain="jaeger", public=True, public_host="traces.example.org",
basic_auth=AUTH)
cf = generate_caddyfile_from_registry(_registry({"jaeger": dep}))
site = cf[cf.index("traces.example.org {"):]
assert site.index("basic_auth") < site.index("reverse_proxy")
def test_it_survives_the_registry_round_trip(self, tmp_path: Path) -> None:
dep = Deployment(manager="systemd", launcher="python", run_cmd=["x"], port=1,
subdomain="jaeger", basic_auth=AUTH)
save_registry(_registry({"jaeger": dep}), tmp_path / "registry.yaml")
(again,) = load_registry(tmp_path / "registry.yaml").named("jaeger")
assert again.basic_auth == AUTH
@pytest.mark.parametrize("bad", [
{"user": "paul", "password_hash": "hiccup"}, # a password, not a hash
{"user": "pa:ul", "password_hash": HASH}, # Basic auth splits on ':'
{"user": "", "password_hash": HASH},
])
def test_only_a_bcrypt_hash_and_a_plain_user(self, bad: dict[str, str]) -> None:
with pytest.raises(ValidationError):
BasicAuthSpec(**bad)
class TestHostBindings:
def _cmd(self, ports: dict[int, int | str]) -> list[str]:
run = RunContainer(launcher="container", image="img:latest", ports=ports)
with patch("wildpc_core.deploy.shutil.which", return_value="/usr/bin/docker"):
return _build_run_cmd("svc", run, {}, [], secret_env_file=None)
def test_a_bare_port_publishes_as_before(self) -> None:
cmd = self._cmd({4000: 4000})
assert cmd[cmd.index("-p") + 1] == "4000:4000"
def test_a_binding_keeps_it_on_one_address(self) -> None:
cmd = self._cmd({4318: "127.0.0.1:4318"})
assert cmd[cmd.index("-p") + 1] == "127.0.0.1:4318:4318"
@pytest.mark.parametrize("bad", ["localhost:4318", "4318", "127.0.0.1", "::1:4318"])
def test_a_binding_is_an_ipv4_and_a_port(self, bad: str) -> None:
with pytest.raises(ValidationError):
RunContainer(launcher="container", image="img", ports={4318: bad})

View File

@@ -0,0 +1,63 @@
"""Tests for `_requires_env` — a `{ref, bind}` requirement projects the target's URL."""
from __future__ import annotations
import wildpc_core.config as C
from wildpc_core.deploy import _requires_env
from wildpc_core.manifest import RemoteDeployment, SystemdDeployment
def _svc(requires: list | None = None, port: int | None = None) -> SystemdDeployment:
spec: dict = {"manager": "systemd", "run": {"launcher": "command", "argv": ["x"]}}
if requires is not None:
spec["requires"] = requires
if port is not None:
spec["expose"] = {"http": {"internal": {"port": port}}}
spec["reach"] = "internal"
return SystemdDeployment.model_validate(spec)
def _cfg(deployments: dict) -> C.WildpcConfig:
return C.WildpcConfig(
root=None,
gateway=C.GatewayConfig(port=9000, tls="acme", domain="node.example"),
repo=None,
programs={},
deployments=deployments,
)
def test_bind_projects_target_url() -> None:
consumer = _svc(requires=[{"ref": "api", "bind": "API_URL"}])
cfg = _cfg({"api": _svc(port=9100), "app": consumer})
assert _requires_env(cfg, consumer) == {"API_URL": "https://api.node.example"}
def test_bind_path_is_appended() -> None:
consumer = _svc(requires=[{"ref": "llm", "bind": "LLM_BASE", "path": "/v1"}])
cfg = _cfg({"llm": _svc(port=8100), "app": consumer})
assert _requires_env(cfg, consumer) == {"LLM_BASE": "https://llm.node.example/v1"}
def test_bind_path_on_reference_base_url() -> None:
ref = RemoteDeployment.model_validate(
{"manager": "none", "base_url": "https://ext.example/"}
)
consumer = _svc(requires=[{"ref": "ext", "bind": "EXT", "path": "/v1"}])
cfg = _cfg({"ext": ref, "app": consumer})
assert _requires_env(cfg, consumer) == {"EXT": "https://ext.example/v1"}
def test_bind_tcp_service_projects_host_port() -> None:
"""A raw-TCP target (e.g. postgres) binds as `<name>.<domain>:<port>`."""
pg = SystemdDeployment.model_validate(
{
"manager": "systemd",
"run": {"launcher": "command", "argv": ["x"]},
"expose": {"tcp": {"port": 5432}},
"reach": "internal",
}
)
consumer = _svc(requires=[{"ref": "postgres", "bind": "DB_HOST"}])
cfg = _cfg({"postgres": pg, "app": consumer})
assert _requires_env(cfg, consumer) == {"DB_HOST": "postgres.node.example:5432"}

View File

@@ -122,6 +122,73 @@ class TestUnitFromDeployed:
unit = generate_unit_from_deployed("test-svc", deployed)
assert "Restart=on-failure" in unit
def test_restart_backoff_and_start_limit(self) -> None:
"""A crash loop must back off and eventually give up.
systemd's defaults (5 starts / 10s) never trip when each attempt outlives
the window: openclaw failed ~5s in, so the counter reset on every cycle and
the unit restarted 260k times, burning a core for days."""
deployed = Deployment(
manager="systemd", launcher="python",
run_cmd=["/home/user/.local/bin/uv", "run", "test-svc"],
env={},
description="Test service",
)
unit = generate_unit_from_deployed("test-svc", deployed)
# Backoff grows the delay instead of hammering every RestartSec.
assert "RestartSteps=8" in unit
assert "RestartMaxDelaySec=300" in unit
# The limit is wide enough that it actually trips (systemd only counts
# starts inside the window).
assert "StartLimitIntervalSec=3600" in unit
assert "StartLimitBurst=10" in unit
def test_start_limit_in_unit_section(self) -> None:
"""StartLimit* belong to [Unit]; systemd ignores them under [Service]."""
deployed = Deployment(
manager="systemd", launcher="python",
run_cmd=["/home/user/.local/bin/uv", "run", "test-svc"],
env={},
description="Test service",
)
unit = generate_unit_from_deployed("test-svc", deployed)
unit_section = unit.split("[Service]")[0]
assert "StartLimitIntervalSec=" in unit_section
assert "StartLimitBurst=" in unit_section
def test_job_has_no_restart_tuning(self) -> None:
"""A timer-driven oneshot never restarts, so it gets no restart keys."""
deployed = Deployment(
manager="systemd", launcher="command",
run_cmd=["/home/user/.local/bin/my-job"],
env={},
description="Nightly job",
schedule="0 2 * * *",
)
unit = generate_unit_from_deployed("my-job", deployed)
for key in ("RestartSteps=", "RestartMaxDelaySec=", "StartLimitBurst="):
assert key not in unit
def test_restart_tuning_is_overridable(self) -> None:
"""A service can widen or tighten the policy in its manifest."""
from wildpc_core.manifest import SystemdSpec
deployed = Deployment(
manager="systemd", launcher="python",
run_cmd=["/home/user/.local/bin/uv", "run", "test-svc"],
env={},
description="Test service",
)
spec = SystemdSpec(
restart_steps=3, restart_max_delay_sec=60,
start_limit_burst=4, start_limit_interval_sec=120,
)
unit = generate_unit_from_deployed("test-svc", deployed, systemd_spec=spec)
assert "RestartSteps=3" in unit
assert "RestartMaxDelaySec=60" in unit
assert "StartLimitIntervalSec=120" in unit
assert "StartLimitBurst=4" in unit
def test_exec_start_from_run_cmd(self) -> None:
"""Unit file ExecStart uses resolved run_cmd."""
deployed = Deployment(

View File

@@ -118,4 +118,84 @@ class TestResolveNodeBin:
def test_default_dir_is_nvm(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("WILDPC_NODE_VERSIONS_DIR", raising=False)
assert toolchains.node_versions_dir() == Path.home() / ".nvm" / "versions" / "node"
assert (
toolchains.node_versions_dir() == Path.home() / ".nvm" / "versions" / "node"
)
class TestSemverRanges:
"""Range pins are evaluated, not reduced to their leading major.
The regression these guard: openclaw pins `>=22.19.0 <23 || >=23.11.0`, and a
leading-major heuristic picked the newest installed 22.x — v22.17.0, *below*
the range's own floor — baking a node into the systemd unit that the program
refuses to run on."""
def test_compound_range_uses_later_alternative(
self, tmp_path: Path, nvm: Path
) -> None:
_install(nvm, "v24.18.0", "v22.17.0", "v20.19.3")
(tmp_path / "package.json").write_text(
'{"engines": {"node": ">=22.19.0 <23 || >=23.11.0"}}'
)
assert resolve_node_bin(tmp_path) == nvm / "v24.18.0" / "bin"
def test_never_picks_below_lower_bound(self, tmp_path: Path, nvm: Path) -> None:
# Only a 22.x that fails the pin's own >=22.19.0 floor is installed.
_install(nvm, "v22.17.0", "v20.19.3")
(tmp_path / "package.json").write_text(
'{"engines": {"node": ">=22.19.0 <23 || >=23.11.0"}}'
)
with pytest.raises(ToolchainError, match="no installed version satisfies"):
resolve_node_bin(tmp_path)
def test_upper_bound_excludes_newer(self, tmp_path: Path, nvm: Path) -> None:
_install(nvm, "v24.18.0", "v22.19.1")
(tmp_path / "package.json").write_text('{"engines": {"node": ">=22.19.0 <23"}}')
assert resolve_node_bin(tmp_path) == nvm / "v22.19.1" / "bin"
def test_error_names_installed_versions(self, tmp_path: Path, nvm: Path) -> None:
_install(nvm, "v22.17.0")
(tmp_path / "package.json").write_text('{"engines": {"node": ">=24"}}')
with pytest.raises(ToolchainError, match=r"installed: 22\.17\.0"):
resolve_node_bin(tmp_path)
def test_tilde_pins_minor(self, tmp_path: Path, nvm: Path) -> None:
_install(nvm, "v24.15.0", "v24.14.9", "v24.14.1")
(tmp_path / ".nvmrc").write_text("~24.14.1")
assert resolve_node_bin(tmp_path) == nvm / "v24.14.9" / "bin"
def test_caret_excludes_next_major(self, tmp_path: Path, nvm: Path) -> None:
_install(nvm, "v25.0.0", "v24.14.1")
(tmp_path / "package.json").write_text('{"engines": {"node": "^24.1.0"}}')
assert resolve_node_bin(tmp_path) == nvm / "v24.14.1" / "bin"
def test_gt_partial_means_next_major(self, tmp_path: Path, nvm: Path) -> None:
# node-semver: ">22" is >=23.0.0, so an installed 22.19.0 does not match.
_install(nvm, "v23.1.0", "v22.19.0")
(tmp_path / "package.json").write_text('{"engines": {"node": ">22"}}')
assert resolve_node_bin(tmp_path) == nvm / "v23.1.0" / "bin"
def test_exact_pin_excludes_neighbours(self, tmp_path: Path, nvm: Path) -> None:
_install(nvm, "v24.14.9", "v24.14.1")
(tmp_path / ".node-version").write_text("=24.14.1")
assert resolve_node_bin(tmp_path) == nvm / "v24.14.1" / "bin"
def test_hyphen_range(self, tmp_path: Path, nvm: Path) -> None:
_install(nvm, "v24.0.0", "v23.5.0", "v22.1.0")
(tmp_path / "package.json").write_text(
'{"engines": {"node": "22.19.0 - 23.11.0"}}'
)
assert resolve_node_bin(tmp_path) == nvm / "v23.5.0" / "bin"
def test_x_wildcard_minor(self, tmp_path: Path, nvm: Path) -> None:
_install(nvm, "v25.0.0", "v24.14.1")
(tmp_path / ".node-version").write_text("24.x")
assert resolve_node_bin(tmp_path) == nvm / "v24.14.1" / "bin"
def test_unparseable_pin_falls_back_to_newest(
self, tmp_path: Path, nvm: Path
) -> None:
_install(nvm, "v24.14.1", "v20.10.0")
(tmp_path / ".nvmrc").write_text("lts/hydrogen")
assert resolve_node_bin(tmp_path) == nvm / "v24.14.1" / "bin"

View File

@@ -1,314 +0,0 @@
# Fleet Mesh Plan — OpenBao + NATS
**Status:** Phases 0–4 complete + verified live across **both nodes** (civil +
primer), including TLS hardening (NATS + OpenBao). Merged to main.
Remaining nice-to-have: the live curl+kill breaker demo (needs a peer-unique
consumed service) — every underlying piece is verified.
## Context
Wild PC's mesh today is a deliberately minimal, read-only gossip: each node
publishes a secret-stripped `NodeRegistry` to Mosquitto MQTT (retained) + an
online/offline LWT, and peers aggregate it into `MeshStateManager`. It is
LAN-only, unauthenticated, carries no secrets, and does nothing with what it
discovers beyond display.
The goal is a **purpose-driven heterogeneous fleet** (media server, security
appliance, TV, comms handler, AI gateway) where nodes come and go as *expected*:
shared LAN-wide config + selected secrets, discoverability + presence, and
binding **by purpose** (a consumer needs `media-index`, not a hostname) with
**circuit-breakers** so churn degrades gracefully instead of hanging.
Two adopted, genuinely-open components (see licensing analysis in chat history):
- **NATS** (CNCF, Apache-2.0) — replaces Mosquitto as the mesh substrate and does
three jobs in one: pub/sub, **JetStream KV** (shared config + registry), and
**TTL keys as presence** (a provider's key vanishes on death → the breaker
trip signal, no separate health plumbing).
- **OpenBao** (Linux Foundation / OpenSSF, MPL-2.0, Vault fork) — the secret
authority behind wildpc's existing `${secret:...}` mechanism.
Design stance: **static single-writer `role` authority, no consensus** —
availability comes from followers running cached local state, not failover.
NATS single-node matches that topology. Consensus stays explicitly out of scope.
## Architecture — planes → components → code seams
| Plane | Component | Primary code seam (verified) |
|-------|-----------|------------------------------|
| Transport + KV + presence | NATS (`wildpc-nats`) | `wildpc-api/.../mqtt_client.py` (whole file), `main.py` lifespan 62-77, `config.py` Settings |
| Shared config + registry + presence | NATS JetStream KV | `mesh.py` `MeshStateManager`; new KV buckets |
| Secret authority | OpenBao (`wildpc-openbao`) | `core/.../config.py:307` `_read_secret` (single chokepoint), `wildpc-api/.../secrets.py` CRUD |
| Binding-by-purpose + breaker | Caddy gateway | `generators/caddyfile.py` `compute_routes` (`remote` kind + ignored `remote_registries` are pre-stubbed), `deploy.py:509-540` `_target_url`/`_requires_env` (local-only today) |
| Authority role | `NodeConfig.role` | `wildpc_core/registry.py` / `manifest.py` |
## Phased plan
Each phase is independently useful; the risky secret-transport work is last.
### Phase 0 — Stand up the two services (no behavior change)
- **`wildpc-nats`**: a `SystemdDeployment` + `RunContainer` (`nats:2` with
JetStream `-js`), data volume `/data/wildpc/wildpc-nats`, `reach: internal`
(`nats.<domain>`). Follow the container YAML shape in `docs/tcp-exposure.md`
(postgres example). Replace the Mosquitto provisioning in `install.sh`
(`setup_mqtt`, `seed_mosquitto_config`, image/port consts) with NATS — or
better, promote it to a bootstrap deployment YAML (today the broker is only a
shell-provisioned container, not a declared deployment).
- **`wildpc-openbao`**: `RunContainer` (`openbao/openbao`), data volume, a file
storage backend, `reach: internal`, **auto-unseal** (local transit/key) so the
node boots unattended (Decision 2).
### Phase 1 — NATS as the mesh transport (swap Mosquitto, behavior-preserving)
- New `nats_client.py` (async-native `nats-py`) replacing `mqtt_client.py`.
Registry → a **JetStream KV bucket `wildpc-registry`** keyed by hostname
(KV last-value replaces MQTT retained). Presence → a **`wildpc-presence`**
bucket with a per-node **TTL key** the node renews (replaces LWT + the 300s
`STALE_TTL` poll; note `prune_stale` is currently uncalled).
- Wire in `main.py` lifespan; rename `mqtt_*` → `nats_*` in `config.py` (keep
`WILDPC_API_` prefix). `MeshStateManager` logic is reused; the KV **watch**
callback drives `update_node`/`set_offline` — and because `nats-py` is
asyncio-native, the `run_coroutine_threadsafe` cross-thread hop for
`broadcast("mesh", …)` **goes away**.
- **Preserve the secret-stripping invariant** from `_registry_to_json` (env /
run_cmd / wildpc_root never on the wire).
- Rename MQTT-named fields: `models.py` `MeshStatus` (217-226) + frontend
`types/index.ts` (326-330), `MeshPanel.tsx`. Drop `paho-mqtt`; add `nats-py`.
Keep zeroconf peer-advert for LAN discovery **and** support explicit NATS seed
URLs for cross-network (Decision 1); drop the dangling unused `_mqtt._tcp`
browse. Rewrite `test_mqtt.py` → `test_nats.py`.
- Fixes a latent gap: today the registry is published **on-connect only** (no
periodic/on-change republish). KV writes on `wildpc apply` fix this naturally.
### Phase 2 — Shared config + presence via JetStream KV
- Buckets: `wildpc-config` (shared LAN config, **authority-written only**),
`wildpc-registry` (per-node), `wildpc-presence` (TTL liveness).
- Add static **`role: authority | follower`** to `NodeConfig` (wildpc.yaml).
Pin **`civil` = authority** (Decision 3); only the authority may write
`wildpc-config`. Authority-down ⇒ shared state read-only, nodes serve cached.
- Followers **watch** `wildpc-config` and reconcile via `wildpc apply`. Presence
key renewal is the churn signal for Phase 3.
### Phase 3 — Cross-node `requires` resolution + gateway binding + breaker (keystone)
- Extend `_target_url`/`_requires_env` (`deploy.py:509-540`) to **fall through to
the mesh registry** when a `ref` isn't satisfied locally (today it's
local-only). The ref is already host-independent — this removes the
"must be local" restriction, not adds a concept.
- Produce the pre-stubbed **`remote` `GatewayRoute`**; thread `remote_registries`
into `compute_routes` (currently accepted-but-ignored) and into the Caddyfile
write at `deploy.py:157` (currently passed nothing). Binding = a **stable local
subdomain route** the gateway re-targets to the remote node's URL — the program
sees one unchanging URL.
- **Circuit-breaker:** gate the remote route on the `wildpc-presence` key
(regenerate + `_reload_gateway` when a provider appears/vanishes), backed by
Caddy passive health (`fail_duration` / `lb_try_duration`) on the
`reverse_proxy` line in `_host_matcher_block`. Consumer degraded-mode =
fallback target or clean 503. (Also carry `health_url` into the registry —
it's currently dropped on `manager: none` refs.)
### Phase 4 — OpenBao as the secret backend (last; needs the security foundation)
- Introduce a **`SecretBackend` seam** at the single read chokepoint
`_read_secret` (`config.py:307`) + the `wildpc-api/.../secrets.py` writer:
`FileSecretBackend` (default, unchanged) and `OpenBaoBackend`. `${secret:NAME}`
syntax is untouched; backend selected in wildpc.yaml or by env.
- Followers auth to OpenBao (token / AppRole); **need-to-know scoping** via
per-deployment policies. Keep the file backend as fallback; migrate secrets in.
### Cross-cutting prerequisite (gates Phase 4 + any cross-network)
- **Transport hardening**: NATS TLS + nkey/user auth (replace Mosquitto's
`allow_anonymous`); OpenBao TLS listener. No secret and no cross-network link
moves until this is in. Cross-network overlay (e.g. Nebula) is orthogonal and
layers under Caddy later.
## Verification (per phase, end-to-end)
- **P1:** bring up two nodes; confirm registry + presence propagate over NATS and
the mesh view (`/mesh/status`, System Map) is identical to the MQTT behavior.
- **P2:** write a key to `wildpc-config` on the authority; observe a follower
watch fire and `wildpc apply` reconcile.
- **P3:** define a service on node A that `requires` a ref provided on node B;
confirm the gateway routes cross-node, then **kill node B** and confirm the
consumer fails fast (curl returns the degraded 503, not a hang) and **recovers**
when B returns.
- **P4:** store a secret in OpenBao; confirm a service renders it via
`EnvironmentFile=` / `--env-file`; then disable the file fallback and confirm
it still resolves.
## Progress
### Phase 0 — DONE + verified (2026-07-07)
Both services stood up **alongside** the live MQTT mesh (no cutover yet):
- `wildpc-nats` (`nats:2`, JetStream) — deployment
`~/.wildpc/deployments/services/wildpc-nats.yaml`, config
`/data/wildpc/wildpc-nats/config/nats-server.conf`. Verified: container active,
`/healthz` ok, JetStream enabled (`/jsz` shows store dir + limits), and a real
**KV round-trip** — created `wildpc-registry` (put/get) and a TTL
`wildpc-presence` bucket (the presence primitive). Test buckets cleaned up.
- `wildpc-openbao` (`openbao/openbao:latest`, v2.5.5) — deployment
`~/.wildpc/deployments/services/wildpc-openbao.yaml`, config
`/data/wildpc/wildpc-openbao/config/openbao.hcl`. Verified: initialized (1
share / threshold 1), unsealed, **KV-v2 secret put/get/delete** round-trip.
Unseal key + root token stored as wildpc secrets `OPENBAO_UNSEAL_KEY` /
`OPENBAO_ROOT_TOKEN`.
**Consciously deferred (tracked):**
1. **Auto-unseal on boot.** OpenBao is unsealed now but a container restart
re-seals it. The manifest `SystemdSpec` has no `exec_start_post` hook, so
proper boot-unseal wiring lands with Phase 4 (either a small manifest
addition or a companion oneshot). Manual recovery: unseal with
`OPENBAO_UNSEAL_KEY`.
2. **`install.sh` bootstrap parity.** Fresh-install provisioning still seeds
Mosquitto; adding NATS/OpenBao there (and retiring Mosquitto) is bootstrap-only
(no runtime impact) and folds in with the Phase 1 cutover.
### Phase 1 — DONE + single-node verified; live cutover done (2026-07-07)
MQTT/Mosquitto transport replaced by NATS JetStream KV.
- New `wildpc_api/mesh_wire.py` — transport-agnostic registry (de)serialization
(moved out of `mqtt_client.py`; preserves the secret-stripping invariant).
- New `wildpc_api/nats_client.py` — `WildpcNATSClient`: connects, PUTs its
registry to the `wildpc-registry` KV bucket, seeds from existing keys, watches
for peer PUT/DELETE, heartbeats a re-PUT (crash liveness via the existing
stale-TTL), and DELETEs its key on graceful stop (immediate peer-offline).
Async-native → the paho cross-thread `run_coroutine_threadsafe` hop is gone.
- `main.py` lifespan, `config.py` (`nats_enabled`/`nats_url`), `models.py`
`MeshStatus` (`connected`/`nats_url`), `nodes.py` `/mesh/status` all rewired.
- Frontend: `types/index.ts` `MeshStatus` + `MeshPanel.tsx` updated to the new
fields. `mqtt_client.py` + `test_mqtt.py` deleted; `test_mesh_wire.py` added.
`mdns.py` dead `_mqtt._tcp` broker-browse removed. `paho-mqtt` → `nats-py`.
- **Live cutover:** `wildpc-api.yaml` env → `WILDPC_API_NATS_*`, `requires:
wildpc-nats`; applied. wildpc-api healthy on NATS, `/mesh/status` connected,
registry (9.7 KiB, 48 deployments) published to KV, **no secrets on the wire**.
- Verification run: ruff clean; 89 api + 196 core tests pass; frontend `tsc`
clean; runtime KV round-trip confirmed against the live `wildpc-nats`.
**Pending second node:** two-node mesh-view parity (peer sees peer, offline on
departure) — needs the second LAN node running the NATS-enabled wildpc-api
pointed at civil's NATS (seed URL) or a clustered `wildpc-nats`. Revert path if
needed: restore `WILDPC_API_MQTT_*` env — but note `mqtt_client.py` is deleted,
so revert = `git checkout main -- wildpc-api` then re-apply. The old Mosquitto
`mqtt` service is left running (dormant) as a safety net; retire it once the
second node is proven on NATS.
### Phase 2 — DONE + single-node verified + tested (2026-07-07)
- **`role` field** on `NodeConfig` + `WildpcConfig`, wired end-to-end:
wildpc.yaml top-level `role:` → config → `_node_config` → registry.yaml →
mesh wire. **`civil` pinned `role: authority`**; default `follower`.
- **Presence** (`wildpc-presence`) — a TTL KV bucket each node renews on the
heartbeat; expiry = the node is gone. Delete-on-stop for immediate departure.
- **Shared config** (`wildpc-config`) — `get_shared_config` / `put_shared_config`
(authority-gated: followers raise `PermissionError`), plus a watch loop that
broadcasts a `config_changed` SSE (the follower `wildpc apply` reconcile hook
hangs off this — inert on one node).
- Hardened `stop()` to bound the NATS drain (can't hang systemd shutdown).
- Verified live: all three buckets present, `civil=online` presence,
`"role": "authority"` on the wire, authority write + follower-deny exercised
against the running `wildpc-nats`.
- **Tests added:** `core/tests/test_fleet_role.py` (role config load + registry
round-trip), `wildpc-api/tests/test_nats_client.py` (role gating, hermetic),
role round-trip in `test_mesh_wire.py`. Suites: **200 core + 93 api** pass.
**Pending second node:** the follower-side reconcile (watch `wildpc-config` →
`wildpc apply`) is wired as an SSE hook but only meaningful with a peer.
### Phase 4 — secret-read backend DONE + verified + tested (2026-07-07)
- New `core/wildpc_core/secret_backends.py`: `SecretBackend` protocol,
`FileSecretBackend` (the historical behavior), `OpenBaoBackend` (KV-v2 read with
file fallback), and `build_backend()` selecting via `WILDPC_SECRET_BACKEND`
(default **file** — production is byte-for-byte unchanged until opted in).
- `_read_secret` (the single chokepoint, `config.py`) now delegates to the active
backend. `${secret:NAME}` syntax untouched.
- OpenBao token bootstraps from the file backend (it can't live in the vault it
unlocks); a missing key / auth failure / unreachable server all fall through to
file, so a partly-migrated vault keeps working.
- Verified live against the running `wildpc-openbao`: a secret stored in the vault
resolves through `${secret:...}` in openbao mode; file-only secrets resolve via
fallback; missing → placeholder; **default file mode unchanged**.
- **Tests:** `core/tests/test_secret_backends.py` (file hit/miss, backend
selection, unreachable-vault + empty-token fallback). Suites: **206 core + 93 api**.
**Hardening:**
1. **Write path — DONE + verified live.** `SecretBackend` gained
`write`/`delete`/`list_names`; `wildpc-api/secrets.py` routes all CRUD through
the active backend, so in openbao mode the dashboard writes to the vault.
Verified: write→vault→read→list→delete round-trip against live OpenBao.
2. **Auto-unseal on boot — DONE + verified.** Added `exec_start_post` to the
systemd spec (general, `-`-prefixed so a hook failure never fails the unit);
`wildpc-openbao` runs `/data/wildpc/wildpc-openbao/unseal.sh` (polls up, unseals
with `OPENBAO_UNSEAL_KEY`). Verified: manual seal → restart → auto-unsealed.
3. **TLS hardening — DONE + verified across both nodes (2026-07-07).**
Reuses the existing ACME **wildcard cert** via `expose.tcp.tls` (the postgres
mechanism) — services present a publicly-trusted cert for
`<name>.civil.payne.io`, so clients verify against the **system CA with no
custom CA to distribute**.
- **NATS:** `tls{}` (wildcard cert in `/tls`) + `authorization{token}`
(`$NATS_TOKEN` from the `NATS_TOKEN` secret). Clients connect to
`tls://wildpc-nats.civil.payne.io:4222` with the token; `WildpcNATSClient`
gained token support (a `tls://` URL → nats-py verifies via system CA).
**civil + primer both cut over**; plaintext now rejected (verified:
`certificate is valid for *.civil.payne.io, not localhost`).
- **OpenBao:** listener `tls_cert_file`/`tls_key_file` from `/tls`; reachable at
`https://wildpc-openbao.civil.payne.io:8200`. Auto-unseal + secret backend
both use the HTTPS URL. Verified: HTTPS serves, **auto-unsealed over HTTPS**,
plaintext rejected, backend reads a secret over HTTPS.
- Coordinated cutover across the whole fleet (both nodes) with a brief,
expected mesh blip; mesh isn't load-bearing so no service impact.
### Phase 3 — cross-node routing + breaker: logic DONE + verified against a real peer (2026-07-07)
**Real second node:** `primer` (192.168.8.129) migrated onto the NATS mesh
(branch checked out, wildpc-api pointed at `nats://civil:4222` via a systemd
drop-in). civil ↔ primer mesh confirmed over the LAN — **Phase 1 two-node parity
done for real**, not simulated. (This also *restored* the civil↔primer mesh my
Phase 1 cutover had split — primer was still on MQTT→civil.)
- `NodeConfig.address` — routable host peers proxy to (wired through registry +
wire; falls back to hostname, which civil resolves for primer).
- `compute_routes` now emits **`remote`** routes for services this node
**consumes** (a local `requires` ref satisfied by an online peer), targeting
`<peer-address>:<port>`. `_host_remote_block` renders a **fail-fast breaker**
(2s dial timeout + passive health) for the there-but-wedged case; **presence
expiry removes the peer from the route set entirely** (gone → no route) — the
primary breaker.
- `wildpc_api/mesh_gateway.py` — on peer join/leave/change (+ startup) the API
re-renders the Caddyfile (same generator `apply` uses + remote routes for
online peers) and reloads the gateway **iff content changed**.
- **Verified:** 4 hermetic tests (route emitted / address fallback / breaker:
peer-absent → no route / unconsumed → no route); **resolution against primer's
real registry** pulled live from the mesh → `wildpc-api → primer:9020`; and the
live integration proven a **no-op** on civil (Caddyfile hash unchanged, gateway
healthy) since civil consumes nothing cross-node yet. Suites: **210 core + 93 api**.
**Remaining:** the full live curl+kill E2E (civil routing to a peer service, then
failing fast on kill) needs a peer-unique service civil consumes — every
underlying piece is verified, but the end-to-end demo needs that provisioning.
primer is now a permanent mesh member on the branch (revert: remove the drop-in +
`git checkout main` on primer).
## Decisions (resolved)
1. **Discovery — both.** Keep mDNS for zero-config LAN peer discovery *and*
support explicit NATS seed URLs so the mesh can span networks. mDNS is the
convenience path on-LAN; seed URLs are the cross-network path. Client connects
via seeds when present, falls back to mDNS-discovered peers on-LAN.
2. **OpenBao unseal — auto for now.** Auto-unseal (local transit/key) so a home
node boots unattended; designed so we can migrate to manual/stricter unseal
later (a supported rekey/unseal-migration, not zero-effort but planned for).
Keep the backend seam clean so the switch is config, not code.
3. **Authority = `civil`.** `civil` (the acme node, `civil.payne.io`) is pinned
as `role: authority`; all others are followers. **Confirmed:** when `civil` is
down, shared config/secrets go **read-only** fleet-wide and every node keeps
serving its own deployments from cached local state.
4. **NATS — single server now, cluster-ready.** Run one `wildpc-nats` today
(matches the single-writer authority). Clustering is deferred but must be a
pure config addition (cluster/routes block + a few more nodes) — no re-architecture.

103
docs/mesh.md Normal file
View File

@@ -0,0 +1,103 @@
# Mesh — runbook
How the nodes find each other, share config and secrets, and route to each
other's services — and what to do when that breaks. The design and its history
are in git: `git log -p -- docs/fleet-mesh-plan.md`.
## What runs where
| piece | where | what it does |
|---|---|---|
| `castle-nats` (civil) | `tls://castle-nats.civil.payne.io:4222`, token `NATS_TOKEN`; monitoring `:8222` | NATS JetStream: the mesh transport and its KV buckets |
| `castle-openbao` (civil) | `https://castle-openbao.civil.payne.io:8200`, KV-v2 mount `castle` | the secret authority behind `${secret:…}` |
| `wildpc-api` (every node) | env `WILDPC_API_NATS_ENABLED/URL/TOKEN` | the mesh client: publishes this node, watches the others |
Config and data: `/data/castle/castle-nats/config/nats-server.conf`,
`/data/castle/castle-openbao/{config/openbao.hcl,unseal.sh,data}`. Both present
the node's ACME wildcard cert (`expose.tcp.tls`), so clients verify against the
system CA; plaintext is refused.
**Nodes.** `civil` is the **authority** (`role: authority` in `wildpc.yaml`);
every other node is a follower (`primer` today). There is no consensus: when
civil is down, shared config and secrets are read-only and every node keeps
running its own deployments from local state.
**KV buckets** on NATS:
| bucket | holds | written by |
|---|---|---|
| `wildpc-registry` | each node's registry, secrets stripped | each node, for itself |
| `wildpc-presence` | a key per node, TTL 90 s, renewed on a heartbeat | each node, for itself |
| `wildpc-config` | shared fleet config | the authority only |
**Secrets.** `wildpc.yaml`'s `secrets:` block selects OpenBao (`backend: openbao`,
`mount: castle`). There is **no file fallback** in that mode — only the bootstrap
tier stays as files in `~/.wildpc/secrets/` (`OPENBAO_ROOT_TOKEN`,
`OPENBAO_UNSEAL_KEY`). civil reads with the root token; followers with a
least-privilege `wildpc-read` token, and `node_prefix` for per-node overrides.
Write with `wildpc secret set`, never by hand (see AGENTS.md §6).
## Is it healthy?
```bash
wildpc mesh status # connected? transport URL, peers
wildpc mesh nodes # each node: online | stale | offline
curl -s localhost:8222/healthz # NATS up
curl -s https://castle-openbao.civil.payne.io:8200/v1/sys/seal-status | jq .sealed
```
- **online** — the node's presence key is live.
- **stale** — no registry update from it for over 5 minutes, but not yet gone.
- **offline** — its presence key expired (90 s) or it left cleanly.
A peer's services are routed only while it is present; when it goes, its routes
drop out and the gateway answers `404 no wildpc route for this host`
immediately, rather than hanging.
## When something is wrong
**A node shows `stale` or `offline`.** On that node: is `wildpc-api` running
(`wildpc service logs wildpc-api`)? Can it reach
`castle-nats.civil.payne.io:4222` by that name (TLS checks the name, so an IP
will not do)? Does its `WILDPC_API_NATS_TOKEN` resolve to the current
`NATS_TOKEN`?
**OpenBao is sealed** (after a restart, if `unseal.sh` failed): every
`${secret:…}` read fails and `wildpc apply` refuses to converge. Unseal by hand:
```bash
curl -s -X POST https://castle-openbao.civil.payne.io:8200/v1/sys/unseal \
-d "{\"key\":\"$(cat ~/.wildpc/secrets/OPENBAO_UNSEAL_KEY)\"}" | jq .sealed
```
**NATS down.** The mesh view freezes and peers go offline; nothing else stops —
each node keeps serving. `wildpc restart castle-nats`, then
`wildpc mesh status` on each node.
**The certificate renewed and a client fails TLS.** `wildpc tls reconcile`
re-materializes the wildcard onto both services and restarts them.
## Routine tasks
**Add a node.** Install Wild PC on it; on its `wildpc-api` set
`WILDPC_API_NATS_ENABLED=true`,
`WILDPC_API_NATS_URL=tls://castle-nats.civil.payne.io:4222`,
`WILDPC_API_NATS_TOKEN=${secret:NATS_TOKEN}`; give it a `secrets:` block with a
`wildpc-read` token and `node_prefix: nodes/<host>`; leave `role` unset
(follower); `wildpc apply wildpc-api`. It appears in `wildpc mesh nodes` within a
heartbeat.
**Use a service another node provides.** `requires: - ref: <service>` in the
consumer's deployment. If no local deployment satisfies it and a present peer
does, the gateway routes `<service>.<domain>` to that peer, with a 2 s dial
timeout so a wedged peer fails fast.
**Shared config.** `wildpc mesh config list|get|set` — `set` works on the
authority only.
**Rotate the NATS token.** `wildpc secret set NATS_TOKEN`, then
`wildpc apply castle-nats` and `wildpc apply wildpc-api` on every node; peers
drop until each is re-applied.
**Cluster NATS** (not done): add a `cluster {}` block and routes to
`nats-server.conf` on each server — configuration only, by design.

View File

@@ -274,7 +274,7 @@ Nested launch spec, discriminated union on `launcher`:
|----------|--------|------------|
| `python` | `uv run --project <source> --no-dev <program>` | `program`, `args` |
| `command` | `which(argv[0])` → resolved path | `argv` |
| `container` | `docker`/`podman` `run` | `image`, `command`, `ports`, `volumes` |
| `container` | `docker`/`podman` `run` (+ `ExecStop=docker stop`) | `image`, `command`, `ports`, `volumes`, `gpus` |
| `compose` | `docker compose -p <project> -f <file> up` (+ `ExecStop=down`) | `file`, `project_name` |
| `node` | `package_manager run script` | `script`, `package_manager` |
@@ -294,6 +294,19 @@ run:
program: my-service # name in [project.scripts]
```
A `container`'s `ports` map container port → host port, published with docker's
`-p`. A bare host port publishes on **every** interface — so it is on the LAN,
whatever `reach:` says. To keep one to this machine, give the host side an
address:
```yaml
run:
launcher: container
image: litellm:main
ports:
4000: 127.0.0.1:4000 # loopback only; `4000: 4000` would be LAN-wide
```
A `compose` launcher supervises a **whole multi-container stack as one systemd
unit** — `ExecStart` runs `docker compose … up` attached (`Type=simple`) and a
generated `ExecStop` runs `… down` so networks/anonymous volumes are reclaimed on
@@ -339,6 +352,20 @@ expose:
health_path: /health # Used by health polling
```
A service with no login of its own can have the gateway ask for one. HTTP Basic
auth, applied wherever the gateway routes it (internal and public names); the
hash is bcrypt, from `caddy hash-password`, so the manifest never holds the
password itself:
```yaml
expose:
http:
internal: { port: 16686 }
basic_auth:
user: paul
password_hash: $2a$14$... # caddy hash-password
```
### `proxy` — Expose the service at a subdomain
`proxy` is a **checkbox** (a bool): `true` means the gateway routes
@@ -404,14 +431,21 @@ route — differing only in whether the target is files on disk or a live proces
The table is shown by `wildpc gateway status`, the dashboard Gateway panel, and
`GET /gateway`; the Caddyfile is generated from it.
**The dashboard and its API.** `wildpc` (the dashboard frontend) and `wildpc-api`
are just two such subdomains (`wildpc.<domain>`, `wildpc-api.<domain>`); the
**The dashboard and its API.** `wild-pc` (the dashboard frontend) and `wildpc-api`
are just two such subdomains (`wild-pc.<domain>`, `wildpc-api.<domain>`); the
dashboard calls the API **cross-origin** (wildpc-api allows CORS `*`). The bare
gateway port (`:9000`) redirects to the dashboard subdomain. On a node with **no
domain** (`gateway.tls: off`), there are no subdomains, so `:9000` serves just the
control plane — the dashboard at `/` plus a `/api` reverse-proxy to wildpc-api —
and other services stay port-only.
**Unrouted subdomains 404.** In `acme` mode every route lives inside one
`*.<domain>` wildcard site, so a host that resolves but has no route still matches
the site and completes its TLS handshake. Each wildcard site therefore ends with an
unmatched `handle` that responds `404` — without it Caddy runs off the end of the
block and returns a bare `200` with an empty body, which makes a typo'd or renamed
subdomain look **up** in a browser while rendering nothing.
#### Host routes need DNS, and the gateway is HTTP-only
A host route only does something once `<host>` resolves **to this node**. For a

View File

@@ -51,6 +51,11 @@ requires:
- { ref: litellm, bind: LITELLM_URL } # bind: project the target's URL into env
```
`path` is appended to the bound URL, for a consumer that wants a sub-path as its
base (e.g. `{ ref: vllm, bind: VLLM_API_BASE, path: /v1 }`).
A raw-TCP target (`expose.tcp`, e.g. postgres) has no URL scheme, so it binds as
`<name>.<domain>:<port>`.
A **program**'s host-package preconditions stay on the program as
`system_dependencies` (a plain list of package names); the model synthesizes the
`{kind: system}` requirements from it for the `functional?` check. This split keeps

View File

@@ -213,15 +213,16 @@ The build is run with `VITE_BASE=/`, so a `vite.config` that reads it (see
@host_my_frontend host my-frontend.example.com
handle @host_my_frontend {
root * /data/repos/my-frontend/dist
try_files {path} /index.html
try_files {path} {path}/index.html /index.html
file_server
}
```
The `try_files {path} /index.html` directive is essential for SPA routing —
it falls back to `index.html` for any path that doesn't match a static file,
letting React Router handle client-side routes. The subdomain is the program name,
not hand-configured.
The `try_files {path} {path}/index.html /index.html` directive handles both SPA
routing and multi-page static sites: it serves a matched static file, then a
nested directory's own `index.html` (so `/sub/dir/` URLs work), and otherwise
falls back to the root `index.html` for any unmatched path — letting React Router
handle client-side routes. The subdomain is the program name, not hand-configured.
## API integration

View File

@@ -233,6 +233,23 @@ def _make_systemd_info(name: str, timer: bool = False) -> SystemdInfo:
return SystemdInfo(unit_name=unit_name, unit_path=unit_path, timer=timer)
def _try_load_config(root: Path | None) -> object | None:
"""Load wildpc.yaml for `root`, or None when there's no config to read.
Listing endpoints call this **once** and reuse the result: load_config()
reparses every program/deployment YAML, so a per-item load turns a listing
into an N x full-config-parse (seconds, not milliseconds).
"""
if root is None:
return None
from wildpc_core.config import load_config
try:
return load_config(root)
except FileNotFoundError:
return None
def _backfill_source(name: str, config: object) -> str | None:
"""Resolve source path from program ref in config."""
if name in config.programs:
@@ -429,6 +446,10 @@ def list_services(include_remote: bool = False) -> list[ServiceSummary]:
summaries: list[ServiceSummary] = []
seen: set[str] = set()
# One config load for the whole listing — the source backfill and the
# non-deployed pass below both read it.
config = _try_load_config(get_wildpc_root())
# Services page shows services (systemd) AND statics (caddy) — both are
# exposed, URL-reachable "services". Not jobs, tools, or remotes.
for _kind, name, deployed in registry.all():
@@ -437,33 +458,19 @@ def list_services(include_remote: bool = False) -> list[ServiceSummary]:
s = _service_from_deployed(name, deployed)
s.node = hostname
# Backfill source
root = get_wildpc_root()
if root and s.source is None:
try:
from wildpc_core.config import load_config
config = load_config(root)
s.source = _backfill_source(name, config)
except FileNotFoundError:
pass
if config is not None and s.source is None:
s.source = _backfill_source(name, config)
summaries.append(s)
seen.add(name)
# Non-deployed from wildpc.yaml
root = get_wildpc_root()
if root:
try:
from wildpc_core.config import load_config
config = load_config(root)
for name, svc in config.services.items():
if name not in seen:
s = _service_from_spec(name, svc, config)
s.node = hostname
summaries.append(s)
seen.add(name)
except FileNotFoundError:
pass
if config is not None:
for name, svc in config.services.items():
if name not in seen:
s = _service_from_spec(name, svc, config)
s.node = hostname
summaries.append(s)
seen.add(name)
# Remote
if include_remote:
@@ -553,39 +560,28 @@ def list_jobs(include_remote: bool = False) -> list[JobSummary]:
summaries: list[JobSummary] = []
seen: set[str] = set()
# One config load for the whole listing (see list_services).
config = _try_load_config(get_wildpc_root())
# Deployed jobs (scheduled)
for _kind, name, deployed in registry.all():
if not deployed.schedule:
continue
s = _job_from_deployed(name, deployed)
s.node = hostname
root = get_wildpc_root()
if root and s.source is None:
try:
from wildpc_core.config import load_config
config = load_config(root)
s.source = _backfill_source(name, config)
except FileNotFoundError:
pass
if config is not None and s.source is None:
s.source = _backfill_source(name, config)
summaries.append(s)
seen.add(name)
# Non-deployed from wildpc.yaml
root = get_wildpc_root()
if root:
try:
from wildpc_core.config import load_config
config = load_config(root)
for name, job in config.jobs.items():
if name not in seen:
s = _job_from_spec(name, job, config)
s.node = hostname
summaries.append(s)
seen.add(name)
except FileNotFoundError:
pass
if config is not None:
for name, job in config.jobs.items():
if name not in seen:
s = _job_from_spec(name, job, config)
s.node = hostname
summaries.append(s)
seen.add(name)
# Remote
if include_remote: