- container launcher: `gpus:` passes through as docker `--gpus`.
- requires: `path:` is appended to the bound URL, for consumers that want a
sub-path as their base (e.g. an OpenAI-compatible server's /v1).
- container services now get `ExecStop=-docker stop wildpc-<name>`. SIGTERM
only reached the `docker run` client, so a stop/restart left the container
running unsupervised and holding its name, and the next start failed with
a name conflict until systemd gave up.
Generated service units took systemd's default start limit (5 starts in
10s), which can never trip when each failed attempt outlives the window.
openclaw failed ~5s in and restarted every ~5.5s, so the counter reset on
every cycle: 260k restarts at ~7.4s CPU each, over a core burned
continuously for days, with the unit stuck in `activating (auto-restart)`
rather than anything a status check would flag.
Service units now get exponential restart backoff (RestartSteps=8,
RestartMaxDelaySec=300) and a start limit sized against it (10 starts per
hour, in [Unit] where systemd reads it). A broken service retries with a
growing delay for ~17min — wide enough to ride out a boot-time dependency
race — then lands in `failed` where it's visible and stops consuming CPU.
All four values are SystemdSpec fields, overridable per deployment.
Timer-driven oneshot jobs are unaffected; they never restart. systemd
older than 254 ignores RestartSteps/RestartMaxDelaySec with a warning and
keeps the flat RestartSec, so the start limit still applies there.
resolve_node_bin() could not parse a range, so it fell back to "pin by the
first number in the string" and took the newest installed version of that
major. For openclaw's engines pin `>=22.19.0 <23 || >=23.11.0` that picked
v22.17.0 — below the range's own floor — and baked it into the service's
unit PATH. The daemon refused to start on it and crash-looped 260k+ times
(~7.4s CPU per attempt, every ~5.5s, since before the journal's retention
window). StartLimitBurst never tripped because five attempts span ~27s,
outside the 10s StartLimitIntervalUSec.
Parse the pin properly: split on `||`, expand each comparator (>=, >, <=,
<, =, ^, ~, hyphen ranges, x/* wildcards, partial versions) into bounds
with node-semver's partial semantics, and pick the newest installed version
satisfying any alternative. openclaw now resolves to v24.18.0 via its
second alternative. A pin this parser cannot read at all still falls back
to newest-installed rather than guessing a version.
When nothing satisfies the pin it now fails with the installed versions
listed, so the next occurrence is diagnosable at apply time rather than
six days later in a restart loop.
/services and /jobs called get_wildpc_root() + load_config() *inside* the
loop over registry.all(), so every item paid a full reparse of wildpc.yaml
and all 77 program/deployment YAMLs (~55ms registry + ~103ms config). With
38 services and 12 jobs on this node that was 6.8s and 3.0s per request,
uncached — the dashboard's /services and /scheduled pages each fire one of
these plus /status, which is where their >10s load came from.
Hoist the load to a single _try_load_config() call per request; the source
backfill and the non-deployed pass both read that one config. Responses are
byte-identical to before; both endpoints now serve in ~0.27s.
The dashboard's deployment kept the pre-rename name `wildpc`, so it was
published at wildpc.<domain> while everything else had moved to Wild PC
naming. Rename the deployment and its program to `wild-pc` (the subdomain
is the deployment name) and follow the `_DASHBOARD` constant in both the
Caddyfile generator and doctor — that constant also drives the `:9000`
redirect and doctor's `wildpc program build` hint, which is why the
program is renamed alongside the deployment.
Browsing the old name exposed a second problem: in acme mode every route
lives inside one `*.<domain>` wildcard site, so an unrouted host still
matches the site and completes its TLS handshake. Caddy then ran off the
end of the block and returned a bare 200 with an empty body — a typo'd or
renamed subdomain looked *up* in a browser while rendering nothing. Each
wildcard site now ends with an unmatched `handle` that responds 404.
The API subdomain is untouched: the dashboard derives its API origin by
stripping the first label, so wild-pc.<domain> still resolves to
wildpc-api.<domain>, and the WebSocket origin gate matches on
gateway.domain rather than a hostname.
Also seed the bootstrap dashboard deployment with the `requires:
wildpc-api` block the live deployment already carried, so a fresh install
declares the dependency.
Orange terminal prompt icon with white stroke-based line art on rounded rect,
matching the shared visual identity across Wild Central, Cloud, and Directory.
A job's systemd unit is castle-<name>-job.service and its EnvironmentFile points
at castle-<name>-job.service.env, but _write_secret_env_file wrote
castle-<name>.service.env (kind defaulted to 'service'). Every secret-using job
failed to start (Failed to load environment files). Thread the deployment kind
through to secret_env_path so the filename matches the unit.
Writing a secret by hand meant knowing the active backend and its layout;
getting it wrong (a file write on an OpenBao fleet) left the value where the
resolver never reads it, so ${secret:NAME} silently degraded to the literal
<MISSING_SECRET:NAME> placeholder that a service then used as its credential.
- `castle secret {list|set|get|rm}` reads/writes the ACTIVE backend, so there's
no wrong store to pick. `set NAME` with no value reads a hidden prompt / stdin.
- `castle apply` (and --plan) now refuses to converge any deployment whose
${secret:...} refs don't resolve in the active backend: exits non-zero, writes
nothing, and names each deployment + secret + the `castle secret set` fix.
- New helpers in core/config.py: active_secret_backend(), active_backend_name(),
secret_refs(). Gate impl: deploy._unresolved_secrets() + ApplyResult.blocked.
- Tests: test_deploy_secret_gate.py, TestSecretRefs, test_secret.py. Docs: AGENTS.md.
Static (manager: caddy) deployments always emitted the SPA fallback
`try_files {path} /index.html`, which serves the root index.html for
every unmatched path. That's correct for React/Vite apps but swallows a
multi-page content site's in-page links back to the homepage, so a Hugo
site (payne.io) couldn't navigate off its landing page.
Add an explicit `spa: bool` on CaddyDeployment (default True, preserving
existing SPA behavior). When False, the gateway serves plain file_server
— resolving directory indexes (/posts/ -> /posts/index.html) and 404ing
missing paths. Threaded through the registry snapshot + mesh and the
route computation so both the internal wildcard and public apex blocks
honor it. `castle program create --stack hugo` now sets spa: false.
Stacks declare host toolchains (uv/pnpm/node/hugo/deno/psql) that can drift
from what's installed and, worse, from what's on a running service's PATH.
Make those dependencies first-class and visible.
Model (core):
- stacks.ToolRequirement + StackHandler.tools declare each stack's toolchains
(command, purpose, phase, install_hint); tools_for() is the single source.
- relations synthesizes them as `kind: tool` requirements so functional?/graph
account for them; checked runtime-env-aware (run-phase tools of a systemd
service are probed against the service's PATH, not the shell) via a shared
generators.systemd.runtime_path helper the unit generator also uses, so the
checker can't drift from the generator. hint_for() makes every unmet
requirement actionable.
- stack_status: the derived per-stack health the CLI/API/UI all render.
- config: add ~/.deno/bin to USER_TOOL_PATH_DIRS so deno (supabase edge fns)
is found by services and the check, same as ~/.local/bin and the pnpm dirs.
Surfaces:
- castle stack list|info (new resource) + GET /stacks/status, /stacks/{name}
(GET /stacks stays a bare name list for the create-form select).
- castle doctor gains a "Stacks & dependencies" section (FAIL for an enabled
deployment's missing tool, WARN otherwise, unused stacks skipped).
- castle apply preflight warns (advisory, like _acme_preflight) when a tool is
missing where a service runs; ConvergePanel renders those warnings.
- Dashboard Stacks page: per-stack tool checklist with versions + copyable
install hints, program links, and verb chips.
Tests: relations (drift + hints), doctor (ok/fail/skip), /stacks endpoints.
The nav label was already "System" and the page is SystemMap; only the URL
was stale. Move it to /system, add a back-compat redirect from /map, and update
the command-palette navigations. Unrelated graph node type "map" left as-is.
Register an existing repo as a program from the /programs page — the web
equivalent of `castle program add`. An inline form with an editable,
autocompleting path bar browses the *server's* filesystem (a browser's own
file dialog only sees the client machine) and also accepts a git URL.
- core: extract the adopt logic (target parsing, stack/command detection,
ProgramSpec build) into castle_core.adopt so the CLI and API share it; the
CLI's add.py now delegates to it.
- castle-api: GET /fs/browse (directory listing, per-entry stat errors skipped
so one unreadable child can't 403 the dir) and POST /programs/adopt.
- app: AddProgramForm + Add-program button on the Programs page; useBrowse /
useAdoptProgram hooks.
Also tidy ProgramCard: deployments render below the description, and the
deployment name label is hidden when it matches the program title.
Swap the canonical KIND_ICONS.program glyph from Package to SquareCode, and route the palette + System-map program affordances through it. (The Programs nav icon in Layout.tsx changes too, committed separately with its own WIP.)
The command palette searched deployments but not the program catalog. List programs too: a deployed program stays one row with a program-source jump; a source-only program gets its own row. Also swap the System-map jump to the Gauge icon and give each row a leading kind icon.
The path-tool lifecycle offered only Enable/Disable (keyed on enabled), so an enabled-but-not-on-PATH tool showed just "Disable" with no way to converge it. Add an Install/Uninstall (apply) action whenever desired (enabled) and actual (installed) disagree.
Every surface hand-mapped deployment kind → lucide icon, so they could (and did) drift. Add KIND_ICONS/kindIcon to lib/labels beside kindLabel, and route KindBadge (→ all detail headers/cards/sections), the Overview tiles, and the program links through it.
_on_path() led with a bare shutil.which(), which resolves against the calling process's PATH. When castle-api runs inside the repo venv (uv run), a tool merely colocated there — e.g. a stray editable a dev uv-pip-installed — read as "installed on PATH", so the UI showed installed and the converge planner reported no drift. Strip the running interpreter's own venv bin from the search path; tools installed anywhere else on PATH still count.
- service/static/tool/job detail pages now link back to their program;
backfill the program ref on registry-only static+job details (the
registry Deployment carries no program ref).
- new RelatedDeployments card surfaces a deployment's requires edges both
directions (Depends on / Required by) as links; references render inert.
- replace the service detail Subdomain/Served-at rows with a launch icon
in the header that opens the gateway URL in a new tab.
Both machines (civil, primer) are on the current on-disk layout, so the
read-compat for older formats is dead weight. Remove it:
- config loader: only deployments/<kind>/<name>.yaml (drop flat deployments/*.yaml
and the services/+jobs/ split); drop the run.runner→manager normalizer.
- registry loader: require composite <kind>/<name> keys + manager/kind (registry.yaml
is regenerated every apply, so it's always current).
- manifest: drop the proxy/public→reach legacy input and the component→program alias
(the derived proxy/public read accessors stay). reach field defaults already match.
- config_editor: parse incoming specs directly (frontend sends the current shape).
- save_config: drop the flat-file migration cleanup.
Bootstrap seeds move to bootstrap/deployments/<kind>/ and install.sh seeds that
per-kind tree directly. The legacy→current translation for terse test fixtures
moves into each test conftest (out of production). All suites green (362).
- endpoint-level: a deployment/program edit via /config/* leaves castle.yaml
globals byte-identical (the exact regression that shipped — the endpoints were
tested but nobody asserted globals survived)
- write_program_file leaves globals untouched (core)
- /secrets/info reports the backend
Still uncovered (tracked): castle mesh CLI, direct-read refactor (dns/stacks),
backend-aware doctor, and all frontend (no test runner configured).
Root-cause hardening for the dropped-globals bug: config edits went through the
full-document save_config, which rewrote castle.yaml globals (+ every resource
file) on every deployment/program edit — so an unmodeled global (role, secrets)
was dropped.
- add write_deployment_file / write_program_file: persist ONE resource file,
never touching globals or other resources
- config_editor deployment/program save/delete/enable now use those instead of
save_config, so a deployment edit can't rewrite castle.yaml
- save_config now preserves ANY unmanaged top-level global (not just secrets)
- _aggregate_yaml surfaces role + secrets (raw-yaml round-trip completeness)
- tests: scoped write leaves globals byte-identical; unmanaged global survives
Verified live: editing lakehouse via the endpoint leaves castle.yaml unchanged.
The deployment form now manages only the wiring (ENV_VAR -> secret name) with a
set/unset status badge and a link to the Secrets page for the value — it no
longer fetches or inline-edits secret values. A deployment only ever stores
${secret:NAME}; values live in the backend and are managed in one place (the
Secrets page). Removes the pre-vault holdover of pulling secret values into the
deployment-editing surface.
save_config rewrites castle.yaml from scratch and only re-emitted gateway/repo/
roots — so any config save (dashboard deployment edit, config_editor) silently
STRIPPED top-level 'role' and the 'secrets:' backend block. Effect: the node
reverted to a file-backend follower, the vault stopped resolving, and a
subsequent apply baked <MISSING_SECRET> into service env (hit castle-lakehouse).
Now re-emit role (from config) + preserve the secrets block (read from the
existing file, since it's not modeled on CastleConfig). Regression test added.
A bare `ssh host 'castle …'` / `'pnpm …'` couldn't find the toolchain: uv,
castle, pnpm, and nvm node live on the interactive PATH (~/.zsh.d/*.sh), which
non-interactive/non-login shells don't load. Add ensure_shell_path() to mirror
just those PATH entries into ~/.zshenv (which non-interactive zsh does load),
so every bootstrapped node can drive castle/pnpm over ssh. Idempotent, marker-
guarded, and skipped for non-zsh users.
Post-migration sweep of legacy file-based secret code:
- deploy.py + doctor.py checked for the token *file* → false 'secret not found'
warnings now that tokens live in the vault; use read_secret (backend-aware)
- drop now-unused SECRETS_DIR imports
- refresh stale docstring/prompt text (${secret} 'reads ~/.castle/secrets' →
'via the active backend'; SecretsEditor add-prompt)
Kept (legitimate file paths): the bootstrap tier (OPENBAO_* token/unseal,
cloudflared creds dir) + the rendered 0600 env files.
- new Secrets page (/secrets, nav entry): lists vault secrets, reveal/copy,
add/edit/delete, shows active backend + addr + role; read-only on followers
- GET /secrets/info (backend/addr/role/writable); fix /secrets CRUD to use the
castle.yaml-selected backend (was defaulting to the now-empty file backend)
- OpenBaoBackend.list_names drops folder entries (node-prefix groups)
- tests: conftest forces file backend so host castle.yaml can't leak into tests
app builds clean (tsc 0); endpoints verified live against the vault.
The System Map subsumes the read-only Graph diagnostic, so remove the page,
its route, and nav entry (the /graph API + useGraph stay — the map and palette
consume them).
Add a host switcher under the sidebar brand: it shows which castle host you're
driving (the is_local node) and, when peers are present, jumps to a peer's
dashboard at castle.<domain>, preserving the current view. Node summaries now
carry gateway_domain so the switcher can build each host's URL.
- castle mesh status|nodes|config (list/get/set) — hits the local castle-api;
surfaces the shared-config write path (authority-gated)
- API: GET/PUT /mesh/config/{key:path}, GET /mesh/config; nats_client gains
list_shared_config
- tests: NATS integration suite against a real broker via a docker fixture
(peer discovery, presence, graceful-offline, shared config, secrets-off-wire);
plus HTTP-layer /mesh/config tests. Closes the runtime-coverage gap.
- AGENTS.md §8 updated for NATS + the mesh CLI
100 api + 211 core tests pass.
NATS tls:// + token and OpenBao HTTPS via the ACME wildcard cert (expose.tcp.tls),
no custom CA. civil + primer both cut over; plaintext rejected on both. Server
config + unseal.sh are instance-side (/data/castle, ~/.castle).
- CastleNATSClient accepts a token; a tls:// server URL makes nats-py verify the
broker against the system CA (trusts the wildcard's Let's Encrypt issuer, no
custom CA needed)
- config: CASTLE_API_NATS_TOKEN; main wires it through
Server side (instance): nats-server.conf gains tls{} (wildcard cert via
expose.tcp.tls) + authorization{token}; clients use tls://castle-nats.<domain>.
- SecretBackend gains write/delete/list_names; FileSecretBackend + OpenBaoBackend
implement them (vault KV-v2 POST/DELETE/LIST); castle-api/secrets.py routes all
CRUD through the active backend (dashboard writes to the vault in openbao mode)
- add systemd exec_start_post (general; '-' prefix so a hook failure can't fail
the unit); castle-openbao runs an unseal.sh on boot using OPENBAO_UNSEAL_KEY
- tests: file write/read/list/delete round-trip
Verified live: write→vault→read→list→delete against OpenBao; seal→restart→
auto-unsealed. TLS hardening remains (deferred; gates cross-network/real secrets).
- secret_backends.py: SecretBackend protocol, FileSecretBackend (historical),
OpenBaoBackend (KV-v2 read + file fallback), build_backend() env-selected
- _read_secret delegates to the active backend; default is file, so production
is unchanged until CASTLE_SECRET_BACKEND=openbao
- OpenBao token bootstraps from the file backend; missing/unreachable falls back
- tests: file hit/miss, backend selection, unreachable + empty-token fallback
Read path verified live against castle-openbao. Write path, auto-unseal-on-boot,
and TLS hardening documented as remaining for full OpenBao production use.
- add static 'role' (authority|follower) to NodeConfig/CastleConfig, wired from
castle.yaml through the registry to the mesh wire; civil pinned authority
- castle-presence: TTL KV bucket each node renews (churn signal); delete-on-stop
- castle-config: authority-gated get/put + change-watch SSE (follower reconcile
hook hangs off it)
- bound the NATS drain on stop() so shutdown can't hang
- tests: role config/registry round-trip, wire round-trip, role write-gating
Single-node verified; follower reconcile pending the second node.
