_on_path() led with a bare shutil.which(), which resolves against the calling process's PATH. When castle-api runs inside the repo venv (uv run), a tool merely colocated there — e.g. a stray editable a dev uv-pip-installed — read as "installed on PATH", so the UI showed installed and the converge planner reported no drift. Strip the running interpreter's own venv bin from the search path; tools installed anywhere else on PATH still count.
- service/static/tool/job detail pages now link back to their program;
backfill the program ref on registry-only static+job details (the
registry Deployment carries no program ref).
- new RelatedDeployments card surfaces a deployment's requires edges both
directions (Depends on / Required by) as links; references render inert.
- replace the service detail Subdomain/Served-at rows with a launch icon
in the header that opens the gateway URL in a new tab.
Both machines (civil, primer) are on the current on-disk layout, so the
read-compat for older formats is dead weight. Remove it:
- config loader: only deployments/<kind>/<name>.yaml (drop flat deployments/*.yaml
and the services/+jobs/ split); drop the run.runner→manager normalizer.
- registry loader: require composite <kind>/<name> keys + manager/kind (registry.yaml
is regenerated every apply, so it's always current).
- manifest: drop the proxy/public→reach legacy input and the component→program alias
(the derived proxy/public read accessors stay). reach field defaults already match.
- config_editor: parse incoming specs directly (frontend sends the current shape).
- save_config: drop the flat-file migration cleanup.
Bootstrap seeds move to bootstrap/deployments/<kind>/ and install.sh seeds that
per-kind tree directly. The legacy→current translation for terse test fixtures
moves into each test conftest (out of production). All suites green (362).
- endpoint-level: a deployment/program edit via /config/* leaves castle.yaml
globals byte-identical (the exact regression that shipped — the endpoints were
tested but nobody asserted globals survived)
- write_program_file leaves globals untouched (core)
- /secrets/info reports the backend
Still uncovered (tracked): castle mesh CLI, direct-read refactor (dns/stacks),
backend-aware doctor, and all frontend (no test runner configured).
Root-cause hardening for the dropped-globals bug: config edits went through the
full-document save_config, which rewrote castle.yaml globals (+ every resource
file) on every deployment/program edit — so an unmodeled global (role, secrets)
was dropped.
- add write_deployment_file / write_program_file: persist ONE resource file,
never touching globals or other resources
- config_editor deployment/program save/delete/enable now use those instead of
save_config, so a deployment edit can't rewrite castle.yaml
- save_config now preserves ANY unmanaged top-level global (not just secrets)
- _aggregate_yaml surfaces role + secrets (raw-yaml round-trip completeness)
- tests: scoped write leaves globals byte-identical; unmanaged global survives
Verified live: editing lakehouse via the endpoint leaves castle.yaml unchanged.
The deployment form now manages only the wiring (ENV_VAR -> secret name) with a
set/unset status badge and a link to the Secrets page for the value — it no
longer fetches or inline-edits secret values. A deployment only ever stores
${secret:NAME}; values live in the backend and are managed in one place (the
Secrets page). Removes the pre-vault holdover of pulling secret values into the
deployment-editing surface.
save_config rewrites castle.yaml from scratch and only re-emitted gateway/repo/
roots — so any config save (dashboard deployment edit, config_editor) silently
STRIPPED top-level 'role' and the 'secrets:' backend block. Effect: the node
reverted to a file-backend follower, the vault stopped resolving, and a
subsequent apply baked <MISSING_SECRET> into service env (hit castle-lakehouse).
Now re-emit role (from config) + preserve the secrets block (read from the
existing file, since it's not modeled on CastleConfig). Regression test added.
A bare `ssh host 'castle …'` / `'pnpm …'` couldn't find the toolchain: uv,
castle, pnpm, and nvm node live on the interactive PATH (~/.zsh.d/*.sh), which
non-interactive/non-login shells don't load. Add ensure_shell_path() to mirror
just those PATH entries into ~/.zshenv (which non-interactive zsh does load),
so every bootstrapped node can drive castle/pnpm over ssh. Idempotent, marker-
guarded, and skipped for non-zsh users.
Post-migration sweep of legacy file-based secret code:
- deploy.py + doctor.py checked for the token *file* → false 'secret not found'
warnings now that tokens live in the vault; use read_secret (backend-aware)
- drop now-unused SECRETS_DIR imports
- refresh stale docstring/prompt text (${secret} 'reads ~/.castle/secrets' →
'via the active backend'; SecretsEditor add-prompt)
Kept (legitimate file paths): the bootstrap tier (OPENBAO_* token/unseal,
cloudflared creds dir) + the rendered 0600 env files.
- new Secrets page (/secrets, nav entry): lists vault secrets, reveal/copy,
add/edit/delete, shows active backend + addr + role; read-only on followers
- GET /secrets/info (backend/addr/role/writable); fix /secrets CRUD to use the
castle.yaml-selected backend (was defaulting to the now-empty file backend)
- OpenBaoBackend.list_names drops folder entries (node-prefix groups)
- tests: conftest forces file backend so host castle.yaml can't leak into tests
app builds clean (tsc 0); endpoints verified live against the vault.
The System Map subsumes the read-only Graph diagnostic, so remove the page,
its route, and nav entry (the /graph API + useGraph stay — the map and palette
consume them).
Add a host switcher under the sidebar brand: it shows which castle host you're
driving (the is_local node) and, when peers are present, jumps to a peer's
dashboard at castle.<domain>, preserving the current view. Node summaries now
carry gateway_domain so the switcher can build each host's URL.
- castle mesh status|nodes|config (list/get/set) — hits the local castle-api;
surfaces the shared-config write path (authority-gated)
- API: GET/PUT /mesh/config/{key:path}, GET /mesh/config; nats_client gains
list_shared_config
- tests: NATS integration suite against a real broker via a docker fixture
(peer discovery, presence, graceful-offline, shared config, secrets-off-wire);
plus HTTP-layer /mesh/config tests. Closes the runtime-coverage gap.
- AGENTS.md §8 updated for NATS + the mesh CLI
100 api + 211 core tests pass.
NATS tls:// + token and OpenBao HTTPS via the ACME wildcard cert (expose.tcp.tls),
no custom CA. civil + primer both cut over; plaintext rejected on both. Server
config + unseal.sh are instance-side (/data/castle, ~/.castle).
- CastleNATSClient accepts a token; a tls:// server URL makes nats-py verify the
broker against the system CA (trusts the wildcard's Let's Encrypt issuer, no
custom CA needed)
- config: CASTLE_API_NATS_TOKEN; main wires it through
Server side (instance): nats-server.conf gains tls{} (wildcard cert via
expose.tcp.tls) + authorization{token}; clients use tls://castle-nats.<domain>.
- SecretBackend gains write/delete/list_names; FileSecretBackend + OpenBaoBackend
implement them (vault KV-v2 POST/DELETE/LIST); castle-api/secrets.py routes all
CRUD through the active backend (dashboard writes to the vault in openbao mode)
- add systemd exec_start_post (general; '-' prefix so a hook failure can't fail
the unit); castle-openbao runs an unseal.sh on boot using OPENBAO_UNSEAL_KEY
- tests: file write/read/list/delete round-trip
Verified live: write→vault→read→list→delete against OpenBao; seal→restart→
auto-unsealed. TLS hardening remains (deferred; gates cross-network/real secrets).
- secret_backends.py: SecretBackend protocol, FileSecretBackend (historical),
OpenBaoBackend (KV-v2 read + file fallback), build_backend() env-selected
- _read_secret delegates to the active backend; default is file, so production
is unchanged until CASTLE_SECRET_BACKEND=openbao
- OpenBao token bootstraps from the file backend; missing/unreachable falls back
- tests: file hit/miss, backend selection, unreachable + empty-token fallback
Read path verified live against castle-openbao. Write path, auto-unseal-on-boot,
and TLS hardening documented as remaining for full OpenBao production use.
- add static 'role' (authority|follower) to NodeConfig/CastleConfig, wired from
castle.yaml through the registry to the mesh wire; civil pinned authority
- castle-presence: TTL KV bucket each node renews (churn signal); delete-on-stop
- castle-config: authority-gated get/put + change-watch SSE (follower reconcile
hook hangs off it)
- bound the NATS drain on stop() so shutdown can't hang
- tests: role config/registry round-trip, wire round-trip, role write-gating
Single-node verified; follower reconcile pending the second node.
Phased plan to evolve the mesh into a purpose-driven fleet: NATS (JetStream KV +
presence) replacing Mosquitto, OpenBao as the secret authority, cross-node
requires resolution + gateway circuit-breaker, static civil=authority role.
Records resolved decisions and Phase 0/1 progress.
The program-level Capability model (provides/consumes) was a vestige of an
earlier design superseded by the deployment-level requires/Requirement graph.
It was read only into a diagnostic node and surfaced as an always-empty
'Capabilities' panel; nothing acted on it. Removed from manifest, relations,
the CLI re-export, and the System Map.
The `castle` CLI and the `castle-api` service are two independent in-process
drivers of `castle_core`. Each resolved DATA_DIR/REPOS_DIR at import time from
its own process env (default /data/castle), persisted nowhere — so they silently
diverged, and `apply`/dashboard-apply crashed on a non-existent /data.
Make the loaded CastleConfig the single source of truth:
- Resolve data_dir/repos_dir only in load_config (env > castle.yaml > default),
anchored to the config root; drop the DATA_DIR/REPOS_DIR module globals and the
import-time file read entirely — no global twin that can disagree with the file.
- Thread config.data_dir/repos_dir through ensure_dirs, _env_context, tls_dir_for
(now unified — deploy no longer inlines the tls path), and create/add/clone.
- ensure_dirs raises an actionable CastleDirError instead of a bare PermissionError;
the api surfaces it as 422.
- doctor: "data dir writable" check + WARN when CASTLE_DATA_DIR/REPOS_DIR env
overrides the file (the one remaining cross-process divergence vector).
- install.sh persists data_dir/repos_dir into castle.yaml (idempotent, non-default).
- Docs: registry.md globals + AGENTS.md roots.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mesh payload didn't include a node's acme domain, so a peer had no way
to build launch URLs for another machine's exposed apps. Publish
gateway_domain in the MQTT registry payload and surface it per-deployment on
/mesh/deployments as `domain`. The dashboard palette and System Map now build
`https://<subdomain>.<domain>` for remote http-exposed apps (references use
their base_url), making primer's apps launchable from civil.
- Mesh endpoints now gate the http port on `subdomain` (the registry's "exposed"
signal), matching the local reach-gated derivation — so a remote reach:off
service (e.g. primer's castle-gateway) no longer shows a phantom :9000.
- The focus/resync effect no longer rebuilds the node set while a connection is
being dragged; a background refetch mid-drag was silently cancelling the
connection, so drag-to-expose/-require intermittently "did nothing".
The keyboard twin of the map's inspect panel: ⌘K (or the "Launch…" nav button)
from any page. Empty query = the "Start Menu" (browsable launchable apps, local +
remote, grouped by machine badge); typing searches every deployment. Enter
launches the app in a new tab; per-result actions jump to it on the map
(/map?focus=<id>, which the map now honors by selecting + centering) or open its
Castle details. Navigation/launch only — no mutation — for v1.
- "Start Menu": frontends and http-exposed services get a launch ↗ that opens
the running app (https://<name>.<domain>) in a new tab — on the node (hover)
and in the panel header.
- The right-click / kebab NodeMenu is gone; its actions (Restart, expose, Delete)
now live in the inspect panel as an Actions row, so one gesture (click a node)
is the whole control surface. Panel header separates Launch (the app) from
Castle details (the management page). Remote nodes stay view-only.
- Local node's deployments render on top (with a "⬡ <host> · this node" label and
the lane headers), then a labeled "other machines" divider, then each remote
machine as a band below — same kind-columns, so cross-node edges align vertically.
- Remote nodes get full UX parity: draggable (persisted), selectable, and a unified
focus adjacency (consumes/consumed-by over local + remote + reference ids) drives
identical highlight + inspect-panel behavior whether a node is local or remote.
The panel shows a machine badge on cross-node deps and disables edit on remotes.
Remote machines' deployments now sit in the same kind-columns as local (so a
cross-node edge runs straight down its column), are draggable with persisted
positions, and the lane headers lift above the topmost band so they head every
machine's columns.
The registry writer/reader are hand-rolled field lists, so the new requires
field was dropped on save — leaving the mesh payload empty. Emit and read it.
Carry each deployment's `requires` (deployment refs, no secrets) from config →
registry → the mesh MQTT payload, and expose it on /mesh/deployments. The System
Map resolves a remote deployment's ref against the provider set across nodes
(same machine → local provider → other machine) and draws a dashed cross-node
edge. This is the multi-node payoff: e.g. primer's castle-api → civil's mqtt.
The consumption audit only matched host:port inside a single value (a URL,
DATABASE_URL), missing the very common split-var shape (X_HOST=localhost +
X_PORT=1883, e.g. castle-api's mqtt config). Resolve those pairs too — factored
into a shared _resolve helper. A bare *_PORT with no *_HOST is ignored so a
deployment's own listen port isn't mistaken for a dependency. Adds test_audit.py.
- Loading spinner while the graph query is in flight.
- Deployment cards drop the "consumed by N" line and external-ref chips — both
now live in the inspect panel, so cards are just name/port/reach.
- A node's external (reference) edges are drawn only while it's selected, then
removed — no permanent web of long cross-map lines.
- LAN exposure edges animate like the Internet ones (animation = live traffic).
_target_url only resolved http_exposed deployments, so a `requires` with a
`bind` pointing at a `manager: none` reference (an external resource) injected
nothing. Return the reference's base_url, so binding an external endpoint into a
consumer's env works — e.g. litellm's vllm-payne reference → VLLM_API_BASE.
Reframes the map around consumes / consumed_by and adds:
- Inspect on select: dim the rest, light a node's consumes/consumed-by, and a
side panel listing both (removable/navigable) with protocol chips.
- Protocol-typed dependency edges (http/pg/bolt/mqtt) + reach badges; TCP infra
now shows its port.
- External resources: a registry lane of reference nodes with an "add external"
form; consumption of one renders as a chip on the consumer (no cross-map line).
- Suggested (undeclared) consumption as dashed amber edges — click to accept.
- Capability/candidates in the panel (alternative providers of the same protocol).
- A "Machines" band rendering other mesh nodes' deployments (read-only).
Adds useSuggestions / useMeshDeployments / useSaveReference hooks and the
GraphEndpoint / GraphSuggestion / MeshDeployment types.
Graph model gains, all derived (GET /graph auto-exposes via asdict):
- endpoints[{protocol,port}] + reach per node — reusing http_exposed/tcp_port,
so raw-TCP infra (postgres/neo4j/mqtt) is finally visible and edges can be
protocol-typed by the target's socket.
- base_url for reference (external) nodes.
- provides/consumes capability types, activating the dormant ProgramSpec fields.
New surfaces:
- core/audit.py + GET /graph/suggestions: SUGGESTS undeclared `requires` by
matching a deployment's env endpoint values against provider sockets. Never
writes; the graph stays declaration-derived (docs/relationships.md).
- kind-scoped PUT/DELETE /config/references/{name} for external resources.
- GET /mesh/deployments (flattened remote deployments + derived endpoints), and
the mesh MQTT payload now carries tcp_port + base_url (forward-compatible,
still no secrets) so peers can resolve cross-node endpoints.
A spatial control surface for the whole node. Tools, jobs, services, and
frontends sit in lanes with program-source grips; edges show requires
(dependency), same-program siblings, and exposure to LAN/Internet targets.
Direct manipulation, each backed by the existing config PUT + apply:
- drag a node to LAN/Internet to set reach (internal/public)
- delete an exposure line or a node (with confirm) to unexpose/remove
- right-click or the per-node ⋯ menu: open, restart, expose, delete
- click the source grip to open the backing program
Nodes are draggable with Shift-/toggle box-select; layout persists to
localStorage with a reset control. Adds useSetReach/useDeleteDeployment
hooks, the @xyflow/react dep, and the nav entry + route.
_reload_gateway validated the generated Caddyfile in castle-api's own
process env, which lacks CLOUDFLARE_API_TOKEN. Under acme the Caddyfile
references {env.CLOUDFLARE_API_TOKEN}, so `caddy validate` failed on an
empty token and the reload was silently skipped — every API-driven apply
left the running Caddy stale (route changes never took effect until a
manual reload). Validate with the gateway service's own resolved env so
the DNS-provider token is present.
Model:
- Drop `requires` from ProgramSpec; requires is now deployment-only.
Collapse Requirement to {ref, bind} with kind defaulting to "deployment"
(drop the unused `version`). System-package preconditions stay on the
program as `system_dependencies`, synthesized into {kind: system} for the
functional check. relations.py/deploy.py read only the deployment's requires.
- create.py seeds a stack's substrate dependency onto the deployment, not the
program. Docs + tests updated.
UI:
- Add a `requires` editor (useRequires) on service, job, and static detail
pages — edit service→service deps (ref + optional bind env var).
- Restore the gateway route table "Open" column: a kind-aware in-app link to
each route's deployment.
- Reformat the reach control: name baked into each option
(localhost:PORT (local) / <name>.<domain> (internal) / <name>.<pubdomain> (public)).
- Services page: sort statics in with systemd services by name, add a
search + kind-filter bar mirroring the programs page.
API: /config/{services,jobs,tools,static}/{name} now pin the twin they
target — _save_deployment/_delete_deployment take an explicit kind so a
patch to a 'backup' service can't bleed into a 'backup' job/tool. Add
/tools and /static endpoints; keep /deployments/{name} kind-agnostic.
New test_kind_twins proves per-kind save/delete isolation on disk.
UI: ConfigPanel and CreateDeploymentForm write to the kind-scoped
resource; GatewayPanel/NodeDetail/DeploymentsSection link via a shared
detailPath(name, kind) helper instead of the ambiguous /deployment/:name.
Includes incidental ruff-format reflow of untouched api files.
Thread kind through every CLI resolver so a tool, service, and job can
share a name. Reads go through config.deployment(kind,name) /
deployments_named(name) / all_deployments(); writes target the per-kind
store (config.services/jobs/tools/statics). delete cascades over
(kind,name) tuples; run/info resolve via registry.named().
Resolve deployments by (name, kind) via the core per-kind stores / registry.get:
- get_service resolves a service-or-static; get_job a job; get_component returns
the first kind for a name (the /services|/jobs|/tools/{name} endpoints are the
unambiguous addresses). _save_deployment stores into the kind store, preferring
the existing same-named deployment on a partial patch (can't derive kind from a
partial payload). delete/set-enabled act across a name's kinds.
- All iterate-all loops use registry.all(); lookups use registry.get/named.
- services.py/logs.py resolve the managed deployment and use kind-aware unit names.
- mqtt registry (de)serialization keyed by composite "<kind>/<name>".
API + core suites green (85 + 182).
A deployment's identity becomes (name, kind), so a tool, service, and job can
share a name (a `backup` trio). Kind is structural — which per-kind store / dir —
not a derived discriminator threaded around.
- CastleConfig holds per-kind stores (services/jobs/tools/statics/references);
all_deployments()/deployments_named()/deployment(kind,name) replace the single
flat `deployments` dict. A construction-only `deployments` InitVar routes a flat
map into the stores by kind_for.
- Storage: deployments/<store>/<name>.yaml (per-kind dirs), with read-compat for
the old flat + services/+jobs/ layouts.
- Registry keyed by composite "<kind>/<name>" + name field; get()/all()/named()/put().
- Job units carry a -job marker (castle-<name>-job.{service,timer}); services and
everything else unchanged — so a service and a job can coexist.
- deploy/apply/lifecycle thread kind through unit naming, activate/deactivate,
is_active; caddyfile/tunnel/relations iterate via all_deployments/registry.all.
- Subdomain uniqueness validated across kinds (a service and a static can't share one).
182 core tests pass, incl. a new same-name-trio suite (round-trip, distinct units,
subdomain guard). API/CLI/UI updated in follow-up phases.
