Carry each deployment's `requires` (deployment refs, no secrets) from config →
registry → the mesh MQTT payload, and expose it on /mesh/deployments. The System
Map resolves a remote deployment's ref against the provider set across nodes
(same machine → local provider → other machine) and draws a dashed cross-node
edge. This is the multi-node payoff: e.g. primer's castle-api → civil's mqtt.
A deployment's identity becomes (name, kind), so a tool, service, and job can
share a name (a `backup` trio). Kind is structural — which per-kind store / dir —
not a derived discriminator threaded around.
- CastleConfig holds per-kind stores (services/jobs/tools/statics/references);
all_deployments()/deployments_named()/deployment(kind,name) replace the single
flat `deployments` dict. A construction-only `deployments` InitVar routes a flat
map into the stores by kind_for.
- Storage: deployments/<store>/<name>.yaml (per-kind dirs), with read-compat for
the old flat + services/+jobs/ layouts.
- Registry keyed by composite "<kind>/<name>" + name field; get()/all()/named()/put().
- Job units carry a -job marker (castle-<name>-job.{service,timer}); services and
everything else unchanged — so a service and a job can coexist.
- deploy/apply/lifecycle thread kind through unit naming, activate/deactivate,
is_active; caddyfile/tunnel/relations iterate via all_deployments/registry.all.
- Subdomain uniqueness validated across kinds (a service and a static can't share one).
182 core tests pass, incl. a new same-name-trio suite (round-trip, distinct units,
subdomain guard). API/CLI/UI updated in follow-up phases.
Frontend builds triggered from the castle web app run inside the castle-api
systemd service, whose PATH omits nvm's versioned node dir — so `pnpm build`
died with `node: not found` even though it worked from an interactive shell.
Introduce a per-program node convention: a program declares its version the
ecosystem-standard way (.node-version / .nvmrc / package.json engines.node),
and castle_core.toolchains.resolve_node_bin() maps it to a concrete nvm bin dir
(CASTLE_NODE_VERSIONS_DIR, default ~/.nvm/versions/node; newest match wins).
The same resolver feeds both sites that run a program's node:
- build time: stacks._build_env() prepends the pinned node for the dev-verb
subprocess (keyed on the source dir), so `castle program build` uses the
program's node regardless of caller.
- run time: deploy._build_deployed() stores it in Deployment.path_prepend,
which the systemd generator puts ahead of the default unit PATH — so a
`launcher: node` service runs its program's node.
A pinned-but-uninstalled version fails loud with an `nvm install` hint instead
of a cryptic `node: not found`. Unpinned → no injection (no guessing).
Also: the systemd generator now honors an explicit PATH in defaults.env as a
full override instead of clobbering it with a trailing Environment=PATH line
(systemd's last-assignment-wins rule had silently defeated the documented
escape hatch).
Pins app/.node-version and documents the convention in the react-vite stack.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expose raw-TCP services (postgres) by name at <name>.<domain>:<port> and
cut the gateway's ACME wildcard cert onto them so they present a trusted
cert. Replaces the proxy/public booleans with a single `reach` enum
(off|internal|public); legacy input still parses via derived accessors.
Core:
- expose.tcp{port,tls} + TlsSpec(material: pair|combined|off, reload)
- tls.py: materialize cert files from Caddy's wildcard, reconcile on
renewal; `castle tls` CLI; optional cert_obtained events-exec hook
(gateway.cert_hook, gated so a plugin-less Caddy still parses)
- apply waits (bounded) for the wildcard to issue before materializing so
a fresh-node TLS service starts with its cert in place, then scopes
materialization to the deployments being applied
- reach: internal|public requires an expose block (no silent no-op);
public raw-TCP guarded until tunnel support lands
- chain.pem (${tls_ca}) is the issuer chain (leaf stripped), a real CA
bundle distinct from cert.pem
- one shared ${...} resolver (resolve_placeholders) for env and container
run fields; run.env now expands like volumes/args; $$ escapes a literal
- validate the generated Caddyfile before reloading the gateway so an
invalid config never degrades routing
Docs: docs/tcp-exposure.md. Tests cover reach/expose validation,
placeholder expansion + escape, issuer-chain material, TLS materialize.
Reconcile the running system to declared config in a single operation,
replacing the old deploy && start plus the per-kind enable/disable/
install/uninstall verbs. The mechanism varies by manager (systemd unit /
PATH install / gateway route); the verb never does.
Core (castle_core.deploy.apply):
- render (units, Caddyfile, tunnel) then reconcile: activate every
enabled deployment that's down, restart any whose unit bytes changed,
deactivate the disabled ones. Returns ApplyResult (the enacted diff).
- --plan computes the diff and touches nothing.
- restart-on-change is exact: _render_unit_files is the single source of
truth for unit bytes, used both to write units and to predict restarts,
so the plan can't drift from what deploy writes.
Desired state:
- DeploymentBase.enabled (default True) — the declarative on/off. apply
converges to it; a disabled deployment is defined but not running and
gets no gateway route (else it would 502). Registry omits enabled when
True, keeping existing registries byte-identical.
CLI:
- add `castle apply [name] [--plan]`; keep `restart` as the one
imperative bounce, plus status/doctor/logs/gateway/program.
- retire deploy, start, stop (top-level); service/job deploy|enable|
disable|start|stop; tool install|uninstall; program install|uninstall.
- next-step strings + doctor hints now say `castle apply`.
Verified: core 129 + cli 31 green; live `castle apply` on a converged
node reports "nothing to do" with no spurious restarts; --plan shows an
empty diff; gateway + api stay up.
Replace the conflated `runner` axis with two orthogonal ones: `manager`
(systemd|caddy|path|none) — who supervises/realizes a deployment — and, for
systemd only, a nested `launcher` (python|command|container|compose|node) — how
the process starts. ServiceSpec and JobSpec collapse into one manager-
discriminated DeploymentSpec union (Systemd/Caddy/Path/Remote); the services/
and jobs/ config dirs collapse into one deployments/ dir. The human "kind"
(service|job|tool|static|reference) is fully derived (kind_for), never stored —
the frontend kind is renamed static. behavior is gone.
- core: DeploymentSpec union + LaunchSpec + kind_for; legacy-aware loader
normalizes old runner shapes; CastleConfig.deployments with derived
services/jobs/tools views; registry.Deployment carries manager/launcher/kind.
- cli: service/job/tool as filtered views + a deployment group; --behavior→--kind,
create --runner→--launcher; lifecycle dispatches over config.deployments.
- castle-api: /deployments primary with /services,/jobs as views; summaries
derive kind; PUT/DELETE /config/deployments/{name} (services/jobs aliased).
- app: KindBadge, frontend→static everywhere, pick-a-kind creation wizard,
per-kind config editors.
- docs: single deployments/ layout, manager/launcher, static kind throughout.
Live migration verified byte-identical: regenerated Caddyfile and every unit
ExecStart line unchanged, so nothing restarted. Suites: core 124, cli 25,
castle-api 55; dashboard build + type-check clean.
Two features built this session, committed together as a checkpoint before the
deployment-model refactor (they share files, so kept as one rollback point):
- Public exposure via Cloudflare tunnel: a `public` toggle on services (requires
proxy), gateway.public_domain/tunnel_id config, a cloudflared ingress generator
(generators/tunnel.py) that maps <name>.<public_domain> to the gateway's
internal host, deploy wiring, and docs/tunnel-setup.md.
- Frontends as `static`-runner services: a RunStatic runner served by Caddy
(file_server), no systemd unit (like `remote`); the Caddyfile generator now
derives static routes from services instead of a behavior==frontend branch.
Collapse the two proxy shapes (path_prefix + host) to a single checkbox: a
service is either port-only, or exposed at <service-name>.<gateway.domain>. Path
routes and their prefix-stripping foot-guns are gone; the subdomain is always the
service name (rename the service to change it).
- Schema: CaddySpec is just `enable` (presence = expose). service_proxy_targets
returns (expose, port, base_url); Deployment carries `subdomain` (was
proxy_path/proxy_host). Registry/deploy/mesh/CLI updated in lockstep.
- Generator: compute_routes emits one host route per exposed service/frontend
(address = name). acme mode = one *.<domain> site with a reverse_proxy matcher
per service + a file_server matcher per frontend; the :<port> site redirects to
the dashboard. off mode = a HTTP control plane on :<port> (dashboard at / +
/api → castle-api); other services are port-only.
- Dashboard/API: castle-app and castle-api are ordinary subdomains. The dashboard
derives the API base at runtime (castle-api.<domain> on a subdomain, else /api)
and calls it cross-origin — castle-api already allows CORS *. Frontends build
with VITE_BASE=/ (served at their subdomain root).
- CLI: `service create` drops --path/--host; --no-proxy makes it port-only.
- Docs/tests reworked for the model; docs use generic placeholders only (no
personal host/domain/IP details).
internal-CA HTTPS forces every device to trust a private root, which breaks down
on some clients (Android browsers). acme mode instead obtains a real Let's Encrypt
wildcard cert (*.<domain>) via a DNS-01 challenge, so browsers trust it with zero
CA install — while services stay internal-only (no inbound exposure, no public A
records; only a transient _acme-challenge TXT touches the public zone, and LAN DNS
resolves the names to a private IP).
- Config: GatewayConfig/NodeConfig gain domain, acme_email, acme_dns_provider
(plumbed through load/save + registry + deploy, omitted-when-empty for
round-trip stability). tls stays the discriminator: off | internal | acme.
- Caddyfile generator: acme branch emits a global {email, acme_dns <provider>
{env.TOKEN}} block + one *.<domain> wildcard site with host matchers derived
from the service name (<name>.<domain>); path/static stay on :<port>. Shared
_host_matcher_block helper (reused by off-mode). CASTLE_ACME_STAGING=1 toggles
the LE staging CA; acme-without-domain falls back to HTTP with a warning.
- deploy(): _acme_preflight warns (never writes) when domain, the gateway-service
token env, or the CLOUDFLARE_API_TOKEN secret is missing.
- install.sh: opt-in --with-dns-plugin[=provider] builds a DNS-plugin Caddy via
xcaddy (pinned) to /usr/local/bin/caddy, which the gateway picks up on deploy.
- Tests: TestCaddyfileTlsAcme (global block, wildcard site, derived host matcher,
path routes on :port, staging toggle, no-domain fallback). Docs: gateway.tls
modes table + full acme/DNS-01 section (setup, wild-central LAN DNS, staging).
Adds "a stack whose default is a substrate": a shared self-hosted Supabase
backend plus a `--stack supabase` that scaffolds per-app projects (migrations +
edge functions + static UI) which deploy against it. Apps own their code and
stay repo-durable; only their rows/blobs live on the shared substrate.
- compose runner: new RunCompose supervises a multi-container stack as one
systemd unit (ExecStart=`compose up`, generated ExecStop=`compose down`);
secrets via EnvironmentFile. Reusable beyond Supabase. Deployment.stop_cmd
carries the teardown command through the registry.
- supabase stack: CLI --stack choice, STACK_DEFAULTS→frontend with
build.outputs=[public], _scaffold_supabase() (migrations/RLS/functions/
static UI/app manifest), and SupabaseHandler with a forward-only idempotent
migration runner (plan_migrations + psql build) and deno dev-verbs.
- docs: docs/stacks/supabase.md, registered in CLAUDE.md; compose runner
documented in registry.md.
- tests: compose run/stop cmd, systemd ExecStop, compose host-route TLS,
migration planner, supabase verb resolution, create --stack supabase.
The gateway was HTTP-only by design, but a plain-HTTP origin is not a browser
"secure context", so apps that need WebCrypto (device identity, E2E crypto)
can't run when proxied — only localhost or HTTPS qualifies.
Add an opt-in `gateway.tls: internal`. When set, each host route
(proxy.caddy.host) is emitted as its own `<host> { tls internal; reverse_proxy }`
site, served over HTTPS by Caddy's local CA (Caddy binds :443 and redirects :80).
Path-prefix/static routes stay on the HTTP :<port> site — give a service a host
route to put it on HTTPS. Default (unset/off) keeps the existing HTTP-only
Caddyfile (single :<port> block, auto_https off, host matchers inline).
Plumbing mirrors gateway_port: GatewayConfig.tls (castle.yaml load/save) →
NodeConfig.gateway_tls (registry load/save, set in deploy) → the Caddyfile
generator. Tests cover both modes; docs/registry.md documents the option plus
the two operational requirements (binding 443/80 via the unprivileged-port
sysctl, and trusting Caddy's root CA on clients) and the host-route-vs-prefix
guidance that motivates it.
Driven by adopting lakehouse (an existing daemon that bundles its own SPA) —
see docs/findings-lakehouse.md.
#3 deploy reloads the gateway. castle deploy regenerated the Caddyfile but
left the running Caddy on the old config, so new proxy routes were silently
dead. Deploy now reloads the gateway when it's running.
#1 castle expose <program>. Turns an adopted program into a service
(run/port/health/proxy/systemd) in one command — the missing
daemon-to-service step. Flags: --port --health --path --run --port-env
--host --no-proxy.
#2 port_env mapping. A service can declare expose.http.internal.port_env so
castle sets the env var the program actually reads (e.g. lakehoused reads
LAKEHOUSED_DAEMON_PORT, not castle's convention LAKEHOUSE_PORT). Castle now
genuinely drives an adopted daemon's bind port.
#4a auto-base for react-vite. The build passes VITE_BASE = the gateway serve
prefix (/<name>/, or / for castle-app); vite.config reads it. A castle-built
frontend now works at its subpath with no hand-tuned base. castle-app's
vite.config updated as the reference.
#4b host-based routing. proxy.caddy.host routes a whole hostname to the backend
root via a host matcher inside the :9000 site, so a root-based SPA (base="/")
serves unchanged — the fix for proxying an app castle can't rebuild. Caddyfile
now emits 'auto_https off' (HTTP-only gateway on a non-standard port).
Nit: activate skips the editable reinstall when the tool is already on PATH.
Tests: core 94, cli 24, api 52; ruff + app build clean. Verified live: lakehouse
runs under systemd, API at /lakehouse, full UI (SPA boots) via host routing.
- Added base_url field to DeployedComponent for external URL proxying
- Updated Caddyfile generator to proxy to base_url when present instead of requiring local port
- Added runner: remote handling in deploy command (returns empty run_cmd, marks as unmanaged)
- Reformatted long ternary expression in registry.py for readability
This enables castle to manage services that proxy to external endpoints without requiring a local process.
Move all instance state into ~/.castle/ with clear separation:
- code/ — user project source (was components/)
- artifacts/ — generated specs, built content (was generated/, static/)
- data/ — service runtime data (was /data/castle/)
- secrets/ — credentials
castle.yaml moves to ~/.castle/castle.yaml as the canonical location.
Source paths use repo: prefix for git repo programs (castle-api, app)
and relative paths for user projects (code/central-context).
Add idempotent install.sh for bootstrapping infrastructure (Docker,
Caddy, MQTT, Postgres, Neo4j) with interactive service detection.
Remove components/ from repo (now in ~/.castle/code/), deinit
submodules, remove .gitmodules.
