Two features built this session, committed together as a checkpoint before the
deployment-model refactor (they share files, so kept as one rollback point):
- Public exposure via Cloudflare tunnel: a `public` toggle on services (requires
proxy), gateway.public_domain/tunnel_id config, a cloudflared ingress generator
(generators/tunnel.py) that maps <name>.<public_domain> to the gateway's
internal host, deploy wiring, and docs/tunnel-setup.md.
- Frontends as `static`-runner services: a RunStatic runner served by Caddy
(file_server), no systemd unit (like `remote`); the Caddyfile generator now
derives static routes from services instead of a behavior==frontend branch.
The proxy->bool refactor (770574f) missed the shared CLI test fixture, which
still used proxy: {caddy: {path_prefix: ...}} — failing ~25 CLI tests with
'Input should be a valid boolean'. Set it to proxy: True.
Also drop the now-dead 'caddy' entry from _STRUCTURAL_KEYS (no nested caddy
key exists anymore) and refresh a stale proxy.caddy docstring reference.
internal-CA HTTPS forces every device to trust a private root, which breaks down
on some clients (Android browsers). acme mode instead obtains a real Let's Encrypt
wildcard cert (*.<domain>) via a DNS-01 challenge, so browsers trust it with zero
CA install — while services stay internal-only (no inbound exposure, no public A
records; only a transient _acme-challenge TXT touches the public zone, and LAN DNS
resolves the names to a private IP).
- Config: GatewayConfig/NodeConfig gain domain, acme_email, acme_dns_provider
(plumbed through load/save + registry + deploy, omitted-when-empty for
round-trip stability). tls stays the discriminator: off | internal | acme.
- Caddyfile generator: acme branch emits a global {email, acme_dns <provider>
{env.TOKEN}} block + one *.<domain> wildcard site with host matchers derived
from the service name (<name>.<domain>); path/static stay on :<port>. Shared
_host_matcher_block helper (reused by off-mode). CASTLE_ACME_STAGING=1 toggles
the LE staging CA; acme-without-domain falls back to HTTP with a warning.
- deploy(): _acme_preflight warns (never writes) when domain, the gateway-service
token env, or the CLOUDFLARE_API_TOKEN secret is missing.
- install.sh: opt-in --with-dns-plugin[=provider] builds a DNS-plugin Caddy via
xcaddy (pinned) to /usr/local/bin/caddy, which the gateway picks up on deploy.
- Tests: TestCaddyfileTlsAcme (global block, wildcard site, derived host matcher,
path routes on :port, staging toggle, no-domain fallback). Docs: gateway.tls
modes table + full acme/DNS-01 section (setup, wild-central LAN DNS, staging).
The gateway was HTTP-only by design, but a plain-HTTP origin is not a browser
"secure context", so apps that need WebCrypto (device identity, E2E crypto)
can't run when proxied — only localhost or HTTPS qualifies.
Add an opt-in `gateway.tls: internal`. When set, each host route
(proxy.caddy.host) is emitted as its own `<host> { tls internal; reverse_proxy }`
site, served over HTTPS by Caddy's local CA (Caddy binds :443 and redirects :80).
Path-prefix/static routes stay on the HTTP :<port> site — give a service a host
route to put it on HTTPS. Default (unset/off) keeps the existing HTTP-only
Caddyfile (single :<port> block, auto_https off, host matchers inline).
Plumbing mirrors gateway_port: GatewayConfig.tls (castle.yaml load/save) →
NodeConfig.gateway_tls (registry load/save, set in deploy) → the Caddyfile
generator. Tests cover both modes; docs/registry.md documents the option plus
the two operational requirements (binding 443/80 via the unprivileged-port
sysctl, and trusting Caddy's root CA on clients) and the host-route-vs-prefix
guidance that motivates it.
- Introduced `_write_castle_config` function to scatter nested config into separate YAML files for programs, services, and jobs.
- Updated `castle_root` fixture to utilize the new config writing method.
- Added tests for configuration aggregation and scattering in `test_health.py`.
- Removed legacy unit handling from `manifest.py` and refactored related code in `config.py`.
- Updated documentation to reflect new configuration directory structure.
- Removed obsolete unit tests related to unit expansion.
Generated systemd units hardcoded a thin PATH (~/.local/bin + system dirs)
while dev-verb builds used a separate _EXTRA_PATH_DIRS list, so a service
could be built and run with different tools on PATH. Hoist a single
USER_TOOL_PATH_DIRS in config.py, consumed by both stacks._build_env
(build) and the systemd unit generator (runtime).
Also prefer pnpm's modern $PNPM_HOME/bin shim over the bare $PNPM_HOME dir
(older installs leave a stale version wrapper there) and add
/usr/local/go/bin so Go-based builds/services resolve their toolchain.
A service/job's env is now exactly its defaults.env — castle injects no hidden
convention vars. Values support ${port}/${data_dir}/${name} placeholders
(resolved at deploy, alongside ${secret:…}), so a program's own env var names
map to castle's computed values without hardcoding.
Why: the auto-injected <PREFIX>_PORT/<PREFIX>_DATA_DIR were a guess at the
program's env names — right for castle-scaffolded services, dead weight for
adopted ones (lakehouse carried two dead vars; notification-bridge/backup jobs
too). They also weren't visible in the config editor (computed at deploy), which
was the source of the 'four env vars but the UI shows none' mystery.
- core: resolve_env_vars gains a context (${port}/${data_dir}/${name});
deploy builds env from defaults.env only — no <PREFIX>_* injection, no
port_env. Removed the port_env field and the dead _env_prefix helper.
- cli: 'service/job create' gains repeatable --env KEY=VALUE (replaces
--port-env); 'program create' scaffolds <PREFIX>_PORT/_DATA_DIR: ${…} for new
daemons.
- app: removed the 'Port env' field; the Environment editor (defaults.env) is
the single place, with a placeholder hint.
- live migration: central-context/castle-api/power-graph/protonmail mapped their
real vars explicitly; lakehouse → just LAKEHOUSED_DAEMON_PORT: ${port}, data
stays in ~/.lakehoused. Verified all services healthy on their ports, dead
vars gone, zero failed units.
- docs: registry.md/design.md/stack guides + findings updated to the explicit
model.
core 94 / cli 24 / api 52 green; ruff + app build clean.
Move all instance state into ~/.castle/ with clear separation:
- code/ — user project source (was components/)
- artifacts/ — generated specs, built content (was generated/, static/)
- data/ — service runtime data (was /data/castle/)
- secrets/ — credentials
castle.yaml moves to ~/.castle/castle.yaml as the canonical location.
Source paths use repo: prefix for git repo programs (castle-api, app)
and relative paths for user projects (code/central-context).
Add idempotent install.sh for bootstrapping infrastructure (Docker,
Caddy, MQTT, Postgres, Neo4j) with interactive service detection.
Remove components/ from repo (now in ~/.castle/code/), deinit
submodules, remove .gitmodules.
- Updated the terminology from "components" to "programs" across the codebase, including in config loading, saving, and manifest specifications.
- Introduced a new `stacks.py` file to handle lifecycle actions for development stacks, implementing handlers for Python and React Vite stacks.
- Adjusted tests to reflect the new program structure and ensure proper functionality.
- Revised documentation to align with the new terminology and structure, ensuring clarity on the purpose and configuration of programs, services, and jobs.
