refactor(env): explicit defaults.env with placeholders; drop auto-injection

A service/job's env is now exactly its defaults.env — castle injects no hidden
convention vars. Values support ${port}/${data_dir}/${name} placeholders
(resolved at deploy, alongside ${secret:…}), so a program's own env var names
map to castle's computed values without hardcoding.

Why: the auto-injected <PREFIX>_PORT/<PREFIX>_DATA_DIR were a guess at the
program's env names — right for castle-scaffolded services, dead weight for
adopted ones (lakehouse carried two dead vars; notification-bridge/backup jobs
too). They also weren't visible in the config editor (computed at deploy), which
was the source of the 'four env vars but the UI shows none' mystery.

- core: resolve_env_vars gains a context (${port}/${data_dir}/${name});
  deploy builds env from defaults.env only — no <PREFIX>_* injection, no
  port_env. Removed the port_env field and the dead _env_prefix helper.
- cli: 'service/job create' gains repeatable --env KEY=VALUE (replaces
  --port-env); 'program create' scaffolds <PREFIX>_PORT/_DATA_DIR: ${…} for new
  daemons.
- app: removed the 'Port env' field; the Environment editor (defaults.env) is
  the single place, with a placeholder hint.
- live migration: central-context/castle-api/power-graph/protonmail mapped their
  real vars explicitly; lakehouse → just LAKEHOUSED_DAEMON_PORT: ${port}, data
  stays in ~/.lakehoused. Verified all services healthy on their ports, dead
  vars gone, zero failed units.
- docs: registry.md/design.md/stack guides + findings updated to the explicit
  model.

core 94 / cli 24 / api 52 green; ruff + app build clean.
This commit is contained in:
2026-06-14 17:06:46 -07:00
parent fd562f7468
commit 7314b5cddb
13 changed files with 122 additions and 89 deletions

View File

@@ -195,9 +195,9 @@ Services and jobs can reference a program via `program:` for description
fallthrough and source code linking. They can also exist independently
(e.g., `castle-gateway` runs Caddy — not our software).
Convention-based env vars (`<PREFIX>_PORT`, `<PREFIX>_DATA_DIR`) are
generated automatically during deploy. Only non-convention values need
`defaults.env`.
A service's env is exactly its `defaults.env` — castle injects nothing
implicitly. Values may use `${port}`/`${data_dir}`/`${name}`/`${secret:…}`
placeholders, which deploy resolves into the registry's flat `env`.
**`$CASTLE_HOME/artifacts/specs/registry.yaml`** (per-node, not in the repo, generated by `castle deploy`) — Node config:
@@ -222,8 +222,8 @@ deployed:
```
The node config says what's deployed *here* and with what concrete
values. `castle deploy` reads the spec from the repo, generates
convention-based env vars, resolves secrets, resolves binary paths,
values. `castle deploy` reads the spec from the repo, resolves the
`defaults.env` placeholders and secrets, resolves binary paths,
and writes the registry. Systemd units and Caddyfile are then generated
from the registry — never from the spec directly.
@@ -521,9 +521,9 @@ What exists today:
`$CASTLE_HOME/artifacts/specs/registry.yaml` (node config). Systemd units and
Caddyfile generated from registry with fully resolved paths. No repo references
in runtime artifacts.
- **Convention-based env generation** — `castle deploy` auto-generates
`<PREFIX>_DATA_DIR=$CASTLE_DATA_DIR/<name>` and `<PREFIX>_PORT` from
the manifest. Only non-convention values need `defaults.env`.
- **Explicit env with placeholders** — a deployment's env is exactly its
`defaults.env`; `castle deploy` resolves `${port}`/`${data_dir}`/`${name}`/
`${secret:…}` into concrete values. No hidden convention injection.
- **Gateway** — Caddy on port 9000, Caddyfile generated from registry
- **API** — `castle-api` on port 9020, reads from registry (optional
castle.yaml fallback for non-deployed programs)

View File

@@ -40,10 +40,13 @@ the daemon's built-in default and we declared the same value. Castle's
convention assumes a castle-aware program; an adopted one doesn't read
`<PREFIX>_PORT`.
**Fix:** let a service declare which env var carries the port, e.g.
`expose.http.port_env: LAKEHOUSED_DAEMON_PORT`, so castle sets *that* to the
configured port and genuinely drives the bind. (`defaults.env` is the manual
workaround.)
**Fixed** (superseding the interim `port_env` field): auto-injection of the
convention vars was dropped entirely. A service's env is now exactly its
`defaults.env`, with `${port}`/`${data_dir}`/`${name}` placeholders for castle's
computed values. lakehouse maps just what it reads —
`LAKEHOUSED_DAEMON_PORT: ${port}` — and keeps its own data root, with no dead
vars. Castle-native services map their `<PREFIX>_PORT`/`_DATA_DIR` the same way
(scaffolded by `castle program create`).
### #3 — `castle deploy` regenerates the Caddyfile but never reloads Caddy (High, bug)

View File

@@ -255,17 +255,36 @@ manage:
exec_reload: "caddy reload ..."
```
### `defaults` — Default environment
### `defaults` — Environment
`defaults.env` is the **single, explicit source** of the env a service/job runs
with — what you write here is exactly what lands in the systemd unit. Castle
does **not** inject hidden convention vars; whatever env var your program reads
for its port, data dir, etc., you map here.
```yaml
expose: { http: { internal: { port: 9001 }, health_path: /health } }
defaults:
env:
MY_SERVICE_PORT: ${port} # the program's own port var ← expose.port
MY_SERVICE_DATA_DIR: ${data_dir} # = $CASTLE_DATA_DIR/<name>
CENTRAL_CONTEXT_URL: http://localhost:9001
API_KEY: ${secret:MY_API_KEY}
```
Castle resolves `${secret:NAME}` by reading `~/.castle/secrets/NAME`.
Never store secrets in castle.yaml or project directories.
Values may contain placeholders that castle resolves at deploy:
| Placeholder | Expands to |
|-------------|------------|
| `${port}` | the service's `expose.http.internal.port` (so it can't drift) |
| `${data_dir}` | `$CASTLE_DATA_DIR/<program-or-name>` (the dedicated data volume) |
| `${name}` | the deployment name |
| `${secret:NAME}` | the contents of `~/.castle/secrets/NAME` |
Hardcode the values instead if you prefer; the placeholders just save you from
repeating castle's computed paths/ports. `castle program create` scaffolds the
`${port}`/`${data_dir}` lines for new services. Never store secrets in
castle.yaml — use `${secret:…}`.
## Job blocks
@@ -431,9 +450,10 @@ variable (both expand `~` and resolve relative paths):
Defined in `core/src/castle_core/config.py`: `CASTLE_HOME` (with derived
`CODE_DIR`, `SECRETS_DIR`, `SPECS_DIR`, `CONTENT_DIR`) and the independent
`DATA_DIR` (`CASTLE_DATA_DIR`). `castle deploy` passes each service its data
path via the generated `<PREFIX>_DATA_DIR` env var. Systemd unit/timer paths are
fixed by systemd's user-unit convention.
`DATA_DIR` (`CASTLE_DATA_DIR`). A service reaches its data path by mapping
`${data_dir}` (= `$CASTLE_DATA_DIR/<name>`) to the env var its program reads, in
`defaults.env`. Systemd unit/timer paths are fixed by systemd's user-unit
convention.
## Manifest models

View File

@@ -129,16 +129,21 @@ services:
systemd: {}
```
Convention-based env vars (`MY_SERVICE_DATA_DIR`, `MY_SERVICE_PORT`) are
generated automatically by `castle deploy`. Only non-convention values
need `defaults.env`:
The env a service runs with is exactly what's in `defaults.env` — castle injects
nothing implicitly. Map the vars your settings read (above, `env_prefix:
"MY_SERVICE_"``MY_SERVICE_PORT`/`MY_SERVICE_DATA_DIR`) to castle's computed
values with the `${port}`/`${data_dir}` placeholders:
```yaml
defaults:
env:
MY_SERVICE_PORT: ${port} # = expose.http.internal.port
MY_SERVICE_DATA_DIR: ${data_dir} # = $CASTLE_DATA_DIR/my-service
CENTRAL_CONTEXT_URL: http://localhost:9001
```
`castle program create` scaffolds the `${port}`/`${data_dir}` lines for you.
## Application entry point
```python
@@ -305,8 +310,9 @@ $CASTLE_DATA_DIR/my-service/ # default /data/castle/my-service/
└── item-name.meta.json
```
The service receives this path via the generated `<PREFIX>_DATA_DIR` env var
it never hardcodes it. Use the `data_dir` setting from your config.
The service receives this path via its `<PREFIX>_DATA_DIR` env var, which
`defaults.env` maps from `${data_dir}` — it never hardcodes it. Use the
`data_dir` setting from your config.
```python
# storage.py