Clean up.

2026-06-13 12:24:41 -07:00
parent 9fe95f6d1e
commit 74a902ee62
11 changed files with 217 additions and 91 deletions
--- a/docs/component-registry.md
+++ b/docs/component-registry.md
@@ -6,7 +6,8 @@ architecture.

 ## castle.yaml

-The single source of truth. Lives at the repo root. Three top-level sections:
+The single source of truth. Lives at `~/.castle/castle.yaml`. Three top-level
+sections:

 ```yaml
 gateway:
@@ -15,7 +16,7 @@ gateway:
 programs:
  my-tool:
    description: Does something useful
-    source: programs/my-tool
+    source: code/my-tool
    stack: python-cli
    behavior: tool
    system_dependencies: [pandoc]
@@ -76,10 +77,24 @@ Explicit declaration of how the program is used:
 ### `source` — Where the source lives

 ```yaml
-source: programs/my-tool
+source: code/my-tool        # your programs, under ~/.castle/code/
+source: repo:castle-api      # castle's own programs, inside the git repo
 ```

-Relative path from repo root to the project directory.
+The `source` path is resolved one of three ways (`core/src/castle_core/config.py`):
+
+| `source:` value | Resolves to | Used for |
+|-----------------|-------------|----------|
+| `code/my-tool` *(relative)* | `$CASTLE_HOME/code/my-tool` | Your own programs |
+| `repo:castle-api` | `<repo>/castle-api` (via the top-level `repo:` field) | Castle's built-in programs |
+| `/abs/path` | as-is | Anything explicitly absolute |
+
+Relative sources resolve against the castle home (`$CASTLE_HOME`, default
+`~/.castle` — see [Infrastructure paths](#infrastructure-paths)). Most programs
+you create live under **`$CASTLE_HOME/code/`** and are recorded as
+`source: code/<name>`. Castle's own programs (CLI, core, castle-api, app) live in
+the git repo and use the `repo:` prefix. When `castle deploy` writes `castle.yaml`
+back out, it rewrites absolute paths into these relative forms.

 ### `stack` — Development toolchain

@@ -215,28 +230,50 @@ Castle generates a systemd `.timer` file alongside the `.service` unit.
 Jobs also support `run` (required), `manage`, and `defaults` — same
 semantics as services.

+## How programs get into `~/.castle/code/`
+
+Every program's source lives in one place — `~/.castle/code/<name>/`. It can
+arrive there a few ways:
+
+1. **Scaffold a new one** with `castle create` — writes the project into
+   `~/.castle/code/<name>/` and registers it in `castle.yaml` with
+   `source: code/<name>`.
+2. **Clone an existing project** — `git clone <url> ~/.castle/code/<name>`,
+   then add a `programs:` entry pointing at `source: code/<name>`.
+3. **Drop files in directly** — a `code/<name>/` directory is just a working
+   tree; it doesn't have to be under version control to be registered and run.
+
+`~/.castle/` and `~/.castle/code/` are **not** themselves git repos, and there
+are no submodules. Each program directory manages its own version control (or
+none) independently — some are standalone git clones, others are just loose
+files.
+
+Castle's own programs (CLI, core, castle-api, app) are the exception: they live
+inside the castle git repo and are referenced with `source: repo:<name>`.
+
 ## Registering a new program

 ### Via `castle create` (recommended)

 ```bash
-# Service — scaffolds project, assigns port, registers in castle.yaml
+# Service — scaffolds into ~/.castle/code/, assigns port, registers in castle.yaml
 castle create my-service --stack python-fastapi --description "Does something"

-# Tool — scaffolds under programs/
+# Tool — scaffolds into ~/.castle/code/
 castle create my-tool --stack python-cli --description "Does something"
 ```

 ### Manually

-Add entries to the appropriate sections of `castle.yaml`:
+Clone or create the project under `~/.castle/code/`, then add entries to the
+appropriate sections of `castle.yaml`:

 ```yaml
 # Tool — only needs a program entry
 programs:
  my-tool:
    description: Does something useful
-    source: programs/my-tool
+    source: code/my-tool
    stack: python-cli
    behavior: tool

@@ -244,7 +281,7 @@ programs:
 programs:
  my-service:
    description: Does something useful
-    source: programs/my-service
+    source: code/my-service
    stack: python-fastapi
    behavior: daemon

@@ -270,7 +307,7 @@ services:

 ```bash
 castle create my-service --stack python-fastapi   # 1. Scaffold + register
-cd programs/my-service && uv sync         # 2. Install deps
+cd ~/.castle/code/my-service && uv sync   # 2. Install deps
 # ... implement ...
 castle test my-service                    # 3. Run tests
 castle service enable my-service          # 4. Generate systemd unit, start
@@ -290,10 +327,10 @@ castle service disable my-service # Stop and remove systemd unit

 ```bash
 castle create my-tool --stack python-cli        # 1. Scaffold + register
-cd programs/my-tool && uv sync           # 2. Install deps
+cd ~/.castle/code/my-tool && uv sync     # 2. Install deps
 # ... implement ...
 castle test my-tool                      # 3. Run tests
-uv tool install --editable programs/my-tool/    # 4. Install to PATH
+uv tool install --editable ~/.castle/code/my-tool/   # 4. Install to PATH
 ```

 ### Job lifecycle
@@ -317,15 +354,33 @@ and a `.timer` file.

 ## Infrastructure paths

+Castle uses **two** independent roots, each overridable by an environment
+variable (both expand `~` and resolve relative paths):
+
+- **`CASTLE_HOME`** — config, code, artifacts, and secrets. Default `~/.castle`.
+- **`CASTLE_DATA_DIR`** — program/service data I/O (potentially large; lives on a
+  dedicated volume). Default `/data/castle`. Decoupled from `CASTLE_HOME` on
+  purpose so bulk data doesn't sit in the home directory.
+
 | What | Where |
 |------|-------|
-| Registry | `castle.yaml` (repo root) |
-| Service data | `/data/castle/<name>/` |
-| Secrets | `~/.castle/secrets/<NAME>` |
-| Generated Caddyfile | `~/.castle/generated/Caddyfile` |
+| Castle home | `$CASTLE_HOME` (default `~/.castle`) |
+| Registry | `$CASTLE_HOME/castle.yaml` |
+| Program source (yours) | `$CASTLE_HOME/code/<name>/` |
+| Program source (castle's) | `<repo>/<name>` (via `source: repo:<name>`) |
+| Secrets | `$CASTLE_HOME/secrets/<NAME>` |
+| Generated Caddyfile | `$CASTLE_HOME/artifacts/specs/Caddyfile` |
+| Built frontends | `$CASTLE_HOME/artifacts/content/<name>/` |
+| **Service data** | **`$CASTLE_DATA_DIR/<name>/` (default `/data/castle/<name>/`)** |
 | Systemd units | `~/.config/systemd/user/castle-*.service` |
 | Systemd timers | `~/.config/systemd/user/castle-*.timer` |

+Defined in `core/src/castle_core/config.py`: `CASTLE_HOME` (with derived
+`CODE_DIR`, `SECRETS_DIR`, `SPECS_DIR`, `CONTENT_DIR`) and the independent
+`DATA_DIR` (`CASTLE_DATA_DIR`). `castle deploy` passes each service its data
+path via the generated `<PREFIX>_DATA_DIR` env var. Systemd unit/timer paths are
+fixed by systemd's user-unit convention.
+
 ## Manifest models

 The Pydantic models live in `core/src/castle_core/manifest.py`. Key classes:
--- a/docs/design.md
+++ b/docs/design.md
@@ -31,8 +31,9 @@ is self-sufficient. The mesh is optional.
   operates above the line.

 5. **Separate source from runtime.** The repo is for development. The
-   runtime lives in standard Unix locations (`~/.castle/`, `/data/castle/`,
-   systemd units). Nothing running should point into the source tree.
+   runtime lives in standard Unix locations (`$CASTLE_HOME`, default
+   `~/.castle/`, plus systemd units). Nothing running should point into the
+   source tree.

 6. **AI-manageable.** The CLI and API exist so that AI assistants can
   discover, create, and manage components programmatically. Humans
@@ -113,9 +114,10 @@ Manages running processes using standard Linux infrastructure.
 - TLS termination

 **Filesystem** handles storage:
- Service data: `/data/castle/<name>/`
- Secrets: `~/.castle/secrets/`
- Generated config: `~/.castle/generated/`
+- Service data: `$CASTLE_DATA_DIR/<name>/` (default `/data/castle/`, on a
+  dedicated volume)
+- Secrets: `$CASTLE_HOME/secrets/` (default `~/.castle/secrets/`)
+- Generated config: `$CASTLE_HOME/artifacts/specs/` (Caddyfile, registry.yaml)

 Castle generates systemd unit files and Caddyfile entries from the
 registry. It doesn't run a daemon itself — it configures OS-level
@@ -123,8 +125,8 @@ infrastructure and gets out of the way.

 Critically, the runtime layer references only standard paths — never
 the source tree. Systemd units point to installed binaries (on PATH
-or in `~/.castle/bin/`), not to repo subdirectories. Caddy serves
-from `~/.castle/static/`, not from build output directories in the repo.
+or in `~/.local/bin/`), not to repo subdirectories. Caddy serves
+from `$CASTLE_HOME/artifacts/content/`, not from build output directories in the repo.

 ### Registry Layer

@@ -156,7 +158,7 @@ These map to two files:
 programs:
  central-context:
    description: Content storage API
-    source: components/central-context
+    source: code/central-context

 services:
  central-context:
@@ -197,7 +199,7 @@ Convention-based env vars (`<PREFIX>_PORT`, `<PREFIX>_DATA_DIR`) are
 generated automatically during deploy. Only non-convention values need
 `defaults.env`.

-**`~/.castle/registry.yaml`** (per-node, not in the repo) — Node config:
+**`$CASTLE_HOME/artifacts/specs/registry.yaml`** (per-node, not in the repo, generated by `castle deploy`) — Node config:

 ```yaml
 node:
@@ -209,7 +211,7 @@ deployed:
    runner: python
    run_cmd: [/home/user/.local/bin/central-context]
    env:
-      CENTRAL_CONTEXT_DATA_DIR: /data/castle/central-context
+      CENTRAL_CONTEXT_DATA_DIR: /home/user/.castle/data/central-context
      CENTRAL_CONTEXT_PORT: "9001"
    behavior: daemon
    stack: python-fastapi
@@ -302,7 +304,7 @@ Local paths always take precedence.
 ### Dashboard

 The web dashboard (`castle-app`) is a React SPA served by Caddy from
-`~/.castle/static/castle-app/`. It talks to `castle-api` via the
+`$CASTLE_HOME/artifacts/content/castle-app/`. It talks to `castle-api` via the
 gateway proxy at `/api`.

 **Layout:**
@@ -417,9 +419,10 @@ Each step is distinct:
   execute them implicitly.

 2. **Install** — Makes the artifact available on the system. For tools:
-   `uv tool install` or compiled binary placed in `~/.castle/bin/`. For
+   `uv tool install` or compiled binary placed in `~/.local/bin/`. For
   services: same — the binary or entry point is on PATH or in a known
-   location. For frontends: built assets copied to `~/.castle/static/`.
+   location. For frontends: built assets copied to
+   `$CASTLE_HOME/artifacts/content/`.

 3. **Deploy** — Materializes the runtime configuration. Reads the
   component spec, merges with node config, generates systemd units
@@ -427,28 +430,32 @@ Each step is distinct:
   the source tree. Enables and starts services.

 For compiled languages (Rust, Go), build produces a standalone binary
-and install is just placing it in `~/.castle/bin/`. For interpreted
+and install is just placing it in `~/.local/bin/`. For interpreted
 languages (Python, Node), the runtime wrapper (uv, node) handles
 finding the installed artifact.

 ## Runtime Filesystem Layout

-What already exists and what the target looks like:
+Two roots, each overridable by an env var: `$CASTLE_HOME` (config, code,
+artifacts, secrets; default `~/.castle`) and `$CASTLE_DATA_DIR` (bulk program
+data; default `/data/castle`, kept on a dedicated volume):

 ```
-~/.castle/                      ← Castle runtime home
-├── registry.yaml               ← Node config (what's deployed here)
-├── generated/                  ← Generated Caddyfile
-│   └── Caddyfile
-├── secrets/                    ← Secret files (NAME → value)
-│   └── PROTONMAIL_API_KEY
-├── bin/                        ← Compiled binaries, shims
-│   └── my-go-tool
-└── static/                     ← Built frontend assets
-    └── castle-app/
-        └── dist/
+$CASTLE_HOME/                   ← Config & artifacts (default ~/.castle)
+├── castle.yaml                 ← Registry spec (programs, services, jobs)
+├── infra.conf                  ← Infrastructure install choices
+├── code/                       ← Program source (your programs)
+│   └── <name>/
+├── artifacts/
+│   ├── specs/                  ← Generated by `castle deploy`
+│   │   ├── Caddyfile
+│   │   └── registry.yaml       ← Node config (what's deployed here)
+│   └── content/                ← Built frontend assets
+│       └── castle-app/         ← (index.html + assets, served at root)
+└── secrets/                    ← Secret files (NAME → value)
+    └── PROTONMAIL_API_KEY

-/data/castle/                   ← Persistent service data
+$CASTLE_DATA_DIR/               ← Persistent service data (default /data/castle)
 └── <name>/

 ~/.config/systemd/user/         ← Systemd units (standard location)
@@ -458,8 +465,11 @@ What already exists and what the target looks like:
 └── ...
 ```

+Compiled-language tools (Rust, Go — planned) install their binaries to the
+standard `~/.local/bin/`, not under `$CASTLE_HOME`.
+
 Source (the repo) is referenced only during build and install. Everything
-the runtime touches lives in `~/.castle/`, `/data/castle/`, or standard
+the runtime touches lives under `$CASTLE_HOME`, `$CASTLE_DATA_DIR`, or standard
 systemd paths.

 ## OTP as Design Guide
@@ -470,7 +480,7 @@ Castle's architecture parallels Erlang/OTP, mapped onto Unix:
 |-------------|------------------|
 | Application | Component (independent, self-contained) |
 | Application resource file | Component spec in `castle.yaml` |
-| Release config (sys.config) | Node config in `~/.castle/registry.yaml` |
+| Release config (sys.config) | Node config in `$CASTLE_HOME/artifacts/specs/registry.yaml` |
 | Release assembly | `castle deploy` (spec + node config → runtime) |
 | Supervisor | systemd (restart policies, ordering) |
 | Process | Running service/worker/job |
@@ -508,17 +518,17 @@ What exists today:
 - **Three packages** — `castle-core` (models, config, generators),
  `castle-cli` (commands), `castle-api` (HTTP API)
 - **Source/runtime split** — `castle.yaml` (spec) → `castle deploy` →
-  `~/.castle/registry.yaml` (node config). Systemd units and Caddyfile
-  generated from registry with fully resolved paths. No repo references
+  `$CASTLE_HOME/artifacts/specs/registry.yaml` (node config). Systemd units and
+  Caddyfile generated from registry with fully resolved paths. No repo references
  in runtime artifacts.
 - **Convention-based env generation** — `castle deploy` auto-generates
-  `<PREFIX>_DATA_DIR=/data/castle/<name>` and `<PREFIX>_PORT` from
+  `<PREFIX>_DATA_DIR=$CASTLE_DATA_DIR/<name>` and `<PREFIX>_PORT` from
  the manifest. Only non-convention values need `defaults.env`.
 - **Gateway** — Caddy on port 9000, Caddyfile generated from registry
 - **API** — `castle-api` on port 9020, reads from registry (optional
  castle.yaml fallback for non-deployed components)
 - **Dashboard** — `castle-app` React/Vite frontend, static assets
-  served from `~/.castle/static/castle-app/`
+  served from `$CASTLE_HOME/artifacts/content/castle-app/`
 - **Services** — central-context (content storage), notification-bridge
  (desktop notification forwarder)
 - **Jobs** — protonmail (email sync every 5 min), backup-collect (nightly),
@@ -531,7 +541,7 @@ What exists today:
  Opt-in via `CASTLE_API_MQTT_ENABLED` / `CASTLE_API_MDNS_ENABLED`.
 - **MQTT broker** — Mosquitto running as `castle-mqtt` Docker container
  on port 1883, managed by systemd. Config and data in
-  `/data/castle/castle-mqtt/`.
+  `$CASTLE_DATA_DIR/castle-mqtt/`.
 - **Node API** — `GET /mesh/status`, `GET /nodes`, `GET /nodes/{hostname}`.
  `GET /components?include_remote=true` for cross-node component listing.
 - **Gateway panel** — Dedicated UI showing route table, health per route,
@@ -559,7 +569,7 @@ What doesn't exist yet:
 | Process supervision | systemd (user units) | Active |
 | HTTP routing | Caddy (port 9000) | Active |
 | Component specs | castle.yaml + Pydantic models | Active |
-| Node config | `~/.castle/registry.yaml` | Active |
+| Node config | `$CASTLE_HOME/artifacts/specs/registry.yaml` | Active |
 | CLI | castle (Python, uv) | Active |
 | API | castle-api (FastAPI) | Active |
 | Dashboard | castle-app (React, Vite, shadcn/ui) | Active |
@@ -569,7 +579,7 @@ What doesn't exist yet:
 | Type checking | pyright (Python), tsc (TS) | Active |
 | Testing | pytest (Python), Vitest (TS) | Active |
 | Secrets | `~/.castle/secrets/` file-based | Active |
-| Data storage | Filesystem (`/data/castle/`) | Active |
+| Data storage | Filesystem (`$CASTLE_DATA_DIR/`, default `/data/castle/`) | Active |
 | Messaging | MQTT (paho-mqtt client, Mosquitto broker) | Active (opt-in) |
 | Node discovery | mDNS (python-zeroconf) + MQTT | Active (opt-in) |
 | Rust packaging | cargo | Planned |
--- a/docs/stacks/python-cli.md
+++ b/docs/stacks/python-cli.md
@@ -24,10 +24,10 @@ How to build CLI tools following Unix philosophy.

 ## Project layout

-Each tool is an independent project under `components/` with its own `pyproject.toml`:
+Each tool is an independent project under `~/.castle/code/` with its own `pyproject.toml`:

 ```
-components/my-tool/
+~/.castle/code/my-tool/
 ├── src/my_tool/
 │   ├── __init__.py
 │   └── main.py       # Entry point
@@ -37,13 +37,13 @@ components/my-tool/
 └── CLAUDE.md
 ```

-Examples: `components/pdf2md/`, `components/gpt/`, `components/protonmail/`
+Examples: `code/pdf2md/`, `code/gpt/`, `code/protonmail/`

 ## Creating a new tool

 ```bash
 castle create my-tool --stack python-cli --description "Does something"
-cd components/my-tool && uv sync
+cd ~/.castle/code/my-tool && uv sync
 ```

 This scaffolds the project and registers it in `castle.yaml`.
@@ -317,7 +317,7 @@ uv run ruff format .        # Format
 programs:
  my-tool:
    description: Does something useful
-    source: components/my-tool
+    source: code/my-tool
    stack: python-cli
    behavior: tool
 ```
@@ -328,7 +328,7 @@ Tools with system dependencies declare them directly on the program:
 programs:
  pdf2md:
    description: Convert PDF files to Markdown
-    source: components/pdf2md
+    source: code/pdf2md
    stack: python-cli
    behavior: tool
    system_dependencies: [pandoc, poppler-utils]
--- a/docs/stacks/python-fastapi.md
+++ b/docs/stacks/python-fastapi.md
@@ -102,7 +102,7 @@ Castle passes config via env vars in castle.yaml:
 programs:
  my-service:
    description: Does something useful
-    source: components/my-service
+    source: code/my-service
    stack: python-fastapi
    behavior: daemon

@@ -292,12 +292,15 @@ Mapping convention:
 Castle services use filesystem storage with JSON metadata sidecars:

 ```
-/data/castle/my-service/
+$CASTLE_DATA_DIR/my-service/     # default /data/castle/my-service/
 └── bucket/
    ├── item-name
    └── item-name.meta.json
 ```

+The service receives this path via the generated `<PREFIX>_DATA_DIR` env var —
+it never hardcodes it. Use the `data_dir` setting from your config.
+
 ```python
 # storage.py
 import hashlib
--- a/docs/stacks/react-vite.md
+++ b/docs/stacks/react-vite.md
@@ -20,7 +20,7 @@ How to build, serve, and manage web frontends as castle components.

 ```bash
 # Create the project
-mkdir components/my-frontend && cd components/my-frontend
+mkdir ~/.castle/code/my-frontend && cd ~/.castle/code/my-frontend
 pnpm create vite . --template react-ts

 # Core dependencies
@@ -116,7 +116,7 @@ handles serving directly from the build output.
 programs:
  my-frontend:
    description: Web dashboard
-    source: components/my-frontend
+    source: code/my-frontend
    build:
      commands:
        - ["pnpm", "build"]
@@ -124,7 +124,8 @@ programs:
        - dist/
 ```

-For production, Caddy serves the static `dist/` output directly — no
+For production, `castle deploy` copies the build output to
+`~/.castle/artifacts/content/<name>/` and Caddy serves it from there — no
 Node process needed. See [Serving with Caddy](#serving-with-caddy) below.

 For development with Vite's dev server, add a service entry:
@@ -148,12 +149,23 @@ See @docs/component-registry.md for the full registry reference.

 ## Serving with Caddy

-For production, serve the static `dist/` output directly from Caddy rather than
-running a Node process. The gateway Caddyfile can serve the files:
+For production, the static build output is served by Caddy rather than a Node
+process. You do **not** write this block by hand — `castle deploy` generates it.
+The flow:
+
+1. `castle deploy` runs the program's `build.commands`, then copies each
+   `build.outputs` directory from `~/.castle/code/<name>/` into
+   `~/.castle/artifacts/content/<name>/` (`core/src/castle_core/deploy.py`,
+   `_copy_app_static`).
+2. The Caddyfile generator scans `~/.castle/artifacts/content/`; any directory
+   containing an `index.html` is served as a SPA at a path prefix matching its
+   name. `castle-app` is special-cased to serve at the root `/`.
+
+The generated block looks like this (written to `~/.castle/artifacts/specs/Caddyfile`):

 ```caddyfile
-handle_path /app/* {
-    root * /data/repos/castle/my-frontend/dist
+handle_path /my-frontend/* {
+    root * /home/payne/.castle/artifacts/content/my-frontend
    try_files {path} /index.html
    file_server
 }
@@ -161,7 +173,8 @@ handle_path /app/* {

 The `try_files {path} /index.html` directive is essential for SPA routing —
 it falls back to `index.html` for any path that doesn't match a static file,
-letting React Router handle client-side routes.
+letting React Router handle client-side routes. The serving prefix is derived
+from the program name, not hand-configured.

 ## API integration