From fd562f74684836b66ff3caef3199661e10fe2f0d Mon Sep 17 00:00:00 2001 From: Paul Payne Date: Sun, 14 Jun 2026 16:32:13 -0700 Subject: [PATCH] docs: clean up pre-existing staleness across the guides MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sweep of accumulated drift unrelated to the CLI reorg: - Source paths: ~/.castle/code/ → /data/repos/ (REPOS_DIR), and source: code/X → absolute /data/repos/X. registry.md source-resolution table and 'how programs get in' section rewritten for the /data/repos layout. - Old field name component: → program: in all YAML examples. - Old term 'component' → 'program' throughout design.md and the stack guides. - Frontend serving: corrected the copy-to-artifacts/content model to serve-in- place from the repo (/), matching Phase 2b; dropped the dead _copy_app_static reference and the 'runtime never references the source tree' claim (frontends are the deliberate exception). - Stale endpoints/commands: /components → /deployments; castle create/add → castle program create/add in the create.py/add.py docstrings. CLAUDE.md was already current. cli 24 green; ruff clean. --- cli/src/castle_cli/commands/add.py | 4 +- cli/src/castle_cli/commands/create.py | 2 +- docs/design.md | 68 ++++++++++++------------- docs/registry.md | 73 +++++++++++++-------------- docs/stacks/python-cli.md | 12 ++--- docs/stacks/python-fastapi.md | 4 +- docs/stacks/react-vite.md | 36 ++++++------- 7 files changed, 98 insertions(+), 101 deletions(-) diff --git a/cli/src/castle_cli/commands/add.py b/cli/src/castle_cli/commands/add.py index 75cdda4..1c312f2 100644 --- a/cli/src/castle_cli/commands/add.py +++ b/cli/src/castle_cli/commands/add.py @@ -1,6 +1,6 @@ -"""castle add — adopt an existing repo as a program (no scaffolding). +"""castle program add — adopt an existing repo as a program (no scaffolding). -`castle create` makes new code from a stack. `castle add` adopts code that +`castle program create` makes new code from a stack. `castle program add` adopts code that already exists — a local path, or a git URL to clone. It detects sensible dev verb commands so a non-castle project becomes usable without writing them by hand. """ diff --git a/cli/src/castle_cli/commands/create.py b/cli/src/castle_cli/commands/create.py index 4fc0f68..6c52244 100644 --- a/cli/src/castle_cli/commands/create.py +++ b/cli/src/castle_cli/commands/create.py @@ -1,4 +1,4 @@ -"""castle create - scaffold a new project from templates.""" +"""castle program create — scaffold a new program from templates.""" from __future__ import annotations diff --git a/docs/design.md b/docs/design.md index 571a4f4..f46620e 100644 --- a/docs/design.md +++ b/docs/design.md @@ -20,7 +20,7 @@ is self-sufficient. The mesh is optional. Castle service is just a well-behaved Unix daemon that happens to be registered in a manifest. -3. **Stack and behavior.** Each component has a *stack* (development +3. **Stack and behavior.** Each program has a *stack* (development toolchain: python-fastapi, python-cli, react-vite) and a *behavior* (runtime role: daemon, tool, frontend). Scheduling, systemd management, and proxying are orthogonal operations — not behaviors. @@ -36,7 +36,7 @@ is self-sufficient. The mesh is optional. source tree. 6. **AI-manageable.** The CLI and API exist so that AI assistants can - discover, create, and manage components programmatically. Humans + discover, create, and manage programs programmatically. Humans use the dashboard. Agents use the CLI and API. 7. **Simple until proven otherwise.** Filesystem over databases. HTTP @@ -123,25 +123,25 @@ Castle generates systemd unit files and Caddyfile entries from the registry. It doesn't run a daemon itself — it configures OS-level infrastructure and gets out of the way. -Critically, the runtime layer references only standard paths — never -the source tree. Systemd units point to installed binaries (on PATH -or in `~/.local/bin/`), not to repo subdirectories. Caddy serves -from `$CASTLE_HOME/artifacts/content/`, not from build output directories in the repo. +Systemd units point to installed binaries (on PATH or in `~/.local/bin/`), +not to repo subdirectories. Frontends are the deliberate exception: rather +than stage a copy, Caddy serves their built assets **in place** from the repo +(`//`), with the serve prefix baked into the build via `VITE_BASE`. ### Registry Layer -The registry is the central concept in Castle. It tracks what components +The registry is the central concept in Castle. It tracks what programs exist, what they can do, and how they're configured. But it's not a single thing — it's three distinct concepts: -**1. Component spec** — what a component *is*. Description, capabilities, +**1. Component spec** — what a program *is*. Description, capabilities, build instructions, default configuration. This is source-level -information, version-controlled in the repo. It answers: "what components +information, version-controlled in the repo. It answers: "what programs could exist?" **2. Node config** — what's *deployed on this machine*, with what concrete ports, data paths, and env vars. This is per-machine. Two Castle nodes -might run different subsets of components with different parameters. It +might run different subsets of programs with different parameters. It answers: "what's running here, and how?" **3. Runtime state** — what's *actually happening*. PIDs, health, uptime, @@ -158,11 +158,11 @@ These map to two files: programs: central-context: description: Content storage API - source: code/central-context + source: /data/repos/central-context services: central-context: - component: central-context + program: central-context run: runner: python tool: central-context @@ -178,7 +178,7 @@ services: jobs: backup-collect: - component: backup-collect + program: backup-collect run: runner: command argv: [backup-collect] @@ -191,7 +191,7 @@ Programs define *what software exists* (identity, source, install, tools). Services define *how daemons run* (run config, expose, proxy, systemd). Jobs define *how scheduled tasks run* (run config, cron schedule, systemd). -Services and jobs can reference a component via `component:` for description +Services and jobs can reference a program via `program:` for description fallthrough and source code linking. They can also exist independently (e.g., `castle-gateway` runs Caddy — not our software). @@ -239,7 +239,7 @@ Three interfaces expose the registry: - **CLI** (`castle`) — For AI agents and terminal users. Structured output via `--json`. Commands for listing, inspecting, creating, - and managing components. + and managing programs. - **API** (`castle-api`) — For programmatic access over HTTP. Used by the dashboard, other nodes, and remote agents. - **Dashboard** (`castle-app`) — For human discoverability. Visual @@ -248,7 +248,7 @@ Three interfaces expose the registry: ### Coordination Layer Coordination handles discovery and communication — both between -components on a single node and across multiple Castle nodes. +programs on a single node and across multiple Castle nodes. **Intra-node coordination:** - Components find each other through the gateway (path-based routing) @@ -257,7 +257,7 @@ components on a single node and across multiple Castle nodes. - No service mesh or message broker required for basic operation. **Inter-node coordination:** -- Each Castle node runs the API, which exposes its component registry. +- Each Castle node runs the API, which exposes its program registry. - Nodes discover each other via MQTT retained messages and mDNS/DNS-SD (python-zeroconf) for LAN environments. - The gateway on each node can proxy to services on other nodes, @@ -290,9 +290,9 @@ Local paths always take precedence. **Why MQTT over custom gossip:** - Standard protocol, every language has a client library. - Retained messages give new nodes an immediate view of the network. -- Topic-based routing maps naturally to `castle/{node}/{component}`. +- Topic-based routing maps naturally to `castle/{node}/{program}`. - Works across networks (not just LAN like mDNS). -- Mosquitto is a single binary, simple to run as a Castle component. +- Mosquitto is a single binary, simple to run as a Castle program. **Why mDNS/DNS-SD as a complement:** - Zero-config LAN discovery via python-zeroconf. @@ -303,9 +303,9 @@ Local paths always take precedence. ### Dashboard -The web dashboard (`castle-app`) is a React SPA served by Caddy from -`$CASTLE_HOME/artifacts/content/castle-app/`. It talks to `castle-api` via the -gateway proxy at `/api`. +The web dashboard (`castle-app`) is a React SPA served by Caddy in place from +its repo build output (`/dist/`) at the root `/`. It talks to +`castle-api` via the gateway proxy at `/api`. **Layout:** @@ -344,7 +344,7 @@ Components · Software catalog backup-collect Python / CLI tool 0 2 * * * — ``` -**Key components:** +**Key programs:** - **GatewayPanel** — Route table with live health badges, reload button, collapsible Caddyfile viewer. Node column appears when multi-node. - **MeshPanel** — MQTT connection status (connected/disconnected badge), @@ -353,7 +353,7 @@ Components · Software catalog - **NodeBar** — Horizontal list of discovered nodes. Hidden in single-node mode. Each node links to `/node/{hostname}`. - **ServiceSection** — Daemon cards in a responsive grid. -- **ComponentTable** — Unified sortable table for all non-daemon components +- **ComponentTable** — Unified sortable table for all non-daemon programs (tools, frontends) with Stack, Behavior, Schedule, and Status columns. **Real-time updates:** @@ -365,11 +365,11 @@ Components · Software catalog **Multi-node behavior:** - NodeBar appears when `GET /nodes` returns >1 node. - GatewayPanel shows a "Node" column when routes span multiple nodes. -- `/node/{hostname}` page shows a specific node's deployed components. +- `/node/{hostname}` page shows a specific node's deployed programs. ## Component Contract -Every Castle component, regardless of language, must satisfy a minimal +Every Castle program, regardless of language, must satisfy a minimal contract. This is what makes the system uniform above the build line. ### Services (long-running daemons) @@ -421,11 +421,11 @@ Each step is distinct: 2. **Install** — Makes the artifact available on the system. For tools: `uv tool install` or compiled binary placed in `~/.local/bin/`. For services: same — the binary or entry point is on PATH or in a known - location. For frontends: built assets copied to - `$CASTLE_HOME/artifacts/content/`. + location. For frontends: assets built in place under the repo + (`//`), served from there — no copy. 3. **Deploy** — Materializes the runtime configuration. Reads the - component spec, merges with node config, generates systemd units + program spec, merges with node config, generates systemd units and Caddyfile entries that reference *installed* artifacts — never the source tree. Enables and starts services. @@ -507,7 +507,7 @@ Key OTP ideas that apply: can remap these across nodes. - **Spec vs. config.** In OTP, an application defines its structure (the `.app` file) and a release provides the deployment config - (`sys.config`). Castle mirrors this: the component spec defines + (`sys.config`). Castle mirrors this: the program spec defines structure, the node config provides deployment values. ## Current State @@ -526,9 +526,9 @@ What exists today: the manifest. Only non-convention values need `defaults.env`. - **Gateway** — Caddy on port 9000, Caddyfile generated from registry - **API** — `castle-api` on port 9020, reads from registry (optional - castle.yaml fallback for non-deployed components) + castle.yaml fallback for non-deployed programs) - **Dashboard** — `castle-app` React/Vite frontend, static assets - served from `$CASTLE_HOME/artifacts/content/castle-app/` + served in place from its repo build output (`/dist/`) - **Services** — central-context (content storage), notification-bridge (desktop notification forwarder) - **Jobs** — protonmail (email sync every 5 min), backup-collect (nightly), @@ -543,7 +543,7 @@ What exists today: on port 1883, managed by systemd. Config and data in `$CASTLE_DATA_DIR/castle-mqtt/`. - **Node API** — `GET /mesh/status`, `GET /nodes`, `GET /nodes/{hostname}`. - `GET /components?include_remote=true` for cross-node component listing. + `GET /deployments?include_remote=true` for cross-node program listing. - **Gateway panel** — Dedicated UI showing route table, health per route, reload button, Caddyfile viewer. Cross-node routes shown when multi-node. - **Mesh panel** — Dashboard UI showing MQTT connection status, broker @@ -555,7 +555,7 @@ What exists today: What doesn't exist yet: -- **Multi-language support** — Rust and Go components (the abstractions +- **Multi-language support** — Rust and Go programs (the abstractions support them via `command` runner, but no examples exist yet) - **Build automation** — Castle records build specs but doesn't orchestrate builds (each project builds independently) diff --git a/docs/registry.md b/docs/registry.md index 77e34e7..1efcb7f 100644 --- a/docs/registry.md +++ b/docs/registry.md @@ -43,14 +43,14 @@ gateway: programs: my-tool: description: Does something useful - source: code/my-tool + source: /data/repos/my-tool stack: python-cli behavior: tool system_dependencies: [pandoc] services: my-service: - component: my-service + program: my-service run: runner: python program: my-service @@ -65,7 +65,7 @@ services: jobs: my-job: - component: my-tool + program: my-tool run: runner: command argv: [my-tool, sync] @@ -82,7 +82,7 @@ jobs: | `services:` | Long-running daemons — how they run | service | | `jobs:` | Scheduled tasks — when they run | job | -Services and jobs can reference a program via `component:` for description +Services and jobs can reference a program via `program:` for description fallthrough and source code linking. They can also exist independently (e.g., `castle-gateway` runs Caddy — not our software). @@ -104,24 +104,23 @@ Explicit declaration of how the program is used: ### `source` — Where the source lives ```yaml -source: code/my-tool # your programs, under ~/.castle/code/ -source: repo:castle-api # castle's own programs, inside the git repo +source: /data/repos/my-tool # your programs, under $CASTLE_REPOS_DIR +source: repo:castle-api # castle's own programs, inside the git repo ``` The `source` path is resolved one of three ways (`core/src/castle_core/config.py`): | `source:` value | Resolves to | Used for | |-----------------|-------------|----------| -| `code/my-tool` *(relative)* | `$CASTLE_HOME/code/my-tool` | Your own programs | +| `/data/repos/my-tool` *(absolute)* | as-is | Your own programs (the default) | | `repo:castle-api` | `/castle-api` (via the top-level `repo:` field) | Castle's built-in programs | -| `/abs/path` | as-is | Anything explicitly absolute | +| `code/my-tool` *(relative)* | `$CASTLE_HOME/code/my-tool` | Legacy — pre-`/data/repos` layout | -Relative sources resolve against the castle home (`$CASTLE_HOME`, default -`~/.castle` — see [Infrastructure paths](#infrastructure-paths)). Most programs -you create live under **`$CASTLE_HOME/code/`** and are recorded as -`source: code/`. Castle's own programs (CLI, core, castle-api, app) live in -the git repo and use the `repo:` prefix. When `castle deploy` writes `castle.yaml` -back out, it rewrites absolute paths into these relative forms. +Programs you create or adopt live under **`$CASTLE_REPOS_DIR`** (default +`/data/repos`, override with `CASTLE_REPOS_DIR`) and are recorded with an +**absolute** `source:`. Castle's own programs (CLI, core, castle-api, app) live +in the git repo and use the `repo:` prefix. A relative `source:` still resolves +against `$CASTLE_HOME` for back-compat, but new programs no longer use it. ### `stack` — Development toolchain (optional) @@ -168,7 +167,7 @@ system_dependencies: [pandoc, poppler-utils] ``` System packages that must be installed for the program to work. Displayed -in `castle list --behavior tool` and the dashboard. +in `castle program list --behavior tool` and the dashboard. ### `version` — Program version @@ -287,23 +286,21 @@ Castle generates a systemd `.timer` file alongside the `.service` unit. Jobs also support `run` (required), `manage`, and `defaults` — same semantics as services. -## How programs get into `~/.castle/code/` +## How programs get into `/data/repos/` -Every program's source lives in one place — `~/.castle/code//`. It can -arrive there a few ways: +Every program's source lives under `$CASTLE_REPOS_DIR` (default `/data/repos//`). +It can arrive there a few ways: 1. **Scaffold a new one** with `castle program create` — writes the project into - `~/.castle/code//` and registers it in `castle.yaml` with - `source: code/`. -2. **Clone an existing project** — `git clone ~/.castle/code/`, - then add a `programs:` entry pointing at `source: code/`. -3. **Drop files in directly** — a `code//` directory is just a working - tree; it doesn't have to be under version control to be registered and run. + `/data/repos//` and registers it in `castle.yaml` with an absolute + `source: /data/repos/`. +2. **Adopt an existing repo** — `castle program add ` registers it + in place (or records its `repo:` URL for `castle program clone`). +3. **Drop files in directly** — a `/data/repos//` directory is just a + working tree; it doesn't have to be under version control to be run. -`~/.castle/` and `~/.castle/code/` are **not** themselves git repos, and there -are no submodules. Each program directory manages its own version control (or -none) independently — some are standalone git clones, others are just loose -files. +`/data/repos/` holds independent repos — each program directory manages its own +version control (or none); some are standalone git clones, others loose files. Castle's own programs (CLI, core, castle-api, app) are the exception: they live inside the castle git repo and are referenced with `source: repo:`. @@ -313,16 +310,16 @@ inside the castle git repo and are referenced with `source: repo:`. ### Via `castle program create` (recommended) ```bash -# Service — scaffolds into ~/.castle/code/, assigns port, registers in castle.yaml +# Service — scaffolds into /data/repos/, assigns port, registers in castle.yaml castle program create my-service --stack python-fastapi --description "Does something" -# Tool — scaffolds into ~/.castle/code/ +# Tool — scaffolds into /data/repos/ castle program create my-tool --stack python-cli --description "Does something" ``` ### Manually -Clone or create the project under `~/.castle/code/`, then add entries to the +Clone or create the project under `/data/repos/`, then add entries to the appropriate sections of `castle.yaml`: ```yaml @@ -330,7 +327,7 @@ appropriate sections of `castle.yaml`: programs: my-tool: description: Does something useful - source: code/my-tool + source: /data/repos/my-tool stack: python-cli behavior: tool @@ -338,13 +335,13 @@ programs: programs: my-service: description: Does something useful - source: code/my-service + source: /data/repos/my-service stack: python-fastapi behavior: daemon services: my-service: - component: my-service + program: my-service run: runner: python program: my-service @@ -364,7 +361,7 @@ services: ```bash castle program create my-service --stack python-fastapi # 1. Scaffold + register -cd ~/.castle/code/my-service && uv sync # 2. Install deps +cd /data/repos/my-service && uv sync # 2. Install deps # ... implement ... castle program test my-service # 3. Run tests castle service enable my-service # 4. Generate systemd unit, start @@ -384,10 +381,10 @@ castle service disable my-service # Stop and remove systemd unit ```bash castle program create my-tool --stack python-cli # 1. Scaffold + register -cd ~/.castle/code/my-tool && uv sync # 2. Install deps +cd /data/repos/my-tool && uv sync # 2. Install deps # ... implement ... castle program test my-tool # 3. Run tests -uv tool install --editable ~/.castle/code/my-tool/ # 4. Install to PATH +uv tool install --editable /data/repos/my-tool/ # 4. Install to PATH ``` ### Job lifecycle @@ -427,7 +424,7 @@ variable (both expand `~` and resolve relative paths): | Program source (castle's) | `/` (via `source: repo:`) | | Secrets | `$CASTLE_HOME/secrets/` | | Generated Caddyfile | `$CASTLE_HOME/artifacts/specs/Caddyfile` | -| Built frontends | `$CASTLE_HOME/artifacts/content//` | +| Built frontends | served in place from `//` (no copy) | | **Service data** | **`$CASTLE_DATA_DIR//` (default `/data/castle//`)** | | Systemd units | `~/.config/systemd/user/castle-*.service` | | Systemd timers | `~/.config/systemd/user/castle-*.timer` | diff --git a/docs/stacks/python-cli.md b/docs/stacks/python-cli.md index 868b8c1..e64e05c 100644 --- a/docs/stacks/python-cli.md +++ b/docs/stacks/python-cli.md @@ -31,10 +31,10 @@ How to build CLI tools following Unix philosophy. ## Project layout -Each tool is an independent project under `~/.castle/code/` with its own `pyproject.toml`: +Each tool is an independent project under `/data/repos/` with its own `pyproject.toml`: ``` -~/.castle/code/my-tool/ +/data/repos/my-tool/ ├── src/my_tool/ │ ├── __init__.py │ └── main.py # Entry point @@ -50,7 +50,7 @@ Examples: `code/pdf2md/`, `code/gpt/`, `code/protonmail/` ```bash castle program create my-tool --stack python-cli --description "Does something" -cd ~/.castle/code/my-tool && uv sync +cd /data/repos/my-tool && uv sync ``` This scaffolds the project and registers it in `castle.yaml`. @@ -324,7 +324,7 @@ uv run ruff format . # Format programs: my-tool: description: Does something useful - source: code/my-tool + source: /data/repos/my-tool stack: python-cli behavior: tool ``` @@ -335,13 +335,13 @@ Tools with system dependencies declare them directly on the program: programs: pdf2md: description: Convert PDF files to Markdown - source: code/pdf2md + source: /data/repos/pdf2md stack: python-cli behavior: tool system_dependencies: [pandoc, poppler-utils] ``` Tools live in the `programs:` section. If a tool also runs on a schedule, -add a separate entry in the `jobs:` section referencing the component. +add a separate entry in the `jobs:` section referencing the program. See @docs/registry.md for the full registry reference. diff --git a/docs/stacks/python-fastapi.md b/docs/stacks/python-fastapi.md index c93a71b..3651c8d 100644 --- a/docs/stacks/python-fastapi.md +++ b/docs/stacks/python-fastapi.md @@ -109,13 +109,13 @@ Castle passes config via env vars in castle.yaml: programs: my-service: description: Does something useful - source: code/my-service + source: /data/repos/my-service stack: python-fastapi behavior: daemon services: my-service: - component: my-service + program: my-service run: runner: python program: my-service diff --git a/docs/stacks/react-vite.md b/docs/stacks/react-vite.md index 04309c1..8c0b6cb 100644 --- a/docs/stacks/react-vite.md +++ b/docs/stacks/react-vite.md @@ -7,7 +7,7 @@ > stack — it declares its own `commands:`. See @docs/registry.md for > `commands:`, `stack:` (optional), and `repo:`. -How to build, serve, and manage web frontends as castle components. +How to build, serve, and manage web frontends as castle programs. ## Stack @@ -27,7 +27,7 @@ How to build, serve, and manage web frontends as castle components. ```bash # Create the project -mkdir ~/.castle/code/my-frontend && cd ~/.castle/code/my-frontend +mkdir /data/repos/my-frontend && cd /data/repos/my-frontend pnpm create vite . --template react-ts # Core dependencies @@ -122,9 +122,9 @@ pnpm run check # lint + type-check + test The `build` output is a static SPA in `dist/` — just HTML, JS, and CSS files. -## Registering as a castle component +## Registering as a program -A frontend component has a `build` spec (produces static output). Register it +A frontend program has a `build` spec (produces static output). Register it in the `programs:` section of `castle.yaml`. No `run` block needed if Caddy handles serving directly from the build output. @@ -133,7 +133,7 @@ handles serving directly from the build output. programs: my-frontend: description: Web dashboard - source: code/my-frontend + source: /data/repos/my-frontend build: commands: - ["pnpm", "build"] @@ -141,16 +141,16 @@ programs: - dist/ ``` -For production, `castle deploy` copies the build output to -`~/.castle/artifacts/content//` and Caddy serves it from there — no -Node process needed. See [Serving with Caddy](#serving-with-caddy) below. +For production, Caddy serves the build output **in place** from the program's +repo (`/`) — no Node process and no copy into a +central directory. See [Serving with Caddy](#serving-with-caddy) below. For development with Vite's dev server, add a service entry: ```yaml services: my-frontend: - component: my-frontend + program: my-frontend run: runner: node script: dev @@ -170,19 +170,19 @@ For production, the static build output is served by Caddy rather than a Node process. You do **not** write this block by hand — `castle deploy` generates it. The flow: -1. `castle deploy` runs the program's `build.commands`, then copies each - `build.outputs` directory from `~/.castle/code//` into - `~/.castle/artifacts/content//` (`core/src/castle_core/deploy.py`, - `_copy_app_static`). -2. The Caddyfile generator scans `~/.castle/artifacts/content/`; any directory - containing an `index.html` is served as a SPA at a path prefix matching its - name. `castle-app` is special-cased to serve at the root `/`. +1. `castle deploy` runs the program's `build.commands` (so `dist/` is current). +2. The Caddyfile generator emits a route per `behavior: frontend` program that + has `build.outputs`, rooted **in place** at `/` — + no copy. A static frontend mounts at `//`; `castle-app` is + special-cased to serve at the root `/`. -The generated block looks like this (written to `~/.castle/artifacts/specs/Caddyfile`): +The build is run with `VITE_BASE` set to that serve prefix, so a `vite.config` +that reads it (see [Vite config](#vite-config)) emits asset URLs that resolve at +the subpath. The generated block (in `~/.castle/artifacts/specs/Caddyfile`): ```caddyfile handle_path /my-frontend/* { - root * /home/payne/.castle/artifacts/content/my-frontend + root * /data/repos/my-frontend/dist try_files {path} /index.html file_server }