refactor: Restructure ~/.castle/ as the instance directory

Move all instance state into ~/.castle/ with clear separation: - code/ — user project source (was components/) - artifacts/ — generated specs, built content (was generated/, static/) - data/ — service runtime data (was /data/castle/) - secrets/ — credentials castle.yaml moves to ~/.castle/castle.yaml as the canonical location. Source paths use repo: prefix for git repo programs (castle-api, app) and relative paths for user projects (code/central-context). Add idempotent install.sh for bootstrapping infrastructure (Docker, Caddy, MQTT, Postgres, Neo4j) with interactive service detection. Remove components/ from repo (now in ~/.castle/code/), deinit submodules, remove .gitmodules.
2026-03-09 22:03:37 -07:00
parent b7628c590b
commit 8be129259c
86 changed files with 1222 additions and 13989 deletions
--- a/docs/working/discussion.md
+++ b/docs/working/discussion.md
@@ -0,0 +1,664 @@
+Yes — that changes the design a lot, in a good way.
+
+If **castle is not a general platform**, but a **curated machine for you + an AI assistant**, then the ontology should be optimized for:
+
+* few concepts
+* strong defaults
+* predictable placement
+* limited extension paths
+* easy generation from natural language
+
+That means castle should **not** try to model all possible machine semantics. Ubuntu already does that. Castle should define a **small opinionated overlay** on top of Ubuntu.
+
+So I’d stop trying to find the perfect universal ontology and instead design a **usefully incomplete one**.
+
+## The design principle
+
+Castle should answer:
+
+> What are the few things I commonly want the machine to have or do?
+
+From your examples, those are roughly:
+
+* a **tool**
+* a **site**
+* a **service**
+* a **worker/daemon**
+* a **job**
+* a **store/broker**
+* maybe a **library/project**
+
+That suggests the core user-facing abstraction should not be “artifact” or “namespace” or even “runtime.”
+
+It should be something more like:
+
+## Castle’s main abstraction: `unit`
+
+A **unit** is a thing the castle machine knows how to host.
+
+That thing may be:
+
+* a Python tool
+* a static site
+* a web service
+* a daemon
+* a scheduled job
+* a database
+* a broker
+
+Then beneath that, castle can have a smaller set of internal dimensions.
+
+---
+
+# A much simpler castle model
+
+I think the practical model is:
+
+```text
+unit
+kind
+stack
+uses
+```
+
+Where:
+
+## `unit`
+
+The named thing on the machine.
+
+Examples:
+
+* `reddit-tool`
+* `docs-site`
+* `api`
+* `transform-worker`
+* `morning-search`
+* `postgres`
+* `mqtt`
+
+## `kind`
+
+The machine role.
+
+Examples:
+
+* `tool`
+* `site`
+* `service`
+* `worker`
+* `job`
+* `store`
+* `broker`
+
+## `stack`
+
+How it is built/run.
+
+Examples:
+
+* `python`
+* `rust`
+* `react-static`
+* `fastapi`
+* `shell`
+
+This is where language/build/runtime conventions collapse into one curated choice.
+
+## `uses`
+
+What castle subsystems or other units it depends on.
+
+Examples:
+
+* `postgres`
+* `neo4j`
+* `mqtt`
+* `caddy`
+* `reddit`
+* `web`
+
+This is much closer to how you actually want to talk to the AI.
+
+---
+
+# Why this is better
+
+You don’t want to say:
+
+> create a deployment of kind service using runtime python with artifact class program and lifecycle systemd
+
+You want to say:
+
+> add a Python service that does X and uses Postgres
+
+So castle should be shaped around **intentional unit kinds**, not abstract machine theory.
+
+The machine theory should stay underneath as implementation.
+
+---
+
+# What castle can standardize
+
+Because castle is curated, you can hardcode a lot:
+
+## Always true on castle
+
+* Ubuntu
+* systemd
+* Caddy
+* standard directory layout
+* standard logging
+* standard service supervision
+* standard way to expose commands on PATH
+* standard way to run Python via `uv`
+* standard way to build JS via `pnpm`
+* standard way to define dependencies between units
+
+That means users and AI do **not** need to specify those every time.
+
+This is the key simplification move.
+
+---
+
+# Recommended fixed subsystems
+
+I’d make these built-in castle subsystems:
+
+* **python** via `uv`
+* **rust** via `cargo`
+* **node/react** via `pnpm`
+* **java** via one standard choice
+* **caddy**
+* **systemd**
+* **postgres**
+* **mqtt**
+* maybe **neo4j**
+* maybe **web-search ingestion** as a castle-native capability
+
+Then the schema does not need to model arbitrary runtimes. It only needs to reference these known capabilities.
+
+---
+
+# Strong defaults for placement
+
+You said castle can always use well-known spots. Good. Do that aggressively.
+
+For example:
+
+```text
+/castle/units/<name>/src
+/castle/units/<name>/build
+/castle/units/<name>/env
+/castle/units/<name>/state
+/castle/units/<name>/config
+/castle/units/<name>/logs
+```
+
+And for exposure:
+
+```text
+/castle/bin/<tool>
+/castle/sites/<site>
+/castle/data/<store>
+```
+
+Then castle itself can generate:
+
+* symlinks into `/usr/local/bin`
+* systemd units
+* Caddy config
+* env files
+* state dirs
+
+So the schema does not need to talk much about placement, because placement is mostly implied.
+
+---
+
+# The actual user-facing kinds
+
+I’d recommend just these:
+
+## 1. `tool`
+
+A command you can run from anywhere.
+
+Examples:
+
+* Python CLI
+* Rust CLI
+* shell helper
+
+Defaults:
+
+* exposed on PATH
+* no long-running process
+* build/install conventions by stack
+
+Example:
+
+```yaml
+units:
+  summarize-reddit:
+    kind: tool
+    stack: python
+```
+
+---
+
+## 2. `site`
+
+A human-facing HTTP site served by Caddy.
+
+Examples:
+
+* React static site
+* docs site
+* maybe proxied app frontend later
+
+Defaults:
+
+* served by Caddy
+* standard root dir
+* standard local hostname or port
+
+Example:
+
+```yaml
+units:
+  castle-web:
+    kind: site
+    stack: react-static
+    port: 8080
+```
+
+---
+
+## 3. `service`
+
+A long-running HTTP or TCP process supervised by systemd.
+
+Examples:
+
+* FastAPI app
+* Rust API
+* Java backend
+
+Defaults:
+
+* systemd-managed
+* logs to journal
+* optional Caddy exposure if HTTP
+* standard environment/config paths
+
+Example:
+
+```yaml
+units:
+  reddit-api:
+    kind: service
+    stack: fastapi
+    port: 9090
+    uses: [postgres, neo4j, reddit]
+```
+
+---
+
+## 4. `worker`
+
+A long-running non-HTTP daemon, often connected to broker/store inputs.
+
+Examples:
+
+* MQTT consumer
+* pubsub transformer
+* sync loop
+* queue processor
+
+Defaults:
+
+* systemd-managed
+* usually not publicly exposed
+* can subscribe to broker/topic or poll/store
+
+Example:
+
+```yaml
+units:
+  topic-transformer:
+    kind: worker
+    stack: python
+    uses: [mqtt]
+    subscribe: channel-b
+    publish: channel-y
+```
+
+This kind seems especially important for your goals.
+
+---
+
+## 5. `job`
+
+A scheduled or one-shot task.
+
+Examples:
+
+* morning search
+* report generation
+* backup
+* fetch and write JSON
+
+Defaults:
+
+* systemd timer
+* writes to known output locations
+* can use web-search capability
+
+Example:
+
+```yaml
+units:
+  morning-z-search:
+    kind: job
+    stack: python
+    schedule: daily@08:00
+    uses: [web]
+    outputs:
+      - /castle/shared/search/z/
+```
+
+---
+
+## 6. `store`
+
+A built-in stateful service.
+
+Examples:
+
+* Postgres
+* Neo4j
+
+Defaults:
+
+* castle manages lifecycle and placement
+* other units reference by name
+
+Example:
+
+```yaml
+units:
+  postgres:
+    kind: store
+    engine: postgres
+
+  neo4j:
+    kind: store
+    engine: neo4j
+```
+
+You may even decide these are so built-in they don’t need to be declared unless customized.
+
+---
+
+## 7. `broker`
+
+A built-in messaging service.
+
+Examples:
+
+* MQTT
+
+Example:
+
+```yaml
+units:
+  mqtt:
+    kind: broker
+    engine: mqtt
+```
+
+Again, maybe built-in by default.
+
+---
+
+# The schema can now be tiny
+
+Instead of a universal schema, use a compact one with per-kind defaults.
+
+Example:
+
+```yaml
+units:
+  reddit-tool:
+    kind: tool
+    stack: python
+    description: CLI for querying reddit summaries
+
+  castle-web:
+    kind: site
+    stack: react-static
+    port: 8080
+
+  reddit-api:
+    kind: service
+    stack: fastapi
+    port: 9090
+    uses: [postgres, neo4j, reddit]
+
+  topic-transformer:
+    kind: worker
+    stack: python
+    uses: [mqtt]
+    subscribe: channel-b
+    publish: channel-y
+
+  morning-z-search:
+    kind: job
+    stack: python
+    schedule: "daily@08:00"
+    uses: [web]
+    output_dir: /castle/shared/search/z
+```
+
+That is probably closer to the right level.
+
+---
+
+# What `stack` should mean now
+
+Since castle is curated, `stack` can absorb a lot:
+
+Instead of:
+
+* build system
+* runtime
+* conventions
+* entrypoint rules
+
+just use a known stack name.
+
+Examples:
+
+* `python`
+* `fastapi`
+* `rust`
+* `react-static`
+* `java-service`
+* `shell`
+
+Each stack implies:
+
+* where source goes
+* how to build
+* how to run
+* what files are expected
+* how dependencies are installed
+
+This is much simpler than separately modeling build/runtime/deployment for every case.
+
+So for castle:
+
+> **stack = curated implementation pattern**
+
+That’s probably the right definition.
+
+---
+
+# What `uses` should mean
+
+This is another very valuable simplification.
+
+Instead of asking the user to define bindings and interfaces, let them say what the unit uses.
+
+Examples:
+
+* `postgres`
+* `neo4j`
+* `mqtt`
+* `caddy`
+* `web`
+* `reddit`
+* another unit name
+
+Then castle knows how to wire it.
+
+Example:
+
+```yaml
+uses: [postgres, neo4j, reddit]
+```
+
+could imply:
+
+* env vars for local connection strings
+* service ordering in systemd
+* generated client config
+* AI awareness that those capabilities are available
+
+This is much more useful than formal dependency graphs.
+
+---
+
+# Under the hood
+
+Internally, castle can still map to:
+
+* artifacts
+* runtimes
+* systemd
+* Caddy
+* mount namespaces
+* env files
+* directories
+
+But that should be hidden.
+
+The AI assistant should primarily think in terms of:
+
+* **What kind of unit is this?**
+* **What stack should it use?**
+* **What does it use?**
+* **What should it be named?**
+
+That’s enough to generate the machine config.
+
+---
+
+# So the real castle ontology may be
+
+## User-facing
+
+```text
+unit
+kind
+stack
+uses
+```
+
+## Machine-facing
+
+```text
+source
+artifact
+runtime
+deployment
+placement
+```
+
+This split is important.
+
+You were trying to find one ontology for both humans and machines. You probably want two:
+
+* a **simple command ontology** for you + AI
+* a **lower-level implementation ontology** for castle internals
+
+That resolves a lot of the tension.
+
+---
+
+# Example natural language mapping
+
+“Add a python tool that does X”
+
+becomes:
+
+```yaml
+name: x-tool
+kind: tool
+stack: python
+```
+
+“Make a web service that does Y”
+
+becomes:
+
+```yaml
+name: y-service
+kind: service
+stack: fastapi
+```
+
+“Make a daemon that watches pubsub channel B and transforms the data and puts it on Y”
+
+becomes:
+
+```yaml
+name: b-to-y
+kind: worker
+stack: python
+uses: [mqtt]
+subscribe: B
+publish: Y
+```
+
+“Every morning, do a web search for Z and put the results in PATH as json files”
+
+becomes:
+
+```yaml
+name: z-search
+kind: job
+stack: python
+uses: [web]
+schedule: daily@08:00
+output_dir: /castle/shared/search/z
+format: json
+```
+
+This is very manageable.
+
+---
+
+# Recommendation
+
+I would make castle intentionally opinionated around:
+
+* **7 unit kinds max**
+* **5–8 stacks max**
+* **a handful of built-in capabilities in `uses`**
+* **fixed placement conventions**
+* **automatic generation of systemd/Caddy/config**
+
+That seems like the right balance of simplicity and extensibility for what you want.
+
+The next useful step is to draft the **minimal v1 castle schema** with only those user-facing fields and strong defaults.