feat: Enhance tool management and documentation in Castle

- Introduced category tools support in the `castle create` command.
- Added detailed guides for creating components in CLAUDE.md.
- Implemented new API endpoints for listing and retrieving tool details.
- Updated component and tool models to include additional metadata.
- Improved error handling and response structures in service actions.
- Enhanced documentation for component registry and web APIs.
This commit is contained in:
2026-02-21 00:09:34 -08:00
parent f39a551aad
commit 08c6f3fa83
34 changed files with 1748 additions and 296 deletions

310
docs/component-registry.md Normal file
View File

@@ -0,0 +1,310 @@
# Component Registry
How castle tracks, configures, and manages components. This is the central
reference for `castle.yaml` structure and the manifest architecture.
## castle.yaml
The single source of truth for all components. Lives at the repo root.
```yaml
gateway:
port: 9000
components:
my-service:
description: Does something useful
run:
runner: python_uv_tool
tool: my-service
cwd: my-service
env:
MY_SERVICE_DATA_DIR: /data/castle/my-service
MY_SERVICE_PORT: "9001"
expose:
http:
internal: { port: 9001 }
health_path: /health
proxy:
caddy: { path_prefix: /my-service }
manage:
systemd: {}
```
## Manifest blocks
Each component declares **what it does** through these optional blocks:
### `run` — How to start it
Discriminated union on `runner`:
| Runner | Use case | Key fields |
|--------|----------|------------|
| `python_uv_tool` | Python service/tool via uv | `tool`, `cwd`, `env` |
| `command` | Shell command | `argv`, `cwd`, `env` |
| `python_module` | Python -m invocation | `module`, `args`, `python` |
| `container` | Docker/Podman | `image`, `command`, `ports`, `volumes` |
| `node` | Node.js script | `script`, `package_manager` (npm/pnpm/yarn) |
| `remote` | External service | `base_url`, `health_url` |
**Services** use `python_uv_tool`:
```yaml
run:
runner: python_uv_tool
tool: my-service # name in [project.scripts]
cwd: my-service # working directory relative to repo root
env:
MY_SERVICE_DATA_DIR: /data/castle/my-service
MY_SERVICE_PORT: "9001"
```
**Tools invoked by castle** (jobs, scheduled tasks) use `command`:
```yaml
run:
runner: command
argv: ["protonmail", "sync"]
cwd: protonmail
env:
PROTONMAIL_USERNAME: user@example.com
```
**Standalone tools** that users invoke directly often have no `run` block at
all — castle just installs them to PATH.
### `expose` — What it exposes
```yaml
expose:
http:
internal:
port: 9001 # Required for services
health_path: /health # Used by health polling
```
Having `expose.http` gives the component the **service** role.
### `proxy` — How to proxy it
```yaml
proxy:
caddy:
path_prefix: /my-service # Proxied at gateway:9000/my-service/
```
Castle generates a Caddyfile from these entries. Only needed for services
accessible through the gateway.
### `manage` — How to manage it
```yaml
manage:
systemd: {}
```
Enables `castle service enable/disable` and `castle logs`. An empty `{}`
uses defaults (enable=true, restart=on-failure, restart_sec=5).
Full options:
```yaml
manage:
systemd:
description: Custom unit description
restart: always # on-failure | always | no
restart_sec: 2
no_new_privileges: true
after: [network.target, castle-other.service]
wanted_by: [default.target]
```
### `install` — How to install it
```yaml
install:
path:
alias: my-tool # Command name in PATH
```
Creates a shim so the tool is available system-wide after
`uv tool install --editable .`.
### `tool` — Tool metadata
```yaml
tool:
tool_type: python_standalone # or "script"
category: document # Grouping for display
source: tools/document/ # Source directory
version: "1.0.0"
system_dependencies: [pandoc, poppler-utils]
```
This block provides metadata for `castle tool list` and the dashboard.
It's separate from `install` (which handles PATH registration) and `run`
(which handles execution).
### `build` — How to build it
```yaml
build:
commands:
- ["pnpm", "build"]
outputs:
- dist/
```
Having build outputs gives the component the **frontend** role.
### `triggers` — What triggers it
```yaml
triggers:
- type: schedule
cron: "*/5 * * * *"
timezone: America/Los_Angeles # default
```
Having a schedule trigger gives the component the **job** role.
Castle generates a systemd .timer file alongside the .service unit.
Other trigger types: `manual`, `event` (source + topic), `request` (protocol).
### `env` with secrets
Environment variables can reference secrets stored in `~/.castle/secrets/`:
```yaml
run:
env:
API_KEY: ${secret:MY_API_KEY}
```
Castle resolves `${secret:NAME}` by reading `~/.castle/secrets/NAME`.
Never store secrets in castle.yaml or project directories.
## Role derivation
Roles are **computed** from manifest declarations, never set manually:
| Role | Derived when |
|------|-------------|
| **service** | Has `expose.http` |
| **tool** | Has `install.path` or has `tool` spec (fallback) |
| **worker** | Has `manage.systemd` but no `expose.http` |
| **job** | Has trigger with `type: schedule` |
| **frontend** | Has `build` with outputs or commands |
| **containerized** | Runner is `container` |
| **remote** | Runner is `remote` |
A component can have multiple roles. For example, `protonmail` is both a
**tool** (installed to PATH) and a **job** (runs on a cron schedule).
## Registering a new component
### Via `castle create` (recommended)
```bash
# Service — scaffolds project, assigns port, registers in castle.yaml
castle create my-service --type service --description "Does something"
# Standalone tool — scaffolds at repo root
castle create my-tool --type tool --description "Does something"
# Category tool — adds to existing tools/<category>/ package
castle create my-tool --type tool --category document --description "Does something"
```
### Manually
Add an entry to the `components:` section of `castle.yaml`:
```yaml
components:
my-tool:
description: Does something useful
tool:
tool_type: python_standalone
category: utilities
source: my-tool/
install:
path:
alias: my-tool
```
## Lifecycle
### Service lifecycle
```bash
castle create my-service --type service # 1. Scaffold + register
cd my-service && uv sync # 2. Install deps
# ... implement ...
castle test my-service # 3. Run tests
castle service enable my-service # 4. Generate systemd unit, start
castle gateway reload # 5. Update Caddy routes
```
After `service enable`, the service starts automatically on boot and restarts
on failure. Manage with:
```bash
castle logs my-service -f # Tail logs
castle run my-service # Run in foreground (for debugging)
castle service disable my-service # Stop and remove systemd unit
```
### Tool lifecycle
```bash
castle create my-tool --type tool # 1. Scaffold + register
cd my-tool && uv sync # 2. Install deps
# ... implement ...
castle test my-tool # 3. Run tests
uv tool install --editable my-tool/ # 4. Install to PATH
```
### Job lifecycle
Jobs are tools or services with a schedule trigger. They need both `run`
(so castle knows how to execute them) and `manage.systemd` (so systemd
handles the timer):
```yaml
my-job:
description: Runs nightly
run:
runner: command
argv: ["my-job"]
triggers:
- type: schedule
cron: "0 2 * * *"
manage:
systemd: {}
```
`castle service enable my-job` generates both a `.service` (Type=oneshot)
and a `.timer` file.
## Infrastructure paths
| What | Where |
|------|-------|
| Component registry | `castle.yaml` (repo root) |
| Service data | `/data/castle/<name>/` |
| Secrets | `~/.castle/secrets/<NAME>` |
| Generated Caddyfile | `~/.castle/generated/Caddyfile` |
| Systemd units | `~/.config/systemd/user/castle-*.service` |
| Systemd timers | `~/.config/systemd/user/castle-*.timer` |
## Manifest models
The Pydantic models live in `cli/src/castle_cli/manifest.py`. Key classes:
- `ComponentManifest` — top-level model, has `roles` computed property
- `RunSpec` — discriminated union (RunPythonUvTool, RunCommand, etc.)
- `TriggerSpec` — union (TriggerSchedule, TriggerManual, TriggerEvent, TriggerRequest)
- `ExposeSpec`, `ProxySpec`, `ManageSpec`, `InstallSpec`, `ToolSpec`, `BuildSpec`
- `CaddySpec`, `SystemdSpec`, `HttpExposeSpec`, `HttpInternal`
Config loading: `cli/src/castle_cli/config.py``load_config()` parses
castle.yaml into `CastleConfig` with typed `components` dict.

View File

@@ -1,7 +1,6 @@
# Python Tools in Castle
How to build CLI tools following Unix philosophy. Based on the patterns in the
[toolkit](https://github.com/payneio/toolkit) project.
How to build CLI tools following Unix philosophy.
## Principles
@@ -18,88 +17,120 @@ How to build CLI tools following Unix philosophy. Based on the patterns in the
| **CLI** | argparse |
| **Package manager** | uv (never pip) |
| **Build** | hatchling |
| **Testing** | unittest + mocking |
| **Linting** | ruff |
| **Type checking** | pyright |
| **Testing** | pytest |
| **Linting** | ruff (shared `ruff.toml` at repo root) |
| **Type checking** | pyright (shared `pyrightconfig.json` at repo root) |
| **Python** | 3.11+ minimum |
## Project layout
## Two kinds of tools
Tools live inside a single package with categories:
### Standalone tools
Independent projects at the repo root with their own `pyproject.toml`:
```
toolkit/
├── tools/
│ ├── document/
│ ├── pdf2md.py # Implementation
│ │ ├── pdf2md.md # Docs + YAML frontmatter (single source of truth)
│ └── test_pdf2md.py # Tests alongside tool
│ ├── search/
│ │ ├── search.py
│ │ ├── search.md
│ │ └── test_search.py
│ ├── system/
│ │ ├── schedule.py
│ │ └── schedule.md
│ └── toolkit/
│ ├── toolkit.py # Meta-tool for discovery/scaffolding
│ └── toolkit.md
my-tool/
├── src/my_tool/
│ ├── __init__.py
└── main.py # Entry point
├── tests/
│ └── test_main.py
├── pyproject.toml
── Makefile
└── README.md
── CLAUDE.md
```
Each tool is a `.py` + `.md` pair. The `.md` file has YAML frontmatter for
metadata — no separate config files needed.
Examples: `devbox-connect/`, `mboxer/`, `protonmail/`
## YAML frontmatter (.md file)
### Category tools
```yaml
---
command: pdf2md
script: document/pdf2md.py
description: Convert PDF files to Markdown
version: 1.0.0
category: document
system_dependencies:
- pandoc
- poppler-utils
---
Multiple tools sharing a single package under `tools/<category>/`:
# pdf2md
Converts PDF files to Markdown format...
```
tools/document/
├── src/document/
│ ├── __init__.py
│ ├── pdf2md.py # Each tool is a module
│ ├── docx2md.py
│ ├── html2text.py
│ └── md2pdf.py
├── tests/
├── pyproject.toml # One pyproject with multiple [project.scripts]
└── CLAUDE.md
```
The toolkit management command discovers tools by scanning for these `.md` files.
Examples: `tools/document/`, `tools/search/`, `tools/system/`
## Creating a new tool
### Standalone
```bash
castle create my-tool --type tool --description "Does something"
cd my-tool && uv sync
```
This scaffolds the project and registers it in `castle.yaml`.
### Category tool
```bash
castle create my-tool --type tool --category document --description "Does something"
cd tools/document && uv sync
```
This adds a `.py` file to the existing category package and updates its
`pyproject.toml` entry points.
## pyproject.toml
### Standalone tool
```toml
[project]
name = "toolkit"
name = "my-tool"
version = "0.1.0"
description = "Does something useful"
requires-python = ">=3.11"
dependencies = [
"requests>=2.28.0",
"pyyaml>=6.0.0",
]
dependencies = []
[project.scripts]
pdf2md = "tools.document.pdf2md:main"
docx2md = "tools.document.docx2md:main"
search = "tools.search.search:main"
toolkit = "tools.toolkit.toolkit:main"
my-tool = "my_tool.main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["tools"]
packages = ["src/my_tool"]
[dependency-groups]
dev = ["pytest>=7.0.0"]
[tool.ruff.lint.isort]
known-first-party = ["my_tool"]
```
Entry points follow `command = "tools.<category>.<tool>:main"`. After
`uv tool install --editable .`, all commands are in PATH.
### Category package
```toml
[project]
name = "castle-document"
version = "0.1.0"
description = "Castle document conversion tools"
requires-python = ">=3.11"
dependencies = []
[project.scripts]
docx2md = "document.docx2md:main"
pdf2md = "document.pdf2md:main"
html2text = "document.html2text:main"
md2pdf = "document.md2pdf:main"
[tool.hatch.build.targets.wheel]
packages = ["src/document"]
```
After `uv tool install --editable .`, all commands are in PATH.
## Tool implementation patterns
@@ -113,8 +144,6 @@ to stdout.
"""
my-tool: Brief description
Detailed usage docs here.
Usage:
my-tool [options] [FILE]
cat input.txt | my-tool
@@ -224,7 +253,7 @@ import sys
def convert(input_file: str, output_file: str) -> int:
try:
result = subprocess.run(
subprocess.run(
["pandoc", input_file, "-o", output_file],
check=True,
capture_output=True,
@@ -245,24 +274,6 @@ def convert(input_file: str, output_file: str) -> int:
Always use `check=True` and `capture_output=True` with subprocess. Handle
`FileNotFoundError` for missing system dependencies.
### Tool with optional dependencies
```python
tantivy_available = False
try:
import tantivy
tantivy_available = True
except ImportError:
tantivy = None
def check_tantivy() -> None:
if not tantivy_available:
print("Error: tantivy not found. Install with: uv add tantivy",
file=sys.stderr)
sys.exit(1)
```
## Error handling
```python
@@ -301,119 +312,94 @@ for f in *.pdf; do pdf2md "$f" > "${f%.pdf}.md"; done
cat doc.txt | text-extractor | jq .content
```
When a tool writes to both a file and stdout, status messages must go to
stderr so they don't contaminate the pipe:
```python
with open(output_file, "w") as f:
f.write(result)
print(result) # stdout (for piping)
print(f"Wrote to {output_file}", file=sys.stderr) # stderr (status)
```
## Testing
Tests live alongside the tool implementation, using unittest with mocking:
Tests use pytest. For standalone tools, test via subprocess to exercise the
real CLI interface:
```python
# tools/gpt/test_gpt.py
import unittest
from unittest.mock import patch, MagicMock
from io import StringIO
from tools.gpt import gpt
import subprocess
import sys
class TestGPT(unittest.TestCase):
@patch("tools.gpt.gpt.get_api_key")
@patch("openai.OpenAI")
def test_generate(self, mock_openai, mock_key):
mock_key.return_value = "fake-key"
mock_client = MagicMock()
mock_openai.return_value = mock_client
mock_client.chat.completions.create.return_value = MagicMock(
choices=[MagicMock(message=MagicMock(content="response"))]
class TestCLI:
def test_version(self) -> None:
result = subprocess.run(
[sys.executable, "-m", "my_tool.main", "--version"],
capture_output=True,
text=True,
)
assert "my-tool" in result.stdout
result = gpt.generate_text("prompt", "gpt-4", 0.7, 500)
self.assertEqual(result, "response")
def test_stdin(self) -> None:
result = subprocess.run(
[sys.executable, "-m", "my_tool.main"],
input="hello\n",
capture_output=True,
text=True,
)
assert result.returncode == 0
assert "hello" in result.stdout
@patch("tools.gpt.gpt.generate_text")
def test_cli(self, mock_gen):
mock_gen.return_value = "output"
with patch("sys.argv", ["gpt", "prompt"]):
with patch("sys.stdout", new=StringIO()) as out:
gpt.main()
self.assertEqual(out.getvalue().strip(), "output")
def test_file_input(self, tmp_path) -> None:
input_file = tmp_path / "input.txt"
input_file.write_text("test data")
result = subprocess.run(
[sys.executable, "-m", "my_tool.main", str(input_file)],
capture_output=True,
text=True,
)
assert result.returncode == 0
assert "test data" in result.stdout
```
Pattern: mock external dependencies (APIs, file I/O, subprocesses), test the
CLI by patching `sys.argv`.
For unit testing core logic, import the function directly and test it as a
pure function.
## Build and install
```makefile
# Makefile
all: build install
build:
uv sync
install:
uv tool install --editable .
check:
uv run ruff format .
uv run ruff check . --fix
uv run pyright
test:
uv run pytest -v
clean:
find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
```
## Commands
```bash
make all # Sync deps + install to PATH
make check # Format, lint, type-check
make test # Run tests
uv sync # Install deps
uv run my-tool --help # Run the tool
uv run pytest tests/ -v # Run tests
uv run ruff check . # Lint
uv run ruff format . # Format
```
## Creating a new tool
## Registering in castle.yaml
Use the toolkit management command:
```bash
toolkit create my-tool --description "Does something" --category document
```
This creates the `.py` template, `.md` with frontmatter, and updates
`pyproject.toml` with the entry point.
Or manually:
1. Create `tools/<category>/my_tool.py` with the argparse pattern above
2. Create `tools/<category>/my_tool.md` with YAML frontmatter
3. Add entry to `pyproject.toml` under `[project.scripts]`:
```toml
my-tool = "tools.category.my_tool:main"
```
4. Run `make install` to register in PATH
## Registering in castle
Standalone tools:
```yaml
# castle.yaml
components:
toolkit:
description: Personal utility scripts
run:
runner: command
argv: ["toolkit"]
cwd: toolkit
my-tool:
description: Does something useful
tool:
tool_type: python_standalone
category: utilities
source: my-tool/
install:
path: { alias: toolkit }
path:
alias: my-tool
```
Tools with `install.path` get the `tool` role. They don't need `expose`,
`proxy`, or `manage` blocks.
Category tools get one entry per tool, all pointing to the same source:
```yaml
components:
pdf2md:
description: Convert PDF files to Markdown
tool:
tool_type: python_standalone
category: document
source: tools/document/
system_dependencies: [pandoc, poppler-utils]
install:
path:
alias: pdf2md
```
Tools with `install.path` get the **tool** role. They don't need `expose`,
`proxy`, or `manage` blocks unless castle also runs them (e.g., scheduled jobs).
See @docs/component-registry.md for the full manifest reference.

View File

@@ -422,3 +422,6 @@ uv run ruff format . # Format
```bash
castle create my-service --type service --description "Does something useful"
```
See @docs/component-registry.md for manifest fields, role derivation, and
the full service lifecycle (enable, logs, gateway reload).

View File

@@ -156,6 +156,9 @@ components:
This gives the component both the `frontend` role (from `build`) and the
`service` role (from `expose.http`) during development.
See @docs/component-registry.md for the full manifest reference and role
derivation rules.
## Serving with Caddy
For production, serve the static `dist/` output directly from Caddy rather than