wild-cloud/wild-pc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Cleanup

Browse Source
This commit is contained in:
Paul Payne
2026-02-21 02:01:45 -08:00
parent 12b54dca4f
commit d43c49d219
6 changed files with 168 additions and 524 deletions
Download Patch File Download Diff File Expand all files Collapse all files

139
README.md
View File

@@ -1,6 +1,6 @@
# Castle
A declarative local control plane. Castle manages a collection of independent services, tools, and libraries from a single CLI, with a unified gateway, systemd integration, and standardized project scaffolding.
A personal software platform. Castle manages independent services, tools, and frontends from a single CLI, with a unified gateway, systemd integration, and a web dashboard.
## Quick Start
@@ -21,87 +21,136 @@ castle services start
open http://localhost:9000
```
## Creating Projects
## Creating Components
```bash
# Service — FastAPI app with health endpoint, systemd unit, gateway route
castle create my-api --type service --description "Does something useful"
cd my-api && uv sync
castle test my-api
castle service enable my-api
castle gateway reload
# Standalone tool — CLI tool with argparse, stdin/stdout, Unix pipes
castle create my-tool --type tool --description "Does something"
# Category tool — adds to existing tools/<category>/ package
castle create my-tool --type tool --category document --description "Converts something"
```
Three project types are supported:
- **service** — FastAPI app with health endpoint, pydantic-settings, systemd unit, gateway route
- **tool** — CLI tool with argparse, stdin/stdout, Unix pipe conventions
- **library** — Python package with src/ layout, no entry point
## CLI Reference
```
castle list [--type TYPE] [--json] List all projects
castle create NAME --type TYPE Scaffold a new project
castle test [PROJECT] Run tests (one or all)
castle lint [PROJECT] Run linter (one or all)
castle list [--role ROLE] [--json] List all components
castle info NAME [--json] Show component details
castle create NAME --type TYPE Scaffold a new component
castle run NAME Run component in foreground
castle test [NAME] Run tests (one or all)
castle lint [NAME] Run linter (one or all)
castle sync Update submodules + install deps
castle logs NAME [-f] [-n 50] View component logs
castle gateway start|stop|reload Manage Caddy reverse proxy
castle service enable|disable NAME Manage a systemd service
castle service status Show all service statuses
castle services start|stop Start/stop everything
castle tool list List tools grouped by category
castle tool info NAME Show tool details + docs
```
## Registry
`castle.yaml` at the repo root is the single source of truth. It defines every project's type, port, gateway path, data directory, command, and environment variables. The CLI reads this for all operations — generating Caddyfiles, systemd units, and dashboard HTML.
`castle.yaml` is the single source of truth. Components declare **what they do** (run, expose, manage, install, build, triggers) and roles are **derived** from those declarations.
```yaml
gateway:
port: 9000
projects:
components:
central-context:
type: service
port: 9001
path: /central-context
command: uv run central-context
working_dir: central-context
data_dir: /data/castle/central-context
description: Content storage API
health: /health
env:
CENTRAL_CONTEXT_DATA_DIR: ${data_dir}
run:
runner: python_uv_tool
tool: central-context
working_dir: central-context
env:
CENTRAL_CONTEXT_DATA_DIR: /data/castle/central-context
CENTRAL_CONTEXT_PORT: "9001"
expose:
http:
internal: { port: 9001 }
health_path: /health
proxy:
caddy: { path_prefix: /central-context }
manage:
systemd: {}
```
## Architecture
```
castle.yaml ← project registry
cli/ ← castle CLI (castle-component)
central-context/ ← content storage API (git submodule)
notification-bridge/ ← notification forwarder (git submodule)
devbox-connect/ ← SSH tunnel manager
mboxer/ ← MBOX converter (git submodule)
toolkit/ ← personal utility scripts (git submodule)
event-bus/ ← inter-service event bus (castle-component)
ruff.toml ← shared lint config
pyrightconfig.json ← shared type checking config
castle.yaml <- component registry (single source of truth)
cli/ <- castle CLI
castle-api/ <- Castle API (dashboard backend)
app/ <- Castle web app (React/Vite frontend)
central-context/ <- content storage API (git submodule)
notification-bridge/ <- desktop notification forwarder (git submodule)
protonmail/ <- email sync tool/job
devbox-connect/ <- SSH tunnel manager
tools/ <- category tool packages (document/, search/, system/, etc.)
ruff.toml <- shared lint config
pyrightconfig.json <- shared type checking config
```
**Independence principle:** Regular projects never depend on castle. They accept configuration (data dir, port, URLs) via environment variables. Only castle-components (CLI, gateway, event bus) know about castle internals like `castle.yaml`. This keeps projects portable and independently publishable.
**Independence principle:** Services never depend on castle. They accept configuration (data dir, port, URLs) via environment variables. Only castle components (CLI, API, gateway) know about castle internals.
**Gateway:** Caddy reverse proxy at port 9000. All services are accessible under one address (`localhost:9000/central-context/*` → `localhost:9001/*`). A dashboard with live health checks is served at the root.
**Gateway:** Caddy reverse proxy at port 9000. Services are proxied under one address (`localhost:9000/central-context/*` -> `localhost:9001/*`). The web app is served at the root.
**Systemd:** The CLI generates user units under `~/.config/systemd/user/castle-*.service`. `castle services start` brings up everything in one command.
**Systemd:** The CLI generates user units under `~/.config/systemd/user/castle-*.service`. Scheduled jobs get `.timer` files alongside.
**Data:** Service data lives in `/data/castle/<service-name>/`, outside the repo. Secrets live in `~/.castle/secrets/`.
## Current Projects
## Components
| Project | Type | Port | Description |
|---------|------|------|-------------|
| central-context | service | 9001 | Content storage API |
| notification-bridge | service | 9002 | Desktop notification forwarder |
| devbox-connect | tool | — | SSH tunnel manager |
| mboxer | tool | — | MBOX to EML converter |
| toolkit | tool | — | Personal utility scripts |
| event-bus | castle-component | 9010 | Inter-service event bus |
### Services
| Component | Port | Description |
|-----------|------|-------------|
| castle-gateway | 9000 | Caddy reverse proxy gateway |
| central-context | 9001 | Content storage API |
| notification-bridge | 9002 | Desktop notification forwarder |
| castle-api | 9020 | Castle API (dashboard backend) |
### Jobs
| Component | Schedule | Description |
|-----------|----------|-------------|
| protonmail | Every 5 min | ProtonMail email sync |
| backup-collect | 2:00 AM | Collect files into backup directory |
| backup-data | 3:30 AM | Restic backup of /data to /storage |
### Tools
| Tool | Category | Description |
|------|----------|-------------|
| docx2md | document | Convert Word .docx to Markdown |
| pdf2md | document | Convert PDF to Markdown |
| html2text | document | Convert HTML to plain text |
| md2pdf | document | Convert Markdown to PDF |
| mbox2eml | document | Convert MBOX mailboxes to .eml files |
| search | search | Manage searchable file collections |
| docx-extractor | search | Extract content from Word files |
| pdf-extractor | search | Extract content from PDF files |
| text-extractor | search | Extract content from text files |
| schedule | system | Manage systemd user timers |
| backup-collect | system | Collect files for backup |
| android-backup | android | Backup Android devices via ADB |
| browser | browser | Browse the web via browser-use |
| gpt | gpt | OpenAI text generation |
| mdscraper | mdscraper | Combine text files into markdown |
| devbox-connect | standalone | SSH tunnel manager |
### Frontends
| Component | Description |
|-----------|-------------|
| castle-app | Castle web dashboard (React/Vite/TypeScript) |

3
castle.yaml
View File

@@ -23,7 +23,8 @@ components:
port: 9000
health_path: /
central-context:
description: Content storage API
description: Content storage API useful to get context into my data dir from anywhere
on the LAN
run:
runner: python_uv_tool
working_dir: central-context

72
protonmail/README.md Normal file
View File

@@ -0,0 +1,72 @@
# protonmail
CLI tool for syncing and managing ProtonMail emails via ProtonMail Bridge. Downloads emails as `.eml` files for fast, offline access. All read/list/search commands work against the local cache.
## Prerequisites
[ProtonMail Bridge](https://proton.me/mail/bridge) must be running locally with IMAP enabled (default port 1143).
## Usage
```bash
# Sync all emails from Bridge to local .eml files
protonmail sync
# Sync to a custom directory
protonmail sync -o ~/emails
# List recent emails (from local cache)
protonmail list
protonmail list Sent
protonmail list -n 50
# Read a specific email
protonmail read "2024-07-17_01-14-40_from_alice_example_com_to_bob_example_com.eml"
# Search by subject or sender
protonmail search "invoice"
protonmail search "alice" -f Sent
# Send an email (interactive, requires Bridge)
protonmail send
```
## Configuration
Set via environment variables (castle manages these automatically):
| Variable | Description | Default |
|----------|-------------|---------|
| `PROTONMAIL_USERNAME` | Bridge username (required) | - |
| `PROTONMAIL_API_KEY` | Bridge API key (required) | - |
| `PROTONMAIL_DATA_DIR` | Where to store synced .eml files | `/data/messages/email/protonmail` |
In castle, the API key is stored as a secret:
```yaml
env:
PROTONMAIL_API_KEY: ${secret:PROTONMAIL_API_KEY}
```
## How sync works
1. Connects to Bridge via IMAP on `127.0.0.1:1143`
2. Lists all folders (INBOX, Sent, Drafts, etc.)
3. For each folder, downloads new messages as `.eml` files
4. Skips emails already in the local cache (matched by Message-ID)
5. Filenames follow the pattern: `YYYY-MM-DD_HH-MM-SS_from_SENDER_to_RECIPIENT_SUBJECT.eml`
## Castle integration
Registered as both a **tool** (installed to PATH) and a **job** (syncs every 5 minutes via systemd timer).
## Development
```bash
uv sync # Install dependencies
uv run protonmail --version # Verify
uv run pytest tests/ -v # Run tests
uv run ruff check . # Lint
```
No third-party dependencies -- stdlib only.

62
working/decisions.md
View File

@@ -1,62 +0,0 @@
# Implementation Decisions
Decisions made during implementation that weren't pre-decided. Paul will review these.
## 1. CLI entry point name
Used `castle` as the command name (via `[project.scripts] castle = "castle_cli.main:main"`).
Package name is `castle-cli` to avoid conflicts, but the command is just `castle`.
## 2. Caddy `handle_path` instead of `handle`
Used `handle_path` in the Caddyfile instead of `handle`. `handle_path` automatically strips
the path prefix before proxying, so `/central-context/health` proxies to `localhost:9000/health`.
Without this, the upstream service would receive the full `/central-context/health` path and
return 404.
## 3. Service auto-port assignment
`castle create --type service` auto-assigns the next available port starting from 9000,
skipping any ports already used by other services or the gateway. This avoids port collisions
without requiring the user to track assignments manually.
## 4. Systemd unit naming convention
All castle systemd units use the prefix `castle-` (e.g., `castle-central-context.service`,
`castle-gateway.service`). This makes them easy to identify and manage as a group.
## 5. `uv` path resolution in systemd units
Systemd user units don't inherit the user's PATH. The CLI resolves `uv` to its absolute
path (via `shutil.which`) when generating unit files so they work regardless of PATH.
## 6. Dashboard health checks use direct ports
The dashboard HTML checks health by fetching directly from each service's port
(e.g., `localhost:9000/health`) rather than going through the gateway. This avoids
circular dependency if the gateway itself is having issues, and gives accurate per-service
health status.
## 7. `castle sync` runs `uv sync` before `git submodule update`
Actually runs submodule update first, then `uv sync` in each project. This way submodules
are at the right commit before dependencies are installed.
## 8. Template test structure
Scaffolded services include a health endpoint test by default. Tools include a placeholder
test. Libraries include an import test. This ensures `castle test` works immediately after
`castle create` without requiring the developer to write tests first.
## 9. `save_config` YAML formatting
When the CLI saves back to `castle.yaml` (e.g., after `castle create`), the YAML output
uses `yaml.dump` with `default_flow_style=False` and `sort_keys=False` to keep the
format readable and maintain insertion order. The format won't be identical to the original
hand-written YAML (e.g., blank lines are lost) but is functionally equivalent.
## 10. Jinja2 dependency
Added jinja2 as a dependency in the CLI's pyproject.toml for potential future template
expansion, but the current scaffold implementation uses plain f-strings. Could be removed
if it's not used soon.

342
working/schema.py
View File

@@ -1,342 +0,0 @@
from __future__ import annotations
from enum import Enum
from typing import Annotated, Dict, List, Literal, Optional, Set, Union
from pydantic import (
BaseModel,
Field,
HttpUrl,
computed_field,
model_validator,
)
# -------------------------
# Shared primitives
# -------------------------
EnvMap = Dict[str, str]
class RestartPolicy(str, Enum):
NO = "no"
ON_FAILURE = "on-failure"
ALWAYS = "always"
class TLSMode(str, Enum):
OFF = "off"
INTERNAL = "internal"
LETSENCRYPT = "letsencrypt"
class Role(str, Enum):
TOOL = "tool"
SERVICE = "service"
WORKER = "worker"
FRONTEND = "frontend"
JOB = "job"
REMOTE = "remote"
CONTAINERIZED = "containerized"
# -------------------------
# Run specs (discriminated union)
# -------------------------
class RunBase(BaseModel):
runner: str
cwd: Optional[str] = None
env: EnvMap = Field(default_factory=dict)
class RunCommand(RunBase):
runner: Literal["command"]
argv: List[str] = Field(min_length=1)
class RunPythonModule(RunBase):
runner: Literal["python_module"]
module: str
args: List[str] = Field(default_factory=list)
python: Optional[str] = None # e.g. "/path/to/python"
class RunPythonUvTool(RunBase):
runner: Literal["python_uv_tool"]
tool: str # the installed tool name
args: List[str] = Field(default_factory=list)
class RunContainer(RunBase):
runner: Literal["container"]
image: str
command: Optional[List[str]] = None # overrides image CMD if provided
args: List[str] = Field(default_factory=list)
ports: Dict[int, int] = Field(default_factory=dict) # container_port -> host_port
volumes: List[str] = Field(default_factory=list) # "host:container[:ro]"
workdir: Optional[str] = None
class RunNode(RunBase):
runner: Literal["node"]
script: str # e.g. "dev", "start", or a file path
package_manager: Literal["npm", "pnpm", "yarn"] = "pnpm"
args: List[str] = Field(default_factory=list)
class RunRemote(RunBase):
runner: Literal["remote"]
base_url: HttpUrl
# Optional: metadata for how Castle should treat it
health_url: Optional[HttpUrl] = None
RunSpec = Annotated[
Union[RunCommand, RunPythonModule, RunPythonUvTool, RunContainer, RunNode, RunRemote],
Field(discriminator="runner"),
]
# -------------------------
# Triggers
# -------------------------
class TriggerManual(BaseModel):
type: Literal["manual"] = "manual"
class TriggerSchedule(BaseModel):
type: Literal["schedule"] = "schedule"
# Keep this simple + explicit. You can later add "cron"/"interval".
cron: str # e.g. "0 * * * *"
timezone: str = "America/Los_Angeles"
class TriggerEvent(BaseModel):
type: Literal["event"] = "event"
source: str # e.g. "kafka", "redis", "webhook", "fs"
topic: Optional[str] = None
queue: Optional[str] = None
class TriggerRequest(BaseModel):
type: Literal["request"] = "request"
protocol: Literal["http", "https", "grpc"] = "http"
TriggerSpec = Union[TriggerManual, TriggerSchedule, TriggerEvent, TriggerRequest]
# -------------------------
# Systemd management intent
# -------------------------
class ReadinessHttpGet(BaseModel):
http_get: HttpUrl | str # allow templated strings like "http://127.0.0.1:${PORT}/healthz"
timeout_seconds: int = 2
interval_seconds: int = 2
success_codes: List[int] = Field(default_factory=lambda: [200])
class SystemdSpec(BaseModel):
enable: bool = True
user: bool = True
description: Optional[str] = None
after: List[str] = Field(default_factory=list)
requires: List[str] = Field(default_factory=list)
wanted_by: List[str] = Field(default_factory=lambda: ["default.target"])
restart: RestartPolicy = RestartPolicy.ON_FAILURE
restart_sec: int = 2
# Optional hardening knobs (you can expand later)
no_new_privileges: bool = True
readiness: Optional[ReadinessHttpGet] = None
class ManageSpec(BaseModel):
systemd: Optional[SystemdSpec] = None
# -------------------------
# Install intent (PATH shims etc.)
# -------------------------
class PathInstallSpec(BaseModel):
enable: bool = True
# If set, Castle creates a shim with this name (e.g. ~/.local/bin/<alias>)
alias: Optional[str] = None
# If true, shim forwards args to `castle run <id> ...`
shim: bool = True
class InstallSpec(BaseModel):
path: Optional[PathInstallSpec] = None
# -------------------------
# HTTP exposure + proxy intent
# -------------------------
class HttpInternal(BaseModel):
host: str = "127.0.0.1"
port: int = Field(ge=1, le=65535)
unix_socket: Optional[str] = None # e.g. "/run/user/1000/notes.sock"
@model_validator(mode="after")
def _exactly_one_target(self):
# allow either (host+port) or unix_socket
if self.unix_socket and (self.host or self.port):
# host/port always exist via defaults; treat unix_socket as alternative
# if unix_socket is set, we ignore host/port semantically
return self
return self
class HttpPublic(BaseModel):
hostnames: List[str] = Field(min_length=1)
path_prefix: str = "/"
tls: TLSMode = TLSMode.INTERNAL
class HttpExposeSpec(BaseModel):
internal: HttpInternal
public: Optional[HttpPublic] = None
health_path: Optional[str] = None # e.g. "/healthz"
class ExposeSpec(BaseModel):
http: Optional[HttpExposeSpec] = None
# Future: cli, grpc, etc.
class CaddySpec(BaseModel):
enable: bool = True
# Optional: extra per-component knobs
extra_snippets: List[str] = Field(default_factory=list)
class ProxySpec(BaseModel):
caddy: Optional[CaddySpec] = None
# -------------------------
# Build spec (mostly for frontends, but generic)
# -------------------------
class BuildSpec(BaseModel):
# e.g. ["pnpm", "build"] or ["uv", "run", "python", "-m", "build"]
commands: List[List[str]] = Field(default_factory=list)
outputs: List[str] = Field(default_factory=list) # paths relative to cwd/repo
# -------------------------
# Capabilities (optional, for dependency graph)
# -------------------------
class Capability(BaseModel):
type: str # e.g. "http.endpoint", "cli.command", "ui.bundle"
name: Optional[str] = None
meta: Dict[str, str] = Field(default_factory=dict)
# -------------------------
# The manifest (source of truth)
# -------------------------
class ComponentManifest(BaseModel):
id: str = Field(pattern=r"^[a-z0-9][a-z0-9\-_.]{1,63}$")
name: Optional[str] = None
description: Optional[str] = None
run: RunSpec
triggers: List[TriggerSpec] = Field(default_factory=list)
manage: Optional[ManageSpec] = None
install: Optional[InstallSpec] = None
expose: Optional[ExposeSpec] = None
proxy: Optional[ProxySpec] = None
build: Optional[BuildSpec] = None
provides: List[Capability] = Field(default_factory=list)
consumes: List[Capability] = Field(default_factory=list)
tags: List[str] = Field(default_factory=list)
# ---- Derived ontology ----
@computed_field
@property
def roles(self) -> List[Role]:
"""
Derive roles purely from declared blocks.
No 'kind/profile' field required.
"""
roles: Set[Role] = set()
# Runner-derived roles
if self.run.runner == "remote":
roles.add(Role.REMOTE)
if self.run.runner == "container":
roles.add(Role.CONTAINERIZED)
# Interface / integration-derived roles
if self.install and self.install.path and self.install.path.enable:
roles.add(Role.TOOL)
if self.expose and self.expose.http:
roles.add(Role.SERVICE)
# Worker vs Service heuristic:
# If it's managed (systemd) but not exposed over HTTP, it's worker-ish.
if (
self.manage
and self.manage.systemd
and self.manage.systemd.enable
and not (self.expose and self.expose.http)
):
roles.add(Role.WORKER)
# Frontend heuristic: build outputs present (or node runner + build)
if self.build and (self.build.outputs or self.build.commands):
# Avoid labeling everything with build as frontend if it's clearly not;
# if it's exposing HTTP and has build, that's typically frontend or full-stack,
# but frontend label is still useful.
roles.add(Role.FRONTEND)
# Job heuristic: any schedule trigger
if any(getattr(t, "type", None) == "schedule" for t in self.triggers):
roles.add(Role.JOB)
# Fallback: if nothing else, treat as worker-ish if it’s supervised,
# otherwise tool-ish if it’s transient. But keep it conservative.
if not roles:
roles.add(Role.TOOL)
return sorted(roles, key=lambda r: r.value)
# ---- Optional consistency checks ----
@model_validator(mode="after")
def _basic_consistency(self):
# If you declare proxy.caddy, you probably mean to expose HTTP publicly.
if self.proxy and self.proxy.caddy and self.proxy.caddy.enable:
if not (self.expose and self.expose.http and self.expose.http.public):
# keep this as a soft constraint by raising a ValueError;
# remove if you prefer permissive manifests
raise ValueError(
"proxy.caddy is enabled but expose.http.public is not set. "
"Either disable caddy or declare public hostnames."
)
# If systemd is enabled, ensure runner isn't something obviously incompatible.
if self.manage and self.manage.systemd and self.manage.systemd.enable:
if self.run.runner == "remote":
raise ValueError("manage.systemd cannot be enabled for runner=remote.")
return self

74
working/validation-log.md
View File

@@ -1,74 +0,0 @@
# Validation Log
## What was implemented
### Castle CLI (`cli/`)
- `castle --version` — shows version
- `castle list [--type] [--json]` — list all projects, filter by type, JSON output
- `castle create <name> --type service|tool|library` — scaffold project, register in castle.yaml
- `castle test [project]` — run tests across one or all projects
- `castle lint [project]` — run ruff across one or all projects
- `castle sync` — git submodule update + uv sync in all projects
- `castle gateway start|stop|reload|status` — manage Caddy reverse proxy
- `castle service enable|disable <name>` — manage individual systemd units
- `castle service status` — show all service statuses
- `castle services start|stop` — start/stop everything
### Registry (`castle.yaml`)
- All 4 existing projects registered with correct types, ports, data dirs
- `castle create` auto-registers new projects
- Auto port assignment for new services
### Gateway (Caddy)
- Caddyfile generated from castle.yaml into `~/.castle/generated/`
- Dashboard HTML with health checks served at root
- Reverse proxy with `handle_path` (strips prefix) for each service
- Gateway managed as its own systemd unit
### Systemd integration
- User units generated under `~/.config/systemd/user/castle-*.service`
- Resolves `uv` to absolute path for systemd
- Resolves `${data_dir}` in env vars
- Creates data dirs via ExecStartPre
### Project templates
- Service: FastAPI, pydantic-settings, lifespan, health endpoint, test fixtures
- Tool: argparse, stdin/stdout, exit codes
- Library: src/ layout, import test
- All templates include CLAUDE.md and pyproject.toml with ruff isort config
### Configuration files
- `castle.yaml` — project registry
- `ruff.toml` — shared ruff config (100-char lines, E/F/I/W rules)
- `pyrightconfig.json` — shared pyright config
- `CLAUDE.md` — updated with full castle system docs
- `recommendations.md` — updated with all decisions
## What was validated
| Test | Result |
|------|--------|
| `castle --version` | 0.1.0 |
| `castle list` | Shows all 4 projects grouped by type |
| `castle list --json` | Valid JSON output |
| `castle list --type service` | Filters correctly |
| `castle create --type service` | Scaffolds, registers, auto-assigns port |
| `castle create --type tool` | Scaffolds, registers |
| `castle create --type library` | Scaffolds, registers |
| `castle create` duplicate | Fails with error message |
| `castle test <project>` | Passes for scaffolded projects |
| `castle lint <project>` | Passes for scaffolded projects |
| `castle sync` | Updates submodules + uv sync in all projects |
| `castle gateway start` | Starts Caddy, dashboard accessible |
| `castle gateway reload` | Regenerates and reloads |
| `castle gateway status` | Shows running/stopped |
| `castle service enable` | Creates unit, enables, starts |
| `castle service disable` | Stops, removes unit |
| `castle service status` | Shows all service statuses with colors |
| `castle services start` | Starts all services + gateway |
| `castle services stop` | Stops everything |
| Gateway reverse proxy | Routes correctly (path prefix stripped) |
| Gateway dashboard | Serves HTML with health check JS |
| Direct service access | All services respond on their ports |
| CLI unit tests | 32/32 passing |
| CLI lint | All checks passed |