diff --git a/.gitmodules b/.gitmodules index 1791c31..65bcc1e 100644 --- a/.gitmodules +++ b/.gitmodules @@ -1,6 +1,6 @@ [submodule "central-context"] - path = central-context + path = components/central-context url = https://github.com/payneio/central-context.git [submodule "notification-bridge"] - path = notification-bridge + path = components/notification-bridge url = https://github.com/payneio/attention-firewall.git diff --git a/CLAUDE.md b/CLAUDE.md index 098d26f..a61e950 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -28,18 +28,18 @@ When creating a new service, tool, or frontend, follow the detailed guides: ```bash # Service castle create my-service --type service --description "Does something" -cd my-service && uv sync +cd components/my-service && uv sync uv run my-service # starts on auto-assigned port castle service enable my-service # register with systemd castle gateway reload # update reverse proxy routes # Tool castle create my-tool --type tool --description "Does something" -cd my-tool && uv sync +cd components/my-tool && uv sync ``` -The `castle create` command scaffolds the project, generates a CLAUDE.md, -and registers it in `castle.yaml` as a `ComponentManifest`. +The `castle create` command scaffolds the project under `components/`, +generates a CLAUDE.md, and registers it in `castle.yaml` as a `ComponentManifest`. ## Castle CLI diff --git a/README.md b/README.md index c546130..257c5ca 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ open http://localhost:9000 ```bash # Service — FastAPI app with health endpoint, systemd unit, gateway route castle create my-api --type service --description "Does something useful" -cd my-api && uv sync +cd components/my-api && uv sync castle test my-api castle service enable my-api castle gateway reload @@ -68,7 +68,7 @@ components: run: runner: python_uv_tool tool: central-context - working_dir: central-context + working_dir: components/central-context env: CENTRAL_CONTEXT_DATA_DIR: /data/castle/central-context CENTRAL_CONTEXT_PORT: "9001" @@ -89,11 +89,13 @@ castle.yaml <- component registry (single source of truth) cli/ <- castle CLI castle-api/ <- Castle API (dashboard backend) app/ <- Castle web app (React/Vite frontend) -central-context/ <- content storage API (git submodule) -notification-bridge/ <- desktop notification forwarder (git submodule) -protonmail/ <- email sync tool/job -devbox-connect/ <- SSH tunnel manager -pdf2md/ <- standalone tool (each tool is its own project) +components/ <- all non-infrastructure components + central-context/ <- content storage API (git submodule) + notification-bridge/ <- desktop notification forwarder (git submodule) + protonmail/ <- email sync tool/job + devbox-connect/ <- SSH tunnel manager + pdf2md/ <- standalone tool (each tool is its own project) + ... ruff.toml <- shared lint config pyrightconfig.json <- shared type checking config ``` diff --git a/castle.yaml b/castle.yaml index 791b3f3..da49e46 100644 --- a/castle.yaml +++ b/castle.yaml @@ -27,7 +27,7 @@ components: on the LAN run: runner: python_uv_tool - working_dir: central-context + working_dir: components/central-context env: CENTRAL_CONTEXT_DATA_DIR: /data/castle/central-context CENTRAL_CONTEXT_PORT: '9001' @@ -48,7 +48,7 @@ components: server. run: runner: python_uv_tool - working_dir: notification-bridge + working_dir: components/notification-bridge env: CENTRAL_CONTEXT_URL: http://localhost:9001 BUCKET_NAME: notifications @@ -86,7 +86,7 @@ components: description: ProtonMail email sync via Bridge run: runner: command - working_dir: protonmail + working_dir: components/protonmail env: PROTONMAIL_USERNAME: paul@payne.io PROTONMAIL_API_KEY: ${secret:PROTONMAIL_API_KEY} @@ -103,7 +103,7 @@ components: path: alias: protonmail tool: - source: protonmail/ + source: components/protonmail/ backup-collect: description: Collect files from various sources into backup directory run: @@ -120,7 +120,7 @@ components: manage: systemd: {} tool: - source: backup-collect/ + source: components/backup-collect/ system_dependencies: - rsync backup-data: @@ -155,21 +155,21 @@ components: path: alias: devbox-connect tool: - source: devbox-connect/ + source: components/devbox-connect/ mbox2eml: description: MBOX to EML email converter install: path: alias: mbox2eml tool: - source: mbox2eml/ + source: components/mbox2eml/ android-backup: description: Backup Android device using ADB install: path: alias: android-backup tool: - source: android-backup/ + source: components/android-backup/ system_dependencies: - adb browser: @@ -178,14 +178,14 @@ components: path: alias: browser tool: - source: browser/ + source: components/browser/ docx-extractor: description: Extract content and metadata from Word .docx files install: path: alias: docx-extractor tool: - source: docx-extractor/ + source: components/docx-extractor/ system_dependencies: - pandoc docx2md: @@ -194,7 +194,7 @@ components: path: alias: docx2md tool: - source: docx2md/ + source: components/docx2md/ system_dependencies: - pandoc gpt: @@ -203,21 +203,21 @@ components: path: alias: gpt tool: - source: gpt/ + source: components/gpt/ html2text: description: Convert HTML content to plain text install: path: alias: html2text tool: - source: html2text/ + source: components/html2text/ md2pdf: description: Convert Markdown files to PDF install: path: alias: md2pdf tool: - source: md2pdf/ + source: components/md2pdf/ system_dependencies: - pandoc - texlive-latex-base @@ -227,21 +227,21 @@ components: path: alias: mdscraper tool: - source: mdscraper/ + source: components/mdscraper/ pdf-extractor: description: Extract content and metadata from PDF files install: path: alias: pdf-extractor tool: - source: pdf-extractor/ + source: components/pdf-extractor/ pdf2md: description: Convert PDF files to Markdown install: path: alias: pdf2md tool: - source: pdf2md/ + source: components/pdf2md/ system_dependencies: - pandoc - poppler-utils @@ -251,18 +251,18 @@ components: path: alias: schedule tool: - source: schedule/ + source: components/schedule/ search: description: Manage self-contained searchable collections of files install: path: alias: search tool: - source: search/ + source: components/search/ text-extractor: description: Extract content and metadata from text files install: path: alias: text-extractor tool: - source: text-extractor/ + source: components/text-extractor/ diff --git a/cli/src/castle_cli/commands/create.py b/cli/src/castle_cli/commands/create.py index 44a8b22..6912f1a 100644 --- a/cli/src/castle_cli/commands/create.py +++ b/cli/src/castle_cli/commands/create.py @@ -47,9 +47,11 @@ def run_create(args: argparse.Namespace) -> int: print(f"Error: component '{name}' already exists in castle.yaml") return 1 - project_dir = config.root / name + components_dir = config.root / "components" + components_dir.mkdir(exist_ok=True) + project_dir = components_dir / name if project_dir.exists(): - print(f"Error: directory '{name}' already exists") + print(f"Error: directory 'components/{name}' already exists") return 1 # Determine port for services @@ -80,7 +82,7 @@ def run_create(args: argparse.Namespace) -> int: run=RunPythonUvTool( runner="python_uv_tool", tool=name, - cwd=name, + cwd=f"components/{name}", env={f"{env_prefix}_DATA_DIR": f"/data/castle/{name}"}, ), expose=ExposeSpec( @@ -96,7 +98,7 @@ def run_create(args: argparse.Namespace) -> int: manifest = ComponentManifest( id=name, description=args.description or f"A castle {proj_type}", - tool=ToolSpec(source=f"{name}/"), + tool=ToolSpec(source=f"components/{name}/"), install=InstallSpec(path=PathInstallSpec(alias=name)), ) else: @@ -114,7 +116,7 @@ def run_create(args: argparse.Namespace) -> int: print(f" Port: {port}") print(" Registered in castle.yaml") print("\nNext steps:") - print(f" cd {name}") + print(f" cd components/{name}") print(" uv sync") if proj_type == "service": print(f" uv run {name} # starts on port {port}") diff --git a/cli/src/castle_cli/commands/dev.py b/cli/src/castle_cli/commands/dev.py index 9008a33..6457586 100644 --- a/cli/src/castle_cli/commands/dev.py +++ b/cli/src/castle_cli/commands/dev.py @@ -14,8 +14,7 @@ def _get_project_dir(config: CastleConfig, project_name: str) -> Path: if project_name not in config.components: raise ValueError(f"Unknown component: {project_name}") manifest = config.components[project_name] - cwd = manifest.run.working_dir if manifest.run else None - working_dir = cwd or project_name + working_dir = manifest.source_dir or project_name return config.root / working_dir @@ -59,8 +58,7 @@ def run_test(args: argparse.Namespace) -> int: # Run all all_passed = True for name, manifest in config.components.items(): - cwd = manifest.run.working_dir if manifest.run else None - working_dir = cwd or name + working_dir = manifest.source_dir or name project_dir = config.root / working_dir tests_dir = project_dir / "tests" if not tests_dir.exists(): @@ -98,8 +96,7 @@ def run_lint(args: argparse.Namespace) -> int: # Run all all_passed = True for name, manifest in config.components.items(): - cwd = manifest.run.working_dir if manifest.run else None - working_dir = cwd or name + working_dir = manifest.source_dir or name project_dir = config.root / working_dir if not _has_pyproject(project_dir): continue diff --git a/cli/src/castle_cli/commands/info.py b/cli/src/castle_cli/commands/info.py index 06b1a90..ba57bd7 100644 --- a/cli/src/castle_cli/commands/info.py +++ b/cli/src/castle_cli/commands/info.py @@ -29,8 +29,7 @@ def run_info(args: argparse.Namespace) -> int: if getattr(args, "json", False): data = manifest.model_dump(exclude_none=True) # Include CLAUDE.md content if it exists - cwd = manifest.run.working_dir if manifest.run else None - claude_md = _find_claude_md(config.root, cwd or name) + claude_md = _find_claude_md(config.root, manifest.source_dir or name) if claude_md: data["claude_md"] = claude_md print(json.dumps(data, indent=2)) diff --git a/cli/src/castle_cli/commands/sync.py b/cli/src/castle_cli/commands/sync.py index cd9178a..d85de1d 100644 --- a/cli/src/castle_cli/commands/sync.py +++ b/cli/src/castle_cli/commands/sync.py @@ -28,8 +28,7 @@ def run_sync(args: argparse.Namespace) -> int: all_ok = True synced_dirs: set[Path] = set() for name, manifest in config.components.items(): - cwd = manifest.run.working_dir if manifest.run else None - working_dir = cwd or name + working_dir = manifest.source_dir or name project_dir = config.root / working_dir pyproject = project_dir / "pyproject.toml" diff --git a/cli/src/castle_cli/manifest.py b/cli/src/castle_cli/manifest.py index 7c376a1..862acca 100644 --- a/cli/src/castle_cli/manifest.py +++ b/cli/src/castle_cli/manifest.py @@ -302,6 +302,21 @@ class ComponentManifest(BaseModel): return sorted(roles, key=lambda r: r.value) + @property + def source_dir(self) -> str | None: + """Best-effort relative directory for this component's source. + + Resolution order: run.working_dir → build.working_dir → tool.source (strip trailing /). + Returns None if no directory can be determined. + """ + if self.run and self.run.working_dir: + return self.run.working_dir + if self.build and self.build.working_dir: + return self.build.working_dir + if self.tool and self.tool.source: + return self.tool.source.rstrip("/") + return None + @model_validator(mode="after") def _basic_consistency(self) -> ComponentManifest: if self.manage and self.manage.systemd and self.manage.systemd.enable: diff --git a/android-backup/pyproject.toml b/components/android-backup/pyproject.toml similarity index 100% rename from android-backup/pyproject.toml rename to components/android-backup/pyproject.toml diff --git a/android-backup/src/android_backup/__init__.py b/components/android-backup/src/android_backup/__init__.py similarity index 100% rename from android-backup/src/android_backup/__init__.py rename to components/android-backup/src/android_backup/__init__.py diff --git a/android-backup/src/android_backup/main.py b/components/android-backup/src/android_backup/main.py similarity index 100% rename from android-backup/src/android_backup/main.py rename to components/android-backup/src/android_backup/main.py diff --git a/backup-collect/pyproject.toml b/components/backup-collect/pyproject.toml similarity index 100% rename from backup-collect/pyproject.toml rename to components/backup-collect/pyproject.toml diff --git a/backup-collect/src/backup_collect/__init__.py b/components/backup-collect/src/backup_collect/__init__.py similarity index 100% rename from backup-collect/src/backup_collect/__init__.py rename to components/backup-collect/src/backup_collect/__init__.py diff --git a/backup-collect/src/backup_collect/main.py b/components/backup-collect/src/backup_collect/main.py similarity index 100% rename from backup-collect/src/backup_collect/main.py rename to components/backup-collect/src/backup_collect/main.py diff --git a/browser/pyproject.toml b/components/browser/pyproject.toml similarity index 100% rename from browser/pyproject.toml rename to components/browser/pyproject.toml diff --git a/browser/src/browser/__init__.py b/components/browser/src/browser/__init__.py similarity index 100% rename from browser/src/browser/__init__.py rename to components/browser/src/browser/__init__.py diff --git a/browser/src/browser/main.py b/components/browser/src/browser/main.py similarity index 100% rename from browser/src/browser/main.py rename to components/browser/src/browser/main.py diff --git a/central-context b/components/central-context similarity index 100% rename from central-context rename to components/central-context diff --git a/devbox-connect/README.md b/components/devbox-connect/README.md similarity index 100% rename from devbox-connect/README.md rename to components/devbox-connect/README.md diff --git a/devbox-connect/pyproject.toml b/components/devbox-connect/pyproject.toml similarity index 100% rename from devbox-connect/pyproject.toml rename to components/devbox-connect/pyproject.toml diff --git a/devbox-connect/service/install-service.ps1 b/components/devbox-connect/service/install-service.ps1 similarity index 100% rename from devbox-connect/service/install-service.ps1 rename to components/devbox-connect/service/install-service.ps1 diff --git a/devbox-connect/service/run-manual.bat b/components/devbox-connect/service/run-manual.bat similarity index 100% rename from devbox-connect/service/run-manual.bat rename to components/devbox-connect/service/run-manual.bat diff --git a/devbox-connect/src/devbox_connect/__init__.py b/components/devbox-connect/src/devbox_connect/__init__.py similarity index 100% rename from devbox-connect/src/devbox_connect/__init__.py rename to components/devbox-connect/src/devbox_connect/__init__.py diff --git a/devbox-connect/src/devbox_connect/cli.py b/components/devbox-connect/src/devbox_connect/cli.py similarity index 100% rename from devbox-connect/src/devbox_connect/cli.py rename to components/devbox-connect/src/devbox_connect/cli.py diff --git a/devbox-connect/src/devbox_connect/config.py b/components/devbox-connect/src/devbox_connect/config.py similarity index 100% rename from devbox-connect/src/devbox_connect/config.py rename to components/devbox-connect/src/devbox_connect/config.py diff --git a/devbox-connect/src/devbox_connect/tunnel.py b/components/devbox-connect/src/devbox_connect/tunnel.py similarity index 100% rename from devbox-connect/src/devbox_connect/tunnel.py rename to components/devbox-connect/src/devbox_connect/tunnel.py diff --git a/devbox-connect/tunnels.example.yaml b/components/devbox-connect/tunnels.example.yaml similarity index 100% rename from devbox-connect/tunnels.example.yaml rename to components/devbox-connect/tunnels.example.yaml diff --git a/devbox-connect/uv.lock b/components/devbox-connect/uv.lock similarity index 100% rename from devbox-connect/uv.lock rename to components/devbox-connect/uv.lock diff --git a/docx-extractor/pyproject.toml b/components/docx-extractor/pyproject.toml similarity index 100% rename from docx-extractor/pyproject.toml rename to components/docx-extractor/pyproject.toml diff --git a/docx-extractor/src/docx_extractor/__init__.py b/components/docx-extractor/src/docx_extractor/__init__.py similarity index 100% rename from docx-extractor/src/docx_extractor/__init__.py rename to components/docx-extractor/src/docx_extractor/__init__.py diff --git a/docx-extractor/src/docx_extractor/main.py b/components/docx-extractor/src/docx_extractor/main.py similarity index 100% rename from docx-extractor/src/docx_extractor/main.py rename to components/docx-extractor/src/docx_extractor/main.py diff --git a/docx2md/pyproject.toml b/components/docx2md/pyproject.toml similarity index 100% rename from docx2md/pyproject.toml rename to components/docx2md/pyproject.toml diff --git a/docx2md/src/docx2md/__init__.py b/components/docx2md/src/docx2md/__init__.py similarity index 100% rename from docx2md/src/docx2md/__init__.py rename to components/docx2md/src/docx2md/__init__.py diff --git a/docx2md/src/docx2md/main.py b/components/docx2md/src/docx2md/main.py similarity index 100% rename from docx2md/src/docx2md/main.py rename to components/docx2md/src/docx2md/main.py diff --git a/gpt/pyproject.toml b/components/gpt/pyproject.toml similarity index 100% rename from gpt/pyproject.toml rename to components/gpt/pyproject.toml diff --git a/gpt/src/gpt/__init__.py b/components/gpt/src/gpt/__init__.py similarity index 100% rename from gpt/src/gpt/__init__.py rename to components/gpt/src/gpt/__init__.py diff --git a/gpt/src/gpt/main.py b/components/gpt/src/gpt/main.py similarity index 100% rename from gpt/src/gpt/main.py rename to components/gpt/src/gpt/main.py diff --git a/html2text/pyproject.toml b/components/html2text/pyproject.toml similarity index 100% rename from html2text/pyproject.toml rename to components/html2text/pyproject.toml diff --git a/html2text/src/html2text/__init__.py b/components/html2text/src/html2text/__init__.py similarity index 100% rename from html2text/src/html2text/__init__.py rename to components/html2text/src/html2text/__init__.py diff --git a/html2text/src/html2text/main.py b/components/html2text/src/html2text/main.py similarity index 100% rename from html2text/src/html2text/main.py rename to components/html2text/src/html2text/main.py diff --git a/mbox2eml/pyproject.toml b/components/mbox2eml/pyproject.toml similarity index 100% rename from mbox2eml/pyproject.toml rename to components/mbox2eml/pyproject.toml diff --git a/mbox2eml/src/mbox2eml/__init__.py b/components/mbox2eml/src/mbox2eml/__init__.py similarity index 100% rename from mbox2eml/src/mbox2eml/__init__.py rename to components/mbox2eml/src/mbox2eml/__init__.py diff --git a/mbox2eml/src/mbox2eml/main.py b/components/mbox2eml/src/mbox2eml/main.py similarity index 100% rename from mbox2eml/src/mbox2eml/main.py rename to components/mbox2eml/src/mbox2eml/main.py diff --git a/md2pdf/pyproject.toml b/components/md2pdf/pyproject.toml similarity index 100% rename from md2pdf/pyproject.toml rename to components/md2pdf/pyproject.toml diff --git a/md2pdf/src/md2pdf/__init__.py b/components/md2pdf/src/md2pdf/__init__.py similarity index 100% rename from md2pdf/src/md2pdf/__init__.py rename to components/md2pdf/src/md2pdf/__init__.py diff --git a/md2pdf/src/md2pdf/main.py b/components/md2pdf/src/md2pdf/main.py similarity index 100% rename from md2pdf/src/md2pdf/main.py rename to components/md2pdf/src/md2pdf/main.py diff --git a/mdscraper/pyproject.toml b/components/mdscraper/pyproject.toml similarity index 100% rename from mdscraper/pyproject.toml rename to components/mdscraper/pyproject.toml diff --git a/mdscraper/src/mdscraper/__init__.py b/components/mdscraper/src/mdscraper/__init__.py similarity index 100% rename from mdscraper/src/mdscraper/__init__.py rename to components/mdscraper/src/mdscraper/__init__.py diff --git a/mdscraper/src/mdscraper/main.py b/components/mdscraper/src/mdscraper/main.py similarity index 100% rename from mdscraper/src/mdscraper/main.py rename to components/mdscraper/src/mdscraper/main.py diff --git a/notification-bridge b/components/notification-bridge similarity index 100% rename from notification-bridge rename to components/notification-bridge diff --git a/pdf-extractor/pyproject.toml b/components/pdf-extractor/pyproject.toml similarity index 100% rename from pdf-extractor/pyproject.toml rename to components/pdf-extractor/pyproject.toml diff --git a/pdf-extractor/src/pdf_extractor/__init__.py b/components/pdf-extractor/src/pdf_extractor/__init__.py similarity index 100% rename from pdf-extractor/src/pdf_extractor/__init__.py rename to components/pdf-extractor/src/pdf_extractor/__init__.py diff --git a/pdf-extractor/src/pdf_extractor/main.py b/components/pdf-extractor/src/pdf_extractor/main.py similarity index 100% rename from pdf-extractor/src/pdf_extractor/main.py rename to components/pdf-extractor/src/pdf_extractor/main.py diff --git a/pdf2md/pyproject.toml b/components/pdf2md/pyproject.toml similarity index 100% rename from pdf2md/pyproject.toml rename to components/pdf2md/pyproject.toml diff --git a/pdf2md/src/pdf2md/__init__.py b/components/pdf2md/src/pdf2md/__init__.py similarity index 100% rename from pdf2md/src/pdf2md/__init__.py rename to components/pdf2md/src/pdf2md/__init__.py diff --git a/pdf2md/src/pdf2md/main.py b/components/pdf2md/src/pdf2md/main.py similarity index 100% rename from pdf2md/src/pdf2md/main.py rename to components/pdf2md/src/pdf2md/main.py diff --git a/protonmail/CLAUDE.md b/components/protonmail/CLAUDE.md similarity index 100% rename from protonmail/CLAUDE.md rename to components/protonmail/CLAUDE.md diff --git a/protonmail/README.md b/components/protonmail/README.md similarity index 100% rename from protonmail/README.md rename to components/protonmail/README.md diff --git a/protonmail/pyproject.toml b/components/protonmail/pyproject.toml similarity index 100% rename from protonmail/pyproject.toml rename to components/protonmail/pyproject.toml diff --git a/protonmail/src/protonmail/__init__.py b/components/protonmail/src/protonmail/__init__.py similarity index 100% rename from protonmail/src/protonmail/__init__.py rename to components/protonmail/src/protonmail/__init__.py diff --git a/protonmail/src/protonmail/main.py b/components/protonmail/src/protonmail/main.py similarity index 100% rename from protonmail/src/protonmail/main.py rename to components/protonmail/src/protonmail/main.py diff --git a/protonmail/tests/__init__.py b/components/protonmail/tests/__init__.py similarity index 100% rename from protonmail/tests/__init__.py rename to components/protonmail/tests/__init__.py diff --git a/protonmail/tests/test_main.py b/components/protonmail/tests/test_main.py similarity index 100% rename from protonmail/tests/test_main.py rename to components/protonmail/tests/test_main.py diff --git a/protonmail/uv.lock b/components/protonmail/uv.lock similarity index 100% rename from protonmail/uv.lock rename to components/protonmail/uv.lock diff --git a/schedule/pyproject.toml b/components/schedule/pyproject.toml similarity index 100% rename from schedule/pyproject.toml rename to components/schedule/pyproject.toml diff --git a/schedule/src/schedule/__init__.py b/components/schedule/src/schedule/__init__.py similarity index 100% rename from schedule/src/schedule/__init__.py rename to components/schedule/src/schedule/__init__.py diff --git a/schedule/src/schedule/main.py b/components/schedule/src/schedule/main.py similarity index 100% rename from schedule/src/schedule/main.py rename to components/schedule/src/schedule/main.py diff --git a/search/pyproject.toml b/components/search/pyproject.toml similarity index 100% rename from search/pyproject.toml rename to components/search/pyproject.toml diff --git a/search/src/search/__init__.py b/components/search/src/search/__init__.py similarity index 100% rename from search/src/search/__init__.py rename to components/search/src/search/__init__.py diff --git a/search/src/search/main.py b/components/search/src/search/main.py similarity index 100% rename from search/src/search/main.py rename to components/search/src/search/main.py diff --git a/text-extractor/pyproject.toml b/components/text-extractor/pyproject.toml similarity index 100% rename from text-extractor/pyproject.toml rename to components/text-extractor/pyproject.toml diff --git a/text-extractor/src/text_extractor/__init__.py b/components/text-extractor/src/text_extractor/__init__.py similarity index 100% rename from text-extractor/src/text_extractor/__init__.py rename to components/text-extractor/src/text_extractor/__init__.py diff --git a/text-extractor/src/text_extractor/main.py b/components/text-extractor/src/text_extractor/main.py similarity index 100% rename from text-extractor/src/text_extractor/main.py rename to components/text-extractor/src/text_extractor/main.py diff --git a/recommendations.md b/recommendations.md deleted file mode 100644 index 2859a11..0000000 --- a/recommendations.md +++ /dev/null @@ -1,187 +0,0 @@ -# Scaling Recommendations - -## Decisions made - -- **Git structure**: Monorepo with submodules. Castle is a git repo; projects that need independent publishing are their own repos added as submodules. Projects without remotes (e.g., devbox-connect) are tracked directly until they get their own repo. -- **Scope**: Castle is a personal software platform — not just services, but tools, libraries, and apps. Toolkit (v1, CLI tools only) is being absorbed and generalized. -- **Gateway**: Caddy reverse proxy + generated dashboard. Single port for all web services. Caddy chosen over Traefik because the service registry is static (no container orchestration), and Caddyfile syntax is trivially simple. -- **Event bus**: A lightweight castle-component for inter-service communication, so services don't hardcode knowledge of each other. -- **Independence principle**: Regular services/tools/libraries must never depend on castle. They accept standard configuration (data dir, port, URLs) via env vars or args. Only "castle-components" (CLI, gateway, event bus) know about castle internals like `castle.yaml`. This keeps services portable and independently publishable. -- **Registry**: `castle.yaml` at the repo root. Centralized — all projects are registered here. `castle create` adds entries automatically. No marker files in projects (would violate independence principle). -- **Discovery**: Centralized via `castle.yaml`. The CLI reads this file to know what projects exist, their types, and how to orchestrate them. -- **CLI location**: `cli/` directory at the repo root, installed via `uv tool install`. -- **Generated files**: `~/.castle/generated/` for Caddyfiles, systemd units, dashboard HTML. Separate from `~/.castle/secrets/`. - -## 1. Build the `castle` CLI - -The top-level CLI lives in `cli/` and is installed via `uv tool install`. It replaces both toolkit's `toolkit` command and the need for a Makefile/justfile. It should: -- Discover projects by type (tool, service, library, app) from `castle.yaml` -- Scaffold new projects from templates: `castle create --type service` -- Run commands across projects: `castle test`, `castle lint`, `castle sync` -- Wrap submodule pain points: `castle sync` does `git submodule update --init --recursive` -- Manage services: `castle service enable/disable/status`, `castle services start/stop` -- Manage gateway: `castle gateway start/reload` -- Register `uv tool` entries for tool-type projects - -This generalizes toolkit's discovery/scaffolding pattern (YAML frontmatter in markdown, `toolkit create`) across all project types. - -## 2. Define project type templates - -Each type encodes best practices: -- **tool**: argparse, stdin/stdout, exit codes, single-purpose (toolkit pattern) -- **service**: FastAPI, pydantic-settings, lifespan, health endpoint -- **library**: src/ layout, typed API, no CLI entry point -- **app**: TBD as needs emerge - -Shared patterns (settings base class, error handling, test fixtures) live in templates rather than a shared library — avoids a runtime dependency that couples all projects. - -## 3. Standardize project layout - -Pick `src//` for all projects. Currently inconsistent: `src/central_context/`, flat `notification_bridge/`, single-file `convert.py`. The castle CLI's discovery and scaffolding depends on predictable structure. - -## 4. Registry, gateway, and systemd - -`castle.yaml` at the repo root is the single source of truth for all projects — their types, ports, paths, data directories, commands, and inter-service relationships. - -The castle CLI generates artifacts from this registry into `~/.castle/generated/`: -- **Caddyfile** — reverse proxy config so all services are accessible under one port (e.g., `localhost:9000/central-context/*` → `localhost:9001/*`) -- **Dashboard HTML** — served at the gateway root (`localhost:9000/`) with links to each service, health status, and docs links -- **Systemd user units** — `.service` files under `~/.config/systemd/user/` - -Example `castle.yaml`: -```yaml -gateway: - port: 9000 - -projects: - # Services (long-running, have ports) - central-context: - type: service - port: 9001 - path: /central-context - command: uv run central-context - working_dir: central-context - data_dir: /data/castle/central-context - description: Content storage API - health: /health - env: - CENTRAL_CONTEXT_DATA_DIR: ${data_dir} - - notification-bridge: - type: service - port: 9001 - path: /notifications - command: uv run notification-bridge - working_dir: notification-bridge - data_dir: /data/castle/notification-bridge - description: Desktop notification forwarder - health: /health - publishes: - - notification.received - - devbox-connect: - type: tool - description: SSH tunnel manager with auto-reconnect - - mboxer: - type: tool - description: MBOX to EML email converter - - # Castle-components - event-bus: - type: castle-component - port: 9010 - path: /events - command: uv run event-bus - description: Inter-service event bus -``` - -### Gateway commands - -``` -castle gateway start # generate Caddyfile + dashboard, start Caddy -castle gateway reload # regenerate after castle.yaml changes -``` - -### Systemd commands - -``` -castle service enable central-context # generate unit, enable, start -castle service disable central-context # stop and disable -castle service status # show status of all services -castle services start # enable + start everything (including gateway) -castle services stop # stop everything -``` - -The gateway itself is also a systemd unit (`castle-gateway.service`), so `castle services start` brings up all services and the Caddy proxy in one command. - -Castle resolves `${data_dir}` references, ensures data directories exist, and passes env vars when generating units. - -## 5. Agent context strategy - -The primary value of castle is that agents can rapidly create and manage software in a standardized way. The conventions must be machine-discoverable. - -- **Top-level `CLAUDE.md`** is the agent's entry point into the entire system. It should reference `castle.yaml`, explain project types, link to templates, and describe the agent workflow (scaffold → register → test → enable). -- **`castle create` updates context automatically** — when a new project is scaffolded, the CLI generates a project-level `CLAUDE.md` from the template and registers the project in `castle.yaml`. -- **Each project type template includes a `CLAUDE.md` template** so agents immediately understand a project's conventions, build commands, and architecture upon reading it. -- **The agent workflow is explicit**: an agent creating a new service follows: `castle create` → implement → `castle test` → `castle service enable`. No tribal knowledge required. - -## 6. Data persistence conventions - -As castle replaces commercial applications, the data these services hold becomes the valuable part. Conventions: - -- **Each service's data dir is configured in `castle.yaml`** — defaults to `/data/castle//`. Castle supplies this to the service at launch (via env var or arg). The service itself just accepts a data dir setting — it has no knowledge of castle. -- **Data directories are never inside submodule trees** — submodules get cloned fresh; persistent data must live outside them. -- **Backup stays generic** — `backup-collect` remains a general-purpose tool. It doesn't read `castle.yaml`. Castle can separately generate a backup manifest from the registry if needed, but that's a castle concern, not a backup-collect concern. - -## 7. Secret management - -API keys, tokens, and credentials will accumulate as services replace commercial apps. Rules: - -- **Secrets live in `~/.castle/secrets/`** — never in project directories (submodules get pushed to GitHub). Agents must be told this explicitly in context. -- **`castle.yaml` can reference secrets by name** — the castle CLI resolves them when generating systemd units or passing env vars at launch. Services themselves just receive env vars — they don't know where the values came from. - -## 8. Event bus - -A castle-component that decouples inter-service communication. Currently notification-bridge hardcodes central-context's URL — this won't scale to dozens of services that need to react to each other's events. - -The event bus is a FastAPI service registered in `castle.yaml`: -- Services **publish** typed events: `POST /events/publish` with `{topic, payload}` -- Services **subscribe** to topics: register a webhook callback in `castle.yaml` or via `POST /events/subscribe` -- The bus delivers events to subscribers via HTTP POST to their registered endpoints - -This keeps services decoupled — a service only knows about the bus, not about other services. Example flow: notification-bridge publishes a `notification.received` event, and any service that cares subscribes to that topic. - -Subscriptions are declared in `castle.yaml` (see example above). The castle CLI configures the bus with the subscription table at startup. - -The bus should be simple — no persistence, no guaranteed delivery, no complex routing. Just HTTP fan-out. Add durability later only if needed. - -## 9. Consistent ruff/pyright configuration - -Each project has its own ruff rules (devbox-connect: `E,F,I,W`; mboxer: `ALL`). Put a shared `ruff.toml` at the repo root (ruff walks up to find config). Same for pyright — only devbox-connect has it currently. - -## 10. Absorb toolkit - -Don't move toolkit in as a monolith. Instead: -1. Add toolkit as a submodule -2. Graduate all tools into independent castle projects (each tool is its own standalone package) -3. Promote toolkit's meta-tooling up into the castle CLI - -## 11. What to defer - -- **Shared runtime library (`castle-core`)** — templates are better than a runtime dependency for now. Revisit if projects start importing shared code at runtime. -- **Containerization/orchestration** — until deploying beyond the local machine -- **API gateway / service mesh** — Caddy handles reverse proxying; a full mesh is premature -- **Distributed tracing / observability** — add when debugging cross-service issues becomes painful -- **Formal API schema sharing** — FastAPI generates OpenAPI already; formalize when consumers need stability guarantees - -## Priority - -1. **Castle CLI** with project discovery and scaffolding -2. **Registry** (`castle.yaml`) + **Caddy gateway** + **systemd integration** -3. **Agent context strategy** — CLAUDE.md generation in templates, agent workflow docs -4. **Data persistence conventions** + **secret management** -5. **Standardize layout** across existing projects -6. **Root-level ruff/pyright config** -7. **Event bus** castle-component -8. **Absorb toolkit** incrementally