feat: Add various tools for document processing and management

- Introduced `docx-extractor` for extracting content and metadata from Word .docx files.
- Added `docx2md` for converting Word .docx files to Markdown format.
- Implemented `gpt` for generating text using OpenAI's GPT models.
- Created `html2text` for converting HTML content to plain text.
- Developed `mbox2eml` for converting MBOX mailbox files to individual .eml files.
- Added `md2pdf` for converting Markdown files to PDF format.
- Introduced `mdscraper` for combining text files into a single markdown document.
- Created `pdf-extractor` for extracting content and metadata from PDF files.
- Developed `pdf2md` for converting PDF files to Markdown format.
- Implemented `protonmail` for managing ProtonMail emails via Bridge.
- Added `schedule` for managing systemd timers and services.
- Introduced `search` for managing searchable collections of files.
- Added `text-extractor` for extracting content and metadata from text files.
- Removed outdated recommendations document.

.gitmodules

@@ -1,6 +1,6 @@
 [submodule "central-context"]
-	path = central-context
+	path = components/central-context
 	url = https://github.com/payneio/central-context.git
 [submodule "notification-bridge"]
-	path = notification-bridge
+	path = components/notification-bridge
 	url = https://github.com/payneio/attention-firewall.git

CLAUDE.md

@@ -28,18 +28,18 @@ When creating a new service, tool, or frontend, follow the detailed guides:
 ```bash
 # Service
 castle create my-service --type service --description "Does something"
-cd my-service && uv sync
+cd components/my-service && uv sync
 uv run my-service               # starts on auto-assigned port
 castle service enable my-service # register with systemd
 castle gateway reload            # update reverse proxy routes
 
 # Tool
 castle create my-tool --type tool --description "Does something"
-cd my-tool && uv sync
+cd components/my-tool && uv sync
 ```
 
-The `castle create` command scaffolds the project, generates a CLAUDE.md,
+The `castle create` command scaffolds the project under `components/`,
-and registers it in `castle.yaml` as a `ComponentManifest`.
+generates a CLAUDE.md, and registers it in `castle.yaml` as a `ComponentManifest`.
 
 ## Castle CLI
 

README.md

@@ -26,7 +26,7 @@ open http://localhost:9000
 ```bash
 # Service — FastAPI app with health endpoint, systemd unit, gateway route
 castle create my-api --type service --description "Does something useful"
-cd my-api && uv sync
+cd components/my-api && uv sync
 castle test my-api
 castle service enable my-api
 castle gateway reload
@@ -68,7 +68,7 @@ components:
     run:
       runner: python_uv_tool
       tool: central-context
-      working_dir: central-context
+      working_dir: components/central-context
       env:
         CENTRAL_CONTEXT_DATA_DIR: /data/castle/central-context
         CENTRAL_CONTEXT_PORT: "9001"
@@ -89,11 +89,13 @@ castle.yaml          <- component registry (single source of truth)
 cli/                 <- castle CLI
 castle-api/          <- Castle API (dashboard backend)
 app/                 <- Castle web app (React/Vite frontend)
+components/          <- all non-infrastructure components
   central-context/   <- content storage API (git submodule)
   notification-bridge/ <- desktop notification forwarder (git submodule)
   protonmail/        <- email sync tool/job
   devbox-connect/    <- SSH tunnel manager
   pdf2md/            <- standalone tool (each tool is its own project)
+  ...
 ruff.toml            <- shared lint config
 pyrightconfig.json   <- shared type checking config
 ```

castle.yaml

@@ -27,7 +27,7 @@ components:
       on the LAN
     run:
       runner: python_uv_tool
-      working_dir: central-context
+      working_dir: components/central-context
       env:
         CENTRAL_CONTEXT_DATA_DIR: /data/castle/central-context
         CENTRAL_CONTEXT_PORT: '9001'
@@ -48,7 +48,7 @@ components:
       server.
     run:
       runner: python_uv_tool
-      working_dir: notification-bridge
+      working_dir: components/notification-bridge
       env:
         CENTRAL_CONTEXT_URL: http://localhost:9001
         BUCKET_NAME: notifications
@@ -86,7 +86,7 @@ components:
     description: ProtonMail email sync via Bridge
     run:
       runner: command
-      working_dir: protonmail
+      working_dir: components/protonmail
       env:
         PROTONMAIL_USERNAME: paul@payne.io
         PROTONMAIL_API_KEY: ${secret:PROTONMAIL_API_KEY}
@@ -103,7 +103,7 @@ components:
       path:
         alias: protonmail
     tool:
-      source: protonmail/
+      source: components/protonmail/
   backup-collect:
     description: Collect files from various sources into backup directory
     run:
@@ -120,7 +120,7 @@ components:
     manage:
       systemd: {}
     tool:
-      source: backup-collect/
+      source: components/backup-collect/
       system_dependencies:
       - rsync
   backup-data:
@@ -155,21 +155,21 @@ components:
       path:
         alias: devbox-connect
     tool:
-      source: devbox-connect/
+      source: components/devbox-connect/
   mbox2eml:
     description: MBOX to EML email converter
     install:
       path:
         alias: mbox2eml
     tool:
-      source: mbox2eml/
+      source: components/mbox2eml/
   android-backup:
     description: Backup Android device using ADB
     install:
       path:
         alias: android-backup
     tool:
-      source: android-backup/
+      source: components/android-backup/
       system_dependencies:
       - adb
   browser:
@@ -178,14 +178,14 @@ components:
       path:
         alias: browser
     tool:
-      source: browser/
+      source: components/browser/
   docx-extractor:
     description: Extract content and metadata from Word .docx files
     install:
       path:
         alias: docx-extractor
     tool:
-      source: docx-extractor/
+      source: components/docx-extractor/
       system_dependencies:
       - pandoc
   docx2md:
@@ -194,7 +194,7 @@ components:
       path:
         alias: docx2md
     tool:
-      source: docx2md/
+      source: components/docx2md/
       system_dependencies:
       - pandoc
   gpt:
@@ -203,21 +203,21 @@ components:
       path:
         alias: gpt
     tool:
-      source: gpt/
+      source: components/gpt/
   html2text:
     description: Convert HTML content to plain text
     install:
       path:
         alias: html2text
     tool:
-      source: html2text/
+      source: components/html2text/
   md2pdf:
     description: Convert Markdown files to PDF
     install:
       path:
         alias: md2pdf
     tool:
-      source: md2pdf/
+      source: components/md2pdf/
       system_dependencies:
       - pandoc
       - texlive-latex-base
@@ -227,21 +227,21 @@ components:
       path:
         alias: mdscraper
     tool:
-      source: mdscraper/
+      source: components/mdscraper/
   pdf-extractor:
     description: Extract content and metadata from PDF files
     install:
       path:
         alias: pdf-extractor
     tool:
-      source: pdf-extractor/
+      source: components/pdf-extractor/
   pdf2md:
     description: Convert PDF files to Markdown
     install:
       path:
         alias: pdf2md
     tool:
-      source: pdf2md/
+      source: components/pdf2md/
       system_dependencies:
       - pandoc
       - poppler-utils
@@ -251,18 +251,18 @@ components:
       path:
         alias: schedule
     tool:
-      source: schedule/
+      source: components/schedule/
   search:
     description: Manage self-contained searchable collections of files
     install:
       path:
         alias: search
     tool:
-      source: search/
+      source: components/search/
   text-extractor:
     description: Extract content and metadata from text files
     install:
       path:
         alias: text-extractor
     tool:
-      source: text-extractor/
+      source: components/text-extractor/

cli/src/castle_cli/commands/create.py

@@ -47,9 +47,11 @@ def run_create(args: argparse.Namespace) -> int:
         print(f"Error: component '{name}' already exists in castle.yaml")
         return 1
 
-    project_dir = config.root / name
+    components_dir = config.root / "components"
+    components_dir.mkdir(exist_ok=True)
+    project_dir = components_dir / name
     if project_dir.exists():
-        print(f"Error: directory '{name}' already exists")
+        print(f"Error: directory 'components/{name}' already exists")
         return 1
 
     # Determine port for services
@@ -80,7 +82,7 @@ def run_create(args: argparse.Namespace) -> int:
             run=RunPythonUvTool(
                 runner="python_uv_tool",
                 tool=name,
-                cwd=name,
+                cwd=f"components/{name}",
                 env={f"{env_prefix}_DATA_DIR": f"/data/castle/{name}"},
             ),
             expose=ExposeSpec(
@@ -96,7 +98,7 @@ def run_create(args: argparse.Namespace) -> int:
         manifest = ComponentManifest(
             id=name,
             description=args.description or f"A castle {proj_type}",
-            tool=ToolSpec(source=f"{name}/"),
+            tool=ToolSpec(source=f"components/{name}/"),
             install=InstallSpec(path=PathInstallSpec(alias=name)),
         )
     else:
@@ -114,7 +116,7 @@ def run_create(args: argparse.Namespace) -> int:
         print(f"  Port: {port}")
     print("  Registered in castle.yaml")
     print("\nNext steps:")
-    print(f"  cd {name}")
+    print(f"  cd components/{name}")
     print("  uv sync")
     if proj_type == "service":
         print(f"  uv run {name}  # starts on port {port}")

cli/src/castle_cli/commands/dev.py

@@ -14,8 +14,7 @@ def _get_project_dir(config: CastleConfig, project_name: str) -> Path:
     if project_name not in config.components:
         raise ValueError(f"Unknown component: {project_name}")
     manifest = config.components[project_name]
-    cwd = manifest.run.working_dir if manifest.run else None
+    working_dir = manifest.source_dir or project_name
-    working_dir = cwd or project_name
     return config.root / working_dir
 
 
@@ -59,8 +58,7 @@ def run_test(args: argparse.Namespace) -> int:
     # Run all
     all_passed = True
     for name, manifest in config.components.items():
-        cwd = manifest.run.working_dir if manifest.run else None
+        working_dir = manifest.source_dir or name
-        working_dir = cwd or name
         project_dir = config.root / working_dir
         tests_dir = project_dir / "tests"
         if not tests_dir.exists():
@@ -98,8 +96,7 @@ def run_lint(args: argparse.Namespace) -> int:
     # Run all
     all_passed = True
     for name, manifest in config.components.items():
-        cwd = manifest.run.working_dir if manifest.run else None
+        working_dir = manifest.source_dir or name
-        working_dir = cwd or name
         project_dir = config.root / working_dir
         if not _has_pyproject(project_dir):
             continue

cli/src/castle_cli/commands/info.py

@@ -29,8 +29,7 @@ def run_info(args: argparse.Namespace) -> int:
     if getattr(args, "json", False):
         data = manifest.model_dump(exclude_none=True)
         # Include CLAUDE.md content if it exists
-        cwd = manifest.run.working_dir if manifest.run else None
+        claude_md = _find_claude_md(config.root, manifest.source_dir or name)
-        claude_md = _find_claude_md(config.root, cwd or name)
         if claude_md:
             data["claude_md"] = claude_md
         print(json.dumps(data, indent=2))

cli/src/castle_cli/commands/sync.py

@@ -28,8 +28,7 @@ def run_sync(args: argparse.Namespace) -> int:
     all_ok = True
     synced_dirs: set[Path] = set()
     for name, manifest in config.components.items():
-        cwd = manifest.run.working_dir if manifest.run else None
+        working_dir = manifest.source_dir or name
-        working_dir = cwd or name
         project_dir = config.root / working_dir
         pyproject = project_dir / "pyproject.toml"
 

cli/src/castle_cli/manifest.py

@@ -302,6 +302,21 @@ class ComponentManifest(BaseModel):
 
         return sorted(roles, key=lambda r: r.value)
 
+    @property
+    def source_dir(self) -> str | None:
+        """Best-effort relative directory for this component's source.
+
+        Resolution order: run.working_dir → build.working_dir → tool.source (strip trailing /).
+        Returns None if no directory can be determined.
+        """
+        if self.run and self.run.working_dir:
+            return self.run.working_dir
+        if self.build and self.build.working_dir:
+            return self.build.working_dir
+        if self.tool and self.tool.source:
+            return self.tool.source.rstrip("/")
+        return None
+
     @model_validator(mode="after")
     def _basic_consistency(self) -> ComponentManifest:
         if self.manage and self.manage.systemd and self.manage.systemd.enable:

recommendations.md

@@ -1,187 +0,0 @@
-# Scaling Recommendations
-
-## Decisions made
-
-- **Git structure**: Monorepo with submodules. Castle is a git repo; projects that need independent publishing are their own repos added as submodules. Projects without remotes (e.g., devbox-connect) are tracked directly until they get their own repo.
-- **Scope**: Castle is a personal software platform — not just services, but tools, libraries, and apps. Toolkit (v1, CLI tools only) is being absorbed and generalized.
-- **Gateway**: Caddy reverse proxy + generated dashboard. Single port for all web services. Caddy chosen over Traefik because the service registry is static (no container orchestration), and Caddyfile syntax is trivially simple.
-- **Event bus**: A lightweight castle-component for inter-service communication, so services don't hardcode knowledge of each other.
-- **Independence principle**: Regular services/tools/libraries must never depend on castle. They accept standard configuration (data dir, port, URLs) via env vars or args. Only "castle-components" (CLI, gateway, event bus) know about castle internals like `castle.yaml`. This keeps services portable and independently publishable.
-- **Registry**: `castle.yaml` at the repo root. Centralized — all projects are registered here. `castle create` adds entries automatically. No marker files in projects (would violate independence principle).
-- **Discovery**: Centralized via `castle.yaml`. The CLI reads this file to know what projects exist, their types, and how to orchestrate them.
-- **CLI location**: `cli/` directory at the repo root, installed via `uv tool install`.
-- **Generated files**: `~/.castle/generated/` for Caddyfiles, systemd units, dashboard HTML. Separate from `~/.castle/secrets/`.
-
-## 1. Build the `castle` CLI
-
-The top-level CLI lives in `cli/` and is installed via `uv tool install`. It replaces both toolkit's `toolkit` command and the need for a Makefile/justfile. It should:
-- Discover projects by type (tool, service, library, app) from `castle.yaml`
-- Scaffold new projects from templates: `castle create <name> --type service`
-- Run commands across projects: `castle test`, `castle lint`, `castle sync`
-- Wrap submodule pain points: `castle sync` does `git submodule update --init --recursive`
-- Manage services: `castle service enable/disable/status`, `castle services start/stop`
-- Manage gateway: `castle gateway start/reload`
-- Register `uv tool` entries for tool-type projects
-
-This generalizes toolkit's discovery/scaffolding pattern (YAML frontmatter in markdown, `toolkit create`) across all project types.
-
-## 2. Define project type templates
-
-Each type encodes best practices:
-- **tool**: argparse, stdin/stdout, exit codes, single-purpose (toolkit pattern)
-- **service**: FastAPI, pydantic-settings, lifespan, health endpoint
-- **library**: src/ layout, typed API, no CLI entry point
-- **app**: TBD as needs emerge
-
-Shared patterns (settings base class, error handling, test fixtures) live in templates rather than a shared library — avoids a runtime dependency that couples all projects.
-
-## 3. Standardize project layout
-
-Pick `src/<package_name>/` for all projects. Currently inconsistent: `src/central_context/`, flat `notification_bridge/`, single-file `convert.py`. The castle CLI's discovery and scaffolding depends on predictable structure.
-
-## 4. Registry, gateway, and systemd
-
-`castle.yaml` at the repo root is the single source of truth for all projects — their types, ports, paths, data directories, commands, and inter-service relationships.
-
-The castle CLI generates artifacts from this registry into `~/.castle/generated/`:
-- **Caddyfile** — reverse proxy config so all services are accessible under one port (e.g., `localhost:9000/central-context/*` → `localhost:9001/*`)
-- **Dashboard HTML** — served at the gateway root (`localhost:9000/`) with links to each service, health status, and docs links
-- **Systemd user units** — `.service` files under `~/.config/systemd/user/`
-
-Example `castle.yaml`:
-```yaml
-gateway:
-  port: 9000
-
-projects:
-  # Services (long-running, have ports)
-  central-context:
-    type: service
-    port: 9001
-    path: /central-context
-    command: uv run central-context
-    working_dir: central-context
-    data_dir: /data/castle/central-context
-    description: Content storage API
-    health: /health
-    env:
-      CENTRAL_CONTEXT_DATA_DIR: ${data_dir}
-
-  notification-bridge:
-    type: service
-    port: 9001
-    path: /notifications
-    command: uv run notification-bridge
-    working_dir: notification-bridge
-    data_dir: /data/castle/notification-bridge
-    description: Desktop notification forwarder
-    health: /health
-    publishes:
-      - notification.received
-
-  devbox-connect:
-    type: tool
-    description: SSH tunnel manager with auto-reconnect
-
-  mboxer:
-    type: tool
-    description: MBOX to EML email converter
-
-  # Castle-components
-  event-bus:
-    type: castle-component
-    port: 9010
-    path: /events
-    command: uv run event-bus
-    description: Inter-service event bus
-```
-
-### Gateway commands
-
-```
-castle gateway start    # generate Caddyfile + dashboard, start Caddy
-castle gateway reload   # regenerate after castle.yaml changes
-```
-
-### Systemd commands
-
-```
-castle service enable central-context   # generate unit, enable, start
-castle service disable central-context  # stop and disable
-castle service status                   # show status of all services
-castle services start                   # enable + start everything (including gateway)
-castle services stop                    # stop everything
-```
-
-The gateway itself is also a systemd unit (`castle-gateway.service`), so `castle services start` brings up all services and the Caddy proxy in one command.
-
-Castle resolves `${data_dir}` references, ensures data directories exist, and passes env vars when generating units.
-
-## 5. Agent context strategy
-
-The primary value of castle is that agents can rapidly create and manage software in a standardized way. The conventions must be machine-discoverable.
-
-- **Top-level `CLAUDE.md`** is the agent's entry point into the entire system. It should reference `castle.yaml`, explain project types, link to templates, and describe the agent workflow (scaffold → register → test → enable).
-- **`castle create` updates context automatically** — when a new project is scaffolded, the CLI generates a project-level `CLAUDE.md` from the template and registers the project in `castle.yaml`.
-- **Each project type template includes a `CLAUDE.md` template** so agents immediately understand a project's conventions, build commands, and architecture upon reading it.
-- **The agent workflow is explicit**: an agent creating a new service follows: `castle create` → implement → `castle test` → `castle service enable`. No tribal knowledge required.
-
-## 6. Data persistence conventions
-
-As castle replaces commercial applications, the data these services hold becomes the valuable part. Conventions:
-
-- **Each service's data dir is configured in `castle.yaml`** — defaults to `/data/castle/<service-name>/`. Castle supplies this to the service at launch (via env var or arg). The service itself just accepts a data dir setting — it has no knowledge of castle.
-- **Data directories are never inside submodule trees** — submodules get cloned fresh; persistent data must live outside them.
-- **Backup stays generic** — `backup-collect` remains a general-purpose tool. It doesn't read `castle.yaml`. Castle can separately generate a backup manifest from the registry if needed, but that's a castle concern, not a backup-collect concern.
-
-## 7. Secret management
-
-API keys, tokens, and credentials will accumulate as services replace commercial apps. Rules:
-
-- **Secrets live in `~/.castle/secrets/`** — never in project directories (submodules get pushed to GitHub). Agents must be told this explicitly in context.
-- **`castle.yaml` can reference secrets by name** — the castle CLI resolves them when generating systemd units or passing env vars at launch. Services themselves just receive env vars — they don't know where the values came from.
-
-## 8. Event bus
-
-A castle-component that decouples inter-service communication. Currently notification-bridge hardcodes central-context's URL — this won't scale to dozens of services that need to react to each other's events.
-
-The event bus is a FastAPI service registered in `castle.yaml`:
-- Services **publish** typed events: `POST /events/publish` with `{topic, payload}`
-- Services **subscribe** to topics: register a webhook callback in `castle.yaml` or via `POST /events/subscribe`
-- The bus delivers events to subscribers via HTTP POST to their registered endpoints
-
-This keeps services decoupled — a service only knows about the bus, not about other services. Example flow: notification-bridge publishes a `notification.received` event, and any service that cares subscribes to that topic.
-
-Subscriptions are declared in `castle.yaml` (see example above). The castle CLI configures the bus with the subscription table at startup.
-
-The bus should be simple — no persistence, no guaranteed delivery, no complex routing. Just HTTP fan-out. Add durability later only if needed.
-
-## 9. Consistent ruff/pyright configuration
-
-Each project has its own ruff rules (devbox-connect: `E,F,I,W`; mboxer: `ALL`). Put a shared `ruff.toml` at the repo root (ruff walks up to find config). Same for pyright — only devbox-connect has it currently.
-
-## 10. Absorb toolkit
-
-Don't move toolkit in as a monolith. Instead:
-1. Add toolkit as a submodule
-2. Graduate all tools into independent castle projects (each tool is its own standalone package)
-3. Promote toolkit's meta-tooling up into the castle CLI
-
-## 11. What to defer
-
-- **Shared runtime library (`castle-core`)** — templates are better than a runtime dependency for now. Revisit if projects start importing shared code at runtime.
-- **Containerization/orchestration** — until deploying beyond the local machine
-- **API gateway / service mesh** — Caddy handles reverse proxying; a full mesh is premature
-- **Distributed tracing / observability** — add when debugging cross-service issues becomes painful
-- **Formal API schema sharing** — FastAPI generates OpenAPI already; formalize when consumers need stability guarantees
-
-## Priority
-
-1. **Castle CLI** with project discovery and scaffolding
-2. **Registry** (`castle.yaml`) + **Caddy gateway** + **systemd integration**
-3. **Agent context strategy** — CLAUDE.md generation in templates, agent workflow docs
-4. **Data persistence conventions** + **secret management**
-5. **Standardize layout** across existing projects
-6. **Root-level ruff/pyright config**
-7. **Event bus** castle-component
-8. **Absorb toolkit** incrementally