feat: Add new tools and configurations for Castle platform
- Introduced `uv.lock` for dependency management with various packages including `pytest`, `colorama`, and `pluggy`. - Added `pyrightconfig.json` for Python type checking configuration. - Expanded `recommendations.md` with detailed scaling recommendations and project management strategies. - Created shared `ruff.toml` for consistent linting across projects. - Developed `Castle Tools` with various utilities including Android backup, browser automation, document conversion, and search tools. - Implemented `backup-collect` and `schedule` tools for system administration tasks. - Enhanced `search` functionality with indexing and querying capabilities using Tantivy. - Added comprehensive documentation for each tool, including usage examples and installation instructions.
This commit is contained in:
@@ -1,46 +1,188 @@
|
||||
# Scaling Recommendations
|
||||
|
||||
## 1. Extract a shared library
|
||||
## Decisions made
|
||||
|
||||
The same patterns are already duplicated in central-context and notification-bridge: `BaseSettings` with `.env`, FastAPI lifespan boilerplate, uvicorn entry points, error-to-HTTP-exception translation, test fixtures with temp dirs and settings overrides. At dozens of services this becomes a maintenance problem — fix a bug in the pattern and you're patching it everywhere.
|
||||
- **Git structure**: Monorepo with submodules. Castle is a git repo; projects that need independent publishing are their own repos added as submodules. Projects without remotes (e.g., devbox-connect) are tracked directly until they get their own repo.
|
||||
- **Scope**: Castle is a personal software platform — not just services, but tools, libraries, and apps. Toolkit (v1, CLI tools only) is being absorbed and generalized.
|
||||
- **Gateway**: Caddy reverse proxy + generated dashboard. Single port for all web services. Caddy chosen over Traefik because the service registry is static (no container orchestration), and Caddyfile syntax is trivially simple.
|
||||
- **Event bus**: A lightweight castle-component for inter-service communication, so services don't hardcode knowledge of each other.
|
||||
- **Independence principle**: Regular services/tools/libraries must never depend on castle. They accept standard configuration (data dir, port, URLs) via env vars or args. Only "castle-components" (CLI, gateway, event bus) know about castle internals like `castle.yaml`. This keeps services portable and independently publishable.
|
||||
- **Registry**: `castle.yaml` at the repo root. Centralized — all projects are registered here. `castle create` adds entries automatically. No marker files in projects (would violate independence principle).
|
||||
- **Discovery**: Centralized via `castle.yaml`. The CLI reads this file to know what projects exist, their types, and how to orchestrate them.
|
||||
- **CLI location**: `cli/` directory at the repo root, installed via `uv tool install`.
|
||||
- **Generated files**: `~/.castle/generated/` for Caddyfiles, systemd units, dashboard HTML. Separate from `~/.castle/secrets/`.
|
||||
|
||||
A `castle-core` (or similar) package that provides a base settings class, standard lifespan wiring, common test fixtures, and health check endpoint would let new services start from ~10 lines of setup code.
|
||||
## 1. Build the `castle` CLI
|
||||
|
||||
## 2. Standardize project layout
|
||||
The top-level CLI lives in `cli/` and is installed via `uv tool install`. It replaces both toolkit's `toolkit` command and the need for a Makefile/justfile. It should:
|
||||
- Discover projects by type (tool, service, library, app) from `castle.yaml`
|
||||
- Scaffold new projects from templates: `castle create <name> --type service`
|
||||
- Run commands across projects: `castle test`, `castle lint`, `castle sync`
|
||||
- Wrap submodule pain points: `castle sync` does `git submodule update --init --recursive`
|
||||
- Manage services: `castle service enable/disable/status`, `castle services start/stop`
|
||||
- Manage gateway: `castle gateway start/reload`
|
||||
- Register `uv tool` entries for tool-type projects
|
||||
|
||||
Right now there are three different layouts: `src/central_context/`, flat `notification_bridge/`, and single-file `convert.py`. Pick one (the `src/` layout is the most robust) and stick with it. This matters because any top-level tooling that iterates over projects needs predictable structure.
|
||||
This generalizes toolkit's discovery/scaffolding pattern (YAML frontmatter in markdown, `toolkit create`) across all project types.
|
||||
|
||||
## 3. Top-level task runner
|
||||
## 2. Define project type templates
|
||||
|
||||
With dozens of projects, there needs to be a way to run commands across all or some of them. A root `Makefile`, `justfile`, or script that can do things like:
|
||||
- `make test` — run all tests
|
||||
- `make test p=central-context` — run one project's tests
|
||||
- `make lint` — lint everything
|
||||
- `make sync` — `uv sync` in all projects
|
||||
Each type encodes best practices:
|
||||
- **tool**: argparse, stdin/stdout, exit codes, single-purpose (toolkit pattern)
|
||||
- **service**: FastAPI, pydantic-settings, lifespan, health endpoint
|
||||
- **library**: src/ layout, typed API, no CLI entry point
|
||||
- **app**: TBD as needs emerge
|
||||
|
||||
Without this, significant time gets spent just navigating and running repetitive commands.
|
||||
Shared patterns (settings base class, error handling, test fixtures) live in templates rather than a shared library — avoids a runtime dependency that couples all projects.
|
||||
|
||||
## 4. Port and service registry
|
||||
## 3. Standardize project layout
|
||||
|
||||
Ports are currently hardcoded defaults (9000, 9001). With dozens of services, there needs to be a single source of truth for port assignments — even if it's just a `services.yaml` at the repo root that maps service names to ports.
|
||||
Pick `src/<package_name>/` for all projects. Currently inconsistent: `src/central_context/`, flat `notification_bridge/`, single-file `convert.py`. The castle CLI's discovery and scaffolding depends on predictable structure.
|
||||
|
||||
## 5. Inter-service configuration
|
||||
## 4. Registry, gateway, and systemd
|
||||
|
||||
notification-bridge hardcodes `http://localhost:9000` as the central-context URL. This pattern doesn't scale — each new service that talks to another service adds more hardcoded URLs in more `.env` files. Consider either:
|
||||
- A convention like `CASTLE_{SERVICE_NAME}_URL` derived from the registry
|
||||
- A shared config that generates per-service `.env` files
|
||||
`castle.yaml` at the repo root is the single source of truth for all projects — their types, ports, paths, data directories, commands, and inter-service relationships.
|
||||
|
||||
## 6. Consistent ruff/pyright configuration
|
||||
The castle CLI generates artifacts from this registry into `~/.castle/generated/`:
|
||||
- **Caddyfile** — reverse proxy config so all services are accessible under one port (e.g., `localhost:9000/central-context/*` → `localhost:9001/*`)
|
||||
- **Dashboard HTML** — served at the gateway root (`localhost:9000/`) with links to each service, health status, and docs links
|
||||
- **Systemd user units** — `.service` files under `~/.config/systemd/user/`
|
||||
|
||||
Each project currently has its own ruff rules (devbox-connect selects `E,F,I,W` while mboxer selects `ALL`). With dozens of projects, either put a shared `ruff.toml` at the repo root (ruff walks up to find config) or decide on one standard. Same for pyright — only devbox-connect has it enabled currently.
|
||||
Example `castle.yaml`:
|
||||
```yaml
|
||||
gateway:
|
||||
port: 9000
|
||||
|
||||
## 7. What to defer
|
||||
projects:
|
||||
# Services (long-running, have ports)
|
||||
central-context:
|
||||
type: service
|
||||
port: 9001
|
||||
path: /central-context
|
||||
command: uv run central-context
|
||||
working_dir: central-context
|
||||
data_dir: /data/castle/central-context
|
||||
description: Content storage API
|
||||
health: /health
|
||||
env:
|
||||
CENTRAL_CONTEXT_DATA_DIR: ${data_dir}
|
||||
|
||||
- **Containerization/orchestration** — until deploying somewhere beyond the local machine
|
||||
- **API gateway / service mesh** — premature until there are actual traffic patterns
|
||||
notification-bridge:
|
||||
type: service
|
||||
port: 9001
|
||||
path: /notifications
|
||||
command: uv run notification-bridge
|
||||
working_dir: notification-bridge
|
||||
data_dir: /data/castle/notification-bridge
|
||||
description: Desktop notification forwarder
|
||||
health: /health
|
||||
publishes:
|
||||
- notification.received
|
||||
|
||||
devbox-connect:
|
||||
type: tool
|
||||
description: SSH tunnel manager with auto-reconnect
|
||||
|
||||
mboxer:
|
||||
type: tool
|
||||
description: MBOX to EML email converter
|
||||
|
||||
# Castle-components
|
||||
event-bus:
|
||||
type: castle-component
|
||||
port: 9010
|
||||
path: /events
|
||||
command: uv run event-bus
|
||||
description: Inter-service event bus
|
||||
```
|
||||
|
||||
### Gateway commands
|
||||
|
||||
```
|
||||
castle gateway start # generate Caddyfile + dashboard, start Caddy
|
||||
castle gateway reload # regenerate after castle.yaml changes
|
||||
```
|
||||
|
||||
### Systemd commands
|
||||
|
||||
```
|
||||
castle service enable central-context # generate unit, enable, start
|
||||
castle service disable central-context # stop and disable
|
||||
castle service status # show status of all services
|
||||
castle services start # enable + start everything (including gateway)
|
||||
castle services stop # stop everything
|
||||
```
|
||||
|
||||
The gateway itself is also a systemd unit (`castle-gateway.service`), so `castle services start` brings up all services and the Caddy proxy in one command.
|
||||
|
||||
Castle resolves `${data_dir}` references, ensures data directories exist, and passes env vars when generating units.
|
||||
|
||||
## 5. Agent context strategy
|
||||
|
||||
The primary value of castle is that agents can rapidly create and manage software in a standardized way. The conventions must be machine-discoverable.
|
||||
|
||||
- **Top-level `CLAUDE.md`** is the agent's entry point into the entire system. It should reference `castle.yaml`, explain project types, link to templates, and describe the agent workflow (scaffold → register → test → enable).
|
||||
- **`castle create` updates context automatically** — when a new project is scaffolded, the CLI generates a project-level `CLAUDE.md` from the template and registers the project in `castle.yaml`.
|
||||
- **Each project type template includes a `CLAUDE.md` template** so agents immediately understand a project's conventions, build commands, and architecture upon reading it.
|
||||
- **The agent workflow is explicit**: an agent creating a new service follows: `castle create` → implement → `castle test` → `castle service enable`. No tribal knowledge required.
|
||||
|
||||
## 6. Data persistence conventions
|
||||
|
||||
As castle replaces commercial applications, the data these services hold becomes the valuable part. Conventions:
|
||||
|
||||
- **Each service's data dir is configured in `castle.yaml`** — defaults to `/data/castle/<service-name>/`. Castle supplies this to the service at launch (via env var or arg). The service itself just accepts a data dir setting — it has no knowledge of castle.
|
||||
- **Data directories are never inside submodule trees** — submodules get cloned fresh; persistent data must live outside them.
|
||||
- **Backup stays generic** — `backup-collect` remains a general-purpose tool. It doesn't read `castle.yaml`. Castle can separately generate a backup manifest from the registry if needed, but that's a castle concern, not a backup-collect concern.
|
||||
|
||||
## 7. Secret management
|
||||
|
||||
API keys, tokens, and credentials will accumulate as services replace commercial apps. Rules:
|
||||
|
||||
- **Secrets live in `~/.castle/secrets/`** — never in project directories (submodules get pushed to GitHub). Agents must be told this explicitly in context.
|
||||
- **`castle.yaml` can reference secrets by name** — the castle CLI resolves them when generating systemd units or passing env vars at launch. Services themselves just receive env vars — they don't know where the values came from.
|
||||
|
||||
## 8. Event bus
|
||||
|
||||
A castle-component that decouples inter-service communication. Currently notification-bridge hardcodes central-context's URL — this won't scale to dozens of services that need to react to each other's events.
|
||||
|
||||
The event bus is a FastAPI service registered in `castle.yaml`:
|
||||
- Services **publish** typed events: `POST /events/publish` with `{topic, payload}`
|
||||
- Services **subscribe** to topics: register a webhook callback in `castle.yaml` or via `POST /events/subscribe`
|
||||
- The bus delivers events to subscribers via HTTP POST to their registered endpoints
|
||||
|
||||
This keeps services decoupled — a service only knows about the bus, not about other services. Example flow: notification-bridge publishes a `notification.received` event, and any service that cares subscribes to that topic.
|
||||
|
||||
Subscriptions are declared in `castle.yaml` (see example above). The castle CLI configures the bus with the subscription table at startup.
|
||||
|
||||
The bus should be simple — no persistence, no guaranteed delivery, no complex routing. Just HTTP fan-out. Add durability later only if needed.
|
||||
|
||||
## 9. Consistent ruff/pyright configuration
|
||||
|
||||
Each project has its own ruff rules (devbox-connect: `E,F,I,W`; mboxer: `ALL`). Put a shared `ruff.toml` at the repo root (ruff walks up to find config). Same for pyright — only devbox-connect has it currently.
|
||||
|
||||
## 10. Absorb toolkit
|
||||
|
||||
Don't move toolkit in as a monolith. Instead:
|
||||
1. Add toolkit as a submodule
|
||||
2. Graduate heavy tools (`search`, `protonmail`, `browser`) into independent castle projects
|
||||
3. Keep lightweight tools (`docx2md`, `html2text`, etc.) grouped in a single `tools` package
|
||||
4. Promote toolkit's meta-tooling up into the castle CLI
|
||||
|
||||
## 11. What to defer
|
||||
|
||||
- **Shared runtime library (`castle-core`)** — templates are better than a runtime dependency for now. Revisit if projects start importing shared code at runtime.
|
||||
- **Containerization/orchestration** — until deploying beyond the local machine
|
||||
- **API gateway / service mesh** — Caddy handles reverse proxying; a full mesh is premature
|
||||
- **Distributed tracing / observability** — add when debugging cross-service issues becomes painful
|
||||
- **Formal API schema sharing** (OpenAPI contracts between services) — FastAPI generates these already; formalize when there are consumers that need stability guarantees
|
||||
- **Formal API schema sharing** — FastAPI generates OpenAPI already; formalize when consumers need stability guarantees
|
||||
|
||||
## Priority
|
||||
|
||||
The highest-leverage first step is the shared library + top-level task runner, since those reduce the marginal cost of adding each new service.
|
||||
1. **Castle CLI** with project discovery and scaffolding
|
||||
2. **Registry** (`castle.yaml`) + **Caddy gateway** + **systemd integration**
|
||||
3. **Agent context strategy** — CLAUDE.md generation in templates, agent workflow docs
|
||||
4. **Data persistence conventions** + **secret management**
|
||||
5. **Standardize layout** across existing projects
|
||||
6. **Root-level ruff/pyright config**
|
||||
7. **Event bus** castle-component
|
||||
8. **Absorb toolkit** incrementally
|
||||
|
||||
Reference in New Issue
Block a user