feat: Add new tools and configurations for Castle platform
- Introduced `uv.lock` for dependency management with various packages including `pytest`, `colorama`, and `pluggy`. - Added `pyrightconfig.json` for Python type checking configuration. - Expanded `recommendations.md` with detailed scaling recommendations and project management strategies. - Created shared `ruff.toml` for consistent linting across projects. - Developed `Castle Tools` with various utilities including Android backup, browser automation, document conversion, and search tools. - Implemented `backup-collect` and `schedule` tools for system administration tasks. - Enhanced `search` functionality with indexing and querying capabilities using Tantivy. - Added comprehensive documentation for each tool, including usage examples and installation instructions.
This commit is contained in:
419
docs/python-tools.md
Normal file
419
docs/python-tools.md
Normal file
@@ -0,0 +1,419 @@
|
||||
# Python Tools in Castle
|
||||
|
||||
How to build CLI tools following Unix philosophy. Based on the patterns in the
|
||||
[toolkit](https://github.com/payneio/toolkit) project.
|
||||
|
||||
## Principles
|
||||
|
||||
- Each tool does one thing well
|
||||
- Read from stdin or file argument, write to stdout
|
||||
- Compose via pipes: `pdf2md doc.pdf | gpt "summarize this"`
|
||||
- Status messages go to stderr (don't interfere with piping)
|
||||
- Exit 0 on success, 1 on error
|
||||
|
||||
## Stack
|
||||
|
||||
| Layer | Choice |
|
||||
|-------|--------|
|
||||
| **CLI** | argparse |
|
||||
| **Package manager** | uv (never pip) |
|
||||
| **Build** | hatchling |
|
||||
| **Testing** | unittest + mocking |
|
||||
| **Linting** | ruff |
|
||||
| **Type checking** | pyright |
|
||||
|
||||
## Project layout
|
||||
|
||||
Tools live inside a single package with categories:
|
||||
|
||||
```
|
||||
toolkit/
|
||||
├── tools/
|
||||
│ ├── document/
|
||||
│ │ ├── pdf2md.py # Implementation
|
||||
│ │ ├── pdf2md.md # Docs + YAML frontmatter (single source of truth)
|
||||
│ │ └── test_pdf2md.py # Tests alongside tool
|
||||
│ ├── search/
|
||||
│ │ ├── search.py
|
||||
│ │ ├── search.md
|
||||
│ │ └── test_search.py
|
||||
│ ├── system/
|
||||
│ │ ├── schedule.py
|
||||
│ │ └── schedule.md
|
||||
│ └── toolkit/
|
||||
│ ├── toolkit.py # Meta-tool for discovery/scaffolding
|
||||
│ └── toolkit.md
|
||||
├── pyproject.toml
|
||||
├── Makefile
|
||||
└── README.md
|
||||
```
|
||||
|
||||
Each tool is a `.py` + `.md` pair. The `.md` file has YAML frontmatter for
|
||||
metadata — no separate config files needed.
|
||||
|
||||
## YAML frontmatter (.md file)
|
||||
|
||||
```yaml
|
||||
---
|
||||
command: pdf2md
|
||||
script: document/pdf2md.py
|
||||
description: Convert PDF files to Markdown
|
||||
version: 1.0.0
|
||||
category: document
|
||||
system_dependencies:
|
||||
- pandoc
|
||||
- poppler-utils
|
||||
---
|
||||
|
||||
# pdf2md
|
||||
|
||||
Converts PDF files to Markdown format...
|
||||
```
|
||||
|
||||
The toolkit management command discovers tools by scanning for these `.md` files.
|
||||
|
||||
## pyproject.toml
|
||||
|
||||
```toml
|
||||
[project]
|
||||
name = "toolkit"
|
||||
version = "0.1.0"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"requests>=2.28.0",
|
||||
"pyyaml>=6.0.0",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
pdf2md = "tools.document.pdf2md:main"
|
||||
docx2md = "tools.document.docx2md:main"
|
||||
search = "tools.search.search:main"
|
||||
toolkit = "tools.toolkit.toolkit:main"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["tools"]
|
||||
```
|
||||
|
||||
Entry points follow `command = "tools.<category>.<tool>:main"`. After
|
||||
`uv tool install --editable .`, all commands are in PATH.
|
||||
|
||||
## Tool implementation patterns
|
||||
|
||||
### Simple tool: stdin/stdout
|
||||
|
||||
The most common pattern. Read from a file argument or stdin, process, write
|
||||
to stdout.
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
my-tool: Brief description
|
||||
|
||||
Detailed usage docs here.
|
||||
|
||||
Usage:
|
||||
my-tool [options] [FILE]
|
||||
cat input.txt | my-tool
|
||||
|
||||
Examples:
|
||||
my-tool input.txt
|
||||
my-tool input.txt -o output.txt
|
||||
cat input.txt | my-tool > output.txt
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
|
||||
|
||||
def process(data: str) -> str:
|
||||
"""Core logic — pure function, easy to test."""
|
||||
return data.upper()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Brief description",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
parser.add_argument("input", nargs="?", help="Input file (default: stdin)")
|
||||
parser.add_argument("-o", "--output", help="Output file (default: stdout)")
|
||||
parser.add_argument("--version", action="version", version="my-tool 1.0.0")
|
||||
args = parser.parse_args()
|
||||
|
||||
# Read
|
||||
if args.input:
|
||||
with open(args.input) as f:
|
||||
data = f.read()
|
||||
else:
|
||||
data = sys.stdin.read()
|
||||
|
||||
# Process
|
||||
result = process(data)
|
||||
|
||||
# Write
|
||||
if args.output:
|
||||
with open(args.output, "w") as f:
|
||||
f.write(result)
|
||||
print(f"Wrote to {args.output}", file=sys.stderr)
|
||||
else:
|
||||
print(result, end="")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
```
|
||||
|
||||
### Tool with subcommands
|
||||
|
||||
For complex tools with multiple operations:
|
||||
|
||||
```python
|
||||
def cmd_init(args: argparse.Namespace) -> int:
|
||||
"""Initialize a collection."""
|
||||
directory = args.directory or "."
|
||||
# ...
|
||||
print(f"Initialized in {directory}")
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_query(args: argparse.Namespace) -> int:
|
||||
"""Search a collection."""
|
||||
# ...
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Manage collections")
|
||||
parser.add_argument("--version", action="version", version="1.0.0")
|
||||
subparsers = parser.add_subparsers(dest="command")
|
||||
|
||||
init_p = subparsers.add_parser("init", help="Initialize")
|
||||
init_p.add_argument("directory", nargs="?")
|
||||
init_p.add_argument("--name", help="Collection name")
|
||||
init_p.add_argument("--force", action="store_true")
|
||||
|
||||
query_p = subparsers.add_parser("query", help="Search")
|
||||
query_p.add_argument("query", help="Search query")
|
||||
query_p.add_argument("--limit", type=int, default=20)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.command == "init":
|
||||
return cmd_init(args)
|
||||
elif args.command == "query":
|
||||
return cmd_query(args)
|
||||
else:
|
||||
parser.print_help()
|
||||
return 1
|
||||
```
|
||||
|
||||
### Tool with external processes
|
||||
|
||||
When wrapping system commands:
|
||||
|
||||
```python
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
def convert(input_file: str, output_file: str) -> int:
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["pandoc", input_file, "-o", output_file],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
return 0
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"Error: {e}", file=sys.stderr)
|
||||
if e.stderr:
|
||||
print(e.stderr, file=sys.stderr)
|
||||
return 1
|
||||
except FileNotFoundError:
|
||||
print("Error: pandoc not found. Install with: apt install pandoc",
|
||||
file=sys.stderr)
|
||||
return 1
|
||||
```
|
||||
|
||||
Always use `check=True` and `capture_output=True` with subprocess. Handle
|
||||
`FileNotFoundError` for missing system dependencies.
|
||||
|
||||
### Tool with optional dependencies
|
||||
|
||||
```python
|
||||
tantivy_available = False
|
||||
try:
|
||||
import tantivy
|
||||
tantivy_available = True
|
||||
except ImportError:
|
||||
tantivy = None
|
||||
|
||||
|
||||
def check_tantivy() -> None:
|
||||
if not tantivy_available:
|
||||
print("Error: tantivy not found. Install with: uv add tantivy",
|
||||
file=sys.stderr)
|
||||
sys.exit(1)
|
||||
```
|
||||
|
||||
## Error handling
|
||||
|
||||
```python
|
||||
def main() -> int:
|
||||
try:
|
||||
result = do_work()
|
||||
print(result)
|
||||
return 0
|
||||
except SpecificError as e:
|
||||
print(f"Error: {e}", file=sys.stderr)
|
||||
return 1
|
||||
except FileNotFoundError as e:
|
||||
print(f"Error: file not found: {e}", file=sys.stderr)
|
||||
return 1
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Normal output goes to **stdout** (enables piping)
|
||||
- Error messages go to **stderr**
|
||||
- Status/progress messages go to **stderr**
|
||||
- Return **0** for success, **1** for error
|
||||
- Entry point: `sys.exit(main())`
|
||||
|
||||
## Piping
|
||||
|
||||
Tools compose naturally via Unix pipes:
|
||||
|
||||
```bash
|
||||
# Convert and summarize
|
||||
pdf2md paper.pdf | gpt "summarize this"
|
||||
|
||||
# Process a batch
|
||||
for f in *.pdf; do pdf2md "$f" > "${f%.pdf}.md"; done
|
||||
|
||||
# Chain extractors
|
||||
cat doc.txt | text-extractor | jq .content
|
||||
```
|
||||
|
||||
When a tool writes to both a file and stdout, status messages must go to
|
||||
stderr so they don't contaminate the pipe:
|
||||
|
||||
```python
|
||||
with open(output_file, "w") as f:
|
||||
f.write(result)
|
||||
print(result) # stdout (for piping)
|
||||
print(f"Wrote to {output_file}", file=sys.stderr) # stderr (status)
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
Tests live alongside the tool implementation, using unittest with mocking:
|
||||
|
||||
```python
|
||||
# tools/gpt/test_gpt.py
|
||||
import unittest
|
||||
from unittest.mock import patch, MagicMock
|
||||
from io import StringIO
|
||||
|
||||
from tools.gpt import gpt
|
||||
|
||||
|
||||
class TestGPT(unittest.TestCase):
|
||||
@patch("tools.gpt.gpt.get_api_key")
|
||||
@patch("openai.OpenAI")
|
||||
def test_generate(self, mock_openai, mock_key):
|
||||
mock_key.return_value = "fake-key"
|
||||
mock_client = MagicMock()
|
||||
mock_openai.return_value = mock_client
|
||||
mock_client.chat.completions.create.return_value = MagicMock(
|
||||
choices=[MagicMock(message=MagicMock(content="response"))]
|
||||
)
|
||||
|
||||
result = gpt.generate_text("prompt", "gpt-4", 0.7, 500)
|
||||
self.assertEqual(result, "response")
|
||||
|
||||
@patch("tools.gpt.gpt.generate_text")
|
||||
def test_cli(self, mock_gen):
|
||||
mock_gen.return_value = "output"
|
||||
with patch("sys.argv", ["gpt", "prompt"]):
|
||||
with patch("sys.stdout", new=StringIO()) as out:
|
||||
gpt.main()
|
||||
self.assertEqual(out.getvalue().strip(), "output")
|
||||
```
|
||||
|
||||
Pattern: mock external dependencies (APIs, file I/O, subprocesses), test the
|
||||
CLI by patching `sys.argv`.
|
||||
|
||||
## Build and install
|
||||
|
||||
```makefile
|
||||
# Makefile
|
||||
all: build install
|
||||
|
||||
build:
|
||||
uv sync
|
||||
|
||||
install:
|
||||
uv tool install --editable .
|
||||
|
||||
check:
|
||||
uv run ruff format .
|
||||
uv run ruff check . --fix
|
||||
uv run pyright
|
||||
|
||||
test:
|
||||
uv run pytest -v
|
||||
|
||||
clean:
|
||||
find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
|
||||
```
|
||||
|
||||
```bash
|
||||
make all # Sync deps + install to PATH
|
||||
make check # Format, lint, type-check
|
||||
make test # Run tests
|
||||
```
|
||||
|
||||
## Creating a new tool
|
||||
|
||||
Use the toolkit management command:
|
||||
|
||||
```bash
|
||||
toolkit create my-tool --description "Does something" --category document
|
||||
```
|
||||
|
||||
This creates the `.py` template, `.md` with frontmatter, and updates
|
||||
`pyproject.toml` with the entry point.
|
||||
|
||||
Or manually:
|
||||
1. Create `tools/<category>/my_tool.py` with the argparse pattern above
|
||||
2. Create `tools/<category>/my_tool.md` with YAML frontmatter
|
||||
3. Add entry to `pyproject.toml` under `[project.scripts]`:
|
||||
```toml
|
||||
my-tool = "tools.category.my_tool:main"
|
||||
```
|
||||
4. Run `make install` to register in PATH
|
||||
|
||||
## Registering in castle
|
||||
|
||||
```yaml
|
||||
# castle.yaml
|
||||
components:
|
||||
toolkit:
|
||||
description: Personal utility scripts
|
||||
run:
|
||||
runner: command
|
||||
argv: ["toolkit"]
|
||||
cwd: toolkit
|
||||
install:
|
||||
path: { alias: toolkit }
|
||||
```
|
||||
|
||||
Tools with `install.path` get the `tool` role. They don't need `expose`,
|
||||
`proxy`, or `manage` blocks.
|
||||
424
docs/web-apis.md
Normal file
424
docs/web-apis.md
Normal file
@@ -0,0 +1,424 @@
|
||||
# Web APIs in Castle
|
||||
|
||||
How to build Python web APIs as castle service components. Based on the
|
||||
patterns used in [wild-cloud/api](https://github.com/civilsociety-dev/wild-cloud)
|
||||
and existing castle services (central-context, notification-bridge, event-bus).
|
||||
|
||||
## Stack
|
||||
|
||||
| Layer | Choice |
|
||||
|-------|--------|
|
||||
| **Framework** | FastAPI |
|
||||
| **Server** | uvicorn |
|
||||
| **Config** | pydantic-settings (env vars) |
|
||||
| **Validation** | Pydantic models |
|
||||
| **HTTP client** | httpx (async) |
|
||||
| **Testing** | pytest + FastAPI TestClient |
|
||||
| **Python** | 3.13+ for services |
|
||||
|
||||
## Project layout
|
||||
|
||||
```
|
||||
my-service/
|
||||
├── src/my_service/
|
||||
│ ├── __init__.py # Package version
|
||||
│ ├── main.py # FastAPI app, lifespan, entry point
|
||||
│ ├── config.py # pydantic-settings
|
||||
│ ├── models.py # Request/response Pydantic models
|
||||
│ ├── routes.py # APIRouter with endpoints
|
||||
│ └── storage.py # Domain logic (no FastAPI imports)
|
||||
├── tests/
|
||||
│ ├── conftest.py # Fixtures (client, temp dirs)
|
||||
│ ├── test_api.py # Endpoint integration tests
|
||||
│ └── test_storage.py # Domain unit tests
|
||||
├── pyproject.toml
|
||||
└── CLAUDE.md
|
||||
```
|
||||
|
||||
Separation of concerns: routes handle HTTP, storage/core handles logic,
|
||||
models define schemas. Domain code never imports FastAPI.
|
||||
|
||||
## pyproject.toml
|
||||
|
||||
```toml
|
||||
[project]
|
||||
name = "my-service"
|
||||
version = "0.1.0"
|
||||
description = "Does something useful"
|
||||
requires-python = ">=3.13"
|
||||
dependencies = [
|
||||
"fastapi>=0.115.0",
|
||||
"uvicorn>=0.34.0",
|
||||
"pydantic-settings>=2.0.0",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
my-service = "my_service.main:run"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/my_service"]
|
||||
|
||||
[dependency-groups]
|
||||
dev = [
|
||||
"pytest>=8.0.0",
|
||||
"httpx>=0.28.0",
|
||||
]
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Use pydantic-settings with an env prefix matching the service name:
|
||||
|
||||
```python
|
||||
# config.py
|
||||
from pathlib import Path
|
||||
from pydantic_settings import BaseSettings
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
data_dir: Path = Path("./data")
|
||||
host: str = "0.0.0.0"
|
||||
port: int = 9001
|
||||
|
||||
model_config = {
|
||||
"env_prefix": "MY_SERVICE_",
|
||||
"env_file": ".env",
|
||||
}
|
||||
|
||||
def ensure_data_dir(self) -> None:
|
||||
self.data_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
settings = Settings()
|
||||
```
|
||||
|
||||
Castle passes config via env vars in castle.yaml:
|
||||
|
||||
```yaml
|
||||
components:
|
||||
my-service:
|
||||
run:
|
||||
runner: python_uv_tool
|
||||
tool: my-service
|
||||
cwd: my-service
|
||||
env:
|
||||
MY_SERVICE_DATA_DIR: /data/castle/my-service
|
||||
expose:
|
||||
http:
|
||||
internal: { port: 9001 }
|
||||
health_path: /health
|
||||
proxy:
|
||||
caddy: { path_prefix: /my-service }
|
||||
manage:
|
||||
systemd: {}
|
||||
```
|
||||
|
||||
## Application entry point
|
||||
|
||||
```python
|
||||
# main.py
|
||||
from collections.abc import AsyncGenerator
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
import uvicorn
|
||||
from fastapi import FastAPI
|
||||
|
||||
from my_service.config import settings
|
||||
from my_service.routes import router
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
settings.ensure_data_dir()
|
||||
yield
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title="my-service",
|
||||
description="Does something useful",
|
||||
version="0.1.0",
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
app.include_router(router)
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
def health() -> dict[str, str]:
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
def run() -> None:
|
||||
uvicorn.run(
|
||||
"my_service.main:app",
|
||||
host=settings.host,
|
||||
port=settings.port,
|
||||
reload=False,
|
||||
)
|
||||
```
|
||||
|
||||
For services with async resources (HTTP clients, connections), initialize
|
||||
them in the lifespan and clean up after yield:
|
||||
|
||||
```python
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
settings.ensure_data_dir()
|
||||
async with httpx.AsyncClient(timeout=10.0) as client:
|
||||
app.state.http_client = client
|
||||
yield
|
||||
```
|
||||
|
||||
## Routes
|
||||
|
||||
Use APIRouter with a prefix and tags. Map domain exceptions to HTTP status codes.
|
||||
|
||||
```python
|
||||
# routes.py
|
||||
from fastapi import APIRouter, HTTPException, status
|
||||
|
||||
from my_service.models import ItemCreate, ItemResponse
|
||||
from my_service.storage import (
|
||||
ItemExistsError,
|
||||
ItemNotFoundError,
|
||||
create_item,
|
||||
get_item,
|
||||
list_items,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/items", tags=["items"])
|
||||
|
||||
|
||||
@router.post("", response_model=ItemResponse, status_code=status.HTTP_201_CREATED)
|
||||
def create(request: ItemCreate) -> ItemResponse:
|
||||
try:
|
||||
return create_item(request)
|
||||
except ItemExistsError as e:
|
||||
raise HTTPException(status_code=status.HTTP_409_CONFLICT, detail=str(e))
|
||||
|
||||
|
||||
@router.get("/{item_id}", response_model=ItemResponse)
|
||||
def get(item_id: str) -> ItemResponse:
|
||||
try:
|
||||
return get_item(item_id)
|
||||
except ItemNotFoundError:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Not found")
|
||||
|
||||
|
||||
@router.get("", response_model=list[ItemResponse])
|
||||
def list_all() -> list[ItemResponse]:
|
||||
return list_items()
|
||||
```
|
||||
|
||||
## Request/response models
|
||||
|
||||
Separate create models (what the client sends) from response models (what
|
||||
comes back). Use inheritance to avoid repetition.
|
||||
|
||||
```python
|
||||
# models.py
|
||||
from datetime import datetime
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class ItemCreate(BaseModel):
|
||||
name: str = Field(..., description="Item name")
|
||||
content: str = Field(..., description="Item content")
|
||||
description: str | None = Field(default=None)
|
||||
|
||||
|
||||
class ItemResponse(BaseModel):
|
||||
name: str
|
||||
description: str | None
|
||||
created_at: datetime
|
||||
size_bytes: int
|
||||
checksum: str
|
||||
|
||||
|
||||
class ItemWithBody(ItemResponse):
|
||||
content: str
|
||||
```
|
||||
|
||||
## Error handling
|
||||
|
||||
Define domain exceptions in the storage/core layer. Map them to HTTP status
|
||||
codes in the route layer.
|
||||
|
||||
```python
|
||||
# storage.py
|
||||
class StorageError(Exception):
|
||||
pass
|
||||
|
||||
class ItemExistsError(StorageError):
|
||||
pass
|
||||
|
||||
class ItemNotFoundError(StorageError):
|
||||
pass
|
||||
|
||||
class InvalidNameError(StorageError):
|
||||
pass
|
||||
```
|
||||
|
||||
Mapping convention:
|
||||
|
||||
| Exception | HTTP Status |
|
||||
|-----------|-------------|
|
||||
| `NotFoundError` | 404 |
|
||||
| `ExistsError` / conflict | 409 |
|
||||
| `InvalidError` / bad input | 400 |
|
||||
| Unexpected | 500 (FastAPI default) |
|
||||
|
||||
## Storage
|
||||
|
||||
Castle services use filesystem storage with JSON metadata sidecars:
|
||||
|
||||
```
|
||||
/data/castle/my-service/
|
||||
└── bucket/
|
||||
├── item-name
|
||||
└── item-name.meta.json
|
||||
```
|
||||
|
||||
```python
|
||||
# storage.py
|
||||
import hashlib
|
||||
import json
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from my_service.config import settings
|
||||
from my_service.models import ItemCreate, ItemResponse
|
||||
|
||||
|
||||
def create_item(request: ItemCreate) -> ItemResponse:
|
||||
checksum = hashlib.sha256(request.content.encode()).hexdigest()
|
||||
path = settings.data_dir / request.name
|
||||
meta_path = path.with_suffix(path.suffix + ".meta.json")
|
||||
|
||||
if path.exists():
|
||||
raise ItemExistsError(f"'{request.name}' already exists")
|
||||
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
now = datetime.now()
|
||||
|
||||
metadata = ItemResponse(
|
||||
name=request.name,
|
||||
description=request.description,
|
||||
created_at=now,
|
||||
size_bytes=len(request.content.encode()),
|
||||
checksum=checksum,
|
||||
)
|
||||
|
||||
path.write_text(request.content, encoding="utf-8")
|
||||
meta_path.write_text(
|
||||
json.dumps(metadata.model_dump(), default=str, indent=2)
|
||||
)
|
||||
return metadata
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
### Fixtures
|
||||
|
||||
```python
|
||||
# tests/conftest.py
|
||||
from collections.abc import Generator
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from my_service import config
|
||||
from my_service.main import app
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def temp_data_dir(tmp_path: Path) -> Generator[Path, None, None]:
|
||||
data_dir = tmp_path / "data"
|
||||
data_dir.mkdir()
|
||||
original = config.settings.data_dir
|
||||
config.settings.data_dir = data_dir
|
||||
yield data_dir
|
||||
config.settings.data_dir = original
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(temp_data_dir: Path) -> Generator[TestClient, None, None]:
|
||||
with TestClient(app) as client:
|
||||
yield client
|
||||
```
|
||||
|
||||
### Endpoint tests
|
||||
|
||||
```python
|
||||
# tests/test_api.py
|
||||
from fastapi import status
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
|
||||
class TestHealth:
|
||||
def test_health(self, client: TestClient) -> None:
|
||||
response = client.get("/health")
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
assert response.json() == {"status": "ok"}
|
||||
|
||||
|
||||
class TestCreateItem:
|
||||
def test_create(self, client: TestClient) -> None:
|
||||
response = client.post(
|
||||
"/items",
|
||||
json={"name": "test", "content": "hello"},
|
||||
)
|
||||
assert response.status_code == status.HTTP_201_CREATED
|
||||
data = response.json()
|
||||
assert data["name"] == "test"
|
||||
assert "checksum" in data
|
||||
|
||||
def test_duplicate_returns_409(self, client: TestClient) -> None:
|
||||
payload = {"name": "dup", "content": "hello"}
|
||||
client.post("/items", json=payload)
|
||||
response = client.post("/items", json=payload)
|
||||
assert response.status_code == status.HTTP_409_CONFLICT
|
||||
```
|
||||
|
||||
### Domain unit tests
|
||||
|
||||
```python
|
||||
# tests/test_storage.py
|
||||
from pathlib import Path
|
||||
|
||||
from my_service.models import ItemCreate
|
||||
from my_service.storage import create_item
|
||||
|
||||
|
||||
class TestCreateItem:
|
||||
def test_creates_files(self, temp_data_dir: Path) -> None:
|
||||
request = ItemCreate(name="test", content="hello")
|
||||
metadata = create_item(request)
|
||||
|
||||
assert metadata.name == "test"
|
||||
assert (temp_data_dir / "test").exists()
|
||||
assert (temp_data_dir / "test.meta.json").exists()
|
||||
```
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
uv sync # Install deps
|
||||
uv run my-service # Run service
|
||||
uv run pytest tests/ -v # Run tests
|
||||
uv run ruff check . # Lint
|
||||
uv run ruff format . # Format
|
||||
```
|
||||
|
||||
## Scaffolding
|
||||
|
||||
`castle create` generates all of this automatically:
|
||||
|
||||
```bash
|
||||
castle create my-service --type service --description "Does something useful"
|
||||
```
|
||||
334
docs/web-frontends.md
Normal file
334
docs/web-frontends.md
Normal file
@@ -0,0 +1,334 @@
|
||||
# Web Frontends in Castle
|
||||
|
||||
How to build, serve, and manage web frontends as castle components. Based on
|
||||
the stack used in [wild-cloud/web](https://github.com/civilsociety-dev/wild-cloud).
|
||||
|
||||
## Stack
|
||||
|
||||
| Layer | Choice |
|
||||
|-------|--------|
|
||||
| **Build** | Vite 6 |
|
||||
| **Language** | TypeScript 5.8 (strict) |
|
||||
| **Framework** | React 19 |
|
||||
| **Routing** | React Router 7 |
|
||||
| **Styling** | Tailwind CSS 4 (`@tailwindcss/vite` plugin) |
|
||||
| **Components** | shadcn/ui (new-york style) + Radix UI primitives |
|
||||
| **Icons** | Lucide React |
|
||||
| **Server state** | TanStack React Query 5 |
|
||||
| **Forms** | React Hook Form + Zod validation |
|
||||
| **Testing** | Vitest + Testing Library |
|
||||
| **Package manager** | pnpm |
|
||||
|
||||
## Scaffolding a new frontend
|
||||
|
||||
```bash
|
||||
# Create the project
|
||||
mkdir my-frontend && cd my-frontend
|
||||
pnpm create vite . --template react-ts
|
||||
|
||||
# Core dependencies
|
||||
pnpm add react-router react-router-dom \
|
||||
@tanstack/react-query \
|
||||
tailwindcss @tailwindcss/vite \
|
||||
class-variance-authority clsx tailwind-merge \
|
||||
lucide-react sonner zod \
|
||||
react-hook-form @hookform/resolvers
|
||||
|
||||
# Dev dependencies
|
||||
pnpm add -D vitest jsdom @testing-library/react @testing-library/jest-dom \
|
||||
@testing-library/user-event
|
||||
|
||||
# shadcn/ui
|
||||
pnpm dlx shadcn@latest init
|
||||
```
|
||||
|
||||
## Vite config
|
||||
|
||||
```ts
|
||||
// vite.config.ts
|
||||
import path from "path"
|
||||
import tailwindcss from "@tailwindcss/vite"
|
||||
import { defineConfig } from "vite"
|
||||
import react from "@vitejs/plugin-react"
|
||||
|
||||
export default defineConfig({
|
||||
plugins: [react(), tailwindcss()],
|
||||
resolve: {
|
||||
alias: {
|
||||
"@": path.resolve(__dirname, "./src"),
|
||||
},
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
## Project layout
|
||||
|
||||
```
|
||||
my-frontend/
|
||||
├── src/
|
||||
│ ├── main.tsx # ReactDOM.createRoot mount
|
||||
│ ├── App.tsx # Root: QueryClientProvider + RouterProvider
|
||||
│ ├── index.css # Tailwind imports + CSS custom properties
|
||||
│ ├── router/
|
||||
│ │ ├── index.tsx # createBrowserRouter
|
||||
│ │ └── routes.tsx # Route tree
|
||||
│ ├── components/
|
||||
│ │ └── ui/ # shadcn/ui components
|
||||
│ ├── services/api/
|
||||
│ │ ├── client.ts # Typed fetch wrapper
|
||||
│ │ └── hooks/ # React Query hooks per resource
|
||||
│ ├── hooks/ # App-level hooks
|
||||
│ ├── contexts/ # React context providers
|
||||
│ ├── lib/
|
||||
│ │ ├── utils.ts # cn() helper
|
||||
│ │ └── queryClient.ts # Query client config
|
||||
│ ├── types/ # Shared TypeScript types
|
||||
│ └── schemas/ # Zod schemas
|
||||
├── public/ # Static assets (favicon, manifest.json)
|
||||
├── index.html # SPA entry point
|
||||
├── package.json
|
||||
├── vite.config.ts
|
||||
├── tsconfig.json
|
||||
├── vitest.config.ts
|
||||
├── components.json # shadcn/ui config
|
||||
└── .env # VITE_API_BASE_URL etc.
|
||||
```
|
||||
|
||||
## Build commands
|
||||
|
||||
```bash
|
||||
pnpm run dev # Vite dev server (:5173), HMR
|
||||
pnpm run build # tsc -b && vite build → dist/
|
||||
pnpm run preview # Serve production build locally
|
||||
pnpm run type-check # tsc --noEmit
|
||||
pnpm run lint # ESLint
|
||||
pnpm run test # Vitest
|
||||
pnpm run check # lint + type-check + test
|
||||
```
|
||||
|
||||
The `build` output is a static SPA in `dist/` — just HTML, JS, and CSS files.
|
||||
|
||||
## Registering as a castle component
|
||||
|
||||
A frontend component has a `build` spec (produces static output) and optionally
|
||||
a `proxy` spec (Caddy serves the built files). No `run` block needed if Caddy
|
||||
handles serving directly from the build output.
|
||||
|
||||
```yaml
|
||||
# castle.yaml
|
||||
components:
|
||||
my-frontend:
|
||||
description: Web dashboard
|
||||
build:
|
||||
commands:
|
||||
- ["pnpm", "build"]
|
||||
outputs:
|
||||
- dist/
|
||||
proxy:
|
||||
caddy:
|
||||
path_prefix: /app
|
||||
```
|
||||
|
||||
For development with Vite's dev server, add a `run` block:
|
||||
|
||||
```yaml
|
||||
components:
|
||||
my-frontend:
|
||||
description: Web dashboard
|
||||
run:
|
||||
runner: node
|
||||
script: dev
|
||||
package_manager: pnpm
|
||||
cwd: my-frontend
|
||||
build:
|
||||
commands:
|
||||
- ["pnpm", "build"]
|
||||
outputs:
|
||||
- dist/
|
||||
expose:
|
||||
http:
|
||||
internal: { port: 5173 }
|
||||
proxy:
|
||||
caddy:
|
||||
path_prefix: /app
|
||||
```
|
||||
|
||||
This gives the component both the `frontend` role (from `build`) and the
|
||||
`service` role (from `expose.http`) during development.
|
||||
|
||||
## Serving with Caddy
|
||||
|
||||
For production, serve the static `dist/` output directly from Caddy rather than
|
||||
running a Node process. The gateway Caddyfile can serve the files:
|
||||
|
||||
```caddyfile
|
||||
handle_path /app/* {
|
||||
root * /data/repos/castle/my-frontend/dist
|
||||
try_files {path} /index.html
|
||||
file_server
|
||||
}
|
||||
```
|
||||
|
||||
The `try_files {path} /index.html` directive is essential for SPA routing —
|
||||
it falls back to `index.html` for any path that doesn't match a static file,
|
||||
letting React Router handle client-side routes.
|
||||
|
||||
## API integration
|
||||
|
||||
Frontends talk to castle services via environment variables injected at build
|
||||
time. Vite exposes variables prefixed with `VITE_`:
|
||||
|
||||
```bash
|
||||
# .env
|
||||
VITE_API_BASE_URL=http://localhost:9001
|
||||
```
|
||||
|
||||
```ts
|
||||
// src/services/api/client.ts
|
||||
const BASE_URL = import.meta.env.VITE_API_BASE_URL ?? "http://localhost:9001"
|
||||
|
||||
class ApiClient {
|
||||
private baseUrl: string
|
||||
|
||||
constructor(baseUrl = BASE_URL) {
|
||||
this.baseUrl = baseUrl
|
||||
}
|
||||
|
||||
async get<T>(path: string): Promise<T> {
|
||||
const resp = await fetch(`${this.baseUrl}${path}`)
|
||||
if (!resp.ok) throw new ApiError(resp.status, await resp.text())
|
||||
return resp.json()
|
||||
}
|
||||
|
||||
// post<T>, put<T>, delete<T>, etc.
|
||||
}
|
||||
|
||||
export const apiClient = new ApiClient()
|
||||
```
|
||||
|
||||
When served behind the castle gateway, the API base URL can use the gateway's
|
||||
proxy paths (e.g., `/central-context/`) instead of direct ports, avoiding CORS.
|
||||
|
||||
## React Query setup
|
||||
|
||||
```ts
|
||||
// src/lib/queryClient.ts
|
||||
import { QueryClient } from "@tanstack/react-query"
|
||||
|
||||
export const queryClient = new QueryClient({
|
||||
defaultOptions: {
|
||||
queries: {
|
||||
staleTime: 5 * 60 * 1000, // 5 minutes
|
||||
gcTime: 10 * 60 * 1000, // 10 minutes
|
||||
retry: 1,
|
||||
},
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
```ts
|
||||
// src/services/api/hooks/useThings.ts
|
||||
import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query"
|
||||
import { apiClient } from "../client"
|
||||
|
||||
export function useThings() {
|
||||
return useQuery({
|
||||
queryKey: ["things"],
|
||||
queryFn: () => apiClient.get<Thing[]>("/things"),
|
||||
})
|
||||
}
|
||||
|
||||
export function useCreateThing() {
|
||||
const qc = useQueryClient()
|
||||
return useMutation({
|
||||
mutationFn: (data: CreateThing) => apiClient.post<Thing>("/things", data),
|
||||
onSuccess: () => qc.invalidateQueries({ queryKey: ["things"] }),
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
## shadcn/ui setup
|
||||
|
||||
Initialize with the `components.json` config:
|
||||
|
||||
```json
|
||||
{
|
||||
"$schema": "https://ui.shadcn.com/schema.json",
|
||||
"style": "new-york",
|
||||
"rsc": false,
|
||||
"tsx": true,
|
||||
"tailwind": {
|
||||
"config": "",
|
||||
"css": "src/index.css",
|
||||
"baseColor": "neutral",
|
||||
"cssVariables": true
|
||||
},
|
||||
"aliases": {
|
||||
"components": "@/components",
|
||||
"utils": "@/lib/utils",
|
||||
"ui": "@/components/ui",
|
||||
"lib": "@/lib",
|
||||
"hooks": "@/hooks"
|
||||
},
|
||||
"iconLibrary": "lucide"
|
||||
}
|
||||
```
|
||||
|
||||
Add components as needed:
|
||||
|
||||
```bash
|
||||
pnpm dlx shadcn@latest add button card dialog sidebar
|
||||
```
|
||||
|
||||
Components are copied into `src/components/ui/` as source files you own and can
|
||||
modify. They use Radix UI primitives underneath, with Tailwind for styling.
|
||||
|
||||
## Dark mode
|
||||
|
||||
Use CSS custom properties with a `.dark` class on `<html>`:
|
||||
|
||||
```css
|
||||
/* src/index.css */
|
||||
@import "tailwindcss";
|
||||
|
||||
:root {
|
||||
--background: oklch(1 0 0);
|
||||
--foreground: oklch(0.145 0 0);
|
||||
--primary: oklch(0.205 0 0);
|
||||
/* ... */
|
||||
}
|
||||
|
||||
.dark {
|
||||
--background: oklch(0.145 0 0);
|
||||
--foreground: oklch(0.985 0 0);
|
||||
--primary: oklch(0.922 0 0);
|
||||
/* ... */
|
||||
}
|
||||
```
|
||||
|
||||
Toggle via a React context that persists the preference to `localStorage`.
|
||||
|
||||
## Testing
|
||||
|
||||
```ts
|
||||
// vitest.config.ts
|
||||
import { defineConfig } from "vitest/config"
|
||||
import path from "path"
|
||||
|
||||
export default defineConfig({
|
||||
test: {
|
||||
environment: "jsdom",
|
||||
setupFiles: ["src/test/setup.ts"],
|
||||
exclude: ["dist/**"],
|
||||
},
|
||||
resolve: {
|
||||
alias: { "@": path.resolve(__dirname, "./src") },
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
```bash
|
||||
pnpm run test # Single run
|
||||
pnpm run test:ui # Interactive UI
|
||||
pnpm run test:coverage # With coverage report
|
||||
```
|
||||
Reference in New Issue
Block a user