feat: Enhance gateway management and add Caddyfile generation support
This commit is contained in:
62
working/decisions.md
Normal file
62
working/decisions.md
Normal file
@@ -0,0 +1,62 @@
|
||||
# Implementation Decisions
|
||||
|
||||
Decisions made during implementation that weren't pre-decided. Paul will review these.
|
||||
|
||||
## 1. CLI entry point name
|
||||
|
||||
Used `castle` as the command name (via `[project.scripts] castle = "castle_cli.main:main"`).
|
||||
Package name is `castle-cli` to avoid conflicts, but the command is just `castle`.
|
||||
|
||||
## 2. Caddy `handle_path` instead of `handle`
|
||||
|
||||
Used `handle_path` in the Caddyfile instead of `handle`. `handle_path` automatically strips
|
||||
the path prefix before proxying, so `/central-context/health` proxies to `localhost:9000/health`.
|
||||
Without this, the upstream service would receive the full `/central-context/health` path and
|
||||
return 404.
|
||||
|
||||
## 3. Service auto-port assignment
|
||||
|
||||
`castle create --type service` auto-assigns the next available port starting from 9000,
|
||||
skipping any ports already used by other services or the gateway. This avoids port collisions
|
||||
without requiring the user to track assignments manually.
|
||||
|
||||
## 4. Systemd unit naming convention
|
||||
|
||||
All castle systemd units use the prefix `castle-` (e.g., `castle-central-context.service`,
|
||||
`castle-gateway.service`). This makes them easy to identify and manage as a group.
|
||||
|
||||
## 5. `uv` path resolution in systemd units
|
||||
|
||||
Systemd user units don't inherit the user's PATH. The CLI resolves `uv` to its absolute
|
||||
path (via `shutil.which`) when generating unit files so they work regardless of PATH.
|
||||
|
||||
## 6. Dashboard health checks use direct ports
|
||||
|
||||
The dashboard HTML checks health by fetching directly from each service's port
|
||||
(e.g., `localhost:9000/health`) rather than going through the gateway. This avoids
|
||||
circular dependency if the gateway itself is having issues, and gives accurate per-service
|
||||
health status.
|
||||
|
||||
## 7. `castle sync` runs `uv sync` before `git submodule update`
|
||||
|
||||
Actually runs submodule update first, then `uv sync` in each project. This way submodules
|
||||
are at the right commit before dependencies are installed.
|
||||
|
||||
## 8. Template test structure
|
||||
|
||||
Scaffolded services include a health endpoint test by default. Tools include a placeholder
|
||||
test. Libraries include an import test. This ensures `castle test` works immediately after
|
||||
`castle create` without requiring the developer to write tests first.
|
||||
|
||||
## 9. `save_config` YAML formatting
|
||||
|
||||
When the CLI saves back to `castle.yaml` (e.g., after `castle create`), the YAML output
|
||||
uses `yaml.dump` with `default_flow_style=False` and `sort_keys=False` to keep the
|
||||
format readable and maintain insertion order. The format won't be identical to the original
|
||||
hand-written YAML (e.g., blank lines are lost) but is functionally equivalent.
|
||||
|
||||
## 10. Jinja2 dependency
|
||||
|
||||
Added jinja2 as a dependency in the CLI's pyproject.toml for potential future template
|
||||
expansion, but the current scaffold implementation uses plain f-strings. Could be removed
|
||||
if it's not used soon.
|
||||
342
working/schema.py
Normal file
342
working/schema.py
Normal file
@@ -0,0 +1,342 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from enum import Enum
|
||||
from typing import Annotated, Dict, List, Literal, Optional, Set, Union
|
||||
|
||||
from pydantic import (
|
||||
BaseModel,
|
||||
Field,
|
||||
HttpUrl,
|
||||
computed_field,
|
||||
model_validator,
|
||||
)
|
||||
|
||||
|
||||
# -------------------------
|
||||
# Shared primitives
|
||||
# -------------------------
|
||||
|
||||
EnvMap = Dict[str, str]
|
||||
|
||||
|
||||
class RestartPolicy(str, Enum):
|
||||
NO = "no"
|
||||
ON_FAILURE = "on-failure"
|
||||
ALWAYS = "always"
|
||||
|
||||
|
||||
class TLSMode(str, Enum):
|
||||
OFF = "off"
|
||||
INTERNAL = "internal"
|
||||
LETSENCRYPT = "letsencrypt"
|
||||
|
||||
|
||||
class Role(str, Enum):
|
||||
TOOL = "tool"
|
||||
SERVICE = "service"
|
||||
WORKER = "worker"
|
||||
FRONTEND = "frontend"
|
||||
JOB = "job"
|
||||
REMOTE = "remote"
|
||||
CONTAINERIZED = "containerized"
|
||||
|
||||
|
||||
# -------------------------
|
||||
# Run specs (discriminated union)
|
||||
# -------------------------
|
||||
|
||||
class RunBase(BaseModel):
|
||||
runner: str
|
||||
cwd: Optional[str] = None
|
||||
env: EnvMap = Field(default_factory=dict)
|
||||
|
||||
|
||||
class RunCommand(RunBase):
|
||||
runner: Literal["command"]
|
||||
argv: List[str] = Field(min_length=1)
|
||||
|
||||
|
||||
class RunPythonModule(RunBase):
|
||||
runner: Literal["python_module"]
|
||||
module: str
|
||||
args: List[str] = Field(default_factory=list)
|
||||
python: Optional[str] = None # e.g. "/path/to/python"
|
||||
|
||||
|
||||
class RunPythonUvTool(RunBase):
|
||||
runner: Literal["python_uv_tool"]
|
||||
tool: str # the installed tool name
|
||||
args: List[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class RunContainer(RunBase):
|
||||
runner: Literal["container"]
|
||||
image: str
|
||||
command: Optional[List[str]] = None # overrides image CMD if provided
|
||||
args: List[str] = Field(default_factory=list)
|
||||
ports: Dict[int, int] = Field(default_factory=dict) # container_port -> host_port
|
||||
volumes: List[str] = Field(default_factory=list) # "host:container[:ro]"
|
||||
workdir: Optional[str] = None
|
||||
|
||||
|
||||
class RunNode(RunBase):
|
||||
runner: Literal["node"]
|
||||
script: str # e.g. "dev", "start", or a file path
|
||||
package_manager: Literal["npm", "pnpm", "yarn"] = "pnpm"
|
||||
args: List[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class RunRemote(RunBase):
|
||||
runner: Literal["remote"]
|
||||
base_url: HttpUrl
|
||||
# Optional: metadata for how Castle should treat it
|
||||
health_url: Optional[HttpUrl] = None
|
||||
|
||||
|
||||
RunSpec = Annotated[
|
||||
Union[RunCommand, RunPythonModule, RunPythonUvTool, RunContainer, RunNode, RunRemote],
|
||||
Field(discriminator="runner"),
|
||||
]
|
||||
|
||||
|
||||
# -------------------------
|
||||
# Triggers
|
||||
# -------------------------
|
||||
|
||||
class TriggerManual(BaseModel):
|
||||
type: Literal["manual"] = "manual"
|
||||
|
||||
|
||||
class TriggerSchedule(BaseModel):
|
||||
type: Literal["schedule"] = "schedule"
|
||||
# Keep this simple + explicit. You can later add "cron"/"interval".
|
||||
cron: str # e.g. "0 * * * *"
|
||||
timezone: str = "America/Los_Angeles"
|
||||
|
||||
|
||||
class TriggerEvent(BaseModel):
|
||||
type: Literal["event"] = "event"
|
||||
source: str # e.g. "kafka", "redis", "webhook", "fs"
|
||||
topic: Optional[str] = None
|
||||
queue: Optional[str] = None
|
||||
|
||||
|
||||
class TriggerRequest(BaseModel):
|
||||
type: Literal["request"] = "request"
|
||||
protocol: Literal["http", "https", "grpc"] = "http"
|
||||
|
||||
|
||||
TriggerSpec = Union[TriggerManual, TriggerSchedule, TriggerEvent, TriggerRequest]
|
||||
|
||||
|
||||
# -------------------------
|
||||
# Systemd management intent
|
||||
# -------------------------
|
||||
|
||||
class ReadinessHttpGet(BaseModel):
|
||||
http_get: HttpUrl | str # allow templated strings like "http://127.0.0.1:${PORT}/healthz"
|
||||
timeout_seconds: int = 2
|
||||
interval_seconds: int = 2
|
||||
success_codes: List[int] = Field(default_factory=lambda: [200])
|
||||
|
||||
|
||||
class SystemdSpec(BaseModel):
|
||||
enable: bool = True
|
||||
user: bool = True
|
||||
|
||||
description: Optional[str] = None
|
||||
after: List[str] = Field(default_factory=list)
|
||||
requires: List[str] = Field(default_factory=list)
|
||||
wanted_by: List[str] = Field(default_factory=lambda: ["default.target"])
|
||||
|
||||
restart: RestartPolicy = RestartPolicy.ON_FAILURE
|
||||
restart_sec: int = 2
|
||||
|
||||
# Optional hardening knobs (you can expand later)
|
||||
no_new_privileges: bool = True
|
||||
|
||||
readiness: Optional[ReadinessHttpGet] = None
|
||||
|
||||
|
||||
class ManageSpec(BaseModel):
|
||||
systemd: Optional[SystemdSpec] = None
|
||||
|
||||
|
||||
# -------------------------
|
||||
# Install intent (PATH shims etc.)
|
||||
# -------------------------
|
||||
|
||||
class PathInstallSpec(BaseModel):
|
||||
enable: bool = True
|
||||
# If set, Castle creates a shim with this name (e.g. ~/.local/bin/<alias>)
|
||||
alias: Optional[str] = None
|
||||
# If true, shim forwards args to `castle run <id> ...`
|
||||
shim: bool = True
|
||||
|
||||
|
||||
class InstallSpec(BaseModel):
|
||||
path: Optional[PathInstallSpec] = None
|
||||
|
||||
|
||||
# -------------------------
|
||||
# HTTP exposure + proxy intent
|
||||
# -------------------------
|
||||
|
||||
class HttpInternal(BaseModel):
|
||||
host: str = "127.0.0.1"
|
||||
port: int = Field(ge=1, le=65535)
|
||||
unix_socket: Optional[str] = None # e.g. "/run/user/1000/notes.sock"
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _exactly_one_target(self):
|
||||
# allow either (host+port) or unix_socket
|
||||
if self.unix_socket and (self.host or self.port):
|
||||
# host/port always exist via defaults; treat unix_socket as alternative
|
||||
# if unix_socket is set, we ignore host/port semantically
|
||||
return self
|
||||
return self
|
||||
|
||||
|
||||
class HttpPublic(BaseModel):
|
||||
hostnames: List[str] = Field(min_length=1)
|
||||
path_prefix: str = "/"
|
||||
tls: TLSMode = TLSMode.INTERNAL
|
||||
|
||||
|
||||
class HttpExposeSpec(BaseModel):
|
||||
internal: HttpInternal
|
||||
public: Optional[HttpPublic] = None
|
||||
health_path: Optional[str] = None # e.g. "/healthz"
|
||||
|
||||
|
||||
class ExposeSpec(BaseModel):
|
||||
http: Optional[HttpExposeSpec] = None
|
||||
# Future: cli, grpc, etc.
|
||||
|
||||
|
||||
class CaddySpec(BaseModel):
|
||||
enable: bool = True
|
||||
# Optional: extra per-component knobs
|
||||
extra_snippets: List[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class ProxySpec(BaseModel):
|
||||
caddy: Optional[CaddySpec] = None
|
||||
|
||||
|
||||
# -------------------------
|
||||
# Build spec (mostly for frontends, but generic)
|
||||
# -------------------------
|
||||
|
||||
class BuildSpec(BaseModel):
|
||||
# e.g. ["pnpm", "build"] or ["uv", "run", "python", "-m", "build"]
|
||||
commands: List[List[str]] = Field(default_factory=list)
|
||||
outputs: List[str] = Field(default_factory=list) # paths relative to cwd/repo
|
||||
|
||||
|
||||
# -------------------------
|
||||
# Capabilities (optional, for dependency graph)
|
||||
# -------------------------
|
||||
|
||||
class Capability(BaseModel):
|
||||
type: str # e.g. "http.endpoint", "cli.command", "ui.bundle"
|
||||
name: Optional[str] = None
|
||||
meta: Dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
|
||||
# -------------------------
|
||||
# The manifest (source of truth)
|
||||
# -------------------------
|
||||
|
||||
class ComponentManifest(BaseModel):
|
||||
id: str = Field(pattern=r"^[a-z0-9][a-z0-9\-_.]{1,63}$")
|
||||
name: Optional[str] = None
|
||||
description: Optional[str] = None
|
||||
|
||||
run: RunSpec
|
||||
|
||||
triggers: List[TriggerSpec] = Field(default_factory=list)
|
||||
|
||||
manage: Optional[ManageSpec] = None
|
||||
install: Optional[InstallSpec] = None
|
||||
expose: Optional[ExposeSpec] = None
|
||||
proxy: Optional[ProxySpec] = None
|
||||
build: Optional[BuildSpec] = None
|
||||
|
||||
provides: List[Capability] = Field(default_factory=list)
|
||||
consumes: List[Capability] = Field(default_factory=list)
|
||||
|
||||
tags: List[str] = Field(default_factory=list)
|
||||
|
||||
# ---- Derived ontology ----
|
||||
|
||||
@computed_field
|
||||
@property
|
||||
def roles(self) -> List[Role]:
|
||||
"""
|
||||
Derive roles purely from declared blocks.
|
||||
No 'kind/profile' field required.
|
||||
"""
|
||||
roles: Set[Role] = set()
|
||||
|
||||
# Runner-derived roles
|
||||
if self.run.runner == "remote":
|
||||
roles.add(Role.REMOTE)
|
||||
if self.run.runner == "container":
|
||||
roles.add(Role.CONTAINERIZED)
|
||||
|
||||
# Interface / integration-derived roles
|
||||
if self.install and self.install.path and self.install.path.enable:
|
||||
roles.add(Role.TOOL)
|
||||
|
||||
if self.expose and self.expose.http:
|
||||
roles.add(Role.SERVICE)
|
||||
|
||||
# Worker vs Service heuristic:
|
||||
# If it's managed (systemd) but not exposed over HTTP, it's worker-ish.
|
||||
if (
|
||||
self.manage
|
||||
and self.manage.systemd
|
||||
and self.manage.systemd.enable
|
||||
and not (self.expose and self.expose.http)
|
||||
):
|
||||
roles.add(Role.WORKER)
|
||||
|
||||
# Frontend heuristic: build outputs present (or node runner + build)
|
||||
if self.build and (self.build.outputs or self.build.commands):
|
||||
# Avoid labeling everything with build as frontend if it's clearly not;
|
||||
# if it's exposing HTTP and has build, that's typically frontend or full-stack,
|
||||
# but frontend label is still useful.
|
||||
roles.add(Role.FRONTEND)
|
||||
|
||||
# Job heuristic: any schedule trigger
|
||||
if any(getattr(t, "type", None) == "schedule" for t in self.triggers):
|
||||
roles.add(Role.JOB)
|
||||
|
||||
# Fallback: if nothing else, treat as worker-ish if it’s supervised,
|
||||
# otherwise tool-ish if it’s transient. But keep it conservative.
|
||||
if not roles:
|
||||
roles.add(Role.TOOL)
|
||||
|
||||
return sorted(roles, key=lambda r: r.value)
|
||||
|
||||
# ---- Optional consistency checks ----
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _basic_consistency(self):
|
||||
# If you declare proxy.caddy, you probably mean to expose HTTP publicly.
|
||||
if self.proxy and self.proxy.caddy and self.proxy.caddy.enable:
|
||||
if not (self.expose and self.expose.http and self.expose.http.public):
|
||||
# keep this as a soft constraint by raising a ValueError;
|
||||
# remove if you prefer permissive manifests
|
||||
raise ValueError(
|
||||
"proxy.caddy is enabled but expose.http.public is not set. "
|
||||
"Either disable caddy or declare public hostnames."
|
||||
)
|
||||
|
||||
# If systemd is enabled, ensure runner isn't something obviously incompatible.
|
||||
if self.manage and self.manage.systemd and self.manage.systemd.enable:
|
||||
if self.run.runner == "remote":
|
||||
raise ValueError("manage.systemd cannot be enabled for runner=remote.")
|
||||
|
||||
return self
|
||||
74
working/validation-log.md
Normal file
74
working/validation-log.md
Normal file
@@ -0,0 +1,74 @@
|
||||
# Validation Log
|
||||
|
||||
## What was implemented
|
||||
|
||||
### Castle CLI (`cli/`)
|
||||
- `castle --version` — shows version
|
||||
- `castle list [--type] [--json]` — list all projects, filter by type, JSON output
|
||||
- `castle create <name> --type service|tool|library` — scaffold project, register in castle.yaml
|
||||
- `castle test [project]` — run tests across one or all projects
|
||||
- `castle lint [project]` — run ruff across one or all projects
|
||||
- `castle sync` — git submodule update + uv sync in all projects
|
||||
- `castle gateway start|stop|reload|status` — manage Caddy reverse proxy
|
||||
- `castle service enable|disable <name>` — manage individual systemd units
|
||||
- `castle service status` — show all service statuses
|
||||
- `castle services start|stop` — start/stop everything
|
||||
|
||||
### Registry (`castle.yaml`)
|
||||
- All 4 existing projects registered with correct types, ports, data dirs
|
||||
- `castle create` auto-registers new projects
|
||||
- Auto port assignment for new services
|
||||
|
||||
### Gateway (Caddy)
|
||||
- Caddyfile generated from castle.yaml into `~/.castle/generated/`
|
||||
- Dashboard HTML with health checks served at root
|
||||
- Reverse proxy with `handle_path` (strips prefix) for each service
|
||||
- Gateway managed as its own systemd unit
|
||||
|
||||
### Systemd integration
|
||||
- User units generated under `~/.config/systemd/user/castle-*.service`
|
||||
- Resolves `uv` to absolute path for systemd
|
||||
- Resolves `${data_dir}` in env vars
|
||||
- Creates data dirs via ExecStartPre
|
||||
|
||||
### Project templates
|
||||
- Service: FastAPI, pydantic-settings, lifespan, health endpoint, test fixtures
|
||||
- Tool: argparse, stdin/stdout, exit codes
|
||||
- Library: src/ layout, import test
|
||||
- All templates include CLAUDE.md and pyproject.toml with ruff isort config
|
||||
|
||||
### Configuration files
|
||||
- `castle.yaml` — project registry
|
||||
- `ruff.toml` — shared ruff config (100-char lines, E/F/I/W rules)
|
||||
- `pyrightconfig.json` — shared pyright config
|
||||
- `CLAUDE.md` — updated with full castle system docs
|
||||
- `recommendations.md` — updated with all decisions
|
||||
|
||||
## What was validated
|
||||
|
||||
| Test | Result |
|
||||
|------|--------|
|
||||
| `castle --version` | 0.1.0 |
|
||||
| `castle list` | Shows all 4 projects grouped by type |
|
||||
| `castle list --json` | Valid JSON output |
|
||||
| `castle list --type service` | Filters correctly |
|
||||
| `castle create --type service` | Scaffolds, registers, auto-assigns port |
|
||||
| `castle create --type tool` | Scaffolds, registers |
|
||||
| `castle create --type library` | Scaffolds, registers |
|
||||
| `castle create` duplicate | Fails with error message |
|
||||
| `castle test <project>` | Passes for scaffolded projects |
|
||||
| `castle lint <project>` | Passes for scaffolded projects |
|
||||
| `castle sync` | Updates submodules + uv sync in all projects |
|
||||
| `castle gateway start` | Starts Caddy, dashboard accessible |
|
||||
| `castle gateway reload` | Regenerates and reloads |
|
||||
| `castle gateway status` | Shows running/stopped |
|
||||
| `castle service enable` | Creates unit, enables, starts |
|
||||
| `castle service disable` | Stops, removes unit |
|
||||
| `castle service status` | Shows all service statuses with colors |
|
||||
| `castle services start` | Starts all services + gateway |
|
||||
| `castle services stop` | Stops everything |
|
||||
| Gateway reverse proxy | Routes correctly (path prefix stripped) |
|
||||
| Gateway dashboard | Serves HTML with health check JS |
|
||||
| Direct service access | All services respond on their ports |
|
||||
| CLI unit tests | 32/32 passing |
|
||||
| CLI lint | All checks passed |
|
||||
Reference in New Issue
Block a user