feat: raw-TCP exposure with castle-managed TLS + reach model

Expose raw-TCP services (postgres) by name at <name>.<domain>:<port> and cut the gateway's ACME wildcard cert onto them so they present a trusted cert. Replaces the proxy/public booleans with a single `reach` enum (off|internal|public); legacy input still parses via derived accessors. Core: - expose.tcp{port,tls} + TlsSpec(material: pair|combined|off, reload) - tls.py: materialize cert files from Caddy's wildcard, reconcile on renewal; `castle tls` CLI; optional cert_obtained events-exec hook (gateway.cert_hook, gated so a plugin-less Caddy still parses) - apply waits (bounded) for the wildcard to issue before materializing so a fresh-node TLS service starts with its cert in place, then scopes materialization to the deployments being applied - reach: internal|public requires an expose block (no silent no-op); public raw-TCP guarded until tunnel support lands - chain.pem (${tls_ca}) is the issuer chain (leaf stripped), a real CA bundle distinct from cert.pem - one shared ${...} resolver (resolve_placeholders) for env and container run fields; run.env now expands like volumes/args; $$ escapes a literal - validate the generated Caddyfile before reloading the gateway so an invalid config never degrades routing Docs: docs/tcp-exposure.md. Tests cover reach/expose validation, placeholder expansion + escape, issuer-chain material, TLS materialize.
2026-07-04 23:04:26 -07:00
parent 0e8bf2571f
commit 3c566540aa
20 changed files with 1382 additions and 107 deletions
--- a/docs/tcp-exposure.md
+++ b/docs/tcp-exposure.md
@@ -0,0 +1,343 @@
+# TCP exposure & castle-managed TLS material
+
+> Status: **design / not yet implemented.** This specs the `reach` exposure
+> ladder, raw-TCP service exposure (postgres, redis, …), and how castle
+> materializes its ACME wildcard cert onto a service so a raw port presents a
+> *trusted* cert — generically, with **no protocol knowledge in castle's code**.
+
+Read `docs/dns-and-tls.md` first — this builds on the acme wildcard model.
+
+---
+
+## 1. Why
+
+The gateway (Caddy) is HTTP-only: it multiplexes subdomains onto `:443` by TLS
+SNI. Raw-TCP services (postgres/5432, redis/6379, …) can't ride that — the
+protocols either send a cleartext preamble before any ClientHello (postgres,
+mysql) so there's no SNI to route on, or they simply aren't HTTP. But they don't
+*need* the gateway: the wildcard DNS record already points **every** subdomain at
+the node, and each service already has its own port. So "expose a TCP service
+internally" reduces to two things castle can do without a proxy:
+
+1. **bind the port** on the LAN, and
+2. **put a trusted cert on the service** — the one wildcard cert we already own,
+   which is valid for `<name>.<domain>` (wildcards match).
+
+The only per-service variation is *what file format the cert takes* and *how the
+service is told to re-read it*. Both are pushed into the deployment yaml. Castle
+stays a cert-mover and a signal-sender.
+
+Public raw-TCP is a separate, Cloudflare-edge concern — see §6.
+
+---
+
+## 2. The `reach` ladder (replaces `proxy` / `public`)
+
+Exposure becomes one protocol-agnostic enum on the deployment:
+
+| `reach` | meaning |
+|---------|---------|
+| `off` *(default)* | reachable only at its own `host:port` (no gateway route, no DNS intent) |
+| `internal` | reachable at `<name>.<domain>` — HTTP via the gateway, or TCP via bind + wildcard DNS |
+| `public` | *also* projected to the internet (HTTP via tunnel origin; TCP via `cloudflared access tcp`) |
+
+`reach` is orthogonal to **protocol**, which is described by `expose.http` /
+`expose.tcp` (ports live there). One field says *how far it reaches*; the other
+says *what it speaks*.
+
+### Legacy mapping (back-compat normalizer)
+
+`castle` accepts the old fields and normalizes them so existing yamls keep
+working; `castle apply` can rewrite them on next write:
+
+| old | new |
+|-----|-----|
+| *(neither)* | `reach: off` |
+| `proxy: true` | `reach: internal` |
+| `proxy: true` + `public: true` | `reach: public` |
+| `public: true` without `proxy` | *(already invalid — unchanged)* |
+
+---
+
+## 3. Manifest models (`core/src/castle_core/manifest.py`)
+
+```python
+class Reach(str, Enum):
+    OFF = "off"
+    INTERNAL = "internal"
+    PUBLIC = "public"
+
+
+class TlsMaterial(str, Enum):
+    OFF = "off"           # service does its own TLS (or none)
+    PAIR = "pair"         # cert.pem + key.pem (postgres, redis, most daemons)
+    COMBINED = "combined" # one file: key+cert concatenated (mongodb, haproxy)
+
+
+class TlsSpec(BaseModel):
+    material: TlsMaterial = TlsMaterial.OFF
+    # Optional zero-downtime reload argv run after re-materialize on renewal.
+    # Default: castle restarts the deployment (fine for a ~60-day cadence).
+    reload: list[str] | None = None
+
+
+class TcpExposeSpec(BaseModel):
+    port: int = Field(ge=1, le=65535)
+    tls: TlsSpec | None = None      # step 3; absent = service does its own TLS
+    # No bind-host field: publishing the port on the LAN is the deployment's own
+    # job (a container's run.ports, or a native service binding 0.0.0.0). castle
+    # doesn't rebind it, so a `host:` here would be an ignored (misleading) field.
+
+
+class ExposeSpec(BaseModel):
+    http: HttpExposeSpec | None = None
+    tcp: TcpExposeSpec | None = None
+
+    @model_validator(mode="after")
+    def _one_protocol(self) -> "ExposeSpec":
+        if self.http and self.tcp:
+            raise ValueError("a deployment exposes http OR tcp, not both")
+        return self
+```
+
+On `SystemdDeployment`, **replace** `proxy: bool` / `public: bool` with:
+
+```python
+    reach: Reach = Reach.OFF
+
+    @model_validator(mode="after")
+    def _validate(self) -> "SystemdDeployment":
+        if self.reach == Reach.PUBLIC and not (self.expose and (self.expose.http or self.expose.tcp)):
+            raise ValueError("reach: public requires an exposed http or tcp port")
+        tls = self.expose and self.expose.tcp and self.expose.tcp.tls
+        if tls and tls.material != TlsMaterial.OFF and self.reach == Reach.OFF:
+            raise ValueError("tls.material needs reach: internal|public")
+        return self
+```
+
+`CaddyDeployment` (static) also gains `reach: Reach = Reach.INTERNAL`
+(constrained to `internal|public` — a static site is inherently served), and its
+old `public: bool` normalizes to `reach`.
+
+> **acme prerequisite:** `tls.material != off` only makes sense when
+> `gateway.tls == acme` with a domain — otherwise there's no wildcard cert to
+> copy. `castle apply` / `castle doctor` warn if `material` is set without acme.
+
+---
+
+## 4. Placeholders (`deploy.py` `_env_context`)
+
+When a deployment declares `expose.tcp.tls.material != off`, castle adds these to
+the placeholder context (alongside the existing `${port}`/`${data_dir}`/… set),
+pointing at the materialized copies under `‹data_dir›/tls/`:
+
+| placeholder | expands to (host path) |
+|-------------|------------------------|
+| `${tls_dir}` | `‹DATA_DIR›/‹config_key›/tls` — mount this into a container |
+| `${tls_cert}` | `${tls_dir}/cert.pem` (leaf + chain) |
+| `${tls_key}` | `${tls_dir}/key.pem` |
+| `${tls_pem}` | `${tls_dir}/combined.pem` (`material: combined`) |
+| `${tls_ca}` | `${tls_dir}/chain.pem` (issuer chain) |
+
+They resolve through the one shared `${...}` resolver (`resolve_placeholders`,
+which `resolve_env_split` also uses) — for a container these `${key}` refs are
+expanded in `run.env`, `run.volumes`, `run.args`, `run.command`, `run.workdir`,
+`run.user`, and `run.tmpfs` alike. To pass a **literal** `${key}` through to the
+container's own shell/env (rather than have castle expand it), write `$${key}`
+(docker-compose-style `$$` escape).
+
+**Native service** (python/command launcher) references the host paths directly:
+
+```yaml
+reach: internal
+expose:
+  tcp: { port: 6379, tls: { material: pair } }
+defaults:
+  env:
+    REDIS_TLS_CERT: ${tls_cert}
+    REDIS_TLS_KEY: ${tls_key}
+```
+
+**Container** mounts `${tls_dir}` and uses in-container paths (the host→container
+path differs, so the author maps it — consistent with our "image specifics live
+in the deployment" rule):
+
+```yaml
+reach: internal
+expose:
+  tcp: { port: 5432, tls: { material: pair } }
+run:
+  launcher: container
+  user: ${uid}:${gid}          # default for castle containers — see below
+  image: postgres:17
+  ports: { '5432': 5432 }
+  tmpfs: [/var/run/postgresql]  # image runtime dir, needs to be writable by our uid
+  volumes:
+    - /data/castle/postgres/data:/var/lib/postgresql/data
+    - ${tls_dir}:/tls:ro
+  args: ['-c','ssl=on','-c','ssl_cert_file=/tls/cert.pem','-c','ssl_key_file=/tls/key.pem']
+```
+
+### Container key ownership — dissolved by uid uniformity (verified)
+
+The naive problem: postgres refuses a key that isn't `0600` **and owned by the
+process uid** (999 in the stock image), while castle writes files as the host
+user (1000), and bind mounts preserve host uid — so 999 can't read them, and
+chowning across uids needs privilege. Rather than patch that with a privileged
+chown, castle **runs containers as the invoking user** (`--user ${uid}:${gid}`,
+the default on `RunContainer`, overridable per deployment). Then the process, its
+data dir, its secrets *and* its certs are all one uid — nothing to chown, for any
+service. Verified end-to-end against `postgres:17`:
+
+```
+docker run --user 1000:1000 --tmpfs /var/run/postgresql \
+  -v …/key.pem:/tls/key.pem:ro (0600, owned 1000) … postgres:17 -c ssl=on …
+→ pg_stat_ssl: t | TLSv1.3     # SSL on, real TLS 1.3, key read fine, zero privilege
+```
+
+The only image-specific detail is a writable runtime dir (`--tmpfs
+/var/run/postgresql`) — declared in the deployment yaml, not castle. `RunContainer`
+gains a `user: str = "${uid}:${gid}"` field and a `tmpfs: list[str]`; castle stays
+privilege-free. (One-time migration cost: an existing data dir owned by 999 gets
+chowned to the runtime uid once when a service adopts this — not per-renewal.)
+
+---
+
+## 5. Cert materialization + the `cert_obtained` hook
+
+### The materializer (`castle_core`, protocol-agnostic)
+
+`materialize_tls(config, name, dep) -> bool` — for one deployment with
+`tls.material != off`:
+
+1. Locate the source wildcard in Caddy's store by glob (prefer prod over staging):
+   `~/.local/share/caddy/certificates/acme-v02*-directory/wildcard_.‹domain›/wildcard_.‹domain›.{crt,key}`
+   (falls back to `acme-staging-v02*` when `CASTLE_ACME_STAGING=1`).
+2. Compare a hash of the source to the materialized copy; **return False if
+   unchanged** (idempotent — safe to call every apply and every renewal).
+3. Write `‹data_dir›/tls/` in the requested format:
+   - `pair` → `cert.pem` (the `.crt`, already leaf+chain), `key.pem`
+   - `combined` → `combined.pem` = `key.pem` + `cert.pem` concatenated
+   - `chain.pem` (for `${tls_ca}`) is written for either format — the **issuer
+     chain** (intermediates only, leaf stripped), a real CA bundle distinct from
+     the leaf-bearing `cert.pem`/`combined.pem`.
+   - files `0600` (key) / `0644` (cert); dir `0700`.
+4. Apply `tls.owner` (via the privileged chown step, §4) when set.
+5. Return True (changed).
+
+`reconcile_tls(config)` — walk all deployments; `materialize_tls` each; for those
+that changed, run `tls.reload` argv **or** `castle restart ‹name›`. Idempotent;
+prints a summary.
+
+### Wiring
+
+- **`castle apply`** calls `materialize_tls` for each affected deployment before
+  (re)starting it — so first deploy has certs in place.
+- **Event-driven (the hook you wanted) — VERIFIED.** Build Caddy with
+  `github.com/mholt/caddy-events-exec` (add the `--with` to `install.sh`'s xcaddy
+  build, alongside `caddy-dns/cloudflare`) and add to the Caddyfile global options:
+  ```
+  {
+      email …
+      acme_dns cloudflare {env.CLOUDFLARE_API_TOKEN}
+      events {
+          on cert_obtained exec castle tls reconcile
+      }
+  }
+  ```
+  Caddy fires `cert_obtained` on every issuance **and renewal** (renewal reuses
+  the same event with `renewal: true`); the handler runs `castle tls reconcile`,
+  which re-copies the rotated wildcard and reloads consumers. Immediate, no polling.
+  > **Proven end-to-end** (2026-07-03): built into our Caddy 2.11.4, an
+  > `on cert_obtained exec` handler fired on internal-CA issuance with event data
+  > (`issuer=local`, `identifier`, `certificate_path`) correctly passed. Since the
+  > emit is issuer-agnostic (CertMagic `config.go`), acme renewal fires it identically.
+  > Two caveats from the test: (a) the module is **experimental** — pin the xcaddy
+  > build and re-verify on Caddy upgrades; (b) `exec` runs the command in the
+  > **background and swallows its exit code** — so `castle tls reconcile` must be
+  > idempotent and log its *own* outcome (don't rely on Caddy surfacing failures).
+- **Safety net:** a nightly `castle-tls-reconcile` job (`manager: systemd` +
+  `schedule`) also runs `castle tls reconcile` — catches a missed event, a Caddy
+  restart, or a manual cert swap. Same idempotent call, so belt-and-suspenders is
+  free.
+
+New CLI: `castle tls reconcile [--plan]` (thin wrapper over `reconcile_tls`), and
+`castle tls status` to show, per deployment, source vs materialized cert fingerprint
+ expiry.
+
+---
+
+## 6. `reach: public` for TCP (later, separable)
+
+Public raw-TCP can't reach an unmodified client below Cloudflare Enterprise
+(Spectrum arbitrary-TCP is Enterprise-gated). The free, minimal path rides the
+**existing** tunnel:
+
+- `tunnel.py` emits an extra ingress entry per public TCP deployment:
+  `{ hostname: ‹name›.‹public_domain›, service: tcp://localhost:‹port› }`.
+- castle ensures a self-hosted **Access** app + policy over that hostname.
+- `castle` prints the client connect line:
+  `cloudflared access tcp --hostname ‹name›.‹public_domain› --url localhost:‹local›`.
+
+The client runs `cloudflared access tcp` (or WARP private-network) and points its
+tool (`psql`/`redis-cli`) at `localhost`. This is the correct posture for a DB —
+authenticated tunnel, never an open port. Build this only when wanted.
+
+---
+
+## 7. Generality check
+
+Same castle code, different yaml — the postgres-ness is two env/arg mappings + a
+format choice, nothing more:
+
+| service | `material` | consumes | reload |
+|---------|-----------|----------|--------|
+| postgres | `pair` | `ssl_cert_file` / `ssl_key_file` args | SIGHUP / restart |
+| redis | `pair` | `--tls-cert-file/--tls-key-file/--tls-ca-cert-file` | restart |
+| mongodb | `combined` | `net.tls.certificateKeyFile: /tls/combined.pem` | restart |
+| arbitrary | `pair` | its own `TLS_CERT`/`TLS_KEY` env | restart |
+
+`pair` vs `combined` is the entire protocol-variation surface.
+
+---
+
+## 8. Build order
+
+1. ~~**`reach` enum + normalizer** for HTTP (replace `proxy`/`public`); migrate
+   existing yamls.~~ **DONE (2026-07-03).** Canonical `reach` on
+   Systemd/CaddyDeployment; `_reach_from_legacy` before-validator accepts legacy
+   input; `proxy`/`public` are derived read-only accessors. 13 yamls migrated
+   (`apply --plan` = already converged); app toggles + `deploy_create` write
+   `reach`; all suites green.
+2. ~~**`expose.tcp` + bind** (internal TCP, `material: off`): `postgres.civil…:5432`
+   works, service does its own TLS.~~ **DONE (2026-07-03).** `TcpExposeSpec` +
+   `ExposeSpec` one-protocol validator; `http_exposed`/`tcp_port` predicates so a
+   TCP `reach: internal` yields **no** Caddy route (correctness); registry carries
+   `tcp_port`; `castle service info` shows the `<name>.<domain>:<port>` endpoint;
+   public-TCP guarded (step 5). Applied to `postgres` — verified: `psql -h
+   postgres.civil.payne.io` connects by name, postgres absent from HTTP routes,
+   `apply --plan` still converged.
+3. **TLS material + placeholders + `materialize_tls`** — **CODE DONE (2026-07-03).**
+   `TlsSpec`/`TlsMaterial`; `${tls_dir|cert|key|pem|ca}` + `${uid}/${gid}`
+   placeholders; `RunContainer.user`/`tmpfs` (uid-uniformity, no chown);
+   `core/tls.py` (`wildcard_cert`/`materialize_tls`/`materialize_all`); wired into
+   `castle apply`; unit-tested in isolation (pair/combined/idempotency/stale-clean).
+4. **`cert_obtained` hook + `castle tls`** — **CODE DONE (2026-07-03).** Generator
+   emits the `events { on cert_obtained exec castle tls reconcile }` block, **gated
+   on `CASTLE_CADDY_CERT_HOOK=1`** so the current (no-plugin) gateway is unaffected;
+   `castle tls reconcile|status` CLI; `reconcile_tls` idempotent + self-logging.
+   *(Nightly safety-net job: TODO.)*
+   → **LIVE ROLLOUT DONE (2026-07-03).** (a) `/usr/local/bin/caddy` rebuilt with
+   cloudflare-dns + `caddy-events-exec` (old binary → `caddy.bak-pre-events`;
+   `install.sh` bakes both plugins). (b) Gate is a durable **`gateway.cert_hook`**
+   config flag (not an env var) → NodeConfig → generator; set true, `apply`'d; the
+   running gateway's admin API confirms the `cert_obtained → castle tls reconcile`
+   subscription is live. (c) postgres enabled: `user: ${uid}:${gid}` + `tmpfs` +
+   `tls.material: pair`; data dir chowned 999→1000 once (cold backup at
+   `/data/castle/postgres-data-backup-pre-tls.tar.gz`); WAL-recovered clean.
+   **Verified:** `psql -h postgres.civil.payne.io sslmode=verify-full sslrootcert=system`
+   → "verify-full OK" against the trusted LE wildcard; `castle` + `litellm` DBs intact.
+   Also landed: `${uid}/${gid}` in the placeholder context + placeholder expansion
+   in container run fields (`volumes`/`args`/`user`/`tmpfs`) so `${tls_dir}` mounts work.
+5. **`reach: public` for TCP** (tunnel `tcp://` + Access + connect line) — when
+   wanted.