From 0d35ac9ffde046df5ff775aaded388688e1c8056 Mon Sep 17 00:00:00 2001 From: Paul Payne Date: Thu, 19 Feb 2026 16:38:11 -0800 Subject: [PATCH] Initial commit: castle personal software platform Add three submodules (central-context, mboxer, notification-bridge), devbox-connect as tracked files, and top-level project docs. --- .gitmodules | 9 + CLAUDE.md | 77 ++++ central-context | 1 + devbox-connect/README.md | 219 +++++++++++ devbox-connect/pyproject.toml | 31 ++ devbox-connect/service/install-service.ps1 | 202 ++++++++++ devbox-connect/service/run-manual.bat | 19 + devbox-connect/src/devbox_connect/__init__.py | 3 + devbox-connect/src/devbox_connect/cli.py | 235 ++++++++++++ devbox-connect/src/devbox_connect/config.py | 162 ++++++++ devbox-connect/src/devbox_connect/tunnel.py | 355 ++++++++++++++++++ devbox-connect/tunnels.example.yaml | 66 ++++ mboxer | 1 + notification-bridge | 1 + recommendations.md | 46 +++ 15 files changed, 1427 insertions(+) create mode 100644 .gitmodules create mode 100644 CLAUDE.md create mode 160000 central-context create mode 100644 devbox-connect/README.md create mode 100644 devbox-connect/pyproject.toml create mode 100644 devbox-connect/service/install-service.ps1 create mode 100644 devbox-connect/service/run-manual.bat create mode 100644 devbox-connect/src/devbox_connect/__init__.py create mode 100644 devbox-connect/src/devbox_connect/cli.py create mode 100644 devbox-connect/src/devbox_connect/config.py create mode 100644 devbox-connect/src/devbox_connect/tunnel.py create mode 100644 devbox-connect/tunnels.example.yaml create mode 160000 mboxer create mode 160000 notification-bridge create mode 100644 recommendations.md diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 0000000..f05cd0c --- /dev/null +++ b/.gitmodules @@ -0,0 +1,9 @@ +[submodule "central-context"] + path = central-context + url = https://github.com/payneio/central-context.git +[submodule "mboxer"] + path = mboxer + url = git@github.com:payneio/mboxer.git +[submodule "notification-bridge"] + path = notification-bridge + url = https://github.com/payneio/attention-firewall.git diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..59b1f88 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,77 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Repository Overview + +Castle is a monorepo of four independent Python projects that form personal infrastructure services. Each project has its own `pyproject.toml`, `uv.lock`, and dependencies. + +| Project | Purpose | Layout | +|---------|---------|--------| +| **central-context** | REST API for storing/retrieving UTF-8 content in buckets | `src/central_context/` | +| **notification-bridge** | Cross-platform desktop notification forwarder | `notification_bridge/` (no src/) | +| **devbox-connect** | SSH tunnel manager with auto-reconnect | `src/devbox_connect/` | +| **mboxer** | MBOX to EML email converter | Single file `convert.py` | + +## Build & Development Commands + +All projects use **uv** as the package manager. Commands must be run from each project's directory. + +### central-context +```bash +cd central-context +uv sync # Install deps +uv run central-context # Run service (port 9000) +uv run pytest tests/ -v # Run tests +uv run pytest tests/test_storage.py -v # Single test file +``` + +### notification-bridge +```bash +cd notification-bridge +uv sync --extra linux # Install deps (use --extra windows on Windows) +uv run notification-bridge # Run service (port 9001) +uv run pytest --cov=notification_bridge # Run tests with coverage +uv run ruff format . # Format +uv run ruff check . # Lint +``` + +### devbox-connect +```bash +cd devbox-connect +uv sync # Install deps +uv tool install . # Install as CLI tool +devbox-connect -c tunnels.yaml start # Start tunnels +devbox-connect -c tunnels.yaml status # Show status +devbox-connect -c tunnels.yaml validate # Validate config +``` + +### mboxer +```bash +cd mboxer +uv sync # Install deps +python convert.py # Run converter (configure via .env) +ruff check . --fix # Lint +``` + +## Architecture + +**central-context** is the hub — notification-bridge forwards captured desktop notifications to it via its REST API. The API organizes content into buckets (filesystem directories), auto-names entries by SHA256 checksum, and stores JSON metadata sidecars alongside content files. + +**notification-bridge** uses a platform adapter pattern: `listeners/base.py` defines a `NotificationListener` protocol, with `linux.py` (D-Bus) and `windows.py` (WinRT) implementations. The server captures notifications and POSTs them to central-context. + +**devbox-connect** manages persistent SSH tunnels defined in YAML config. It supports two config formats: simple (flat list) and grouped (by host). Tunnels auto-reconnect with exponential backoff. Has Windows service support via NSSM. + +## Configuration + +- **central-context**: Env vars with `CENTRAL_CONTEXT_` prefix, pydantic-settings +- **notification-bridge**: `.env` file (`CENTRAL_CONTEXT_URL`, `BUCKET_NAME`, `PORT`) +- **devbox-connect**: YAML config file (`tunnels.yaml`) +- **mboxer**: `.env` file (`MBOX_PATH`, `OUTPUT_DIR`) + +## Code Style + +- **Linting/formatting**: ruff (project-specific configs in each `pyproject.toml`) +- **devbox-connect**: 100-char line length, pyright type checking at standard level, Python 3.10+ +- **central-context / notification-bridge**: Python 3.13, FastAPI +- **Testing**: pytest with pytest-asyncio for async tests diff --git a/central-context b/central-context new file mode 160000 index 0000000..495e6d7 --- /dev/null +++ b/central-context @@ -0,0 +1 @@ +Subproject commit 495e6d7c1f29e9319f670041cf5d0d98277d7a5a diff --git a/devbox-connect/README.md b/devbox-connect/README.md new file mode 100644 index 0000000..7291fa6 --- /dev/null +++ b/devbox-connect/README.md @@ -0,0 +1,219 @@ +# devbox-connect + +SSH tunnel manager for connecting local ports to your devbox. + +## Features + +- **Simple YAML configuration** - Define tunnels in a readable config file +- **Auto-reconnect** - Automatically reconnects with exponential backoff when connections drop +- **Multiple tunnels** - Manage many port forwards to one or more hosts +- **Windows service support** - Run as a background service with auto-start + +## Installation + +```bash +# Install with uv +uv tool install git+https://github.com/YOUR_USERNAME/devbox-connect + +# Or install from local directory +cd devbox-connect +uv tool install . +``` + +## Quick Start + +1. Create a configuration file `tunnels.yaml`: + +```yaml +user: your-username +key_file: ~/.ssh/id_rsa + +tunnels: + - name: web-dev + host: devbox.example.com + remote_port: 8080 + + - name: database + host: devbox.example.com + remote_port: 5432 +``` + +2. Start the tunnels: + +```bash +devbox-connect -c tunnels.yaml start +``` + +3. Access your devbox services locally: + - `localhost:8080` → devbox:8080 + - `localhost:5432` → devbox:5432 + +## Usage + +``` +devbox-connect [-c CONFIG] COMMAND + +Commands: + start Start tunnels and keep running (default) + status Show configured tunnels + validate Validate configuration file + +Options: + -c, --config PATH Path to config file (default: tunnels.yaml) + -v, --verbose Enable verbose output +``` + +## Configuration + +See `tunnels.example.yaml` for a complete example. + +### Simple Format (single host) + +```yaml +user: username +key_file: ~/.ssh/id_rsa # Optional + +tunnels: + - name: web + host: devbox.example.com + remote_port: 8080 + local_port: 8080 # Optional, defaults to remote_port + + - name: jupyter + host: devbox.example.com + remote_port: 8888 + local_port: 9999 # Use different local port +``` + +### Grouped Format (multiple hosts) + +```yaml +hosts: + - host: devbox1.example.com + user: username + tunnels: + - name: web + remote_port: 8080 + + - host: devbox2.example.com + user: username + tunnels: + - name: api + remote_port: 3000 +``` + +### Configuration Options + +| Option | Description | Default | +|--------|-------------|---------| +| `user` | SSH username | (required) | +| `host` | Remote hostname | (required) | +| `key_file` | Path to SSH private key | SSH agent/default | +| `remote_port` | Port on remote host | (required) | +| `local_port` | Local port to listen on | Same as remote_port | +| `remote_host` | Host on remote side | `localhost` | +| `reconnect_delay` | Initial reconnect delay (seconds) | `5` | +| `max_reconnect_delay` | Max reconnect delay | `60` | + +### Forwarding Through Devbox + +You can access services on other hosts through your devbox: + +```yaml +tunnels: + - name: internal-db + host: devbox.example.com + remote_port: 5432 + remote_host: internal-db.corp # Accessed via devbox +``` + +## Windows Service + +To run devbox-connect as a Windows service that starts automatically: + +### Prerequisites + +1. Install NSSM (Non-Sucking Service Manager): + ```powershell + winget install nssm + ``` + +2. Install devbox-connect: + ```powershell + uv tool install . + ``` + +### Install Service + +Run PowerShell as Administrator: + +```powershell +.\service\install-service.ps1 -ConfigPath C:\path\to\tunnels.yaml +``` + +### Manage Service + +```powershell +# Check status +Get-Service DevboxConnect + +# Start/Stop +Start-Service DevboxConnect +Stop-Service DevboxConnect + +# View logs +Get-Content $env:LOCALAPPDATA\devbox-connect\service.log -Tail 50 + +# Uninstall +.\service\install-service.ps1 -Uninstall +``` + +### Manual Run (without service) + +```batch +service\run-manual.bat C:\path\to\tunnels.yaml +``` + +## SSH Key Setup + +devbox-connect uses SSH key authentication. Ensure your key is set up: + +1. Generate a key (if needed): + ```bash + ssh-keygen -t ed25519 -f ~/.ssh/devbox_key + ``` + +2. Copy to devbox: + ```bash + ssh-copy-id -i ~/.ssh/devbox_key user@devbox.example.com + ``` + +3. Reference in config: + ```yaml + key_file: ~/.ssh/devbox_key + ``` + +Or use SSH agent (key_file not needed if agent has your key loaded). + +## Troubleshooting + +### Connection refused +- Check the remote service is running on the specified port +- Verify you can SSH to the host manually: `ssh user@devbox` + +### Permission denied +- Check your SSH key is correct and has proper permissions +- On Windows, ensure key file isn't world-readable + +### Port already in use +- Change `local_port` to an unused port +- Check what's using the port: `netstat -an | findstr :8080` + +### Tunnels disconnect frequently +- Check network stability +- Increase `reconnect_delay` and `max_reconnect_delay` +- Some networks/firewalls drop idle connections; the remote service may need keepalive + +## License + +MIT diff --git a/devbox-connect/pyproject.toml b/devbox-connect/pyproject.toml new file mode 100644 index 0000000..93dd1ea --- /dev/null +++ b/devbox-connect/pyproject.toml @@ -0,0 +1,31 @@ +[project] +name = "devbox-connect" +version = "0.1.0" +description = "SSH tunnel manager for connecting to devbox ports" +readme = "README.md" +requires-python = ">=3.10" +dependencies = [ + "paramiko>=3.4.0", + "pyyaml>=6.0", +] + +[project.scripts] +devbox-connect = "devbox_connect.cli:main" + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["src/devbox_connect"] + +[tool.ruff] +line-length = 100 +target-version = "py310" + +[tool.ruff.lint] +select = ["E", "F", "I", "W"] + +[tool.pyright] +pythonVersion = "3.10" +typeCheckingMode = "standard" diff --git a/devbox-connect/service/install-service.ps1 b/devbox-connect/service/install-service.ps1 new file mode 100644 index 0000000..6f88928 --- /dev/null +++ b/devbox-connect/service/install-service.ps1 @@ -0,0 +1,202 @@ +# install-service.ps1 - Install devbox-connect as a Windows service using NSSM +# +# Prerequisites: +# 1. Install NSSM: winget install nssm +# 2. Install devbox-connect: uv tool install . +# +# Usage (run as Administrator): +# .\install-service.ps1 -ConfigPath C:\path\to\tunnels.yaml +# .\install-service.ps1 -Uninstall +# .\install-service.ps1 -Status + +param( + [string]$ConfigPath, + [switch]$Uninstall, + [switch]$Status, + [string]$ServiceName = "DevboxConnect" +) + +$ErrorActionPreference = "Stop" + +function Test-Administrator { + $identity = [Security.Principal.WindowsIdentity]::GetCurrent() + $principal = New-Object Security.Principal.WindowsPrincipal($identity) + return $principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator) +} + +function Get-NssmPath { + $nssm = Get-Command nssm -ErrorAction SilentlyContinue + if ($nssm) { + return $nssm.Source + } + + # Check common locations + $paths = @( + "C:\Program Files\nssm\nssm.exe", + "C:\Program Files (x86)\nssm\nssm.exe", + "$env:LOCALAPPDATA\Microsoft\WinGet\Packages\*nssm*\nssm.exe" + ) + + foreach ($path in $paths) { + $found = Get-Item $path -ErrorAction SilentlyContinue | Select-Object -First 1 + if ($found) { + return $found.FullName + } + } + + return $null +} + +function Get-DevboxConnectPath { + # Find the uv-installed devbox-connect + $uvToolPath = "$env:LOCALAPPDATA\uv\tools\devbox-connect" + if (Test-Path $uvToolPath) { + # Find the Python executable in the venv + $pythonPath = Join-Path $uvToolPath "Scripts\python.exe" + if (Test-Path $pythonPath) { + return @{ + Python = $pythonPath + Module = "devbox_connect.cli" + } + } + } + + # Try to find via uv tool list + $uvPath = Get-Command uv -ErrorAction SilentlyContinue + if ($uvPath) { + $toolDir = & uv tool dir 2>$null + if ($toolDir -and (Test-Path "$toolDir\devbox-connect")) { + $pythonPath = Join-Path $toolDir "devbox-connect\Scripts\python.exe" + if (Test-Path $pythonPath) { + return @{ + Python = $pythonPath + Module = "devbox_connect.cli" + } + } + } + } + + return $null +} + +# Check for admin rights +if (-not (Test-Administrator)) { + Write-Error "This script must be run as Administrator" + exit 1 +} + +# Handle status check +if ($Status) { + $service = Get-Service -Name $ServiceName -ErrorAction SilentlyContinue + if ($service) { + Write-Host "Service '$ServiceName' status: $($service.Status)" + Write-Host "Startup type: $($service.StartType)" + } else { + Write-Host "Service '$ServiceName' is not installed" + } + exit 0 +} + +# Handle uninstall +if ($Uninstall) { + $nssm = Get-NssmPath + if (-not $nssm) { + Write-Error "NSSM not found. Install with: winget install nssm" + exit 1 + } + + Write-Host "Stopping service..." + & $nssm stop $ServiceName 2>$null + + Write-Host "Removing service..." + & $nssm remove $ServiceName confirm + + Write-Host "Service '$ServiceName' removed" + exit 0 +} + +# Install service +if (-not $ConfigPath) { + Write-Error "ConfigPath is required. Usage: .\install-service.ps1 -ConfigPath C:\path\to\tunnels.yaml" + exit 1 +} + +if (-not (Test-Path $ConfigPath)) { + Write-Error "Config file not found: $ConfigPath" + exit 1 +} + +$ConfigPath = (Resolve-Path $ConfigPath).Path + +# Find NSSM +$nssm = Get-NssmPath +if (-not $nssm) { + Write-Error @" +NSSM (Non-Sucking Service Manager) not found. + +Install with: + winget install nssm + +Or download from: https://nssm.cc/download +"@ + exit 1 +} + +# Find devbox-connect +$devboxConnect = Get-DevboxConnectPath +if (-not $devboxConnect) { + Write-Error @" +devbox-connect not found. Install with: + uv tool install . + +Or from a directory containing the project: + uv tool install /path/to/devbox-connect +"@ + exit 1 +} + +Write-Host "Found devbox-connect at: $($devboxConnect.Python)" +Write-Host "Config file: $ConfigPath" +Write-Host "" + +# Remove existing service if present +$existing = Get-Service -Name $ServiceName -ErrorAction SilentlyContinue +if ($existing) { + Write-Host "Removing existing service..." + & $nssm stop $ServiceName 2>$null + & $nssm remove $ServiceName confirm +} + +# Install the service +Write-Host "Installing service '$ServiceName'..." + +& $nssm install $ServiceName $devboxConnect.Python "-m" $devboxConnect.Module "-c" $ConfigPath "start" + +# Configure service +& $nssm set $ServiceName DisplayName "Devbox Connect SSH Tunnels" +& $nssm set $ServiceName Description "Maintains SSH tunnels to devbox for port forwarding" +& $nssm set $ServiceName Start SERVICE_AUTO_START +& $nssm set $ServiceName AppStdout "$env:LOCALAPPDATA\devbox-connect\service.log" +& $nssm set $ServiceName AppStderr "$env:LOCALAPPDATA\devbox-connect\service.log" +& $nssm set $ServiceName AppRotateFiles 1 +& $nssm set $ServiceName AppRotateBytes 1048576 + +# Create log directory +$logDir = "$env:LOCALAPPDATA\devbox-connect" +if (-not (Test-Path $logDir)) { + New-Item -ItemType Directory -Path $logDir | Out-Null +} + +Write-Host "" +Write-Host "Service '$ServiceName' installed successfully!" +Write-Host "" +Write-Host "Commands:" +Write-Host " Start: Start-Service $ServiceName" +Write-Host " Stop: Stop-Service $ServiceName" +Write-Host " Status: Get-Service $ServiceName" +Write-Host " Logs: Get-Content $logDir\service.log -Tail 50" +Write-Host "" +Write-Host "Starting service..." + +Start-Service $ServiceName +Get-Service $ServiceName diff --git a/devbox-connect/service/run-manual.bat b/devbox-connect/service/run-manual.bat new file mode 100644 index 0000000..5e35be1 --- /dev/null +++ b/devbox-connect/service/run-manual.bat @@ -0,0 +1,19 @@ +@echo off +REM run-manual.bat - Run devbox-connect manually (not as a service) +REM +REM Usage: +REM run-manual.bat - Uses tunnels.yaml in current directory +REM run-manual.bat C:\path\to\config - Uses specified config file + +setlocal + +set CONFIG=%1 +if "%CONFIG%"=="" set CONFIG=tunnels.yaml + +echo Starting devbox-connect with config: %CONFIG% +echo Press Ctrl+C to stop +echo. + +devbox-connect -c "%CONFIG%" start + +pause diff --git a/devbox-connect/src/devbox_connect/__init__.py b/devbox-connect/src/devbox_connect/__init__.py new file mode 100644 index 0000000..0612a4c --- /dev/null +++ b/devbox-connect/src/devbox_connect/__init__.py @@ -0,0 +1,3 @@ +"""SSH tunnel manager for connecting to devbox ports.""" + +__version__ = "0.1.0" diff --git a/devbox-connect/src/devbox_connect/cli.py b/devbox-connect/src/devbox_connect/cli.py new file mode 100644 index 0000000..a14d4ce --- /dev/null +++ b/devbox-connect/src/devbox_connect/cli.py @@ -0,0 +1,235 @@ +"""Command-line interface for devbox-connect.""" + +import argparse +import logging +import signal +import sys +import time +from pathlib import Path + +from .config import ConfigError, load_config +from .tunnel import TunnelManager, TunnelState + +# ANSI color codes +GREEN = "\033[92m" +RED = "\033[91m" +YELLOW = "\033[93m" +CYAN = "\033[96m" +RESET = "\033[0m" +BOLD = "\033[1m" + +# State to color mapping +STATE_COLORS = { + TunnelState.CONNECTED: GREEN, + TunnelState.CONNECTING: YELLOW, + TunnelState.DISCONNECTED: RED, + TunnelState.ERROR: RED, +} + +STATE_SYMBOLS = { + TunnelState.CONNECTED: "[OK]", + TunnelState.CONNECTING: "[..]", + TunnelState.DISCONNECTED: "[--]", + TunnelState.ERROR: "[!!]", +} + + +def setup_logging(verbose: bool) -> None: + """Configure logging.""" + level = logging.DEBUG if verbose else logging.INFO + format_str = "%(asctime)s - %(levelname)s - %(message)s" if verbose else "%(message)s" + logging.basicConfig(level=level, format=format_str) + + +def print_banner() -> None: + """Print application banner.""" + print(f"{BOLD}{CYAN}devbox-connect{RESET} - SSH Tunnel Manager") + print() + + +def print_status(manager: TunnelManager) -> None: + """Print current tunnel status.""" + status = manager.get_status() + + for host_key, tunnels in status.items(): + print(f"{BOLD}{host_key}{RESET}") + for name, tunnel_status in tunnels.items(): + state = tunnel_status.state + color = STATE_COLORS[state] + symbol = STATE_SYMBOLS[state] + local_port = tunnel_status.config.local_port or tunnel_status.config.remote_port + remote = f"{tunnel_status.config.remote_host}:{tunnel_status.config.remote_port}" + + line = f" {color}{symbol}{RESET} {name}: localhost:{local_port} -> {remote}" + + if tunnel_status.connections > 0: + line += f" ({tunnel_status.connections} active)" + + if tunnel_status.error: + line += f" {RED}({tunnel_status.error}){RESET}" + + print(line) + print() + + +def on_status_change(host: str, tunnel: str, state: TunnelState) -> None: + """Callback for tunnel status changes.""" + color = STATE_COLORS[state] + symbol = STATE_SYMBOLS[state] + logging.info(f"{color}{symbol}{RESET} {host}/{tunnel}: {state.value}") + + +def run_manager(config_path: Path, verbose: bool) -> int: + """Run the tunnel manager.""" + setup_logging(verbose) + print_banner() + + # Load configuration + try: + config = load_config(config_path) + except ConfigError as e: + logging.error(f"Configuration error: {e}") + return 1 + + if not config.hosts: + logging.error("No tunnels configured") + return 1 + + # Count tunnels + total_tunnels = sum(len(h.tunnels) for h in config.hosts) + logging.info(f"Loaded {total_tunnels} tunnel(s) for {len(config.hosts)} host(s)") + print() + + # Create and start manager + manager = TunnelManager(config, on_status_change=on_status_change) + + # Handle shutdown signals + shutdown_requested = False + + def signal_handler(signum: int, frame: object) -> None: + nonlocal shutdown_requested + if shutdown_requested: + # Force exit on second signal + sys.exit(1) + shutdown_requested = True + print(f"\n{YELLOW}Shutting down...{RESET}") + manager.stop() + + signal.signal(signal.SIGINT, signal_handler) + signal.signal(signal.SIGTERM, signal_handler) + + # Start tunnels + manager.start() + + # Wait a moment for connections + time.sleep(2) + + # Print initial status + print_status(manager) + print(f"{CYAN}Press Ctrl+C to stop{RESET}") + print() + + # Keep running until shutdown + try: + while not shutdown_requested: + time.sleep(1) + except KeyboardInterrupt: + pass + + manager.stop() + print(f"{GREEN}Stopped{RESET}") + return 0 + + +def show_status(config_path: Path) -> int: + """Show status of configured tunnels (quick check).""" + setup_logging(verbose=False) + + try: + config = load_config(config_path) + except ConfigError as e: + logging.error(f"Configuration error: {e}") + return 1 + + print_banner() + print(f"Config: {config_path}") + print() + + for host in config.hosts: + print(f"{BOLD}{host.user}@{host.host}:{host.port}{RESET}") + for tunnel in host.tunnels: + local_port = tunnel.local_port or tunnel.remote_port + remote = f"{tunnel.remote_host}:{tunnel.remote_port}" + print(f" - {tunnel.name}: localhost:{local_port} -> {remote}") + print() + + return 0 + + +def validate_config(config_path: Path) -> int: + """Validate configuration file.""" + setup_logging(verbose=False) + + try: + config = load_config(config_path) + total_tunnels = sum(len(h.tunnels) for h in config.hosts) + print(f"{GREEN}Configuration valid{RESET}") + print(f" Hosts: {len(config.hosts)}") + print(f" Tunnels: {total_tunnels}") + return 0 + except ConfigError as e: + print(f"{RED}Configuration invalid: {e}{RESET}") + return 1 + + +def main() -> int: + """Main entry point.""" + parser = argparse.ArgumentParser( + prog="devbox-connect", + description="SSH tunnel manager for connecting to devbox ports", + ) + parser.add_argument( + "-c", + "--config", + type=Path, + default=Path("tunnels.yaml"), + help="Path to configuration file (default: tunnels.yaml)", + ) + parser.add_argument( + "-v", + "--verbose", + action="store_true", + help="Enable verbose output", + ) + + subparsers = parser.add_subparsers(dest="command", help="Commands") + + # start command (default) + start_parser = subparsers.add_parser("start", help="Start tunnels (default)") + start_parser.add_argument("-v", "--verbose", action="store_true") + + # status command + subparsers.add_parser("status", help="Show configured tunnels") + + # validate command + subparsers.add_parser("validate", help="Validate configuration file") + + args = parser.parse_args() + + # Handle default command + command = args.command or "start" + + if command == "start": + verbose = getattr(args, "verbose", False) + return run_manager(args.config, verbose) + elif command == "status": + return show_status(args.config) + elif command == "validate": + return validate_config(args.config) + else: + parser.print_help() + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/devbox-connect/src/devbox_connect/config.py b/devbox-connect/src/devbox_connect/config.py new file mode 100644 index 0000000..1fe2c46 --- /dev/null +++ b/devbox-connect/src/devbox_connect/config.py @@ -0,0 +1,162 @@ +"""Configuration loading and validation.""" + +from dataclasses import dataclass, field +from pathlib import Path + +import yaml + + +@dataclass +class TunnelConfig: + """Configuration for a single SSH tunnel.""" + + name: str + host: str + remote_port: int + local_port: int | None = None # Defaults to remote_port if not specified + user: str | None = None # Defaults to global user + key_file: str | None = None # Defaults to global key_file + remote_host: str = "localhost" # The host on the remote side to connect to + + def __post_init__(self) -> None: + if self.local_port is None: + self.local_port = self.remote_port + + +@dataclass +class HostConfig: + """Configuration for a host with multiple tunnels.""" + + host: str + user: str + key_file: str | None = None + port: int = 22 + tunnels: list[TunnelConfig] = field(default_factory=list) + + +@dataclass +class Config: + """Root configuration.""" + + hosts: list[HostConfig] = field(default_factory=list) + reconnect_delay: int = 5 # Seconds between reconnection attempts + max_reconnect_delay: int = 60 # Max delay with exponential backoff + + +class ConfigError(Exception): + """Configuration error.""" + + pass + + +def load_config(path: Path) -> Config: + """Load configuration from a YAML file.""" + if not path.exists(): + raise ConfigError(f"Config file not found: {path}") + + with open(path) as f: + data = yaml.safe_load(f) + + if not data: + raise ConfigError("Empty configuration file") + + return _parse_config(data) + + +def _parse_config(data: dict) -> Config: + """Parse configuration dictionary into Config object.""" + config = Config( + reconnect_delay=data.get("reconnect_delay", 5), + max_reconnect_delay=data.get("max_reconnect_delay", 60), + ) + + # Handle simple format: list of tunnels with host info per tunnel + if "tunnels" in data: + config.hosts = _parse_simple_format(data["tunnels"], data) + # Handle grouped format: hosts with nested tunnels + elif "hosts" in data: + config.hosts = _parse_grouped_format(data["hosts"]) + else: + raise ConfigError("Config must contain either 'tunnels' or 'hosts' key") + + return config + + +def _parse_simple_format(tunnels: list[dict], global_config: dict) -> list[HostConfig]: + """Parse simple format where each tunnel specifies its host.""" + # Group tunnels by host + hosts_map: dict[str, HostConfig] = {} + + global_user = global_config.get("user") + global_key_file = global_config.get("key_file") + + for t in tunnels: + if "host" not in t: + raise ConfigError(f"Tunnel '{t.get('name', 'unnamed')}' missing 'host'") + if "remote_port" not in t: + raise ConfigError(f"Tunnel '{t.get('name', 'unnamed')}' missing 'remote_port'") + + host = t["host"] + user = t.get("user", global_user) + if not user: + raise ConfigError(f"Tunnel '{t.get('name', 'unnamed')}' missing 'user'") + + key_file = t.get("key_file", global_key_file) + ssh_port = t.get("ssh_port", 22) + + # Create host key for grouping + host_key = f"{user}@{host}:{ssh_port}" + + if host_key not in hosts_map: + hosts_map[host_key] = HostConfig( + host=host, + user=user, + key_file=key_file, + port=ssh_port, + ) + + tunnel = TunnelConfig( + name=t.get("name", f"{host}:{t['remote_port']}"), + host=host, + remote_port=t["remote_port"], + local_port=t.get("local_port"), + remote_host=t.get("remote_host", "localhost"), + ) + hosts_map[host_key].tunnels.append(tunnel) + + return list(hosts_map.values()) + + +def _parse_grouped_format(hosts: list[dict]) -> list[HostConfig]: + """Parse grouped format with hosts containing nested tunnels.""" + result = [] + + for h in hosts: + if "host" not in h: + raise ConfigError("Host entry missing 'host' field") + if "user" not in h: + raise ConfigError(f"Host '{h['host']}' missing 'user' field") + + host_config = HostConfig( + host=h["host"], + user=h["user"], + key_file=h.get("key_file"), + port=h.get("port", 22), + ) + + for t in h.get("tunnels", []): + if "remote_port" not in t: + raise ConfigError(f"Tunnel in host '{h['host']}' missing 'remote_port'") + + tunnel = TunnelConfig( + name=t.get("name", f"{h['host']}:{t['remote_port']}"), + host=h["host"], + remote_port=t["remote_port"], + local_port=t.get("local_port"), + remote_host=t.get("remote_host", "localhost"), + ) + host_config.tunnels.append(tunnel) + + result.append(host_config) + + return result diff --git a/devbox-connect/src/devbox_connect/tunnel.py b/devbox-connect/src/devbox_connect/tunnel.py new file mode 100644 index 0000000..aceacaf --- /dev/null +++ b/devbox-connect/src/devbox_connect/tunnel.py @@ -0,0 +1,355 @@ +"""SSH tunnel manager using paramiko.""" + +import logging +import socket +import threading +import time +from dataclasses import dataclass, field +from enum import Enum +from pathlib import Path +from typing import Callable + +import paramiko + +from .config import Config, HostConfig, TunnelConfig + +logger = logging.getLogger(__name__) + + +class TunnelState(Enum): + """State of a tunnel.""" + + DISCONNECTED = "disconnected" + CONNECTING = "connecting" + CONNECTED = "connected" + ERROR = "error" + + +@dataclass +class TunnelStatus: + """Status of a single tunnel.""" + + config: TunnelConfig + state: TunnelState = TunnelState.DISCONNECTED + error: str | None = None + connections: int = 0 # Active forwarded connections + + +@dataclass +class HostConnection: + """Manages SSH connection and tunnels for a single host.""" + + config: HostConfig + tunnels: dict[str, TunnelStatus] = field(default_factory=dict) + client: paramiko.SSHClient | None = None + transport: paramiko.Transport | None = None + _forward_threads: list[threading.Thread] = field(default_factory=list) + _forward_servers: list[socket.socket] = field(default_factory=list) + _stop_event: threading.Event = field(default_factory=threading.Event) + + def __post_init__(self) -> None: + for tunnel in self.config.tunnels: + self.tunnels[tunnel.name] = TunnelStatus(config=tunnel) + + +class TunnelManager: + """Manages SSH tunnels based on configuration.""" + + def __init__( + self, + config: Config, + on_status_change: Callable[[str, str, TunnelState], None] | None = None, + ): + self.config = config + self.on_status_change = on_status_change + self.hosts: dict[str, HostConnection] = {} + self._stop_event = threading.Event() + self._reconnect_threads: list[threading.Thread] = [] + + # Initialize host connections + for host_config in config.hosts: + key = f"{host_config.user}@{host_config.host}" + self.hosts[key] = HostConnection(config=host_config) + + def start(self) -> None: + """Start all tunnels.""" + self._stop_event.clear() + for host_key, host_conn in self.hosts.items(): + self._connect_host(host_key, host_conn) + + def stop(self) -> None: + """Stop all tunnels.""" + logger.info("Stopping all tunnels...") + self._stop_event.set() + + for host_conn in self.hosts.values(): + self._disconnect_host(host_conn) + + # Wait for reconnect threads to finish + for thread in self._reconnect_threads: + thread.join(timeout=2) + self._reconnect_threads.clear() + + def get_status(self) -> dict[str, dict[str, TunnelStatus]]: + """Get status of all tunnels.""" + return {host_key: host.tunnels for host_key, host in self.hosts.items()} + + def _connect_host(self, host_key: str, host_conn: HostConnection) -> bool: + """Connect to a host and establish all tunnels.""" + config = host_conn.config + + # Update all tunnel states to connecting + for tunnel_status in host_conn.tunnels.values(): + self._update_state(host_key, tunnel_status, TunnelState.CONNECTING) + + try: + # Create SSH client + client = paramiko.SSHClient() + client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + + # Prepare connection kwargs + connect_kwargs: dict = { + "hostname": config.host, + "port": config.port, + "username": config.user, + } + + # Add key file if specified + if config.key_file: + key_path = Path(config.key_file).expanduser() + if not key_path.exists(): + raise FileNotFoundError(f"SSH key file not found: {key_path}") + connect_kwargs["key_filename"] = str(key_path) + + logger.info(f"Connecting to {config.user}@{config.host}:{config.port}...") + client.connect(**connect_kwargs) + + host_conn.client = client + host_conn.transport = client.get_transport() + + if host_conn.transport is None: + raise ConnectionError("Failed to get transport") + + # Start port forwarding for each tunnel + host_conn._stop_event.clear() + for tunnel_config in config.tunnels: + self._start_tunnel(host_key, host_conn, tunnel_config) + + logger.info(f"Connected to {config.host}") + return True + + except Exception as e: + logger.error(f"Failed to connect to {config.host}: {e}") + for tunnel_status in host_conn.tunnels.values(): + tunnel_status.error = str(e) + self._update_state(host_key, tunnel_status, TunnelState.ERROR) + self._schedule_reconnect(host_key, host_conn) + return False + + def _disconnect_host(self, host_conn: HostConnection) -> None: + """Disconnect from a host.""" + host_conn._stop_event.set() + + # Close forward servers + for server in host_conn._forward_servers: + try: + server.close() + except Exception: + pass + host_conn._forward_servers.clear() + + # Wait for forward threads + for thread in host_conn._forward_threads: + thread.join(timeout=1) + host_conn._forward_threads.clear() + + # Close SSH connection + if host_conn.client: + try: + host_conn.client.close() + except Exception: + pass + host_conn.client = None + host_conn.transport = None + + def _start_tunnel( + self, host_key: str, host_conn: HostConnection, tunnel_config: TunnelConfig + ) -> None: + """Start a single port forward tunnel.""" + tunnel_status = host_conn.tunnels[tunnel_config.name] + + try: + # Create local listening socket + local_port = tunnel_config.local_port or tunnel_config.remote_port + server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + server.bind(("127.0.0.1", local_port)) + server.listen(5) + server.settimeout(1.0) # Allow checking stop event + + host_conn._forward_servers.append(server) + + # Start accept thread + thread = threading.Thread( + target=self._accept_loop, + args=(host_key, host_conn, tunnel_config, server), + daemon=True, + ) + thread.start() + host_conn._forward_threads.append(thread) + + self._update_state(host_key, tunnel_status, TunnelState.CONNECTED) + logger.info( + f"Tunnel '{tunnel_config.name}' open: " + f"localhost:{local_port} -> {tunnel_config.remote_host}:{tunnel_config.remote_port}" + ) + + except Exception as e: + logger.error(f"Failed to start tunnel '{tunnel_config.name}': {e}") + tunnel_status.error = str(e) + self._update_state(host_key, tunnel_status, TunnelState.ERROR) + + def _accept_loop( + self, + host_key: str, + host_conn: HostConnection, + tunnel_config: TunnelConfig, + server: socket.socket, + ) -> None: + """Accept loop for forwarding connections.""" + tunnel_status = host_conn.tunnels[tunnel_config.name] + + while not host_conn._stop_event.is_set() and not self._stop_event.is_set(): + try: + client_socket, addr = server.accept() + except socket.timeout: + continue + except OSError: + break # Socket was closed + + # Check if transport is still active + if host_conn.transport is None or not host_conn.transport.is_active(): + client_socket.close() + self._update_state(host_key, tunnel_status, TunnelState.ERROR) + tunnel_status.error = "SSH connection lost" + break + + # Open channel to remote + try: + channel = host_conn.transport.open_channel( + "direct-tcpip", + (tunnel_config.remote_host, tunnel_config.remote_port), + client_socket.getpeername(), + ) + except Exception as e: + logger.error(f"Failed to open channel for {tunnel_config.name}: {e}") + client_socket.close() + continue + + if channel is None: + logger.error(f"Failed to open channel for {tunnel_config.name}") + client_socket.close() + continue + + # Start forwarding thread + tunnel_status.connections += 1 + thread = threading.Thread( + target=self._forward_data, + args=(client_socket, channel, tunnel_status), + daemon=True, + ) + thread.start() + + # Connection lost, trigger reconnect + if not self._stop_event.is_set(): + self._schedule_reconnect(host_key, host_conn) + + def _forward_data( + self, + client_socket: socket.socket, + channel: paramiko.Channel, + tunnel_status: TunnelStatus, + ) -> None: + """Forward data between client socket and SSH channel.""" + try: + while True: + # Check both directions for data + r_ready = [] + + # Use select for multiplexing + import select + + try: + r_ready, _, _ = select.select([client_socket, channel], [], [], 1.0) + except Exception: + break + + if client_socket in r_ready: + data = client_socket.recv(4096) + if not data: + break + channel.send(data) + + if channel in r_ready: + data = channel.recv(4096) + if not data: + break + client_socket.send(data) + + except Exception as e: + logger.debug(f"Forward connection closed: {e}") + finally: + tunnel_status.connections -= 1 + try: + client_socket.close() + except Exception: + pass + try: + channel.close() + except Exception: + pass + + def _schedule_reconnect(self, host_key: str, host_conn: HostConnection) -> None: + """Schedule a reconnection attempt.""" + if self._stop_event.is_set(): + return + + thread = threading.Thread( + target=self._reconnect_loop, + args=(host_key, host_conn), + daemon=True, + ) + thread.start() + self._reconnect_threads.append(thread) + + def _reconnect_loop(self, host_key: str, host_conn: HostConnection) -> None: + """Reconnection loop with exponential backoff.""" + delay = self.config.reconnect_delay + + while not self._stop_event.is_set(): + logger.info(f"Reconnecting to {host_conn.config.host} in {delay}s...") + + # Wait with stop check + for _ in range(delay): + if self._stop_event.is_set(): + return + time.sleep(1) + + # Disconnect first + self._disconnect_host(host_conn) + + # Try to connect + if self._connect_host(host_key, host_conn): + return # Success + + # Exponential backoff + delay = min(delay * 2, self.config.max_reconnect_delay) + + def _update_state(self, host_key: str, tunnel_status: TunnelStatus, state: TunnelState) -> None: + """Update tunnel state and notify callback.""" + tunnel_status.state = state + if state != TunnelState.ERROR: + tunnel_status.error = None + + if self.on_status_change: + self.on_status_change(host_key, tunnel_status.config.name, state) diff --git a/devbox-connect/tunnels.example.yaml b/devbox-connect/tunnels.example.yaml new file mode 100644 index 0000000..fd3c307 --- /dev/null +++ b/devbox-connect/tunnels.example.yaml @@ -0,0 +1,66 @@ +# devbox-connect configuration example +# +# Two formats are supported: +# 1. Simple format - flat list of tunnels (good for single host) +# 2. Grouped format - tunnels organized by host (good for multiple hosts) + +# ============================================================================= +# SIMPLE FORMAT (recommended for single devbox) +# ============================================================================= + +# Global settings (can be overridden per tunnel) +user: your-username +key_file: ~/.ssh/id_rsa # Optional: defaults to SSH agent or default keys + +# Reconnection settings +reconnect_delay: 5 # Initial delay between reconnect attempts (seconds) +max_reconnect_delay: 60 # Maximum delay with exponential backoff + +tunnels: + # Web development server + - name: web-dev + host: devbox.example.com + remote_port: 8080 + # local_port: 8080 # Optional: defaults to remote_port + + # Database + - name: postgres + host: devbox.example.com + remote_port: 5432 + + # Jupyter notebook + - name: jupyter + host: devbox.example.com + remote_port: 8888 + local_port: 9999 # Use different local port to avoid conflicts + + # Access a service running on a different host via devbox + - name: internal-api + host: devbox.example.com + remote_port: 8000 + remote_host: internal-server.local # Forward to a different host through SSH + + +# ============================================================================= +# GROUPED FORMAT (for multiple hosts) +# ============================================================================= +# Uncomment below and comment out the 'tunnels' section above to use this format + +# hosts: +# - host: devbox1.example.com +# user: your-username +# key_file: ~/.ssh/id_rsa +# port: 22 # SSH port, default 22 +# tunnels: +# - name: web +# remote_port: 8080 +# - name: db +# remote_port: 5432 +# +# - host: devbox2.example.com +# user: your-username +# tunnels: +# - name: api +# remote_port: 3000 +# - name: redis +# remote_port: 6379 diff --git a/mboxer b/mboxer new file mode 160000 index 0000000..22f46b6 --- /dev/null +++ b/mboxer @@ -0,0 +1 @@ +Subproject commit 22f46b6fd16248cf3dfc44950f3ec163df6eac36 diff --git a/notification-bridge b/notification-bridge new file mode 160000 index 0000000..3087d40 --- /dev/null +++ b/notification-bridge @@ -0,0 +1 @@ +Subproject commit 3087d4066baa55ce37f9e73c75092a031eb1fd0b diff --git a/recommendations.md b/recommendations.md new file mode 100644 index 0000000..e486b7b --- /dev/null +++ b/recommendations.md @@ -0,0 +1,46 @@ +# Scaling Recommendations + +## 1. Extract a shared library + +The same patterns are already duplicated in central-context and notification-bridge: `BaseSettings` with `.env`, FastAPI lifespan boilerplate, uvicorn entry points, error-to-HTTP-exception translation, test fixtures with temp dirs and settings overrides. At dozens of services this becomes a maintenance problem — fix a bug in the pattern and you're patching it everywhere. + +A `castle-core` (or similar) package that provides a base settings class, standard lifespan wiring, common test fixtures, and health check endpoint would let new services start from ~10 lines of setup code. + +## 2. Standardize project layout + +Right now there are three different layouts: `src/central_context/`, flat `notification_bridge/`, and single-file `convert.py`. Pick one (the `src/` layout is the most robust) and stick with it. This matters because any top-level tooling that iterates over projects needs predictable structure. + +## 3. Top-level task runner + +With dozens of projects, there needs to be a way to run commands across all or some of them. A root `Makefile`, `justfile`, or script that can do things like: +- `make test` — run all tests +- `make test p=central-context` — run one project's tests +- `make lint` — lint everything +- `make sync` — `uv sync` in all projects + +Without this, significant time gets spent just navigating and running repetitive commands. + +## 4. Port and service registry + +Ports are currently hardcoded defaults (9000, 9001). With dozens of services, there needs to be a single source of truth for port assignments — even if it's just a `services.yaml` at the repo root that maps service names to ports. + +## 5. Inter-service configuration + +notification-bridge hardcodes `http://localhost:9000` as the central-context URL. This pattern doesn't scale — each new service that talks to another service adds more hardcoded URLs in more `.env` files. Consider either: +- A convention like `CASTLE_{SERVICE_NAME}_URL` derived from the registry +- A shared config that generates per-service `.env` files + +## 6. Consistent ruff/pyright configuration + +Each project currently has its own ruff rules (devbox-connect selects `E,F,I,W` while mboxer selects `ALL`). With dozens of projects, either put a shared `ruff.toml` at the repo root (ruff walks up to find config) or decide on one standard. Same for pyright — only devbox-connect has it enabled currently. + +## 7. What to defer + +- **Containerization/orchestration** — until deploying somewhere beyond the local machine +- **API gateway / service mesh** — premature until there are actual traffic patterns +- **Distributed tracing / observability** — add when debugging cross-service issues becomes painful +- **Formal API schema sharing** (OpenAPI contracts between services) — FastAPI generates these already; formalize when there are consumers that need stability guarantees + +## Priority + +The highest-leverage first step is the shared library + top-level task runner, since those reduce the marginal cost of adding each new service.