# Wild Cloud App Lifecycle: State and Operations ## Overview Wild Cloud manages applications across multiple independent systems with different consistency guarantees. Understanding these systems, their interactions, and how app packages are structured is critical for reliable app lifecycle management. This document covers: - **System architecture**: The three independent systems managing app state - **User workflows**: Two distinct approaches (git-based vs Web UI) - **App package structure**: How apps are defined in Wild Directory - **State lifecycle**: Complete state transitions from add to delete - **Operations**: How each lifecycle operation works across systems - **Edge cases**: Common failure modes and automatic recovery ## User Workflows Wild Cloud supports two fundamentally different workflows for managing app lifecycle: ### Advanced Users: Git-Based Infrastructure-as-Code **Target Audience**: DevOps engineers, systems administrators, users comfortable with git and command-line tools. **Key Characteristics**: - Instance data directory is a git repository - Wild Directory tracked as upstream remote - Manual edits tracked in git with commit messages - Wild Directory updates merged using standard git workflows - Full version control and audit trail - SSH/command-line access to Wild Central device **Typical Workflow**: ```bash # Clone instance repository git clone user@wild-central:/var/lib/wild-central/instances/my-cloud # Make custom changes vim apps/myapp/deployment.yaml git commit -m "Increase CPU limits for production" # Merge upstream Wild Directory updates git remote add wild-directory https://github.com/wildcloud/wild-directory.git git fetch wild-directory git merge wild-directory/main # Resolve conflicts if needed # Deploy changes wild app deploy myapp ``` **Philosophy**: Treat cluster configuration like application code - version controlled, reviewed, tested, and deployed through established git workflows. ### Regular Users: Web UI-Based Management **Target Audience**: Non-technical users, small teams, users who prefer graphical interfaces. **Key Characteristics**: - All management through Web UI or CLI (no SSH access) - Configuration changes via forms (config.yaml, secrets.yaml) - Wild Directory updates applied automatically with config merging - Cannot directly edit manifest files (prevents divergence) - Simplified workflow with automatic safety checks **Typical Workflow**: 1. Browse available apps in Web UI 2. Click "Add" to add app to instance 3. Configure via form fields (port, storage, domain, etc.) 4. Click "Deploy" to deploy to cluster 5. System notifies when Wild Directory updates available 6. Click "Update" to merge changes (config preserved) 7. Review changes in diff view 8. Click "Deploy" to apply updates **Philosophy**: Abstract away complexity - users manage apps like installing software, not like managing infrastructure code. ### Key Differences | Aspect | Advanced Users (Git) | Regular Users (Web UI) | |--------|----------------------|------------------------| | **Access** | SSH + command line | Web UI + CLI | | **Manifest Editing** | Direct file editing | Via config forms only | | **Version Control** | Git (full history) | System managed | | **Wild Directory Updates** | Manual git merge | Automatic merge with review | | **Customization** | Unlimited | Configuration-based only | | **Drift** | Intentional (git-tracked) | Unintentional (reconcile) | | **Collaboration** | Git branches/PRs | Shared Web UI access | | **Rollback** | `git revert` | Re-deploy previous state | The rest of this document covers both workflows, with sections clearly marked for each user type where behavior differs. ## System Architecture ### The Multi-System Challenge Wild Cloud app state spans **three independent systems**: 1. **Wild Directory** (Source of Truth) - Location: `/path/to/wild-directory/{app-name}/` - Consistency: Immutable, version controlled - Purpose: Template definitions shared across all instances 2. **Instance Data** (Local State) - Location: `/path/to/data-dir/instances/{instance}/` - Consistency: Immediately consistent, file-system based - Purpose: Instance-specific configuration and compiled manifests 3. **Kubernetes Cluster** (Runtime State) - Location: Kubernetes API and etcd - Consistency: Eventually consistent - Purpose: Running application workloads **Critical Insight**: These systems have fundamentally different consistency models, creating inherent challenges for atomic operations across system boundaries. ## State Components ### 1. Wild Directory (Immutable Source) ``` wild-directory/ └── {app-name}/ ├── manifest.yaml # App metadata, dependencies, defaults ├── kustomization.yaml # Kustomize configuration ├── deployment.yaml # Kubernetes workload (template) ├── service.yaml # Kubernetes service (template) ├── ingress.yaml # Kubernetes ingress (template) ├── namespace.yaml # Namespace definition (template) ├── pvc.yaml # Storage claims (template) ├── db-init-job.yaml # Database initialization (optional) └── README.md # Documentation ``` **Characteristics**: - Read-only during operations - Contains gomplate template variables: `{{ .cloud.domain }}`, `{{ .app.port }}` - Shared across all Wild Cloud instances - Version controlled (git) #### App Manifest Structure The `manifest.yaml` file defines everything about an app: ```yaml name: myapp # App identifier (matches directory name) is: myapp # Unique app type identifier description: Brief description version: 1.0.0 # Follow upstream versioning icon: https://example.com/icon.svg requires: # Dependencies (optional) - name: postgres # Dependency app type (matches 'is' field) alias: db # Optional: reference name in templates - name: redis # No alias = use 'redis' as reference defaultConfig: # Merged into instance config.yaml namespace: myapp image: myapp/myapp:latest port: "8080" storage: 10Gi domain: myapp.{{ .cloud.domain }} # Can reference dependencies: dbHost: "{{ .apps.db.host }}" redisHost: "{{ .apps.redis.host }}" defaultSecrets: # App's own secrets - key: apiKey # Auto-generated random if no default - key: dbUrl # Can use template with config/secrets default: "postgresql://{{ .app.dbUser }}:{{ .secrets.dbPassword }}@{{ .app.dbHost }}/{{ .app.dbName }}" requiredSecrets: # Secrets from dependencies - db.password # Format: . - redis.auth # Copied from dependency's secrets ``` **Template Variable Resolution**: In `manifest.yaml` only: - `{{ .cloud.* }}` - Infrastructure config (domain, smtp, etc.) - `{{ .cluster.* }}` - Cluster config (IPs, versions, etc.) - `{{ .operator.* }}` - Operator info (email) - `{{ .app.* }}` - This app's config from defaultConfig - `{{ .apps..* }}` - Dependency app's config (via requires mapping) - `{{ .secrets.* }}` - This app's secrets (in defaultSecrets default only) In `*.yaml` resource templates: - `{{ .* }}` - Only this app's config (all from defaultConfig) - No access to secrets, cluster config, or other apps **Dependency Resolution**: 1. `requires` lists app types needed (matches `is` field) 2. At add time, user maps to actual installed apps 3. System stores mapping in `installedAs` field in instance manifest 4. Templates resolve `{{ .apps.db.* }}` using this mapping ### 2. Instance Data (Local State) ``` data-dir/instances/{instance}/ ├── config.yaml # App configuration (user-editable) ├── secrets.yaml # App secrets (generated + user-editable) ├── kubeconfig # Cluster access credentials ├── apps/ │ └── {app-name}/ │ ├── manifest.yaml # Copy with installedAs mappings │ ├── deployment.yaml # Compiled (variables resolved) │ ├── service.yaml # Compiled │ ├── ingress.yaml # Compiled │ └── ... # All manifests compiled └── operations/ └── op_{action}_app_{app-name}_{timestamp}.json ``` #### config.yaml Structure ```yaml apps: postgres: namespace: postgres image: pgvector/pgvector:pg15 port: "5432" storage: 10Gi host: postgres.postgres.svc.cluster.local # ... all defaultConfig values from manifest ``` #### secrets.yaml Structure ```yaml apps: postgres: password: ghost: dbPassword: adminPassword: smtpPassword: # defaultSecrets + requiredSecrets ``` **Characteristics**: - Immediately consistent (filesystem) - File-locked during updates (`config.yaml.lock`, `secrets.yaml.lock`) - Version controlled (recommended but optional) - User-editable (advanced users can SSH and modify) ### 3. Kubernetes Cluster (Runtime State) ``` Kubernetes Cluster └── Namespace: {app-name} ├── Deployment: {app-name}-* ├── ReplicaSet: {app-name}-* ├── Pod: {app-name}-* ├── Service: {app-name} ├── Ingress: {app-name} ├── PVC: {app-name}-pvc ├── Secret: {app-name}-secrets ├── ConfigMap: {app-name}-* (optional) └── Job: {app-name}-db-init (optional) ``` **Namespace Lifecycle**: - `Active`: Normal operating state - `Terminating`: Deletion in progress (may take time) - Finalizers: `[kubernetes]` prevents deletion until resources cleaned **Characteristics**: - Eventually consistent (distributed system) - Cascade deletion: Deleting namespace deletes all child resources - Finalizers block deletion until cleared - May enter stuck states requiring automatic intervention #### Kubernetes Resource Labeling All Wild Cloud apps use standard labels automatically applied via Kustomize: ```yaml # In kustomization.yaml labels: - includeSelectors: true # Apply to resources AND selectors pairs: app: myapp # App name managedBy: kustomize partOf: wild-cloud ``` This auto-expands selectors: ```yaml # You write: selector: component: web # Kustomize expands to: selector: app: myapp managedBy: kustomize partOf: wild-cloud component: web ``` **Important**: Use simple component labels (`component: web`), not Helm-style labels (`app.kubernetes.io/name`). ### 4. External System State (Kubernetes Controller-Managed) These systems are not directly controlled by Wild Cloud API but are integral to app lifecycle: #### External DNS (via external-dns controller) **Location**: External DNS provider (Cloudflare, Route53, etc.) **Trigger**: Ingress with external-dns annotations ```yaml annotations: external-dns.alpha.kubernetes.io/target: {{ .domain }} ``` **State Flow**: ``` 1. Deploy creates Ingress with annotations 2. external-dns controller watches Ingress resources 3. Controller creates DNS records at provider 4. DNS propagates (eventual consistency, 30-300 seconds) 5. Domain resolves to cluster load balancer IP ``` **Lifecycle**: - **Create**: Automatic when Ingress deployed - **Update**: Automatic when Ingress annotations change - **Delete**: Automatic when Ingress deleted (DNS records cleaned up) **Eventual Consistency**: DNS changes take 30s-5min to propagate globally. **Edge Cases**: - DNS propagation delay (app deployed but domain not resolving yet) - Provider rate limits (too many updates) - Stale records if external-dns controller down during deletion - Multiple ingresses with same hostname (last write wins) **Debugging**: ```bash # View external-dns logs kubectl logs -n external-dns deployment/external-dns # Check what DNS records external-dns is managing kubectl get ingress -A -o yaml | grep external-dns ``` #### TLS Certificates (via cert-manager) **Location**: Both cluster (Kubernetes Secret) and external CA (Let's Encrypt) **Trigger**: Ingress with cert-manager annotations ```yaml annotations: cert-manager.io/cluster-issuer: letsencrypt-prod spec: tls: - hosts: - myapp.cloud.example.com secretName: myapp-tls ``` **State Flow**: ``` 1. Deploy creates Ingress with TLS config 2. cert-manager creates Certificate resource 3. cert-manager creates Order with ACME DNS-01 challenge 4. cert-manager updates DNS via provider (for challenge) 5. Let's Encrypt validates domain ownership via DNS 6. cert-manager receives certificate and stores in Secret 7. Ingress controller uses Secret for TLS termination ``` **Lifecycle**: - **Create**: Automatic when Ingress with cert-manager annotation deployed - **Renew**: Automatic (starts 30 days before expiry) - **Delete**: Secret deleted with namespace, CA record persists **Eventual Consistency**: Certificate issuance takes 30s-2min (DNS challenge + CA validation). **Edge Cases**: - DNS-01 challenge timeout (DNS not propagated yet) - Rate limits (Let's Encrypt: 50 certs/domain/week, 5 failed validations/hour) - Expired certificates (cert-manager should auto-renew but may fail) - Namespace stuck terminating (cert-manager challenges may block finalizers) **Debugging**: ```bash # View certificates and their status kubectl get certificate -n myapp kubectl describe certificate myapp-tls -n myapp # View ACME challenge progress kubectl get certificaterequest -n myapp kubectl get order -n myapp kubectl get challenge -n myapp # Check cert-manager logs kubectl logs -n cert-manager deployment/cert-manager ``` #### Wildcard Certificates (Shared Resource Pattern) Wild Cloud uses **two shared wildcard certificates** to avoid rate limits: **1. Public Wildcard Certificate** (`wildcard-wild-cloud-tls`) ```yaml # Created once in cert-manager namespace apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: wildcard-wild-cloud-tls namespace: cert-manager spec: secretName: wildcard-wild-cloud-tls dnsNames: - "*.cloud.example.com" ``` **2. Internal Wildcard Certificate** (`wildcard-internal-wild-cloud-tls`) ```yaml # For internal-only apps not exposed via external-dns apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: wildcard-internal-wild-cloud-tls namespace: cert-manager spec: secretName: wildcard-internal-wild-cloud-tls dnsNames: - "*.internal.cloud.example.com" ``` **Usage Pattern**: - **Public apps** (exposed externally): Use `wildcard-wild-cloud-tls` - Domain: `myapp.cloud.example.com` - Has external-dns annotation (creates public DNS record) - **Internal apps** (cluster-only): Use `wildcard-internal-wild-cloud-tls` - Domain: `myapp.internal.cloud.example.com` - No external-dns annotation (only accessible within cluster/LAN) - Examples: Docker registry, internal dashboards **Shared Pattern**: 1. One wildcard cert per domain covers all subdomains 2. Apps reference via `tlsSecretName: wildcard-wild-cloud-tls` (or `wildcard-internal-wild-cloud-tls`) 3. Deploy operation copies secret from cert-manager namespace to app namespace 4. All apps on same domain share the certificate **Advantages**: - Avoids Let's Encrypt rate limits (50 certs/domain/week) - Faster deployment (no ACME challenge per app) - Survives app delete/redeploy (cert persists in cert-manager namespace) **Trade-offs**: - All apps on same domain share same cert (if compromised, affects all apps) - Cert must be copied to each app namespace (handled by Deploy operation) **Copy Operation**: ```go // In apps.Deploy() // Copies both wildcard certs if referenced by ingress wildcardSecrets := []string{"wildcard-wild-cloud-tls", "wildcard-internal-wild-cloud-tls"} for _, secretName := range wildcardSecrets { if bytes.Contains(ingressContent, []byte(secretName)) { utilities.CopySecretBetweenNamespaces(kubeconfigPath, secretName, "cert-manager", appName) } } ``` #### Load Balancer IPs (via MetalLB) **Location**: MetalLB controller state + cluster network **Trigger**: Service with `type: LoadBalancer` ```yaml apiVersion: v1 kind: Service metadata: name: traefik namespace: traefik spec: type: LoadBalancer loadBalancerIP: 192.168.8.80 # Optional: request specific IP ``` **State Flow**: ``` 1. Service created with type: LoadBalancer 2. MetalLB controller assigns IP from configured pool 3. MetalLB announces IP via ARP (Layer 2) or BGP (Layer 3) 4. Network routes traffic to assigned IP 5. kube-proxy on nodes routes to service endpoints ``` **Lifecycle**: - **Create**: Automatic when LoadBalancer Service deployed - **Persist**: IP sticky (same IP across pod restarts) - **Delete**: IP returned to pool when Service deleted **Eventual Consistency**: ARP cache clearing takes 0-60 seconds. **Edge Cases**: - IP pool exhaustion (no IPs available from MetalLB pool) - IP conflicts (pool overlaps with DHCP or static assignments) - ARP cache issues (old MAC address cached, traffic fails until cleared) - Split-brain scenarios (multiple nodes announce same IP) **Debugging**: ```bash # View services with assigned IPs kubectl get svc -A --field-selector spec.type=LoadBalancer # Check MetalLB IP pools kubectl get ipaddresspool -n metallb-system # View MetalLB controller state kubectl logs -n metallb-system deployment/controller kubectl logs -n metallb-system daemonset/speaker ``` ### Cross-System Dependency Chain A complete app deployment triggers this cascade across systems: ``` Wild Cloud API (Deploy) ↓ Kubernetes (kubectl apply) ↓ Namespace + Resources Created ↓ ┌─────────────────┬──────────────────┬─────────────────┐ │ │ │ │ external-dns cert-manager MetalLB watches Ingress watches Ingress watches Service ↓ ↓ ↓ DNS Provider Let's Encrypt Network ARP/BGP (Cloudflare) (ACME CA) (Local Network) ↓ ↓ ↓ CNAME Record TLS Certificate IP Address Created Issued Announced (30s-5min) (30s-2min) (0-60s) ↓ ↓ ↓ Domain Resolves + HTTPS Works + Traffic Routes ``` **Total Time to Fully Operational**: - Kubernetes resources: 5-30 seconds (image pull + pod start) - DNS propagation: 30 seconds - 5 minutes - TLS certificate: 30 seconds - 2 minutes - Network ARP: 0-60 seconds **Worst case**: 5-7 minutes from deploy command to app fully accessible via HTTPS. ## App Lifecycle States ### State 0: NOT_ADDED ``` Wild Directory: {app-name}/ exists (templates) Instance Apps: (does not exist) config.yaml: (no apps.{app-name} entry) secrets.yaml: (no apps.{app-name} entry) Cluster: (no namespace) ``` **Invariants**: - App can be added from Wild Directory - No local or cluster state exists --- ### State 1: ADDED **After**: `wild app add {app-name}` ``` Wild Directory: {app-name}/ (unchanged) Instance Apps: {app-name}/ created with compiled manifests manifest.yaml has installedAs dependency mappings config.yaml: apps.{app-name} populated from defaultConfig secrets.yaml: apps.{app-name} populated with generated secrets Cluster: (no namespace yet) ``` **Operations**: 1. Read `wild-directory/{app-name}/manifest.yaml` 2. Resolve gomplate variables using instance config 3. Generate random secrets for `defaultSecrets` (if no default provided) 4. Copy secrets from dependencies for `requiredSecrets` 5. Compile templates → write to `instance/apps/{app-name}/` 6. Append to `config.yaml` (file-locked) 7. Append to `secrets.yaml` (file-locked) **Invariants**: - Local state consistent: config, secrets, and compiled manifests all exist - Cluster state empty: nothing deployed yet - Idempotent: Can re-add without side effects (overwrites local state) --- ### State 2: DEPLOYING **During**: `wild app deploy {app-name}` ``` Wild Directory: (unchanged) Instance Apps: (unchanged) config.yaml: (unchanged) secrets.yaml: (unchanged) Cluster: namespace: Active (or being created) resources: Creating/Pending secret/{app-name}-secrets: Created ``` **Operations**: 1. Check namespace status (pre-flight check) 2. Create/update namespace (idempotent) 3. Create Kubernetes secret from `secrets.yaml` (overwrite if exists) 4. Copy dependency secrets (e.g., postgres-secrets) 5. Copy TLS certificates (e.g., wildcard certs from cert-manager) 6. Apply manifests: `kubectl apply -k instance/apps/{app-name}/` **Invariants Being Established**: - Namespace must be `Active` or `NotFound` (not `Terminating`) - Kubernetes secret created before workloads - All dependencies deployed first --- ### State 3: DEPLOYED **After successful deploy**: ``` Wild Directory: (unchanged) Instance Apps: (unchanged) config.yaml: (unchanged) secrets.yaml: (unchanged) Cluster: namespace: Active deployment: Ready (replicas running) pods: Running service: Available (endpoints ready) ingress: Ready (external-dns created DNS) pvc: Bound (storage provisioned) secret: Exists ``` **Invariants**: - **Strong consistency**: Local state matches cluster intent - All pods healthy and running - Services have endpoints - DNS records created (via external-dns) - TLS certificates valid (via cert-manager) **Health Checks**: ```bash kubectl get pods -n {app-name} kubectl get ingress -n {app-name} kubectl get pvc -n {app-name} ``` --- ### State 3a: UPDATING (Configuration/Secret Changes) **Scenario**: User modifies config.yaml or secrets.yaml and redeploys. **Operations**: #### Update Configuration Only ``` 1. User edits config.yaml (e.g., change port, storage size) 2. User runs: wild app deploy {app-name} 3. System re-compiles templates with new config 4. System applies updated manifests: kubectl apply -k 5. Kubernetes performs rolling update (if applicable) ``` **State Flow**: ``` config.yaml: Modified (new values) Instance Apps: Templates re-compiled with new config secrets.yaml: (unchanged) Cluster: Rolling update (pods recreated with new config) ``` **Important**: Config changes trigger template recompilation. The `.package` directory preserves original templates, but deployed manifests are regenerated. #### Update Secrets Only ``` 1. User edits secrets.yaml (e.g., change password) 2. User runs: wild app deploy {app-name} 3. System deletes old Kubernetes secret 4. System creates new Kubernetes secret with updated values 5. Pods must be restarted to pick up new secrets ``` **State Flow**: ``` config.yaml: (unchanged) Instance Apps: (unchanged - no template changes) secrets.yaml: Modified (new secrets) Cluster: Secret updated, pods may need manual restart ``` **Critical**: Most apps don't auto-reload secrets. May require manual pod restart: ```bash kubectl rollout restart deployment/{app-name} -n {app-name} ``` #### Update Both Config and Secrets ``` 1. User edits both config.yaml and secrets.yaml 2. User runs: wild app deploy {app-name} 3. System re-compiles templates + updates secrets 4. System applies manifests (rolling update) 5. Pods restart with new config and secrets ``` --- ### State 3b: UPDATING (Manifest/Template Changes) **Scenario**: User directly edits Kustomize files in instance apps directory. This workflow differs significantly for **advanced users** (git-based) vs **regular users** (Web UI/CLI). #### Advanced User Workflow (Git-Based) **Instance directory as git repository**: ```bash # Instance data directory is a git repo cd /var/lib/wild-central/instances/my-cloud git status git log ``` **Operations**: ``` 1. User SSHs to Wild Central device (or uses VSCode Remote SSH) 2. User edits: apps/{app-name}/deployment.yaml 3. User commits changes: git add . && git commit -m "Custom resource limits" 4. User runs: wild app deploy {app-name} OR kubectl apply -k apps/{app-name}/ 5. Changes applied to cluster ``` **State Flow**: ``` Wild Directory: (unchanged - original templates intact) Instance Apps: Modified and git-tracked (intentional divergence) config.yaml: (unchanged) secrets.yaml: (unchanged) Cluster: Updated with manual changes .package/: (unchanged - preserves original templates) Git History: Tracks all manual edits with commit messages ``` **Benefits**: - **Version Control**: Full audit trail of all changes - **Rollback**: `git revert` to undo changes - **Infrastructure as Code**: Instance config managed like application code - **Collaboration**: Multiple admins can work on same cluster config - **Merge Workflow**: Wild Directory updates handled as upstream merges **Example Git Workflow**: ```bash # Make custom changes vim apps/myapp/deployment.yaml git add apps/myapp/deployment.yaml git commit -m "Increase CPU limit for production load" # Deploy changes wild app deploy myapp # Later, merge upstream Wild Directory updates git pull upstream main # Pull Wild Directory changes git merge upstream/main # Merge with local customizations # Resolve any conflicts git push origin main ``` #### Regular User Workflow (Web UI/CLI) **Operations**: ``` 1. User cannot directly edit manifests (no SSH access) 2. User modifies config.yaml or secrets.yaml via Web UI 3. System re-compiles templates automatically 4. User deploys via Web UI ``` **State Flow**: ``` Wild Directory: (unchanged) Instance Apps: Re-compiled from templates (stays in sync) config.yaml: Modified via Web UI secrets.yaml: Modified via Web UI Cluster: Updated via Web UI deploy ``` **Protection**: - No manual manifest editing (prevents divergence) - All changes through config/secrets (stays synchronized) - Wild Directory updates apply cleanly (no merge conflicts) --- ### State 3c: UPDATING (Wild Directory Version Update) **Scenario**: Wild Directory app updated (bug fix, new version, new features). This workflow differs significantly for **advanced users** (git-based) vs **regular users** (Web UI/CLI). #### Advanced User Workflow (Git Merge) **Wild Directory as upstream remote**: ```bash # Add Wild Directory as upstream remote (one-time setup) git remote add wild-directory https://github.com/wildcloud/wild-directory.git git fetch wild-directory ``` **Detection**: ```bash # Check for upstream updates git fetch wild-directory git log HEAD..wild-directory/main --oneline # See what changed in specific app git diff HEAD wild-directory/main -- apps/myapp/ ``` **Merge Operations**: ```bash # 1. Fetch latest Wild Directory changes git fetch wild-directory # 2. Merge upstream changes with local customizations git merge wild-directory/main # 3. Resolve any conflicts # Git will show conflicts in manifest files, config, etc. # User resolves conflicts preserving their custom changes # 4. Test changes wild app deploy myapp --dry-run # 5. Deploy updated app wild app deploy myapp # 6. Commit merge git push origin main ``` **Conflict Resolution Example**: ```yaml # Conflict in apps/myapp/deployment.yaml <<<<<<< HEAD resources: limits: cpu: "2000m" # Local customization memory: "4Gi" # Local customization ======= resources: limits: cpu: "1000m" # Wild Directory default memory: "2Gi" # Wild Directory default >>>>>>> wild-directory/main ``` User resolves by keeping their custom values or adopting new defaults. **Benefits**: - **Full Control**: User decides what to merge and when - **Conflict Resolution**: Git's standard merge tools handle conflicts - **Audit Trail**: Git history shows what changed and why - **Selective Updates**: Can cherry-pick specific app updates **State Flow**: ``` Wild Directory: (tracked as remote, fetched regularly) Instance Apps: Merged with git (custom + upstream changes) config.yaml: Manually merged (conflicts resolved by user) secrets.yaml: Preserved (not in Wild Directory) .package/: Updated after merge Git History: Shows merge commits and conflict resolutions Cluster: Updated when user deploys after merge ``` #### Regular User Workflow (Automated Merge) **Detection Methods**: **Method 1: Compare .package with Wild Directory** ```bash # System compares checksums/timestamps diff -r instance/apps/{app-name}/.package/ wild-directory/{app-name}/ ``` If differences exist: New version available in Wild Directory. **Method 2: Check manifest version field** ```yaml # wild-directory/{app-name}/manifest.yaml version: 2.0.0 # instance/apps/{app-name}/manifest.yaml version: 1.0.0 # Older version ``` **Safe Update (Preserves Local Config)** ``` 1. System detects Wild Directory changes 2. User initiates update (via UI or CLI) 3. System backs up current instance state: - Saves current config.yaml section - Saves current secrets.yaml section - Saves current manifest.yaml (with installedAs mappings) 4. System re-adds app from Wild Directory: - Copies new templates to instance/apps/{app-name}/ - Updates .package/ with new source files - Merges new defaultConfig with existing config - Preserves existing secrets (doesn't regenerate) 5. System re-compiles templates with preserved config 6. User reviews changes (diff shown in UI) 7. User deploys updated app ``` **State Flow**: ``` Wild Directory: (unchanged - new version available) Instance Apps: Updated templates + recompiled manifests config.yaml: Merged (new fields added, existing preserved) secrets.yaml: (unchanged - existing secrets preserved) .package/: Updated with new source files Cluster: (not changed until user deploys) ``` **Merge Strategy for Config**: ```yaml # Old config.yaml (version 1.0.0) apps: myapp: port: "8080" storage: 10Gi # New Wild Directory manifest (version 2.0.0) adds "replicas" field defaultConfig: port: "8080" storage: 10Gi replicas: "3" # New field # Merged config.yaml (after update) apps: myapp: port: "8080" # Preserved storage: 10Gi # Preserved replicas: "3" # Added ``` **Breaking Changes**: If Wild Directory update has breaking changes (renamed fields, removed features): - System cannot auto-merge - User must manually reconcile - UI shows conflicts and requires resolution #### Destructive Update (Fresh Install) ``` 1. User deletes app: wild app delete {app-name} 2. User re-adds app: wild app add {app-name} 3. Config and secrets regenerated (loses customizations) 4. User must manually reconfigure ``` **Use When**: - Major version upgrade with breaking changes - Significant manifest restructuring - User wants clean slate --- ### State 3d: DEPLOYED with Drift **Scenario**: Cluster state diverged from instance state. This state has different meanings for **advanced users** vs **regular users**. #### Advanced Users: Intentional Drift (Git-Tracked) **Scenario**: User made direct cluster changes and committed them to git. **Example**: ```bash # User edits deployment directly kubectl edit deployment myapp -n myapp # User documents change in git vim apps/myapp/deployment.yaml # Update manifest to match git add apps/myapp/deployment.yaml git commit -m "Emergency CPU limit increase for production incident" ``` **State Flow**: ``` Instance Apps: Updated and git-tracked (intentional) Git History: Documents why change was made Cluster: Matches updated instance state ``` **This is NOT drift** - it's infrastructure-as-code in action. The instance directory reflects the true desired state, tracked in git. **Reconciliation**: Not needed (intentional state). #### Regular Users: Unintentional Drift **Scenario**: Cluster state diverged from instance state (unexpected). **Causes**: - User ran `kubectl edit` directly (shouldn't happen - no SSH access) - Another admin modified cluster resources - Partial deployment failure (some resources applied, others failed) - Kubernetes controller modified resources (e.g., HPA changed replicas) **Detection**: ```bash # Compare desired vs actual state kubectl diff -k instance/apps/{app-name}/ # Or use declarative check kubectl apply -k instance/apps/{app-name}/ --dry-run=server ``` **State Flow**: ``` Instance Apps: Unchanged (desired state) Cluster: Diverged (actual state differs) ``` **Reconciliation**: ``` 1. User runs: wild app deploy {app-name} 2. kubectl apply re-applies desired state 3. Kubernetes reconciles differences (three-way merge) 4. Cluster returns to matching instance state ``` **Important**: `kubectl apply` is idempotent and safe for reconciliation. #### Distinguishing Intentional vs Unintentional Drift **Advanced users (git-based)**: - Check git status: `git status` shows no uncommitted changes → intentional - Check git log: `git log -- apps/myapp/` shows recent commits → intentional - Cluster state matches git-tracked files → intentional **Regular users (Web UI)**: - Any divergence is unintentional (no way to edit manifests directly) - Reconcile immediately by redeploying --- ### State 4: DELETING **During**: `wild app delete {app-name}` ``` Wild Directory: (unchanged) Instance Apps: Being removed config.yaml: apps.{app-name} being removed secrets.yaml: apps.{app-name} being removed Cluster: namespace: Active → Terminating resources: Deleting (cascade) ``` **Operations** (Two-Phase): **Phase 1: Cluster Cleanup (Best Effort)** ```bash # Try graceful deletion kubectl delete namespace {app-name} --timeout=30s --wait=true # If stuck, force cleanup kubectl patch namespace {app-name} --type=merge -p '{"metadata":{"finalizers":null}}' ``` **Phase 2: Local Cleanup (Always Succeeds)** ```bash rm -rf instance/apps/{app-name}/ yq delete config.yaml '.apps.{app-name}' yq delete secrets.yaml '.apps.{app-name}' ``` **Critical Design Decision**: - **Don't wait indefinitely for cluster cleanup** - Local state is immediately consistent after Phase 2 - Cluster cleanup is eventually consistent --- ### State 5: DELETED **After successful delete**: ``` Wild Directory: (unchanged - still available for re-add) Instance Apps: (removed) config.yaml: (no apps.{app-name} entry) secrets.yaml: (no apps.{app-name} entry) Cluster: namespace: NotFound all resources: (removed) ``` **Invariants**: - Local state has no trace of app - Cluster has no namespace or resources - App can be re-added cleanly --- ### State X: STUCK_TERMINATING (Edge Case) **Problematic state when namespace won't delete**: ``` Wild Directory: (unchanged) Instance Apps: May or may not exist (depends on delete progress) config.yaml: May or may not have entry secrets.yaml: May or may not have entry Cluster: namespace: Terminating (STUCK!) finalizers: Blocking deletion resources: Some exist, some terminating ``` **Why This Happens**: 1. Resources with custom finalizers 2. Webhooks or admission controllers blocking deletion 3. Network issues during deletion 4. StatefulSet with orphaned PVCs **Resolution**: - Handled automatically by Deploy pre-flight checks - Force cleanup finalizers after retries - User never needs manual intervention ## System Boundaries and Consistency ### Consistency Guarantees by System | System | Consistency Model | Synchronization | |--------|------------------|-----------------| | Wild Directory | Immutable | Read-only | | Instance Data | Immediately Consistent | File locks | | Kubernetes | Eventually Consistent | Reconciliation loops | ### Cross-System Operations #### Delete Operation (Spans 2 Systems) ``` ┌─────────────────────────────────────────────────┐ │ Delete Operation Timeline │ ├─────────────────────────────────────────────────┤ │ │ │ T=0s: kubectl delete namespace (initiated) │ │ └─ Cluster enters eventual consistency │ │ │ │ T=1s: rm apps/{app-name}/ (completes) │ │ yq delete config.yaml (completes) │ │ yq delete secrets.yaml (completes) │ │ └─ Local state immediately consistent │ │ │ │ T=2s: Return success to user │ │ │ │ T=30s: Namespace still terminating in cluster │ │ └─ This is OK! Eventually consistent │ │ │ │ T=60s: Cluster cleanup completes │ │ └─ Both systems now consistent │ └─────────────────────────────────────────────────┘ ``` **Key Insight**: We accept temporary inconsistency at the system boundary. #### Deploy Operation (Spans 2 Systems) ``` ┌─────────────────────────────────────────────────┐ │ Deploy Operation Timeline │ ├─────────────────────────────────────────────────┤ │ │ │ T=0s: Check namespace status (pre-flight) │ │ If Terminating: Force cleanup + retry │ │ │ │ T=5s: Create namespace (idempotent) │ │ Create secrets │ │ Apply manifests │ │ └─ Cluster enters reconciliation │ │ │ │ T=30s: Pods starting, images pulling │ │ │ │ T=60s: All pods Running, services ready │ │ └─ Deployment successful │ └─────────────────────────────────────────────────┘ ``` **Key Insight**: Deploy owns making cluster match local state. ## Idempotency and Safety ### Idempotent Operations | Operation | Idempotent? | Why | |-----------|-------------|-----| | `app add` | ✅ Yes | Overwrites local state | | `app deploy` | ✅ Yes | `kubectl apply` is idempotent | | `app delete` | ✅ Yes | `kubectl delete --ignore-not-found` | ### Non-Idempotent Danger Zones 1. **Secret Generation**: Regenerating secrets breaks running apps - Solution: Only generate if key doesn't exist 2. **Database Initialization**: Running twice can cause conflicts - Solution: Job uses `CREATE IF NOT EXISTS`, `ALTER IF EXISTS` 3. **Finalizer Removal**: Skips cleanup logic - Solution: Only as last resort after graceful attempts ## Edge Cases and Error Handling ### Edge Case 1: Namespace Stuck Terminating **Scenario**: Previous delete left namespace in Terminating state. **Detection**: ```bash kubectl get namespace {app-name} -o jsonpath='{.status.phase}' # Returns: "Terminating" ``` **Resolution** (Automatic): 1. Deploy pre-flight check detects Terminating state 2. Attempts force cleanup: removes finalizers 3. Waits 5 seconds 4. Retries up to 3 times 5. If still stuck, returns clear error message **Code**: ```go if status == "Terminating" { forceNamespaceCleanup(kubeconfigPath, appName) time.Sleep(5 * time.Second) // Retry deploy } ``` ### Edge Case 2: Concurrent Delete + Deploy **Scenario**: User deletes app, then immediately redeploys. **Timeline**: ``` T=0s: Delete initiated T=1s: Local state cleaned up T=2s: User clicks "Deploy" T=3s: Deploy detects Terminating namespace T=4s: Deploy force cleanups and retries T=10s: Deploy succeeds ``` **Why This Works**: - Delete doesn't block on cluster cleanup - Deploy handles any namespace state - Eventual consistency at system boundary ### Edge Case 3: Dependency Not Deployed **Scenario**: User tries to deploy app requiring postgres, but postgres isn't deployed. **Current Behavior**: Deployment succeeds but pods crash (CrashLoopBackOff). **Detection**: ```bash kubectl get pods -n {app-name} # Shows: CrashLoopBackOff kubectl logs {pod-name} -n {app-name} # Shows: "Connection refused to postgres.postgres.svc.cluster.local" ``` **Future Enhancement**: Pre-flight dependency check in Deploy operation. ### Edge Case 4: Secrets Out of Sync **Scenario**: User manually updates password in Kubernetes but not in `secrets.yaml`. **Impact**: - Next deploy overwrites Kubernetes secret - App may lose access if password changed elsewhere **Best Practice**: Always update `secrets.yaml` as source of truth. ### Edge Case 5: PVC Retention **Scenario**: Delete removes namespace but PVCs may persist (depends on reclaim policy). **Behavior**: - PVC with `ReclaimPolicy: Retain` stays after delete - Redeploy creates new PVC (data orphaned) **Resolution**: Document PVC backup/restore procedures. ## App Package Development Best Practices ### Security Requirements **All pods must include security contexts**: ```yaml spec: template: spec: securityContext: runAsNonRoot: true runAsUser: 999 # Use appropriate non-root UID runAsGroup: 999 seccompProfile: type: RuntimeDefault containers: - name: app securityContext: allowPrivilegeEscalation: false capabilities: drop: [ALL] readOnlyRootFilesystem: false # true when possible ``` Common UIDs: PostgreSQL/Redis use 999. ### Database Initialization Pattern Apps requiring databases should include `db-init-job.yaml`: ```yaml apiVersion: batch/v1 kind: Job metadata: name: myapp-db-init spec: template: spec: restartPolicy: OnFailure containers: - name: db-init image: postgres:15 command: - /bin/bash - -c - | # Create database if doesn't exist # Create/update user with password # Grant permissions ``` **Critical**: Use idempotent SQL: - `CREATE DATABASE IF NOT EXISTS` - `CREATE USER IF NOT EXISTS ... ELSE ALTER USER ... WITH PASSWORD` - Jobs retry on failure until success ### Database URL Secrets Never use runtime variable substitution - it doesn't work with Kustomize: ```yaml # ❌ Wrong: - name: DB_URL value: "postgres://user:$(DB_PASSWORD)@host/db" # ✅ Correct: - name: DB_URL valueFrom: secretKeyRef: name: myapp-secrets key: dbUrl ``` Define `dbUrl` in manifest's `defaultSecrets` with template: ```yaml defaultSecrets: - key: dbUrl default: "postgres://{{ .app.dbUser }}:{{ .secrets.dbPassword }}@{{ .app.dbHost }}/{{ .app.dbName }}" ``` ### External DNS Integration Ingresses should include external-dns annotations: ```yaml metadata: annotations: external-dns.alpha.kubernetes.io/target: {{ .domain }} external-dns.alpha.kubernetes.io/cloudflare-proxied: "false" ``` This creates: `myapp.cloud.example.com` → `cloud.example.com` (CNAME) ### Converting from Helm Charts 1. Extract and render Helm chart: ```bash helm fetch --untar --untardir charts repo/chart-name helm template --output-dir base --namespace myapp myapp charts/chart-name ``` 2. Create Wild Cloud structure: - Add `namespace.yaml` - Run `kustomize create --autodetect` - Create `manifest.yaml` - Replace values with gomplate variables - Update labels (remove Helm-style, add Wild Cloud standard) - Add security contexts - Add external-dns annotations ## Testing Strategies ### Unit Tests Test individual operations in isolation: ```go func TestDelete_NamespaceNotFound(t *testing.T) { // Test delete when namespace doesn't exist // Should succeed without error } func TestDelete_NamespaceTerminating(t *testing.T) { // Test delete when namespace stuck terminating // Should force cleanup and succeed } func TestDeploy_NamespaceTerminating(t *testing.T) { // Test deploy when namespace terminating // Should retry and eventually succeed } ``` ### Integration Tests Test cross-system operations: ```go func TestDeleteThenDeploy(t *testing.T) { // 1. Deploy app // 2. Delete app // 3. Immediately redeploy // Should succeed without manual intervention } func TestConcurrentOperations(t *testing.T) { // Test multiple operations on same app // File locks should prevent corruption } ``` ### Chaos Tests Test resilience to failures: ```go func TestDeleteWithNetworkPartition(t *testing.T) { // Simulate network failure during delete // Local state should still be cleaned up } func TestDeployWithStuckFinalizer(t *testing.T) { // Manually add finalizer to namespace // Deploy should detect and force cleanup } ``` ## Operational Procedures ### Manual Inspection **Check all state locations**: ```bash # 1. Local state ls instance/apps/{app-name}/ yq eval '.apps.{app-name}' config.yaml yq eval '.apps.{app-name}' secrets.yaml # 2. Cluster state kubectl get namespace {app-name} kubectl get all -n {app-name} kubectl get pvc -n {app-name} kubectl get secrets -n {app-name} kubectl get ingress -n {app-name} ``` **Check operation status**: ```bash ls -lt instance/operations/ | head -5 cat instance/operations/op_deploy_app_{app-name}_*.json ``` ### Manual Recovery **If namespace stuck terminating**: ```bash # This should never be needed - Deploy handles automatically # But for understanding: kubectl get namespace {app-name} -o json | \ jq '.spec.finalizers = []' | \ kubectl replace --raw /api/v1/namespaces/{app-name}/finalize -f - ``` **If local state corrupted**: ```bash # Re-add from Wild Directory wild app add {app-name} # This regenerates local state from source ``` **If secrets lost**: ```bash # Secrets are auto-generated on add # If lost, must re-add app (regenerates new secrets) # Apps will need reconfiguration with new credentials ``` ## Design Principles ### 1. Eventual Consistency at Boundaries Accept that cluster state and local state may temporarily diverge. Design operations to handle any state. ### 2. Local State as Source of Truth Instance data (config.yaml, secrets.yaml) is authoritative for intended state. Cluster reflects current state. ### 3. Idempotent Everything Every operation should be safely repeatable. Use: - `kubectl apply` (not `create`) - `kubectl delete --ignore-not-found` - `CREATE IF NOT EXISTS` in SQL ### 4. Fail Forward, Not Backward If operation partially completes, retry should make progress (not start over). ### 5. No Indefinite Waits Operations timeout and fail explicitly rather than hanging forever. ### 6. User Never Needs Manual Intervention Automated recovery from all known edge cases (stuck namespaces, etc.). ## Future Enhancements ### 1. Dependency Validation Pre-flight check that required apps are deployed: ```go if manifest.Requires != nil { for _, dep := range manifest.Requires { if !isAppDeployed(dep.Name) { return fmt.Errorf("dependency %s not deployed", dep.Name) } } } ``` ### 2. State Reconciliation Periodic background job to ensure consistency: ```go func ReconcileAppState(appName string) { localState := readLocalState(appName) clusterState := readClusterState(appName) if !statesMatch(localState, clusterState) { // Alert or auto-correct } } ``` ### 3. Backup/Restore Workflows Built-in PVC backup before delete: ```bash wild app backup {app-name} wild app restore {app-name} --from-backup {timestamp} ``` ### 4. Dry-Run Mode Preview changes without applying: ```bash wild app deploy {app-name} --dry-run # Shows: resources that would be created/updated ``` ## Git Workflow Best Practices (Advanced Users) This section provides operational guidance for advanced users managing Wild Cloud instances as git repositories. ### Initial Repository Setup ```bash # Initialize instance directory as git repo cd /var/lib/wild-central/instances/my-cloud git init git add . git commit -m "Initial Wild Cloud instance configuration" # Add Wild Directory as upstream remote git remote add wild-directory https://github.com/wildcloud/wild-directory.git git fetch wild-directory # Add origin for your team's instance repo git remote add origin git@github.com:myorg/wild-cloud-instances.git git push -u origin main ``` ### .gitignore Configuration ```bash # Create .gitignore for instance directory cat > .gitignore <>>>>>> wild-directory/main # Resolution: Keep our production values resources: limits: cpu: "4000m" memory: "16Gi" requests: cpu: "2000m" memory: "8Gi" ``` ### Commit Message Conventions **Format**: `(): ` **Types**: - `feat`: New app or feature - `fix`: Bug fix or correction - `config`: Configuration change - `scale`: Resource scaling - `upgrade`: Version upgrade - `security`: Security-related change - `docs`: Documentation change **Examples**: ```bash git commit -m "feat(redis): Add Redis cache for session storage" git commit -m "scale(postgres): Increase CPU limits for production load" git commit -m "fix(ghost): Correct domain configuration for SSL" git commit -m "upgrade(immich): Update to v1.2.0 with new ML features" git commit -m "security(all): Rotate database passwords" git commit -m "config(mastodon): Enable SMTP for email notifications" ``` ### Rollback Procedures **Rollback entire app configuration**: ```bash # Find commit to rollback to git log --oneline -- apps/myapp/ # Revert specific commit git revert abc123 # Or rollback to specific point git checkout abc123 -- apps/myapp/ git commit -m "rollback(myapp): Revert to stable configuration" # Deploy reverted state wild app deploy myapp ``` **Emergency rollback (production incident)**: ```bash # Immediately revert to last known good state git log --oneline -5 git reset --hard abc123 # Last working commit wild app deploy myapp # Document the incident git commit --allow-empty -m "emergency: Rolled back myapp due to production incident" git push --force origin main # Force push to update remote ``` ### Collaboration Patterns **Multiple admins working on same cluster**: ```bash # Always pull before making changes git pull origin main # Use descriptive branch names git checkout -b alice/add-monitoring git checkout -b bob/upgrade-postgres # Push branches for review git push origin alice/add-monitoring # Use PRs/MRs for review before merging to main # This prevents conflicts and ensures peer review ``` **Code review checklist**: - [ ] Changes tested in non-production environment - [ ] Resource limits appropriate for workload - [ ] Secrets not committed - [ ] Dependencies deployed (if new app) - [ ] Commit message follows conventions - [ ] Breaking changes documented ### Backup and Disaster Recovery **Regular backups**: ```bash # Create tagged backup of current state git tag -a backup-$(date +%Y%m%d) -m "Daily backup" git push origin backup-$(date +%Y%m%d) # Automated daily backup (cron) 0 2 * * * cd /var/lib/wild-central/instances/my-cloud && git tag backup-$(date +%Y%m%d-%H%M) && git push origin --tags ``` **Disaster recovery**: ```bash # Clone instance repository to new Wild Central device git clone git@github.com:myorg/wild-cloud-instances.git /var/lib/wild-central/instances/my-cloud # Restore secrets from secure backup (NOT in git) # (From password manager, Vault, encrypted backup, etc.) cp ~/secure-backup/secrets.yaml /var/lib/wild-central/instances/my-cloud/ # Deploy all apps cd /var/lib/wild-central/instances/my-cloud for app in apps/*/; do wild app deploy $(basename $app) done ``` ### Git Workflow vs Web UI **When git is better**: - Complex changes requiring review - Multi-app updates - Compliance/audit requirements - Team collaboration - Emergency rollbacks **When Web UI is better**: - Quick configuration tweaks - Adding single app - Viewing current state - Non-technical team members **Hybrid approach**: Advanced users can use git for complex changes, Web UI for quick operations. The two workflows coexist peacefully since both modify the same instance directory. ## Conclusion Wild Cloud's app lifecycle management spans three independent systems with different consistency guarantees. By understanding these systems and their boundaries, we can design operations that are: - **Reliable**: Handle edge cases automatically - **Simple**: Two-phase operations (cluster + local) - **Safe**: Idempotent and recoverable - **Fast**: Don't wait unnecessarily for eventual consistency Additionally, for advanced users, the git-based workflow provides: - **Auditable**: Full version control history - **Collaborative**: Standard git workflows for team management - **Recoverable**: Git revert/rollback capabilities - **Professional**: Infrastructure-as-code best practices The key insight is accepting eventual consistency at system boundaries while maintaining immediate consistency within each system. This allows operations to complete quickly for users while ensuring the system eventually reaches a consistent state.