wild-directory/ADDING-APPS.md

# Adding Wild Cloud Apps

This guide is for contributors and maintainers who want to create or modify Wild Cloud apps. If you're looking to use existing apps, see [README.md](README.md).

## Overview

Wild Cloud apps are Kubernetes applications packaged as Kustomize configurations with standardized conventions for configuration management, secrets handling, and deployment.

## Directory Structure

Each app has a two-level structure: an `app.yaml` meta file at the root, and version-specific files inside `versions/`. Version directories are named by **slot** (typically the major version), not by the full version string. The actual version lives in `manifest.yaml` inside the slot.

```
myapp/
├── app.yaml               # App identity, latest slot pointer, upgrade routing
└── versions/
    ├── 2/                  # Current latest slot (manifest.yaml has version: 2.3.1)
    │   ├── manifest.yaml   # Version-specific config (requires, defaultConfig, etc.)
    │   ├── kustomization.yaml
    │   └── *.yaml          # Kubernetes resource templates
    └── 1/                  # Waypoint slot (only if upgrade routing needs it)
        ├── manifest.yaml
        ├── kustomization.yaml
        └── *.yaml
```

Most apps have **one** version directory. A second appears only when a waypoint is needed for upgrade routing.

## Required Files

Each app directory must contain:

1. **`app.yaml`** - App identity, latest slot pointer, and upgrade routing rules
2. **`versions/{slot}/manifest.yaml`** - Version-specific configuration schema
3. **`versions/{slot}/kustomization.yaml`** - Kustomize configuration with Wild Cloud labels
4. **`versions/{slot}/*.yaml`** - Kubernetes resource templates

## App Meta (`app.yaml`)

The `app.yaml` file at the app root defines identity, display info, and upgrade routing. These fields are version-independent.

```yaml
name: immich
is: immich
description: Immich is a self-hosted photo and video backup solution that allows you to store, manage, and share your media files securely.
icon: https://immich.app/assets/images/logo.png
latest: "1"
```

### App Meta Fields

| Field | Required | Description |
|-------|----------|-------------|
| `name` | Yes | App identifier (must match directory name) |
| `is` | Yes | Unique id for this app. Used for `requires` mapping |
| `description` | Yes | Brief app description shown in listings |
| `icon` | No | URL to app icon for UI display |
| `category` | Yes | Category for filtering in the app directory — use an established one or introduce a new one (see below) |
| `latest` | Yes | Slot name -- directory name under `versions/` (not a version string) |
| `upgrade` | No | Upgrade routing rules (see Upgrade Metadata below) |

### Categories

The UI uses `category` to populate the filter buttons in the App Directory. Use an established category when one fits — new categories are fine when an app clearly doesn't belong anywhere existing. Current categories:

| Category | Use for |
|----------|---------|
| `services` | Wild Cloud infrastructure services (traefik, metallb, cert-manager, etc.) |
| `database` | Database engines (postgres, mysql, redis, memcached) |
| `productivity` | Wikis, project management, file sharing, collaboration (nextcloud, outline, taiga, etc.) |
| `business` | ERP, CRM, accounting, forms, marketing automation (odoo, akaunting, mautic, etc.) |
| `communication` | Chat, messaging, video conferencing, email (mattermost, jitsi, listmonk, etc.) |
| `social` | Social networks and fediverse apps (mastodon, pixelfed, peertube, ghost, etc.) |
| `community` | Community organizing, civic tech, decision-making (loomio, decidim, discourse, etc.) |
| `education` | Learning management systems (moodle, chamilo) |
| `developer` | Dev tools, code hosting, package management (gitea, aptly, etc.) |
| `security` | VPN, password management (vaultwarden, headscale) |
| `ai` | AI/ML inference and interfaces (vllm, open-webui) |
| `media` | Photo and video management (immich) |

## Version Manifest (`versions/{slot}/manifest.yaml`)

Each version slot contains a `manifest.yaml` with version-specific installation details: dependencies, configuration schema, and secret requirements.

```yaml
version: 1.135.3-1
requires:
  - name: pg
    alias: db        # Use a different reference name in templates
  - name: redis      # 'alias' and 'installedAs' default to 'name' value
defaultConfig:
  namespace: immich
  externalDnsDomain: "{{ .cloud.domain }}"
  storage: 250Gi
  cacheStorage: 10Gi
  domain: immich.{{ .cloud.domain }}
  tlsSecretName: wildcard-wild-cloud-tls
  db: # Configuration can be nested
    host: "{{ .apps.pg.host }}" # Can reference 'requires' app configurations
    name: immich
    user: immich
  redis:
    host: "{{ .apps.redis.host }}"
defaultSecrets:
  - key: password      # Random value will be generated if empty
  - key: dbUrl
    default: "postgresql://{{ .app.db.user }}:{{ .secrets.dbPassword }}@{{ .app.db.host }}:{{ .app.db.port }}/{{ .app.db.name }}?pool=30"
requiredSecrets:
  - db.password   # References postgres app via 'db' alias
  - redis.auth    # References redis app via 'redis' name (no alias)
```

### Version Manifest Fields

| Field | Required | Description |
|-------|----------|-------------|
| `version` | Yes | App version (see Versioning Convention below) |
| `requires` | No | List of dependency apps with optional aliases |
| `defaultConfig` | Yes | Default configuration values merged into operator's `config.yaml` |
| `defaultSecrets` | No | This app's secrets (no 'default' = auto-generated) |
| `requiredSecrets` | No | List of secrets from dependency apps (format: `<app-ref>.<key>`) |

### Versioning Convention

Wild Cloud uses a two-part version scheme inspired by Debian packaging: `<upstream>-<revision>`.

- **Upstream version** tracks the third-party software version (e.g., `v4.0.18`, `1.120.2`)
- **Packaging revision** tracks Wild Cloud packaging changes (template fixes, manifest cleanup, config restructuring) that don't change the upstream software version

**Examples:**
- `v4.0.18` — initial packaging of upstream v4.0.18
- `v4.0.18-1` — first packaging fix (no upstream change)
- `v4.0.18-2` — second packaging fix
- `v4.0.19` — upstream version bump, revision resets

**When to bump the packaging revision:** Any change to the app package that doesn't correspond to an upstream software update — manifest field changes, template improvements, kustomize restructuring, security context fixes, label corrections, etc.

**When to bump the upstream version:** When updating the container image tag or deploying a new version of the third-party software.

The web UI uses version comparison to detect available updates. If the deployed version differs from the wild-directory version, operators see an update indicator and can apply it from the app detail panel.

### Slot Naming Convention

Version directory names are **slot names**, not version strings. The slot is a stable label; the actual version lives in `manifest.yaml` inside the slot.

**Rules:**
- Use the **major version** as the slot name (e.g., `1`, `2`, `5`, `v3`)
- Preserve the `v` prefix if the upstream project uses it (e.g., `v1` for cert-manager)
- **Never** put packaging revisions (`-1`, `-2`) in directory names
- **Never** put minor/patch versions in directory names unless creating a waypoint that needs to be distinct from another slot at the same major version

**Examples:**

| App | Slot name | Version in manifest |
|-----|-----------|-------------------|
| Ghost 5.118.1-2 | `5` | `5.118.1-2` |
| cert-manager v1.17.2 | `v1` | `v1.17.2` |
| Immich 1.135.3-1 | `1` | `1.135.3-1` |
| Traefik v3.4 | `v3` | `v3.4` |

When bumping versions (upstream or packaging), update files inside the existing slot. Only create a new directory when you need a new waypoint.

### Upgrade Metadata

Most apps can upgrade from any version to any other version directly — no special metadata is needed. The `upgrade` field is **optional** and only required when an app has breaking changes that need controlled upgrade paths.

**When you don't need `upgrade:`** Simple apps (Ghost, Redis, most stateless apps) where any version can safely replace any other version. This is the 90% case — just bump the version and the system handles it as a single-step update.

**When you need `upgrade:`** Apps with breaking database schema changes, incompatible config formats, or upstream requirements for sequential version upgrades (e.g., Discourse requires stepping through major versions).

#### The `upgrade` block in `app.yaml`

Upgrade routing rules live in `app.yaml`, centralized for all versions. The system iteratively re-evaluates these rules after each waypoint step.

```yaml
# app.yaml
name: myapp
latest: "3"
upgrade:
  from:
    - version: ">=3.5.0"       # Can upgrade directly from 3.5.x
    - version: ">=3.4.0"
      via: "2"                  # Must pass through slot "2" first (a waypoint)
    - version: "<3.4.0"
      blocked: true
      notes: "Requires sequential major upgrades. See upstream docs."
  preUpgrade:
    backup: required            # "none", "recommended", or "required"
```

Note: `latest` and `via` are **slot names** (directory names), not version strings. The system reads the actual version from the manifest inside each slot.

Version-specific upgrade behavior (migrations, configMigrations) lives in the version's `manifest.yaml`:

```yaml
# versions/3/manifest.yaml
version: 3.6.0
upgrade:
  migrations:
    pre:
      - migrations/pre-deploy.yaml   # K8s Job YAML paths relative to version dir
    post:
      - migrations/post-deploy.yaml
  configMigrations:
    oldKeyName: newKeyName      # Renames config keys automatically
```

**`app.yaml` upgrade fields:**

| Field | Description |
|-------|-------------|
| `from` | List of version constraint rules, evaluated in order (first match wins) |
| `from[].version` | Version constraint: `>=`, `>`, `<=`, `<`, `=`, or `>0` (matches any) |
| `from[].via` | Waypoint slot name in `versions/` — upgrade must pass through this slot first |
| `from[].blocked` | If true, upgrade is blocked with an error message |
| `from[].notes` | Human-readable message shown when blocked or as context |
| `preUpgrade.backup` | Backup requirement: `"required"` blocks upgrade until backup is done, `"recommended"` shows a warning |

**Version `manifest.yaml` upgrade fields:**

| Field | Description |
|-------|-------------|
| `migrations.pre` | K8s Job YAMLs to run before deploying this version step |
| `migrations.post` | K8s Job YAMLs to run after deploying this version step |
| `configMigrations` | Map of old config key → new config key for automatic renaming |

#### Waypoint versions

When an upgrade requires passing through an intermediate version, add that version's files as a new slot in the `versions/` directory alongside the latest:

```
myapp/
├── app.yaml               # Routing rules + latest pointer
└── versions/
    ├── 3/                  # Latest slot (version: 3.6.0)
    │   ├── manifest.yaml
    │   ├── kustomization.yaml
    │   └── *.yaml
    └── 2/                  # Waypoint slot (version: 2.8.0)
        ├── manifest.yaml
        ├── kustomization.yaml
        └── *.yaml
```

Each waypoint is a complete app package. The system computes a chain automatically — for example, upgrading from 2.3.0 to 3.6.0 might produce: `2.3.0 → 2.8.0 (slot "2") → 3.6.0 (slot "3")`.

**Creating a waypoint:** The current latest slot becomes the waypoint (leave it in place), then create a new slot for the new major version:

```bash
# Current slot "2" (with version 2.8.0) stays as a waypoint
# Create the new slot for the next major version
mkdir -p wild-directory/myapp/versions/3
# ... add manifest.yaml, kustomization.yaml, *.yaml for 3.0.0 ...
# Update app.yaml: set latest to "3", add upgrade routing rules with via: "2"
```

#### Migration jobs

Migration jobs are K8s Job manifests that run database migrations or other one-time tasks during an upgrade step. They must be **idempotent** (safe to re-run) since a failed upgrade might be retried.

Place migration job files in the version slot directory and reference them from that version's `manifest.yaml`:

```yaml
# versions/3/migrations/db-migrate.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: myapp-db-migrate
spec:
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: migrate
          image: myapp:3.6.0
          command: ["bundle", "exec", "rake", "db:migrate"]
```

Each migration step belongs to the version that introduces the breaking change. If version 3.6.0 requires a schema migration, the migration lives in the slot `3/` directory.

#### Example: simple app with waypoint

```yaml
# myapp/app.yaml
name: myapp
latest: "2"
upgrade:
  from:
    - version: ">=1.0.0"
      via: "1"
    - version: "<1.0.0"
      blocked: true
      notes: "Versions before 1.0.0 are not supported"
  preUpgrade:
    backup: recommended
```

This creates a 2-step upgrade path: `1.x → slot "1" (e.g., version 1.0.0-1) → slot "2" (e.g., version 2.0.0)`. The waypoint at `versions/1/` is a complete app package used as an intermediate step.

### Adding a New Version

When an upstream app releases a new version, you update the Wild Directory package to track it. The process depends on whether the new version has breaking changes.

#### Simple version bump (no breaking changes)

Most version updates are simple — update the container image tag, adjust any changed config, and update the version in `manifest.yaml`. No directory rename or `app.yaml` change needed.

```bash
# 1. Update files inside the existing slot
#    - Bump version in manifest.yaml (e.g., 1.2.0 → 1.3.0)
#    - Update container image tags in deployment YAMLs
#    - Adjust defaultConfig if the new version adds/changes config
vi wild-directory/myapp/versions/1/manifest.yaml
vi wild-directory/myapp/versions/1/deployment.yaml

# 2. app.yaml doesn't change — latest still points to slot "1"

# 3. Test
wild app add myapp && wild app deploy myapp
```

The directory structure stays the same:
```
myapp/
├── app.yaml               # latest: "1" (unchanged)
└── versions/
    └── 1/
        ├── manifest.yaml   # version: 1.3.0 (bumped)
        └── *.yaml
```

#### Version bump with breaking changes (waypoint required)

When the new version can't safely upgrade from all previous versions — e.g., a database schema change requires stepping through an intermediate version — create a new slot for the new major version, keep the old slot as a waypoint, and add routing rules.

```bash
# 1. The current slot (2/) becomes a waypoint — leave it in place
# 2. Create a new slot for the new major version
mkdir -p wild-directory/myapp/versions/3
# ... add new version files (manifest.yaml, kustomization.yaml, *.yaml) ...

# 3. Update app.yaml: point latest to new slot, add upgrade routing rules
```

```yaml
# app.yaml
name: myapp
latest: "3"
upgrade:
  from:
    - version: ">=2.5.0"            # 2.5.x can upgrade directly
    - version: ">=2.0.0"
      via: "2"                       # Older 2.x must pass through slot 2 first
    - version: "<2.0.0"
      blocked: true
      notes: "Upgrade to 2.x first. See upstream migration guide."
  preUpgrade:
    backup: recommended
```

The resulting directory:
```
myapp/
├── app.yaml               # latest: "3", upgrade routing rules
└── versions/
    ├── 3/                  # New latest (manifest.yaml has version: 3.0.0)
    │   ├── manifest.yaml
    │   └── *.yaml
    └── 2/                  # Waypoint (manifest.yaml has version: 2.5.0)
        ├── manifest.yaml
        └── *.yaml
```

#### Version bump with database migrations

When the new version requires a schema migration (e.g., `ALTER TABLE`, new indexes, data transformations), add migration job files to the slot directory and reference them from the version's `manifest.yaml`. Since this is a minor/patch update within the same major version, update files in-place in the existing slot.

```bash
# 1. Update files inside the existing slot
#    - Bump version in manifest.yaml (e.g., 2.0.0 → 2.1.0)
#    - Update container image tags in deployment YAMLs
vi wild-directory/myapp/versions/2/manifest.yaml
vi wild-directory/myapp/versions/2/deployment.yaml

# 2. Add migration job files
mkdir -p wild-directory/myapp/versions/2/migrations
```

Create the migration job:
```yaml
# versions/2/migrations/pre-deploy.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: myapp-migrate-2-1-0
  namespace: myapp
spec:
  backoffLimit: 3
  template:
    spec:
      restartPolicy: OnFailure
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: migrate
          image: myapp:2.1.0
          command: ["bundle", "exec", "rake", "db:migrate"]
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: myapp-secrets
                  key: dbUrl
```

Reference the migration in the version manifest:
```yaml
# versions/2/manifest.yaml
version: 2.1.0
upgrade:
  migrations:
    pre:
      - migrations/pre-deploy.yaml
defaultConfig:
  # ...
```

`app.yaml` doesn't change — `latest` still points to slot `"2"`.

Migration jobs must be **idempotent** — safe to re-run if an upgrade is retried after a partial failure. Use `CREATE IF NOT EXISTS`, `ALTER TABLE IF NOT EXISTS`, etc.

**Pre vs post migrations:**
- `pre` — runs before deploying the new version's manifests (schema changes that the new code needs)
- `post` — runs after deploying (data backfills, cleanup that the old code didn't need)

#### Version bump with config key renames

When a version renames config keys (e.g., `dbHost` → `db.host`), use `configMigrations` to automatically rename them during upgrade:

```yaml
# versions/2/manifest.yaml
version: 2.1.0
upgrade:
  configMigrations:
    dbHost: db.host
    dbPort: db.port
    dbName: db.name
defaultConfig:
  db:
    host: "{{ .apps.pg.host }}"
    port: "5432"
    name: myapp
```

The system renames the keys in the instance's `config.yaml` before recompiling templates with the new version.

### Dependency Configuration

- Each dependency in `requires` can have:
  - `name`: The app name to depend on (any app with a matching `is` field can satisfy this requirement)
  - `alias`: Optional reference name for templates (defaults to `name`)

### Manifest Template Variables (configuration and secrets)

#### Manifest Template Variable Sources

1. Standard Wild Cloud variables: `{{ .cloud.* }}`, `{{ .cluster.* }}`, `{{ .operator.* }}`
2. App-specific variables: `{{ .app.* }}` - resolved from current app's config
3. Dependency variables: `{{ .apps.<ref>.* }}` - resolved using app reference mapping
4. App-specific secrets (in 'defaultSecrets' ONLY): `{{ secrets.* }}`

#### Available Configuration Variiables

Here's a comprehensive rundown of all config variables that get set during cluster and service setup in config.yaml:

##### operator (Set during initial setup)

  - operator.email - Email for cluster operator/admin

##### cloud (Infrastructure-level settings)

###### DNS Configuration:
- cloud.dnsmasq.ip - IP address of the DNS server (Wild Central)
- cluster.internalDns.externalResolver - External DNS resolver (e.g., 1.1.1.1, 8.8.8.8)

###### Network Configuration:

- cloud.router.ip - Router gateway IP
- cloud.router.dynamicDns - Dynamic DNS hostname (optional)
- cloud.dhcpRange - DHCP range for the network (e.g., "192.168.8.34,192.168.8.79")
- cloud.dnsmasq.interface - Network interface for dnsmasq

###### Domain Configuration:

- cloud.baseDomain - Base domain for the cloud (e.g., "payne.io")
- cloud.domain - Full cloud domain (e.g., "cloud2.payne.io")
- cloud.internalDomain - Internal cluster domain (e.g., "internal.cloud2.payne.io")

###### Storage Configuration (NFS Service):

- cloud.nfs.host - NFS server hostname/IP
- cloud.nfs.mediaPath - NFS export path for media storage
- cloud.nfs.storageCapacity - NFS storage capacity (e.g., "50Gi", "1Ti")

###### Registry Configuration (Docker Registry Service):

- cloud.dockerRegistryHost - Docker registry hostname (e.g., "registry.internal.cloud2.payne.io")

###### Backup Configuration:

- cloud.backup.root - Root path for backups

##### cluster (Kubernetes cluster settings)

###### Basic Cluster Info:

- cluster.name - Cluster name identifier
- cluster.hostnamePrefix - Prefix for node hostnames

###### Node Configuration:

- cluster.nodes.talos.version - Talos Linux version (e.g., "v1.11.5")
- cluster.nodes.talos.schematicId - Talos Image Factory schematic ID
- cluster.nodes.control.vip - Virtual IP for control plane
- cluster.nodes.active.* - Individual node configurations with:
  - role - "controlplane" or "worker"
  - interface - Network interface name
  - disk - Disk device path
  - currentIp - Current IP address
  - targetIp - Target IP address
  - configured - Configuration status
  - applied - Applied status
  - maintenance - Maintenance mode
  - schematicId - Node-specific schematic ID
  - version - Node-specific Talos version

###### MetalLB Service:

- cluster.ipAddressPool - IP range for MetalLB (e.g., "192.168.8.80-192.168.8.89")
- cluster.loadBalancerIp - Primary load balancer IP (e.g., "192.168.8.80")

###### Cert-Manager Service:

- cluster.certManager.cloudflare.domain - Cloudflare domain for DNS-01 challenge
- cluster.certManager.cloudflare.zoneID - Cloudflare zone ID

###### ExternalDNS Service:

- cluster.externalDns.ownerId - Unique identifier for this cluster's DNS records

###### Docker Registry Service:

- cluster.dockerRegistry.storage - Storage size for registry (e.g., "10Gi")

##### apps (Application configurations)

Each app added to the cluster gets its own section under apps.<app-name> with app-specific configuration from the app's manifest. Common patterns include:

Standard app fields:
- apps.<name>.namespace - Kubernetes namespace
- apps.<name>.domain - App domain (e.g., "ghost.cloud2.payne.io")
- apps.<name>.externalDnsDomain - Domain for external DNS
- apps.<name>.tlsSecretName - TLS certificate secret name
- apps.<name>.image - Container image
- apps.<name>.port - Service port
- apps.<name>.storage - Persistent volume size
- apps.<name>.timezone - Timezone setting

Database-dependent apps:
- apps.<name>.dbHost / dbHostname - Database hostname
- apps.<name>.dbPort - Database port
- apps.<name>.dbName - Database name
- apps.<name>.dbUser / dbUsername - Database user

SMTP-enabled apps:
- apps.<name>.smtp.host - SMTP server
- apps.<name>.smtp.port - SMTP port
- apps.<name>.smtp.user - SMTP username
- apps.<name>.smtp.from - From address
- apps.<name>.smtp.tls - TLS enabled
- apps.<name>.smtp.startTls - STARTTLS enabled

Configuration Flow

1. Initial Setup: operator.email, basic cloud.* settings
2. Cluster Bootstrap: cluster.name, cluster.nodes.* settings
3. Infrastructure Services: Each service prompts for its serviceConfig from its manifest
   - MetalLB → cluster.ipAddressPool, cluster.loadBalancerIp
   - Cert-Manager → cluster.certManager.*
   - ExternalDNS → cluster.externalDns.ownerId
   - NFS → cloud.nfs.*
   - Docker Registry → cloud.dockerRegistryHost, cluster.dockerRegistry.storage
4. Apps: Each app adds its configuration under apps.<name>.* based on its manifest (including SMTP as an infrastructure app at apps.smtp.*)

#### Manifest App Reference Resolution:

When you use `{{ .apps.<ref>.* }}` in templates:
1. System checks if `<ref>` matches any dependency's `alias` field
2. If no alias match, checks if `<ref>` matches any dependency's `name` field
3. Uses the `installedAs` value (automatically added when the app is added) to find actual app configuration in `config.yaml`

All manifest template variables must be defined in one of these locations.

**Important:** In the rest of the app templates, ALL configuration keys referenced in templates (via `{{ .key }}`) must be defined in `defaultConfig`. Only the app config is available to app templates.

### Kustomization (`kustomization.yaml`)

The kustomization file defines how Kubernetes resources are built and applies Wild Cloud's standard labels.

```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: immich
labels:
  - includeSelectors: true
    pairs:
      app: immich
      managedBy: kustomize
      partOf: wild-cloud
resources:
  - deployment-server.yaml
  - deployment-machine-learning.yaml
  - deployment-microservices.yaml
  - ingress.yaml
  - namespace.yaml
  - pvc.yaml
  - service.yaml
  - db-init-job.yaml
```

#### Kustomization Requirements

- **Namespace**: Must match the app name
- **Labels**: Must include standard Wild Cloud labels with `includeSelectors: true`
- **Resources**: List all Kubernetes manifest files

#### Labeling Strategy

Wild Cloud uses Kustomize's `includeSelectors: true` feature to automatically apply standard labels to all resources AND their selectors:

```yaml
labels:
  - includeSelectors: true
    pairs:
      app: myapp              # App name (matches directory)
      managedBy: kustomize
      partOf: wild-cloud
```

This means individual resources can use simple, component-specific selectors like `component: web`, and Kustomize will automatically expand them to include all Wild Cloud labels.

**Do NOT use Helm-style labels** (`app.kubernetes.io/name`, `app.kubernetes.io/instance`). Use simple component labels (`component: web`, `component: worker`, etc.) instead.

## Configuration Templates

### Gomplate Templating

Resource files in this repository are **templates** that get compiled when users add apps via the web app, CLI, or API. Only variables defined in the manifest file's 'defaultConfig' section are available to the resource templates. Use gomplate syntax to reference configuration:

### Ingress

Use this as the standard ingress template for all apps:

```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
  annotations:
    external-dns.alpha.kubernetes.io/target: {{ .externalDnsDomain }}
    external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
    external-dns.alpha.kubernetes.io/ttl: "60"
spec:
  ingressClassName: traefik
  rules:
  - host: {{ .domain }}
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp
            port:
              number: 80
  tls:
  - hosts:
    - {{ .domain }}
    secretName: {{ .tlsSecretName }}
```

**Rules:**
- Always use `spec.ingressClassName: traefik` — do NOT use the `kubernetes.io/ingress.class` annotation (deprecated)
- Do NOT add `cert-manager.io/cluster-issuer` — the wildcard TLS cert is pre-distributed to app namespaces and managed centrally; adding this annotation causes cert-manager to issue a wrong per-app cert that overwrites the wildcard secret
- Do NOT add Traefik redirect annotations — HTTPS redirect is handled globally by Traefik
- `externalDnsDomain`, `domain`, and `tlsSecretName` must be defined in `defaultConfig`

### External DNS

The `external-dns.alpha.kubernetes.io/target` annotation creates a CNAME from the app subdomain to the cluster domain (e.g., `myapp.cloud.example.com` → `cloud.example.com`). Always set `cloudflare-proxied: "false"` since Wild Cloud manages its own TLS.

## App Dependencies and Reference Mapping

### How Dependency References Work

When an app depends on other apps, the reference system allows flexibility in naming while maintaining clear relationships:

1. **Define dependencies** in your manifest with optional aliases:
```yaml
requires:
  - name: postgres      # Actual app to depend on
    alias: db          # Optional: how to reference it in templates
  - name: redis        # No alias means use 'redis' as reference
```

2. **At installation time**, the system:
   - Prompts user to map dependencies to actual installed apps
   - Sets `installedAs` field in the local app manifest to track the mapping
   - Example: User might have `postgres-primary` installed, mapped to the `db` dependency

### Example: Multiple Database Instances

If a user has multiple PostgreSQL instances:
```yaml
# User's config.yaml
apps:
  postgres-primary:
    hostname: primary.postgres.svc.cluster.local
  postgres-analytics:
    hostname: analytics.postgres.svc.cluster.local
```

When adding an app that requires postgres, they can choose which instance to use, and the system tracks this in the manifest's `installedAs` field.

## Database Patterns

### Database Initialization Jobs

Apps requiring PostgreSQL or MySQL should include a database initialization job (`db-init-job.yaml`):

**Purpose:**
- Creates the application database (if it doesn't exist)
- Creates/updates the application user with proper credentials
- Grants necessary permissions
- Installs required database extensions (e.g., PostgreSQL's `vector`, `cube`, `earthdistance`)

**Implementation requirements:**
- Use `restartPolicy: OnFailure`
- Include in `kustomization.yaml` resources
- Use appropriate security context (e.g., `runAsUser: 999` for PostgreSQL)
- **Write idempotent SQL**: For MySQL, use `CREATE USER IF NOT EXISTS ... ; ALTER USER ...` so re-running the job after a redeploy doesn't leave stale passwords. See "MySQL db-init User Password Idempotency" in Common Gotchas.

**Example apps:** `immich`, `gitea`, `openproject`, `discourse`

### Database URL Configuration

When apps need database URLs with embedded credentials, **always use a dedicated `dbUrl` secret**.

❌ **Wrong** - Kustomize cannot process runtime env var substitution:
```yaml
- name: DB_URL
  value: "postgresql://user:$(DB_PASSWORD)@host/db"  # This won't work!
```

✅ **Correct** - Use a dedicated secret:
```yaml
- name: DB_URL
  valueFrom:
    secretKeyRef:
      name: myapp-secrets
      key: apps.myapp.dbUrl
```

Add `apps.myapp.dbUrl` to your manifest's `defaultSecrets`, and the system will generate the complete URL with embedded credentials automatically when the app is added.

### Backup/Restore Database Name Conventions

Wild Cloud's backup/restore system uses blue-green deployments. During restore, a standby copy of the app is created with a colored database name (e.g., `myapp_green`). The system automatically patches env vars in your Kubernetes resources to point to the standby database.

**How it works:** The restore system compiles your kustomize resources, finds env vars whose values match the original database name, and generates kustomize JSON patches to replace them with the standby database name. It uses env var naming conventions to distinguish database name fields from username fields (since both often have the same value).

**Env var naming guidelines for database-related fields:**

- **Database name env vars** should contain one of: `DATABASE`, `DB_NAME`, `DBNAME`, or `__DATABASE` in the env var name (e.g., `LISTMONK_db__database`, `DB_NAME`, `POSTGRES_DB`)
- **Database URL env vars** are detected by containing `://` in the value (e.g., `postgresql://user:pass@host/dbname`)
- **Username env vars** should contain `USER` in the name (e.g., `DB_USER`, `LISTMONK_db__user`) — these will NOT be patched even if the value matches the database name
- Avoid env var names that are ambiguous about whether they hold a database name or username

**Example — correct naming:**
```yaml
env:
  - name: DB_NAME           # Will be patched (contains "DB_NAME")
    value: myapp
  - name: DB_USER           # Will NOT be patched (contains "USER")
    value: myapp
  - name: DATABASE_URL      # Will be patched (contains "://")
    value: "postgresql://myapp:secret@postgres/myapp"
```

## Deployment Strategy

Apps using `ReadWriteOnce` (RWO) persistent volumes **must** set `strategy: type: Recreate` on their Deployment. RWO volumes can only be attached to one pod at a time, so the default `RollingUpdate` strategy will cause Multi-Attach errors during updates (the new pod can't mount the volume while the old pod still holds it).

```yaml
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      component: web
```

## Security Requirements

### Security Contexts

**All pods must comply with Pod Security Standards.** Include security contexts at both pod and container levels:

```yaml
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 999        # Use appropriate non-root UID
        runAsGroup: 999       # Use appropriate GID
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: container-name
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: [ALL]
          readOnlyRootFilesystem: false  # Set to true when possible
```

**Common user IDs:**
- PostgreSQL: `runAsUser: 999`
- Redis: `runAsUser: 999`
- MySQL: Consult the container image documentation

### Secrets Management

Secrets are managed through two mechanisms: default secrets for the app itself and required secrets from dependencies.

**In manifest:**
```yaml
defaultSecrets:
  key: dbPassword      # This app's database password
  key: apiKey          # This app's API key
requiredSecrets:
  - db.password       # Password from postgres dependency (aliased as 'db')
  - redis.auth        # Auth from redis dependency
```

**In resources:**
```yaml
env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: myapp-secrets
        key: dbPassword    # Points to the default secret
  - name: POSTGRES_PASSWORD
    valueFrom:
      secretKeyRef:
        name: myapp-secrets
        key: db.password    # Points to the required secret
```

**Secret workflow:**
1. Define app's own secrets in `defaultSecrets` (key, default mappings)
2. Reference dependency secrets in `requiredSecrets` (list)
3. When adding an app, the system:
   - Generates random values for empty `defaultSecrets`
   - Copies referenced secrets from dependencies
   - Stores all in the instance's `secrets.yaml`
4. When deploying, creates a Kubernetes Secret named `<app-name>-secrets` containing:
   - All `defaultSecrets` with key format: `<key>`
   - All `requiredSecrets` with key format: `<app-ref>.<key>`

**Key collision handling:** If the same key exists in both `defaultSecrets` and `requiredSecrets`, the `requiredSecrets` value takes precedence. Authors should ensure their local secrets don't collide with their required secrets.

**Important:** Never commit `secrets.yaml` to Git. Templates should only reference secrets, never contain actual secret values.

## Converting from Helm Charts

Wild Cloud prefers Kustomize over Helm for simplicity and Git-friendliness. When an official Helm chart exists, convert it rather than creating manifests from scratch.

### Conversion Process

1. **Extract and render the Helm chart:**
```bash
helm fetch --untar --untardir charts repo/chart-name
helm template --output-dir base --namespace myapp --values values.yaml myapp charts/chart-name
cd base/chart-name
```

2. **Add namespace manifest:**
```bash
cat <<EOF > namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: myapp
EOF
```

3. **Create kustomization:**
```bash
kustomize create --autodetect
```

4. **Convert to Wild Cloud format:**
   - Create `manifest.yaml` with app metadata
   - Replace hardcoded values with gomplate variables (e.g., `{{ .cloud.domain }}`)
   - Update secrets to use dotted-path convention
   - Replace Helm labels with Wild Cloud standard labels
   - Add `includeSelectors: true` to kustomization
   - Use simple component labels (`component: web`, not `app.kubernetes.io/name`)
   - Add security contexts to all pods
   - Add external-dns annotations to ingresses

### Example Label Migration

❌ **Helm style:**
```yaml
labels:
  app.kubernetes.io/name: myapp
  app.kubernetes.io/instance: release-name
  app.kubernetes.io/component: server
```

✅ **Wild Cloud style:**
```yaml
# In kustomization.yaml (applied automatically)
labels:
  - includeSelectors: true
    pairs:
      app: myapp
      managedBy: kustomize
      partOf: wild-cloud

# In individual resources
labels:
  component: server  # Simple component label
```

## Common Gotchas

Lessons learned from deploying apps to Wild Cloud. These are the non-obvious issues that cause failures.

### Always Verify Docker Image Availability

Before packaging an app, confirm the image is **publicly accessible without authentication**:

```bash
docker pull registry.example.com/org/image:tag
# or check via curl for ghcr.io:
curl -s "https://ghcr.io/token?scope=repository:org/image:pull&service=ghcr.io" | python3 -c "import json,sys; print(json.load(sys.stdin).get('token','no token')[:20])"
```

Common pitfalls:
- **Bitnami images** moved away from Docker Hub — their `bitnami/appname` images may no longer be available; check `oci.bitnami.com` or use the official upstream image instead
- **ghcr.io images** frequently require authentication even when the repo is "public" on GitHub — test anonymous pull before committing to them
- **compdemocracy/polis** and some other community apps use private AWS ECR or require GitHub tokens — these cannot be packaged for general use without finding a public mirror

If no public image exists, note it clearly in the wild-directory `app.yaml` description rather than packaging a broken app.

### Re-run `wild app add` After Template Changes

`wild app deploy <app>` applies whatever is already compiled in the instance's `apps/` directory. It does **not** recompile templates from wild-directory. After modifying any template file in wild-directory, always re-run:

```bash
wild app add <app>   # recompiles templates into the instance dir
wild app deploy <app>  # applies the recompiled files
```

### PostgreSQL Connection Strings: Always Include `?sslmode=disable`

Wild Cloud's internal PostgreSQL does not have SSL enabled. Any app that constructs a PostgreSQL connection string must include `?sslmode=disable`, or the connection will fail with "The server does not support SSL connections."

```yaml
defaultSecrets:
  - key: dbUrl
    default: "postgresql://{{ .app.db.user }}:{{ .secrets.dbPassword }}@{{ .app.db.host }}:{{ .app.db.port }}/{{ .app.db.name }}?sslmode=disable"
```

Also add `PGSSLMODE=disable` as an env var for apps that use libpq directly.

### Redis URLs Must Include the Password

Wild Cloud's Redis requires authentication. Apps that accept a Redis URL must include the password:

```yaml
# In manifest.yaml requiredSecrets:
requiredSecrets:
  - redis.password

# In deployment.yaml env — use K8s env var expansion for the URL:
- name: REDIS_PASSWORD
  valueFrom:
    secretKeyRef:
      name: myapp-secrets
      key: redis.password
- name: REDIS_URL
  value: "redis://:$(REDIS_PASSWORD)@{{ .redis.host }}:6379"
```

The `$(VAR_NAME)` syntax is evaluated by Kubernetes at pod start from other env vars in the same container spec.

### Runtime-Specific Resource Defaults

Default resource requests vary by runtime. The standard 256Mi/50m is too low for heavier runtimes:

| Runtime | Suggested request | Notes |
|---------|------------------|-------|
| Go/Rust | 64–128Mi | Very efficient |
| Python/Ruby/PHP | 128–256Mi | Typical range |
| Node.js | 256–512Mi | Add `NODE_OPTIONS=--max-old-space-size=<N>` to cap heap |
| JVM (Java/Kotlin/Scala) | 512Mi–1Gi | Set `-Xmx` to keep within limit |
| Celery/Sidekiq workers | 256–512Mi | Full worker process per instance |

For Node.js apps, cap memory use explicitly:
```yaml
- name: NODE_OPTIONS
  value: "--max-old-space-size=768"  # ~75% of 1Gi limit
```

### Health Check Probes for Django/FastAPI (ALLOWED_HOSTS)

Django and similar frameworks reject health probe requests because Kubernetes sends them directly to the pod IP, which is not in `ALLOWED_HOSTS`. Fix by adding a `Host` header to the probe:

```yaml
livenessProbe:
  httpGet:
    path: /
    port: 8000
    httpHeaders:
      - name: Host
        value: "{{ .domain }}"
```

### Images That Need Root + Special Capabilities

Some images run an init system (nginx, s6-overlay, custom entrypoint scripts) that requires elevated privileges to set up directories and drop to a lower-privileged user. These fail when `runAsNonRoot: true` or `capabilities.drop: [ALL]` is set.

**Pattern for nginx-based images** (e.g., community app frontends):
```yaml
spec:
  securityContext:
    runAsNonRoot: false
    runAsUser: 0
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: [ALL]
        add: [CHOWN, SETUID, SETGID]  # needed for nginx to drop to worker user
      readOnlyRootFilesystem: false
```

**Pattern for linuxserver.io / s6-overlay images** (e.g., BookStack, most linuxserver images):
```yaml
spec:
  securityContext:
    runAsNonRoot: false
    runAsUser: 0
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    securityContext:
      readOnlyRootFilesystem: false  # Only this — do NOT drop capabilities
```

Signs you need this pattern:
- `chown(...) failed (1: Operation not permitted)` — add `CHOWN` capability
- `Permission denied` on a directory during startup — try running as root
- Container exits immediately with code 1 despite image being "official"

### Hex-Format Secrets

Some apps require secrets in specific formats (e.g., exactly 64 hex chars for Outline). The default Wild Cloud secret generator produces random alphanumeric strings. Use `crypto.SHA256` in the manifest `default` field to derive a hex string from another secret:

```yaml
defaultSecrets:
  - key: utilsSecret           # generated first (alphanumeric)
  - key: secretKey
    default: '{{ crypto.SHA256 .secrets.utilsSecret }}'  # hex derivative
```

The order of `defaultSecrets` matters — a referenced secret must appear before the entry that references it.

### Odoo (and Other Apps) Requiring Explicit Database Initialization

Some apps don't auto-initialize their database — they need to be told which database to use AND to install their schema. Odoo is the most common case: without explicit `-d DATABASE -i base` args, it starts in multi-database manager mode and all health checks fail.

```yaml
# In the container spec:
args: ["-d", "{{ .db.name }}", "-i", "base"]
```

The `-i base` flag is **idempotent**: it installs on first run and updates on subsequent runs. The downside is slightly slower startup on subsequent restarts. For very large apps, use an initContainer to run initialization separately from the main container.

Combine with a `startupProbe` to give the initialization time to complete:

```yaml
startupProbe:
  httpGet:
    path: /web/health
    port: 8069
  failureThreshold: 40    # 40 × 30s = 20 minutes
  periodSeconds: 30
  timeoutSeconds: 10
livenessProbe:
  httpGet:
    path: /web/health
    port: 8069
  initialDelaySeconds: 0  # startupProbe takes over startup detection
  periodSeconds: 30
  failureThreshold: 6
```

The `startupProbe` runs until it succeeds (or exhausts `failureThreshold`), at which point the liveness and readiness probes kick in with `initialDelaySeconds: 0`. This is cleaner than inflating `initialDelaySeconds` on the main probes.

### Gancio Re-deploy: "Non empty db" Crash

Gancio stores setup state in `config.json` on its data PVC. If the PVC is deleted or the app is re-added without wiping the database, gancio finds tables in PostgreSQL but no `config.json` and refuses to start.

**Fix**: Drop the schema before restarting:
```bash
kubectl exec -n postgres <postgres-pod> -- psql -U postgres gancio \
  -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public; GRANT ALL ON SCHEMA public TO gancio; GRANT ALL ON SCHEMA public TO public;"
```

Scale the deployment to 0 before dropping the schema to avoid active connection errors.

### Heavy-Framework Startup Times and Probe Delays

Frameworks with heavy startup initialization (Laravel, Rails, Spring Boot, Symfony) run tasks like package discovery, autoload generation, database migrations, and asset compilation before accepting HTTP requests. This takes 2–3 minutes — much longer than the default 60-second initial delay.

The failure mode is a restart loop: pod runs for ~2 minutes, liveness probe fires and fails, container is killed, new pod starts. The app never gets far enough to become Ready.

For these runtimes, use extended probe delays:

```yaml
livenessProbe:
  tcpSocket:
    port: 8080
  initialDelaySeconds: 120    # Give the framework time to fully initialize
  periodSeconds: 30
  failureThreshold: 6         # Tolerate transient failures during startup
readinessProbe:
  tcpSocket:
    port: 8080
  initialDelaySeconds: 60
  periodSeconds: 15
  failureThreshold: 3
```

**Runtimes that typically need this:**
| Runtime | Suggested liveness initialDelaySeconds |
|---------|----------------------------------------|
| PHP/Laravel/Symfony | 120s |
| Ruby on Rails | 90–120s |
| Spring Boot (JVM) | 90–120s |
| Django (with migrations) | 60–90s |
| Node.js | 60s (usually sufficient) |

### MySQL db-init User Password Idempotency

The common pattern `CREATE USER IF NOT EXISTS 'user'@'%' IDENTIFIED BY '${PASSWORD}'` only sets the password when the user is first created. If the MySQL user already exists from a previous deployment (e.g., after deleting and re-adding an app without dropping the database), the password is silently unchanged and the new deployment fails with "Access denied".

Use the `CREATE ... IF NOT EXISTS` + `ALTER USER` pattern to make db-init jobs idempotent regardless of prior state:

```sql
CREATE DATABASE IF NOT EXISTS ${DB_DATABASE_NAME};
CREATE USER IF NOT EXISTS '${DB_USERNAME}'@'%' IDENTIFIED BY '${DB_PASSWORD}';
ALTER USER '${DB_USERNAME}'@'%' IDENTIFIED BY '${DB_PASSWORD}';
GRANT ALL PRIVILEGES ON ${DB_DATABASE_NAME}.* TO '${DB_USERNAME}'@'%';
FLUSH PRIVILEGES;
```

The `ALTER USER` line is safe even when the user was just created — it's a no-op in that case. This applies to MySQL/MariaDB only; PostgreSQL's `CREATE USER IF NOT EXISTS` does not exist but `DO $$ BEGIN IF NOT EXISTS ... END $$` patterns serve the same purpose.

## Validation Checklist

Before submitting a new or modified app, verify:

- [ ] **App Meta (`app.yaml`)**
  - [ ] `name` matches directory name
  - [ ] `latest` points to a valid version in `versions/`
  - [ ] `description` present
  - [ ] `upgrade` rules correct (if applicable)

- [ ] **Version Manifest (`versions/{slot}/manifest.yaml`)**
  - [ ] `version` field present with full version string (e.g., `1.135.3-1`)
  - [ ] Slot directory follows naming convention (major version, e.g., `1`, `v1`)
  - [ ] All required fields present (`version`, `defaultConfig`)
  - [ ] All template variables defined in `defaultConfig`
  - [ ] `defaultSecrets` uses maps with 'key' and 'default' attributes
  - [ ] `requiredSecrets` references use `<app-ref>.<key>` format
  - [ ] Dependencies listed in `requires` with optional `alias` fields
  - [ ] Manifest template references match dependency aliases or names

- [ ] **Kustomization**
  - [ ] Includes standard Wild Cloud labels with `includeSelectors: true`
  - [ ] Namespace matches app name
  - [ ] All resource files listed under `resources:`

- [ ] **Resources**
  - [ ] Security contexts on all pods (both pod-level and container-level)
  - [ ] `strategy: type: Recreate` on deployments with ReadWriteOnce PVCs
  - [ ] Simple component labels, no Helm-style labels
  - [ ] Ingresses include external-dns annotations
  - [ ] Database apps include init jobs (if applicable)

- [ ] **Images**
  - [ ] Docker images are publicly accessible (test anonymous pull)
  - [ ] Image tags are specific versions, not `latest` or branch names
  - [ ] Architecture constraints (amd64-only) noted in manifest if applicable

- [ ] **Testing**
  - [ ] Templates compile successfully with sample config
  - [ ] App deploys without errors in test cluster
  - [ ] All dependencies work correctly

## Contributing

Contributions are welcome! To contribute:

1. Fork the repository
2. Create a new app directory following the structure above
3. Test your app thoroughly
4. Submit a pull request with:
   - Description of the app and its purpose
   - Any special configuration notes
   - Dependencies required

## Notice: Third-Party Software

The Kubernetes manifests and Kustomize files in this directory are designed to deploy **third-party software**.

Unless otherwise stated, the software deployed by these manifests **is not authored or maintained** by this project. All copyrights, licenses, and responsibilities for that software remain with the respective upstream authors.

These files are provided solely for convenience and automation. Users are responsible for reviewing and complying with the licenses of the software they deploy.

This project is licensed under the GNU AGPLv3 or later, but this license does **not apply** to the third-party software being deployed.

See individual deployment directories for upstream project links and container sources.