wild-directory/ADDING-APPS-NOTES.md

# Adding Apps - Notes & Documentation Improvements

## Documentation Improvement Candidates

Issues and gaps discovered while adding apps. These should be folded back into ADDING-APPS.md.

### 1. Node.js app memory requirements
Node.js apps (NocoDB, Outline, etc.) need more than 512Mi memory. NocoDB crashed with OOM at 512Mi.
- Recommended: 1Gi limit, 512Mi request for Node.js apps
- Set `NODE_OPTIONS=--max-old-space-size=768` to keep heap within the limit
- **Doc suggestion**: Add a section in ADDING-APPS.md about runtime-specific resource defaults:
  - Node.js apps: 1Gi limit, 512Mi request; add NODE_OPTIONS env var
  - JVM apps (Java/Kotlin): 1Gi+ limit
  - Python/Ruby apps: 512Mi typically sufficient
  - Go/Rust apps: 256-512Mi typically sufficient

### 2. db-init-job is repeated boilerplate
The db-init-job.yaml for PostgreSQL apps is nearly identical across all apps. Only the app name, namespace, and secret name change. 
- **Doc suggestion**: Provide a copy-paste template for the standard PostgreSQL db-init-job with placeholders marked clearly.

### 4. Apps requiring hex-encoded secrets
Some apps (Outline) require secrets in specific formats (e.g., exactly 64 hex chars). The default Wild Cloud `GenerateSecret` produces alphanumeric (not hex) strings.
- **Solution**: Use `crypto.SHA256` gomplate function in the manifest `default` field to generate a valid 64-char hex string from another secret:
  ```yaml
  defaultSecrets:
    - key: utilsSecret
    - key: secretKey
      default: '{{ crypto.SHA256 .secrets.utilsSecret }}'
  ```
- **Doc suggestion**: Add this pattern to ADDING-APPS.md with examples of apps that need it. Note that the order of `defaultSecrets` matters — referenced secrets must come first.

### 6. Always use `?sslmode=disable` in PostgreSQL connection strings
The Wild Cloud internal PostgreSQL does not have SSL enabled. Apps that construct PostgreSQL connection strings must include `?sslmode=disable`, and deployments should set `PGSSLMODE=disable`. Without it, apps crash with "The server does not support SSL connections."
- Already done correctly in: listmonk, gitea, discourse
- Missing from: outline (fixed), any new app
- **Doc suggestion**: Add to the ADDING-APPS.md dbUrl template examples: always append `?sslmode=disable` to postgres connection strings.

### 5. Re-running `wild app add` is required after template changes
When you modify files in wild-directory (e.g., fix a template), you must re-run `wild app add <app>` before deploying. The compiled templates in the instance `apps/` directory are NOT automatically updated when wild-directory changes. Only then will `wild app deploy` use the updated templates.
- **Doc suggestion**: Add to ADDING-APPS.md: "After modifying templates in wild-directory, always re-run `wild app add <app>` to recompile before deploying."

### 7. WriteFreely requires root user (jrasanen/writefreely image)
The `jrasanen/writefreely` image startup script creates `config.ini` in `/writefreely/` which is a root-owned directory. The image's default user is non-root, causing "Permission denied" errors at startup.
- **Fix**: Set `runAsUser: 0` and `runAsNonRoot: false` in the pod securityContext
- Still safe with `allowPrivilegeEscalation: false` and `capabilities.drop: ALL`
- **Doc suggestion**: Note that some community images require root. When an image fails with permission denied on startup, check if it needs root and add the security context override with capabilities dropped.

### 8. Redis URL must include password for authenticated Redis
Wild Cloud's Redis requires authentication. Apps that take a Redis URL must include the password in the URL: `redis://:password@host:6379`. 
- **Pattern**: Add `redis.password` to `requiredSecrets`, then use K8s env var expansion:
  ```yaml
  - name: REDIS_PASSWORD
    valueFrom:
      secretKeyRef:
        name: myapp-secrets
        key: redis.password
  - name: REDIS_URL
    value: redis://:$(REDIS_PASSWORD)@{{ .redis.host }}:6379
  ```
- The `$(VAR_NAME)` syntax is supported by Kubernetes in container env values (evaluated at pod start)
- **Doc suggestion**: Add this pattern to ADDING-APPS.md under "Redis Authentication" section.

### 3. Consistent health check paths across apps
Different apps use different health check paths (`/health`, `/healthz`, `/_health`, `/api/v1/health`). This should be documented per app type or noted that you must verify the actual path from the app's Docker docs.
- For apps where no valid HTTP health path exists (e.g., Typebot viewer which only serves chatbot URLs), use `tcpSocket` probe instead of `httpGet`.

### 9. Linuxserver.io and s6-overlay images require root + capabilities
Images using s6-overlay init system (most linuxserver.io images) must run as root and need full capabilities to switch to their internal "abc" user.
- Set `runAsUser: 0, runAsNonRoot: false` in pod securityContext
- Do NOT set `allowPrivilegeEscalation: false` or `capabilities.drop: ALL` in container securityContext
- Only keep `readOnlyRootFilesystem: false`
- Same applies to Nextcloud (uses rsync/chown during setup)
- **Affects**: BookStack (linuxserver.io), any linuxserver.io image

### 10. CryptPad needs emptyDir + initContainer for config directory
The CryptPad image's `/cryptpad/config/` directory is in the image overlay filesystem and not writable by the container process even when running as root.
- **Fix**: Mount an emptyDir at `/cryptpad/config`, and use an initContainer to pre-seed `config.example.js` from the image into the emptyDir.
- The initContainer mounts the emptyDir at a different path (e.g., `/config-dest`) and copies the file.
- **CPAD_CONF** env var must be set to `/cryptpad/config/config.js` (startup script copies config.example.js to config.js if config.js doesn't exist).

### 11. Pixelfed image requires authentication (ghcr.io access restriction)
~~`ghcr.io/pixelfed/pixelfed` returns 403 Forbidden for anonymous pulls.~~
- **Resolved**: Use `ghcr.io/mattlqx/docker-pixelfed:v0.12.7-nginx` (community image, publicly accessible)
- Note: the mattlqx image uses port 80 (nginx), not 8080. Update `containerPort` and service `targetPort` accordingly.

### 12. Bitnami images no longer on Docker Hub
Bitnami moved their images from Docker Hub (`bitnami/appname`) to their own OCI registry. Apps that were packaged using Bitnami images (e.g., Ghost) need to be updated to use the official upstream image or Bitnami's new registry.
- **Ghost fix**: Switched from `bitnami/ghost:5.x` to official `ghost:5.130.6-alpine`
- The official ghost image uses different env vars (`database__connection__host` instead of `GHOST_DATABASE_HOST`) and a different mount path (`/var/lib/ghost/content` instead of `/bitnami/ghost`)
- Check image availability before packaging: `docker manifest inspect docker.io/bitnami/appname:tag`

### 13. Required secrets belong in app-secrets, not dep-secrets
Apps that use `requiredSecrets` (e.g., `mysql.rootPassword`) receive those secrets copied into their own `<app>-secrets` K8s Secret, under the key `<dep>.<key>` (e.g., `mysql.rootPassword`). db-init jobs and deployments must reference `<app>-secrets`, NOT `mysql-secrets`.

```yaml
# WRONG - mysql-secrets doesn't exist in the app namespace
secretKeyRef:
  name: mysql-secrets
  key: rootPassword

# CORRECT - the required secret is copied into ghost-secrets
secretKeyRef:
  name: ghost-secrets
  key: mysql.rootPassword
```

### 15. mattlqx/docker-pixelfed requires APP_PORT env var
The `ghcr.io/mattlqx/docker-pixelfed` community image generates nginx config from env vars using `sed`. The `listen` directive uses `APP_PORT`, which must be set explicitly or nginx crashes with "invalid number of arguments in 'listen' directive".
- Add `APP_PORT: "80"` to the deployment env vars
- The image listens on port 80 (nginx), not 8080 as the original pixelfed image did
- Both the web and worker deployments must mount the shared storage PVC — use ReadWriteMany access mode, not ReadWriteOnce, since both pods need it simultaneously

### 14. Your Priorities and Polis have no public Docker images
- `ghcr.io/citizensfoundation/your-priorities-app` — 403 Forbidden, images require auth
- `compdemocracy/polis-server` — private, no public Docker Hub or ghcr.io images
- Both apps require building custom images from source before they can be packaged
- Attempting to add these to wild-directory results in ImagePullBackOff

### 16. PHP/Laravel and other heavy-framework apps need extended probe delays
Laravel's startup process runs package discovery, autoload optimization, and encryption key generation before the HTTP server is ready. This commonly takes 2–3 minutes on cluster restart or first boot.
- The default `initialDelaySeconds: 60` is too short — the liveness probe fires before the app is ready, kills the container, and a restart loop begins
- Set `initialDelaySeconds: 120` and `failureThreshold: 6` for liveness probes on Laravel/Symfony apps
- Set `initialDelaySeconds: 60` for readiness probes
- The symptom is: pod shows `Running` for ~2 minutes then gets `Killing` due to liveness probe failure, followed by a new pod starting the same cycle
- The same applies to other frameworks with heavy startup initialization: Rails (`bundle exec`, asset precompilation), Spring Boot (JVM + Spring context loading), Django with migrations
- **Affects**: Ushahidi API (Laravel) — fixed with 120s initial delay + failureThreshold: 6

### 17. MySQL db-init `CREATE USER IF NOT EXISTS` doesn't update passwords
The common db-init pattern `CREATE USER IF NOT EXISTS 'user'@'%' IDENTIFIED BY '${PASSWORD}'` only sets the password at creation time. If the MySQL user already exists from a previous deployment (e.g., after `wild app delete` + `wild app add` without dropping the database), the password is silently left unchanged and the new deployment fails with "Access denied for user".
- **Symptom**: App starts, connects to MySQL, gets "Access denied" even though the db-init job completed successfully
- **Quick fix**: Exec into the MySQL pod and run: `ALTER USER 'user'@'%' IDENTIFIED BY 'current_password'; FLUSH PRIVILEGES;`
- **Better pattern** — make db-init jobs truly idempotent for both user creation and password:
  ```sql
  CREATE USER IF NOT EXISTS '${DB_USERNAME}'@'%' IDENTIFIED BY '${DB_PASSWORD}';
  ALTER USER '${DB_USERNAME}'@'%' IDENTIFIED BY '${DB_PASSWORD}';
  GRANT ALL PRIVILEGES ON ${DB_DATABASE_NAME}.* TO '${DB_USERNAME}'@'%';
  FLUSH PRIVILEGES;
  ```
- This `CREATE ... IF NOT EXISTS` + `ALTER USER` pattern is safe to run repeatedly regardless of whether the user existed before
- **Affects**: Ghost (MySQL backend) — hit this when redeploying after a previous partial test run

### 18. Gancio "Non empty db" crash on re-deploy
Gancio stores its setup state in `config.json` on the data PVC. If the PVC is lost or the app is deleted and redeployed with an existing database, gancio finds a non-empty DB but no `config.json` and refuses to start with "Non empty db! Please move your current db elsewhere than retry."
- **Fix**: Drop the database schema and let gancio recreate it:
  ```bash
  kubectl exec -n postgres <postgres-pod> -- psql -U postgres gancio \
    -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public; GRANT ALL ON SCHEMA public TO gancio; GRANT ALL ON SCHEMA public TO public;"
  kubectl scale deployment -n gancio gancio --replicas=0
  # (wait for pod to terminate)
  kubectl scale deployment -n gancio gancio --replicas=1
  ```
- Scale down first to release the DB connection before dropping the schema
- This only affects re-deployments; fresh installs work fine

### 19. Odoo requires explicit database name and `-i base` on first run
The official Odoo Docker image does not auto-initialize the database. Without specifying the database name and installing the base module, Odoo starts in multi-database manager mode and health checks fail with "Database not initialized".
- Add `args: ["-d", "DATABASE_NAME", "-i", "base"]` to the container spec
- The `-i base` flag installs Odoo's base module and creates all database tables on first run
- Subsequent runs with `-i base` are safe (idempotent) — it updates rather than reinstalls
- Use a `startupProbe` with `failureThreshold: 40, periodSeconds: 30` (20 minutes) since first-run initialization takes 10–15 minutes
- Once startup probe passes, the regular liveness/readiness probes with `initialDelaySeconds: 0` take over

### 20. Headscale v0.29+ CLI: inconsistent user flag types
In headscale v0.29, different commands accept different user identifiers:
- `preauthkeys create --user <id>` — requires numeric ID
- `auth register --user <username>` — accepts username string
- `users destroy --identifier <id>` — requires numeric ID, not a positional arg

```bash
# Always list users first to get the ID
headscale users list

headscale preauthkeys create --user 1           # numeric ID
headscale auth register --auth-id <id> --user payne  # username string OK
headscale users destroy --identifier 1          # numeric ID
```

Node registration flow: when a Tailscale client connects via browser (not pre-auth key), headscale
shows a command like `headscale auth register --auth-id hskey-authreq-XXXX --user USERNAME`.
Run it via `kubectl exec -n headscale deploy/headscale -- headscale auth register --auth-id <id> --user <username>`.

### 21. "No available server" from Traefik — kustomize label selector mismatch
If a deployment was originally created without kustomize labels in its selector (or with stale templates), re-deploying via `wild app deploy` cannot fix it because `spec.selector` on a Deployment is immutable once set. The Service selector will update (it's mutable) but pods won't match it, leaving the Service with no endpoints.
- **Symptom**: Traefik returns "No available server" even though pods are Running
- **Diagnosis**: `kubectl get endpoints <app> -n <namespace>` shows `<none>`; pod labels don't include `app: <name>` / `managedBy: kustomize` / `partOf: wild-cloud`
- **Fix**: Delete the affected deployments and re-apply kustomize — `kubectl delete deployment <name> -n <ns>` then `kubectl apply -k <instance>/apps/<app>/`
- **Root cause**: The Deployment selector is set correctly on first deploy, but if the resource already existed with only `component: web` (e.g., from a pre-kustomize deployment), the selector is stuck

### 22. Apps using same-origin iframes need SAMEORIGIN frame policy
The global Traefik `security-headers` middleware applies `X-Frame-Options: SAMEORIGIN` to all responses (changed from `DENY`). Apps that use iframes internally (like Etherpad's pad editor, which loads `../static/empty.html` in an iframe) were broken by the stricter `DENY` policy.
- **Root cause**: `DENY` prevents even same-origin iframes. `SAMEORIGIN` still blocks cross-origin embedding while allowing apps to embed their own sub-pages.
- **Symptoms**: JavaScript error `Blocked a frame with origin "https://..." from accessing a cross-origin frame` — occurs when an app creates a same-origin iframe and then the `X-Frame-Options: DENY` header causes the browser to block access to it.
- **Important**: Route-level Traefik middlewares run BEFORE entrypoint middlewares on the response path. A per-app middleware cannot override a global entrypoint middleware — the entrypoint middleware always runs last.
- **Fix applied globally**: Changed `frameDeny: true` to `X-Frame-Options: SAMEORIGIN` in `customResponseHeaders` in the crowdsec `security-headers` middleware.
- **Affects**: Etherpad (pad editor iframe), any app that loads sub-pages in iframes on the same domain.

### 24. nginx proxy containers: Docker DNS resolver (`127.0.0.11`) doesn't work in Kubernetes

Apps that use nginx as a frontend proxy (e.g., karrot-frontend) often ship with an nginx config template that uses Docker's embedded DNS resolver:

```nginx
resolver 127.0.0.11 valid=3s;
set $backend app-backend:8000;
proxy_pass http://$backend$request_uri;
```

This fails in Kubernetes with `send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11`. There's a second trap: **any nginx variable anywhere in the `proxy_pass` URL** (even `$request_uri` in the path) forces runtime DNS resolution and requires a `resolver` directive. Removing the resolver line but keeping `proxy_pass http://backend:8000$request_uri;` still fails with `no resolver defined to resolve backend`.

**Fix**: override the nginx config template via a ConfigMap mounted with `subPath`, and remove the variable from `proxy_pass` entirely:

```nginx
# Instead of:
resolver 127.0.0.11 valid=3s;
set $backend ${BACKEND};
proxy_pass http://$backend$request_uri;

# Use:
proxy_pass http://${BACKEND};
```

With no nginx variable in `proxy_pass`, nginx resolves the hostname at startup using the pod's `/etc/resolv.conf` (which points to CoreDNS). The full request URI is still forwarded automatically in a regex `location ~` block. Mount the ConfigMap with `subPath` to override just the template file without replacing the whole directory.

### 25. ConfigMap changes don't restart pods — and `subPath` mounts never auto-update

When you update a ConfigMap and run `wild app deploy` (`kubectl apply`), the ConfigMap object is updated but **pods are not restarted**. You must explicitly restart them:

```bash
kubectl rollout restart deployment/<name> -n <namespace>
```

There's a further subtlety: ConfigMaps mounted with `subPath` are **never** live-updated in the pod filesystem, even if you wait. The pod must be restarted to pick up changes regardless of the `subPath` mount update behavior.

This means: after any ConfigMap change, always follow up with a rollout restart for affected deployments.

### 26. The `includeSelectors: true` mismatch affects every deployment in the app — check them all

When diagnosing "no available server" from Traefik (note #21), it's easy to focus only on the deployment that serves the HTTP route and miss other deployments in the same namespace. The `includeSelectors: true` label propagation affects every resource in the kustomization — if any deployment pre-existed with only `component: X` labels, its Service will have `<none>` endpoints and dependent services (backends, workers, Redis) will silently fail.

**Always run this after any selector-related redeploy**:
```bash
kubectl get endpoints --all-namespaces | grep "<none>"
```

Check every deployment in the affected namespace, not just the one Traefik is routing to. A Redis pod with `<none>` endpoints will cause the backend to crash with `ConnectionError` even though the Redis pod itself shows `1/1 Running`.

**Checklist when fixing a selector mismatch in an app**:
1. `kubectl get endpoints -n <app>` — identify all services with `<none>`
2. `kubectl get pods -n <app> --show-labels` — confirm which pods have old labels
3. Delete ALL affected deployments (not just the web one), then redeploy
4. Verify all endpoints are populated before declaring the app fixed

### 23. Post-deploy admin scripts for apps that require manual first-run steps

Some apps can't complete setup through the web UI until an initial admin account is created via a CLI command. Package these as shell scripts in `scripts/` alongside the kustomize files so users have a repeatable, documented way to run them.

**When to add a script**: If the app's README or upstream docs require running a management command after deploy (creating a superuser, registering a node, inviting the first user, etc.) — wrap it in a script.

**Always register scripts in `manifest.yaml`** — the web UI automatically shows a button with a parameter form for every script listed there. Without this, users have to drop to the CLI even though the UI fully supports it. Format:
```yaml
scripts:
  - name: create-superuser
    path: scripts/create-superuser.sh
    description: "Short description shown in the UI."
    params:
      - name: EMAIL
        description: Description shown in the param form
        required: true
      - name: PASSWORD
        description: Leave blank to generate a random one
```

**Script conventions** (follow `synapse/versions/v1/scripts/create-user.sh` as the reference):
- Require `KUBECONFIG`, `WILD_INSTANCE`, and `WILD_API_DATA_DIR` env vars; exit with a clear error if missing
- Read `namespace` from `config.yaml` via `yq` so the script doesn't hardcode values
- Auto-generate passwords with `openssl rand` if `PASSWORD` is not supplied
- Find the running pod by label (don't hardcode a pod name); fail clearly if none found
- Print credentials at the end with a "save this — it won't be shown again" warning
- Reference the script in the app README's "First-Time Setup" section in place of a raw `kubectl exec` command

**Non-interactive Django `createsuperuser`**:
```bash
kubectl exec -n <ns> <pod> -c <container> -- \
    env DJANGO_SUPERUSER_PASSWORD="${PASSWORD}" \
    python manage.py createsuperuser --email "${EMAIL}" --noinput
```
The `--noinput` flag reads the password from `DJANGO_SUPERUSER_PASSWORD`.

**Affected apps so far**: Eventyay (`scripts/create-superuser.sh`), Synapse (`scripts/create-user.sh`), Headscale (`scripts/register-node.sh`).

## App Status Tracking

| App | Status | Notes |
|-----|--------|-------|
| Firefly III | ✅ working | |
| Wiki.js | ✅ working | |
| Outline | ✅ working | Redis URL needs password via K8s env var expansion |
| WriteFreely | ✅ working | Needs runAsUser: 0 (community image quirk) |
| NocoDB | ✅ working | Node.js needs 2Gi memory |
| Mattermost | ✅ working | |
| Etherpad | ✅ working | |
| CryptPad | ✅ working | Needs emptyDir + initContainer for config seeding |
| MediaWiki | ✅ working | Post-deploy web installer required for LocalSettings.php |
| Nextcloud | ✅ working | |
| PeerTube | ✅ working | |
| BookStack | ✅ working | linuxserver.io needs root + no capabilities.drop |
| Typebot | ✅ working | Builder: /signin health path; Viewer: tcpSocket probe |
| Pixelfed | ✅ working | Use ghcr.io/mattlqx/docker-pixelfed:v0.12.7-nginx (community), port 80 not 8080 |
| Wekan | pending | needs MongoDB |
| Mautic | ✅ working | |
| Taiga | ✅ working | |
| Pol.is | ❌ blocked | Private AWS ECR images, no public alternative |
| Zulip | ✅ working | Bundles own postgres/rabbitmq/memcached; first-run migrations take ~10min |
| Rocket.Chat | pending | needs MongoDB |
| LimeSurvey | ✅ working | |
| Gancio | ✅ working | Re-deploy: must drop DB schema first if prior data exists (see note 18) |
| Mobilizon | ✅ working | |
| Formbricks | ✅ working | |
| Jitsi | ✅ working | |
| Mastodon | already done | |
| Chamilo | ✅ working | |
| Moodle | ✅ working | |
| Open edX | pending | complex |
| Pretix | ✅ working | |
| Indico | ✅ working | |
| Eventyay | ✅ working | No versioned Docker tags upstream — main tag is intentional |
| Leihs | pending | |
| LibreBooking | ✅ working | |
| uMap | ✅ working | Host header needed in health probes (Django ALLOWED_HOSTS) |
| MapComplete | pending | |
| Terrastories | pending | |
| Karrot | ✅ working | nginx frontend: override template via ConfigMap to remove Docker DNS resolver (see note 24) |
| LiquidFeedback | pending | |
| Your Priorities | ❌ blocked | No public Docker images (ghcr.io requires auth) |
| Helios Voting | pending | |
| Belenios | pending | |
| Decidim | already done | |
| Alaveteli | pending | |
| FixMyStreet | pending | |
| Ghost | ✅ working | Bitnami image gone; use official ghost:alpine; different env vars + mount path (see note 12) |
| Ushahidi | ✅ working | Client: runAsUser:0 + CHOWN/SETUID/SETGID; API: liveness initialDelay 120s (Laravel slow start) |
| CiviCRM | pending | |
| Open Collective | pending | |
| Open Food Network | pending | |
| Resonate | pending | |
| BookWyrm | ✅ working | |
| Lemmy | already done | |
| Akaunting | ✅ working | |
| GnuCash | skip | desktop app, not containerizable |
| OhMyForm | ✅ working | |
| Odoo | ✅ working | Needs `-d DATABASE -i base` args; uses startupProbe (20min window) for first-run DB init |
| Headscale | ✅ working | SQLite, ConfigMap-based config, no secrets; v0.29 CLI uses numeric user IDs (see note 20) |
| Baserow | ✅ working | |
| Keila | already done | |
| Listmonk | already done | |
| Discourse | already done | |
| Gitea | already done | |
| Immich | already done | |
| Loomio | already done | |
| Synapse | already done | App renamed from matrix to synapse |
| Open WebUI | already done | |
| OpenProject | already done | |
| vLLM | already done | |