wild-directory/ADDING-APPS-NOTES.md

# Adding Apps - Notes & Documentation Improvements

## Documentation Improvement Candidates

Issues and gaps discovered while adding apps. These should be folded back into ADDING-APPS.md.

### 1. Node.js app memory requirements
Node.js apps (NocoDB, Outline, etc.) need more than 512Mi memory. NocoDB crashed with OOM at 512Mi.
- Recommended: 1Gi limit, 512Mi request for Node.js apps
- Set `NODE_OPTIONS=--max-old-space-size=768` to keep heap within the limit
- **Doc suggestion**: Add a section in ADDING-APPS.md about runtime-specific resource defaults:
  - Node.js apps: 1Gi limit, 512Mi request; add NODE_OPTIONS env var
  - JVM apps (Java/Kotlin): 1Gi+ limit
  - Python/Ruby apps: 512Mi typically sufficient
  - Go/Rust apps: 256-512Mi typically sufficient

### 2. db-init-job is repeated boilerplate
The db-init-job.yaml for PostgreSQL apps is nearly identical across all apps. Only the app name, namespace, and secret name change. 
- **Doc suggestion**: Provide a copy-paste template for the standard PostgreSQL db-init-job with placeholders marked clearly.

### 4. Apps requiring hex-encoded secrets
Some apps (Outline) require secrets in specific formats (e.g., exactly 64 hex chars). The default Wild Cloud `GenerateSecret` produces alphanumeric (not hex) strings.
- **Solution**: Use `crypto.SHA256` gomplate function in the manifest `default` field to generate a valid 64-char hex string from another secret:
  ```yaml
  defaultSecrets:
    - key: utilsSecret
    - key: secretKey
      default: '{{ crypto.SHA256 .secrets.utilsSecret }}'
  ```
- **Doc suggestion**: Add this pattern to ADDING-APPS.md with examples of apps that need it. Note that the order of `defaultSecrets` matters — referenced secrets must come first.

### 6. Always use `?sslmode=disable` in PostgreSQL connection strings
The Wild Cloud internal PostgreSQL does not have SSL enabled. Apps that construct PostgreSQL connection strings must include `?sslmode=disable`, and deployments should set `PGSSLMODE=disable`. Without it, apps crash with "The server does not support SSL connections."
- Already done correctly in: listmonk, gitea, discourse
- Missing from: outline (fixed), any new app
- **Doc suggestion**: Add to the ADDING-APPS.md dbUrl template examples: always append `?sslmode=disable` to postgres connection strings.

### 5. Re-running `wild app add` is required after template changes
When you modify files in wild-directory (e.g., fix a template), you must re-run `wild app add <app>` before deploying. The compiled templates in the instance `apps/` directory are NOT automatically updated when wild-directory changes. Only then will `wild app deploy` use the updated templates.
- **Doc suggestion**: Add to ADDING-APPS.md: "After modifying templates in wild-directory, always re-run `wild app add <app>` to recompile before deploying."

### 7. WriteFreely requires root user (jrasanen/writefreely image)
The `jrasanen/writefreely` image startup script creates `config.ini` in `/writefreely/` which is a root-owned directory. The image's default user is non-root, causing "Permission denied" errors at startup.
- **Fix**: Set `runAsUser: 0` and `runAsNonRoot: false` in the pod securityContext
- Still safe with `allowPrivilegeEscalation: false` and `capabilities.drop: ALL`
- **Doc suggestion**: Note that some community images require root. When an image fails with permission denied on startup, check if it needs root and add the security context override with capabilities dropped.

### 8. Redis URL must include password for authenticated Redis
Wild Cloud's Redis requires authentication. Apps that take a Redis URL must include the password in the URL: `redis://:password@host:6379`. 
- **Pattern**: Add `redis.password` to `requiredSecrets`, then use K8s env var expansion:
  ```yaml
  - name: REDIS_PASSWORD
    valueFrom:
      secretKeyRef:
        name: myapp-secrets
        key: redis.password
  - name: REDIS_URL
    value: redis://:$(REDIS_PASSWORD)@{{ .redis.host }}:6379
  ```
- The `$(VAR_NAME)` syntax is supported by Kubernetes in container env values (evaluated at pod start)
- **Doc suggestion**: Add this pattern to ADDING-APPS.md under "Redis Authentication" section.

### 3. Consistent health check paths across apps
Different apps use different health check paths (`/health`, `/healthz`, `/_health`, `/api/v1/health`). This should be documented per app type or noted that you must verify the actual path from the app's Docker docs.
- For apps where no valid HTTP health path exists (e.g., Typebot viewer which only serves chatbot URLs), use `tcpSocket` probe instead of `httpGet`.

### 9. Linuxserver.io and s6-overlay images require root + capabilities
Images using s6-overlay init system (most linuxserver.io images) must run as root and need full capabilities to switch to their internal "abc" user.
- Set `runAsUser: 0, runAsNonRoot: false` in pod securityContext
- Do NOT set `allowPrivilegeEscalation: false` or `capabilities.drop: ALL` in container securityContext
- Only keep `readOnlyRootFilesystem: false`
- Same applies to Nextcloud (uses rsync/chown during setup)
- **Affects**: BookStack (linuxserver.io), any linuxserver.io image

### 10. CryptPad needs emptyDir + initContainer for config directory
The CryptPad image's `/cryptpad/config/` directory is in the image overlay filesystem and not writable by the container process even when running as root.
- **Fix**: Mount an emptyDir at `/cryptpad/config`, and use an initContainer to pre-seed `config.example.js` from the image into the emptyDir.
- The initContainer mounts the emptyDir at a different path (e.g., `/config-dest`) and copies the file.
- **CPAD_CONF** env var must be set to `/cryptpad/config/config.js` (startup script copies config.example.js to config.js if config.js doesn't exist).

### 11. Pixelfed image requires authentication (ghcr.io access restriction)
~~`ghcr.io/pixelfed/pixelfed` returns 403 Forbidden for anonymous pulls.~~
- **Resolved**: Use `ghcr.io/mattlqx/docker-pixelfed:v0.12.7-nginx` (community image, publicly accessible)
- Note: the mattlqx image uses port 80 (nginx), not 8080. Update `containerPort` and service `targetPort` accordingly.

### 12. Bitnami images no longer on Docker Hub
Bitnami moved their images from Docker Hub (`bitnami/appname`) to their own OCI registry. Apps that were packaged using Bitnami images (e.g., Ghost) need to be updated to use the official upstream image or Bitnami's new registry.
- **Ghost fix**: Switched from `bitnami/ghost:5.x` to official `ghost:5.130.6-alpine`
- The official ghost image uses different env vars (`database__connection__host` instead of `GHOST_DATABASE_HOST`) and a different mount path (`/var/lib/ghost/content` instead of `/bitnami/ghost`)
- Check image availability before packaging: `docker manifest inspect docker.io/bitnami/appname:tag`

### 13. Required secrets belong in app-secrets, not dep-secrets
Apps that use `requiredSecrets` (e.g., `mysql.rootPassword`) receive those secrets copied into their own `<app>-secrets` K8s Secret, under the key `<dep>.<key>` (e.g., `mysql.rootPassword`). db-init jobs and deployments must reference `<app>-secrets`, NOT `mysql-secrets`.

```yaml
# WRONG - mysql-secrets doesn't exist in the app namespace
secretKeyRef:
  name: mysql-secrets
  key: rootPassword

# CORRECT - the required secret is copied into ghost-secrets
secretKeyRef:
  name: ghost-secrets
  key: mysql.rootPassword
```

### 15. mattlqx/docker-pixelfed requires APP_PORT env var
The `ghcr.io/mattlqx/docker-pixelfed` community image generates nginx config from env vars using `sed`. The `listen` directive uses `APP_PORT`, which must be set explicitly or nginx crashes with "invalid number of arguments in 'listen' directive".
- Add `APP_PORT: "80"` to the deployment env vars
- The image listens on port 80 (nginx), not 8080 as the original pixelfed image did
- Both the web and worker deployments must mount the shared storage PVC — use ReadWriteMany access mode, not ReadWriteOnce, since both pods need it simultaneously

### 14. Your Priorities and Polis have no public Docker images
- `ghcr.io/citizensfoundation/your-priorities-app` — 403 Forbidden, images require auth
- `compdemocracy/polis-server` — private, no public Docker Hub or ghcr.io images
- Both apps require building custom images from source before they can be packaged
- Attempting to add these to wild-directory results in ImagePullBackOff

### 16. PHP/Laravel and other heavy-framework apps need extended probe delays
Laravel's startup process runs package discovery, autoload optimization, and encryption key generation before the HTTP server is ready. This commonly takes 2–3 minutes on cluster restart or first boot.
- The default `initialDelaySeconds: 60` is too short — the liveness probe fires before the app is ready, kills the container, and a restart loop begins
- Set `initialDelaySeconds: 120` and `failureThreshold: 6` for liveness probes on Laravel/Symfony apps
- Set `initialDelaySeconds: 60` for readiness probes
- The symptom is: pod shows `Running` for ~2 minutes then gets `Killing` due to liveness probe failure, followed by a new pod starting the same cycle
- The same applies to other frameworks with heavy startup initialization: Rails (`bundle exec`, asset precompilation), Spring Boot (JVM + Spring context loading), Django with migrations
- **Affects**: Ushahidi API (Laravel) — fixed with 120s initial delay + failureThreshold: 6

### 17. MySQL db-init `CREATE USER IF NOT EXISTS` doesn't update passwords
The common db-init pattern `CREATE USER IF NOT EXISTS 'user'@'%' IDENTIFIED BY '${PASSWORD}'` only sets the password at creation time. If the MySQL user already exists from a previous deployment (e.g., after `wild app delete` + `wild app add` without dropping the database), the password is silently left unchanged and the new deployment fails with "Access denied for user".
- **Symptom**: App starts, connects to MySQL, gets "Access denied" even though the db-init job completed successfully
- **Quick fix**: Exec into the MySQL pod and run: `ALTER USER 'user'@'%' IDENTIFIED BY 'current_password'; FLUSH PRIVILEGES;`
- **Better pattern** — make db-init jobs truly idempotent for both user creation and password:
  ```sql
  CREATE USER IF NOT EXISTS '${DB_USERNAME}'@'%' IDENTIFIED BY '${DB_PASSWORD}';
  ALTER USER '${DB_USERNAME}'@'%' IDENTIFIED BY '${DB_PASSWORD}';
  GRANT ALL PRIVILEGES ON ${DB_DATABASE_NAME}.* TO '${DB_USERNAME}'@'%';
  FLUSH PRIVILEGES;
  ```
- This `CREATE ... IF NOT EXISTS` + `ALTER USER` pattern is safe to run repeatedly regardless of whether the user existed before
- **Affects**: Ghost (MySQL backend) — hit this when redeploying after a previous partial test run

### 18. Gancio "Non empty db" crash on re-deploy
Gancio stores its setup state in `config.json` on the data PVC. If the PVC is lost or the app is deleted and redeployed with an existing database, gancio finds a non-empty DB but no `config.json` and refuses to start with "Non empty db! Please move your current db elsewhere than retry."
- **Fix**: Drop the database schema and let gancio recreate it:
  ```bash
  kubectl exec -n postgres <postgres-pod> -- psql -U postgres gancio \
    -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public; GRANT ALL ON SCHEMA public TO gancio; GRANT ALL ON SCHEMA public TO public;"
  kubectl scale deployment -n gancio gancio --replicas=0
  # (wait for pod to terminate)
  kubectl scale deployment -n gancio gancio --replicas=1
  ```
- Scale down first to release the DB connection before dropping the schema
- This only affects re-deployments; fresh installs work fine

### 19. Odoo requires explicit database name and `-i base` on first run
The official Odoo Docker image does not auto-initialize the database. Without specifying the database name and installing the base module, Odoo starts in multi-database manager mode and health checks fail with "Database not initialized".
- Add `args: ["-d", "DATABASE_NAME", "-i", "base"]` to the container spec
- The `-i base` flag installs Odoo's base module and creates all database tables on first run
- Subsequent runs with `-i base` are safe (idempotent) — it updates rather than reinstalls
- Use a `startupProbe` with `failureThreshold: 40, periodSeconds: 30` (20 minutes) since first-run initialization takes 10–15 minutes
- Once startup probe passes, the regular liveness/readiness probes with `initialDelaySeconds: 0` take over

### 20. Headscale v0.29+ CLI: inconsistent user flag types
In headscale v0.29, different commands accept different user identifiers:
- `preauthkeys create --user <id>` — requires numeric ID
- `auth register --user <username>` — accepts username string
- `users destroy --identifier <id>` — requires numeric ID, not a positional arg

```bash
# Always list users first to get the ID
headscale users list

headscale preauthkeys create --user 1           # numeric ID
headscale auth register --auth-id <id> --user payne  # username string OK
headscale users destroy --identifier 1          # numeric ID
```

Node registration flow: when a Tailscale client connects via browser (not pre-auth key), headscale
shows a command like `headscale auth register --auth-id hskey-authreq-XXXX --user USERNAME`.
Run it via `kubectl exec -n headscale deploy/headscale -- headscale auth register --auth-id <id> --user <username>`.

## App Status Tracking

| App | Status | Notes |
|-----|--------|-------|
| Firefly III | ✅ working | |
| Wiki.js | ✅ working | |
| Outline | ✅ working | Redis URL needs password via K8s env var expansion |
| WriteFreely | ✅ working | Needs runAsUser: 0 (community image quirk) |
| NocoDB | ✅ working | Node.js needs 2Gi memory |
| Mattermost | ✅ working | |
| Etherpad | ✅ working | |
| CryptPad | ✅ working | Needs emptyDir + initContainer for config seeding |
| MediaWiki | ✅ working | Post-deploy web installer required for LocalSettings.php |
| Nextcloud | ✅ working | |
| PeerTube | ✅ working | |
| BookStack | ✅ working | linuxserver.io needs root + no capabilities.drop |
| Typebot | ✅ working | Builder: /signin health path; Viewer: tcpSocket probe |
| Pixelfed | ✅ working | Use ghcr.io/mattlqx/docker-pixelfed:v0.12.7-nginx (community), port 80 not 8080 |
| Wekan | pending | needs MongoDB |
| Mautic | ✅ working | |
| Taiga | ✅ working | |
| Pol.is | ❌ blocked | Private AWS ECR images, no public alternative |
| Zulip | ✅ working | Bundles own postgres/rabbitmq/memcached; first-run migrations take ~10min |
| Rocket.Chat | pending | needs MongoDB |
| LimeSurvey | ✅ working | |
| Gancio | ✅ working | Re-deploy: must drop DB schema first if prior data exists (see note 18) |
| Mobilizon | ✅ working | |
| Formbricks | ✅ working | |
| Jitsi | ✅ working | |
| Mastodon | already done | |
| Chamilo | ✅ working | |
| Moodle | ✅ working | |
| Open edX | pending | complex |
| Pretix | ✅ working | |
| Indico | ✅ working | |
| Eventyay | ✅ working | No versioned Docker tags upstream — main tag is intentional |
| Leihs | pending | |
| LibreBooking | ✅ working | |
| uMap | ✅ working | Host header needed in health probes (Django ALLOWED_HOSTS) |
| MapComplete | pending | |
| Terrastories | pending | |
| Karrot | already done | |
| LiquidFeedback | pending | |
| Your Priorities | ❌ blocked | No public Docker images (ghcr.io requires auth) |
| Helios Voting | pending | |
| Belenios | pending | |
| Decidim | already done | |
| Alaveteli | pending | |
| FixMyStreet | pending | |
| Ghost | ✅ working | Bitnami image gone; use official ghost:alpine; different env vars + mount path (see note 12) |
| Ushahidi | ✅ working | Client: runAsUser:0 + CHOWN/SETUID/SETGID; API: liveness initialDelay 120s (Laravel slow start) |
| CiviCRM | pending | |
| Open Collective | pending | |
| Open Food Network | pending | |
| Resonate | pending | |
| BookWyrm | ✅ working | |
| Lemmy | already done | |
| Akaunting | ✅ working | |
| GnuCash | skip | desktop app, not containerizable |
| OhMyForm | ✅ working | |
| Odoo | ✅ working | Needs `-d DATABASE -i base` args; uses startupProbe (20min window) for first-run DB init |
| Headscale | ✅ working | SQLite, ConfigMap-based config, no secrets; v0.29 CLI uses numeric user IDs (see note 20) |
| Baserow | ✅ working | |
| Keila | already done | |
| Listmonk | already done | |
| Discourse | already done | |
| Gitea | already done | |
| Immich | already done | |
| Loomio | already done | |
| Synapse | already done | App renamed from matrix to synapse |
| Open WebUI | already done | |
| OpenProject | already done | |
| vLLM | already done | |