wild-directory/ADDING-APPS-NOTES.md

Adding Apps - Notes & Documentation Improvements

Documentation Improvement Candidates

Issues and gaps discovered while adding apps. These should be folded back into ADDING-APPS.md.

1. Node.js app memory requirements

Node.js apps (NocoDB, Outline, etc.) need more than 512Mi memory. NocoDB crashed with OOM at 512Mi.

• Recommended: 1Gi limit, 512Mi request for Node.js apps
• Set NODE_OPTIONS=--max-old-space-size=768 to keep heap within the limit
• Doc suggestion: Add a section in ADDING-APPS.md about runtime-specific resource defaults:
  • Node.js apps: 1Gi limit, 512Mi request; add NODE_OPTIONS env var
  • JVM apps (Java/Kotlin): 1Gi+ limit
  • Python/Ruby apps: 512Mi typically sufficient
  • Go/Rust apps: 256-512Mi typically sufficient

2. db-init-job is repeated boilerplate

The db-init-job.yaml for PostgreSQL apps is nearly identical across all apps. Only the app name, namespace, and secret name change.

• Doc suggestion: Provide a copy-paste template for the standard PostgreSQL db-init-job with placeholders marked clearly.

4. Apps requiring hex-encoded secrets

Some apps (Outline) require secrets in specific formats (e.g., exactly 64 hex chars). The default Wild Cloud GenerateSecret produces alphanumeric (not hex) strings.

• Solution: Use crypto.SHA256 gomplate function in the manifest default field to generate a valid 64-char hex string from another secret:

  defaultSecrets:
    - key: utilsSecret
    - key: secretKey
      default: '{{ crypto.SHA256 .secrets.utilsSecret }}'
  

• Doc suggestion: Add this pattern to ADDING-APPS.md with examples of apps that need it. Note that the order of defaultSecrets matters — referenced secrets must come first.

6. Always use ?sslmode=disable in PostgreSQL connection strings

The Wild Cloud internal PostgreSQL does not have SSL enabled. Apps that construct PostgreSQL connection strings must include ?sslmode=disable, and deployments should set PGSSLMODE=disable. Without it, apps crash with "The server does not support SSL connections."

• Already done correctly in: listmonk, gitea, discourse
• Missing from: outline (fixed), any new app
• Doc suggestion: Add to the ADDING-APPS.md dbUrl template examples: always append ?sslmode=disable to postgres connection strings.

5. Re-running wild app add is required after template changes

When you modify files in wild-directory (e.g., fix a template), you must re-run wild app add <app> before deploying. The compiled templates in the instance apps/ directory are NOT automatically updated when wild-directory changes. Only then will wild app deploy use the updated templates.

• Doc suggestion: Add to ADDING-APPS.md: "After modifying templates in wild-directory, always re-run wild app add <app> to recompile before deploying."

7. WriteFreely requires root user (jrasanen/writefreely image)

The jrasanen/writefreely image startup script creates config.ini in /writefreely/ which is a root-owned directory. The image's default user is non-root, causing "Permission denied" errors at startup.

• Fix: Set runAsUser: 0 and runAsNonRoot: false in the pod securityContext
• Still safe with allowPrivilegeEscalation: false and capabilities.drop: ALL
• Doc suggestion: Note that some community images require root. When an image fails with permission denied on startup, check if it needs root and add the security context override with capabilities dropped.

8. Redis URL must include password for authenticated Redis

Wild Cloud's Redis requires authentication. Apps that take a Redis URL must include the password in the URL: redis://:password@host:6379.

• Pattern: Add redis.password to requiredSecrets, then use K8s env var expansion:

  - name: REDIS_PASSWORD
    valueFrom:
      secretKeyRef:
        name: myapp-secrets
        key: redis.password
  - name: REDIS_URL
    value: redis://:$(REDIS_PASSWORD)@{{ .redis.host }}:6379
  

• The $(VAR_NAME) syntax is supported by Kubernetes in container env values (evaluated at pod start)
• Doc suggestion: Add this pattern to ADDING-APPS.md under "Redis Authentication" section.

3. Consistent health check paths across apps

Different apps use different health check paths (/health, /healthz, /_health, /api/v1/health). This should be documented per app type or noted that you must verify the actual path from the app's Docker docs.

• For apps where no valid HTTP health path exists (e.g., Typebot viewer which only serves chatbot URLs), use tcpSocket probe instead of httpGet.

9. Linuxserver.io and s6-overlay images require root + capabilities

Images using s6-overlay init system (most linuxserver.io images) must run as root and need full capabilities to switch to their internal "abc" user.

• Set runAsUser: 0, runAsNonRoot: false in pod securityContext
• Do NOT set allowPrivilegeEscalation: false or capabilities.drop: ALL in container securityContext
• Only keep readOnlyRootFilesystem: false
• Same applies to Nextcloud (uses rsync/chown during setup)
• Affects: BookStack (linuxserver.io), any linuxserver.io image

10. CryptPad needs emptyDir + initContainer for config directory

The CryptPad image's /cryptpad/config/ directory is in the image overlay filesystem and not writable by the container process even when running as root.

• Fix: Mount an emptyDir at /cryptpad/config, and use an initContainer to pre-seed config.example.js from the image into the emptyDir.
• The initContainer mounts the emptyDir at a different path (e.g., /config-dest) and copies the file.
• CPAD_CONF env var must be set to /cryptpad/config/config.js (startup script copies config.example.js to config.js if config.js doesn't exist).

11. Pixelfed image requires authentication (ghcr.io access restriction)

ghcr.io/pixelfed/pixelfed returns 403 Forbidden for anonymous pulls.

• Resolved: Use ghcr.io/mattlqx/docker-pixelfed:v0.12.7-nginx (community image, publicly accessible)
• Note: the mattlqx image uses port 80 (nginx), not 8080. Update containerPort and service targetPort accordingly.

12. Bitnami images no longer on Docker Hub

Bitnami moved their images from Docker Hub (bitnami/appname) to their own OCI registry. Apps that were packaged using Bitnami images (e.g., Ghost) need to be updated to use the official upstream image or Bitnami's new registry.

• Ghost fix: Switched from bitnami/ghost:5.x to official ghost:5.130.6-alpine
• The official ghost image uses different env vars (database__connection__host instead of GHOST_DATABASE_HOST) and a different mount path (/var/lib/ghost/content instead of /bitnami/ghost)
• Check image availability before packaging: docker manifest inspect docker.io/bitnami/appname:tag

13. Required secrets belong in app-secrets, not dep-secrets

Apps that use requiredSecrets (e.g., mysql.rootPassword) receive those secrets copied into their own <app>-secrets K8s Secret, under the key <dep>.<key> (e.g., mysql.rootPassword). db-init jobs and deployments must reference <app>-secrets, NOT mysql-secrets.

# WRONG - mysql-secrets doesn't exist in the app namespace
secretKeyRef:
  name: mysql-secrets
  key: rootPassword

# CORRECT - the required secret is copied into ghost-secrets
secretKeyRef:
  name: ghost-secrets
  key: mysql.rootPassword

15. mattlqx/docker-pixelfed requires APP_PORT env var

The ghcr.io/mattlqx/docker-pixelfed community image generates nginx config from env vars using sed. The listen directive uses APP_PORT, which must be set explicitly or nginx crashes with "invalid number of arguments in 'listen' directive".

• Add APP_PORT: "80" to the deployment env vars
• The image listens on port 80 (nginx), not 8080 as the original pixelfed image did
• Both the web and worker deployments must mount the shared storage PVC — use ReadWriteMany access mode, not ReadWriteOnce, since both pods need it simultaneously

14. Your Priorities and Polis have no public Docker images

• ghcr.io/citizensfoundation/your-priorities-app — 403 Forbidden, images require auth
• compdemocracy/polis-server — private, no public Docker Hub or ghcr.io images
• Both apps require building custom images from source before they can be packaged
• Attempting to add these to wild-directory results in ImagePullBackOff

16. PHP/Laravel and other heavy-framework apps need extended probe delays

Laravel's startup process runs package discovery, autoload optimization, and encryption key generation before the HTTP server is ready. This commonly takes 2–3 minutes on cluster restart or first boot.

• The default initialDelaySeconds: 60 is too short — the liveness probe fires before the app is ready, kills the container, and a restart loop begins
• Set initialDelaySeconds: 120 and failureThreshold: 6 for liveness probes on Laravel/Symfony apps
• Set initialDelaySeconds: 60 for readiness probes
• The symptom is: pod shows Running for ~2 minutes then gets Killing due to liveness probe failure, followed by a new pod starting the same cycle
• The same applies to other frameworks with heavy startup initialization: Rails (bundle exec, asset precompilation), Spring Boot (JVM + Spring context loading), Django with migrations
• Affects: Ushahidi API (Laravel) — fixed with 120s initial delay + failureThreshold: 6

17. MySQL db-init CREATE USER IF NOT EXISTS doesn't update passwords

The common db-init pattern CREATE USER IF NOT EXISTS 'user'@'%' IDENTIFIED BY '${PASSWORD}' only sets the password at creation time. If the MySQL user already exists from a previous deployment (e.g., after wild app delete + wild app add without dropping the database), the password is silently left unchanged and the new deployment fails with "Access denied for user".

• Symptom: App starts, connects to MySQL, gets "Access denied" even though the db-init job completed successfully
• Quick fix: Exec into the MySQL pod and run: ALTER USER 'user'@'%' IDENTIFIED BY 'current_password'; FLUSH PRIVILEGES;
• Better pattern — make db-init jobs truly idempotent for both user creation and password:

  CREATE USER IF NOT EXISTS '${DB_USERNAME}'@'%' IDENTIFIED BY '${DB_PASSWORD}';
  ALTER USER '${DB_USERNAME}'@'%' IDENTIFIED BY '${DB_PASSWORD}';
  GRANT ALL PRIVILEGES ON ${DB_DATABASE_NAME}.* TO '${DB_USERNAME}'@'%';
  FLUSH PRIVILEGES;
  

• This CREATE ... IF NOT EXISTS + ALTER USER pattern is safe to run repeatedly regardless of whether the user existed before
• Affects: Ghost (MySQL backend) — hit this when redeploying after a previous partial test run

18. Gancio "Non empty db" crash on re-deploy

Gancio stores its setup state in config.json on the data PVC. If the PVC is lost or the app is deleted and redeployed with an existing database, gancio finds a non-empty DB but no config.json and refuses to start with "Non empty db! Please move your current db elsewhere than retry."

• Fix: Drop the database schema and let gancio recreate it:

  kubectl exec -n postgres <postgres-pod> -- psql -U postgres gancio \
    -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public; GRANT ALL ON SCHEMA public TO gancio; GRANT ALL ON SCHEMA public TO public;"
  kubectl scale deployment -n gancio gancio --replicas=0
  # (wait for pod to terminate)
  kubectl scale deployment -n gancio gancio --replicas=1
  

• Scale down first to release the DB connection before dropping the schema
• This only affects re-deployments; fresh installs work fine

19. Odoo requires explicit database name and -i base on first run

The official Odoo Docker image does not auto-initialize the database. Without specifying the database name and installing the base module, Odoo starts in multi-database manager mode and health checks fail with "Database not initialized".

• Add args: ["-d", "DATABASE_NAME", "-i", "base"] to the container spec
• The -i base flag installs Odoo's base module and creates all database tables on first run
• Subsequent runs with -i base are safe (idempotent) — it updates rather than reinstalls
• Use a startupProbe with failureThreshold: 40, periodSeconds: 30 (20 minutes) since first-run initialization takes 10–15 minutes
• Once startup probe passes, the regular liveness/readiness probes with initialDelaySeconds: 0 take over

20. Headscale v0.29+ CLI: inconsistent user flag types

In headscale v0.29, different commands accept different user identifiers:

• preauthkeys create --user <id> — requires numeric ID
• auth register --user <username> — accepts username string
• users destroy --identifier <id> — requires numeric ID, not a positional arg

# Always list users first to get the ID
headscale users list

headscale preauthkeys create --user 1           # numeric ID
headscale auth register --auth-id <id> --user payne  # username string OK
headscale users destroy --identifier 1          # numeric ID

Node registration flow: when a Tailscale client connects via browser (not pre-auth key), headscale shows a command like headscale auth register --auth-id hskey-authreq-XXXX --user USERNAME. Run it via kubectl exec -n headscale deploy/headscale -- headscale auth register --auth-id <id> --user <username>.

21. "No available server" from Traefik — kustomize label selector mismatch

If a deployment was originally created without kustomize labels in its selector (or with stale templates), re-deploying via wild app deploy cannot fix it because spec.selector on a Deployment is immutable once set. The Service selector will update (it's mutable) but pods won't match it, leaving the Service with no endpoints.

• Symptom: Traefik returns "No available server" even though pods are Running
• Diagnosis: kubectl get endpoints <app> -n <namespace> shows <none>; pod labels don't include app: <name> / managedBy: kustomize / partOf: wild-cloud
• Fix: Delete the affected deployments and re-apply kustomize — kubectl delete deployment <name> -n <ns> then kubectl apply -k <instance>/apps/<app>/
• Root cause: The Deployment selector is set correctly on first deploy, but if the resource already existed with only component: web (e.g., from a pre-kustomize deployment), the selector is stuck

22. Apps using same-origin iframes need SAMEORIGIN frame policy

The global Traefik security-headers middleware applies X-Frame-Options: SAMEORIGIN to all responses (changed from DENY). Apps that use iframes internally (like Etherpad's pad editor, which loads ../static/empty.html in an iframe) were broken by the stricter DENY policy.

• Root cause: DENY prevents even same-origin iframes. SAMEORIGIN still blocks cross-origin embedding while allowing apps to embed their own sub-pages.
• Symptoms: JavaScript error Blocked a frame with origin "https://..." from accessing a cross-origin frame — occurs when an app creates a same-origin iframe and then the X-Frame-Options: DENY header causes the browser to block access to it.
• Important: Route-level Traefik middlewares run BEFORE entrypoint middlewares on the response path. A per-app middleware cannot override a global entrypoint middleware — the entrypoint middleware always runs last.
• Fix applied globally: Changed frameDeny: true to X-Frame-Options: SAMEORIGIN in customResponseHeaders in the crowdsec security-headers middleware.
• Affects: Etherpad (pad editor iframe), any app that loads sub-pages in iframes on the same domain.

24. nginx proxy containers: Docker DNS resolver (127.0.0.11) doesn't work in Kubernetes

Apps that use nginx as a frontend proxy (e.g., karrot-frontend) often ship with an nginx config template that uses Docker's embedded DNS resolver:

resolver 127.0.0.11 valid=3s;
set $backend app-backend:8000;
proxy_pass http://$backend$request_uri;

This fails in Kubernetes with send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11. There's a second trap: any nginx variable anywhere in the proxy_pass URL (even $request_uri in the path) forces runtime DNS resolution and requires a resolver directive. Removing the resolver line but keeping proxy_pass http://backend:8000$request_uri; still fails with no resolver defined to resolve backend.

Fix: override the nginx config template via a ConfigMap mounted with subPath, and remove the variable from proxy_pass entirely:

# Instead of:
resolver 127.0.0.11 valid=3s;
set $backend ${BACKEND};
proxy_pass http://$backend$request_uri;

# Use:
proxy_pass http://${BACKEND};

With no nginx variable in proxy_pass, nginx resolves the hostname at startup using the pod's /etc/resolv.conf (which points to CoreDNS). The full request URI is still forwarded automatically in a regex location ~ block. Mount the ConfigMap with subPath to override just the template file without replacing the whole directory.

25. ConfigMap changes don't restart pods — and subPath mounts never auto-update

When you update a ConfigMap and run wild app deploy (kubectl apply), the ConfigMap object is updated but pods are not restarted. You must explicitly restart them:

kubectl rollout restart deployment/<name> -n <namespace>

There's a further subtlety: ConfigMaps mounted with subPath are never live-updated in the pod filesystem, even if you wait. The pod must be restarted to pick up changes regardless of the subPath mount update behavior.

This means: after any ConfigMap change, always follow up with a rollout restart for affected deployments.

26. The includeSelectors: true mismatch affects every deployment in the app — check them all

When diagnosing "no available server" from Traefik (note #21), it's easy to focus only on the deployment that serves the HTTP route and miss other deployments in the same namespace. The includeSelectors: true label propagation affects every resource in the kustomization — if any deployment pre-existed with only component: X labels, its Service will have <none> endpoints and dependent services (backends, workers, Redis) will silently fail.

Always run this after any selector-related redeploy:

kubectl get endpoints --all-namespaces | grep "<none>"

Check every deployment in the affected namespace, not just the one Traefik is routing to. A Redis pod with <none> endpoints will cause the backend to crash with ConnectionError even though the Redis pod itself shows 1/1 Running.

Checklist when fixing a selector mismatch in an app:

1. kubectl get endpoints -n <app> — identify all services with <none>
2. kubectl get pods -n <app> --show-labels — confirm which pods have old labels
3. Delete ALL affected deployments (not just the web one), then redeploy
4. Verify all endpoints are populated before declaring the app fixed

23. Post-deploy admin scripts for apps that require manual first-run steps

Some apps can't complete setup through the web UI until an initial admin account is created via a CLI command. Package these as shell scripts in scripts/ alongside the kustomize files so users have a repeatable, documented way to run them.

When to add a script: If the app's README or upstream docs require running a management command after deploy (creating a superuser, registering a node, inviting the first user, etc.) — wrap it in a script.

Always register scripts in manifest.yaml — the web UI automatically shows a button with a parameter form for every script listed there. Without this, users have to drop to the CLI even though the UI fully supports it. Format:

scripts:
  - name: create-superuser
    path: scripts/create-superuser.sh
    description: "Short description shown in the UI."
    params:
      - name: EMAIL
        description: Description shown in the param form
        required: true
      - name: PASSWORD
        description: Leave blank to generate a random one

Script conventions (follow synapse/versions/v1/scripts/create-user.sh as the reference):

• Require KUBECONFIG, WILD_INSTANCE, and WILD_API_DATA_DIR env vars; exit with a clear error if missing
• Read namespace from config.yaml via yq so the script doesn't hardcode values
• Auto-generate passwords with openssl rand if PASSWORD is not supplied
• Find the running pod by label (don't hardcode a pod name); fail clearly if none found
• Print credentials at the end with a "save this — it won't be shown again" warning
• Reference the script in the app README's "First-Time Setup" section in place of a raw kubectl exec command

Non-interactive Django createsuperuser:

kubectl exec -n <ns> <pod> -c <container> -- \
    env DJANGO_SUPERUSER_PASSWORD="${PASSWORD}" \
    python manage.py createsuperuser --email "${EMAIL}" --noinput

The --noinput flag reads the password from DJANGO_SUPERUSER_PASSWORD.

Affected apps so far: Eventyay (scripts/create-superuser.sh), Synapse (scripts/create-user.sh), Headscale (scripts/register-node.sh).

App Status Tracking

App	Status	Notes
Firefly III	✅ working
Wiki.js	✅ working
Outline	✅ working	Redis URL needs password via K8s env var expansion
WriteFreely	✅ working	Needs runAsUser: 0 (community image quirk)
NocoDB	✅ working	Node.js needs 2Gi memory
Mattermost	✅ working
Etherpad	✅ working
CryptPad	✅ working	Needs emptyDir + initContainer for config seeding
MediaWiki	✅ working	Post-deploy web installer required for LocalSettings.php
Nextcloud	✅ working
PeerTube	✅ working
BookStack	✅ working	linuxserver.io needs root + no capabilities.drop
Typebot	✅ working	Builder: /signin health path; Viewer: tcpSocket probe
Pixelfed	✅ working	Use ghcr.io/mattlqx/docker-pixelfed:v0.12.7-nginx (community), port 80 not 8080
Wekan	pending	needs MongoDB
Mautic	✅ working
Taiga	✅ working
Pol.is	❌ blocked	Private AWS ECR images, no public alternative
Zulip	✅ working	Bundles own postgres/rabbitmq/memcached; first-run migrations take ~10min
Rocket.Chat	pending	needs MongoDB
LimeSurvey	✅ working
Gancio	✅ working	Re-deploy: must drop DB schema first if prior data exists (see note 18)
Mobilizon	✅ working
Formbricks	✅ working
Jitsi	✅ working
Mastodon	already done
Chamilo	✅ working
Moodle	✅ working
Open edX	pending	complex
Pretix	✅ working
Indico	✅ working
Eventyay	✅ working	No versioned Docker tags upstream — main tag is intentional
Leihs	pending
LibreBooking	✅ working
uMap	✅ working	Host header needed in health probes (Django ALLOWED_HOSTS)
MapComplete	pending
Terrastories	pending
Karrot	✅ working	nginx frontend: override template via ConfigMap to remove Docker DNS resolver (see note 24)
LiquidFeedback	pending
Your Priorities	❌ blocked	No public Docker images (ghcr.io requires auth)
Helios Voting	pending
Belenios	pending
Decidim	already done
Alaveteli	pending
FixMyStreet	pending
Ghost	✅ working	Bitnami image gone; use official ghost:alpine; different env vars + mount path (see note 12)
Ushahidi	✅ working	Client: runAsUser:0 + CHOWN/SETUID/SETGID; API: liveness initialDelay 120s (Laravel slow start)
CiviCRM	pending
Open Collective	pending
Open Food Network	pending
Resonate	pending
BookWyrm	✅ working
Lemmy	already done
Akaunting	✅ working
GnuCash	skip	desktop app, not containerizable
OhMyForm	✅ working
Odoo	✅ working	Needs -d DATABASE -i base args; uses startupProbe (20min window) for first-run DB init
Headscale	✅ working	SQLite, ConfigMap-based config, no secrets; v0.29 CLI uses numeric user IDs (see note 20)
Baserow	✅ working
Keila	already done
Listmonk	already done
Discourse	already done
Gitea	already done
Immich	already done
Loomio	already done
Synapse	already done	App renamed from matrix to synapse
Open WebUI	already done
OpenProject	already done
vLLM	already done