Reduces apps default resource usage.

2026-06-19 21:38:56 +00:00
parent 63dcd1b82b
commit c223e2b5e0
55 changed files with 661 additions and 174 deletions
--- a/ADDING-APPS.md
+++ b/ADDING-APPS.md
@@ -736,6 +736,7 @@ Apps requiring PostgreSQL or MySQL should include a database initialization job
 - Use `restartPolicy: OnFailure`
 - Include in `kustomization.yaml` resources
 - Use appropriate security context (e.g., `runAsUser: 999` for PostgreSQL)
+- **Write idempotent SQL**: For MySQL, use `CREATE USER IF NOT EXISTS ... ; ALTER USER ...` so re-running the job after a redeploy doesn't leave stale passwords. See "MySQL db-init User Password Idempotency" in Common Gotchas.

 **Example apps:** `immich`, `gitea`, `openproject`, `discourse`

@@ -935,6 +936,247 @@ labels:
  component: server  # Simple component label
 ```

+## Common Gotchas
+
+Lessons learned from deploying apps to Wild Cloud. These are the non-obvious issues that cause failures.
+
+### Always Verify Docker Image Availability
+
+Before packaging an app, confirm the image is **publicly accessible without authentication**:
+
+```bash
+docker pull registry.example.com/org/image:tag
+# or check via curl for ghcr.io:
+curl -s "https://ghcr.io/token?scope=repository:org/image:pull&service=ghcr.io" | python3 -c "import json,sys; print(json.load(sys.stdin).get('token','no token')[:20])"
+```
+
+Common pitfalls:
+- **Bitnami images** moved away from Docker Hub — their `bitnami/appname` images may no longer be available; check `oci.bitnami.com` or use the official upstream image instead
+- **ghcr.io images** frequently require authentication even when the repo is "public" on GitHub — test anonymous pull before committing to them
+- **compdemocracy/polis** and some other community apps use private AWS ECR or require GitHub tokens — these cannot be packaged for general use without finding a public mirror
+
+If no public image exists, note it clearly in the wild-directory `app.yaml` description rather than packaging a broken app.
+
+### Re-run `wild app add` After Template Changes
+
+`wild app deploy <app>` applies whatever is already compiled in the instance's `apps/` directory. It does **not** recompile templates from wild-directory. After modifying any template file in wild-directory, always re-run:
+
+```bash
+wild app add <app>   # recompiles templates into the instance dir
+wild app deploy <app>  # applies the recompiled files
+```
+
+### PostgreSQL Connection Strings: Always Include `?sslmode=disable`
+
+Wild Cloud's internal PostgreSQL does not have SSL enabled. Any app that constructs a PostgreSQL connection string must include `?sslmode=disable`, or the connection will fail with "The server does not support SSL connections."
+
+```yaml
+defaultSecrets:
+  - key: dbUrl
+    default: "postgresql://{{ .app.db.user }}:{{ .secrets.dbPassword }}@{{ .app.db.host }}:{{ .app.db.port }}/{{ .app.db.name }}?sslmode=disable"
+```
+
+Also add `PGSSLMODE=disable` as an env var for apps that use libpq directly.
+
+### Redis URLs Must Include the Password
+
+Wild Cloud's Redis requires authentication. Apps that accept a Redis URL must include the password:
+
+```yaml
+# In manifest.yaml requiredSecrets:
+requiredSecrets:
+  - redis.password
+
+# In deployment.yaml env — use K8s env var expansion for the URL:
+- name: REDIS_PASSWORD
+  valueFrom:
+    secretKeyRef:
+      name: myapp-secrets
+      key: redis.password
+- name: REDIS_URL
+  value: "redis://:$(REDIS_PASSWORD)@{{ .redis.host }}:6379"
+```
+
+The `$(VAR_NAME)` syntax is evaluated by Kubernetes at pod start from other env vars in the same container spec.
+
+### Runtime-Specific Resource Defaults
+
+Default resource requests vary by runtime. The standard 256Mi/50m is too low for heavier runtimes:
+
+| Runtime | Suggested request | Notes |
+|---------|------------------|-------|
+| Go/Rust | 64–128Mi | Very efficient |
+| Python/Ruby/PHP | 128–256Mi | Typical range |
+| Node.js | 256–512Mi | Add `NODE_OPTIONS=--max-old-space-size=<N>` to cap heap |
+| JVM (Java/Kotlin/Scala) | 512Mi–1Gi | Set `-Xmx` to keep within limit |
+| Celery/Sidekiq workers | 256–512Mi | Full worker process per instance |
+
+For Node.js apps, cap memory use explicitly:
+```yaml
+- name: NODE_OPTIONS
+  value: "--max-old-space-size=768"  # ~75% of 1Gi limit
+```
+
+### Health Check Probes for Django/FastAPI (ALLOWED_HOSTS)
+
+Django and similar frameworks reject health probe requests because Kubernetes sends them directly to the pod IP, which is not in `ALLOWED_HOSTS`. Fix by adding a `Host` header to the probe:
+
+```yaml
+livenessProbe:
+  httpGet:
+    path: /
+    port: 8000
+    httpHeaders:
+      - name: Host
+        value: "{{ .domain }}"
+```
+
+### Images That Need Root + Special Capabilities
+
+Some images run an init system (nginx, s6-overlay, custom entrypoint scripts) that requires elevated privileges to set up directories and drop to a lower-privileged user. These fail when `runAsNonRoot: true` or `capabilities.drop: [ALL]` is set.
+
+**Pattern for nginx-based images** (e.g., community app frontends):
+```yaml
+spec:
+  securityContext:
+    runAsNonRoot: false
+    runAsUser: 0
+    seccompProfile:
+      type: RuntimeDefault
+  containers:
+  - name: app
+    securityContext:
+      allowPrivilegeEscalation: false
+      capabilities:
+        drop: [ALL]
+        add: [CHOWN, SETUID, SETGID]  # needed for nginx to drop to worker user
+      readOnlyRootFilesystem: false
+```
+
+**Pattern for linuxserver.io / s6-overlay images** (e.g., BookStack, most linuxserver images):
+```yaml
+spec:
+  securityContext:
+    runAsNonRoot: false
+    runAsUser: 0
+    seccompProfile:
+      type: RuntimeDefault
+  containers:
+  - name: app
+    securityContext:
+      readOnlyRootFilesystem: false  # Only this — do NOT drop capabilities
+```
+
+Signs you need this pattern:
+- `chown(...) failed (1: Operation not permitted)` — add `CHOWN` capability
+- `Permission denied` on a directory during startup — try running as root
+- Container exits immediately with code 1 despite image being "official"
+
+### Hex-Format Secrets
+
+Some apps require secrets in specific formats (e.g., exactly 64 hex chars for Outline). The default Wild Cloud secret generator produces random alphanumeric strings. Use `crypto.SHA256` in the manifest `default` field to derive a hex string from another secret:
+
+```yaml
+defaultSecrets:
+  - key: utilsSecret           # generated first (alphanumeric)
+  - key: secretKey
+    default: '{{ crypto.SHA256 .secrets.utilsSecret }}'  # hex derivative
+```
+
+The order of `defaultSecrets` matters — a referenced secret must appear before the entry that references it.
+
+### Odoo (and Other Apps) Requiring Explicit Database Initialization
+
+Some apps don't auto-initialize their database — they need to be told which database to use AND to install their schema. Odoo is the most common case: without explicit `-d DATABASE -i base` args, it starts in multi-database manager mode and all health checks fail.
+
+```yaml
+# In the container spec:
+args: ["-d", "{{ .db.name }}", "-i", "base"]
+```
+
+The `-i base` flag is **idempotent**: it installs on first run and updates on subsequent runs. The downside is slightly slower startup on subsequent restarts. For very large apps, use an initContainer to run initialization separately from the main container.
+
+Combine with a `startupProbe` to give the initialization time to complete:
+
+```yaml
+startupProbe:
+  httpGet:
+    path: /web/health
+    port: 8069
+  failureThreshold: 40    # 40 × 30s = 20 minutes
+  periodSeconds: 30
+  timeoutSeconds: 10
+livenessProbe:
+  httpGet:
+    path: /web/health
+    port: 8069
+  initialDelaySeconds: 0  # startupProbe takes over startup detection
+  periodSeconds: 30
+  failureThreshold: 6
+```
+
+The `startupProbe` runs until it succeeds (or exhausts `failureThreshold`), at which point the liveness and readiness probes kick in with `initialDelaySeconds: 0`. This is cleaner than inflating `initialDelaySeconds` on the main probes.
+
+### Gancio Re-deploy: "Non empty db" Crash
+
+Gancio stores setup state in `config.json` on its data PVC. If the PVC is deleted or the app is re-added without wiping the database, gancio finds tables in PostgreSQL but no `config.json` and refuses to start.
+
+**Fix**: Drop the schema before restarting:
+```bash
+kubectl exec -n postgres <postgres-pod> -- psql -U postgres gancio \
+  -c "DROP SCHEMA public CASCADE; CREATE SCHEMA public; GRANT ALL ON SCHEMA public TO gancio; GRANT ALL ON SCHEMA public TO public;"
+```
+
+Scale the deployment to 0 before dropping the schema to avoid active connection errors.
+
+### Heavy-Framework Startup Times and Probe Delays
+
+Frameworks with heavy startup initialization (Laravel, Rails, Spring Boot, Symfony) run tasks like package discovery, autoload generation, database migrations, and asset compilation before accepting HTTP requests. This takes 2–3 minutes — much longer than the default 60-second initial delay.
+
+The failure mode is a restart loop: pod runs for ~2 minutes, liveness probe fires and fails, container is killed, new pod starts. The app never gets far enough to become Ready.
+
+For these runtimes, use extended probe delays:
+
+```yaml
+livenessProbe:
+  tcpSocket:
+    port: 8080
+  initialDelaySeconds: 120    # Give the framework time to fully initialize
+  periodSeconds: 30
+  failureThreshold: 6         # Tolerate transient failures during startup
+readinessProbe:
+  tcpSocket:
+    port: 8080
+  initialDelaySeconds: 60
+  periodSeconds: 15
+  failureThreshold: 3
+```
+
+**Runtimes that typically need this:**
+| Runtime | Suggested liveness initialDelaySeconds |
+|---------|----------------------------------------|
+| PHP/Laravel/Symfony | 120s |
+| Ruby on Rails | 90–120s |
+| Spring Boot (JVM) | 90–120s |
+| Django (with migrations) | 60–90s |
+| Node.js | 60s (usually sufficient) |
+
+### MySQL db-init User Password Idempotency
+
+The common pattern `CREATE USER IF NOT EXISTS 'user'@'%' IDENTIFIED BY '${PASSWORD}'` only sets the password when the user is first created. If the MySQL user already exists from a previous deployment (e.g., after deleting and re-adding an app without dropping the database), the password is silently unchanged and the new deployment fails with "Access denied".
+
+Use the `CREATE ... IF NOT EXISTS` + `ALTER USER` pattern to make db-init jobs idempotent regardless of prior state:
+
+```sql
+CREATE DATABASE IF NOT EXISTS ${DB_DATABASE_NAME};
+CREATE USER IF NOT EXISTS '${DB_USERNAME}'@'%' IDENTIFIED BY '${DB_PASSWORD}';
+ALTER USER '${DB_USERNAME}'@'%' IDENTIFIED BY '${DB_PASSWORD}';
+GRANT ALL PRIVILEGES ON ${DB_DATABASE_NAME}.* TO '${DB_USERNAME}'@'%';
+FLUSH PRIVILEGES;
+```
+
+The `ALTER USER` line is safe even when the user was just created — it's a no-op in that case. This applies to MySQL/MariaDB only; PostgreSQL's `CREATE USER IF NOT EXISTS` does not exist but `DO $$ BEGIN IF NOT EXISTS ... END $$` patterns serve the same purpose.
+
 ## Validation Checklist

 Before submitting a new or modified app, verify:
@@ -967,6 +1209,11 @@ Before submitting a new or modified app, verify:
  - [ ] Ingresses include external-dns annotations
  - [ ] Database apps include init jobs (if applicable)

+- [ ] **Images**
+  - [ ] Docker images are publicly accessible (test anonymous pull)
+  - [ ] Image tags are specific versions, not `latest` or branch names
+  - [ ] Architecture constraints (amd64-only) noted in manifest if applicable
+
 - [ ] **Testing**
  - [ ] Templates compile successfully with sample config
  - [ ] App deploys without errors in test cluster