fix(karrot): fix nginx DNS resolver for Kubernetes and add README

The karrot-frontend nginx template hardcodes Docker's DNS resolver
(127.0.0.11) which doesn't exist in Kubernetes. Added a ConfigMap to
override the template, removing the Docker DNS resolver by using a
direct proxy_pass with no nginx variable (which would force runtime
resolution). Also adds a README with usage instructions.

Documents lessons learned in ADDING-APPS-NOTES.md:
- Note 24: nginx Docker DNS resolver doesn't work in Kubernetes; any
  nginx variable in proxy_pass (including $request_uri) forces runtime
  DNS resolution requiring a resolver directive
- Note 25: ConfigMap changes don't restart pods; subPath mounts never
  auto-update; always follow with kubectl rollout restart
- Note 26: includeSelectors mismatch affects every deployment in an
  app — check all endpoints, not just the web-facing one

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

ADDING-APPS-NOTES.md

@@ -179,6 +179,114 @@ Node registration flow: when a Tailscale client connects via browser (not pre-au
 shows a command like `headscale auth register --auth-id hskey-authreq-XXXX --user USERNAME`.
 Run it via `kubectl exec -n headscale deploy/headscale -- headscale auth register --auth-id <id> --user <username>`.
 
+### 21. "No available server" from Traefik — kustomize label selector mismatch
+If a deployment was originally created without kustomize labels in its selector (or with stale templates), re-deploying via `wild app deploy` cannot fix it because `spec.selector` on a Deployment is immutable once set. The Service selector will update (it's mutable) but pods won't match it, leaving the Service with no endpoints.
+- **Symptom**: Traefik returns "No available server" even though pods are Running
+- **Diagnosis**: `kubectl get endpoints <app> -n <namespace>` shows `<none>`; pod labels don't include `app: <name>` / `managedBy: kustomize` / `partOf: wild-cloud`
+- **Fix**: Delete the affected deployments and re-apply kustomize — `kubectl delete deployment <name> -n <ns>` then `kubectl apply -k <instance>/apps/<app>/`
+- **Root cause**: The Deployment selector is set correctly on first deploy, but if the resource already existed with only `component: web` (e.g., from a pre-kustomize deployment), the selector is stuck
+
+### 22. Apps using same-origin iframes need SAMEORIGIN frame policy
+The global Traefik `security-headers` middleware applies `X-Frame-Options: SAMEORIGIN` to all responses (changed from `DENY`). Apps that use iframes internally (like Etherpad's pad editor, which loads `../static/empty.html` in an iframe) were broken by the stricter `DENY` policy.
+- **Root cause**: `DENY` prevents even same-origin iframes. `SAMEORIGIN` still blocks cross-origin embedding while allowing apps to embed their own sub-pages.
+- **Symptoms**: JavaScript error `Blocked a frame with origin "https://..." from accessing a cross-origin frame` — occurs when an app creates a same-origin iframe and then the `X-Frame-Options: DENY` header causes the browser to block access to it.
+- **Important**: Route-level Traefik middlewares run BEFORE entrypoint middlewares on the response path. A per-app middleware cannot override a global entrypoint middleware — the entrypoint middleware always runs last.
+- **Fix applied globally**: Changed `frameDeny: true` to `X-Frame-Options: SAMEORIGIN` in `customResponseHeaders` in the crowdsec `security-headers` middleware.
+- **Affects**: Etherpad (pad editor iframe), any app that loads sub-pages in iframes on the same domain.
+
+### 24. nginx proxy containers: Docker DNS resolver (`127.0.0.11`) doesn't work in Kubernetes
+
+Apps that use nginx as a frontend proxy (e.g., karrot-frontend) often ship with an nginx config template that uses Docker's embedded DNS resolver:
+
+```nginx
+resolver 127.0.0.11 valid=3s;
+set $backend app-backend:8000;
+proxy_pass http://$backend$request_uri;
+```
+
+This fails in Kubernetes with `send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11`. There's a second trap: **any nginx variable anywhere in the `proxy_pass` URL** (even `$request_uri` in the path) forces runtime DNS resolution and requires a `resolver` directive. Removing the resolver line but keeping `proxy_pass http://backend:8000$request_uri;` still fails with `no resolver defined to resolve backend`.
+
+**Fix**: override the nginx config template via a ConfigMap mounted with `subPath`, and remove the variable from `proxy_pass` entirely:
+
+```nginx
+# Instead of:
+resolver 127.0.0.11 valid=3s;
+set $backend ${BACKEND};
+proxy_pass http://$backend$request_uri;
+
+# Use:
+proxy_pass http://${BACKEND};
+```
+
+With no nginx variable in `proxy_pass`, nginx resolves the hostname at startup using the pod's `/etc/resolv.conf` (which points to CoreDNS). The full request URI is still forwarded automatically in a regex `location ~` block. Mount the ConfigMap with `subPath` to override just the template file without replacing the whole directory.
+
+### 25. ConfigMap changes don't restart pods — and `subPath` mounts never auto-update
+
+When you update a ConfigMap and run `wild app deploy` (`kubectl apply`), the ConfigMap object is updated but **pods are not restarted**. You must explicitly restart them:
+
+```bash
+kubectl rollout restart deployment/<name> -n <namespace>
+```
+
+There's a further subtlety: ConfigMaps mounted with `subPath` are **never** live-updated in the pod filesystem, even if you wait. The pod must be restarted to pick up changes regardless of the `subPath` mount update behavior.
+
+This means: after any ConfigMap change, always follow up with a rollout restart for affected deployments.
+
+### 26. The `includeSelectors: true` mismatch affects every deployment in the app — check them all
+
+When diagnosing "no available server" from Traefik (note #21), it's easy to focus only on the deployment that serves the HTTP route and miss other deployments in the same namespace. The `includeSelectors: true` label propagation affects every resource in the kustomization — if any deployment pre-existed with only `component: X` labels, its Service will have `<none>` endpoints and dependent services (backends, workers, Redis) will silently fail.
+
+**Always run this after any selector-related redeploy**:
+```bash
+kubectl get endpoints --all-namespaces | grep "<none>"
+```
+
+Check every deployment in the affected namespace, not just the one Traefik is routing to. A Redis pod with `<none>` endpoints will cause the backend to crash with `ConnectionError` even though the Redis pod itself shows `1/1 Running`.
+
+**Checklist when fixing a selector mismatch in an app**:
+1. `kubectl get endpoints -n <app>` — identify all services with `<none>`
+2. `kubectl get pods -n <app> --show-labels` — confirm which pods have old labels
+3. Delete ALL affected deployments (not just the web one), then redeploy
+4. Verify all endpoints are populated before declaring the app fixed
+
+### 23. Post-deploy admin scripts for apps that require manual first-run steps
+
+Some apps can't complete setup through the web UI until an initial admin account is created via a CLI command. Package these as shell scripts in `scripts/` alongside the kustomize files so users have a repeatable, documented way to run them.
+
+**When to add a script**: If the app's README or upstream docs require running a management command after deploy (creating a superuser, registering a node, inviting the first user, etc.) — wrap it in a script.
+
+**Always register scripts in `manifest.yaml`** — the web UI automatically shows a button with a parameter form for every script listed there. Without this, users have to drop to the CLI even though the UI fully supports it. Format:
+```yaml
+scripts:
+  - name: create-superuser
+    path: scripts/create-superuser.sh
+    description: "Short description shown in the UI."
+    params:
+      - name: EMAIL
+        description: Description shown in the param form
+        required: true
+      - name: PASSWORD
+        description: Leave blank to generate a random one
+```
+
+**Script conventions** (follow `synapse/versions/v1/scripts/create-user.sh` as the reference):
+- Require `KUBECONFIG`, `WILD_INSTANCE`, and `WILD_API_DATA_DIR` env vars; exit with a clear error if missing
+- Read `namespace` from `config.yaml` via `yq` so the script doesn't hardcode values
+- Auto-generate passwords with `openssl rand` if `PASSWORD` is not supplied
+- Find the running pod by label (don't hardcode a pod name); fail clearly if none found
+- Print credentials at the end with a "save this — it won't be shown again" warning
+- Reference the script in the app README's "First-Time Setup" section in place of a raw `kubectl exec` command
+
+**Non-interactive Django `createsuperuser`**:
+```bash
+kubectl exec -n <ns> <pod> -c <container> -- \
+    env DJANGO_SUPERUSER_PASSWORD="${PASSWORD}" \
+    python manage.py createsuperuser --email "${EMAIL}" --noinput
+```
+The `--noinput` flag reads the password from `DJANGO_SUPERUSER_PASSWORD`.
+
+**Affected apps so far**: Eventyay (`scripts/create-superuser.sh`), Synapse (`scripts/create-user.sh`), Headscale (`scripts/register-node.sh`).
+
 ## App Status Tracking
 
 | App | Status | Notes |
@@ -220,7 +328,7 @@ Run it via `kubectl exec -n headscale deploy/headscale -- headscale auth registe
 | uMap | ✅ working | Host header needed in health probes (Django ALLOWED_HOSTS) |
 | MapComplete | pending | |
 | Terrastories | pending | |
-| Karrot | already done | |
+| Karrot | ✅ working | nginx frontend: override template via ConfigMap to remove Docker DNS resolver (see note 24) |
 | LiquidFeedback | pending | |
 | Your Priorities | ❌ blocked | No public Docker images (ghcr.io requires auth) |
 | Helios Voting | pending | |

karrot/versions/1/README.md

@@ -0,0 +1,35 @@
+# Karrot
+
+Karrot is a community platform for sharing resources, coordinating food-saving activities, and organizing volunteers. Groups can manage pickups, track saved food, and communicate through a shared feed.
+
+## Dependencies
+
+- **PostgreSQL** - Database for storing groups, users, and activity data
+- **SMTP** - For account verification and notifications
+
+## Configuration
+
+Key settings in `config.yaml`:
+
+- **domain** - Where Karrot will be accessible (default: `karrot.{your-cloud-domain}`)
+- **storage** - Persistent volume size for uploaded files (default: `2Gi`)
+- **redis.host** - Redis instance for caching and background jobs (default: app-local `karrot-redis`)
+- **smtp** - Email settings inherited from your Wild Cloud SMTP service
+
+## First-Time Setup
+
+1. Add and deploy the app:
+   ```bash
+   wild app add karrot
+   wild app deploy karrot
+   ```
+
+2. Visit `https://karrot.{your-cloud-domain}` and register an account. The first user to register becomes the site admin.
+
+3. Log in and create your first group to start coordinating.
+
+## Notes
+
+- Karrot supports federation — groups can connect with other Karrot instances
+- The `secretKey` and `dbPassword` are auto-generated in `secrets.yaml`
+- Uploaded files (photos, attachments) are stored in the persistent volume and shared between the frontend and backend pods

karrot/versions/1/deployment-frontend.yaml

@@ -14,9 +14,9 @@ spec:
         component: frontend
     spec:
       securityContext:
-        runAsNonRoot: true
+        runAsNonRoot: false
-        runAsUser: 101
+        runAsUser: 0
-        runAsGroup: 101
+        runAsGroup: 0
         seccompProfile:
           type: RuntimeDefault
       containers:
@@ -60,8 +60,6 @@ spec:
             failureThreshold: 3
           securityContext:
             allowPrivilegeEscalation: false
-            capabilities:
-              drop: [ALL]
             readOnlyRootFilesystem: false
           volumeMounts:
             - name: karrot-uploads
@@ -73,6 +71,9 @@ spec:
               mountPath: /etc/nginx/conf.d
             - name: nginx-run
               mountPath: /var/run
+            - name: nginx-template
+              mountPath: /etc/nginx/templates/default.conf.template
+              subPath: default.conf.template
       volumes:
         - name: karrot-uploads
           persistentVolumeClaim:
@@ -83,4 +84,7 @@ spec:
           emptyDir: {}
         - name: nginx-run
           emptyDir: {}
+        - name: nginx-template
+          configMap:
+            name: karrot-nginx-template
       restartPolicy: Always

karrot/versions/1/kustomization.yaml

@@ -9,6 +9,7 @@ labels:
       partOf: wild-cloud
 resources:
   - namespace.yaml
+  - nginx-template-configmap.yaml
   - db-init-job.yaml
   - deployment-backend.yaml
   - deployment-frontend.yaml

karrot/versions/1/nginx-template-configmap.yaml

@@ -0,0 +1,116 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: karrot-nginx-template
+  namespace: {{ .namespace }}
+data:
+  default.conf.template: |
+    map $http_upgrade $connection_upgrade {
+        default upgrade;
+        '' close;
+    }
+
+    client_max_body_size ${FILE_UPLOAD_MAX_SIZE};
+
+    server {
+
+        listen ${LISTEN};
+
+        set_real_ip_from 10.0.0.0/8;
+        set_real_ip_from 172.16.0.0/12;
+        set_real_ip_from 192.168.0.0/16;
+        real_ip_header X-Forwarded-For;
+
+        root /usr/share/nginx/html;
+
+        location /assets {
+          expires max;
+        }
+
+        location /css {
+          expires max;
+        }
+
+        location /js {
+          expires max;
+        }
+
+        location /img {
+          expires max;
+        }
+
+        location /fonts {
+          expires max;
+        }
+
+        location / {
+
+            try_files $uri /index.html;
+
+            add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
+            add_header X-Content-Type-Options nosniff;
+
+            add_header Last-Modified $date_gmt;
+            add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
+
+            add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; connect-src 'self' https://nominatim.openstreetmap.org https://sentry.io https://*.ingest.sentry.io ${CSP_CONNECT_SRC} blob:; style-src 'self' 'unsafe-inline'; font-src 'self' data:; img-src 'self' https: data: blob:;";
+
+            if_modified_since off;
+            expires off;
+            etag off;
+        }
+
+        location /bundlesize.html {
+            add_header Content-Security-Policy "default-src 'self' 'unsafe-inline';";
+            add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
+        }
+
+        location ~ ^\/(api(\-auth)?|docs|silk|static)\/ {
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $remote_addr;
+            proxy_set_header Host $http_host;
+
+            proxy_pass http://${BACKEND};
+
+            proxy_redirect off;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection $connection_upgrade;
+            proxy_set_header Sec-WebSocket-Extensions $http_sec_websocket_extensions;
+            proxy_set_header Sec-WebSocket-Key $http_sec_websocket_key;
+            proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol;
+            proxy_set_header Sec-WebSocket-Version $http_sec_websocket_version;
+        }
+
+        location /media/ {
+            alias ${FILE_UPLOAD_DIR};
+            expires max;
+        }
+
+        location /media/tmp/ {
+            deny all;
+            return 404;
+        }
+
+        location ~ /media/conversation_message_attachment_(files|previews|thumbnails)/ {
+            deny all;
+            return 404;
+        }
+
+        location /uploads/ {
+            internal;
+            alias ${FILE_UPLOAD_DIR};
+            etag on;
+        }
+
+        location /community_proxy/ {
+            proxy_pass https://community.karrot.world/;
+        }
+
+        error_page 503 @maintenance;
+
+        location @maintenance {
+          try_files /maintenance.html =503;
+        }
+
+    }