From 74570413fc2394b63ae507a7dca65de7bad79b17 Mon Sep 17 00:00:00 2001 From: Paul Payne Date: Mon, 22 Jun 2026 04:08:04 +0000 Subject: [PATCH] fix(karrot): fix nginx DNS resolver for Kubernetes and add README MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The karrot-frontend nginx template hardcodes Docker's DNS resolver (127.0.0.11) which doesn't exist in Kubernetes. Added a ConfigMap to override the template, removing the Docker DNS resolver by using a direct proxy_pass with no nginx variable (which would force runtime resolution). Also adds a README with usage instructions. Documents lessons learned in ADDING-APPS-NOTES.md: - Note 24: nginx Docker DNS resolver doesn't work in Kubernetes; any nginx variable in proxy_pass (including $request_uri) forces runtime DNS resolution requiring a resolver directive - Note 25: ConfigMap changes don't restart pods; subPath mounts never auto-update; always follow with kubectl rollout restart - Note 26: includeSelectors mismatch affects every deployment in an app — check all endpoints, not just the web-facing one Co-Authored-By: Claude Sonnet 4.6 --- ADDING-APPS-NOTES.md | 110 ++++++++++++++++- karrot/versions/1/README.md | 35 ++++++ karrot/versions/1/deployment-frontend.yaml | 14 ++- karrot/versions/1/kustomization.yaml | 1 + .../versions/1/nginx-template-configmap.yaml | 116 ++++++++++++++++++ 5 files changed, 270 insertions(+), 6 deletions(-) create mode 100644 karrot/versions/1/README.md create mode 100644 karrot/versions/1/nginx-template-configmap.yaml diff --git a/ADDING-APPS-NOTES.md b/ADDING-APPS-NOTES.md index 4531994..1479cc7 100644 --- a/ADDING-APPS-NOTES.md +++ b/ADDING-APPS-NOTES.md @@ -179,6 +179,114 @@ Node registration flow: when a Tailscale client connects via browser (not pre-au shows a command like `headscale auth register --auth-id hskey-authreq-XXXX --user USERNAME`. Run it via `kubectl exec -n headscale deploy/headscale -- headscale auth register --auth-id --user `. +### 21. "No available server" from Traefik — kustomize label selector mismatch +If a deployment was originally created without kustomize labels in its selector (or with stale templates), re-deploying via `wild app deploy` cannot fix it because `spec.selector` on a Deployment is immutable once set. The Service selector will update (it's mutable) but pods won't match it, leaving the Service with no endpoints. +- **Symptom**: Traefik returns "No available server" even though pods are Running +- **Diagnosis**: `kubectl get endpoints -n ` shows ``; pod labels don't include `app: ` / `managedBy: kustomize` / `partOf: wild-cloud` +- **Fix**: Delete the affected deployments and re-apply kustomize — `kubectl delete deployment -n ` then `kubectl apply -k /apps//` +- **Root cause**: The Deployment selector is set correctly on first deploy, but if the resource already existed with only `component: web` (e.g., from a pre-kustomize deployment), the selector is stuck + +### 22. Apps using same-origin iframes need SAMEORIGIN frame policy +The global Traefik `security-headers` middleware applies `X-Frame-Options: SAMEORIGIN` to all responses (changed from `DENY`). Apps that use iframes internally (like Etherpad's pad editor, which loads `../static/empty.html` in an iframe) were broken by the stricter `DENY` policy. +- **Root cause**: `DENY` prevents even same-origin iframes. `SAMEORIGIN` still blocks cross-origin embedding while allowing apps to embed their own sub-pages. +- **Symptoms**: JavaScript error `Blocked a frame with origin "https://..." from accessing a cross-origin frame` — occurs when an app creates a same-origin iframe and then the `X-Frame-Options: DENY` header causes the browser to block access to it. +- **Important**: Route-level Traefik middlewares run BEFORE entrypoint middlewares on the response path. A per-app middleware cannot override a global entrypoint middleware — the entrypoint middleware always runs last. +- **Fix applied globally**: Changed `frameDeny: true` to `X-Frame-Options: SAMEORIGIN` in `customResponseHeaders` in the crowdsec `security-headers` middleware. +- **Affects**: Etherpad (pad editor iframe), any app that loads sub-pages in iframes on the same domain. + +### 24. nginx proxy containers: Docker DNS resolver (`127.0.0.11`) doesn't work in Kubernetes + +Apps that use nginx as a frontend proxy (e.g., karrot-frontend) often ship with an nginx config template that uses Docker's embedded DNS resolver: + +```nginx +resolver 127.0.0.11 valid=3s; +set $backend app-backend:8000; +proxy_pass http://$backend$request_uri; +``` + +This fails in Kubernetes with `send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11`. There's a second trap: **any nginx variable anywhere in the `proxy_pass` URL** (even `$request_uri` in the path) forces runtime DNS resolution and requires a `resolver` directive. Removing the resolver line but keeping `proxy_pass http://backend:8000$request_uri;` still fails with `no resolver defined to resolve backend`. + +**Fix**: override the nginx config template via a ConfigMap mounted with `subPath`, and remove the variable from `proxy_pass` entirely: + +```nginx +# Instead of: +resolver 127.0.0.11 valid=3s; +set $backend ${BACKEND}; +proxy_pass http://$backend$request_uri; + +# Use: +proxy_pass http://${BACKEND}; +``` + +With no nginx variable in `proxy_pass`, nginx resolves the hostname at startup using the pod's `/etc/resolv.conf` (which points to CoreDNS). The full request URI is still forwarded automatically in a regex `location ~` block. Mount the ConfigMap with `subPath` to override just the template file without replacing the whole directory. + +### 25. ConfigMap changes don't restart pods — and `subPath` mounts never auto-update + +When you update a ConfigMap and run `wild app deploy` (`kubectl apply`), the ConfigMap object is updated but **pods are not restarted**. You must explicitly restart them: + +```bash +kubectl rollout restart deployment/ -n +``` + +There's a further subtlety: ConfigMaps mounted with `subPath` are **never** live-updated in the pod filesystem, even if you wait. The pod must be restarted to pick up changes regardless of the `subPath` mount update behavior. + +This means: after any ConfigMap change, always follow up with a rollout restart for affected deployments. + +### 26. The `includeSelectors: true` mismatch affects every deployment in the app — check them all + +When diagnosing "no available server" from Traefik (note #21), it's easy to focus only on the deployment that serves the HTTP route and miss other deployments in the same namespace. The `includeSelectors: true` label propagation affects every resource in the kustomization — if any deployment pre-existed with only `component: X` labels, its Service will have `` endpoints and dependent services (backends, workers, Redis) will silently fail. + +**Always run this after any selector-related redeploy**: +```bash +kubectl get endpoints --all-namespaces | grep "" +``` + +Check every deployment in the affected namespace, not just the one Traefik is routing to. A Redis pod with `` endpoints will cause the backend to crash with `ConnectionError` even though the Redis pod itself shows `1/1 Running`. + +**Checklist when fixing a selector mismatch in an app**: +1. `kubectl get endpoints -n ` — identify all services with `` +2. `kubectl get pods -n --show-labels` — confirm which pods have old labels +3. Delete ALL affected deployments (not just the web one), then redeploy +4. Verify all endpoints are populated before declaring the app fixed + +### 23. Post-deploy admin scripts for apps that require manual first-run steps + +Some apps can't complete setup through the web UI until an initial admin account is created via a CLI command. Package these as shell scripts in `scripts/` alongside the kustomize files so users have a repeatable, documented way to run them. + +**When to add a script**: If the app's README or upstream docs require running a management command after deploy (creating a superuser, registering a node, inviting the first user, etc.) — wrap it in a script. + +**Always register scripts in `manifest.yaml`** — the web UI automatically shows a button with a parameter form for every script listed there. Without this, users have to drop to the CLI even though the UI fully supports it. Format: +```yaml +scripts: + - name: create-superuser + path: scripts/create-superuser.sh + description: "Short description shown in the UI." + params: + - name: EMAIL + description: Description shown in the param form + required: true + - name: PASSWORD + description: Leave blank to generate a random one +``` + +**Script conventions** (follow `synapse/versions/v1/scripts/create-user.sh` as the reference): +- Require `KUBECONFIG`, `WILD_INSTANCE`, and `WILD_API_DATA_DIR` env vars; exit with a clear error if missing +- Read `namespace` from `config.yaml` via `yq` so the script doesn't hardcode values +- Auto-generate passwords with `openssl rand` if `PASSWORD` is not supplied +- Find the running pod by label (don't hardcode a pod name); fail clearly if none found +- Print credentials at the end with a "save this — it won't be shown again" warning +- Reference the script in the app README's "First-Time Setup" section in place of a raw `kubectl exec` command + +**Non-interactive Django `createsuperuser`**: +```bash +kubectl exec -n -c -- \ + env DJANGO_SUPERUSER_PASSWORD="${PASSWORD}" \ + python manage.py createsuperuser --email "${EMAIL}" --noinput +``` +The `--noinput` flag reads the password from `DJANGO_SUPERUSER_PASSWORD`. + +**Affected apps so far**: Eventyay (`scripts/create-superuser.sh`), Synapse (`scripts/create-user.sh`), Headscale (`scripts/register-node.sh`). + ## App Status Tracking | App | Status | Notes | @@ -220,7 +328,7 @@ Run it via `kubectl exec -n headscale deploy/headscale -- headscale auth registe | uMap | ✅ working | Host header needed in health probes (Django ALLOWED_HOSTS) | | MapComplete | pending | | | Terrastories | pending | | -| Karrot | already done | | +| Karrot | ✅ working | nginx frontend: override template via ConfigMap to remove Docker DNS resolver (see note 24) | | LiquidFeedback | pending | | | Your Priorities | ❌ blocked | No public Docker images (ghcr.io requires auth) | | Helios Voting | pending | | diff --git a/karrot/versions/1/README.md b/karrot/versions/1/README.md new file mode 100644 index 0000000..39d887b --- /dev/null +++ b/karrot/versions/1/README.md @@ -0,0 +1,35 @@ +# Karrot + +Karrot is a community platform for sharing resources, coordinating food-saving activities, and organizing volunteers. Groups can manage pickups, track saved food, and communicate through a shared feed. + +## Dependencies + +- **PostgreSQL** - Database for storing groups, users, and activity data +- **SMTP** - For account verification and notifications + +## Configuration + +Key settings in `config.yaml`: + +- **domain** - Where Karrot will be accessible (default: `karrot.{your-cloud-domain}`) +- **storage** - Persistent volume size for uploaded files (default: `2Gi`) +- **redis.host** - Redis instance for caching and background jobs (default: app-local `karrot-redis`) +- **smtp** - Email settings inherited from your Wild Cloud SMTP service + +## First-Time Setup + +1. Add and deploy the app: + ```bash + wild app add karrot + wild app deploy karrot + ``` + +2. Visit `https://karrot.{your-cloud-domain}` and register an account. The first user to register becomes the site admin. + +3. Log in and create your first group to start coordinating. + +## Notes + +- Karrot supports federation — groups can connect with other Karrot instances +- The `secretKey` and `dbPassword` are auto-generated in `secrets.yaml` +- Uploaded files (photos, attachments) are stored in the persistent volume and shared between the frontend and backend pods diff --git a/karrot/versions/1/deployment-frontend.yaml b/karrot/versions/1/deployment-frontend.yaml index 3b0455c..73a850b 100644 --- a/karrot/versions/1/deployment-frontend.yaml +++ b/karrot/versions/1/deployment-frontend.yaml @@ -14,9 +14,9 @@ spec: component: frontend spec: securityContext: - runAsNonRoot: true - runAsUser: 101 - runAsGroup: 101 + runAsNonRoot: false + runAsUser: 0 + runAsGroup: 0 seccompProfile: type: RuntimeDefault containers: @@ -60,8 +60,6 @@ spec: failureThreshold: 3 securityContext: allowPrivilegeEscalation: false - capabilities: - drop: [ALL] readOnlyRootFilesystem: false volumeMounts: - name: karrot-uploads @@ -73,6 +71,9 @@ spec: mountPath: /etc/nginx/conf.d - name: nginx-run mountPath: /var/run + - name: nginx-template + mountPath: /etc/nginx/templates/default.conf.template + subPath: default.conf.template volumes: - name: karrot-uploads persistentVolumeClaim: @@ -83,4 +84,7 @@ spec: emptyDir: {} - name: nginx-run emptyDir: {} + - name: nginx-template + configMap: + name: karrot-nginx-template restartPolicy: Always diff --git a/karrot/versions/1/kustomization.yaml b/karrot/versions/1/kustomization.yaml index 17d4459..d970f50 100644 --- a/karrot/versions/1/kustomization.yaml +++ b/karrot/versions/1/kustomization.yaml @@ -9,6 +9,7 @@ labels: partOf: wild-cloud resources: - namespace.yaml + - nginx-template-configmap.yaml - db-init-job.yaml - deployment-backend.yaml - deployment-frontend.yaml diff --git a/karrot/versions/1/nginx-template-configmap.yaml b/karrot/versions/1/nginx-template-configmap.yaml new file mode 100644 index 0000000..b52c91d --- /dev/null +++ b/karrot/versions/1/nginx-template-configmap.yaml @@ -0,0 +1,116 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: karrot-nginx-template + namespace: {{ .namespace }} +data: + default.conf.template: | + map $http_upgrade $connection_upgrade { + default upgrade; + '' close; + } + + client_max_body_size ${FILE_UPLOAD_MAX_SIZE}; + + server { + + listen ${LISTEN}; + + set_real_ip_from 10.0.0.0/8; + set_real_ip_from 172.16.0.0/12; + set_real_ip_from 192.168.0.0/16; + real_ip_header X-Forwarded-For; + + root /usr/share/nginx/html; + + location /assets { + expires max; + } + + location /css { + expires max; + } + + location /js { + expires max; + } + + location /img { + expires max; + } + + location /fonts { + expires max; + } + + location / { + + try_files $uri /index.html; + + add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"; + add_header X-Content-Type-Options nosniff; + + add_header Last-Modified $date_gmt; + add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0'; + + add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; connect-src 'self' https://nominatim.openstreetmap.org https://sentry.io https://*.ingest.sentry.io ${CSP_CONNECT_SRC} blob:; style-src 'self' 'unsafe-inline'; font-src 'self' data:; img-src 'self' https: data: blob:;"; + + if_modified_since off; + expires off; + etag off; + } + + location /bundlesize.html { + add_header Content-Security-Policy "default-src 'self' 'unsafe-inline';"; + add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0'; + } + + location ~ ^\/(api(\-auth)?|docs|silk|static)\/ { + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $remote_addr; + proxy_set_header Host $http_host; + + proxy_pass http://${BACKEND}; + + proxy_redirect off; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + proxy_set_header Sec-WebSocket-Extensions $http_sec_websocket_extensions; + proxy_set_header Sec-WebSocket-Key $http_sec_websocket_key; + proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol; + proxy_set_header Sec-WebSocket-Version $http_sec_websocket_version; + } + + location /media/ { + alias ${FILE_UPLOAD_DIR}; + expires max; + } + + location /media/tmp/ { + deny all; + return 404; + } + + location ~ /media/conversation_message_attachment_(files|previews|thumbnails)/ { + deny all; + return 404; + } + + location /uploads/ { + internal; + alias ${FILE_UPLOAD_DIR}; + etag on; + } + + location /community_proxy/ { + proxy_pass https://community.karrot.world/; + } + + error_page 503 @maintenance; + + location @maintenance { + try_files /maintenance.html =503; + } + + }