wild-cloud/wild-directory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(karrot): fix nginx DNS resolver for Kubernetes and add README

Browse Source 
The karrot-frontend nginx template hardcodes Docker's DNS resolver
(127.0.0.11) which doesn't exist in Kubernetes. Added a ConfigMap to
override the template, removing the Docker DNS resolver by using a
direct proxy_pass with no nginx variable (which would force runtime
resolution). Also adds a README with usage instructions.

Documents lessons learned in ADDING-APPS-NOTES.md:
- Note 24: nginx Docker DNS resolver doesn't work in Kubernetes; any
  nginx variable in proxy_pass (including $request_uri) forces runtime
  DNS resolution requiring a resolver directive
- Note 25: ConfigMap changes don't restart pods; subPath mounts never
  auto-update; always follow with kubectl rollout restart
- Note 26: includeSelectors mismatch affects every deployment in an
  app — check all endpoints, not just the web-facing one

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Paul Payne
2026-06-22 04:08:04 +00:00
parent 7aa3e04829
commit 74570413fc
5 changed files with 270 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

110
ADDING-APPS-NOTES.md
View File

@@ -179,6 +179,114 @@ Node registration flow: when a Tailscale client connects via browser (not pre-au
shows a command like `headscale auth register --auth-id hskey-authreq-XXXX --user USERNAME`.
Run it via `kubectl exec -n headscale deploy/headscale -- headscale auth register --auth-id <id> --user <username>`.
### 21. "No available server" from Traefik — kustomize label selector mismatch
If a deployment was originally created without kustomize labels in its selector (or with stale templates), re-deploying via `wild app deploy` cannot fix it because `spec.selector` on a Deployment is immutable once set. The Service selector will update (it's mutable) but pods won't match it, leaving the Service with no endpoints.
- **Symptom**: Traefik returns "No available server" even though pods are Running
- **Diagnosis**: `kubectl get endpoints <app> -n <namespace>` shows `<none>`; pod labels don't include `app: <name>` / `managedBy: kustomize` / `partOf: wild-cloud`
- **Fix**: Delete the affected deployments and re-apply kustomize — `kubectl delete deployment <name> -n <ns>` then `kubectl apply -k <instance>/apps/<app>/`
- **Root cause**: The Deployment selector is set correctly on first deploy, but if the resource already existed with only `component: web` (e.g., from a pre-kustomize deployment), the selector is stuck
### 22. Apps using same-origin iframes need SAMEORIGIN frame policy
The global Traefik `security-headers` middleware applies `X-Frame-Options: SAMEORIGIN` to all responses (changed from `DENY`). Apps that use iframes internally (like Etherpad's pad editor, which loads `../static/empty.html` in an iframe) were broken by the stricter `DENY` policy.
- **Root cause**: `DENY` prevents even same-origin iframes. `SAMEORIGIN` still blocks cross-origin embedding while allowing apps to embed their own sub-pages.
- **Symptoms**: JavaScript error `Blocked a frame with origin "https://..." from accessing a cross-origin frame` — occurs when an app creates a same-origin iframe and then the `X-Frame-Options: DENY` header causes the browser to block access to it.
- **Important**: Route-level Traefik middlewares run BEFORE entrypoint middlewares on the response path. A per-app middleware cannot override a global entrypoint middleware — the entrypoint middleware always runs last.
- **Fix applied globally**: Changed `frameDeny: true` to `X-Frame-Options: SAMEORIGIN` in `customResponseHeaders` in the crowdsec `security-headers` middleware.
- **Affects**: Etherpad (pad editor iframe), any app that loads sub-pages in iframes on the same domain.
### 24. nginx proxy containers: Docker DNS resolver (`127.0.0.11`) doesn't work in Kubernetes
Apps that use nginx as a frontend proxy (e.g., karrot-frontend) often ship with an nginx config template that uses Docker's embedded DNS resolver:
```nginx
resolver 127.0.0.11 valid=3s;
set $backend app-backend:8000;
proxy_pass http://$backend$request_uri;
```
This fails in Kubernetes with `send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11`. There's a second trap: **any nginx variable anywhere in the `proxy_pass` URL** (even `$request_uri` in the path) forces runtime DNS resolution and requires a `resolver` directive. Removing the resolver line but keeping `proxy_pass http://backend:8000$request_uri;` still fails with `no resolver defined to resolve backend`.
**Fix**: override the nginx config template via a ConfigMap mounted with `subPath`, and remove the variable from `proxy_pass` entirely:
```nginx
# Instead of:
resolver 127.0.0.11 valid=3s;
set $backend ${BACKEND};
proxy_pass http://$backend$request_uri;
# Use:
proxy_pass http://${BACKEND};
```
With no nginx variable in `proxy_pass`, nginx resolves the hostname at startup using the pod's `/etc/resolv.conf` (which points to CoreDNS). The full request URI is still forwarded automatically in a regex `location ~` block. Mount the ConfigMap with `subPath` to override just the template file without replacing the whole directory.
### 25. ConfigMap changes don't restart pods — and `subPath` mounts never auto-update
When you update a ConfigMap and run `wild app deploy` (`kubectl apply`), the ConfigMap object is updated but **pods are not restarted**. You must explicitly restart them:
```bash
kubectl rollout restart deployment/<name> -n <namespace>
```
There's a further subtlety: ConfigMaps mounted with `subPath` are **never** live-updated in the pod filesystem, even if you wait. The pod must be restarted to pick up changes regardless of the `subPath` mount update behavior.
This means: after any ConfigMap change, always follow up with a rollout restart for affected deployments.
### 26. The `includeSelectors: true` mismatch affects every deployment in the app — check them all
When diagnosing "no available server" from Traefik (note #21), it's easy to focus only on the deployment that serves the HTTP route and miss other deployments in the same namespace. The `includeSelectors: true` label propagation affects every resource in the kustomization — if any deployment pre-existed with only `component: X` labels, its Service will have `<none>` endpoints and dependent services (backends, workers, Redis) will silently fail.
**Always run this after any selector-related redeploy**:
```bash
kubectl get endpoints --all-namespaces | grep "<none>"
```
Check every deployment in the affected namespace, not just the one Traefik is routing to. A Redis pod with `<none>` endpoints will cause the backend to crash with `ConnectionError` even though the Redis pod itself shows `1/1 Running`.
**Checklist when fixing a selector mismatch in an app**:
1. `kubectl get endpoints -n <app>` — identify all services with `<none>`
2. `kubectl get pods -n <app> --show-labels` — confirm which pods have old labels
3. Delete ALL affected deployments (not just the web one), then redeploy
4. Verify all endpoints are populated before declaring the app fixed
### 23. Post-deploy admin scripts for apps that require manual first-run steps
Some apps can't complete setup through the web UI until an initial admin account is created via a CLI command. Package these as shell scripts in `scripts/` alongside the kustomize files so users have a repeatable, documented way to run them.
**When to add a script**: If the app's README or upstream docs require running a management command after deploy (creating a superuser, registering a node, inviting the first user, etc.) — wrap it in a script.
**Always register scripts in `manifest.yaml`** — the web UI automatically shows a button with a parameter form for every script listed there. Without this, users have to drop to the CLI even though the UI fully supports it. Format:
```yaml
scripts:
- name: create-superuser
path: scripts/create-superuser.sh
description: "Short description shown in the UI."
params:
- name: EMAIL
description: Description shown in the param form
required: true
- name: PASSWORD
description: Leave blank to generate a random one
```
**Script conventions** (follow `synapse/versions/v1/scripts/create-user.sh` as the reference):
- Require `KUBECONFIG`, `WILD_INSTANCE`, and `WILD_API_DATA_DIR` env vars; exit with a clear error if missing
- Read `namespace` from `config.yaml` via `yq` so the script doesn't hardcode values
- Auto-generate passwords with `openssl rand` if `PASSWORD` is not supplied
- Find the running pod by label (don't hardcode a pod name); fail clearly if none found
- Print credentials at the end with a "save this — it won't be shown again" warning
- Reference the script in the app README's "First-Time Setup" section in place of a raw `kubectl exec` command
**Non-interactive Django `createsuperuser`**:
```bash
kubectl exec -n <ns> <pod> -c <container> -- \
env DJANGO_SUPERUSER_PASSWORD="${PASSWORD}" \
python manage.py createsuperuser --email "${EMAIL}" --noinput
```
The `--noinput` flag reads the password from `DJANGO_SUPERUSER_PASSWORD`.
**Affected apps so far**: Eventyay (`scripts/create-superuser.sh`), Synapse (`scripts/create-user.sh`), Headscale (`scripts/register-node.sh`).
## App Status Tracking
| App | Status | Notes |
@@ -220,7 +328,7 @@ Run it via `kubectl exec -n headscale deploy/headscale -- headscale auth registe
| uMap | ✅ working | Host header needed in health probes (Django ALLOWED_HOSTS) |
| MapComplete | pending | |
| Terrastories | pending | |
| Karrot | already done | |
| Karrot | ✅ working | nginx frontend: override template via ConfigMap to remove Docker DNS resolver (see note 24) |
| LiquidFeedback | pending | |
| Your Priorities | ❌ blocked | No public Docker images (ghcr.io requires auth) |
| Helios Voting | pending | |

35
karrot/versions/1/README.md Normal file
View File

@@ -0,0 +1,35 @@
# Karrot
Karrot is a community platform for sharing resources, coordinating food-saving activities, and organizing volunteers. Groups can manage pickups, track saved food, and communicate through a shared feed.
## Dependencies
- **PostgreSQL** - Database for storing groups, users, and activity data
- **SMTP** - For account verification and notifications
## Configuration
Key settings in `config.yaml`:
- **domain** - Where Karrot will be accessible (default: `karrot.{your-cloud-domain}`)
- **storage** - Persistent volume size for uploaded files (default: `2Gi`)
- **redis.host** - Redis instance for caching and background jobs (default: app-local `karrot-redis`)
- **smtp** - Email settings inherited from your Wild Cloud SMTP service
## First-Time Setup
1. Add and deploy the app:
```bash
wild app add karrot
wild app deploy karrot
```
2. Visit `https://karrot.{your-cloud-domain}` and register an account. The first user to register becomes the site admin.
3. Log in and create your first group to start coordinating.
## Notes
- Karrot supports federation — groups can connect with other Karrot instances
- The `secretKey` and `dbPassword` are auto-generated in `secrets.yaml`
- Uploaded files (photos, attachments) are stored in the persistent volume and shared between the frontend and backend pods

14
karrot/versions/1/deployment-frontend.yaml
View File

@@ -14,9 +14,9 @@ spec:
component: frontend
spec:
securityContext:
runAsNonRoot: true
runAsUser: 101
runAsGroup: 101
runAsNonRoot: false
runAsUser: 0
runAsGroup: 0
seccompProfile:
type: RuntimeDefault
containers:
@@ -60,8 +60,6 @@ spec:
failureThreshold: 3
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
readOnlyRootFilesystem: false
volumeMounts:
- name: karrot-uploads
@@ -73,6 +71,9 @@ spec:
mountPath: /etc/nginx/conf.d
- name: nginx-run
mountPath: /var/run
- name: nginx-template
mountPath: /etc/nginx/templates/default.conf.template
subPath: default.conf.template
volumes:
- name: karrot-uploads
persistentVolumeClaim:
@@ -83,4 +84,7 @@ spec:
emptyDir: {}
- name: nginx-run
emptyDir: {}
- name: nginx-template
configMap:
name: karrot-nginx-template
restartPolicy: Always

1
karrot/versions/1/kustomization.yaml
View File

@@ -9,6 +9,7 @@ labels:
partOf: wild-cloud
resources:
- namespace.yaml
- nginx-template-configmap.yaml
- db-init-job.yaml
- deployment-backend.yaml
- deployment-frontend.yaml

116
karrot/versions/1/nginx-template-configmap.yaml Normal file
View File

@@ -0,0 +1,116 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: karrot-nginx-template
namespace: {{ .namespace }}
data:
default.conf.template: |
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
client_max_body_size ${FILE_UPLOAD_MAX_SIZE};
server {
listen ${LISTEN};
set_real_ip_from 10.0.0.0/8;
set_real_ip_from 172.16.0.0/12;
set_real_ip_from 192.168.0.0/16;
real_ip_header X-Forwarded-For;
root /usr/share/nginx/html;
location /assets {
expires max;
}
location /css {
expires max;
}
location /js {
expires max;
}
location /img {
expires max;
}
location /fonts {
expires max;
}
location / {
try_files $uri /index.html;
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
add_header X-Content-Type-Options nosniff;
add_header Last-Modified $date_gmt;
add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; connect-src 'self' https://nominatim.openstreetmap.org https://sentry.io https://*.ingest.sentry.io ${CSP_CONNECT_SRC} blob:; style-src 'self' 'unsafe-inline'; font-src 'self' data:; img-src 'self' https: data: blob:;";
if_modified_since off;
expires off;
etag off;
}
location /bundlesize.html {
add_header Content-Security-Policy "default-src 'self' 'unsafe-inline';";
add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
}
location ~ ^\/(api(\-auth)?|docs|silk|static)\/ {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $http_host;
proxy_pass http://${BACKEND};
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Sec-WebSocket-Extensions $http_sec_websocket_extensions;
proxy_set_header Sec-WebSocket-Key $http_sec_websocket_key;
proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol;
proxy_set_header Sec-WebSocket-Version $http_sec_websocket_version;
}
location /media/ {
alias ${FILE_UPLOAD_DIR};
expires max;
}
location /media/tmp/ {
deny all;
return 404;
}
location ~ /media/conversation_message_attachment_(files|previews|thumbnails)/ {
deny all;
return 404;
}
location /uploads/ {
internal;
alias ${FILE_UPLOAD_DIR};
etag on;
}
location /community_proxy/ {
proxy_pass https://community.karrot.world/;
}
error_page 503 @maintenance;
location @maintenance {
try_files /maintenance.html =503;
}
}