wild-cloud/wild-directory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(karrot): fix nginx DNS resolver for Kubernetes and add README

Browse Source 
The karrot-frontend nginx template hardcodes Docker's DNS resolver
(127.0.0.11) which doesn't exist in Kubernetes. Added a ConfigMap to
override the template, removing the Docker DNS resolver by using a
direct proxy_pass with no nginx variable (which would force runtime
resolution). Also adds a README with usage instructions.

Documents lessons learned in ADDING-APPS-NOTES.md:
- Note 24: nginx Docker DNS resolver doesn't work in Kubernetes; any
  nginx variable in proxy_pass (including $request_uri) forces runtime
  DNS resolution requiring a resolver directive
- Note 25: ConfigMap changes don't restart pods; subPath mounts never
  auto-update; always follow with kubectl rollout restart
- Note 26: includeSelectors mismatch affects every deployment in an
  app — check all endpoints, not just the web-facing one

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Paul Payne
2026-06-22 04:08:04 +00:00
parent 7aa3e04829
commit 74570413fc
5 changed files with 270 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

110
ADDING-APPS-NOTES.md
View File

@@ -179,6 +179,114 @@ Node registration flow: when a Tailscale client connects via browser (not pre-au
shows a command like `headscale auth register --auth-id hskey-authreq-XXXX --user USERNAME`.
Run it via `kubectl exec -n headscale deploy/headscale -- headscale auth register --auth-id <id> --user <username>`.
### 21. "No available server" from Traefik — kustomize label selector mismatch
If a deployment was originally created without kustomize labels in its selector (or with stale templates), re-deploying via `wild app deploy` cannot fix it because `spec.selector` on a Deployment is immutable once set. The Service selector will update (it's mutable) but pods won't match it, leaving the Service with no endpoints.
- **Symptom**: Traefik returns "No available server" even though pods are Running
- **Diagnosis**: `kubectl get endpoints <app> -n <namespace>` shows `<none>`; pod labels don't include `app: <name>` / `managedBy: kustomize` / `partOf: wild-cloud`
- **Fix**: Delete the affected deployments and re-apply kustomize — `kubectl delete deployment <name> -n <ns>` then `kubectl apply -k <instance>/apps/<app>/`
- **Root cause**: The Deployment selector is set correctly on first deploy, but if the resource already existed with only `component: web` (e.g., from a pre-kustomize deployment), the selector is stuck
### 22. Apps using same-origin iframes need SAMEORIGIN frame policy
The global Traefik `security-headers` middleware applies `X-Frame-Options: SAMEORIGIN` to all responses (changed from `DENY`). Apps that use iframes internally (like Etherpad's pad editor, which loads `../static/empty.html` in an iframe) were broken by the stricter `DENY` policy.
- **Root cause**: `DENY` prevents even same-origin iframes. `SAMEORIGIN` still blocks cross-origin embedding while allowing apps to embed their own sub-pages.
- **Symptoms**: JavaScript error `Blocked a frame with origin "https://..." from accessing a cross-origin frame` — occurs when an app creates a same-origin iframe and then the `X-Frame-Options: DENY` header causes the browser to block access to it.
- **Important**: Route-level Traefik middlewares run BEFORE entrypoint middlewares on the response path. A per-app middleware cannot override a global entrypoint middleware — the entrypoint middleware always runs last.
- **Fix applied globally**: Changed `frameDeny: true` to `X-Frame-Options: SAMEORIGIN` in `customResponseHeaders` in the crowdsec `security-headers` middleware.
- **Affects**: Etherpad (pad editor iframe), any app that loads sub-pages in iframes on the same domain.
### 24. nginx proxy containers: Docker DNS resolver (`127.0.0.11`) doesn't work in Kubernetes
Apps that use nginx as a frontend proxy (e.g., karrot-frontend) often ship with an nginx config template that uses Docker's embedded DNS resolver:
```nginx
resolver 127.0.0.11 valid=3s;
set $backend app-backend:8000;
proxy_pass http://$backend$request_uri;
```
This fails in Kubernetes with `send() failed (111: Connection refused) while resolving, resolver: 127.0.0.11`. There's a second trap: **any nginx variable anywhere in the `proxy_pass` URL** (even `$request_uri` in the path) forces runtime DNS resolution and requires a `resolver` directive. Removing the resolver line but keeping `proxy_pass http://backend:8000$request_uri;` still fails with `no resolver defined to resolve backend`.
**Fix**: override the nginx config template via a ConfigMap mounted with `subPath`, and remove the variable from `proxy_pass` entirely:
```nginx
# Instead of:
resolver 127.0.0.11 valid=3s;
set $backend ${BACKEND};
proxy_pass http://$backend$request_uri;
# Use:
proxy_pass http://${BACKEND};
```
With no nginx variable in `proxy_pass`, nginx resolves the hostname at startup using the pod's `/etc/resolv.conf` (which points to CoreDNS). The full request URI is still forwarded automatically in a regex `location ~` block. Mount the ConfigMap with `subPath` to override just the template file without replacing the whole directory.
### 25. ConfigMap changes don't restart pods — and `subPath` mounts never auto-update
When you update a ConfigMap and run `wild app deploy` (`kubectl apply`), the ConfigMap object is updated but **pods are not restarted**. You must explicitly restart them:
```bash
kubectl rollout restart deployment/<name> -n <namespace>
```
There's a further subtlety: ConfigMaps mounted with `subPath` are **never** live-updated in the pod filesystem, even if you wait. The pod must be restarted to pick up changes regardless of the `subPath` mount update behavior.
This means: after any ConfigMap change, always follow up with a rollout restart for affected deployments.
### 26. The `includeSelectors: true` mismatch affects every deployment in the app — check them all
When diagnosing "no available server" from Traefik (note #21), it's easy to focus only on the deployment that serves the HTTP route and miss other deployments in the same namespace. The `includeSelectors: true` label propagation affects every resource in the kustomization — if any deployment pre-existed with only `component: X` labels, its Service will have `<none>` endpoints and dependent services (backends, workers, Redis) will silently fail.
**Always run this after any selector-related redeploy**:
```bash
kubectl get endpoints --all-namespaces | grep "<none>"
```
Check every deployment in the affected namespace, not just the one Traefik is routing to. A Redis pod with `<none>` endpoints will cause the backend to crash with `ConnectionError` even though the Redis pod itself shows `1/1 Running`.
**Checklist when fixing a selector mismatch in an app**:
1. `kubectl get endpoints -n <app>` — identify all services with `<none>`
2. `kubectl get pods -n <app> --show-labels` — confirm which pods have old labels
3. Delete ALL affected deployments (not just the web one), then redeploy
4. Verify all endpoints are populated before declaring the app fixed
### 23. Post-deploy admin scripts for apps that require manual first-run steps
Some apps can't complete setup through the web UI until an initial admin account is created via a CLI command. Package these as shell scripts in `scripts/` alongside the kustomize files so users have a repeatable, documented way to run them.
**When to add a script**: If the app's README or upstream docs require running a management command after deploy (creating a superuser, registering a node, inviting the first user, etc.) — wrap it in a script.
**Always register scripts in `manifest.yaml`** — the web UI automatically shows a button with a parameter form for every script listed there. Without this, users have to drop to the CLI even though the UI fully supports it. Format:
```yaml
scripts:
- name: create-superuser
path: scripts/create-superuser.sh
description: "Short description shown in the UI."
params:
- name: EMAIL
description: Description shown in the param form
required: true
- name: PASSWORD
description: Leave blank to generate a random one
```
**Script conventions** (follow `synapse/versions/v1/scripts/create-user.sh` as the reference):
- Require `KUBECONFIG`, `WILD_INSTANCE`, and `WILD_API_DATA_DIR` env vars; exit with a clear error if missing
- Read `namespace` from `config.yaml` via `yq` so the script doesn't hardcode values
- Auto-generate passwords with `openssl rand` if `PASSWORD` is not supplied
- Find the running pod by label (don't hardcode a pod name); fail clearly if none found
- Print credentials at the end with a "save this — it won't be shown again" warning
- Reference the script in the app README's "First-Time Setup" section in place of a raw `kubectl exec` command
**Non-interactive Django `createsuperuser`**:
```bash
kubectl exec -n <ns> <pod> -c <container> -- \
env DJANGO_SUPERUSER_PASSWORD="${PASSWORD}" \
python manage.py createsuperuser --email "${EMAIL}" --noinput
```
The `--noinput` flag reads the password from `DJANGO_SUPERUSER_PASSWORD`.
**Affected apps so far**: Eventyay (`scripts/create-superuser.sh`), Synapse (`scripts/create-user.sh`), Headscale (`scripts/register-node.sh`).
## App Status Tracking
| App | Status | Notes |
@@ -220,7 +328,7 @@ Run it via `kubectl exec -n headscale deploy/headscale -- headscale auth registe
| uMap | ✅ working | Host header needed in health probes (Django ALLOWED_HOSTS) |
| MapComplete | pending | |
| Terrastories | pending | |
| Karrot | already done | |
| Karrot | ✅ working | nginx frontend: override template via ConfigMap to remove Docker DNS resolver (see note 24) |
| LiquidFeedback | pending | |
| Your Priorities | ❌ blocked | No public Docker images (ghcr.io requires auth) |
| Helios Voting | pending | |