wild-cloud/docs/guides/making-backups.md

Making Backups

This guide covers how to create backups of your wild-cloud infrastructure using the integrated backup system.

Overview

The wild-cloud backup system creates encrypted, deduplicated snapshots using restic. It backs up three main components:

• Applications: Database dumps and persistent volume data
• Cluster: Kubernetes resources and etcd state
• Configuration: Wild-cloud repository and settings

Prerequisites

Before making backups, ensure you have:

1. Environment configured: Run source env.sh to load backup configuration
2. Restic repository: Backup repository configured in config.yaml
3. Backup password: Set in wild-cloud secrets
4. Staging directory: Configured path for temporary backup files

Backup Components

Applications (wild-app-backup)

Backs up individual applications including:

• Database dumps: PostgreSQL/MySQL databases in compressed custom format
• PVC data: Application files streamed directly for restic deduplication
• Auto-discovery: Finds databases and PVCs based on app manifest.yaml

Cluster Resources (wild-backup --cluster-only)

Backs up cluster-wide resources:

• Kubernetes resources: All pods, services, deployments, secrets, configmaps
• Storage definitions: PersistentVolumes, PVCs, StorageClasses
• etcd snapshot: Complete cluster state for disaster recovery

Configuration (wild-backup --home-only)

Backs up wild-cloud configuration:

• Repository contents: All app definitions, manifests, configurations
• Settings: Wild-cloud configuration files and customizations

Making Backups

Full System Backup (Recommended)

Create a complete backup of everything:

# Backup all components (apps + cluster + config)
wild-backup

This is equivalent to:

wild-backup --home --apps --cluster

Selective Backups

Applications Only

# All applications
wild-backup --apps-only

# Single application  
wild-app-backup discourse

# Multiple applications
wild-app-backup discourse gitea immich

Cluster Only

# Kubernetes resources + etcd
wild-backup --cluster-only

Configuration Only

# Wild-cloud repository
wild-backup --home-only

Excluding Components

Skip specific components:

# Skip config, backup apps + cluster
wild-backup --no-home

# Skip applications, backup config + cluster  
wild-backup --no-apps

# Skip cluster resources, backup config + apps
wild-backup --no-cluster

Backup Process Details

Application Backup Process

1. Discovery: Parses manifest.yaml to find database and PVC dependencies
2. Database backup: Creates compressed custom-format dumps
3. PVC backup: Streams files directly to staging for restic deduplication
4. Staging: Organizes files in clean directory structure
5. Upload: Creates individual restic snapshots per application

Cluster Backup Process

1. Resource export: Exports all Kubernetes resources to YAML
2. etcd snapshot: Creates point-in-time etcd backup via talosctl
3. Upload: Creates single restic snapshot for cluster state

Restic Snapshots

Each backup creates tagged restic snapshots:

# View all snapshots
restic snapshots

# Filter by component
restic snapshots --tag discourse    # Specific app
restic snapshots --tag cluster      # Cluster resources
restic snapshots --tag wc-home      # Wild-cloud config

Where Backup Files Are Staged

Before uploading to your restic repository, backup files are organized in a staging directory. This temporary area lets you see exactly what's being backed up and helps with deduplication.

Here's what the staging area looks like:

backup-staging/
├── apps/
│   ├── discourse/
│   │   ├── database_20250816T120000Z.dump
│   │   ├── globals_20250816T120000Z.sql  
│   │   └── discourse/
│   │       └── data/         # All the actual files
│   ├── gitea/
│   │   ├── database_20250816T120000Z.dump
│   │   └── gitea-data/
│   │       └── data/         # Git repositories, etc.
│   └── immich/
│       ├── database_20250816T120000Z.dump
│       └── immich-data/
│           └── upload/       # Photos and videos
└── cluster/
    ├── all-resources.yaml    # All running services
    ├── secrets.yaml          # Passwords and certificates
    ├── configmaps.yaml       # Configuration data
    └── etcd-snapshot.db      # Complete cluster state

This staging approach means you can examine backup contents before they're uploaded, and restic can efficiently deduplicate files that haven't changed.

Advanced Usage

Custom Backup Scripts

Applications can provide custom backup logic:

# Create apps/myapp/backup.sh for custom behavior
chmod +x apps/myapp/backup.sh

# wild-app-backup will use custom script if present
wild-app-backup myapp

Monitoring Backup Status

# Check recent snapshots
restic snapshots | head -20

# Check specific app backups
restic snapshots --tag discourse

# Verify backup integrity
restic check

Backup Automation

Set up automated backups with cron:

# Daily full backup at 2 AM
0 2 * * * cd /data/repos/payne-cloud && source env.sh && wild-backup

# Hourly app backups during business hours  
0 9-17 * * * cd /data/repos/payne-cloud && source env.sh && wild-backup --apps-only

Performance Considerations

Large PVCs (like Immich photos)

The streaming backup approach provides:

• First backup: Full transfer time (all files processed)
• Subsequent backups: Only changed files processed (dramatically faster)
• Storage efficiency: Restic deduplication reduces storage usage

Network Usage

• Database dumps: Compressed at source, efficient transfer
• PVC data: Uncompressed transfer, but restic handles deduplication
• etcd snapshots: Small files, minimal impact

Troubleshooting

Common Issues

"No databases or PVCs found"

• App has no manifest.yaml with database dependencies
• No PVCs with matching labels in app namespace
• Create custom backup.sh script for special cases

"kubectl not found"

• Ensure kubectl is installed and configured
• Check cluster connectivity with kubectl get nodes

"Staging directory not set"

• Configure cloud.backup.staging in config.yaml
• Ensure directory exists and is writable

"Could not create etcd backup"

• Ensure talosctl is installed for Talos clusters
• Check control plane node connectivity
• Verify etcd pods are accessible in kube-system namespace

Backup Verification

Always verify backups periodically:

# Check restic repository integrity
restic check

# List recent snapshots
restic snapshots --compact

# Test restore to different directory
restic restore latest --target /tmp/restore-test

Security Notes

• Encryption: All backups are encrypted with your backup password
• Secrets: Kubernetes secrets are included in cluster backups
• Access control: Secure your backup repository and passwords
• Network: Consider bandwidth usage for large initial backups

Next Steps

• Restoring Backups - Learn how to restore from backups
• Configure automated backup schedules
• Set up backup monitoring and alerting
• Test disaster recovery procedures