Files

Paul Payne bc7a168851 Reorganized for new stable/waypoint versioning design.

2026-05-24 18:28:47 +00:00

22 KiB

Raw Permalink Blame History

Wild Directory Versioning and Upgrade System

Design specification for how Wild Cloud packages its third-party applications, tracks versions, and manages upgrades between versions.

Problem

Wild Cloud packages third-party applications as Kustomize templates. Each application needs:

Identity that doesn't change between versions (name, description, icon)
Version-specific files (k8s manifests, config schema, secrets schema)
Upgrade routing when breaking changes prevent direct version jumps
A clear, low-friction workflow for maintainers bumping versions

The system must handle both simple apps (any version replaces any other) and complex apps (database migrations, mandatory stepping stones between major versions).

Design

Two-level structure: `app.yaml` + `versions/`

Each app in the Wild Directory has a root-level app.yaml for identity and a versions/ directory containing one or more version slots:

ghost/
+-- app.yaml
+-- versions/
    +-- 5/
        +-- manifest.yaml
        +-- kustomization.yaml
        +-- deployment.yaml
        +-- ...

app.yaml holds version-independent fields: name, description, icon, category, the latest pointer, and upgrade routing rules. These are facts about the app itself, not about any particular release.

versions/{slot}/manifest.yaml holds version-specific fields: the precise version string, dependency declarations, default config, default secrets, deploy configuration, and per-version migration jobs. These are facts about a particular release.

When the API installs an app, it reads both files and merges them. The installed manifest in the instance data directory contains the complete picture (name, description, icon, version, config, etc.).

Version slots vs version strings

A version directory is a slot, not a precise version identifier.

The directory name is a stable label chosen by the maintainer. The actual version string lives in manifest.yaml inside the directory. These are intentionally decoupled:

Concept	Where it lives	Example
Slot name	Directory name under `versions/`	`5`
Actual version	`version` field in `manifest.yaml`	`5.118.1-2`
Latest pointer	`latest` field in `app.yaml`	`5`
Waypoint pointer	`via` field in upgrade rules	`2`

The slot name should be the simplest stable identifier that distinguishes it from other slots. For semver apps, use the major version (1, 2, 3). For apps with non-semver schemes, use whatever upstream version boundary makes sense (5, v4, etc.).

Packaging revisions (-1, -2, etc.) never appear in directory names. They only appear in manifest.yaml's version field. A packaging revision is a Wild Cloud-side fix (template improvement, security context change, config restructure) that doesn't change the upstream software.

Minor and patch versions also don't require new directories unless they introduce a breaking change that requires a waypoint. Updating Ghost from 5.118.1 to 5.119.0 means editing files inside versions/5/ and bumping the version field. The directory stays the same.

This follows how established package systems work:

System	Directory/file identity	Where version lives
Debian source packages	Package name (stable)	`debian/changelog`
Helm charts	Chart name directory	`Chart.yaml`
Homebrew	Formula file per package	Version attribute in file
Nix packages	Package name directory	Derivation attribute
FreeBSD ports	Port name directory	`Makefile` variable
Wild Cloud	Slot directory	`manifest.yaml`

Wild Directory is a source package collection (templates compiled at install time), not an artifact repository (pre-built binaries stored per version). The source package pattern is the right fit.

When directories are created and destroyed

Most apps have exactly one version directory. This is the common case for apps where any version can replace any other (Ghost, Redis, most stateless services).

A second directory appears only when a waypoint is needed -- when a breaking change means some installed versions can't jump directly to the latest and must pass through an intermediate version first.

A directory is removed when it's no longer needed as a waypoint. If version 2/ was a waypoint for upgrading from 1.x to 3.x, but you later decide to drop support for 1.x entirely, you can remove 2/ and update the routing rules to block <2.0.0.

`app.yaml` specification

# Required
name: ghost                    # Must match directory name
is: ghost                      # Unique type id, used for `requires` matching
description: "Ghost is a..."   # Shown in app listings
latest: "5"                    # Slot name pointing to a directory in versions/

# Optional
icon: "https://..."            # URL to app icon
category: infrastructure       # Category for filtering

# Optional -- only needed for apps with breaking upgrade paths
upgrade:
  from:
    - version: ">=3.0.0"       # Constraint against installed version
    - version: ">=2.0.0"
      via: "2"                 # Slot name of waypoint directory
    - version: "<2.0.0"
      blocked: true
      notes: "Upgrade to 2.x first"
  preUpgrade:
    backup: recommended        # "none", "recommended", or "required"

Fields:

Field	Type	Required	Description
`name`	string	yes	App identifier, must match directory name
`is`	string	yes	Type id for dependency matching
`description`	string	yes	Human-readable description
`latest`	string	yes	Slot name -- directory name under `versions/`
`icon`	string	no	URL to icon image
`category`	string	no	Grouping category (e.g. `infrastructure`)
`upgrade`	object	no	Routing rules for version upgrades
`upgrade.from`	list	no	Ordered list of version constraint rules
`upgrade.from[].version`	string	yes	Version constraint: `>=`, `>`, `<=`, `<`, `=`, or `>0`
`upgrade.from[].via`	string	no	Slot name of waypoint to pass through
`upgrade.from[].blocked`	bool	no	If true, upgrade is blocked
`upgrade.from[].notes`	string	no	Human-readable message
`upgrade.preUpgrade`	object	no	Pre-upgrade requirements
`upgrade.preUpgrade.backup`	string	no	`"none"`, `"recommended"`, or `"required"`

latest is a slot name, not a version string. It tells the API which directory to look in. The actual version is read from the manifest inside that directory.

via is also a slot name. It tells the upgrade planner which waypoint directory to route through.

Version `manifest.yaml` specification

# Required
version: 5.118.1-2

# Optional
requires:
  - name: pg
    alias: db
  - name: redis
defaultConfig:
  namespace: ghost
  domain: ghost.{{ .cloud.domain }}
  # ...
defaultSecrets:
  - key: password
  - key: dbUrl
    default: "postgresql://..."
requiredSecrets:
  - db.password
deploy:
  # ...
scripts:
  # ...

# Optional -- only when this version has step-specific upgrade behavior
upgrade:
  migrations:
    pre:
      - migrations/pre-deploy.yaml
    post:
      - migrations/post-deploy.yaml
  configMigrations:
    oldKey: new.key

Fields:

Field	Type	Required	Description
`version`	string	yes	Precise version string (e.g. `5.118.1-2`)
`requires`	list	no	Dependencies with optional aliases
`defaultConfig`	map	yes	Configuration schema merged into instance config
`defaultSecrets`	list	no	App's own secrets
`requiredSecrets`	list	no	Secrets needed from dependencies
`deploy`	object	no	Deployment behavior (CRDs, phases, etc.)
`scripts`	list	no	Operational scripts
`upgrade.migrations.pre`	list	no	K8s Job YAMLs to run before deploying this version
`upgrade.migrations.post`	list	no	K8s Job YAMLs to run after deploying this version
`upgrade.configMigrations`	map	no	Old config key -> new config key renames

Note the field split: Identity and routing live in app.yaml. Installation details live here. The version manifest never contains name, is, description, icon, category, or upgrade.from routing rules.

Versioning convention

Wild Cloud uses a two-part version scheme: <upstream>-<revision>.

Upstream tracks the third-party software version: 5.118.1, v4.0.18, 1.135.3
Revision tracks Wild Cloud packaging changes: -1, -2, etc.

A version without a revision suffix (e.g., 5.118.1) is the initial packaging. Revision -1 is the first packaging fix that doesn't change upstream software.

This is the same convention Debian uses for its source packages.

Upgrade system

The 90% case: no routing rules needed

Most apps can upgrade from any version to any other directly. The app has no upgrade block in app.yaml. When the API detects a version mismatch between installed and latest, it produces a single-step plan: install the latest version over the current one.

The 10% case: routing rules in `app.yaml`

Apps with breaking changes between versions need routing rules. These live in app.yaml, centralized for all versions. The rules are evaluated iteratively:

Compare installed version against upgrade.from rules (first match wins)
If the matching rule has via, step to that waypoint
Re-evaluate the same rules with the waypoint's version as the new "installed" version
Repeat until reaching latest or hitting a block

This is iterative re-evaluation, not recursive descent into waypoint manifests. The routing logic lives in one place (app.yaml), not scattered across version manifests.

Rule evaluation

Rules are evaluated in order. First match wins. This means more-specific rules must come before less-specific ones:

upgrade:
  from:
    - version: ">=3.0.0"       # Already past the breaking change, direct
    - version: ">=2.0.0"
      via: "2"                  # Must step through 2.x waypoint first
    - version: "<2.0.0"
      blocked: true             # Too old, no supported path

An installed version of 2.5.0 matches >=2.0.0, steps to waypoint 2/ (which contains, say, version 2.8.0). Re-evaluation with 2.8.0 matches >=2.0.0 again... but this time the waypoint IS the current version, so the visited-set check catches the cycle.

To avoid this, ensure a rule exists that matches post-waypoint versions and routes them directly:

upgrade:
  from:
    - version: ">=2.5.0"       # Post-waypoint versions go direct
    - version: ">=2.0.0"
      via: "2"                  # Pre-waypoint versions step through
    - version: "<2.0.0"
      blocked: true

After stepping to waypoint 2/ (version 2.8.0), re-evaluation matches >=2.5.0 (no via), which means direct upgrade to latest.

Version constraints

Supported operators: >=, >, <=, <, =, and the special >0 (matches any version). Constraints compare major.minor.patch numerically. The packaging revision is ignored in constraint matching so >=5.118.0 matches 5.118.1-2.

Waypoints

A waypoint is a version slot that exists solely as a stepping stone in an upgrade path. It contains a complete app package (manifest, kustomize, k8s resources) that can be installed and run.

Waypoints are created when an upstream app has breaking changes that require sequential upgrades. For example, Discourse requires stepping through each major version. A database schema migration might require running version N before jumping to version N+2.

Waypoint directories are referenced by slot name in via fields:

# app.yaml
latest: "3"
upgrade:
  from:
    - version: ">=2.5.0"
    - version: ">=2.0.0"
      via: "2"

myapp/
+-- app.yaml
+-- versions/
    +-- 3/              # Latest (version: 3.0.0)
    +-- 2/              # Waypoint (version: 2.8.0)

Per-version migrations

Migration jobs (database schema changes, data transformations) are version- specific, not app-level. They live in the version's directory and are referenced from that version's manifest.yaml:

# versions/3/manifest.yaml
version: 3.0.0
upgrade:
  migrations:
    pre:
      - migrations/pre-deploy.yaml
    post:
      - migrations/post-deploy.yaml

versions/3/
+-- manifest.yaml
+-- migrations/
|   +-- pre-deploy.yaml     # K8s Job: runs before deploying 3.0.0
|   +-- post-deploy.yaml    # K8s Job: runs after deploying 3.0.0
+-- kustomization.yaml
+-- deployment.yaml
+-- ...

Migration jobs must be idempotent (safe to re-run on retry). Use CREATE IF NOT EXISTS, ALTER TABLE IF NOT EXISTS, etc.

Pre-migrations run before deploying the new version's manifests. Use for schema changes the new code depends on.

Post-migrations run after deploying. Use for data backfills or cleanup the old code didn't need.

Config migrations

When a version renames config keys, configMigrations in the version manifest tells the system to rename them automatically in the instance's config.yaml before recompiling templates:

# versions/2/manifest.yaml
version: 2.0.0
upgrade:
  configMigrations:
    dbHost: db.host
    dbPort: db.port

API path resolution

The API resolves app paths through resolveAppDir():

Read app.yaml from the app root
Use latest (or a specific requested slot) to find versions/{slot}/
Verify manifest.yaml exists in that directory
Return the directory path and parsed AppMeta

For old-style apps (no app.yaml, manifest.yaml at root), the function falls back to the legacy path. This provides backward compatibility during migration.

When installing, applyMeta() merges identity fields from app.yaml onto the version manifest:

app.yaml          +  versions/5/manifest.yaml  =  installed manifest.yaml
(name, is, desc,     (version, requires,          (complete picture)
 icon, category)      defaultConfig, ...)

The installed manifest in the instance data directory always contains all fields because the instance needs the complete picture -- it doesn't have access to the Wild Directory's app.yaml at runtime.

Drift detection

The API detects when an installed app's version differs from the Wild Directory's latest. For new-style apps:

Read app.yaml to get the latest slot name
Read versions/{latest}/manifest.yaml to get the actual version string
Compare against the installed manifest's version

If they differ, the app is marked as having source drift with the available version noted. The upgrade planner then computes whether an upgrade path exists and how many steps it requires.

Maintainer workflows

Simple version bump

The app upstream releases a new version with no breaking changes. Edit files in-place:

# 1. Update files inside the existing slot
vi wild-directory/ghost/versions/5/manifest.yaml   # bump version: 5.119.0
vi wild-directory/ghost/versions/5/deployment.yaml  # update image tag

# 2. app.yaml doesn't change -- latest still points to "5"

# 3. Test
wild app add ghost && wild app deploy ghost

No directory created, renamed, or deleted. No app.yaml change.

Packaging fix

A Wild Cloud template fix, no upstream change:

# 1. Bump packaging revision in manifest
vi wild-directory/ghost/versions/5/manifest.yaml   # version: 5.118.1-3

# 2. Fix whatever needs fixing
vi wild-directory/ghost/versions/5/deployment.yaml

# 3. app.yaml doesn't change

Breaking upstream version (new waypoint)

The app upstream releases a major version with schema changes:

# 1. Current slot becomes a waypoint -- leave it in place
#    versions/2/ stays, containing version 2.8.0

# 2. Create the new slot
mkdir -p wild-directory/myapp/versions/3
# ... add manifest.yaml, kustomization.yaml, *.yaml for 3.0.0 ...

# 3. Update app.yaml

# app.yaml
name: myapp
latest: "3"
upgrade:
  from:
    - version: ">=2.5.0"         # Post-waypoint, direct
    - version: ">=2.0.0"
      via: "2"                    # Step through waypoint
    - version: "<2.0.0"
      blocked: true

Adding a new app

mkdir -p wild-directory/newapp/versions/1

# Create app.yaml
cat > wild-directory/newapp/app.yaml <<EOF
name: newapp
is: newapp
description: My new application
latest: "1"
EOF

# Create version manifest + k8s resources in versions/1/
# ... manifest.yaml, kustomization.yaml, deployment.yaml, etc.

Examples

Ghost (simple app, one slot)

ghost/
+-- app.yaml
|     name: ghost
|     is: ghost
|     description: Ghost is a powerful app for...
|     icon: https://...
|     latest: "5"
+-- versions/
    +-- 5/
        +-- manifest.yaml    # version: 5.118.1-2
        +-- kustomization.yaml
        +-- deployment.yaml
        +-- service.yaml
        +-- ingress.yaml
        +-- namespace.yaml
        +-- pvc.yaml
        +-- db-init-job.yaml

Upgrading from 5.100.0 to 5.118.1-2: single step, no routing rules needed.

e2e-test-app (two slots, waypoint)

e2e-test-app/
+-- app.yaml
|     name: e2e-test-app
|     is: e2e-test-app
|     description: End-to-end test application...
|     latest: "2"
|     upgrade:
|       from:
|         - version: ">=1.0.0"
|           via: "1"
|         - version: "<1.0.0"
|           blocked: true
|           notes: "Versions before 1.0.0 are not supported"
|       preUpgrade:
|         backup: recommended
+-- versions/
    +-- 2/
    |   +-- manifest.yaml    # version: 2.0.0
    |   +-- kustomization.yaml
    |   +-- ...
    +-- 1/
        +-- manifest.yaml    # version: 1.0.0-1
        +-- kustomization.yaml
        +-- ...

Upgrading from 0.5.0: blocked ("Versions before 1.0.0 are not supported"). Upgrading from 1.2.0: two steps -- 1.2.0 -> 1.0.0-1 (waypoint) -> 2.0.0.

SMTP (infrastructure service, one slot)

smtp/
+-- app.yaml
|     name: smtp
|     is: smtp
|     description: SMTP relay service...
|     category: infrastructure
|     latest: "1"
+-- versions/
    +-- 1/
        +-- manifest.yaml    # version: 1.0.0

Complex app with migrations

discourse/
+-- app.yaml
|     name: discourse
|     is: discourse
|     description: Discourse forum...
|     latest: "3"
|     upgrade:
|       from:
|         - version: ">=2.5.0"
|         - version: ">=2.0.0"
|           via: "2"
|         - version: "<2.0.0"
|           blocked: true
|           notes: "See upstream migration guide"
|       preUpgrade:
|         backup: required
+-- versions/
    +-- 3/
    |   +-- manifest.yaml    # version: 3.6.0
    |   +-- migrations/
    |   |   +-- pre-deploy.yaml
    |   +-- kustomization.yaml
    |   +-- ...
    +-- 2/
        +-- manifest.yaml    # version: 2.8.0
        +-- kustomization.yaml
        +-- ...

Implementation notes

Go types

// app.yaml
type AppMeta struct {
    Name        string         `yaml:"name"`
    Is          string         `yaml:"is,omitempty"`
    Description string         `yaml:"description"`
    Icon        string         `yaml:"icon,omitempty"`
    Category    string         `yaml:"category,omitempty"`
    Latest      string         `yaml:"latest"`           // slot name
    Upgrade     *UpgradeConfig `yaml:"upgrade,omitempty"`
}

// version manifest.yaml (version-specific fields only)
type AppManifest struct {
    Version         string                 `yaml:"version"`  // precise version string
    Requires        []AppDependency        `yaml:"requires,omitempty"`
    DefaultConfig   map[string]interface{} `yaml:"defaultConfig,omitempty"`
    DefaultSecrets  []SecretDefinition     `yaml:"defaultSecrets,omitempty"`
    RequiredSecrets []string               `yaml:"requiredSecrets,omitempty"`
    Deploy          *DeployConfig          `yaml:"deploy,omitempty"`
    Scripts         []Script               `yaml:"scripts,omitempty"`
    Upgrade         *UpgradeConfig         `yaml:"upgrade,omitempty"` // migrations only
    // Identity fields (Name, Is, Description, etc.) populated by applyMeta()
}

Backward compatibility

The API supports both new-style (app.yaml + versions/) and old-style (manifest.yaml at root, .versions/ for waypoints). resolveAppDir() checks for app.yaml first and falls back to the old layout. ComputeUpgradePlan() similarly checks for app.yaml before falling back to recursive manifest-based routing.

Old-style support exists for migration purposes. New apps should always use the new-style layout.

Source URI in installed manifests

The source field in installed manifests points to the app root (file:///wild-directory/ghost), not the version directory. Fetch() and Update() resolve through resolveAppDir() from the root, which reads app.yaml to find the current latest slot.

Design rationale

Why decouple directory name from version string?

Coupling them creates unnecessary churn. Every packaging fix requires creating a new directory and deleting the old one (5.118.1/ -> 5.118.1-1/). Every minor upstream release does the same. This churn has no benefit -- the files inside are what matter, not the directory name.

Decoupling follows the source package pattern used by every major package system. The directory is a stable container. The version is metadata inside it.

Why centralize routing rules in `app.yaml`?

Scattering upgrade.from rules across version manifests means each waypoint must know how to route TO itself. Adding a new waypoint requires editing multiple manifests. Reading the upgrade path requires opening every version manifest in the chain.

Centralizing in app.yaml means one file describes the complete routing topology. The upgrade planner reads one file and iteratively applies rules. Adding a new waypoint means editing one file.

Why iterative re-evaluation instead of recursive descent?

Recursive descent reads the target manifest, finds a via, recurses into the waypoint, which has its own via, and so on. This scatters routing logic across files and makes the path hard to reason about.

Iterative re-evaluation reads app.yaml once, applies rules, steps to a waypoint, then applies the SAME rules again with the new version. The routing table is evaluated like firewall rules -- same table, different input. This is simpler to implement, debug, and maintain.

Why not use a flat file for all versions?

A single YAML file listing all versions and their configs would centralize everything but would grow unwieldy for apps with many k8s resource files. Each version slot needs its own kustomization.yaml, deployment specs, service definitions, etc. Directories are the natural unit.

22 KiB Raw Permalink Blame History