OpenClaw EnterpriseDOCSGitHub

Upgrade images on a persistent local k3d installation

Upgrade an existing Helm-installed OpenClaw Control Plane (OCC) and Kubernetes Compute fleet in the same k3d cluster. This procedure retains PostgreSQL, Namespaces, Agents, version history, Secrets, and Agent workspace and gateway PersistentVolumeClaims (PVCs). Take verified backups first: retaining a volume is not a backup, and migrations or runtime changes can make rollback unsafe. Complete the upgrade checklist before using this procedure.

This path requires the production Helm chart, Kubernetes Compute, and a trusted HTTPS OCC endpoint. The production image upgrade owns the contract. Use the selected publication's source and the live installation's inputs. A cluster from local operations is suitable only after the production installation sequence is complete.

./scripts/k3d has no upgrade command. Its interactive demo is a disposable test harness, not a persistent Helm installation: it uses loopback HTTP and creates its own topology. The production upgrade script cannot upgrade that demo. Ctrl-C, reset and down remove or replace its resources; never use them to upgrade or preserve one. See the demo lifecycle.

Prepare the existing installation

From a secure operator shell, replace placeholders with the existing paths and names. The evidence parent must exist; the final directory must be new.

bash
set -o pipefailumask 077export KUBECONFIG_FILE='/secure/occ/kubeconfig'export CONTEXT='<existing-k3d-context>'export NAMESPACE='<existing-helm-namespace>'export RELEASE='<existing-helm-release>'export VALUES='/secure/occ/values.yaml'export INSTALLATION='/secure/occ/installation.yaml'export OCC_URL='https://<trusted-occ-host>'export OCC_SERVICE_KEY_FILE='/secure/occ/operator-service-key.json'export OCC_CA_BUNDLE='/secure/occ/occ-ca.pem' # omit if the system trust store is sufficientexport UPGRADE_EVIDENCE="/secure/occ/upgrades/$(date -u +%Y%m%dT%H%M%SZ)"

For missing YAML, recover to unused, protected VALUES/INSTALLATION paths. Resolve errors before retrying. Validate Installation separately. Compare both files with operator records; never use example templates.

bash
(  set +x  set -o pipefail  umask 077  fail() { echo "$1" >&2; exit 1; }  mapping() {    [ -s "$1" ] && yq ea --header-preprocess=false -e '[.] | select(length == 1) | .[0] | tag == "!!map"' "$1" &>/dev/null  }  setting() {    yq -o=json "$1"' | select(tag == "!!str" or tag == "!!null")' "$values_stage" 2>/dev/null  }  [ ! -e "$VALUES" ] && [ ! -L "$VALUES" ] &&    [ ! -e "$INSTALLATION" ] && [ ! -L "$INSTALLATION" ] ||    fail "Choose unused output paths."  values_stage=$(mktemp "${VALUES}.XXXXXX") || fail "Values staging failed."  installation_stage=''  trap 'rm -f -- "$values_stage"; [ -z "$installation_stage" ] || rm -f -- "$installation_stage"' EXIT  installation_stage=$(mktemp "${INSTALLATION}.XXXXXX") || fail "Installation staging failed."   helm --kubeconfig "$KUBECONFIG_FILE" --kube-context "$CONTEXT" \    --namespace "$NAMESPACE" get values "$RELEASE" --output yaml > "$values_stage" 2>/dev/null ||    fail "Helm read failed; check release, context, permissions."  mapping "$values_stage" || fail "Helm YAML: require one mapping."  INSTALLATION_SECRET=$(setting '.installation.secretName' |    jq -er 'if . == null then "occ-installation-startup" else . end |      select(length <= 253 and      test("\\A[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*\\z"))' 2>/dev/null) &&    INSTALLATION_KEY=$(setting '.installation.key' |      jq -er 'if . == null then "installation.yaml" else . end |        select(. != "." and (startswith("..") | not)) |        select(length > 0 and length <= 253 and        test("\\A[-._a-zA-Z0-9]+\\z"))' 2>/dev/null) &&    test -n "$INSTALLATION_SECRET" && test -n "$INSTALLATION_KEY" ||    fail "Invalid Secret name/key; correct Helm values."   secret_json=$(kubectl --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" \    --namespace "$NAMESPACE" get secret "$INSTALLATION_SECRET" -o json 2>/dev/null) ||    fail "Secret read failed; check name, namespace, permissions."  installation_data=$(printf '%s' "$secret_json" |    jq -er --arg key "$INSTALLATION_KEY" '.data[$key] | select(type == "string" and length > 0)' 2>/dev/null) &&    [ -n "$installation_data" ] || fail "Missing/invalid Secret data; check configured key."  printf '%s' "$installation_data" |    python3 -c 'import base64,sys; sys.stdout.buffer.write(base64.b64decode(sys.stdin.buffer.read(), validate=True))' > "$installation_stage" 2>/dev/null ||    fail "Decode failed; correct Secret base64 data."  mapping "$installation_stage" || fail "Installation YAML: require one mapping."   ln -- "$values_stage" "$VALUES" || fail "Values publication failed."  ln -- "$installation_stage" "$INSTALLATION" ||    { rm -f -- "$VALUES"; fail "Installation publication failed."; })

Record the Installation ID, Agents, PVC UIDs and bound volumes:

bash
kubectl --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" get --raw=/readyzhelm --kubeconfig "$KUBECONFIG_FILE" --kube-context "$CONTEXT" \  --namespace "$NAMESPACE" status "$RELEASE"occ --output json installation get > /secure/occ/installation-before.jsonkubectl --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" \  get pvc --all-namespaces \  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,UID:.metadata.uid,VOLUME:.spec.volumeName' \  > /secure/occ/pvcs-before.txt

The first command returns ok, Helm reports the release, and OCC authenticates. For every upgrade, save occ --output json installation deployment-inventory before and after the change using the required inventory permissions. Keep inventories private. Check data and backups with the database and storage owners; object names alone do not prove retention.

Apply Installation changes

This procedure keeps the existing Installation and refuses every Installation change except the Plugin Driver selection. A release that changes launcher or profile Installation values, such as the Gateway and Harness memory sizes, does not change an existing installation. Diff internal/occdev/kubernetes.go, scripts/render-installation-profile.ts and deploy/examples/production/installation.yaml between the deployed and candidate source (pre-migration releases: deployed .mjs implementation versus candidate .ts). Apply adopted changes as in apply other Installation changes, with this page's INSTALLATION, VALUES, kubeconfig, context, release and namespace, including for an installation created by dev-up. Do not apply with the launcher state's helm-values.json after this page has upgraded the installation: it still names the bring-up images and lacks the upgraded values, such as the selected images and checksum, so Helm would roll the controller back. Gateway and Harness resources apply when an Agent is next deployed.

It also never adds fields that scripts/dev-up writes only at bring-up. An installation created by dev-up before network.pluginStatusProxySourceCidrs existed still lacks it after an upgrade: plugin status and diagnostics stay unavailable, and each dedicated Codex first deploy starts its Gateway twice. Check the live Installation:

bash
yq -er '.drivers.compute.configuration.network.pluginStatusProxySourceCidrs' "$INSTALLATION"

If it is missing, find the address the API server proxies from: the k3d server node's cni0 bridge. Use the node's Pod CIDR plus 2 (10.42.0.2 for 10.42.0.0/24), and podman exec for a Podman-backed cluster:

bash
export K3D_SERVER='<existing-k3d-server-0-container>'kubectl --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" \  get node "$K3D_SERVER" -o jsonpath='{.spec.podCIDR}{"\n"}'docker exec "$K3D_SERVER" ip route get 10.42.0.2

The route must name dev cni0; another device means no Pod runs on the node yet. Add its src address as a single /32 entry, for example 10.42.0.1/32, under drivers.compute.configuration.network.pluginStatusProxySourceCidrs in INSTALLATION and apply it as above. That leaves INSTALLATION and VALUES equal to the live state, so the script accepts them. Agents deployed afterward get the API-proxy rule. Recreating with occ dev down and scripts/dev-up also adds the field, but discards the database, Agents, and volumes.

Select and make the published images available

In GitHub Actions, inspect Enterprise Containers runs on main in newest first order. Select the newest successful run that actually published both images (publish: true), not a no-push preparation or a partially failed run. Download its container-publication-<run-id>-<attempt> receipt and compare both entries' sourceSha, destination, and digest with the run input, job summary, and successful exact-source CI. If the receipt has expired, stop until you can recover verifiable publication evidence. See container publication for the evidence and partial publication recovery. Use one verified source SHA and each full destination@digest; do not use latest, a source tag, or a digest copied from an older example.

bash
export RELEASE_SOURCE_SHA='<full-40-character-source-commit>'export CONTROLLER_IMAGE='ghcr.io/openclaw/<controller-package>@sha256:<64-hex-digest>'export RUNTIME_IMAGE='ghcr.io/openclaw/<runtime-package>@sha256:<64-hex-digest>'export REPOSITORY_ENABLED="$(yq -er '.repositoryCredentials.enabled // false' "$VALUES")"BROKER_ARGS=()RUNTIME_REPOSITORY_ARGS=()if [ "$REPOSITORY_ENABLED" = true ]; then  # Select the reviewed broker artifact compatible with RELEASE_SOURCE_SHA.  export BROKER_IMAGE='<registry>/repository-credentials@sha256:<64-hex-digest>'  export CURRENT_CONTROLLER_IMAGE="$(yq -er '.images.controller' "$VALUES")"  BROKER_ARGS=(--broker-image "$BROKER_IMAGE")  RUNTIME_REPOSITORY_ARGS=(    --controller-image "$CURRENT_CONTROLLER_IMAGE"    --broker-image "$BROKER_IMAGE"  )fi

The Enterprise Containers receipt covers the controller and runtime images. A repository-enabled release also needs separate provenance and review evidence for BROKER_IMAGE; do not infer it from the runtime digest. The runtime-only command uses the live controller digest because restarting the worker also restarts its broker. A controller or combined release uses the selected candidate controller.

Public GHCR controller and runtime pulls do not require a registry login. If BROKER_IMAGE points at a private broker package, or if you selected a private mirror instead of public GHCR, authenticate the local registry client for that registry before checking digests. Verify the registry's raw index bytes for each receipt digest:

bash
IMAGES=("$CONTROLLER_IMAGE" "$RUNTIME_IMAGE")[ "$REPOSITORY_ENABLED" = false ] || IMAGES+=("$BROKER_IMAGE")for image in "${IMAGES[@]}"; do  expected="${image##*@sha256:}"  actual="$(skopeo inspect --raw "docker://$image" | python3 -c 'import hashlib,sys; print(hashlib.sha256(sys.stdin.buffer.read()).hexdigest())')"  [ "$actual" = "$expected" ] || { echo "Registry digest mismatch: $image" >&2; exit 1; }done

Public GHCR controller and runtime pulls need no node pull credentials. Configure approved pull credentials on every existing k3d node only for a private broker image or private mirror that may run OCC initialization, API, worker, gateway, or Agent Pods. Host docker login alone does not authenticate those nodes. Use the existing node credential mechanism or K3s private registry configuration; protect credential files and avoid putting tokens in commands, logs, or shell history. K3s reads its registry configuration at startup: coordinate any required node restart with the interruption window and preserve the existing cluster and volumes. Check each node can pull the exact references before upgrading. For Docker-backed k3d, repeat with each existing node container name; use podman exec for a Podman-backed cluster:

bash
export K3D_NODE='<existing-k3d-node-container>'docker exec "$K3D_NODE" crictl pull "$CONTROLLER_IMAGE"docker exec "$K3D_NODE" crictl pull "$RUNTIME_IMAGE"[ "$REPOSITORY_ENABLED" = false ] || docker exec "$K3D_NODE" crictl pull "$BROKER_IMAGE"

A successful pull by digest establishes node access to that immutable reference; resolve pull errors before continuing. Do not relabel a platform-specific local image with the published multi-platform index digest. Local image imports and registry index digests can differ.

Run the upgrade

Use a separate clean checkout at RELEASE_SOURCE_SHA, with its chart, upgrade script, and a compatible installed occ CLI. Confirm git rev-parse HEAD equals RELEASE_SOURCE_SHA. Stop if the upgrade script is missing. Do not overwrite a working checkout. Install Node.js, helm, kubectl, jq, yq v4, and Python 3. Follow engines.node in this checkout's package.json even without repository credentials. Use a fresh UPGRADE_EVIDENCE directory for each new release; retain it when resuming an interrupted release. The script changes the protected input files as well as the cluster; it saves their previous contents in that private evidence directory before cluster mutation.

Older dev-up installations set agentNativeAdmin.sharedCookieDomain, which the chart refuses; pass a copy without it:

bash
CANDIDATE_ARGS=()if [[ "$(yq '.agentNativeAdmin | has("sharedCookieDomain")' "$VALUES")" == true ]]; then  (umask 077; yq 'del(.agentNativeAdmin.sharedCookieDomain)' "$VALUES" > /secure/occ/candidate-values.yaml)  CANDIDATE_ARGS=(--candidate-values /secure/occ/candidate-values.yaml)fi

Choose exactly one invocation below. Both options together perform a combined release; omitting one leaves that image selection unchanged.

bash
# Controller only: no Agent revisions are requested.scripts/upgrade-production-images \  --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" \  --namespace "$NAMESPACE" --release "$RELEASE" \  --values "$VALUES" --installation "$INSTALLATION" \  --source-revision "$RELEASE_SOURCE_SHA" --evidence-dir "$UPGRADE_EVIDENCE" \  --controller-image "$CONTROLLER_IMAGE" "${BROKER_ARGS[@]}" "${CANDIDATE_ARGS[@]}" # Runtime only: use a fresh UPGRADE_EVIDENCE for a new release.scripts/upgrade-production-images \  --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" \  --namespace "$NAMESPACE" --release "$RELEASE" \  --values "$VALUES" --installation "$INSTALLATION" \  --source-revision "$RELEASE_SOURCE_SHA" --evidence-dir "$UPGRADE_EVIDENCE" \  --runtime-image "$RUNTIME_IMAGE" "${RUNTIME_REPOSITORY_ARGS[@]}" "${CANDIDATE_ARGS[@]}" # Combined: use a fresh UPGRADE_EVIDENCE for a new release.scripts/upgrade-production-images \  --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" \  --namespace "$NAMESPACE" --release "$RELEASE" \  --values "$VALUES" --installation "$INSTALLATION" \  --source-revision "$RELEASE_SOURCE_SHA" --evidence-dir "$UPGRADE_EVIDENCE" \  --controller-image "$CONTROLLER_IMAGE" --runtime-image "$RUNTIME_IMAGE" \  "${BROKER_ARGS[@]}" "${CANDIDATE_ARGS[@]}"

The helper stops the API and worker and waits for their Pods to terminate. Helm then runs initialization, including database migration with the migrator role, before rolling out OCC. Runtime-only also restarts OCC on its existing controller image to load the updated Installation. The script deploys all recorded running Agents concurrently; stopped and deleting Agents are left alone. See the production contract for version, readiness, and Doctor checks.

Verify and recover

Inspect the private evidence directory's Helm status, workload inventories, Installation records, and, for runtime upgrades, deployments.jsonl and status/. Confirm the selected controller digest in both API and worker Deployment templates and their successful rollouts. For a runtime change, confirm both image slots in the live Installation Secret, each running Agent's new selected version, and its ready gateway/Agent Pods on the selected digest. For controller-only, confirm existing versions and gateways are ready again; the new worker can restart their Pods once. Inspect the live images, rollouts, initialization Job, and OCC access:

bash
helm --kubeconfig "$KUBECONFIG_FILE" --kube-context "$CONTEXT" \  --namespace "$NAMESPACE" status "$RELEASE"kubectl --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" \  --namespace "$NAMESPACE" get jobsselected_controller=$(cat "$UPGRADE_EVIDENCE/selected-controller") || exit 1for component in api worker; do  kubectl --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" \    --namespace "$NAMESPACE" rollout status "deployment/openclaw-enterprise-$component" || exit 1  actual=$(kubectl --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" \    --namespace "$NAMESPACE" get "deployment/openclaw-enterprise-$component" -o json |    jq -er --arg component "$component" --arg repository "$REPOSITORY_ENABLED" '      .spec.template.spec |      (.containers[]?, (.initContainers[]? |        select($component == "worker" and $repository == "true" and .restartPolicy == "Always"))) |      select(.name == $component) | .image') ||    { echo "$component: missing or invalid controller image readback" >&2; exit 1; }  printf '%s: %s\n' "$component" "$actual"  [ "$actual" = "$selected_controller" ] ||    { echo "$component: controller image differs from upgrade evidence" >&2; exit 1; }doneocc --output json installation get > /secure/occ/installation-after.jsonkubectl --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" \  get pvc --all-namespaces \  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,UID:.metadata.uid,VOLUME:.spec.volumeName' \  > /secure/occ/pvcs-after.txt

Helm must be deployed, initialization successful, both rollouts complete, and both displayed images equal the selected controller image (or the previous one for runtime-only). To inspect the live runtime selection after a runtime change, recover the Installation Secret with the commands above into a separate private file and check drivers.compute.configuration.images.gateway and .agent with yq; both must equal RUNTIME_IMAGE. Compare the before and after Installation ID, Namespace and Agent inventory, and each PVC's UID and bound volume name. A different UID for the same namespace and claim name identifies a replacement claim. Verify representative PostgreSQL records, Secrets, workspace and gateway data, and a fresh model response. Pod image IDs may show a platform-specific digest rather than the published multi-platform index. Use workload verification and the upgrade verification for application checks. An empty fleet cannot prove a runtime starts.

If Helm or migration fails, inspect the initialization Job and rollouts, retain the evidence and backups, and prefer a forward fix. Helm rollback does not undo database migrations. If OCC succeeds but a runtime deployment fails, the fleet can be mixed: runtime rollout is not transactional. Inspect each dispatch, deployment status, and version before retrying; an uncertain request may have already created a version. Check compatibility before selecting an older image or restoring state, and follow partial-failure recovery. Do not delete the cluster, database volume, Namespaces, Agents, versions, Secrets, bootstrap volume, or Agent PVCs to force an upgrade or recovery.

Rehearse a compatible controller return

Use the populated controller return rehearsal to check a compatible release pair with a disposable installation and database.

Search documentation