OpenClaw EnterpriseDOCSGitHub

Recover a partial production upgrade

Resume from the original checkout and private evidence directory. Aliases share its lock; changed directory identity refuses live access.

For runtime upgrades, preserve <evidence-directory>.dispatch-state separately from evidence backups. Missing or invalid journals stop resume before live access. After possible restoration, reconcile OCC history and audit records.

--preflight-only leaves writers serving without a freeze. It cannot resume; reconcile its resources before deploying with fresh evidence and checks.

Keep one operator and the freeze for unfinished deployments. Before prepared, writers and Helm are unchanged: reconcile, then use fresh evidence. Then reconcile processes, release, migration, rollout and dispatch; add --resume with unchanged arguments, protected paths, kubeconfig contents, OCC URL, scripts, flow and chart. Keep candidates semantically unchanged. Resume rechecks images, nodes and live state against baseline or candidate.

For an immutable reference switch, retain both Secrets' recorded UIDs and data; resume refuses replacements.

Resume uses immutable controller-compatibility-results.json, when selected. Changing or removing this prepared copy refuses recovery.

After verified completion, clean up without repeating rollout or Doctor. Any completed entry refuses resume before live access; preserve it. New upgrades require fresh evidence. Reconcile live state after output or marker failures: neither an absent marker nor prepared proves unfinished work. Older runs may lack markers.

Correct unusable comparison inputs

Cannot compare upgrade YAML requires readable YAML mappings (including candidates) without empty documents and writable evidence. Correct inputs before retrying; this does not establish drift.

Clean up preflight resources

Follow preflight cleanup to remove recorded Jobs and dependencies after an interrupted preflight. Preserve the original evidence and .upgrade-lock; first confirm the helper, children and accepted remote operations have settled. Successful cleanup does not authorize resume, unlock the upgrade or establish database state.

Resume an unfinished prepared upgrade

Failed application identity or migration preflight prevents further shutdown and Helm submission; writers may already be stopped. Inspect the current upgrade-preflight-resume.*/preflight-{api,worker,migration}.log and corresponding preflight-*-logs.err. Reconcile the application connection or migration failure and unknown creates; retain the original evidence, lock and dispatch journal for --resume. Historical migration-preflight-resume.* attempts remain valid for --cleanup-preflight using their original, hash-matching helper checkout.

With --verify-agent, inspect agent-continuity.json and agent-continuity-after*.json. The former binds the original revision and workload roles; never edit it. Restore that revision's readiness, then resume with the same selection. Successful Helm work is not repeated. A replacement revision cannot pass; if redeployment is necessary, preserve this incomplete evidence and verify that separate recovery. The helper neither redeploys Agents nor rolls back the control plane.

Read native-command-failures.tsv for failed scale, Helm or rollout commands: rows retain the operation, component (api, worker, release) and exit code. Client failures do not establish whether requests took effect; other failures may leave no row. Reconcile live state before resuming.

Both Deployments are validated before stopping either. Later UID/version/replica conflicts can leave the API stopped. Reconcile current-<component>-deployment.json and quiescence-<component>-patch.json against live state before resuming.

Helm failure or disconnection does not prove migration rollback. Inspect live Helm status/history, initialization Job/Pod outcomes and the database; current-helm-status.json may be stale. Keep the freeze. Reconcile the stopped Helm client before retry or recovery. Resolve pending-* releases with the release owner. Revision checks cannot fence concurrent clients.

Ordinary resume skips Helm at the expected deployed revision with candidate values/Installation. It can retry the recorded prior or failed expected revision after the checks below. Invalid intents and other revisions refuse. For completed initialization, see same-candidate Helm recovery.

Outside this pair, preserve intent and evidence. The release owner must verify the recovered manifest, values, Installation, required hook outcomes and migration history, then finish verification separately.

Never start old writers against a migrated database. For a retry within the recorded pair, wait until the initialization Job and Pods are terminal. Run the candidate controller image with node scripts/migrate-production.mjs --check against the retained database, with its migrator credential and required CA. The Kubernetes maintenance procedure supplies these: set CONTROLLER_IMAGE and args ["scripts/migrate-production.mjs","--check"]. Retain exit-zero migration.checked output. Separate migrations require --require-current. Attest the checked outcome with --resume --migration-history-checked. The helper returns OCC to zero replicas and records the next expected revision before retrying. Prefer a reviewed forward fix; neither Helm rollback nor the helper reverses committed migrations.

After an unknown Agent dispatch, requests in the same batch may still finish. Before resuming, inspect the Agent's revision history, status and audit records. A read showing the earlier revision cannot establish rejection. If you cannot confirm an accepted revision, or the journal is incomplete, preserve all records and contact the release owner. The helper cannot resume those attempts.

If you can attribute a revision to the interrupted request, set OCC_NAMESPACE, OCC_AGENT and REVISION_ID to the confirmed IDs. Set DISPATCH_PREFIX to the private evidence path dispatch/<namespace>--<agent>. Keep the intent, errors and partial responses.

bash
umask 077if test -n "$REVISION_ID" && test ! -e "$DISPATCH_PREFIX.json" && test ! -L "$DISPATCH_PREFIX.json" &&  occ --output json --namespace "$OCC_NAMESPACE" agent deployment-status "$OCC_AGENT" "$REVISION_ID" > "$DISPATCH_PREFIX.confirmed-status.json" &&  jq -e --arg namespace "$OCC_NAMESPACE" --arg agent "$OCC_AGENT" --arg revision "$REVISION_ID" \    '.namespaceId == $namespace and .agentId == $agent and .deploymentId == $revision' "$DISPATCH_PREFIX.confirmed-status.json" &&  response=$(mktemp "$DISPATCH_PREFIX.XXXXXXXX.tmp") &&  jq -n --arg id "$REVISION_ID" '{id: $id}' > "$response"; then  mv "$response" "$DISPATCH_PREFIX.json"else  printf '%s\n' 'No response recorded. Inspect the receipt and deployment status before resuming.' >&2fi

For a failed revision, inspect the failure and correct the drafts before an authorized manual deploy. Keep the earlier receipt and new response separately. Verify in revision history that the replacement follows the failed revision selected in the journal. Write {"id":"<replacement ID>","replacesDeploymentId":"<current journaled ID>"} to a private temporary file beside the receipt, then rename it to $DISPATCH_PREFIX.json. Resume checks both deployments before recording the replacement. A restored receipt for a superseded revision stays in place for inspection. Resume uses the journal's current revision.

If the candidate API and worker refuse to start because of split-layout Gateway storage, --resume cannot succeed. Keep both namespaces and their storage, and return to the previous controller image after the check below.

Before selecting an older image, verify it can read candidate state. Restore the retained candidate database if needed. Preserve Agents, versions, PVCs and the bootstrap volume. After return, check the original conversation.

For stalled ReplicaSet creation, check namespace object capacity.

Correct a refused Agent deployment

The startup preflight refuses some such drafts before any writer stops. If OCC still refuses an Agent's deployment with a 4xx answer, such as HTTP 400 ... must use reviewed task actions, it saved no new version. The same applies when occ sends no request because it cannot establish the active revision's Harness overrides. The command lists each such Agent with the message, keeps it in dispatch/*.rejected, and stops. Correct the cause, for example with occ configuration update, and repeat the command with --resume. The resume deploys each refused Agent once; do not deploy it yourself. A 408 or 5xx answer stays an unknown dispatch. When OCC rejects the command's service key, the command lists the Agents it did not deploy; get a new key, point OCC_SERVICE_KEY_FILE at it, and resume.

Correct a Doctor failure

Doctor shares the gateway container's memory limit. At the default heap cap, the command waits for two consecutive readings with at least 640 MiB free or one with at least 1024 MiB free. Initially unavailable cgroup data produces a warning and Doctor runs. Once insufficient memory is observed, missing or failed reads cannot admit Doctor and reset consecutive evidence. Fresh sufficient readings must arrive within that Doctor's --timeout-seconds. Wait for memory use to settle, then resume; consider a larger gateway memory limit if it stays low. These readings do not reserve memory or guarantee that the gateway avoids an OOM kill.

A Doctor failure reports its command exit separately from gateway restart history. It also names each gateway or agent Pod evicted, deleted, replaced, restarted, or not Ready since then: an eviction kills Doctor (exit 137) without a restart. A higher restart count on the same Pod with last termination OOMKilled means an OOM restart occurred after the Ready baseline; it does not prove Doctor caused it. The restart history is unknown when the gateway Pod was replaced, its container status is missing, or its restart count went down. Wait until the gateway is Ready again, then resume; the resume runs Doctor again without new deployments.

did not stay on the same Ready Pod without restarts through OpenClaw Doctor means that every Doctor passed, but afterward a named Agent's gateway Pod, or a dedicated Agent's agent Pod, was evicted, deleted, replaced, restarted, or not Ready. Doctors run one at a time, but node memory pressure from other Pods can still evict them. Inspect the node's events, wait until that Agent's Pods are Ready, then resume. The resume runs Doctor in the current gateway without new deployments.

OpenClaw Doctor ran out of its 384 MiB heap means Doctor reached its heap cap; check the separate gateway restart observation. Repeat the command with --resume and --doctor-heap-mib 768, doubling again if needed. Each extra MiB of heap needs the same free gateway memory before Doctor starts, so raise the gateway memory limit first if the command then waits for memory. The command refuses a cap too large for its 64-bit memory arithmetic before it changes anything.

Correct an invalid Installation name

The stored Installation name must follow the API Name rule: 1 to 200 characters, with no leading or trailing whitespace, control characters, or line or paragraph separators. The upgrade command's startup preflight reads the name through OCC and checks it with the selected controller image's rule before any writer stops. A failure prints the rule and INSTALLATION_NAME_INVALID, attempts best-effort deletion of preflight resources, and stops; the old release keeps serving. Follow safe cleanup before retrying. To check before the maintenance window, run this from the candidate checkout:

bash
occ --output json installation get | node --input-type=module -e 'import { isName, NAME_RULE } from "./packages/contracts/src/api/plain-text.ts";let s = ""; for await (const c of process.stdin) s += c;if (!isName(JSON.parse(s).name)) { console.error(NAME_RULE); process.exit(1); }'

No API renames an Installation, so correct occ.installation.name with the dedicated migrator credential (OCC_MIGRATION_DATABASE_URL as in the upgrade baseline). Shell-quote the name; psql's :'name' quotes it for SQL:

bash
psql "$OCC_MIGRATION_DATABASE_URL" -v ON_ERROR_STOP=1 -v name='<intended name>' <<'SQL'UPDATE occ.installation SET name = :'name';SQL

psql prints UPDATE 1. The running API keeps the name it read at startup, so restart it before you check again:

bash
kubectl --kubeconfig /secure/occ/kubeconfig --context '<reviewed-context>' \  --namespace openclaw-system rollout restart deployment/openclaw-enterprise-apikubectl --kubeconfig /secure/occ/kubeconfig --context '<reviewed-context>' \  --namespace openclaw-system rollout status deployment/openclaw-enterprise-api

The database accepts some names the rule refuses, so run the upgrade command again with a new evidence directory; its preflight checks the name again. If the candidate API and worker log INSTALLATION_NAME_INVALID after the helper stopped OCC (the name changed after the preflight), Helm's --wait has marked the candidate release failed: rename as above, follow the Helm failure steps above, then repeat the command with --resume --migration-history-checked.

Correct values the candidate refuses

Before maintenance, use the candidate checkout with live release, namespace and values:

bash
helm template --is-upgrade oce deploy/helm/openclaw-enterprise \  --namespace openclaw-system \  -f /secure/occ/values.yaml > /dev/null

The helper renders and runs its startup preflight, refusing these values while old writers serve. helm template cannot check every runtime value in the mounted Installation; Compose has no chart. On a stock-chart upgrade, the initialization Job runs installation-check, then the migration init container, then bootstrap, before API and worker rollout. The check can refuse before migration starts; later failures can follow a committed migration. Inspect the initialization Job and its Pod's container statuses and logs, and the retained database's migration history, before choosing recovery. Keep old controllers off a database that may have migrated.

The helper refuses these corrections in candidate files. Apply live values or Installation YAML with the installed release's checkout per Apply other Installation changes, keeping baseline files matched to live state.

Refused value Correction
auth.google.allowedDomains set while Google is disabled Remove the list.
"" or {} for those allowlists, or {} for a provider's egressCidrs Write [] or remove the key.
An IPv4-mapped api.trustedProxy.cidrs entry with a prefix from 97 to 128, such as ::ffff:a00:0/104 Write the IPv4 CIDR with the prefix minus 96: 10.0.0.0/8. A Compose install's API refuses the same entry in OCC_AUTH_TRUSTED_PROXY_CIDRS at startup.
An IPv6 api.trustedProxy.cidrs entry that contains ::ffff:0:0/96, such as ::ffff:0:0/96, ::ffff:a00:0/64 or ::/64 Such an entry trusts forwarded headers from every IPv4 peer. List your proxies' own CIDRs instead.
A Unicode auth.baseUrl host, such as https://bücher.example.com, with a punycode Agent suffix Write the host in punycode: https://xn--bcher-kva.example.com.
An auth.baseUrl with a /. or /%2e path, a short loopback such as http://127.1, or U+200B (zero-width space) Remove the path and invisible characters; write 127.0.0.1.
An auth.baseUrl with a path or query, such as https://example.com/occ Serve OCC at the root of its own origin and set that origin, such as https://occ.example.com. Point browsers, CLI and API clients, and your proxy at the new origin.
A presets.files Preset name with leading or trailing Unicode spaces (such as U+00A0), U+2028, U+2029 or C1 control characters Correct the name in the file, or remove the entry. Startup adds a Preset with the corrected name; delete any saved copy with the old name you no longer need.
A Backend id with a C1 control character (U+0080 to U+009F) Rename the Backend. No Agent can store such an ID, but ServiceAccounts created through that Backend keep it in a binding that cannot change: delete them before the rename and recreate them after the upgrade.
With requireImmutableDigest: true, an upper-case gateway or agent digest in the Installation's drivers.compute.configuration.images, such as @sha256:ABC… or @SHA256: Write sha256 and the hex digits in lower case; the digest is the same.

Roll back across human sign-in

Helm rollback does not reverse migration 0037. Never run both controllers together.

If the target supports the configured provider and enforces session bindings, keep that configuration. Rehearse the return. Verify a fresh sign-in reaches the original account and a resource it could read before.

For a target that issues unbound sessions:

Search documentation