Troubleshoot the platform

Use this page when local startup, the Kubernetes installation, the control plane, or several Agents are affected. For a problem with one Agent, start with Agent troubleshooting.

For pnpm and Helm setup issues, see Troubleshoot local k3d setup.

Local K3s cannot find the cpuset controller

If the k3d server logs report failed to find cpuset cgroup (v2), inspect /sys/fs/cgroup/cgroup.controllers inside that server. Docker running inside a containerized development host needs the outer host to delegate cpuset; a running Docker daemon does not prove delegation. On native rootless Podman, delegating cpuset is not enough: k3d still fails with mkdir /var/run/docker.sock: permission denied. Use the rootful setup requirements.

Use the host management service's documented delegation procedure. If delegation changes fail, check that service before concluding that an outer-host change is required. Restore any paused management service and verify its health before continuing; do not disable the K3s check. Confirm cpuset is available inside the k3d server and that K3s starts successfully.

Local K3s image lookup fails

The Compose control-plane profile without OpenShell resolves a K3s channel through update.k3s.io unless OCC_DEVELOPMENT_K3S_IMAGE selects an explicit image. If that lookup fails or times out, startup stops with k3d failed and k3d's error getting K3s version for channel line. Select an approved Kubernetes 1.35-or-newer image through the profile settings.

After a failed creation, wait for startup to exit, then run ./scripts/dev-down from the repository root with the same profile and state directory. Retry the Compose profile startup with the selected image and wait for the development stack to report ready.

Local K3s cannot load the legacy iptables nat table

Both local k3d profiles start their node with IPTABLES_MODE=legacy. A host whose Docker uses nftables never loads the legacy netfilter modules, and the node cannot load them itself. kube-proxy then exits with can't initialize iptables table 'nat': Table does not exist, K3s shuts down, and cluster creation fails when the startup timeout expires with k3d failed: exit status 1: ten minutes by default for the Kubernetes-only profile, five for the Compose profile.

When the container engine runs on the host's own Linux kernel (not Docker Desktop, a Podman machine, or a remote engine), occ dev up checks that kernel first. If it can tell the kernel has not loaded the modules, it stops before cluster creation with the host kernel has not loaded the legacy iptables modules (ip_tables, iptable_nat) that the local k3d node needs, followed by the modprobe command below. A kernel that ships no iptable_nat module stops with provides no legacy iptables nat table (iptable_nat) instead; see the end of this section. If it cannot tell, it prints Warning: could not confirm that the host kernel provides the legacy iptables nat table (iptable_nat) and continues, and the timeout above still applies. If creation then fails, the error repeats the modprobe command below unless k3d's last error names another cause.

In a second terminal, while that wait is still running, read the node log. <cluster> is the cluster name from the startup output.

docker logs 'k3d-<cluster>-server-0' 2>&1 | grep -i 'iptables table'

On the host, lsmod | grep '^iptable_nat' prints nothing while the module is unloaded. Load the legacy modules K3s uses and keep them across reboots:

sudo modprobe --all iptable_nat iptable_filter iptable_mangle br_netfiltersudo tee /etc/modules-load.d/oce-k3d-legacy-iptables.conf >/dev/null <<'EOF'iptable_natiptable_filteriptable_manglebr_netfilterEOF

A failed creation preserves the state directory and reports it for occ dev down. Wait until that startup exits, then run ./scripts/dev-down from the repository root with the same profile and state directory, and start the profile again. lsmod | grep -E '^iptable_(nat|filter)' lists both modules, and startup reaches OpenClaw Enterprise development stack is ready. A host whose kernel ships no iptable_nat module cannot run these profiles.

Local startup stalls on cert-manager

On some Linux hosts, especially Ubuntu with Docker 29, the k3d node cannot resolve container registries. Startup then waits on the cert-manager rollout while its pods stay in ContainerCreating. The launcher does not print those Pod events, and it prints the kubeconfig path only after startup succeeds.

In a second terminal, while that wait is still running, describe the cert-manager Pods. The context is k3d- plus the cluster name from the first startup line, Creating Kubernetes-only k3d cluster .... <state-directory> belongs to the process that started OCC, not to this second terminal. Use OCC_DEVELOPMENT_STATE_DIRECTORY when that process set it. Otherwise use that process's temporary directory plus openclaw-development: its TMPDIR when set, and /tmp on Linux when TMPDIR is unset. Do not substitute this terminal's TMPDIR. The development settings define that directory.

kubectl --kubeconfig '<state-directory>/kubeconfig' \  --context 'k3d-<cluster>' \  -n cert-manager describe pods

A registry DNS failure shows up as a lookup error in the Pod events. Follow Resolve node DNS failures and set a reachable resolver for a fresh start. That recovery changes only the owned node's resolver. If startup rolls the cluster back, wait until that command exits before starting again.

Local Agent Sandbox controller image pull fails

Default startup preloads the pinned controller. Check registry access and the

and retry occ dev down; uncertain/replaced tags retain recovery state.

For custom manifests, inspect Pod events. ImagePullBackOff with cdn.registry.k8s.io resets means node delivery failed. Restore connectivity or preload a verified cache. Preserve version pins, TLS verification and network policies.

Local Codex sandbox check fails

Kubernetes Compute startup with OCC_DEVELOPMENT_SANDBOX_DRIVER=none verifies the dedicated Codex sandbox. Retain startup errors and bubblewrap diagnostics if that check fails.

Follow Identify the restriction to distinguish container seccomp policy from host user-namespace, AppArmor, or SELinux restrictions. Operation not permitted alone does not identify the cause or justify a host-wide sysctl change. Correct the confirmed restriction, wait for startup and cleanup to finish, then retry. Keep the sandbox check enabled.

When needed, the local launcher installs the profile only on its own k3d node. Production and shared-cluster profile installation belongs to node provisioning; use the Codex sandbox profile procedure there.

Source build fails TLS certificate verification

If HTTPS inspection causes a certificate error during a runtime download, install the approved proxy CA in the host trust store. Before retrying occ dev up, select a PEM bundle containing both public roots and the proxy CA. The build mounts it only for downloads.

# Fedora 44 system bundle after the approved CA has been installed.export OCC_DEVELOPMENT_BUILD_CA_FILE=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

The local console reports a certificate error

For NET::ERR_CERT_AUTHORITY_INVALID or an unknown issuer, check the Console URL against this installation’s public CA. Set BROWSER_CA to the local certificate path and use the exact Browser console URL printed at startup.

BROWSER_CA='<local-browser-ca.crt-path>'CONSOLE_URL='<printed-HTTPS-browser-console-URL>'openssl x509 -in "$BROWSER_CA" -noout -subject -dates -fingerprint -sha256curl --fail --show-error --max-time 10 --cacert "$BROWSER_CA" \  -o /dev/null -w 'HTTP %{http_code}; TLS verify %{ssl_verify_result}\n' "$CONSOLE_URL"

HTTP 200 and TLS verify 0 confirm the chain and hostname. If the browser still rejects the issuer, enable SSL trust for this installation’s CA in the browser’s trust store. occ dev up leaves browser trust settings unchanged.

Trust the CA on macOS

Safari and Chrome use explicit local trust settings in macOS Keychain Access. For an unmanaged Mac where local CA trust is permitted:

1. Open Keychain Access, select the login keychain, and import only the printed browser-ca.crt using File → Import Items.
2. Open the imported OCC development browser CA. Match its SHA-256 fingerprint to the command above; different installations use the same name.
3. Expand Trust, set Secure Sockets Layer (SSL) to Always Trust, and close the certificate window. Complete any macOS authentication prompt locally.
4. Reload the printed HTTPS URL. If the browser cached the old trust decision, quit and reopen it. Expect the console sign-in page without a certificate warning.

See Apple's certificate trust settings and Chrome's local trust-store behavior. Other browsers may use a separate certificate store. On managed computers, use the administrator-approved CA trust procedure; report a policy restriction instead of bypassing the warning or disabling TLS verification.

If the curl --cacert check fails, inspect its error before importing anything: an expired certificate, wrong hostname, or CA from another installation needs that cause resolved. Do not use curl -k as a successful verification result.

Remove trust when discarding the installation

Before occ dev down deletes the state files, record the CA fingerprint. Remove trust and delete only the matching certificate. Other installations use the same CA name. Trust each new installation’s CA separately.

Open the console from another machine

With the browser port enabled, Kubernetes-only startup prints an HTTPS console URL and its public CA, including when using OpenShell. For example: https://console.<cluster>.oce.localhost:8443/console/. <cluster> is the k3d cluster name from startup. The local OpenShell quickstart explicitly sets OCC_DEVELOPMENT_BROWSER_PORT=8443; use the printed port when it differs. When the browser port is disabled, OpenShell startup prints only the HTTP API Console URL. Enable the browser port for the HTTPS workflow below.

Open that URL on the machine that ran occ dev up. The hostname ends in .localhost, so the computer that looks it up resolves it to its own loopback address. The launcher publishes the console port on 127.0.0.1 of the startup machine only. A browser on another computer reports a connection error, such as ERR_CONNECTION_REFUSED, because nothing is listening there.

From the computer that will run the browser, forward the printed port to the startup machine's loopback and bind the local listener to loopback. Leave this session open:

ssh -N -L localhost:8443:127.0.0.1:8443 <user>@<startup-host>

Replace 8443 in both places when startup printed a different port. The localhost bind keeps the forwarded port on the browser machine's loopback even when that machine's SSH configuration sets GatewayPorts yes.

Copy only the printed public CA to that same computer:

scp <user>@<startup-host>:<printed-ca-path> ./browser-ca.crtBROWSER_CA='./browser-ca.crt'CONSOLE_URL='<printed-HTTPS-browser-console-URL>'

<printed-ca-path> is the browser-ca.crt path from startup. Follow local browser CA trust on the browser computer using the local BROWSER_CA value above instead of the startup machine's CA path. Keep the printed hostname and port in CONSOLE_URL. Leave the state directory, which holds the TLS key, on the startup machine. Open the printed URL. The console sign-in page loads. Stop the forward when you are done. Do not publish the console port on an address other than loopback. This name and certificate are for the private development installation.

Bootstrap volume preparation failed

If prepare-bootstrap-volume reports a retained Pod, run its printed Inspect claim command for the phase, StorageClass and bound volume. Ask the storage administrator to check a Pending claim's class, provisioning and placement. WaitForFirstConsumer claims can remain Pending until the consumer is scheduled.

The Helm installation did not complete

Use Helm and kubectl with read access to openclaw-system. Set KUBECONFIG_FILE and CONTEXT to the affected cluster. Replace the default release oce if needed.

Start with the release and the initialization Job:

HELM_RELEASE=ocehelm status "$HELM_RELEASE" --namespace openclaw-system \  --kubeconfig "$KUBECONFIG_FILE" --kube-context "$CONTEXT"kubectl --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" -n openclaw-system \  get jobs,pods,pvcINITIALIZATION_JOB=$(kubectl --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" -n openclaw-system \  get jobs --selector "app.kubernetes.io/instance=$HELM_RELEASE,app.kubernetes.io/component=initialization" \  -o name)kubectl --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" -n openclaw-system \  describe "$INITIALIZATION_JOB"

Labels locate Jobs whose names were shortened for long releases. If no Job exists, inspect the Helm error before describe or logs; the hook may not exist.

If the Job started, inspect the failing container:

kubectl --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" -n openclaw-system \  logs "$INITIALIZATION_JOB" -c installation-check # upgrades onlykubectl --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" -n openclaw-system \  logs "$INITIALIZATION_JOB" -c migrationkubectl --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" -n openclaw-system \  logs "$INITIALIZATION_JOB" -c bootstrap

On upgrades, installation-check first loads any Console theme bundle and the Installation as API startup does, naming refused settings or files. The API keeps serving. Migration precedes bootstrap; unstarted containers have no logs. For image, scheduling or storage errors, run kubectl describe pod <pod-name> with the same context and namespace, and resolve the reported Pod events. Follow bootstrap recovery before retrying bootstrap failures. Do not delete the database or credential volume to force a rerun. Confirm recovery with a successful authenticated occ installation get; a completed Job alone is insufficient.

The control plane is unavailable or Namespaces do not become ready

Check the API and worker separately:

kubectl --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" -n openclaw-system \  get deployment openclaw-enterprise-api openclaw-enterprise-workerkubectl --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" -n openclaw-system \  logs deployment/openclaw-enterprise-api --tail=100kubectl --kubeconfig "$KUBECONFIG_FILE" --context "$CONTEXT" -n openclaw-system \  logs deployment/openclaw-enterprise-worker --tail=100

If a Deployment has no available replicas, inspect its Pods and their events. If the API is available but a newly created Namespace stays unready, check the worker logs and the tenant RoleBindings. After fixing the cause, repeat occ installation get and confirm the Namespace reports ready before creating an Agent there. A healthy control plane does not prove an Agent has deployed or can run a model.

A deployment fails because Kubernetes refused a Secret

KUBERNETES_SECRET_WRITE_FORBIDDEN means Kubernetes answered 403 to one of the version's private Secret writes or readbacks, so the worker failed the deployment without retrying. A preparation failure's worker.compute-prepare-failed log line names the step. Check that the worker's tenant RoleBindings grant get, create and update on secrets in the Agent's Kubernetes namespaces (with an execution cluster, also the control cluster's Gateway namespace), and that no admission policy or ResourceQuota refuses Secrets there. Then deploy again. A running version's maintenance does not fail: it retries and recovers by itself once the grant is fixed.

An Agent's Gateway or Harness Pod stays unready

Run kubectl describe pod <pod-name> in the Agent's tenant namespace. Each Readiness probe failed: event reports the private HTTP endpoint's status or a connection failure. A 503 means the runtime's native, plugin, authentication, or identity gate has not passed. Check the container's startup-phase logs for the failing stage. See Harness authentication for the probe codes.

A Gateway log asks for a restart to apply the model catalog

OpenClaw downloads its hosted model catalog in the background and logs remote model catalog downloaded; restart the Gateway to apply it. No action is needed: the Gateway keeps serving with the catalog it started with, and the download applies, if at all, at its next start. To stop this background traffic, set models.catalogRefresh.enabled to false in the Agent's Configuration and deploy it; the Gateway then uses only the catalog bundled in the runtime image.

Authentication fails after installation

Use the intended credential: occ installation get uses the protected service key; the browser console uses a human administrator sign-in. Check OCC_URL against the installation's approved HTTPS origin and confirm OCC_SERVICE_KEY_FILE points to the intended key. For a missing or exposed key, follow service-key recovery. Repeat the failed operation after recovery; do not copy tokens or key-file contents into an incident report.

Agent workspace files are unavailable

A 503 DEPENDENCY_UNAVAILABLE can mean routing is not configured or the proxy or gateway is unavailable. Start with the Gateway, certificate, security policy, and tenant HTTPRoute in workspace-routing verification. After fixing the failed resource, repeat the original OCC file request. A 404 NOT_FOUND on an otherwise working route means that specific file is missing; do not create or overwrite it just to clear the console message.

For missing exported logs, use observability troubleshooting. When escalating, record the release, cluster context, affected Namespace IDs, failure timestamps, exact error code, and relevant redacted events. Keep service keys and raw Secret values out of the report.