Troubleshoot Agents

Give operators the Namespace, Agent/version IDs, time and request ID. Never include API keys, tokens, or Secret values.

Codex refuses the required permission profile

For RUNTIME_CODEX_PERMISSION_POLICY_INCOMPATIBLE, find the revision's worker.completed event. Its failure field (occ.failure in the Collector) lists sandbox-mode-denied, named-profile-denied, or reviewer-denied, separated by underscores. Ask the policy owner to check OpenShell's requirements. For RUNTIME_CODEX_REQUIREMENTS_TOO_LARGE, ask the policy owner to reduce the reply to at most 1 MiB, then redeploy.

If a dedicated Codex Agent on OpenShell reports Codex did not grant the requested :danger-full-access profile, inspect the error's comparison category:

• sandbox-readOnly or sandbox-workspaceWrite: Codex returned a restricted sandbox. Ask the account administrator to check its managed permission requirements against the OpenShell permission-profile requirement.
• sandbox-externalSandbox: Codex returned a different sandbox mode. Ask the operator to check the selected native profile and runtime compatibility.
• nonobject-result, field-absent, field-null, field-not-object, type-not-string, or type-unrecognized: the response did not provide a recognized sandbox. Check the packaged runtime version and response contract before attributing the failure to account policy.

Without this category, the error cannot distinguish a restricted grant from an incompatible response.

Login, source readiness, and a model reply under another profile cannot prove compatibility. Use an approved compatible credential or policy change. Update copied API-key or PAT sources. For device login, sign in again and replace the Agent binding. Redeploy, then verify a model response from the Agent. Keep the Gateway permission check and OpenShell isolation in place.

Deploy requires a Sandbox Driver that supports protected Codex artifact admission

OCE stops before creating an Agent version when the selected Sandbox Driver cannot validate artifact access for protected Codex repository execution. Ask your Installation administrator to check the drivers.sandbox selection and whether the installed Driver version provides protected Codex artifact admission. For OpenShell, see how to configure the artifact rules.

The configured model is unavailable

Open the failed version in the Agent’s Deployment tab. If the startup model check reports an unavailable model, choose a model ID for the same credential.

For the dedicated Codex quickstart, open Configuration → Edit current Configuration. In the draft, replace the old model ID in these three places:

• Set agents.defaults.model to codex/<new-id>.
• Rename its entry in agents.defaults.models to the same prefixed ID. Keep the entry’s Codex runtime settings.
• Set the corresponding models.providers.codex.models entry’s id and name to the new ID without the prefix.

Save the Configuration and select Deploy new version on the same Agent. When deployment succeeds, open Open Agent UI and send a prompt to confirm the model replies.

The Agent or Namespace is missing

Open Namespaces and confirm that you selected the intended Namespace. The console and HTTP API only list resources your identity can read. If the Namespace is absent, ask your administrator to check your access. If it is provisioning or failed, an operator must check its provisioning status before deployment. Once access is fixed, refresh and confirm the Namespace is ready.

Deployment is blocked or has no selected version

active is the Agent lifecycle, not workload health. An Agent can be active without a selected version. If you did not retain the version ID, inspect the latest deployment you can read:

occ --namespace NAMESPACE_ID agent deployment-status AGENT_ID --output json

Without a deployment ID, this command returns the status of the latest readable Agent revision. Inspect the deployment error before changing credentials. If it failed, follow the next step in error.message. See the deployment status reference.

On Kubernetes, open the Agent's Open draft draft, then Credentials. Select Reload authentication source, then confirm under Harness authentication that Authentication source is the source you expect, not None. Check that the saved model authentication matches the execution mode. For an OpenAI API key, the person selecting it and the Agent's own identity both need operate on the exact platform Secret. See Harness authentication.

If the deployment was accepted, use the returned version ID to check its deployment status:

GET /namespaces/:namespaceId/agents/:agentId/deployments/:revisionId

For Kubernetes workload problems, operators can use the Compute failure checks. When a Kubernetes deployment fails with DEPENDENCY_UNAVAILABLE after its retries, the worker log's worker.compute-prepare-failed events for that version name the code, preparation step and reason. The log Collector exports the code but not the step or reason. KUBERNETES_OWNERSHIP_CONFLICT means the driver refused an object it does not own, such as a Deployment with the Agent's workload name but no OCE ownership labels. The driver never adopts such an object. Delete it, then deploy again.

For a dedicated Agent, a connected workspace node alone does not establish enrollment: OCC must record its device through the setup handoff. If deployment stays pending, give the operator the version ID and worker's reason code. Keep the node's retained identity and workspace intact; repeated restarts do not prove that handoff completed. Unconfirmed credential possession after setup expiry requires the scoped node-Secret recovery; do not delete the node identity or workspace. See workspace-node enrollment.

A second node runner using the same local profile fails immediately. Keep one runner per profile. Its lock remains held until clients, runtime shutdown and pending endpoint writes finish. Recovery requires a definitely dead owner or a process-start identity mismatch; age alone never permits reclaiming a live or ambiguous owner. Different profiles remain independent. This local runner lock does not coordinate other hosts or arbitrary credential writers; workload exclusion remains the execution provider's responsibility.

For an OpenShell workspace-baseline failure with created directory admission failed and insecure-permissions, ask the operator to check the node-state directory's owner and mode. Its private root must be 0700; OCE prepares it at startup, including on storage that adds setgid. After correcting the startup or storage problem, deploy the Agent again. Keep its retained state and shared workspace intact. See writable runtime state.

Verify runtime health after deployment

The console's Deployment tab shows persisted status, with no live gateway-health or chat view. Current version identifies the control plane's selection, even after a failed deployment: an embedded replacement can reject its key during the model check after replacing its predecessor. Selection does not establish credential validity or model responses. Verify the Agent's model response or ask an operator to verify the workload. Require an answer from the configured model; a successful file read is insufficient.

Check workload capacity

For Kubernetes, compare configured requests/limits, admitted Agent Pod limits and enclosing cgroup constraints with the infrastructure owner. Sandbox-visible host totals do not establish usable CPU/memory; an unlimited visible cgroup does not rule out hidden limits. Unreadable outer limits remain unknown until verified through authorized infrastructure tooling. OCE runtime descriptions and diagnostics do not measure effective capacity.

An embedded Gateway reports an unverifiable state owner

A hard stop can leave the old Gateway's locks on the Agent's volume. With Gateway managed mode on or off, OCE can recover a verified Docker predecessor or the first eligible Kubernetes container in the same Pod. A new Pod cannot verify the previous Pod's owner. An older image may also lack a record of the original PID 1 start time. If the replacement reports an unverifiable owner, use the following procedure.

A dedicated Gateway does not recover its locks. After an unclean exit it stays unready with the failed check gateway-state-lock, and its log quotes the holder. Use the same procedure.

A dedicated Harness Pod that stops without a clean shutdown during native setup, for example on a node failure, can leave the lock in its workspace node state. Each later version then fails the check workspace-baseline, and its log says cannot verify Gateway ownership. This applies to the Harness Deployment that Kubernetes Compute creates; a Harness run by a SandboxDriver has no such Deployment. Use the same procedure, with these changes:

• Step 1: The holder is usually an earlier version's Pod that is already deleted. Record the Pod name from the log's holder N at POD_NAME/pid-ns text, not the current Pod. Before you stop the Agent (stopping deletes the Harness Deployment), find the Deployment's volume mount at /home/node/.openclaw-node and record its claim, workspace-node-… subPath and the Pod's securityContext. After the stop, kubectl get pods -A -l openclaw.dev/agent=AGENT_ID must list no Pod, including Terminating Pods, and the holder Pod must not exist. Kubernetes removes a Pod only after its kubelet reports that the containers stopped, unless someone force-deleted the Pod, deleted its node or tainted the node node.kubernetes.io/out-of-service; kubectl get nodes shows every node Ready without that taint. If one of these occurred, or a node is unreachable, keep the Agent stopped until the infrastructure operator verifies on the node that the holder's containers exited.
• Step 2: Start the maintenance container from the Harness image with the Harness Pod's runAsUser and runAsGroup. Do not set fsGroup, unless you also copy the Harness Pod's fsGroupChangePolicy: OnRootMismatch. With the default policy, the kubelet gives the group read and write access to every file on the claim, and the next version fails workspace-baseline with Runtime pin store owner is unsafe. Mount only the subPath, not the claim root, at the same path, and set OPENCLAW_STATE_DIR to /home/node/.openclaw-node. The lock file names depend on this path. A Harness has no configuration mount or profile; skip those items.
• Step 3: Run the inspection and --confirm in the same container, for example a Pod that sleeps, with kubectl exec for each command. A new Pod can change the lock files' metadata, and OpenClaw then reports a changed record.
• If a version already fails with Runtime pin store owner is unsafe, stop the Agent and verify as in step 1. From a maintenance container without fsGroup, run chmod -R go-rwx,g-s /home/node/.openclaw-node, then deploy again.

1. Record the Agent, state volume and original container ID. On Kubernetes, also record the Pod UID and node. Stop the Agent in OCC and verify with the compute provider that the exact container exited. A missing Pod or expired heartbeat alone cannot establish this. If its node is unreachable, keep the Agent stopped until the infrastructure operator verifies shutdown. Stop and resume

2. Keep other writers off the volume. Start a maintenance container from the Gateway's pinned image as its runtime user. Mount the retained volume and configuration at the Gateway's paths and use its state directory and profile. Inspect the holders without starting a Gateway.

   node /app/openclaw.mjs gateway recover-lock --json

3. Match every reported holder to the stopped workload. If any holder could still write or cannot be identified, leave all locks in place. Otherwise repeat the command with --confirm and the exact confirmation fingerprint from the inspection. If OpenClaw reports a changed record, inspect and verify its holders again.

4. Exit the maintenance container, deploy the Agent again and send a chat message to verify it can use its retained state.

A save or deployment lost its response

In the original Namespace, check before retrying.

• Draft edit. Compare the edited fields in occ agent get AGENT_ID --output json. For concurrent edits, check the audit evidence.
• Credentials. Reload Credentials and check the saved bindings. Operators can inspect generated transport storage through the Agent's runtime-credentials API.
• Deployment. Check Viewing version and its status. Deploying again creates another version. For deploy-first, use the printed read-only recovery command.
• Workspace file. Reload and compare its contents with your edit.

For partial Agent creation, see Console recovery.

A plugin does not work after deployment

Look for PLUGIN_INSTALL_FAILED or PLUGIN_AUTH_REQUIRED in the deployment status. A deployment can succeed while the affected plugin is disabled for that startup. Inspect the plugin configuration before redeploying; a successful deployment does not confirm that a third-party connector is usable.

An OpenShell Agent stops answering

For a dedicated Codex Agent on OpenShell, if the Sandbox is running but replies time out, ask the operator to check the Agent Gateway's route to OpenShell: Gateway egress, matching OpenShell ingress and, for secure WebSocket (WSS) connections, the served certificate's coverage of the advertised service hostname. A successful control-endpoint check does not establish that the certificate covers the advertised hostname. Keep hostname verification enabled; correct the owning network or certificate configuration, then verify a model response.

On OpenShell, OCE asks the gateway to restart the Harness whenever its process exits. occ agent runtime AGENT_ID shows Harness Sandbox: starting (HARNESS_RESTARTING, last exit code N, restart N) for the few seconds this takes, then running; chat usually answers again within a minute. A plain starting, with no code, is a first start. A Harness that keeps exiting stays HARNESS_RESTARTING with a growing restart number while OpenShell backs off (up to three minutes between attempts), and the version's diagnostics lead with a failed agent sandbox check with that code. OpenShell reports a signal as 128 plus its number, such as 137 for a killed process. Read the Sandbox logs for the cause. A version deployed by an older controller keeps the old behavior until the Agent is deployed again.

OCE does not replace a Sandbox whose Pod was deleted or evicted, or that a node restart stopped. The Agent stays active while every chat fails. occ agent get and occ agent runtime show Harness Sandbox: lost (CODE) (SANDBOX_FAILED here, SANDBOX_STOPPED after a node restart, or HARNESS_EXITED for an older version whose Harness exited), the Console's Live serving shows Sandbox lost, and the version's diagnostics lead with a failed agent sandbox check carrying the same code. Deploy the Agent again (occ agent deploy AGENT_ID): the new version gets a new Sandbox and the old one is removed. unknown (UNAVAILABLE) means OCC could not read the Sandbox record.

An older development Sandbox can also fail exact-content adoption after a controller upgrade changes its immutable command. Deploy a new Agent revision with occ agent deploy AGENT_ID; OCE does not rewrite the existing Sandbox command.

Read OpenShell sandbox and supervisor logs

The Sandbox source shows only what the OpenShell gateway recorded. An operator with pods/log access to the Agent's Kubernetes namespace can read the Harness output and the supervisor's own log. Each version's Sandbox has two Pods that share an openshell.ai/sandbox-id: the Harness Pod, which keeps OCC's openclaw.dev/revision label, and the supervisor Pod, labeled openshell.ai/boundary-role=supervisor:

REVISION_ID=<revision-id>HARNESS_SELECTOR="openclaw.dev/workload-role=agent,openclaw.dev/revision=$REVISION_ID"NS="$(kubectl get pod -A -l "$HARNESS_SELECTOR" \  -o jsonpath='{.items[*].metadata.namespace}' | cut -d' ' -f1)"SANDBOX_ID="$(kubectl -n "${NS:?no Harness Pod for this revision}" get pod -l "$HARNESS_SELECTOR" \  -o jsonpath='{.items[*].metadata.annotations.openshell\.ai/sandbox-id}' | cut -d' ' -f1)"kubectl -n "$NS" logs --all-containers --tail=200 -l "$HARNESS_SELECTOR"kubectl -n "$NS" logs --all-containers --tail=200 \  -l "openshell.ai/sandbox-id=${SANDBOX_ID:?no Sandbox ID on the Harness Pod},openshell.ai/boundary-role=supervisor"

The first command finds the Agent's Kubernetes namespace from its Harness Pod. no Harness Pod for this revision means the version never started a Sandbox or its Pods are gone; read the deployment status instead.

kubectl -n "$NS" get sandbox,pod -l "openshell.ai/sandbox-id=$SANDBOX_ID" shows the Sandbox and its supervisor Pod. This output bypasses OCC's log redaction, so handle it as sensitive.