Kubernetes tests

Overview

Use disposable local k3d with enforcing NetworkPolicies. Fixtures exclude real gateways/models; runtime tests include them.

Local Kubernetes installation

Install k3d, kubectl, Helm, certutil, and Chromium with sandbox support. Use a disposable container engine with image-build capacity. The Keycloak cases need free, bindable loopback port 443 and must not run beside the standalone Keycloak lane. Run these commands from the trusted checkout root:

pnpm install --frozen-lockfilepnpm exec playwright install chromiumpnpm cli:buildOCC_TEST_DEV_UP_K3D_REAL=1 node --test --test-concurrency=1 tests/integration/dev-up-k3d-real.test.mjs tests/integration/dev-up-k3d-refusals-real.test.ts tests/integration/dev-up-k3d-keycloak-real.test.ts

The tests cover installation, cleanup, workspace boundaries, NetworkPolicy, sandboxing, and Keycloak. Model responses are fixtures. Failed cleanup retains state for occ dev down. See hosted execution.

Kubernetes HTTP fixture

Use Kubernetes 1.35+, Docker, k3d, kubectl, and a dedicated migrated database. Choose another cluster name if oce exists. Default kubeconfig and context remain unchanged. Set OCC_TEST_KUBERNETES_PLUGIN_STATUS_PROXY_CIDRS to the actual API-server Pod-proxy source CIDRs as a comma-separated list before running the tests.

mkdir -m 700 -p /tmp/oce-k3dk3d cluster create oce \  --image +v1.35 \  --api-port 127.0.0.1:6443 \  --kubeconfig-update-default=false \  --kubeconfig-switch-context=falsek3d kubeconfig get oce > /tmp/oce-k3d/kubeconfigchmod 600 /tmp/oce-k3d/kubeconfig docker build --pull=false -t oce-fixture:local tests/fixtures/kubernetesk3d image import oce-fixture:local -c oce docker exec k3d-oce-server-0 ctr -n k8s.io images list -q |  grep -Fx 'docker.io/library/oce-fixture:local' &&OCC_TEST_KUBERNETES_KUBECONFIG=/tmp/oce-k3d/kubeconfig \OCC_TEST_KUBERNETES_CONTEXT=k3d-oce \OCC_TEST_KUBERNETES_IMAGE=oce-fixture:local \OCC_TEST_KUBERNETES_PLUGIN_STATUS_PROXY_CIDRS="${OCC_TEST_KUBERNETES_PLUGIN_STATUS_PROXY_CIDRS:?Set the API-server Pod-proxy source CIDRs}" \OCC_TEST_DATABASE_URL=postgresql://occ_app:occ-app-local@127.0.0.1:55432/openclaw_k8s_local \  node --test --test-concurrency=1 tests/integration/kubernetes-compute-real.test.ts \    tests/integration/kubernetes-compute-provisioning-real.test.mjs \    tests/integration/kubernetes-compute-driver-real.test.ts

Run sequentially against this database without skips. Missing selectors or database skip cases; partial selectors fail. Coverage: API, worker, Driver, RBAC, storage, lifecycle and network isolation. Verify cloud volume detach and node fencing separately.

The State and Lifecycle check traces a deploy from request to worker success and checks the direct Agent audit grant.

For live OAuth, use OpenShell OAuth and credentialed runtime. Delete only your disposable cluster:

k3d cluster delete oce

Installation handoff

With Helm, yq, the go.mod toolchain, and the cluster above:

OCC_TEST_KUBERNETES_KUBECONFIG=/tmp/oce-k3d/kubeconfig \OCC_TEST_KUBERNETES_CONTEXT=k3d-oce \  node --test tests/integration/installation-handoff-k3d-real.test.ts

Checks named-Job access, replacement, interrupted Helm dispatch, CRDs/controllers, quota headroom, address drift and denials. EKS connectivity and Envoy acceptance remain unproved. The production observability lane runs installation-receipt-k3d-real.test.ts with PostgreSQL to check bootstrap after interruption before and after UID capture.

Production workload network access

Use an exclusive fixture cluster with oce-system and envoy-gateway-system available. Install Helm and yq, then run:

OCC_TEST_KUBERNETES_KUBECONFIG=/tmp/oce-k3d/kubeconfig \OCC_TEST_KUBERNETES_CONTEXT=k3d-oce \OCC_TEST_KUBERNETES_IMAGE=oce-fixture:local \  node --test tests/integration/production-network-access.test.mjs

Checks cover isolation, management DNS/API/Git TCP reachability, denied ports and destinations, and grant withdrawal. TLS-preserving API proxies verify withdrawal; a controlled discovery snapshot tests removal of the last backend. Its CI lane owns a separate cluster and awaits slice and namespace cleanup. No database or model credentials are required.

Kubernetes model turns and Secrets

The real-runtime suite needs an authorized model credential, real gateway and Codex images, a disposable cluster with enforcing NetworkPolicies, and a migrated openclaw_k8s_* database. A fixture pass is not a model turn. Follow Codex sandbox setup for seccomp prerequisites.

Develop with local containers and k3d

Install k3d, kubectl, Helm, OpenSSL, and Docker Compose or podman-compose. Start the engine's API socket and set OPENAI_API_KEY privately. The helper builds the images and prepares its own cluster and database:

./scripts/k3d --model-id gpt-6-astra

The demo completes a model turn and serves Control UI at http://127.0.0.1:18888 and OCC at http://127.0.0.1:18889. It prints a password-copy command. Keep the process running to use the interfaces. Without OPENAI_API_KEY, the interactive demo prompts without echo; test requires the variable. For a custom Codex Responses endpoint, use:

./scripts/k3d test --harness codex \  --base-url https://openrouter.ai/api/v1 --model-id z-ai/glm-5.3-flash

./scripts/k3d info shows connections; reset retains images. Fresh preparation pins images against kubelet collection. Recreate older environments with ./scripts/k3d down --harness codex, then reprepare (openclaw for that harness). This deletes that environment; use before image changes or after preparation failures. Default kubeconfig/context and unrelated resources stay unchanged. OCC_K3D_CONTAINER_ENGINE selects docker/podman; OCC_K3D_STATE_DIR sets its directory. See local k3d image upgrades.

Run the real-runtime suite directly

Import approved local gateway and Codex images into the fixture cluster and register their immutable references in k3s:

k3d image import '<local-gateway-tag>' '<local-codex-tag>' -c ocedocker exec k3d-oce-server-0 ctr -n k8s.io images tag \  '<imported-gateway-image>' '<gateway-image>@sha256:<digest>'docker exec k3d-oce-server-0 ctr -n k8s.io images tag \  '<imported-codex-image>' '<codex-image>@sha256:<digest>'

Create a private environment file, set OPENAI_API_KEY privately, and select the actual API-server Pod-proxy source CIDR:

OCC_TEST_KUBERNETES_KUBECONFIG=/tmp/oce-k3d/kubeconfigOCC_TEST_KUBERNETES_CONTEXT=k3d-oceOCC_TEST_DATABASE_URL=postgresql://occ_app:occ-app-local@127.0.0.1:55432/openclaw_k8s_localOCC_TEST_KUBERNETES_GATEWAY_IMAGE=<gateway-image>@sha256:<digest>OCC_TEST_KUBERNETES_AGENT_IMAGE=<codex-image>@sha256:<digest>OCC_TEST_KUBERNETES_PLUGIN_STATUS_PROXY_CIDRS=<api-server-proxy-source>/32OCC_TEST_OPENAI_MODEL=gpt-6-astra

OCC_TEST_HARNESS_K3D_REAL=1 OCC_TEST_SLACK_LIVE=0 \  node --env-file="$TEST_ENV_FILE" --test tests/integration/harness-topology-k3d-real.test.mjs

Seven non-Slack cases must pass: dedicated Codex, dedicated Skill source uploads, embedded OpenClaw, Secret lifecycle, Default user deploy, and startup-failure evidence with and without plugins. The Skill source case needs local zip. A selected suite fails on missing prerequisites; an unselected suite skips. The suite does not install OCC with Helm. Credentials belong only in the selected runtime Pods, not controllers, logs, or shell history. For private workspace routing, run the separate gateway routing suite.

For custom provider and image settings see the test's source and network setup. Transcript persistence requires a runtime with SQLite transcripts. A version number alone does not establish storage support.

Gateway and Harness resources use separate managed namespaces. Default fixture placement does not prove production node-pool isolation. Use the separate Slack and observability suites for those outcomes.

Keycloak OIDC tests

The keycloak-oidc lane checks real browser sign-in, key rotation, rejection, and sign-out against a disposable Keycloak server. It uses generated fixture credentials and an in-process production API with PostgreSQL.

Verified Keycloak flows

The cases check:

A skipped or missing case fails the lane.

Run the Keycloak lane

You need Node.js 24, Docker, OpenSSL, Chromium, and a free, bindable 127.0.0.1:443. Run only one Keycloak lane per host.

state="$PWD/.ci-state/keycloak-oidc.json"node scripts/ci/prepare.mjs --lane keycloak-oidc --state "$state" &&  node scripts/ci/run-tests.ts run keycloak-oidc --state "$state" --results "$PWD/.ci-state/results.json"node scripts/ci/cleanup.ts --state "$state"

Preparation creates a private CA, fixture credentials and a Keycloak container. It may add a tagged line to /etc/hosts using sudo -n; cleanup removes its container, private files, and the line it added. If passwordless sudo is unavailable, add this line yourself before preparation and remove it afterward:

echo '127.0.0.1 keycloak.oce.localhost keycloak.oce.test' | sudo tee -a /etc/hosts

Resolve both names to 127.0.0.1, optionally also ::1, with no nonloopback answers. Readiness uses IPv4 to match Docker. For preparation errors, check the named step: image pull, port availability, placeholders, hosts resolution, or the printed Keycloak log for readiness errors.

Keycloak local launcher coverage

The local installation test checks sign-in, recovery and rollback. It uses its own CA and does not edit /etc/hosts.

Private gateway routing

This suite tests private workspace routing through real Envoy Gateway and cert-manager in a disposable k3d cluster. It verifies file access, model consumption after Gateway replacement, authentication denial, key rotation, certificate renewal, and the dedicated node route. It does not prove a full Helm installation or Harness node credential persistence.

Run private gateway routing

Install the k3d helper prerequisites and provide an authorized OPENAI_API_KEY privately:

./scripts/k3d test./scripts/k3d down

The helper prepares a cluster, database, images, Envoy Gateway, cert-manager, and test CA. Direct execution also requires OCC_TEST_GATEWAY_ROUTING_REAL=1, OCC_TEST_PRODUCTION_CONTROLLER_IMAGE, OCC_TEST_GATEWAY_CA_CERT_PATH, OCC_TEST_GATEWAY_CA_KEY_PATH, NODE_EXTRA_CA_CERTS, and the runtime setup above. Keep the CA key private.

For CI-shaped setup, supply OPENAI_API_KEY, OCC_TEST_OPENAI_MODEL, and NODE_BASE_IMAGE in the environment. In a GitHub Actions runner, execute:

node scripts/ci/prepare.mjs \  --lane gateway-routing \  --state "$RUNNER_TEMP/state/gateway-routing.json" \  --github-env "$GITHUB_ENV" &&node scripts/ci/run-tests.ts run gateway-routing \  --state "$RUNNER_TEMP/state/gateway-routing.json" \  --results "$RUNNER_TEMP/results/gateway-routing.json"

Use scripts/ci/cleanup.ts --state "$RUNNER_TEMP/state/gateway-routing.json" to remove the lane's resources.

Native workspace identity

Prepare native runtime and Envoy images, then set both OCC_TEST_RUNTIME_IMAGE and OCC_TEST_WORKSPACE_ENVOY_IMAGE:

node --test tests/integration/workspace-files-native.test.ts

This checks file access without model calls. It does not prove Kubernetes routing policies, PostgreSQL or worker execution. Selecting only one image fails.

Native Gateway sharing

The gateway-routing lane checks native browser access, agent:operate chat, administration denial, and WebSocket closure after permission withdrawal or session expiry. Its combined QA journey adds real Keycloak sign-in. See Native Agent browser tests for selection, prerequisites, cleanup, and proof boundaries. These tests do not qualify automatic account enrollment, Git, or OpenShell.

Production TUI tests

This Helm test exercises production API and native TUI revision cutover with a real model on disposable k3d. Each gateway container needs at least 4 GiB.

Production TUI Helm test environment

In a private environment file, set:

• OCC_TEST_KUBERNETES_KUBECONFIG and OCC_TEST_KUBERNETES_CONTEXT: a dedicated k3d kubeconfig and k3d-* context with a loopback HTTPS API.
• OCC_TEST_PRODUCTION_CONTROLLER_IMAGE, OCC_TEST_KUBERNETES_RUNTIME_IMAGE, OCC_TEST_PRODUCTION_POSTGRES_IMAGE, and OCC_TEST_PRODUCTION_NODE_IMAGE: immutable references already imported into that cluster.
• OPENAI_API_KEY: existing authorized credential; optionally set OCC_TEST_OPENAI_MODEL (default gpt-6-astra).

Teardown requests Helm uninstall and asynchronous namespace deletion, suppressing errors. Verify resource removal. OCC_TEST_PRODUCTION_TUI_KEEP=1 retains the setup for rehearsal with attach.sh and proof.json. Require a completed run for proof.

Run the production TUI suite

OCC_TEST_PRODUCTION_TUI_REAL=1 \  node --env-file="$TEST_ENV_FILE" --test tests/integration/production-tui-k3d-real.test.mjs

For independent upgrades, set both OCC_TEST_PRODUCTION_UPGRADE_CONTROLLER_IMAGE and OCC_TEST_PRODUCTION_UPGRADE_RUNTIME_IMAGE. One input fails. Omit both to skip upgrades.

Run an isolated upgrade qualification lane

Set OCC_TEST_OPENAI_MODEL. Verify provenance for compatible prior and distinct reviewed candidate controller and runtime digests. A disposable Pod selects the incumbent image’s worker probe for installation. The upgrade uses the candidate chart. Install Docker, k3d, kubectl, Helm, yq, and Go.

run_dir="$(mktemp -d)"node --env-file="$TEST_ENV_FILE" scripts/ci/prepare.mjs --lane production-tui --state "$run_dir/state.json" && \  TMPDIR="$run_dir" node --env-file="$TEST_ENV_FILE" scripts/ci/run-tests.ts run production-tui \    --state "$run_dir/state.json" --results "$run_dir/results.json"

Before removing the runner, retain both private availability receipts.

After success or failure, save the private image mapping if needed and clean up:

install -m 600 "$run_dir/state.json" "$run_dir/state-before-cleanup.json"node scripts/ci/cleanup.ts --state "$run_dir/state.json"

Retain the state if cleanup fails. Do not publish the state or environment file. Confirm evidence includes both upgrades and a fresh model reply. Imported local references can differ from registry digests; this does not prove exact registry reference deployment or live EKS behavior.

Two-cluster validation

This experimental profile runs OCE and dedicated Codex Harnesses in separate clusters. It excludes embedded execution, cloud provisioning and repository credential delivery. Re-run it against current images before claiming acceptance.

Prepare isolated infrastructure

Use two disposable k3d clusters with Kubernetes 1.35 or later, distinct Pod and Service CIDRs, enforcing NetworkPolicies, and loopback kubeconfigs. Select an existing disposable Linux VM through its Docker context. Do not change the default Docker context or kubeconfig. Use unused names and ports:

umask 077export OCE_DOCKER_CONTEXT='your-isolated-local-context'export OCE_TEST_DIR="$(mktemp -d "${TMPDIR:-/tmp}/oce-two-cluster.XXXXXX")"OCE_DOCKER_SOCKET="$(docker context inspect "$OCE_DOCKER_CONTEXT" \  --format '{{.Endpoints.docker.Host}}')"case "$OCE_DOCKER_SOCKET" in unix://*) ;; *) exit 1 ;; esack3d_local() {  env -u DOCKER_CONTEXT DOCKER_HOST="$OCE_DOCKER_SOCKET" k3d "$@"}OCE_K3S_IMAGE='rancher/k3s@sha256:59fe491fd3b73204e499e40b325240d85c42c7189c3ae50150d37b78243f3b32'docker --context "$OCE_DOCKER_CONTEXT" network create oce-two-cluster-localk3d_local cluster create openclaw-k8s-local-cp \  --image "$OCE_K3S_IMAGE" --network oce-two-cluster-local \  --api-port 127.0.0.1:56641 --servers 1 --agents 0 --no-lb \  --kubeconfig-update-default=false --kubeconfig-switch-context=false \  --k3s-arg '--disable=traefik@server:*' \  --k3s-arg '--cluster-cidr=10.70.0.0/16@server:*' \  --k3s-arg '--service-cidr=10.71.0.0/16@server:*'k3d_local cluster create openclaw-k8s-local-dp \  --image "$OCE_K3S_IMAGE" --network oce-two-cluster-local \  --api-port 127.0.0.1:56642 --servers 1 --agents 0 --no-lb \  --kubeconfig-update-default=false --kubeconfig-switch-context=false \  --k3s-arg '--disable=traefik@server:*' \  --k3s-arg '--cluster-cidr=10.72.0.0/16@server:*' \  --k3s-arg '--service-cidr=10.73.0.0/16@server:*'k3d_local kubeconfig get openclaw-k8s-local-cp > "$OCE_TEST_DIR/cp.kubeconfig"k3d_local kubeconfig get openclaw-k8s-local-dp > "$OCE_TEST_DIR/dp.kubeconfig"kubectl --kubeconfig "$OCE_TEST_DIR/cp.kubeconfig" label \  node k3d-openclaw-k8s-local-cp-server-0 oce-role=control-planekubectl --kubeconfig "$OCE_TEST_DIR/dp.kubeconfig" label \  node k3d-openclaw-k8s-local-dp-server-0 oce-role=agents

Keep the private kubeconfigs out of Git and runtime images. Install Envoy Gateway in both clusters and cert-manager in the control plane (CP). Provide independent RWO Gateway storage in CP and Harness storage in the execution cluster (DP). Import approved immutable images and install the Codex seccomp profile on DP.

After CoreDNS installs, place coredns.yaml.skip under /var/lib/rancher/k3s/server/manifests/ on each owned test node to retain local DNS settings across restarts. This is fixture setup, not production DNS setup.

Configure the network and credentials

All cross-cluster traffic uses verified HTTPS. Set reachable DNS names, certificates, observed CIDRs, and cluster API access. Do not use cluster-local .svc addresses across clusters. CP API and worker use separate scoped DP kubeconfigs; Gateway uses the exact-Agent transport token to reach the Harness, and the workspace node uses its node-only identity to reach Gateway.

Create a DP TLS Secret and install deploy/helm/openclaw-execution with routing.hostname, gatewayClassName, tlsSecretName, and controlPlaneCidrs. Provision distinct DP API and worker kubeconfigs, then store them in separate CP Secrets with a kubeconfig key. Administrators own credential issuance, rotation, and tenant grants. The worker must not have RoleBinding-write or impersonation permission.

Add this to the Installation Compute configuration:

executionCluster:  authentication:    mode: kubeconfig    kubeconfigPath: /etc/openclaw/execution/kubeconfig    context: execution  harnessRouting:    hostname: harness.example.test    gatewayName: oce-harnesses    gatewayNamespace: openclaw-system    envoyNamespace: envoy-gateway-system  network:    dns:      namespace: kube-system      podLabels: { k8s-app: kube-dns }    harnessEndpointCidrs: ["<DP ingress destination CIDR>"]    gatewayEndpointCidrs: ["<CP ingress destination CIDR>"]    pluginStatusProxySourceCidrs: ["<DP API proxy source CIDR>"]  caBundle: |    <public DP certificate authority PEM, if privately issued>

Existing authentication, network, and gatewayRouting settings select CP. Supply public CA certificates only. Enable the CP chart's executionCluster, set apiKubeconfigSecretName, workerKubeconfigSecretName, apiCidrs, and apiPort, and set gatewayRouting.remoteNodeCidrs to the observed DP sources. Workloads do not receive kubeconfigs. Canonical model, channel, and transport Secrets stay in CP; selected model and transport material is delivered to data-plane Secrets for each Agent revision.

Use a coredns-custom ConfigMap with a separate *.server zone. Resolve the CP Gateway hostname to its Service IP in CP and its reachable ingress address in DP, with the same hostname and certificate identity. Restart CoreDNS and verify DNS from controller and runtime Pods before creating Agents.

Install and exercise the platform

Follow production installation for PostgreSQL, restricted roles, bootstrap and the CP Helm release. Prepare bootstrap PVC ownership before initialization. If initialization fails, check commit state and follow bootstrap recovery; do not reset the database or delete protected output automatically.

Grant the DP tenant worker and API bindings and the three CP bindings in Agent preparation. Wait for Namespace ready and deployment succeeded.

Create a private JSON fixture with apiUrl (loopback), serviceKeyFile, dockerContext (an explicit local Unix-socket context), agentConfiguration (the dedicated native Configuration create body), and control and execution objects. Each object needs kubeconfigPath, kubernetesContext, release, and systemNamespace. Set harnessAuthMethod to api_key (default) or codex_pat; provide its credential through OPENAI_API_KEY or CODEX_ACCESS_TOKEN, not the fixture. The pass covers only the selected method.

OCC_TEST_TWO_CLUSTER_CONFIG=/private/path/two-cluster.json \  node --test tests/integration/kubernetes-two-cluster-real.test.mjs

The selected test fails on missing inputs. It checks real model turns, workspace RPC, replacement and recovery. It stops and restores the selected DP node; do not run other tests against it concurrently. A failed run retains its Namespace. Delete its Agent, Configuration, Secret and seeded Presets through the API before deleting the Namespace. The test does not establish recovery after worker retry exhaustion or an aged node reconnect after its pairing code expires. Remote Codex plugin installation needs a separate ChatGPT-backed credential and proof.

When finished, remove only the clusters and network created above:

k3d_local cluster delete openclaw-k8s-local-cp openclaw-k8s-local-dpdocker --context "$OCE_DOCKER_CONTEXT" network rm oce-two-cluster-local

Remove the private test directory when you no longer need its kubeconfigs.

Upgrade the execution chart

Upgrade the execution release before the control-plane image upgrade:

helm upgrade <execution-release> deploy/helm/openclaw-execution \  --kube-context <execution-context> --namespace <execution-system-namespace> \  -f <execution-values.yaml>

Pass the installation values; do not use --reuse-values. Keep agentRuntimeLogs.enabled equal to the CP chart value. The control-plane upgrade preflight checks tenant grants before stopping the old controller.