Kubernetes tests

Prepare shared prerequisites for Kubernetes HTTP fixtures or real-runtime gateway, Codex, model, and Secret tests.

Local Kubernetes installation

Build the CLI and run the selected real test to create and clean up a separate k3d cluster:

pnpm cli:buildOCC_TEST_DEV_UP_K3D_REAL=1 node --test tests/integration/dev-up-k3d-real.test.mjs

The test exercises the regular launcher, in-cluster PostgreSQL and OCE, authenticated Namespace readiness, standard Presets, and curated plugin discovery. It provisions a dedicated Codex Agent with a synthetic model Secret and checks workspace-write and outside-write behavior inside its real sandbox. It does not run a model or prove Codex WebSocket tool execution. If cleanup fails, it preserves the recorded state directory for recovery with occ dev down.

See two-cluster validation.

Kubernetes HTTP fixture

Requires Docker, k3d, kubectl, and the migrated openclaw_k8s_local database from PostgreSQL. Create a new disposable cluster; if oce already exists, use a new name consistently throughout these commands.

mkdir -m 700 -p /tmp/oce-k3dk3d cluster create oce \  --image +v1.35 \  --api-port 127.0.0.1:6443 \  --kubeconfig-update-default=false \  --kubeconfig-switch-context=falsek3d kubeconfig get oce > /tmp/oce-k3d/kubeconfigchmod 600 /tmp/oce-k3d/kubeconfig docker build --pull=false -t oce-fixture:local tests/fixtures/kubernetesk3d image import oce-fixture:local -c oce OCC_TEST_KUBERNETES_KUBECONFIG=/tmp/oce-k3d/kubeconfig \OCC_TEST_KUBERNETES_CONTEXT=k3d-oce \OCC_TEST_KUBERNETES_IMAGE=oce-fixture:local \OCC_TEST_DATABASE_URL=postgresql://occ_app:occ-app-local@127.0.0.1:55432/openclaw_k8s_local \  node --test tests/integration/kubernetes-compute-real.test.mjs

All four fixture cases must run: Driver lifecycle/isolation, externally managed namespace preservation, provisioning handoff, and PostgreSQL API-plus-worker reconciliation. No model key is needed. Missing all cluster selectors skips the suite; partial selectors fail, and a missing database skips the API-plus-worker case.

Set OCC_TEST_KUBERNETES_RUNTIME_IMAGE to an imported immutable runtime image reference to extend the API-plus-worker case through real runtime credential Secret and private-state claim deletion. The case uses nonfunctional fixture credentials and performs no model turn.

The tests require an explicit loopback k3d-* context and enforcing NetworkPolicies. They create scoped RBAC and resources and use the stock local-path provisioner for RWO Harness workspaces. The API-plus-worker case verifies that replacement retains the PVC UID and a file written by the old Harness. The HTTP fixture can fail native readiness using a workspace marker; a later deployment must retain both earlier files and writes from the failed candidate. This proves serial replacement on local storage, not cloud CSI detach, node fencing, or data movement between nodes. Use a disposable cluster.

Fixture images and security controls

The disposable tests/fixtures/kubernetes image runs as nonroot and uses the Compute Driver's generated Namespace labels, ResourceQuota, LimitRange, NetworkPolicies, Pod and container security settings, and bounded resources. Its local mutable tag and unpinned docker.io/library/node:24-bookworm base are limited to this disposable fixture; production images still require the documented pinning and review.

The suite checks tenant isolation, resource bounds, nonroot execution, seccomp, dropped capabilities, and a read-only root filesystem. Skipped cases prove no enforcement. API-plus-worker coverage uses synthetic Secrets for binding admission and gateway projection; genuine channel runtime needs the images and credentials below. The Driver lifecycle case clones a workload Pod; removing, emptying or changing its network profile must deny DNS between successful controls.

Live Configuration ConfigMap CRUD and least-privilege RBAC cases require the selected disposable cluster and tenant credentials. Without those inputs, they skip explicitly. Schema, controller, and SDK fixtures do not exercise that live cluster behavior.

The plugin-status fixture tests wait for Driver readiness, a ready gateway Pod, and its plugin status before asserting startup or restart results. A later Pod status read does not establish that an earlier Driver observation was ready. CI preparation first waits up to 120 seconds for the server's route to the worker Pod CIDR to use flannel.1, then admits that route's source /32 for API-server Pod proxy requests. Node readiness alone can precede this route; selecting the container network's default-route source would leave ready Pods unreachable through the proxy. An absent overlay route fails preparation before it publishes the test environment.

Kubernetes model turns and Secrets

Follow Codex sandbox setup for seccomp prerequisites. CI checks workspace writes and outside-write denial; the native workspace case additionally requires tool-history evidence with approvalPolicy: never. Credentialed repository access requires separate proof.

Develop with local containers and k3d

The helper prepares disposable k3d, isolated PostgreSQL, and gateway/Codex images. Start Docker or Podman's API socket (Podman Machine on macOS), then run:

export OCC_TEST_OPENAI_MODEL=gpt-6-astra./scripts/k3d

When OPENAI_API_KEY is not already set, the interactive demo command prompts for it without echoing the value. test requires the variable explicitly; reset does not require it. A prompted value exists only in the helper process and its children; the helper never writes it to state files.

The helper requires k3d, kubectl, Helm, OpenSSL, and Docker Compose or podman-compose. It prefers a running Podman API socket unless DOCKER_HOST selects an engine. Set OCC_K3D_CONTAINER_ENGINE=podman or docker to override detection.

Preparation state is private to the selected engine under ${XDG_STATE_HOME:-$HOME/.local/state}/openclaw-enterprise/k3d-<engine>-codex. Set OCC_K3D_STATE_DIR to an absolute path to override that location. Later runs reuse the prepared cluster, database, images, Envoy Gateway, cert-manager, and the disposable private-routing CA. This helper has no image upgrade command; its demo resources are disposable. For a separate persistent Helm installation, see local k3d image upgrades. The helper builds the current checkout and ignores Kubernetes image selectors inherited from an earlier test shell. Run ./scripts/k3d down before reusing state prepared without workspace routing.

To clear an interrupted test or rerun against a fresh database while preserving the PostgreSQL service, cluster, and imported images:

./scripts/k3d reset

Reset deletes only helper test Namespaces, such as oce-production-*, oce-openshell-*, and oce-ns-*, from the helper-owned cluster. It drops and recreates only the database recorded in the helper's private state.

The default command starts the OCC API in Kubernetes, creates a dedicated Codex Agent, and completes a model turn. It serves the OpenClaw Control UI at http://127.0.0.1:18888 and the OCC console at http://127.0.0.1:18889. The command prints the temporary OCC username and a command to copy its password from the mode-0600 demo.json file, without printing passwords.

Pass --harness openclaw for the verification-only native OpenClaw Harness.

The development login is admin@openclaw.local with openclaw-development-password. Override it with OPENCLAW_DEV_EMAIL or OPENCLAW_DEV_PASSWORD; the database retains the account, so reset before restarting the demo after changing its password. Use ./scripts/k3d get openclaw-control-ui for the Control UI URL and ./scripts/k3d copy openclaw-password for its Gateway secret. This separate password preserves direct loopback access while OCC workspace files use trusted-proxy authentication.

Keep the command running while using either interface. Ctrl-C stops the local controller and worker, closes port-forwards, and removes the private state file and demo Namespaces. The prepared cluster, images, routing controllers, and PostgreSQL remain; rerun ./scripts/k3d to recreate demo resources.

Inspect the current demo and cluster details without parsing the private state files directly:

./scripts/k3d info./scripts/k3d copy openclaw-password./scripts/k3d copy occ-password

info reports the engine, state directory, status, connection values, password copy commands, and host/container processes. get prints a selected non-sensitive value; run ./scripts/k3d help for fields. copy sends either password to the clipboard with pbcopy, wl-copy, or xclip, never to standard output. The OCC console's Workspace files panel uses the same private Envoy route exercised by the focused gateway-routing integration. Demo fields become available after the foreground command reports readiness. Cluster fields remain available while its prepared state exists.

To run the dedicated Codex gateway-routing integration instead:

./scripts/k3d test

The test proves model turns, OCC workspace access through Envoy, routing credential enforcement and rotation, certificate renewal, Pod replacement, and workspace retention. It does not cover credential recovery or embedded OpenClaw.

Remove only resources recorded in the helper's owned state when finished:

./scripts/k3d down

If preparation fails, run the same cleanup command before retrying. The helper does not use or modify the default kubeconfig, active context, the development database on port 55432, or unrelated container-engine resources.

Use the disposable cluster and openclaw_k8s_* database above, an exported OPENAI_API_KEY, and approved real gateway/Codex images. Import local image tags, then register their corresponding immutable references inside k3s. Replace the placeholders with the exact tags and digest references for your images:

k3d image import '<local-gateway-tag>' '<local-codex-tag>' -c ocedocker exec k3d-oce-server-0 ctr -n k8s.io images tag \  '<imported-gateway-image>' '<gateway-image>@sha256:<digest>'docker exec k3d-oce-server-0 ctr -n k8s.io images tag \  '<imported-codex-image>' '<codex-image>@sha256:<digest>'

Prepare a private runtime environment file with the model key and these nonsecret settings, using the actual digest references:

OCC_TEST_KUBERNETES_KUBECONFIG=/tmp/oce-k3d/kubeconfigOCC_TEST_KUBERNETES_CONTEXT=k3d-oceOCC_TEST_DATABASE_URL=postgresql://occ_app:occ-app-local@127.0.0.1:55432/openclaw_k8s_localOCC_TEST_KUBERNETES_GATEWAY_IMAGE=<gateway-image>@sha256:<digest>OCC_TEST_KUBERNETES_AGENT_IMAGE=<codex-image>@sha256:<digest>OCC_TEST_KUBERNETES_PLUGIN_STATUS_PROXY_CIDRS=<api-server-proxy-source>/32OCC_TEST_OPENAI_MODEL=gpt-6-astra

The startup failure cases use dedicated Codex with plugins enabled and disabled. They assert the saved failure through deployment GET after failed Pod deletion and controller restart. This proves retained startup evidence, not live health. Set the private status proxy CIDRs to the actual API-server Pod-proxy source; CI preparation supplies them. For manual clusters, follow the networking setup.

Private workspace-file routing has a separate gateway-routing suite with additional Envoy Gateway, cert-manager, and test-CA setup.

Run the ordinary runtime cases independently of Slack:

OCC_TEST_HARNESS_K3D_REAL=1 OCC_TEST_SLACK_LIVE=0 \  node --env-file="$TEST_ENV_FILE" --test tests/integration/harness-topology-k3d-real.test.mjs

Three non-Slack runtime cases must pass: dedicated Codex, embedded OpenClaw, and the extended Secret lifecycle case. Both topologies use OCC Secret-backed Agent harnessAuth bindings. The Secret API case verifies native SecretRefs, exact grants and denial, shared Secrets, rotation, and redeployment. It prepares those Secrets and grants itself. Routing, Slack and OTLP cases live in separate files, so this invocation contains only its three required runtime cases.

The ordinary suite uses the real production API and worker in the Node test process. The gateway-routing suite runs the API as a Kubernetes Deployment so it reaches Envoy through the normal ClusterIP Service endpoint; its worker and test coordinator remain in the Node test process. Neither suite installs the controller with Helm. Missing selected-suite prerequisites fail; an unselected suite skips. Default Codex version expectation is 0.158.0; see runtime settings for version assertions and alternate image variables.

Transcript persistence

Both Harness topologies require a gateway image that stores transcripts in SQLite. The persistence cases query the test conversation through session_nodes and transcript_events, then verify its history and media after gateway Pod replacement. An older image that writes JSONL transcripts cannot exercise this storage path, even if it uses SQLite for authentication or memory. Setting OCC_TEST_KUBERNETES_OPENCLAW_VERSION alone does not verify transcript storage behavior.

For Secret changes, run the API and PostgreSQL suites as well as the real Kubernetes runtime cases. Route/schema checks and documentation checks alone do not verify Kubernetes Secret storage and delivery. These suites exercise the selected disposable resources; deployments need their own runtime verification.

Kubernetes fixture test environment

Real-cluster integration is opt-in for ordinary development and required when explicitly requested or validating the production-capable Kubernetes driver for release. The test harness requires a dedicated loopback-only k3d context. The driver itself also supports verified remote HTTPS API servers and in-cluster ServiceAccount authentication. These variables do not configure server.mjs, worker.mjs, the normal controller, or its default Compute Driver.

CI selects the Kubernetes 1.35 family so the fixture proves the supported minimum line; a manually selected server must be 1.35 or later. The test exercises the real version endpoint through its scoped controller identity before creating tenant resources.

The HTTP fixture procedure covers setup and PostgreSQL-backed testing; the API-and-worker case rejects the ordinary openclaw_enterprise database. The fixture does not prove a real gateway, authenticated Codex connection, or model turn; use the real-runtime suite for model-turn proof.

CI keeps the project-pinned k3d 5.8.3 binary and passes --image +v1.35 when it creates ordinary disposable clusters. k3d resolves the K3s v1.35 release channel at cluster creation, so these lanes follow the current Kubernetes 1.35.z patch rather than one immutable node image. Preparation rejects a server outside the 1.35 family. The CI kubectl client is pinned to 1.35.0. The separately prepared OpenShell lane retains its own pinned K3s and kubectl versions.

Kubernetes real-runtime test environment

harness-topology-k3d-real.test.mjs is independently opt-in. Set OCC_TEST_HARNESS_K3D_REAL=1 or explicitly select a real runtime image to enable the ordinary runtime suite. Once selected, missing cluster, image, database, credential, or NetworkPolicy prerequisites fail instead of skipping. The ordinary suite verifies dedicated Codex, embedded OpenClaw with a persisted provider credential, and embedded OpenClaw with the Secret API through real Enterprise gateways on an explicitly selected disposable k3d cluster. It does not prove Agent workspace-file private routing until Compute HTTPRoutes, real Envoy Gateway, cert-manager, OCC, and the native Agent runtime are tested together. The default is gpt-6-astra; for dedicated Codex coverage, any OCC_TEST_OPENAI_MODEL override must be authorized and support Codex custom tools.

The separate harness-topology-k3d-routing-real.test.mjs requires OCC_TEST_GATEWAY_ROUTING_REAL=1 and the same runtime prerequisites. It also requires ready Envoy Gateway and cert-manager controllers, OCC_TEST_GATEWAY_CA_CERT_PATH, OCC_TEST_GATEWAY_CA_KEY_PATH, NODE_EXTRA_CA_CERTS, and an imported OCC_TEST_PRODUCTION_CONTROLLER_IMAGE. The suite starts the OCC API in Kubernetes and uses the Envoy ClusterIP Service on its standard HTTPS port; no host Envoy port is published. Controller namespace overrides are OCC_TEST_ENVOY_GATEWAY_NAMESPACE (default envoy-gateway-system) and OCC_TEST_CERT_MANAGER_NAMESPACE (default cert-manager). See the focused routing proof for the disposable CA and command. The CA private key is test setup only; the production OCC API mounts only a public trust bundle.

Related

• Choose another test suite.
• Results, cleanup, and troubleshooting.
• Network access.

Dedicated Gateway placement

Dedicated Gateway resources now live in the logical Namespace's managed Gateway runtime namespace. Fixture bootstrap must grant the worker scoped access there as well as in the Harness namespace before waiting for Namespace readiness. Runtime helper results expose gatewayPlacement separately from placement. Use the former for Gateway Pods, routes, private PVCs and port-forwards; use the latter for Harness execution, model credentials and workspace storage.

Disposable runtime helpers accept OCC_TEST_KUBERNETES_GATEWAY_NODE_SELECTOR as a JSON selector and default to Linux nodes. The default tests namespace and credential separation; it does not prove production node-pool isolation. Configure separate reviewed node pools for that proof. Current tests must still pass with the actual supported Gateway/Codex images and authenticated node reconnect; fixture readiness is not a substitute for model-backed acceptance.

Production observability

See smoke tests and model-log validation.