GitHub Actions testing

Select automated or manually dispatched test lanes and understand the coverage reported by each workflow.

GitHub Actions

Metrics HTTP/persistence coverage belongs to the postgres-application lane, including a separate migrator-role connection for test-only table contention. The logging-collector lane also runs real Prometheus/Grafana collection and dashboard provisioning. See metrics testing for local setup.

The suite index holds ordered lane references and coverage groups. Each scripts/ci/test-suites/<lane>.json owns that lane's test files, required inputs, environment, and preparation resources. Edit the owning lane file when adding or renaming tests; update the index when adding a lane or changing a group. The shared loader assembles these definitions for the runner and preparation tools. Check that every active test file has exactly one lane owner:

node scripts/ci/run-tests.mjs audit

CI workflows reuse the run-ci-lane action for setup, tests, cleanup, and per-job environment isolation.

Compare per-file wallDurationMs, preparation [ci-timing] phases, and Actions step timestamps to find slow setup or tests. Preparation timings include image archive save/import times. Image imports copy the archive into each owned k3d node and run node-local ctr image import; k3d tools-node can hide per-node failures while exiting successfully. Imports targeting one cluster stay serialized, then preparation verifies digest and CRI references on owned nodes.

The checks-baseline lane runs pnpm docs:check: pages above 1,500 visible words are flagged for review and pages above 2,500 fail, except the approved API reference and AGENTS.md instruction files. The generated API, site build, navigation, and links must pass. Run pnpm docs:check-length for the word-count check alone.

The checks-baseline lane runs the dependency policy.

Suite Audit and the eleven PR lanes start independently on ephemeral runners. Kubernetes fixture and observability lanes use ubuntu-22.04 for bridge netfilter support. The repository credential platform lane uses blacksmith-16vcpu-ubuntu-2404 because it builds the delivered runtime image and the repository platform fixture in one job; other lanes and the audit use blacksmith-8vcpu-ubuntu-2404. CI Required uses ubuntu-22.04 and still requires the audit and every lane to pass, including result-artifact accounting. This avoids serial runner allocation before test lanes without changing selection or failure handling.

The repository credential platform lane proves HTTP, PostgreSQL, Unix control and credential material inside Kubernetes; compatible fixture lanes prove NetworkPolicy enforcement. The images packaging lane uses the full tool profile so preparation can derive the reviewed Codex seccomp profile in an owned k3d cluster and export OCC_TEST_CODEX_SECCOMP_PROFILE before the native runtime image smoke tests run.

Full Integration is manual and uses the immutable event commit. All lanes require main except k3d-model, which also accepts an integration-model branch allowlist. Environment gates apply only to lanes that declare one; helper-timeout and standalone logging-collector declare none. The ChatGPT provider-account lane stays main-only without per-run approval. Other model, routing, Slack, OpenShell, and additional OpenTelemetry lanes require approval. Missing selected prerequisites fail. A PR aggregate is not full credentialed coverage; targeted protected runs also report only their selected lanes.

The postgres lane owns migration compatibility tests; postgres-application owns the remaining PostgreSQL files. Each has its own disposable PostgreSQL server. Kubernetes fixture files run in k3d-fixture-configuration, k3d-fixture-state, and k3d-fixture-plugins, each with independent cluster, database, image, and cleanup state. Files still execute sequentially inside a lane. The suite audit requires every file to have exactly one owner, and both workflow aggregates require all eleven lanes.

The repository-credentials-container lane builds .build/repository-credentials/{service,client} using Dockerfiles under deploy/runtime/repository-credentials/ and records source, gh version and three image IDs. It selects immutable IDs through REPOSITORY_CREDENTIALS_TEST_IMAGE, REPOSITORY_CREDENTIALS_SERVICE_IMAGE and REPOSITORY_CREDENTIALS_CLIENT_IMAGE; its real Git/gh fixtures also receive REPOSITORY_CREDENTIALS_NODE_IMAGE and the extracted, version-checked REPOSITORY_CREDENTIALS_GH_BINARY. The credential test guide separates detached artifacts, the combined image, rendered Compose, running container isolation and authorized live proof. CI preparation and suite ownership alone establish no result: inspect executed cases and skips at the exact tested commit, including whether a pull-request run tested a merge commit.

The Kubernetes fixture lanes load bridge netfilter and enable IPv4 bridge filtering before cluster creation so K3s enforces NetworkPolicies on bridged Pod traffic. Setup fails if this cannot be enabled; deny-traffic assertions remain required.

Each Kubernetes fixture lane owns an independent server/worker cluster with shared test-owned local-path storage. Preparation registers and verifies the fixture image's digest on both nodes and derives the API server's proxy source /32 from its route to the worker Pod network. It supplies that address to the plugin status tests, which exercise the private status endpoint across nodes with NetworkPolicy enforcement.

Kubernetes fixture startup logs phase timings plus host resource and pressure snapshots. On cluster or readiness failure, preparation collects bounded node, system Pod, event and redacted node-container diagnostics before cleanup; k3d rollback is disabled long enough to keep those logs. Inspect the diagnostics-<artifact-prefix>-<lane> artifact or local <state-file>.diagnostics.json. Failed diagnostic commands are marked unavailable or timed out; collection preserves the original failure. Raw kubeconfig, environment values and Pod specs are excluded. Local callers must run node scripts/ci/cleanup.mjs --state <state-file> after failed prepared runs. Diagnostics explain setup failures without establishing coverage.

The k3d-model, gateway-routing, slack, and k3d-otel lanes prepare the controller image and workspace routing for dedicated Harness node enrollment. Supply an immutable NODE_BASE_IMAGE for the controller build. Preparation supplies the imported controller digest and private routing CA paths; the Slack lane still requires approved runtime images and credentials.

Implementation status: routing, OpenShell, and logging have concrete CI preparation contracts. Routing installs pinned Gateway API, cert-manager v1.18.4 and Envoy Gateway v1.6.7 manifests, then generates a private test CA. OpenShell creates an owned K3s v1.36.4 cluster, installs a matched kubectl, configures and smoke-tests the selected RuntimeClass with the cluster's runc handler, installs CLI/chart and Agent Sandbox assets, and imports gateway and supervisor images. Only that disposable cluster exempts the selected RuntimeClass from Pod Security Admission; preparation proves an ordinary violating Pod is rejected and the same Pod is admitted with the selected class. The full OpenShell suite proves provider-owned supervisor filesystem, endpoint/L7 network, and process enforcement while the sidecar policy remains binary-unaware. Logging preparation owns a real OpenTelemetry Collector backend with JSONL evidence, so OCC_TEST_OTEL_LOGS_URL is no longer an external input. The Collector and Docker-model jobs use the shared setup-test-docker action to pin Docker 29.4.0 for the production fluentd-write-timeout option. The action replaces the preinstalled daemon and shares /var/run/docker.sock across the CLI, Compose, and Driver; other jobs keep the runner daemon. Full-suite acceptance still requires main-only protected hosted execution of every selected lane. See the delivery status for proof boundaries and live gaps.

Each lane runs whole test files. The runner validates actual Node case results and required names; any skip or TODO fails a selected lane. Missing results, zero cases, failures and cleanup errors also fail. The aggregate checks required job and lane results at the same source commit without repeating case validation. Ordinary pull_request jobs may save pnpm-store caches within the PR merge-ref scope; protected jobs use the approved event commit and do not promote PR build artifacts.

Prepare infrastructure only on a disposable host or through reviewed CI helpers. Each run owns its Compose project, databases, cluster and temp files. CI writes private cleanup state under RUNNER_TEMP and uploads sanitized results plus bootstrap diagnostics; hosted-runner cleanup state disappears after the job. The images-packaging lane attempts to retain sanitized cleanup records for its prepared controller and runtime tags; a planned record does not prove an image exists. It does not cover the separate tag created by the runtime-images test. Export or upload failure and runner loss can prevent retention. Missing state or an empty inventory does not prove cleanup; the tag name is metadata, not authentication or authority to delete an image. Results include source commit, case outcomes, cleanup status and available image digests by role, excluding private registry names and prepared environment values. Local failures can retain cleanup state while the host and state path exist. On Docker Desktop or similar VM-backed hosts, run one Kubernetes lane at a time when measured disk or network pressure has caused instability. Model/service tests still require the approved credentials and spend policy in the implementation specification. The GitHub matrix remains parallel.

See the execution flow for entrypoints, result accounting, cleanup and failure interpretation. Use the suite-specific guides to reproduce a run locally.

Failed browser tests upload diagnostics.

A lane retry replaces that lane's result artifact within the workflow run so the aggregate reads its latest result. Other lanes keep their existing artifacts. Preserve a failed result before retrying if it is needed for investigation; earlier attempt logs remain available. Reruns still require every selected lane and the aggregate to pass.

Select immutable images for local preparation

Set OPENCLAW_CI_K3S_IMAGE to an approved image@sha256:<digest> reference before running node scripts/ci/prepare.mjs --lane <lane> --state <private-state-file> to bypass k3d's online release-channel lookup. Ordinary Kubernetes lanes default to the +v1.35 channel when this variable is absent. Both paths require the running API server to report Kubernetes 1.35.x. OpenShell retains its separately pinned cluster image. Invalid mutable overrides fail before resource creation. Clean up a failed run's owned resources before preparing again with its state path.

Supplied immutable workload images can already exist in the local Docker daemon. Preparation reuses one only when docker image inspect records the requested digest in RepoDigests; a mutable tag or unverified local image is insufficient. Missing or mismatched images are pulled and checked again before import. Other Docker inspection failures stop preparation. Cleanup removes owned import tags and preserves the supplied source image.

On GitHub-hosted runners, both observability lanes remove unused SDKs and require 36 GiB free before building and importing images. SDK removals run concurrently with a ten-minute deadline and per-directory timing receipts. Local runs do not invoke this guarded cleanup. Both use single-node clusters and overlap independent pulls, builds, and cluster setup, then serialize k3d imports for each cluster to avoid shared importer races. The demo lane imports only its three services and a Node image for protocol fixtures; it does not build OCC. State writes remain serialized, and all in-flight operations settle before failure cleanup.

Image imports time out after ten minutes. Preparation verifies each immutable reference on every schedulable node. Errors or timeouts fail preparation; normal lane cleanup removes the owned cluster and partial imports.

Integration coverage by trigger

The CI workflow runs eleven noncredentialed lanes on pull requests, pushes to main, merge groups, and manual dispatch. Full Integration runs only through manual dispatch, using the requested lane or all. The k3d-model branch exception below does not enable other lanes outside main. It does not run on pushes or merges. The provider-account lane remains manual because its configured admin credential cannot authenticate from the hosted runner.

Run Kubernetes model tests before merge

A repository administrator must add the exact branch name to the integration-model environment's deployment branch rules, retaining main, required reviewers, and self-review prevention. Wildcard rules do not satisfy the preflight. This grants the reviewed branch access to the existing model credential only after a reviewer approves the run.

gh workflow run full-integration.yml --ref '<approved-branch>' -f lane=k3d-model

The reviewer must inspect the run's commit before approval. Every job checks out that immutable github.sha; moving the branch does not change an existing run. The dispatcher cannot approve their own run. Have a different collaborator perform one of those actions. Remove the branch rule after the proof completes. Other lanes, including all and provider-account, remain main-only. This lane runs the real Kubernetes topology tests, including embedded invalid-credential cutover and recovery. It also runs the local first-Agent proof: a fresh installer deploys and reuses their own Agent, verifies real model responses, and cannot replace the credential after external changes. Ordinary fixture CI does not run these tests.

Integration tests outside automatic CI

The following integration files have no automatic workflow entrypoint. A green CI Required check does not establish their coverage. This inventory describes workflow selection, not whether a test has ever passed in a local or hosted run.

Manual Full Integration lanes

In GitHub Actions, these ten files run only when explicitly selected in Full Integration, using the listed lane or all. The model/service lanes require their configured credentials and infrastructure. All credentialed lanes except provider-account require protected-environment approval; provider-account remains restricted to main without per-run approval. helper-timeout has no environment approval gate; it is separate because it spends five minutes testing the real helper deadline.

No GitHub workflow entrypoint

dev-up-k3d-real.test.mjs belongs to the CLI-only dev-up-k3d lane, outside both workflow groups and Full Integration dispatch. See run the local installation lane.

repository-credentials-k3d-real.test.mjs belongs to the explicitly selected repository-credentials-installed CLI lane. It is excluded from both workflow groups and Full Integration dispatch options. Follow the installed repository credential qualification procedure for protected App inputs, authorized live writes, model execution, and cleanup.

repository-credentials-live.test.mjs belongs to the repository-credentials-live lane, excluded from both workflow groups and Full Integration dispatch options. Follow the repository credential qualification guide for the authorized disposable repository, protected service setup, and cleanup. The automatic container lane exercises controlled provider behavior and separate container credential isolation. A passing run establishes only its selected checks at its recorded source and images; it does not establish installed platform or live-provider qualification.

postgres-azure-workload-identity.test.mjs belongs to the postgres-azure-workload-identity lane, excluded from both the ci and full groups and from Full Integration dispatch options. Follow the Azure PostgreSQL test procedure for private input setup and result handling. The ordinary constructor, security-rejection, and password cases in postgres-connection-auth.test.mjs run in the mandatory postgres lane.

ssh-compute-real.test.mjs belongs to the ssh-host lane, which is excluded from both the ci and full groups and is not a Full Integration dispatch option. No current workflow provisions its disposable Linux/systemd SSH host or invokes that lane. The readiness-only selector proves real-host readiness, revision cutover, state isolation/persistence, and deletion without a model call. The optional OCC_TEST_SSH_MODEL=1 selector adds real provider execution and runtime credential proof. Follow SSH raw hosts for the disposable host, required environment settings, and direct test command.

Related

• Choose another test suite.
• Results, cleanup, and troubleshooting.

Production observability lane

k3d-observability checks raw metrics and OTLP exports in ordinary PR/main CI on Ubuntu 22.04. The separate Observability Demo workflow runs k3d-observability-demo for relevant changes, merge groups, and manual dispatch; Full Integration includes it with all. Both retain strict case counts, image digests, and cleanup.

Gateway/Codex model-log proof remains in protected k3d-otel; ordinary CI does not establish it. See local commands, scope, and prerequisites.