QA Matrix Advisory

Overview

Scheduled QA runs Console Secret management on Compose/Kubernetes, OpenShell security and lifecycle files, local installation, and OIDC Agent chat. Jobs use real dependencies on disposable installations. Results are advisory, not merge or publication gates.

Trigger

Maintainers label reviewed same-repository PRs. Dependabot authors and triggers are ineligible. Narrow labels select their union. full-qa overrides them and runs the default Codex/console-secrets cells. The focused Compose job checks native redeployment and required-worker execution on Kubernetes workloads.

PR runs test the merge commit. New commits replace opted-in runs; unrelated labels cannot restart or cancel them. Remove labels before pushing unreviewed code; removal stops future selection but does not cancel active runs. Eligible main pushes finish independently. The author of a breaking change owns triage.

Keycloak needs no model credential and allows 180 minutes; OpenClaw allows 120. Together they allow 300 runtime runner-minutes per attempt, plus selection—not a duration or billing estimate. Both require sandboxed packaged Chrome.

OpenClaw requires OPENAI_API_KEY and nonblank OCC_TEST_OPENAI_MODEL in integration-qa-pr, mapped to OCC_TEST_QA_OPENAI_MODEL. Missing inputs fail before the lane. Each cell materializes only its provider credential, removed after success or failure.

Push execution is opt-in through QA_ADVISORY_ON_PUSH; the schedule does not require that variable. Credentialed QA and OpenShell jobs use integration-qa-pr. Dependabot cannot start a manual advisory run.

Scenarios

The QA matrix creates separate Agents on disposable control planes with Kubernetes compute, preserving existing clusters and kubeconfig. Both Plugin catalogs stay enabled across presets; setup checks Google Workspace for OpenClaw and Linear for Codex.

QA matrix and scenario selection own these checks.

Scenario details

To run OpenShell scenarios locally, prepare the matching fixture for network policy, scratch storage, or plugin installation.

The OpenClaw model-ui case checks the example Console theme through sign-in and Agent use. It also redeploys the Agent with a provider alias, the existing OCC Secret, and OCE_MODEL_API_KEY, then requires a real model turn.

Harness destination overrides

The explicit harness-overrides QA scenario runs dedicated Codex and OpenClaw through an installed Kubernetes/OpenShell workflow. It deploys two Agents with distinct authorized destinations and a third inheriting the default, performs real model turns, restarts the controllers with changed and removed defaults, and withdraws one Agent's source while the other Agents continue serving.

Select OCC_TEST_QA_INSTALLATION=kubernetes, OCC_TEST_QA_PRESET=all, and OCC_TEST_QA_SCENARIOS=harness-overrides. Provide two authorized HTTPS port-443 endpoints in OCC_TEST_QA_HARNESS_ENDPOINT_A and _B, protected API-key file paths in OCC_TEST_QA_HARNESS_KEY_FILE_A and _B, and a model authorized at both endpoints in OCC_TEST_QA_HARNESS_MODEL. Use the normal source-built QA images and disposable cluster prerequisites. Missing inputs fail before setup. This scenario is explicit until hosted credentials for both destinations are configured; ordinary PR checks run HTTP and PostgreSQL override coverage. The scenario does not collect independent upstream access logs.

OpenShell file coverage

OpenShell needs Landlock, unavailable on QA's Blacksmith runners. Each file runs on an Ubuntu 24.04 VM with a Landlock check, Docker daemon, cluster and private state. Job names use QA advisory (OpenShell / <test-file-basename>).

Each OpenShell manifest file gets a run-ci-lane job with lane: openshell and its selected test-file. The qa-openshell-compose label selects only the first row below.

For finite Git conversation checks, persisted reply; read=… means the model returned the expected nonce. The readback category identifies the failed check. For example, answer_missing means the reader found the prompt but no matching assistant answer in its bounded transcript window. The prompt and terminal answer must belong to the logical run returned by the Gateway. Codex can finish that run on a new native turn after an internal retry. The check sends each authenticated nonce prompt once.

The runner requires the selected files' credentials, expected tests, and cleanup. Local installation also uses one runner per manifest file: input and network-policy refusals, Kubernetes and Compose profiles, and Keycloak sign-in and recovery. The six original installation cases remain required. A failed job does not cancel its peers. Hosted preparation builds first-Agent images from the current checkout using read-only caches. Local Setup then creates and cleans up each installation without duplicate clusters. See OpenShell CI preparation and timing for image selection, runner requirements, and interpretation of setup and model-execution timings.

Full Integration can also run this suite with lane=openshell or lane=all; that workflow runs the same files serially. The separately prepared OAuth and Linear tests are not included in advisory coverage.

Execution

To run the same coverage as the schedule, configure the hosted credentials, then start the workflow:

gh workflow run qa-advisory.yml --repo openclaw/openclaw-enterprise --ref main

For only OpenShell, use Full Integration with lane=openshell.

Local instructions run the QA matrix only. For other suites, follow OpenShell setup or local installation tests.

Select scenarios

OCC_TEST_QA_INSTALLATION accepts all (default), compose, kubernetes; OCC_TEST_QA_PRESET accepts all (default), OpenClaw, Codex. OCC_TEST_QA_SCENARIOS accepts all (default) or comma-separated selectors. all excludes namespace-lifecycle, sandbox-policy, scratch-volume, plugin-install, openshell-linear-read, codex-search-config, harness-overrides and oidc-agent-chat selectors. Invalid or wholly inapplicable selections fail before provisioning; missing inputs fail selected checks. Omitting OCC_TEST_QA_MATRIX=1 skips the suite.

Select oidc-agent-chat alone with kubernetes and OpenClaw. It uses the gateway-routing lane to prepare Keycloak, PostgreSQL, native routing and runtime images. Follow the native browser proof for inputs, execution and proof limits. It does not use the ordinary matrix's installation launcher.

Select scratch-volume with kubernetes and Codex. The fixture uses an OpenAI API key for https://api.openai.com/v1. NVMe and node-loss recovery need separate proof.

Select codex-search-config alone. It requires OCC_TEST_QA_CODEX_MODEL_BASE_URL, OCC_TEST_QA_CODEX_MODEL, and an API key through OCC_TEST_QA_MODEL_API_KEY or OCC_TEST_QA_OPENAI_KEY_FILE. Environment keys are redacted and excluded from launcher children; key files must be private regular files, not symlinks. This scenario checks configuration persistence and native startup, not live web-search results. Its API-key setup differs from the scheduled account-login tests.

Ordinary matrix cells check startup and authenticated console login. Only Git scenarios start the broker. openshell-linear-read instead starts its own OpenShell stack and needs a connected identity and Linear read fixture. OCC_TEST_QA_CONCURRENCY=1..4 selects workers (default 2); installations and Slack run sequentially. Allow up to 4 GiB per gateway beyond shared services.

Local inputs

Prepare Kubernetes and Compose: workspace dependencies, Docker, k3d, kubectl, Helm, Go, OpenSSL, Chromium/Playwright, and certutil. Store selected inputs in a private environment file. Credential files must be regular files without group/other permissions. Hosted settings are under integration environments.

Slack sender and gateway bot must be distinct channel members; the sender needs post/history/reply access. Reserve the Socket Mode app for this run.

Git's private input directory contains registry.json, private-key.pem, and upstream-cidrs.json. Follow fixture setup: exactly one authorized repository, ${OCC_INITIAL_NAMESPACE_ID}, approved IPv4 /32 endpoints, and run-branch push permissions. The run grants git-write with git-only transport on one branch, oce-qa-<run>, and opens no PR. The run generates the worker's custody key. OCC_TEST_QA_GITHUB_OBSERVER_APP_INPUT_DIRECTORY instead supplies an isolated observer App's registry.json and private-key.pem. Observer credentials stay outside Agents and clean only verified run-owned refs.

OCC_TEST_QA_ARTIFACTS selects private results; otherwise the runner prints its temporary path. Set OCC_TEST_QA_CONTROLLER_IMAGE and OCC_TEST_QA_RUNTIME_IMAGE together to reuse prebuilt images; OCC_TEST_QA_REPOSITORY_IMAGE selects the broker. Model defaults are in shared configuration; the manifest lists overrides.

Prebuild local QA images

Prebuild Kubernetes QA images from a clean checkout with immutable references and matching source labels. Use the same local Unix-socket Docker context with the containerd image store for both build and QA. Use Buildx, Node.js 24+, and CLI and daemon support for docker image save --platform (API 1.48+). The commands select the daemon's Linux amd64 or arm64 platform.

In a fresh Bash shell at the repository root, replace the builder name with an existing builder you own. No QA credentials are needed.

set -euo pipefailumask 077qa_builder="your-existing-builder"qa_platform="$(docker version --format '{{.Server.Os}}/{{.Server.Arch}}')"[[ "$qa_platform" == linux/amd64 || "$qa_platform" == linux/arm64 ]]unset OCC_TEST_QA_CONTROLLER_IMAGE OCC_TEST_QA_RUNTIME_IMAGEqa_commit="$(git rev-parse HEAD)"qa_status="$(git status --porcelain --untracked-files=all)"[[ -z "$qa_status" ]]qa_metadata="$(mktemp -d)"qa_prefix="localhost/oce-qa-$(date +%s)-$$"qa_node_base="$(node -p 'require("./scripts/ci/test-suites/images-packaging.json").prepare.defaultEnv.NODE_BASE_IMAGE')"qa_source="$(node scripts/lib/openclaw-source.mjs build-args)"qa_runtime_args=()while IFS= read -r argument; do  qa_runtime_args+=(--build-arg "$argument")done <<< "$qa_source"docker buildx inspect "$qa_builder" for role in controller runtime; do  qa_recipe=(--target runtime)  if [[ "$role" == runtime ]]; then    qa_recipe=(-f deploy/runtime/Dockerfile "${qa_runtime_args[@]}")  fi  docker buildx build --builder "$qa_builder" --platform "$qa_platform" \    --load --provenance=false --sbom=false "${qa_recipe[@]}" \    --build-arg NODE_BASE_IMAGE="$qa_node_base" \    --build-arg OCC_BUILD_REVISION="$qa_commit" \    --label "org.opencontainers.image.revision=$qa_commit" \    --metadata-file "$qa_metadata/$role.json" \    --tag "$qa_prefix/$role:source" .  qa_digest="$(node -p 'JSON.parse(require("node:fs").readFileSync(process.argv[1]))["containerimage.digest"]' "$qa_metadata/$role.json")"  [[ "$qa_digest" =~ ^sha256:[a-f0-9]{64}$ ]]  qa_ref="$qa_prefix/$role@$qa_digest"  [[ "$(docker image inspect --format '{{.Os}}/{{.Architecture}}' "$qa_ref")" == "$qa_platform" ]]  [[ "$(docker image inspect --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}' "$qa_ref")" == "$qa_commit" ]]  if [[ "$role" == controller ]]; then    CONTROLLER_IMAGE="$qa_ref"  else    RUNTIME_IMAGE="$qa_ref"  fidoneqa_status="$(git status --porcelain --untracked-files=all)"qa_final_commit="$(git rev-parse HEAD)"[[ "$qa_final_commit" == "$qa_commit" && -z "$qa_status" ]]printf 'Build metadata: %s\n' "$qa_metadata"

Failed digest resolution or inspection stops the procedure. Do not substitute tags or invent digests. These images omit publication attestations; source labels prove consistency, not provenance. Retain metadata through qualification. Keep CONTROLLER_IMAGE and RUNTIME_IMAGE unchanged. In the image checks, skip both pulls and run the command starting with OCC_TEST_PRODUCTION_IMAGE=. After all required cases pass without skips, select the pair:

export OCC_TEST_QA_CONTROLLER_IMAGE="$CONTROLLER_IMAGE"export OCC_TEST_QA_RUNTIME_IMAGE="$RUNTIME_IMAGE"

Reuse the builder's intermediate cache, including Node compilation, with consistent platform and build arguments. Loading final images does not restore intermediate cache. For portability, use Docker's cache export/import options; mode=max includes intermediate stages. Hosted CI cache is separate. These commands do not export cache, change networking, or select a global builder.

Run local QA

After preparing those inputs, run from the repository root:

pnpm cli:buildOCC_TEST_QA_MATRIX=1 node --env-file="$TEST_ENV_FILE" \  --test tests/integration/qa-matrix-real.test.mjs

TEST_ENV_FILE is the private environment file described above.

Results and limits

Check the tested commit, selected scenarios, failed or skipped tests, and cleanup. Missing, filtered, or interrupted tests are not passes. The selection summary records the run, attempt and tested SHA. Focused PR runs also verify and record the merge ref and ordered base/head parents before starting jobs. Compare the leaf result's sourceSha with that merge SHA; a passing earlier head is not proof of the current composition.

For focused Keycloak, require both manifest cases and their cleanup to pass. For OpenClaw, inspect matrix.json: one kubernetes/OpenClaw cell with only model-ui, successful real model/browser stages, and Agent/installation cleanup. Its partial:selected scope is intentional; missing stages or artifacts leave verification incomplete. Preserve the job link and attempt-specific results.

Startup failures report elapsed time, the 1,100,000 ms limit, and the observed exit code or signal. timed out identifies deadline expiry. A later SIGPIPE can follow teardown closing output pipes. Failure detail is redacted and command arguments and protected output stay private.

When available, startup timings in the failure message names a JSON file in the same artifact. It retains the last 64 recognized CLI timing records: stage, elapsed milliseconds, and failure status, plus the number of older records omitted. These describe completed commands, including commands that failed; they do not identify an unfinished command. Durations can overlap, so do not sum them as wall time. Unknown or malformed records and other stdout remain private. Commands with protected output produce no timing artifact.

Artifacts expire after seven days. Each OpenShell job reports scope: focused because it runs one file; complete advisory OpenShell coverage requires every manifest file to pass.

In matrix.json, scope: full means both installations and presets with the all scenarios; advisory QA has narrower coverage. See Select scenarios for the exclusions from all.

First-Agent timing results include overlapping stages; do not sum them as elapsed time. A Codex timeout alone does not permit a retry: the client must report model capacity exhaustion before any action started.

Cleanup stops the Agents and installations created by the run. If credential disposal is uncertain, cleanup keeps the installation and reports the required recovery. Keep its broker until sessions are DISPOSED without active uses or pending, uncertain or auxiliary cleanup. Preserve other runs' resources. For Git TLS errors, compare broker DNS with approved endpoints/NetworkPolicy, refresh approved inputs, complete disposal, then recreate only this run's installation.

Files

• Advisory workflow: eligibility, parallel OpenShell runners, and artifact names.
• QA manifest and OpenShell manifest: exact files and requirements.
• QA matrix test: Agent scenarios.
• Scenario selection: presets, selectors, and required inputs.
• Hosted credentials: environment preparation.