Rehearse a populated controller return

Use this local test to check whether the previous OpenClaw Control Plane (OCC) controller can serve a draft that the candidate saved after recovering an interrupted upgrade. The test creates a new installation and PostgreSQL instance, then uses the local image upgrade command to change the controller image.

Prepare the cluster and images

Use native Linux with rootful Docker 29 or newer at /var/run/docker.sock. Install k3d, Go, kubectl, Helm, OpenSSL, jq, yq v4, and Python 3. From the test checkout, install the workspace dependencies.

Create a fresh, disposable k3d cluster for this run. Use one server, no agents, and these k3d cluster create flags:

--no-lb --api-port 127.0.0.1:<free-port>--kubeconfig-update-default=false --kubeconfig-switch-context=false

Keep the cluster exclusive through cleanup. The test derives a private kubeconfig from the cluster and checks that the API endpoint matches the local Docker server container. You are responsible for the cluster's freshness and exclusive use.

Stage two distinct, compatible controller images in Docker and k3d. Use verified publications or transfer the OCI artifacts from successful native image checks. Public registry publication is not required. For transferred artifacts, retain and verify native export evidence before loading the images; the source revision label alone does not establish approval.

Use owned local image names with their actual SHA-256 digests. Load the same verified OCI archive into Docker and the owned k3d node's containerd, retaining its original manifest or index target. Do not use an export that replaces an index with a selected platform manifest. Verify the imported image identity: the rehearsal compares Docker's reported root digest with the selected digest, hashes the selected platform manifest and configuration, then checks the running controller against those identities. Stop if transport changes or loses the expected identity; do not assign a published name to an unverified local image. Keep one approved runtime image fixed across both controllers.

In a private environment file, set the values below. Run from a local shell without model or Git credentials. Release checkouts must contain only their committed files, with no ignored build artifacts, hidden index changes, or additional files. Keep them unchanged throughout the run; workspace dependencies belong in the separate test checkout.

Run the test and check the result

TEST_ENV_FILE="$HOME/.config/oce/rehearsal.env"OCC_TEST_PRODUCTION_TUI_REHEARSAL=1 \  node --env-file="$TEST_ENV_FILE" --test tests/integration/production-tui-k3d-real.test.mjs

Require the rehearsal to pass without a skip. During the forward upgrade, the test suspends the exact Helm client after observing its attempt-bound initializer Job. Kubernetes finishes the real initialization while Helm remains pending. The test terminates and reaps that client, checks the committed PostgreSQL history with the candidate's migrator, and runs the supported --resume --recover-helm --migration-history-checked --recovery-reviewed path. The baseline selects an unused documentation-range directory proxy; the candidate clears it. Recovery must first refuse the retained egress NetworkPolicy, then refuse the retained API environment entry. The fixture explicitly reconciles those two removals with UID/resource-version guards while writers remain stopped, then retries the same recovery command. Product recovery performs no deletion. Recovery must advance exactly one revision without replacing the completed initializer Job or Pod or changing the committed migration history. The test does not edit Helm storage or disable hooks. If Helm finishes before the interruption, the rehearsal fails rather than claiming recovery coverage.

The archive taken before the upgrade must omit the candidate's acknowledged draft. The archive taken after the write must contain it. The previous controller must accept the restored migration history. After the return, both OCC components must use the restored database. OCC must return both drafts and their acknowledged saved settings, including execution mode, with the original service key. Metadata-only responses with unreadable settings fail the rehearsal. OCC must also deny an anonymous read of the Installation.

See proof.json for completed checks and timings and return-progress.json for the last phase started. Both are in the private evidence directory reported by the test. The forward upgrade records authenticated HTTPS GET /installation samples before, during, and after the upgrade, including a failed upgrade. Each request has a five-second deadline. Requests run serially on a nominal one-second cadence during the upgrade, with at least one second between actual starts. Slow requests, evidence writes, and late timer wakes can leave slots unprobed; the sampler does not replay those slots. The evidence includes timestamps, request durations, received HTTP status even for invalid response bodies, and outcomes. Only HTTP 200 with the original Installation ID counts as a successful observation. A successful sample includes requestId when the x-request-id header and meta.requestId match and have the controller’s request ID format. The client omits the ID if it contains a known protected value. To identify the serving Pod, find that ID in a corresponding http.completed record from a known Pod. Samples without the ID still count as successful. Response bodies and credentials are omitted. If recording fails, the test reports the write failure alongside any upgrade failure, preserving the upgrade as the primary cause.

Each completed observation also updates forward-api-availability.json in that same directory by atomic replacement. If the runner stops, this file retains the last saved observations. upgradeSucceeded: null means the action has not been recorded as finished; do not infer success or recovery from an incomplete file. A final boolean records whether the upgrade action succeeded, independently of the observed HTTP results. Process interruption does not add an after sample. This protects completed writes against process termination, not host or disk loss.

In receipt version 2, nominalIntervalMs replaces the old sleep interval (intervalAfterResponseMs is null). Compare cadence.probedSlots with cadence.scheduledSlots to see how many one-second opportunities received a completed first attempt. unprobedSlots counts the remainder. Slot 0 begins when the helper invokes the upgrade. Each during sample records the slot in which it started and its delay within that slot. Boundary reads have no slot. An HTTP error or timeout still counts as a probe.

gapMs measures the previous finish to the next start. Use startGapMs for the interval between starts. maxStartGapMs includes the boundary reads. tailGapMs covers the last start through the last completed response. All offsets use the same monotonic clock. Wall-clock timestamps support log correlation.

An interrupted receipt covers completed observations through throughOffsetMs. An open slot can still receive a probe. On settlement, cadence.endOffsetMs records when the helper observed the upgrade finish, including an exact slot boundary. The last in-flight request and the final read can finish later. The final file write and time after the last response are outside coverage.

These observations do not add an outage-free pass requirement. They measure the fixture's HTTPS route, not exact downtime, public ingress, Slack, or model continuity. Availability between requests remains unobserved.

Inspect retained fixture resources after a failure. Set OCC_TEST_PRODUCTION_TUI_KEEP=1 to retain fixture resources after a success. Remove the disposable cluster when you finish inspecting the evidence.

A result applies only to the selected source and image pair. A pair with identical migration catalogs proves the real idempotent initialization and committed-history path, not a schema-changing migration. This draft-only rehearsal does not prove active Agent, model, or Slack continuity. A historical compatible pair does not qualify later schema changes or a deployment of current main. Recheck migration, restore and saved-configuration compatibility when selecting another pair; retain those limits with the evidence.

Check a separate migration while the incumbent serves

Use the same local fixture and image inputs with OCC_TEST_PRODUCTION_TUI_ONLINE_MIGRATION=1 in place of OCC_TEST_PRODUCTION_TUI_REHEARSAL=1. Select an image pair whose candidate appends entries to the incumbent's canonical PostgreSQL migration catalog.

The test installs the previous release and runs the candidate chart's migrator in a separate Job. It holds the real PostgreSQL migration lock and confirms the candidate is waiting for it on both sides of a successful authenticated HTTPS request. After releasing the lock, it checks the Job exit, verifies the complete ordered migration history against the candidate catalog, and repeats the read through the original API endpoint. The original API and worker Pod identities and Helm release must remain unchanged at the measured checks.

This test does not change how the production upgrade command runs. The reads show that the incumbent served before, while the candidate waited, and after the migration completed. They do not measure availability during SQL execution or qualify overlapping API or worker versions, browser sessions, or rollback.