OpenClaw EnterpriseDOCSGitHub

Production Startup Flow

Overview

Helm initializes OCE from protected storage and approved inputs. After private API/worker readiness, authenticate /installation with the service key.

Deployment guide.

Entry Points

Flow

First install requires empty OCE history.

graph TD
    subgraph Operator["Operator-owned preparation"]
        A["Prepare values, Installation YAML and bootstrap PVC"]
        B["Create namespace and Secrets; check prerequisites"]
        C["Create fresh bootstrap PVC"]
        D["prepare-bootstrap-volume verifies empty output and root permissions"]
        D --> RC["Snapshot inputs and sync intent"]
        RC --> RN{"Receipt notice write failed?"}
        RN -->|Yes| RF["Preserve intent; stop before dispatch"]
        RN -->|No| RD["Sync dispatch marker"]
    end
    subgraph Helm["Helm-owned startup"]
        RD --> E["Run Helm install once with attempt marker"]
        E --> F["Initialization Job validates mounted Installation"]
        F -->|Startup checks pass| V{"PostgreSQL before 17?"}
        V -->|Yes| P["Refuse initialization before migration DDL"]
        V -->|No| O{"Canonical migration history?"}
        O -->|No| P
        O -->|Yes| G["Apply migrations, bootstrap administrators and write protected key output"]
        G --> H["Start private API Deployment"]
        G --> I["Start independent worker Deployment"]
        I --> R{"Repository credentials enabled?"}
        R -->|Yes| S["Sidecar copies protected inputs and starts private control"]
        S --> Q["Sidecar probe gates worker Pod readiness"]
        H --> L["Run Kubernetes Compute preflight"]
        I --> L
        L --> M{"Kubernetes older than 1.35?"}
        M -->|Yes| N["Emit advisory warning and continue"]
    end
    subgraph Handoff["Read-only operator handoff"]
        F -->|Inspect retained Job| T["occ installation doctor or plan reads named Job"]
        T --> U["Check target, Job identity and terminal status"]
        U --> W["Report checkpoint; operator reconciles remaining work"]
    end
    subgraph Proof["Operator-owned authenticated proof"]
        M -->|No| J["Retrieve service-key response from protected storage"]
        N --> J
        J --> K["occ installation get from approved client"]
    end

Execution Trace

Migration, bootstrap, API and worker use createPostgresPool with password or Azure workload-identity authentication. For each password-mode connection, the factory parses the URL before applying explicit caller limits and startup options. It retains TLS-file reads and the effective IP-host certificate check. Failover, maintenance restart, idle_session_timeout or proxy reset discards the idle client and logs only its error code to stderr as database.idle-client-error. The process continues; the next query opens a new connection.

1. Prepare native production inputs

scripts/diagnose-kubernetes-prerequisites.ts:diagnose reads CRDs, controllers, addresses and object quota headroom. EKS binds account, ARN, endpoint and CA. Resolve failed/unknown checks before Helm.

deploy/helm/openclaw-enterprise/values.yaml:1

Keep values, Installation YAML and bootstrap PVC outside the checkout. Helm selects images, endpoint, Secrets, claim, selectors and egress. openclaw.validate requires a DNS-1123 release Namespace label (maximum 63 characters). Installation YAML selects Drivers, Backends, runtime images, projected identity, networking and storage.

File-backed Secrets supply Installation startup, database URLs/optional CA, Better Auth signing, and optional ChatGPT administrator credentials. Helm neither synchronizes inputs, rewrites Driver configuration, nor infers runtime images.

2. Prepare the fresh bootstrap volume

scripts/prepare-bootstrap-volume:124 takes an existing bootstrap.password.claimName PVC, explicit kubeconfig/context/namespace/claim, an approved Node-capable image and optional --node-selector KEY=VALUE labels. It preserves namespace/selector strings, applying selectors before WaitForFirstConsumer binding. It requires an empty root except filesystem-owned lost+found and sets UID/GID 1000, mode 0700.

If forbidden, storage administrators prepare the claim through approved procedures. The helper cannot create PVCs, repair used claims, retrieve credentials or change controller configuration.

3. Run Helm initialization

internal/occcli/installation_receipt.go:installationInstallCommand snapshots kubeconfig/chart/values, checks release states and syncs intent and its directory. A failed stdout notice preserves and names intent, stopping before dispatch or Helm. Otherwise it syncs the unknown marker and starts Helm. observeInstallationJob records the first recognized attempt-matched Job UID (installs only). Helm's outcome stays separate; final report write errors fail the command without changing that outcome or replaying install.

Reported template failures add static offline-render guidance, including settings for the suffix refusal. Stderr stays in memory (4 MiB); excess is discarded without affecting Helm. Raw diagnostics are never printed or saved. Receipt and recovery decisions stay unchanged.

deploy/helm/openclaw-enterprise/templates/bootstrap-networkpolicies.yaml:1 installs isolation before the Job; it and later policies grant configured DNS peers UDP/TCP 53 and 5353 (Helm DNS contract).

deploy/helm/openclaw-enterprise/templates/jobs.yaml:8 refuses existing or unreadable Jobs during connected installs. Offline rendering cannot check this; lookup neither locks names nor verifies database/volume freshness.

openclaw.validate in deploy/helm/openclaw-enterprise/templates/_helpers.tpl checks routing, ports and peer labels. Operators supply GatewayClass, CRDs, controllers and Secrets. Disabling routing omits its resources/mounts and requires disabling native admin and sandbox routing. Database/API ports accept 1–65535 without leading zeros; worker metrics/repository ports must differ. The Compute Driver rejects malformed IPv4 hostnames and numeric/hexadecimal sandbox suffixes. See refused values.

On install and upgrade, deploy/helm/openclaw-enterprise/files/check-installation.mjs validates Installation offline before migration, bootstrap and API rollout. It uses apps/controller/src/startup-failure.ts:formatStartupConfigurationFailure and the API/worker filter apps/controller/src/logging.ts:configurationLogDetails: at most 32 field diagnostics, without submitted values, except apps/controller/src/composition/installation-fields.ts:requireChartListener: CHART_LISTENER_MISMATCH names ports, hosts or paths refused when OCC_EGRESS_AUTHORITY_PORT/OCC_OPENSHELL_GIT_PORT is set.

For chart-managed routing, openclaw.listenerEnv passes OCC_CHART_GATEWAY_ROUTING to the hook, API and worker. requireChartGatewayRouting compares the Kubernetes Compute Gateway name, Gateway and Envoy namespaces, and effective hostname. A mismatch reports the field without its value.

The hook keeps the release name and bounds its suffix to 63 characters. Both database containers optionally mount the CA. Helm validates distinct application and migration Secret keys against Kubernetes syntax before use. installation.key and executionCluster.kubeconfigKey must be Secret keys at most 253 characters, not . or a .. prefix.

For handoff, internal/occcli/installation_doctor.go:installationHandoffCommand verifies receipt namespace/release/cluster connection before reading the named Job. initializationEvidence checks marker, captured UID, chart steps, controller and terminal status. Missing UID, replacement, refusal or inconclusive status leaves the checkpoint unknown. Doctor separately reports Helm's outcome; plan adds read-only recovery. Both check the buffered report's WriteTo before returning Job status. Write errors reach cmd/occ/main.go:main for stderr diagnostics and exit 1.

scripts/migrate-production.mjs:1, scripts/migration-history.mjs:migrateWithHistory

After source verification, all modes reject PostgreSQL before 17 with fixed MIGRATION_SERVER_UNSUPPORTED recovery guidance, before locking or catalog access (compatibility). Under the advisory lock, the same connection verifies roles and receipts, then catalogs and grants against native or reviewed restore digests. Unsupported history fails before DDL/bootstrap. After option guards, completed apply skips Drizzle only when SHOW transaction_read_only returns off. Otherwise Drizzle retains pending migration and read-only refusal. finally releases the lock and client (history).

scripts/lib/bootstrap-configuration.ts:bootstrapConfiguration requires an HTTPS Console origin before database access, including during --verify-existing.

scripts/bootstrap-installation.mjs creates or verifies Installation, human/service administrators, IAM seed, audit evidence, and initial service key. Fresh bootstrap creates default via OpenClawController.createNamespace as the bootstrap Principal. Queued reconciliation commits with Installation/IAM state and audit. Existing Installations add no Namespace; this is not Kubernetes' default. The worker provisions Driver-owned infrastructure; operators provide tenant RoleBindings. The bootstrap container alone writes protected PVC password/service-key files. Operators can invoke the same command with --verify-existing to check a populated candidate using the incumbent ID and a read-only database snapshot. The Helm initialization Job continues to use ordinary bootstrap. Existing output, unsafe permissions, or inconsistent accounts/IAM identity fail the Job. Helm failure does not imply database-hook rollback.

4. Start private API and worker Deployments

apps/controller/src/api-startup.ts:configuration, apps/controller/src/worker.ts:312

apps/controller/src/drivers/compute/kubernetes/index.ts:KubernetesComputeDriver.preflight

With OIDC CA settings, generated assemble-api-ca.mjs combines validated IdP and Gateway roots for NODE_EXTRA_CA_CERTS. The check-api-database-ca.mjs preload requires the selected CA and DNS hostname verification for PostgreSQL.

apps/controller/src/auth/configuration.ts:validateAuthSecret rejects short secrets before Driver construction or PostgreSQL access. composeProduction and the auth factory repeat it. Endpoints include /healthz and /readyz. apps/controller/src/composition/postgres-readiness.ts:createPostgresReadinessCheck shares one application-pool SELECT 1 per instance across overlapping GET and implicit HEAD probes. The query stays active when probes disconnect. Subsequent probes query again once PostgreSQL returns. API startup allows 120 one-second failures. API readiness times out after one second. Worker startup allows 24 five-second failures.

apps/controller/src/index.ts:createFastifyApp

On SIGTERM, the API stops accepting connections and drains requests with Connection: close, avoiding the 72-second keep-alive wait. Its single Recreate replica has no peer or preStop; the 30-second grace cuts off unfinished requests. Logs show shutdown.started (signal), then shutdown.completed (durationMs), or shutdown.failed (exit 1). Only shutdown.started means an interrupted drain.

The PostgreSQL API compositions close the exchange binding, protected Driver, prepared GitHub owner, and database in order, continuing after a close fails. Shutdown reports the first failure, wrapping a falsey thrown value as an error with the original value as its cause. Failed startup preserves the startup error. The exchange binding is independent of the protected Driver. See apps/controller/src/composition/production.ts:composeProduction and apps/controller/src/composition/development-postgres.ts:composePostgresDevelopment.

The worker closes the repository Git sender binding, prepared Git owner, receipt listener, metrics, and prepared GitHub owner in order, continuing after a close fails, then stops its loop and database. A listener left open would keep the Pod running until its grace ends. Any failure logs worker.error SHUTDOWN_FAILED (exit 1). See apps/controller/src/worker/shutdown.ts:closeWorkerComposition.

When set, controlPlane.nodeSelector places the API, worker, migration, bootstrap, and private proxy on reviewed control-plane nodes. The deploy/helm/openclaw-enterprise/templates/gateway-routing.yaml template sets EnvoyProxy.spec.provider.kubernetes.envoyDeployment.pod. Compute owns tenant Gateway and Agent placement.

With database.caSecretName, _helpers.tpl rejects CA paths that would mask active client mounts. It normalizes relative paths, dot segments, and trailing slashes for comparison and emits accepted paths as supplied.

The shared egress policy selects only api, worker, and initialization Pods with the release identity. It allows DNS, database egress to every database.cidrs host, and Kubernetes API egress to every cluster.cidrs host. Collectors use their separate DNS, API, and exporter policy; unknown or missing component labels retain default-deny. Pre-install initialization has only its hook DNS/database grants. Destinations must be explicit IPv4 /32 hosts; refresh them when managed endpoint addresses change. Chart ports take no leading zeros: 0443 fails.

deploy/helm/openclaw-enterprise/templates/networkpolicies.yaml renders separate API-only TCP 443 policies for api.modelDiscoveryCidrs and api.pluginDiscoveryCidrs. Each accepts explicit IPv4 /32 hosts and defaults to no egress. Operators maintain addresses for model discovery and MCP tool discovery independently. Console model selection and Harness egress do not depend on these policies.

When api.channelDirectoryProxyUrl names an approved HTTP(S) proxy at a literal IPv4 address and port, the chart passes it to the API and grants only that Pod egress to the proxy's exact /32 and TCP port. The proxy must permit CONNECT to slack.com:443. The empty default renders no rule and leaves production Slack directory lookup unavailable with manual exact-ID entry.

The Kubernetes Compute Driver queries the API server version and verifies authenticated Namespace access. Below the supported 1.35 baseline, the API logs compute.preflight-warning with both versions and continues. An invalid version response, unreachable API, or failed Namespace access fails preflight.

The worker independently validates production settings, opens the same application-role database, loads selected Drivers, validates IAM, runs Compute preflight with the same advisory warning, emits worker.started, and polls durable Namespace/AgentRevision work. Readiness requires fresh queue-health observations; neither process mounts the bootstrap PVC.

apps/controller/src/composition/installation-config.ts:loadInstallationConfiguration

With repository selected, both processes load the same registry and construct GitHubRepositoryPolicy; the GitHub Backend's drivers.repo member must equal repository.id. Only native delivery loads the Unix client and CA reader used by GitHubNativeRepositorySender. Startup refuses the retired Installation drivers.repo. API startup performs no control operation and loads no App key. With prepared repository Git, the worker reads the App key at startup and exits before readiness with PREPARED_GITHUB_APP_KEY_INVALID if it cannot sign. The worker owns session lifecycle; API readiness remains database-backed.

apps/controller/src/composition/repository-credentials/projected-inputs.ts:prepareProjectedInputs

Beside the single worker, the optional service pins a projection generation, copies config/key/TLS/registry inputs into owned private regular files, validates the selected Backend and exact Service origin, then starts both listeners through the protected loader. App/TLS private material stays in its container; only the worker receives a projected Kubernetes API token. They share only the private control volume. The Pod uses Recreate and a 75-second termination grace period. Readiness requires both the service's Unix-listener probe and the worker's queue-health probe. The worker retries unavailable session operations through its Driver contract. Neither component has independent high availability. Repository runtime reconciliation continues in the Agent repository credential flow.

5. Retrieve the key and prove authenticated access

Fresh bootstrap checks both output paths before creating credentials.

internal/occclient/client.go:Client.GetInstallation

After Helm readiness, retrieve initial-admin-service-key.json from protected bootstrap storage through an approved reader path into an owner-readable file. A completed Job is not an exec endpoint, and the API and worker cannot retrieve this file for the operator.

occ installation get reads the protected key file and displays the Installation. Its ID must match the key response's meta.installationId. The operator records that ID in the Installation startup Secret's openclaw.dev/installation-id annotation; coordinated upgrades use this marker to bind their OCC endpoint to the selected Installation. The quickstart's docs/snippets/openshell-quickstart/operator-session.sh:upgrade_installation checks Helm's deployed revision and API Pod binding, initializes missing annotations, patches using UID/resourceVersion preconditions, and sets controlPlane.installationChecksum. Agent runtime, Gateway WebSocket authentication and model calls remain unproven until tenant deployment/TUI procedures run.

Debugging and Verification

Search documentation