OpenClaw EnterpriseDOCSGitHub

Common Operational Logging Flow

Overview

Trusted startup configuration selects the OCC logging level. Authorized Agent deployment freezes runtime logging in an immutable AgentRevision. Optional Collectors export reviewed operational records; PostgreSQL audit remains separate durable evidence.

Entry Points

Flow

graph TD
  subgraph OCC["OCC control plane"]
    A["OCC process starts"] --> B["Parse startup snapshot once"]
    B --> C["Create OCC Pino logger"]
    C --> D["Emit fixed JSON operational events"]
    B --> E["Authorized Agent deployment starts"]
    E --> F["Sandbox may transform a Configuration copy"]
    F --> G["Admission stamps platform-owned native logging fields"]
    G --> H["Persist immutable AgentRevision"]
  end

  subgraph Runtime["Managed runtime"]
    H --> I["Compute renders gateway and Codex settings"]
    I --> J["Gateway JSON console and Codex JSON stderr"]
    K -->|"no"| L["Local container logs only"]
  end

  subgraph Collector["Bundled Collector boundary"]
    D --> K{"Collector enabled?"}
    J --> K
    K -->|"yes"| M["Collector reads container output and protected metadata"]
    M --> N["Classify records and strip unapproved content"]
    N --> O["Bounded queue and OTLP HTTP exporter"]
  end
  O --> P["Optional demo Loki stores event body and structured metadata"]
  P --> Q["Grafana filters and formats retained metadata"]

Execution Trace

1. Startup parses one configuration snapshot

apps/controller/src/composition/installation-config.ts:loadStartupConfigurationSnapshot

API and worker parse trusted YAML once and pass startupConfiguration.logging to driver composition. Invalid settings fail startup before requests or work. See the settings reference for YAML shape and values.

2. Processes log fixed sanitized events

apps/controller/src/server.mjs:start

Worker (apps/controller/src/worker.mjs), bootstrap (scripts/bootstrap-installation.mjs), and migration (scripts/migrate-production.mjs) also create Pino loggers at the selected level. The API disables Fastify request logging. Bootstrap and migration separate success protocol output from structured failure diagnostics. Before Pino writes, apps/controller/src/logging.ts:emitOccLogEvent keeps reviewed scalar fields and drops unapproved fields, credentials, provider payloads, request/reply objects, and unsafe strings. This source boundary precedes the separate Collector filter in step 7. For worker records, the Collector retains allowlisted work.operation values and bounded work.id shapes. Agent stop keys include the operation UUID; deletion keys have no operation suffix. Unsupported values and key shapes are excluded.

3. Admission freezes runtime logging

packages/occ/src/index.ts:OpenClawController.deployAgent

If the trusted ComputeDriver declares runtimeLogging: "driver", admission validates and freezes the native document without rewriting logging fields. The Compute contract owns that pipeline; OCC logging and audit remain unchanged.

Otherwise, the SandboxDriver may transform a frozen copy of the Namespace-owned Configuration. Deployment stamps native fields before validation: matching logging.level and logging.consoleLevel, JSON console style, and diagnostics.otel.logs=false. It drops the retired logging.redactSensitive key. Runtime code owns console and tool redaction. The source Configuration is unchanged; the immutable AgentRevision retains the admitted policy across restarts and later edits.

4. Compute renders settings from the revision

apps/controller/src/drivers/compute/docker/index.ts:DockerComputeDriver.prepareRevision

Kubernetes uses apps/controller/src/drivers/compute/kubernetes/index.ts:KubernetesComputeDriver.deployment. Both Drivers require consistent admitted logging fields. Kubernetes mounts the document read-only at /etc/openclaw/openclaw.json. Gateway logs JSON to console; dedicated Codex app-servers use JSON stderr and host-owned arguments that disable OTLP export and prompt logging. Lifecycle hooks and SecretBindings cannot override those destinations.

5. Docker collection is an explicit development override

apps/controller/src/drivers/compute/docker/index.ts:DockerComputeDriver.prepareRevision

The optional compose.logging.yaml routes OCC, gateway, and Codex containers through Docker's nonblocking fluentd driver to a pinned Collector. Docker Compute sets runtime LogConfig from OCC_DOCKER_LOGGING_ADDRESS, which must be reachable from the Engine. See the Docker procedure.

6. Kubernetes collection is bundled or equivalent

deploy/helm/openclaw-enterprise/templates/collector.yaml:logging.collector.enabled

The Helm Collector DaemonSet reads node CRI files and uses Pod metadata to associate records with managed workloads. See the Kubernetes observability procedure for enablement and existing-Collector reuse, and the security reference for isolation limits.

k8sattributes maps identity before transform/kubernetes-resource removes internal Pod labels; removing shared labels per record would lose identity for later records in the batch. It also blocks pipeline start until its Pod cache syncs: filelog reads existing CRI files immediately, and a record processed without Pod identity is filtered out while its offset is still committed, so startup events such as worker.started would otherwise be lost for good.

The chart validates one exporter destination: an IPv4 /32 or paired namespace/Pod selectors, with a bounded TCP port. It renders exporter egress alongside DNS/API access. Empty Collector metrics selectors grant no ingress; paired selectors admit port 8888. Policies are additive. The demo can export privately to Loki using the bundled Collector or an external Collector with its own filtering policy. Driver-owned pipelines also have separate guarantees.

7. Collector exports only operational classes

deploy/logging/collector.yaml:transform/operational

The bundled Collector keeps transport-derived identity before parsing untrusted JSON. It classifies fixed OCC event names, gateway subsystem records, and Codex stderr records from codex_app_server. For retained records it keeps allowlisted attributes and replaces the body with the event name, stripping arbitrary content. OCC compute.preflight-warning records retain WARN severity and bounded occ.code; the local diagnostic message is excluded from remote export. It drops malformed, oversized, unclassified, unspecified-severity, and Codex protocol stdout records. Collector-only configuration holds exporter credentials and TLS settings. Finite queues and retries make logs best-effort; outage or overflow cannot block API service, worker reconciliation, or PostgreSQL audit persistence.

8. The demo dashboard presents existing metadata

deploy/helm/openclaw-observability-demo/templates/grafana.yaml:logs.json

For the bundled Collector path, Loki retains event names and normalizes attributes as structured metadata. Grafana formats metadata at query time without changing records. External and Driver-owned pipelines require separate operator review. See the demo guide for panels, correlation, and authorization limits.

Debugging and Verification

Search documentation