Credential source refresh Flow
Overview
OCC registers an empty OAuth2 provider. The Refresh Driver requests a token; the gateway renews it before expiry. See source lifecycle for binding and withdrawal.
In Codex device flow, OpenShell keeps the refresh grant; the Harness gets a placeholder and account metadata. Authorized discovery can retrieve an access token. These OAuth checks do not qualify protected egress.
The Token Service owns GitHub issuance. Git and draft pull requests use committed tokens and cannot trigger issuance.
Entry Points
- Trigger:
POST /namespaces/:namespaceId/credential-sourceswith arefreshtype,PATCH,GET, orDELETE …/credential-sources/:credentialSourceId, andPOST …/credential-sources/:credentialSourceId/rotate. - Source:
packages/occ/src/index.ts:createCredentialSource - Source:
packages/occ/src/index.ts:rotateCredentialSource - Source:
apps/controller/src/drivers/credential-refresh/openshell.ts:OpenShellCredentialRefreshDriver - Assumptions: the OpenShell Backend declares
credential_gatewayandcredential_refresh; the Installation selects both Drivers. GatewaytoolBinariesexpose OAuth2 types; the gateway trusts and reachestoken_url.
Flow
graph TD
A{"<b>Validated refresh type</b><br/>Prepared issuer?"} -- "yes" --> U["<b>501 NOT_IMPLEMENTED</b><br/>No registration"]
A -- "Gateway-owned" --> R["<b>Select Refresh Driver</b>"]
R --> B["<b>registerSource</b><br/>profile with refresh, empty provider"]
B -- "failed" --> X["<b>removeRefresh, removeSource</b><br/>delete record"]
B -- "ready" --> C["<b>configureRefresh</b><br/>material, stable request ID"]
C --> D["<b>rotate</b><br/>first mint"]
D -- "not ready" --> X
D -- "ready" --> E["<b>Mark ready</b><br/>with audit"]
E --> F["<b>Gateway re-mints</b><br/>before expiry, no OCC call"]
E --> G["<b>PATCH</b><br/>configure, then rotate"]
E --> H["<b>POST rotate</b><br/>forced mint"]
E --> I["<b>GET</b><br/>status.refresh"]
E --> J["<b>DELETE</b><br/>removeRefresh, then removeSource"]Execution Trace
1. Registration
packages/occ/src/index.ts:createCredentialSource
OCC authorizes creation and validates config and Secret reference fields against
the selected type. If the type declares rotation: refresh and its provider has
a prepared issuer, OCC returns 501 NOT_IMPLEMENTED
before selecting Refresh, reading Secret values, creating the record or configuring either owner. Prepared issuers cannot rotate on demand.
Gateway-owned types must match the Gateway catalog. OCC selects Refresh, reads Secrets, commits
registering, and sends only config to registerSource. Only Refresh receives
issuer material. Device login defers minting until completion.
The gateway posts issuer material to token_url. Before Secret reads,
validateOAuth2Config rejects IPs in any spelling, localhost and localdomain
names, and single-label names (Pod /etc/hosts aliases). Profile construction
also rejects these stored hosts and port 0, preventing attachment.
Deployer-configured hostAliases (chart setting server.hostGatewayIP) remain reachable.
The check also refuses names that reach the gateway Pod itself, which egress NetworkPolicy does not police (finding 1356):
- A Pod DNS name (
10-42-0-5.idp.pod.cluster.local) in any namespace: CoreDNS's defaultpods insecureanswers it with the written address. CoreDNS ignores one or two leading labels before it, so anypodlabel after the second is refused, whatever the cluster domain (finding 1359). - When the OpenShell Backend names the gateway's namespace
(
apps/controller/src/backends/openshell.ts:placement), any name in that namespace'ssvcorpodzone, whatever the cluster domain, and the shorter names the gateway Pod's DNS search list completes into it:<x>.<namespace>,<x>.<y>.<namespace>, and any name of up to three labels ending in the gateway Service or<service>-*. CoreDNS reads the leading labels as an endpoint (<pod>.<service>-peer, the StatefulSet Pod's own name) or a port and protocol (grpc.tcp.<service>), so the label count decides, not the port name (finding 1366). Anendpointhost<service>.<namespace>.svc[.<domain>]or a dottedserviceNamenames one namespace at save. A bareserviceNameruns in each tenant namespace, so only registration and attach know it. Registration refuses the name before any gateway write with400naming/config/token_url. Attach runs only in the worker: it refuses a stored one with a message that names the replacement, and the worker retries the revision. An IP address, node address, two-label<service>.<namespace>or otherendpointhost names no namespace, and only Pod names are refused.
Other namespaces retain their issuer Service names. OCC stores new URL(token_url).href
so the gateway receives the validated spelling.
apps/controller/src/drivers/credential-gateway/openshell.ts:registerSource
imports a per-source profile and creates a provider without a token. The profile declares the OAuth2
strategy, token_url, and refresh-material names for access_token;
OpenShell resolves the credential at runtime.
2. First mint
packages/occ/src/index.ts:mintFirstRefreshToken
OCC calls configureRefresh with the source config and resolved Secret values,
then rotate. Both request IDs are name-based UUIDs derived from the source ID
and step, so a replay is not applied twice. The OpenShell Driver sends
ConfigureProviderRefresh with client_id, scope, and the secrets as
material, and RotateProviderCredential; OpenShell calls the issuer and stores
the access token on the provider. A thrown call leaves the outcome unknown, and
OCC keeps the record deleting after cleanup. A definite non-ready mint is
terminal: OCC removes the refresh material and provider, deletes the record, and
returns 503 with the Driver's failure code. Otherwise the second transaction
marks the source ready with its audit event.
Device authorization and configuration
packages/occ/src/index.ts:startAgentDeviceAuthorization,
pollAgentDeviceAuthorization, withPluginDiscoveryCredential
deviceAuthorization.harnessId sources register before login.
source_registering and source_ready audit events commit with state, identifying
actor, request, and source. Audit failure blocks login; failed starts retain audited sources.
OCC commits the session Secret's handle, actor, source, and Driver IDs with
session_pending. Known commit failure deletes metadata and private Driver value;
unknown outcomes skip compensation. Session events identify Agent or Namespace
and source ID. Audit excludes handles, credentials, and user codes.
apps/controller/src/console/agents/device-login.mts:createDeviceLogin records
start_unconfirmed before a start; both authentication forms retain it across
navigation. Agent detail keeps its Credentials tab during a start; its draft
records a later reply unless a newer tab owns it. Before the page is left, a
usable reply supplies the challenge; a 501, or no Codex device-login type
(deviceLoginOffered), disables Sign in; other refusals permit retry.
Unconfirmed attempts cannot start again, poll, or cancel. See
interrupted sign-in recovery.
Compare-and-swap serializes known-session polls. OCC rechecks scope and both
Driver selections before completion. Uncertain exchanges close the handle but
keep the source.
apps/controller/src/drivers/credential-refresh/codex-device-authorization.ts:startCodexDeviceAuthorization
uses fixed auth.openai.com endpoints from API Pods for start, poll, and exchange.
Redirects are refused. DeviceAuthorizationStartError reports connection failure
or timeout as unreachable; HTTP rejection, malformed replies, and AbortError
as unavailable. HTTP returns 503 DEPENDENCY_UNAVAILABLE, naming the host and
API Pod egress only for unreachable. The device_authorization.start_failed
warning includes the Refresh boundary and failure class, never provider messages,
response bodies, or nested causes.
apps/controller/src/drivers/credential-refresh/openshell.ts:pollDeviceAuthorization
performs the Codex exchange, configures OpenShell with refresh material, and
stores trusted account metadata. Before acquiring a code, its issuer polling
returns pending on HTTP 429, 5xx, or a fetch connection failure; OCC keeps
the handle and interval. An aborted initial poll, including OCC's internal
timeout, is not pending: it and any failure from code acquisition onward close
the session without replay. OpenShell owns
future refresh and successor persistence. After handoff, polls read existing state:
unfinished refresh stays pending; a persistent uncertain marker needs operator inspection.
Ready confirms a usable token. Later versions attach the existing provider
without reseeding credentials; account metadata changes require a new login and
version.
For plugin configuration, the Gateway reads ownership/account metadata with its
Backend identity, then calls GetProviderCredentials through the
operator-TLS channel. OpenShell may refresh to keep five minutes of lifetime.
Only access credentials enter the callback; it
rechecks the caller's authority and saved-Agent binding/grant before provider I/O.
Closing or expiring the login session leaves source-based discovery enabled.
Cancellation or a failed recheck cannot undo a started gateway refresh.
3. Background refresh
apps/controller/src/drivers/credential-refresh/openshell.ts:refreshStatus
OpenShell re-mints the token before it expires and writes it to the provider.
The Sandbox proxy substitutes the current token for the stable placeholder on
each request, so a running Harness needs no restart. OCC makes no call.
GET on the source adds refreshStatus to the gateway status; OpenShell's
refreshed state reports ready; its error states (with a recovery action) and
missing established refresh state report failed. OCC finds the
type through refreshDriverForSource, which tolerates a type the catalog no
longer offers: that source keeps its gateway status without refresh.
4. Update and rotation
packages/occ/src/index.ts:updateCredentialSource
PATCH locks the source, not the Namespace, so other Namespace writes do not
wait on the issuer. A field the catalog marks issuerRotated (refresh_token)
must name a different Secret than the recorded one, or the update returns 409
before any Secret read: OpenShell may hold a newer token. PATCH then reads
its Secrets and checks the gateway status. For a refresh type it calls
configureRefresh with a new request ID, then rotate, instead of the
gateway's updateSource. A non-ready mint returns 503 before OCC replaces
the Secret references, but OpenShell keeps the new material, the next GET
reports the failed mint, and OCC records a failure audit event. Each
reconfiguration, even one whose mint fails, starts a new OpenShell authorization
epoch that revokes running Sandboxes' stable placeholders, so Agents need a
redeploy.
packages/occ/src/index.ts:rotateCredentialSource
POST …/rotate requires credential_source:update, a ready source, and a
refresh type; a static type returns 409. It calls rotate under the source
lock and returns the new status. The HTTP handler commits the audit event in
the same transaction; a failed mint commits a failure event instead. A commit
whose outcome is unknown records neither event.
5. Deletion
packages/occ/src/index.ts:deleteCredentialSource
After the usual reference check and deleting mark, OCC calls removeRefresh,
then the gateway's removeSource. Before DeleteProviderRefresh,
apps/controller/src/drivers/credential-refresh/openshell.ts:ownedProvider
checks the exact source Namespace and paired Gateway ID, then the current
provider's manager/source labels and profile identity. A missing provider is
already clean; a foreign provider or failed lookup refuses cleanup. The deletion
RPC retains allow_missing. Explicit rotation uses the same guard but refuses absence. It guards against
out-of-band provider drift; lookup and mutation are not atomic. A failure leaves
the record deleting for retry.
6. Issue and renew a GitHub installation token
apps/controller/src/drivers/repo/github/access/prepared-owner.ts:createGitHubPreparationOwner
The Token Service records issuing, signs a five-minute App JWT and rechecks ownership before issuance.
Issuer diagnostics record the first failure stage. requestEndCalled indicates a possible send.
graph TD
W["<b>Token Service</b><br/>Reads registered App key"] -->|Signed JWT| G["<b>GitHub</b><br/>Issues token"]
G -->|Installation token| W
W -->|Verify and commit| P[("<b>PostgreSQL</b><br/>Encrypted token")]
P -->|Eligible token| R["<b>OCC and resolver</b>"]
R -->|Credential header| S["<b>Metadata sender</b><br/>Needs a send permit"]
classDef owner fill:#e7f1ec,stroke:#789884,color:#263e33
classDef issuer fill:#f0eaf6,stroke:#9584ab,color:#392c42
classDef stored fill:#e8eff6,stroke:#8498a9,color:#283746
class W,R,S owner
class G issuer
class P stored
linkStyle default stroke:#89919aOCC registers github_app_installation with the scoped grant and signing-key reference. PostgreSQL encrypts tokens with a persistent 32-byte key. Authorized metadata requests reuse valid tokens or join issuance until their deadline. Native Git uses its private broker.
provider-transport/request-options.ts:prepareProviderScope freezes scope. The issuer captures before validation, retaining expiry bounds locally. Transport clears synchronously borrowed bytes; publication requires successful close and current signing authority/admission. github-installation inherits scope with {}. OCC rechecks the grant and version before sending.
| Limit | Effect |
|---|---|
| Repository approval (Git) | Removal or expiry stops new authorizations. Standing approval permits another finite session for the same running Agent and scope. |
| Execution's repository session (Git) | The session ends at its deadline even if the token is fresh. |
Local cutoff (usableUntil) |
Readers stop selecting the token 60 seconds before GitHub expiry. A request may require more remaining time. |
GitHub expiry (expires_at) |
GitHub sets a one-hour installation token lifetime. |
Metadata reads share both token limits. Withdrawing the credential-source binding stops new authorizations. A previously authorized send can finish.
Key rotation and withdrawal. When the signing key changes version, the worker
advances the owner epoch and retires old material before publishing its replacement.
Deleting the Agent blocks use and renewal immediately. On its next pass, the
worker disables the Agent-bound metadata family and retires the token. GitHub
revocation can outlast Agent deletion. Source deletion waits up to 25 seconds
for revocation, returning 409 if custody remains unsettled.
Uncertain outcomes. Lost receipts block replay; readback can recover eligible committed material. The parser
retains unique cleanup tokens despite ambiguous scope; duplicate root tokens leave none. Capture and definite refusals survive transport errors.
Retirement claims precede DELETE /installation/token. Complete 204 and successful close confirm revocation, separately from durable
settlement. Expiry plus skew can settle uncertainty without confirming revocation. See App key recovery.
apps/controller/src/backends/token-service/retirement.ts:retireToken clears
detached bytes before settling the original receipt; combined errors name stages.
apps/controller/src/drivers/repo/github/access/prepared-postgres.ts:store
clears publication buffers on success/failure. Callers own detached reads; clearing cannot revoke tokens.
Debugging and Verification
-
node --test tests/integration/repository-credentials-github-prepared.test.mjsexercises producer → transaction commit → warm consumer over loopback HTTPS, with a generated App key and actual JWT signature verification. It covers renewal overlap, duplicate attempts, latest-admission fences, rejected provider scope/expiry, cancellation, and transaction failures before/after commit. Its serializable in-memory fixture reconnects owners to shared committed state; it is not production durability, process-crash, live-provider, installed-runtime, cluster or regular Agent workflow proof.postgres-prepared-github-lifecycle.test.mjsexercises the durable lifecycle, signed protected exchange, issuer rotation, withdrawal, Agent deletion and quarantine over loopback. -
Read the source:
status.refresh.state,lastRefreshAt,failureCode, andrecoveryActionexplain the last mint without exposing a token. -
oauth_token_endpoint_unavailablemeans the OpenShell gateway Pod could not reachtoken_url; check its NetworkPolicy and trust of the issuer's CA. -
reauthorizerequires new authorization: replace Secret-backed material withPATCH; for device-authorized sources, sign in again and rebind the Agent. -
tests/helpers/openshell-sandbox-workflows.mjsproves minting, background re-minting, forced rotation, reauthorization, a failed update, and deletion against a real Keycloak.tests/conformance/credential-source-occ.test.mjscovers OCC ordering and cleanup.
