github-actions[bot] commented on code in PR #66119:
URL: https://github.com/apache/doris/pull/66119#discussion_r3664929301
##########
.github/workflows/code-review-runner.yml:
##########
@@ -101,26 +127,138 @@ jobs:
OSS_CODEX_GOAL_FALLBACK_OBJECT:
oss://doris-community-ci/codex/codex-goal
- name: Configure Codex auth
+ id: configure_auth
run: |
- install -m 700 -d "$RUNNER_TEMP/codex-home"
- printf 'CODEX_HOME=%s\n' "$RUNNER_TEMP/codex-home" >> "$GITHUB_ENV"
+ defer_review() {
+ local retry_at="$1" request_id
+ request_id="${DEFERRED_REQUEST_ID:-$GITHUB_RUN_ID}"
+ local request_file
+ request_file="$(mktemp "$RUNNER_TEMP/codex-review-pending.XXXXXX")"
+ jq -n \
+ --arg pr_number "$PR_NUMBER" \
+ --arg head_sha "$HEAD_SHA" \
+ --arg base_sha "$BASE_SHA" \
+ --arg review_focus "$REVIEW_FOCUS" \
+ --arg not_before "$retry_at" \
+ --arg request_id "$request_id" \
+ '{pr_number: $pr_number, head_sha: $head_sha, base_sha:
$base_sha,
+ review_focus: $review_focus, not_before: $not_before,
request_id: $request_id}' > "$request_file"
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f \
+ "$request_file"
"$OSS_CODEX_REVIEW_PENDING_PREFIX/${PR_NUMBER}-${HEAD_SHA}-${request_id}.json"
+ rm -f "$request_file"
+ }
+
+ codex_review_user=codex-review
+ sudo useradd --create-home --shell /usr/sbin/nologin
"$codex_review_user"
+ codex_home="$(mktemp -d /tmp/codex-home.XXXXXX)"
+ sudo chown "$codex_review_user:$codex_review_user" "$codex_home"
+ sudo chgrp "$(id -gn)" "$codex_home"
+ sudo chmod 770 "$codex_home"
+ control_dir="$RUNNER_TEMP/codex-auth-control"
+ install -m 700 -d "$control_dir"
+ printf 'CODEX_HOME=%s\n' "$codex_home" >> "$GITHUB_ENV"
+ printf 'CODEX_REVIEW_USER=%s\n' "$codex_review_user" >> "$GITHUB_ENV"
+
+ auth_prefix='oss://doris-community-ci/codex/auth.json.'
+ auth_listing="$RUNNER_TEMP/codex-auth-objects.list"
+ if ! ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" ls
"$auth_prefix" -s > "$auth_listing"; then
+ echo 'Failed to list Codex auth objects from OSS.'
+ exit 1
+ fi
+
+ auth_objects_file="$control_dir/auth-objects.txt"
+ candidate_auth_file="$(mktemp "$RUNNER_TEMP/codex-auth.XXXXXX")"
+ trap 'rm -f "$candidate_auth_file"' EXIT
+ while IFS= read -r candidate; do
+ if ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f
"$candidate" "$candidate_auth_file" \
+ && test -s "$candidate_auth_file" \
+ && jq -e '
+ .auth_mode == "chatgpt"
+ and (.tokens.access_token | type == "string" and length > 0)
Review Comment:
[P1] Keep cooldown identity stable across token refreshes
`auth_state_key` includes a SHA-256 of the entire `auth.json`, but Codex
legitimately rewrites access/refresh tokens during a run. This workflow uploads
those new bytes before recording a quota/rate/auth result under the old hash,
so the next job discovers the same OSS object under a new key and treats it as
immediately available. A refresh followed by quota exhaustion therefore
bypasses the persisted cooldown and can repeatedly select the exhausted
account. Key availability by a stable validated account/object identity (with
an explicit replacement generation if needed), and add a cross-job
refresh-then-quota test.
##########
.github/workflows/code-review-runner.yml:
##########
@@ -101,26 +127,138 @@ jobs:
OSS_CODEX_GOAL_FALLBACK_OBJECT:
oss://doris-community-ci/codex/codex-goal
- name: Configure Codex auth
+ id: configure_auth
run: |
- install -m 700 -d "$RUNNER_TEMP/codex-home"
- printf 'CODEX_HOME=%s\n' "$RUNNER_TEMP/codex-home" >> "$GITHUB_ENV"
+ defer_review() {
+ local retry_at="$1" request_id
+ request_id="${DEFERRED_REQUEST_ID:-$GITHUB_RUN_ID}"
+ local request_file
+ request_file="$(mktemp "$RUNNER_TEMP/codex-review-pending.XXXXXX")"
+ jq -n \
+ --arg pr_number "$PR_NUMBER" \
+ --arg head_sha "$HEAD_SHA" \
+ --arg base_sha "$BASE_SHA" \
+ --arg review_focus "$REVIEW_FOCUS" \
+ --arg not_before "$retry_at" \
+ --arg request_id "$request_id" \
+ '{pr_number: $pr_number, head_sha: $head_sha, base_sha:
$base_sha,
+ review_focus: $review_focus, not_before: $not_before,
request_id: $request_id}' > "$request_file"
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f \
+ "$request_file"
"$OSS_CODEX_REVIEW_PENDING_PREFIX/${PR_NUMBER}-${HEAD_SHA}-${request_id}.json"
+ rm -f "$request_file"
+ }
+
+ codex_review_user=codex-review
+ sudo useradd --create-home --shell /usr/sbin/nologin
"$codex_review_user"
+ codex_home="$(mktemp -d /tmp/codex-home.XXXXXX)"
+ sudo chown "$codex_review_user:$codex_review_user" "$codex_home"
+ sudo chgrp "$(id -gn)" "$codex_home"
+ sudo chmod 770 "$codex_home"
+ control_dir="$RUNNER_TEMP/codex-auth-control"
+ install -m 700 -d "$control_dir"
+ printf 'CODEX_HOME=%s\n' "$codex_home" >> "$GITHUB_ENV"
+ printf 'CODEX_REVIEW_USER=%s\n' "$codex_review_user" >> "$GITHUB_ENV"
+
+ auth_prefix='oss://doris-community-ci/codex/auth.json.'
+ auth_listing="$RUNNER_TEMP/codex-auth-objects.list"
+ if ! ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" ls
"$auth_prefix" -s > "$auth_listing"; then
+ echo 'Failed to list Codex auth objects from OSS.'
+ exit 1
+ fi
+
+ auth_objects_file="$control_dir/auth-objects.txt"
+ candidate_auth_file="$(mktemp "$RUNNER_TEMP/codex-auth.XXXXXX")"
+ trap 'rm -f "$candidate_auth_file"' EXIT
+ while IFS= read -r candidate; do
Review Comment:
[P1] Validate the ID token before admitting an account
This predicate accepts an object with no `tokens.id_token`, but current
Codex requires that field and parses it as a JWT while deserializing
`auth.json` ([schema and
deserializer](https://github.com/openai/codex/blob/e597169e9a783156e50ae9765d891a3dd74df064/codex-rs/login/src/token_data.rs#L10-L23)).
Such an object is selected as “valid,” then Codex fails to load it; that
schema error is classified as fatal, so the workflow never rotates to the
remaining valid objects. Validate every field required by the installed Codex
schema, including a parseable ID token, both during discovery and later
installs.
##########
.github/scripts/codex_auth_state.py:
##########
@@ -0,0 +1,453 @@
+#!/usr/bin/env python3
+"""Persist and select the rotating credentials used by code review jobs."""
+
+from __future__ import annotations
+
+import argparse
+import copy
+import json
+import re
+import tempfile
+from dataclasses import dataclass
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from typing import Any
+
+
+STATE_VERSION = 1
+AVAILABLE = "available"
+QUOTA_EXHAUSTED = "quota_exhausted"
+RATE_LIMITED = "rate_limited"
+AUTHENTICATION_FAILED = "authentication_failed"
+TRANSIENT_FAILURE = "transient_failure"
+STATUS_VALUES = {AVAILABLE, QUOTA_EXHAUSTED, RATE_LIMITED,
AUTHENTICATION_FAILED}
+QUOTA_RETRY_DELAY = timedelta(days=1)
+RATE_LIMIT_RETRY_DELAY = timedelta(hours=1)
+PERMANENT_AUTH_FAILURE_PATTERN = re.compile(
+ r"refresh[\s_-]*token.*(?:revoked|expired|already\s+used)|"
+ r"(?:revoked|expired).*refresh[\s_-]*token",
+ re.IGNORECASE,
+)
+USAGE_LIMIT_MESSAGE_PATTERN = re.compile(
+ r"you(?:'|\u2019)ve hit your usage limit|usage limit",
+ re.IGNORECASE,
+)
+USAGE_LIMIT_RETRY_PATTERN = re.compile(
+ r"try again
at\s+(?:(?P<date>[A-Za-z]{3,9}\s+\d{1,2}(?:st|nd|rd|th)?,\s+\d{4}\s+"
+ r"\d{1,2}:\d{2}\s+(?:AM|PM))|(?P<time>\d{1,2}:\d{2}\s+(?:AM|PM)))",
+ re.IGNORECASE | re.DOTALL,
+)
+
+
+@dataclass(frozen=True)
+class Selection:
+ auth_object: str | None
+ all_quota_exhausted: bool
+ next_retry_at: str | None
+
+
+@dataclass(frozen=True)
+class FailureClassification:
+ kind: str
+ http_status: int | None
+ retry_after: str | None
+
+
+def utc_now() -> datetime:
+ return datetime.now(timezone.utc).replace(microsecond=0)
+
+
+def parse_timestamp(value: str) -> datetime:
+ parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
+ if parsed.tzinfo is None:
+ raise ValueError(f"Timestamp must include a timezone: {value!r}")
+ return parsed.astimezone(timezone.utc).replace(microsecond=0)
+
+
+def format_timestamp(value: datetime) -> str:
+ return
value.astimezone(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00",
"Z")
+
+
+def default_account_state() -> dict[str, Any]:
+ return {
+ "status": AVAILABLE,
+ "last_failure_at": None,
+ "last_http_status": None,
+ "last_success_at": None,
+ "last_recovered_at": None,
+ "retry_after": None,
+ }
+
+
+def default_state() -> dict[str, Any]:
+ return {"version": STATE_VERSION, "next_account": 0, "accounts": {}}
+
+
+def load_state(path: Path) -> dict[str, Any]:
+ if not path.exists():
+ return default_state()
+
+ state = json.loads(path.read_text())
+ if not isinstance(state, dict):
+ raise ValueError("Authentication state must be a JSON object")
+ if state.get("version") != STATE_VERSION:
+ raise ValueError(f"Unsupported authentication state version:
{state.get('version')!r}")
+ if not isinstance(state.get("next_account"), int):
+ raise ValueError("Authentication state next_account must be an
integer")
+ if not isinstance(state.get("accounts"), dict):
+ raise ValueError("Authentication state accounts must be an object")
+ return state
+
+
+def save_state(path: Path, state: dict[str, Any]) -> None:
+ with tempfile.NamedTemporaryFile("w", encoding="utf-8", dir=path.parent,
delete=False) as temporary:
+ json.dump(state, temporary, indent=2, sort_keys=True)
+ temporary.write("\n")
+ temporary_path = Path(temporary.name)
+ temporary_path.replace(path)
+
+
+def promote_auth(baseline: dict[str, Any], candidate: dict[str, Any]) ->
dict[str, Any]:
+ if baseline.get("auth_mode") != "chatgpt" or candidate.get("auth_mode") !=
"chatgpt":
+ raise ValueError("Codex credentials must use ChatGPT authentication")
+
+ baseline_tokens = baseline.get("tokens")
+ candidate_tokens = candidate.get("tokens")
+ if not isinstance(baseline_tokens, dict) or not
isinstance(candidate_tokens, dict):
+ raise ValueError("Codex credentials must contain a tokens object")
+ for token_name in ("access_token", "refresh_token"):
+ if not isinstance(baseline_tokens.get(token_name), str) or not
baseline_tokens[token_name]:
+ raise ValueError(f"Baseline {token_name} must be a non-empty
string")
+ if not isinstance(candidate_tokens.get(token_name), str) or not
candidate_tokens[token_name]:
+ raise ValueError(f"Candidate {token_name} must be a non-empty
string")
+
+ immutable_baseline = copy.deepcopy(baseline)
+ immutable_candidate = copy.deepcopy(candidate)
+ for credentials in (immutable_baseline, immutable_candidate):
+ del credentials["tokens"]["access_token"]
+ del credentials["tokens"]["refresh_token"]
+ credentials.pop("last_refresh", None)
+ if immutable_candidate != immutable_baseline:
Review Comment:
[P1] Promote the complete refresh-owned credential bundle
Current Codex can replace `id_token` during refresh and always advances
`last_refresh` ([upstream
implementation](https://github.com/openai/codex/blob/e597169e9a783156e50ae9765d891a3dd74df064/codex-rs/login/src/auth/manager.rs#L1308-L1331)).
Here the new ID token remains in the “immutable” comparison, so a valid
same-account refresh aborts the workflow; when it is unchanged, the returned
timestamp is still discarded, leaving stale refresh state. Validate stable
account/workspace claims instead, then atomically promote `id_token` when
returned together with access token, refresh token, and `last_refresh`.
##########
.github/workflows/code-review-retry.yml:
##########
@@ -0,0 +1,96 @@
+name: Retry Deferred Code Reviews
+
+on:
+ schedule:
+ - cron: '*/15 * * * *'
+ workflow_dispatch:
+
+permissions:
+ actions: write
+ contents: read
+ pull-requests: read
+
+concurrency:
+ group: code-review-deferred-dispatch
+ cancel-in-progress: false
+
+jobs:
+ dispatch-deferred-reviews:
+ runs-on: ubuntu-latest
+ steps:
+ - name: Install ossutil
+ run: |
+ tmp_dir="$(mktemp -d)"
+ trap 'rm -rf "$tmp_dir"' EXIT
+ curl -fsSL -o "$tmp_dir/ossutil.zip"
https://gosspublic.alicdn.com/ossutil/1.7.19/ossutil-v1.7.19-linux-amd64.zip
+ unzip -q "$tmp_dir/ossutil.zip" -d "$tmp_dir"
+ sudo install -m 0755 "$tmp_dir/ossutil-v1.7.19-linux-amd64/ossutil"
/usr/local/bin/ossutil
+
+ - name: Dispatch reviews that have reached their retry time
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ OSS_AK: ${{ secrets.OSS_AK }}
+ OSS_SK: ${{ secrets.OSS_SK }}
+ OSS_ENDPOINT: oss-cn-hongkong.aliyuncs.com
+ OSS_CODEX_REVIEW_PENDING_PREFIX:
oss://doris-community-ci/codex/review-pending
+ REPO: ${{ github.repository }}
+ run: |
+ pending_listing="$RUNNER_TEMP/codex-review-pending.list"
+ if ! ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" ls \
+ "$OSS_CODEX_REVIEW_PENDING_PREFIX/" -s > "$pending_listing"; then
+ echo 'Failed to list deferred Codex review requests.' >&2
+ exit 1
+ fi
+
+ now="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
+ while IFS= read -r pending_object; do
+ pending_file="$(mktemp "$RUNNER_TEMP/codex-review-pending.XXXXXX")"
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f \
+ "$pending_object" "$pending_file"
+ if ! jq -e '
+ (.pr_number | type == "string" and test("^[0-9]+$"))
+ and (.head_sha | type == "string" and length > 0)
+ and (.base_sha | type == "string" and length > 0)
+ and (.review_focus | type == "string")
+ and (.not_before | type == "string")
+ and (.request_id | type == "string" and test("^[0-9]+$"))
+ ' "$pending_file" >/dev/null; then
+ echo "Removing malformed deferred review request:
${pending_object##*/}"
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" rm -f
"$pending_object"
+ rm -f "$pending_file"
+ continue
+ fi
+
+ not_before="$(jq -r '.not_before' "$pending_file")"
+ if [ -n "$not_before" ] && [[ "$not_before" > "$now" ]]; then
+ rm -f "$pending_file"
+ continue
+ fi
+
+ pr_number="$(jq -r '.pr_number' "$pending_file")"
+ head_sha="$(jq -r '.head_sha' "$pending_file")"
+ base_sha="$(jq -r '.base_sha' "$pending_file")"
+ review_focus="$(jq -r '.review_focus' "$pending_file")"
+ request_id="$(jq -r '.request_id' "$pending_file")"
+ pull="$(gh api "repos/${REPO}/pulls/${pr_number}")"
+ if [ "$(jq -r '.state' <<<"$pull")" != 'open' ] \
+ || [ "$(jq -r '.head.sha' <<<"$pull")" != "$head_sha" ] \
+ || [ "$(jq -r '.base.sha' <<<"$pull")" != "$base_sha" ]; then
+ echo "Removing stale deferred review request:
${pending_object##*/}"
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" rm -f
"$pending_object"
+ rm -f "$pending_file"
+ continue
+ fi
+
+ gh workflow run code-review-runner.yml --ref "$GITHUB_REF_NAME" \
Review Comment:
[P1] Resolve the status after a deferred review
The initial `/review` workflow leaves the `code-review` context pending when
no account is eligible, but this scheduled dispatch bypasses that caller.
Neither this scheduler nor `code-review-runner.yml` has `statuses: write` or
posts a terminal status, so even a later successful review leaves the head
permanently pending and can keep a required context blocking merge. Give the
deferred run end-to-end status ownership: pending only while durably
queued/leased, success after verified submission, and an explicit failure/error
when a same-head request is abandoned.
##########
.github/workflows/code-review-comment.yml:
##########
@@ -51,6 +51,7 @@ jobs:
needs:
- resolve-pr
- mark-review-pending
+ - enqueue-review
Review Comment:
[P1] Recover when durable enqueue fails
`mark-review-pending` runs independently of `enqueue-review`. If the status
POST succeeds but the OSS upload fails, this job is skipped because a
dependency failed, and `refresh-required-check` also skips when `code-review`
is skipped. The head is then left permanently pending even though no runner or
deferred object can recover it. Publish pending only after durable enqueue
succeeds, or add an `always()` recovery path that reports the enqueue failure
and resolves the status.
##########
.github/workflows/code-review-runner.yml:
##########
@@ -487,53 +635,309 @@ jobs:
HEAD_SHA: ${{ inputs.head_sha }}
BASE_SHA: ${{ inputs.base_sha }}
+ - name: Grant isolated Codex user access to review context
+ run: |
+ sudo chown -R "$CODEX_REVIEW_USER:$CODEX_REVIEW_USER"
"$REVIEW_CONTEXT_DIR"
+
- name: Run automated code review
id: review
timeout-minutes: 115
continue-on-error: true
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ OSS_AK: ${{ secrets.OSS_AK }}
+ OSS_SK: ${{ secrets.OSS_SK }}
+ OSS_ENDPOINT: oss-cn-hongkong.aliyuncs.com
+ OSS_CODEX_AUTH_STATUS_OBJECT:
oss://doris-community-ci/codex/auth-status.json
+ OSS_CODEX_REVIEW_PENDING_PREFIX:
oss://doris-community-ci/codex/review-pending
+ AUTH_UNAVAILABLE: ${{ steps.configure_auth.outputs.auth_unavailable
}}
+ CONFIGURED_ALL_QUOTA_EXHAUSTED: ${{
steps.configure_auth.outputs.all_quota_exhausted }}
REPO: ${{ github.repository }}
PR_NUMBER: ${{ inputs.pr_number }}
HEAD_SHA: ${{ inputs.head_sha }}
+ BASE_SHA: ${{ inputs.base_sha }}
+ REVIEW_FOCUS: ${{ inputs.review_focus }}
+ DEFERRED_REQUEST_ID: ${{ inputs.deferred_request_id }}
run: |
- GOAL_PROMPT="$(cat "$REVIEW_CONTEXT_DIR/codex_goal_prompt.txt")"
- review_started_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
+ write_review_output() {
+ local key="$1"
+ local value="$2"
+ {
+ echo "${key}<<EOF"
+ printf '%s\n' "$value"
+ echo "EOF"
+ } >> "$GITHUB_OUTPUT"
+ }
+
+ defer_review() {
+ local retry_at="$1" request_id
+ request_id="${DEFERRED_REQUEST_ID:-$GITHUB_RUN_ID}"
+ local request_file
+ request_file="$(mktemp "$RUNNER_TEMP/codex-review-pending.XXXXXX")"
+ jq -n \
+ --arg pr_number "$PR_NUMBER" \
+ --arg head_sha "$HEAD_SHA" \
+ --arg base_sha "$BASE_SHA" \
+ --arg review_focus "$REVIEW_FOCUS" \
+ --arg not_before "$retry_at" \
+ --arg request_id "$request_id" \
+ '{pr_number: $pr_number, head_sha: $head_sha, base_sha:
$base_sha,
+ review_focus: $review_focus, not_before: $not_before,
request_id: $request_id}' > "$request_file"
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f \
+ "$request_file"
"$OSS_CODEX_REVIEW_PENDING_PREFIX/${PR_NUMBER}-${HEAD_SHA}-${request_id}.json"
+ rm -f "$request_file"
+ }
+
+ auth_object_args=()
+ known_auth_object_args=()
+ while IFS= read -r auth_object; do
+ auth_object_args+=(--auth-object "$auth_object")
+ known_auth_object_args+=(--known-auth-object "$auth_object")
+ done < "$CODEX_AUTH_OBJECTS_FILE"
+ if [ "${#auth_object_args[@]}" -eq 0 ]; then
+ write_review_output "all_quota_exhausted" "false"
+ write_review_output "failure_reason" "No valid Codex auth account
is available for review."
+ exit 1
+ fi
- set +e
- # GitHub-hosted runners are ephemeral. Avoid workspace-write here
because
- # Codex uses bubblewrap for that mode and uid maps can be
unavailable.
- codex exec --goal "$GOAL_PROMPT" \
- --cd "$GITHUB_WORKSPACE" \
- --model "gpt-5.6-sol" \
- --config "model_reasoning_effort=xhigh" \
- --sandbox danger-full-access \
- --color never \
- --json \
- --output-last-message
"$REVIEW_CONTEXT_DIR/codex-final-message.txt" \
- > "$REVIEW_CONTEXT_DIR/codex-events.jsonl" \
- 2> >(tee "$REVIEW_CONTEXT_DIR/codex-stderr.log" >&2)
- status=$?
- set -e
+ sync_auth_state() {
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f \
+ "$CODEX_AUTH_STATE_FILE" "$OSS_CODEX_AUTH_STATUS_OBJECT"
+ }
+
+ record_auth_result() {
+ local result="$1"
+ local http_status="$2"
+ local retry_after="$3"
+ local record_args=(
+ record
+ --state "$CODEX_AUTH_STATE_FILE"
+ --auth-object "$CODEX_AUTH_STATE_KEY"
+ "${known_auth_object_args[@]}"
+ --result "$result"
+ )
+ if [ -n "$http_status" ]; then
+ record_args+=(--http-status "$http_status")
+ fi
+ if [ -n "$retry_after" ]; then
+ record_args+=(--retry-after "$retry_after")
+ fi
+ "$CODEX_AUTH_STATE_HELPER" "${record_args[@]}"
+ }
+
+ install_selected_auth() {
+ local candidate_auth_file
+ candidate_auth_file="$(mktemp "$RUNNER_TEMP/codex-auth.XXXXXX")"
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f \
+ "$CODEX_AUTH_OSS_OBJECT" "$candidate_auth_file"
+ jq -e '
+ .auth_mode == "chatgpt"
+ and (.tokens.access_token | type == "string" and length > 0)
+ and (.tokens.refresh_token | type == "string" and length > 0)
+ ' "$candidate_auth_file" >/dev/null
+ sudo install -o "$CODEX_REVIEW_USER" -g "$CODEX_REVIEW_USER" -m
600 \
+ "$candidate_auth_file" "$CODEX_HOME/auth.json"
+ install -m 600 "$candidate_auth_file" "$CODEX_AUTH_SNAPSHOT"
+ rm -f "$candidate_auth_file"
+ }
+
+ sync_refreshed_auth() {
+ local refreshed_auth_file promoted_auth_file
+ refreshed_auth_file="$(mktemp
"$RUNNER_TEMP/codex-refreshed-auth.XXXXXX")"
+ promoted_auth_file="$(mktemp
"$RUNNER_TEMP/codex-promoted-auth.XXXXXX")"
+ sudo -u "$CODEX_REVIEW_USER" cat "$CODEX_HOME/auth.json" >
"$refreshed_auth_file"
+ "$CODEX_AUTH_STATE_HELPER" promote \
+ --baseline "$CODEX_AUTH_SNAPSHOT" \
+ --candidate "$refreshed_auth_file" \
+ --output "$promoted_auth_file"
+ if ! cmp -s "$refreshed_auth_file" "$CODEX_AUTH_SNAPSHOT"; then
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f \
+ "$promoted_auth_file" "$CODEX_AUTH_OSS_OBJECT"
+ install -m 600 "$promoted_auth_file" "$CODEX_AUTH_SNAPSHOT"
+ fi
+ rm -f "$refreshed_auth_file" "$promoted_auth_file"
+ }
+
+ review_side_effects_present() {
+ local review_sync_dir="$RUNNER_TEMP/codex-review-sync"
+ local reviews_file="$review_sync_dir/reviews.json"
+ local inline_comments_file="$review_sync_dir/inline-comments.json"
+ local issue_comments_file="$review_sync_dir/issue-comments.json"
+ install -m 700 -d "$review_sync_dir"
+
+ for attempt in 1 2 3 4 5 6; do
+ if ! gh api --paginate --slurp
"repos/${REPO}/pulls/${PR_NUMBER}/reviews" > "$reviews_file" \
+ || ! gh api --paginate --slurp
"repos/${REPO}/pulls/${PR_NUMBER}/comments" > "$inline_comments_file" \
+ || ! gh api --paginate --slurp
"repos/${REPO}/issues/${PR_NUMBER}/comments" > "$issue_comments_file"; then
+ return 2
+ fi
+ if jq -s -e --arg started_at "$review_started_at" --arg head_sha
"$HEAD_SHA" '
+ map(add // []) | add // []
+ | any(.[]?; ((.submitted_at // .created_at // "") >=
$started_at)
+ and ((.commit_id // $head_sha) == $head_sha))
+ ' "$reviews_file" "$inline_comments_file" "$issue_comments_file"
>/dev/null; then
+ return 0
+ fi
+ sleep 5
+ done
+ return 1
+ }
+
+ select_next_auth() {
+ local selection_json
+ selection_json="$("$CODEX_AUTH_STATE_HELPER" select \
+ --state "$CODEX_AUTH_STATE_FILE" \
+ "${auth_object_args[@]}")"
+ sync_auth_state
+ selected_auth_state_key="$(jq -r '.auth_object // empty'
<<<"$selection_json")"
+ selected_auth_object="${selected_auth_state_key%%#*}"
+ all_quota_exhausted="$(jq -r '.all_quota_exhausted'
<<<"$selection_json")"
+ next_retry_at="$(jq -r '.next_retry_at // empty'
<<<"$selection_json")"
+ }
+
+ if [ "$AUTH_UNAVAILABLE" = "true" ]; then
+ write_review_output "all_quota_exhausted"
"$CONFIGURED_ALL_QUOTA_EXHAUSTED"
+ write_review_output "failure_reason" "No Codex auth account is
currently eligible for review."
+ exit 1
+ fi
+ GOAL_PROMPT="$(cat "$REVIEW_CONTEXT_DIR/codex_goal_prompt.txt")"
failure_reason=""
- if [ "$status" -ne 0 ]; then
- if [ -s "$REVIEW_CONTEXT_DIR/codex-events.jsonl" ]; then
- failure_reason="$(jq -r 'select(.type == "turn.failed") |
.error.message // empty' "$REVIEW_CONTEXT_DIR/codex-events.jsonl" | tail -n 1)"
+ all_quota_exhausted="${CONFIGURED_ALL_QUOTA_EXHAUSTED:-false}"
+ transient_attempt=0
Review Comment:
[P2] Preserve root telemetry for every retry attempt
Every iteration redirects the root stream to the same `codex-events.jsonl`
and `codex-stderr.log`, truncating the prior attempt. Litefuse's JSONL input
therefore loses account A's root events, usage, and terminal failure when
account B is attempted; the separate subagent-session scan cannot reconstruct
that root context. Use per-attempt root files and emit or aggregate them with
attempt/account metadata while keeping the final output separate.
##########
.github/workflows/code-review-runner.yml:
##########
@@ -487,53 +635,309 @@ jobs:
HEAD_SHA: ${{ inputs.head_sha }}
BASE_SHA: ${{ inputs.base_sha }}
+ - name: Grant isolated Codex user access to review context
+ run: |
+ sudo chown -R "$CODEX_REVIEW_USER:$CODEX_REVIEW_USER"
"$REVIEW_CONTEXT_DIR"
+
- name: Run automated code review
id: review
timeout-minutes: 115
continue-on-error: true
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ OSS_AK: ${{ secrets.OSS_AK }}
+ OSS_SK: ${{ secrets.OSS_SK }}
+ OSS_ENDPOINT: oss-cn-hongkong.aliyuncs.com
+ OSS_CODEX_AUTH_STATUS_OBJECT:
oss://doris-community-ci/codex/auth-status.json
+ OSS_CODEX_REVIEW_PENDING_PREFIX:
oss://doris-community-ci/codex/review-pending
+ AUTH_UNAVAILABLE: ${{ steps.configure_auth.outputs.auth_unavailable
}}
+ CONFIGURED_ALL_QUOTA_EXHAUSTED: ${{
steps.configure_auth.outputs.all_quota_exhausted }}
REPO: ${{ github.repository }}
PR_NUMBER: ${{ inputs.pr_number }}
HEAD_SHA: ${{ inputs.head_sha }}
+ BASE_SHA: ${{ inputs.base_sha }}
+ REVIEW_FOCUS: ${{ inputs.review_focus }}
+ DEFERRED_REQUEST_ID: ${{ inputs.deferred_request_id }}
run: |
- GOAL_PROMPT="$(cat "$REVIEW_CONTEXT_DIR/codex_goal_prompt.txt")"
- review_started_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
+ write_review_output() {
+ local key="$1"
+ local value="$2"
+ {
+ echo "${key}<<EOF"
+ printf '%s\n' "$value"
+ echo "EOF"
+ } >> "$GITHUB_OUTPUT"
+ }
+
+ defer_review() {
+ local retry_at="$1" request_id
+ request_id="${DEFERRED_REQUEST_ID:-$GITHUB_RUN_ID}"
+ local request_file
+ request_file="$(mktemp "$RUNNER_TEMP/codex-review-pending.XXXXXX")"
+ jq -n \
+ --arg pr_number "$PR_NUMBER" \
+ --arg head_sha "$HEAD_SHA" \
+ --arg base_sha "$BASE_SHA" \
+ --arg review_focus "$REVIEW_FOCUS" \
+ --arg not_before "$retry_at" \
+ --arg request_id "$request_id" \
+ '{pr_number: $pr_number, head_sha: $head_sha, base_sha:
$base_sha,
+ review_focus: $review_focus, not_before: $not_before,
request_id: $request_id}' > "$request_file"
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f \
+ "$request_file"
"$OSS_CODEX_REVIEW_PENDING_PREFIX/${PR_NUMBER}-${HEAD_SHA}-${request_id}.json"
+ rm -f "$request_file"
+ }
+
+ auth_object_args=()
+ known_auth_object_args=()
+ while IFS= read -r auth_object; do
+ auth_object_args+=(--auth-object "$auth_object")
+ known_auth_object_args+=(--known-auth-object "$auth_object")
+ done < "$CODEX_AUTH_OBJECTS_FILE"
+ if [ "${#auth_object_args[@]}" -eq 0 ]; then
+ write_review_output "all_quota_exhausted" "false"
+ write_review_output "failure_reason" "No valid Codex auth account
is available for review."
+ exit 1
+ fi
- set +e
- # GitHub-hosted runners are ephemeral. Avoid workspace-write here
because
- # Codex uses bubblewrap for that mode and uid maps can be
unavailable.
- codex exec --goal "$GOAL_PROMPT" \
- --cd "$GITHUB_WORKSPACE" \
- --model "gpt-5.6-sol" \
- --config "model_reasoning_effort=xhigh" \
- --sandbox danger-full-access \
- --color never \
- --json \
- --output-last-message
"$REVIEW_CONTEXT_DIR/codex-final-message.txt" \
- > "$REVIEW_CONTEXT_DIR/codex-events.jsonl" \
- 2> >(tee "$REVIEW_CONTEXT_DIR/codex-stderr.log" >&2)
- status=$?
- set -e
+ sync_auth_state() {
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f \
+ "$CODEX_AUTH_STATE_FILE" "$OSS_CODEX_AUTH_STATUS_OBJECT"
+ }
+
+ record_auth_result() {
+ local result="$1"
+ local http_status="$2"
+ local retry_after="$3"
+ local record_args=(
+ record
+ --state "$CODEX_AUTH_STATE_FILE"
+ --auth-object "$CODEX_AUTH_STATE_KEY"
+ "${known_auth_object_args[@]}"
+ --result "$result"
+ )
+ if [ -n "$http_status" ]; then
+ record_args+=(--http-status "$http_status")
+ fi
+ if [ -n "$retry_after" ]; then
+ record_args+=(--retry-after "$retry_after")
+ fi
+ "$CODEX_AUTH_STATE_HELPER" "${record_args[@]}"
+ }
+
+ install_selected_auth() {
+ local candidate_auth_file
+ candidate_auth_file="$(mktemp "$RUNNER_TEMP/codex-auth.XXXXXX")"
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f \
+ "$CODEX_AUTH_OSS_OBJECT" "$candidate_auth_file"
+ jq -e '
+ .auth_mode == "chatgpt"
+ and (.tokens.access_token | type == "string" and length > 0)
+ and (.tokens.refresh_token | type == "string" and length > 0)
+ ' "$candidate_auth_file" >/dev/null
+ sudo install -o "$CODEX_REVIEW_USER" -g "$CODEX_REVIEW_USER" -m
600 \
+ "$candidate_auth_file" "$CODEX_HOME/auth.json"
+ install -m 600 "$candidate_auth_file" "$CODEX_AUTH_SNAPSHOT"
+ rm -f "$candidate_auth_file"
+ }
+
+ sync_refreshed_auth() {
+ local refreshed_auth_file promoted_auth_file
+ refreshed_auth_file="$(mktemp
"$RUNNER_TEMP/codex-refreshed-auth.XXXXXX")"
+ promoted_auth_file="$(mktemp
"$RUNNER_TEMP/codex-promoted-auth.XXXXXX")"
+ sudo -u "$CODEX_REVIEW_USER" cat "$CODEX_HOME/auth.json" >
"$refreshed_auth_file"
+ "$CODEX_AUTH_STATE_HELPER" promote \
+ --baseline "$CODEX_AUTH_SNAPSHOT" \
+ --candidate "$refreshed_auth_file" \
+ --output "$promoted_auth_file"
+ if ! cmp -s "$refreshed_auth_file" "$CODEX_AUTH_SNAPSHOT"; then
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f \
+ "$promoted_auth_file" "$CODEX_AUTH_OSS_OBJECT"
+ install -m 600 "$promoted_auth_file" "$CODEX_AUTH_SNAPSHOT"
+ fi
+ rm -f "$refreshed_auth_file" "$promoted_auth_file"
+ }
+
+ review_side_effects_present() {
+ local review_sync_dir="$RUNNER_TEMP/codex-review-sync"
+ local reviews_file="$review_sync_dir/reviews.json"
+ local inline_comments_file="$review_sync_dir/inline-comments.json"
+ local issue_comments_file="$review_sync_dir/issue-comments.json"
+ install -m 700 -d "$review_sync_dir"
+
+ for attempt in 1 2 3 4 5 6; do
+ if ! gh api --paginate --slurp
"repos/${REPO}/pulls/${PR_NUMBER}/reviews" > "$reviews_file" \
+ || ! gh api --paginate --slurp
"repos/${REPO}/pulls/${PR_NUMBER}/comments" > "$inline_comments_file" \
Review Comment:
[P1] Reconcile only artifacts owned by this attempt
This treats every review or comment after `review_started_at` as a Codex
side effect without checking its actor or an attempt-owned artifact ID. Issue
comments have no `commit_id`, and the fallback assigns them `$head_sha`, so an
unrelated maintainer comment aborts otherwise safe retries. The final verifier
at lines 946-949 repeats the ownership omission for reviews, allowing an
unrelated maintainer review to falsely prove that a zero-exit Codex attempt
submitted one. Snapshot IDs before the attempt and require artifacts
attributable to this workflow/request in both consumers.
##########
.github/workflows/code-review-runner.yml:
##########
@@ -101,26 +127,138 @@ jobs:
OSS_CODEX_GOAL_FALLBACK_OBJECT:
oss://doris-community-ci/codex/codex-goal
- name: Configure Codex auth
+ id: configure_auth
run: |
- install -m 700 -d "$RUNNER_TEMP/codex-home"
- printf 'CODEX_HOME=%s\n' "$RUNNER_TEMP/codex-home" >> "$GITHUB_ENV"
+ defer_review() {
+ local retry_at="$1" request_id
+ request_id="${DEFERRED_REQUEST_ID:-$GITHUB_RUN_ID}"
+ local request_file
+ request_file="$(mktemp "$RUNNER_TEMP/codex-review-pending.XXXXXX")"
+ jq -n \
+ --arg pr_number "$PR_NUMBER" \
+ --arg head_sha "$HEAD_SHA" \
+ --arg base_sha "$BASE_SHA" \
+ --arg review_focus "$REVIEW_FOCUS" \
+ --arg not_before "$retry_at" \
+ --arg request_id "$request_id" \
+ '{pr_number: $pr_number, head_sha: $head_sha, base_sha:
$base_sha,
+ review_focus: $review_focus, not_before: $not_before,
request_id: $request_id}' > "$request_file"
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f \
+ "$request_file"
"$OSS_CODEX_REVIEW_PENDING_PREFIX/${PR_NUMBER}-${HEAD_SHA}-${request_id}.json"
+ rm -f "$request_file"
+ }
+
+ codex_review_user=codex-review
+ sudo useradd --create-home --shell /usr/sbin/nologin
"$codex_review_user"
+ codex_home="$(mktemp -d /tmp/codex-home.XXXXXX)"
+ sudo chown "$codex_review_user:$codex_review_user" "$codex_home"
+ sudo chgrp "$(id -gn)" "$codex_home"
+ sudo chmod 770 "$codex_home"
+ control_dir="$RUNNER_TEMP/codex-auth-control"
+ install -m 700 -d "$control_dir"
+ printf 'CODEX_HOME=%s\n' "$codex_home" >> "$GITHUB_ENV"
+ printf 'CODEX_REVIEW_USER=%s\n' "$codex_review_user" >> "$GITHUB_ENV"
+
+ auth_prefix='oss://doris-community-ci/codex/auth.json.'
+ auth_listing="$RUNNER_TEMP/codex-auth-objects.list"
+ if ! ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" ls
"$auth_prefix" -s > "$auth_listing"; then
+ echo 'Failed to list Codex auth objects from OSS.'
+ exit 1
+ fi
+
+ auth_objects_file="$control_dir/auth-objects.txt"
+ candidate_auth_file="$(mktemp "$RUNNER_TEMP/codex-auth.XXXXXX")"
+ trap 'rm -f "$candidate_auth_file"' EXIT
+ while IFS= read -r candidate; do
+ if ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f
"$candidate" "$candidate_auth_file" \
+ && test -s "$candidate_auth_file" \
+ && jq -e '
+ .auth_mode == "chatgpt"
+ and (.tokens.access_token | type == "string" and length > 0)
+ and (.tokens.refresh_token | type == "string" and length > 0)
+ ' "$candidate_auth_file" >/dev/null; then
+ candidate_hash="$(sha256sum "$candidate_auth_file" | awk '{print
$1}')"
+ printf '%s#%s\n' "$candidate" "$candidate_hash" >>
"$auth_objects_file"
+ else
+ echo "Skipping invalid Codex auth object: ${candidate##*/}"
+ fi
+ done < <(
+ awk '$0 ~
/^oss:\/\/doris-community-ci\/codex\/auth\.json\.[0-9]+$/ { print }'
"$auth_listing" | sort
+ )
+
+ if [ ! -s "$auth_objects_file" ]; then
+ echo 'No valid Codex auth object found in OSS.'
+ exit 1
+ fi
+ chmod 600 "$auth_objects_file"
+
+ auth_object_args=()
+ while IFS= read -r auth_object; do
+ auth_object_args+=(--auth-object "$auth_object")
+ done < "$auth_objects_file"
+
+ state_file="$control_dir/auth-status.json"
+ set +e
+ state_stat_output="$(ossutil -i "$OSS_AK" -k "$OSS_SK" -e
"$OSS_ENDPOINT" stat "$OSS_CODEX_AUTH_STATUS_OBJECT" 2>&1)"
+ state_stat_status=$?
+ set -e
+ if [ "$state_stat_status" -eq 0 ]; then
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f
"$OSS_CODEX_AUTH_STATUS_OBJECT" "$state_file"
+ elif grep -Eqi 'NoSuchKey|NoSuchObject|404' <<<"$state_stat_output";
then
+ "$CODEX_AUTH_STATE_HELPER" initialize \
+ --state "$state_file" \
+ "${auth_object_args[@]}"
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f
"$state_file" "$OSS_CODEX_AUTH_STATUS_OBJECT"
+ echo "Created Codex auth status file in OSS."
+ else
+ echo "Unable to determine whether the Codex auth status file
exists." >&2
+ exit 1
+ fi
+
+ selection_json="$("$CODEX_AUTH_STATE_HELPER" select \
+ --state "$state_file" \
+ "${auth_object_args[@]}")"
+ ossutil -i "$OSS_AK" -k "$OSS_SK" -e "$OSS_ENDPOINT" cp -f
"$state_file" "$OSS_CODEX_AUTH_STATUS_OBJECT"
+ chmod 600 "$state_file"
+ printf 'CODEX_AUTH_OBJECTS_FILE=%s\n' "$auth_objects_file" >>
"$GITHUB_ENV"
+ printf 'CODEX_AUTH_STATE_FILE=%s\n' "$state_file" >> "$GITHUB_ENV"
+
+ auth_state_key="$(jq -r '.auth_object // empty'
<<<"$selection_json")"
+ auth_object="${auth_state_key%%#*}"
+ all_quota_exhausted="$(jq -r '.all_quota_exhausted'
<<<"$selection_json")"
+ printf 'all_quota_exhausted=%s\n' "$all_quota_exhausted" >>
"$GITHUB_OUTPUT"
+ if [ -z "$auth_object" ]; then
+ next_retry_at="$(jq -r '.next_retry_at // empty'
<<<"$selection_json")"
+ if [ -n "$next_retry_at" ]; then
+ defer_review "$next_retry_at"
+ fi
+ echo "No Codex auth account is currently eligible; next retry:
${next_retry_at:-unknown}"
+ printf 'auth_unavailable=true\n' >> "$GITHUB_OUTPUT"
+ exit 0
+ fi
Review Comment:
[P1] Claim the deferred request before dispatch
The scheduler dispatches this request while leaving its immediately-due OSS
object pending, and this best-effort deletion occurs only after
checkout/install/setup. A scheduler tick before this line can queue the same
request again; a delete failure leaves the same replay path afterward. The
concurrency group merely serializes those runs, and the queued run does not
require the object still to exist, so a second complete review can execute.
Introduce a unique `pending -> claimed` transition with a lease/generation that
the runner validates, then acknowledge/delete only on terminal completion.
##########
.github/scripts/codex_auth_state.py:
##########
@@ -0,0 +1,453 @@
+#!/usr/bin/env python3
+"""Persist and select the rotating credentials used by code review jobs."""
+
+from __future__ import annotations
+
+import argparse
+import copy
+import json
+import re
+import tempfile
+from dataclasses import dataclass
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from typing import Any
+
+
+STATE_VERSION = 1
+AVAILABLE = "available"
+QUOTA_EXHAUSTED = "quota_exhausted"
+RATE_LIMITED = "rate_limited"
+AUTHENTICATION_FAILED = "authentication_failed"
+TRANSIENT_FAILURE = "transient_failure"
+STATUS_VALUES = {AVAILABLE, QUOTA_EXHAUSTED, RATE_LIMITED,
AUTHENTICATION_FAILED}
+QUOTA_RETRY_DELAY = timedelta(days=1)
+RATE_LIMIT_RETRY_DELAY = timedelta(hours=1)
+PERMANENT_AUTH_FAILURE_PATTERN = re.compile(
+ r"refresh[\s_-]*token.*(?:revoked|expired|already\s+used)|"
+ r"(?:revoked|expired).*refresh[\s_-]*token",
+ re.IGNORECASE,
+)
+USAGE_LIMIT_MESSAGE_PATTERN = re.compile(
+ r"you(?:'|\u2019)ve hit your usage limit|usage limit",
+ re.IGNORECASE,
+)
+USAGE_LIMIT_RETRY_PATTERN = re.compile(
+ r"try again
at\s+(?:(?P<date>[A-Za-z]{3,9}\s+\d{1,2}(?:st|nd|rd|th)?,\s+\d{4}\s+"
+ r"\d{1,2}:\d{2}\s+(?:AM|PM))|(?P<time>\d{1,2}:\d{2}\s+(?:AM|PM)))",
+ re.IGNORECASE | re.DOTALL,
+)
+
+
+@dataclass(frozen=True)
+class Selection:
+ auth_object: str | None
+ all_quota_exhausted: bool
+ next_retry_at: str | None
+
+
+@dataclass(frozen=True)
+class FailureClassification:
+ kind: str
+ http_status: int | None
+ retry_after: str | None
+
+
+def utc_now() -> datetime:
+ return datetime.now(timezone.utc).replace(microsecond=0)
+
+
+def parse_timestamp(value: str) -> datetime:
+ parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
+ if parsed.tzinfo is None:
+ raise ValueError(f"Timestamp must include a timezone: {value!r}")
+ return parsed.astimezone(timezone.utc).replace(microsecond=0)
+
+
+def format_timestamp(value: datetime) -> str:
+ return
value.astimezone(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00",
"Z")
+
+
+def default_account_state() -> dict[str, Any]:
+ return {
+ "status": AVAILABLE,
+ "last_failure_at": None,
+ "last_http_status": None,
+ "last_success_at": None,
+ "last_recovered_at": None,
+ "retry_after": None,
+ }
+
+
+def default_state() -> dict[str, Any]:
+ return {"version": STATE_VERSION, "next_account": 0, "accounts": {}}
+
+
+def load_state(path: Path) -> dict[str, Any]:
+ if not path.exists():
+ return default_state()
+
+ state = json.loads(path.read_text())
+ if not isinstance(state, dict):
+ raise ValueError("Authentication state must be a JSON object")
+ if state.get("version") != STATE_VERSION:
+ raise ValueError(f"Unsupported authentication state version:
{state.get('version')!r}")
+ if not isinstance(state.get("next_account"), int):
+ raise ValueError("Authentication state next_account must be an
integer")
+ if not isinstance(state.get("accounts"), dict):
+ raise ValueError("Authentication state accounts must be an object")
+ return state
+
+
+def save_state(path: Path, state: dict[str, Any]) -> None:
+ with tempfile.NamedTemporaryFile("w", encoding="utf-8", dir=path.parent,
delete=False) as temporary:
+ json.dump(state, temporary, indent=2, sort_keys=True)
+ temporary.write("\n")
+ temporary_path = Path(temporary.name)
+ temporary_path.replace(path)
+
+
+def promote_auth(baseline: dict[str, Any], candidate: dict[str, Any]) ->
dict[str, Any]:
+ if baseline.get("auth_mode") != "chatgpt" or candidate.get("auth_mode") !=
"chatgpt":
+ raise ValueError("Codex credentials must use ChatGPT authentication")
+
+ baseline_tokens = baseline.get("tokens")
+ candidate_tokens = candidate.get("tokens")
+ if not isinstance(baseline_tokens, dict) or not
isinstance(candidate_tokens, dict):
+ raise ValueError("Codex credentials must contain a tokens object")
+ for token_name in ("access_token", "refresh_token"):
+ if not isinstance(baseline_tokens.get(token_name), str) or not
baseline_tokens[token_name]:
+ raise ValueError(f"Baseline {token_name} must be a non-empty
string")
+ if not isinstance(candidate_tokens.get(token_name), str) or not
candidate_tokens[token_name]:
+ raise ValueError(f"Candidate {token_name} must be a non-empty
string")
+
+ immutable_baseline = copy.deepcopy(baseline)
+ immutable_candidate = copy.deepcopy(candidate)
+ for credentials in (immutable_baseline, immutable_candidate):
+ del credentials["tokens"]["access_token"]
+ del credentials["tokens"]["refresh_token"]
+ credentials.pop("last_refresh", None)
+ if immutable_candidate != immutable_baseline:
+ raise ValueError("Refreshed credentials changed an immutable identity
field")
+
+ promoted = copy.deepcopy(baseline)
+ promoted["tokens"]["access_token"] = candidate_tokens["access_token"]
+ promoted["tokens"]["refresh_token"] = candidate_tokens["refresh_token"]
+ return promoted
+
+
+def promote_auth_file(baseline_path: Path, candidate_path: Path, output_path:
Path) -> None:
+ baseline = json.loads(baseline_path.read_text())
+ candidate = json.loads(candidate_path.read_text())
+ if not isinstance(baseline, dict) or not isinstance(candidate, dict):
+ raise ValueError("Codex credentials must be JSON objects")
+ save_state(output_path, promote_auth(baseline, candidate))
+
+
+def account_state(state: dict[str, Any], auth_object: str) -> dict[str, Any]:
+ accounts = state["accounts"]
+ entry = accounts.setdefault(auth_object, default_account_state())
+ if not isinstance(entry, dict):
+ raise ValueError(f"Authentication state for {auth_object!r} must be an
object")
+ status = entry.get("status", AVAILABLE)
+ if status not in STATUS_VALUES:
+ raise ValueError(f"Unknown authentication status for {auth_object!r}:
{status!r}")
+ entry.setdefault("status", AVAILABLE)
+ entry.setdefault("last_failure_at", None)
+ entry.setdefault("last_http_status", None)
+ entry.setdefault("last_success_at", None)
+ entry.setdefault("last_recovered_at", None)
+ entry.setdefault("retry_after", None)
+ return entry
+
+
+def recover_expired_accounts(state: dict[str, Any], auth_objects: list[str],
now: datetime) -> None:
+ for auth_object in auth_objects:
+ entry = account_state(state, auth_object)
+ retry_after = entry["retry_after"]
+ if retry_after is None:
+ continue
+ if parse_timestamp(retry_after) <= now:
+ entry["status"] = AVAILABLE
+ entry["retry_after"] = None
+ entry["last_recovered_at"] = format_timestamp(now)
+
+
+def next_retry_at(state: dict[str, Any], auth_objects: list[str]) -> str |
None:
+ retry_at = [
+ entry["retry_after"]
+ for auth_object in auth_objects
+ for entry in [account_state(state, auth_object)]
+ if entry["retry_after"] is not None
+ ]
+ return min(retry_at, key=parse_timestamp, default=None)
+
+
+def all_quota_exhausted(state: dict[str, Any], auth_objects: list[str]) ->
bool:
+ return bool(auth_objects) and all(
+ account_state(state, auth_object)["status"] == QUOTA_EXHAUSTED for
auth_object in auth_objects
+ )
+
+
+def select_auth_object(state: dict[str, Any], auth_objects: list[str], now:
datetime) -> Selection:
+ if not auth_objects:
+ raise ValueError("At least one auth object is required")
+
+ auth_objects = sorted(set(auth_objects))
+ recover_expired_accounts(state, auth_objects, now)
+ starting_index = state["next_account"] % len(auth_objects)
+ for offset in range(len(auth_objects)):
+ index = (starting_index + offset) % len(auth_objects)
+ auth_object = auth_objects[index]
+ if account_state(state, auth_object)["status"] == AVAILABLE:
+ state["next_account"] = (index + 1) % len(auth_objects)
+ return Selection(auth_object, False, None)
+
+ return Selection(None, all_quota_exhausted(state, auth_objects),
next_retry_at(state, auth_objects))
+
+
+def record_result(
+ state: dict[str, Any],
+ auth_object: str,
+ result: str,
+ now: datetime,
+ http_status: int | None = None,
+ retry_after: datetime | None = None,
+) -> None:
+ entry = account_state(state, auth_object)
+ timestamp = format_timestamp(now)
+ if result == AVAILABLE:
+ entry["status"] = AVAILABLE
+ entry["last_success_at"] = timestamp
+ entry["retry_after"] = None
+ return
+
+ if result == QUOTA_EXHAUSTED:
+ http_status = http_status or 403
+ retry_at = retry_after or now + QUOTA_RETRY_DELAY
+ elif result == RATE_LIMITED:
+ http_status = http_status or 429
+ elif result == AUTHENTICATION_FAILED:
+ http_status = http_status or 401
+ retry_at = None
+ elif result == TRANSIENT_FAILURE:
+ entry["status"] = AVAILABLE
+ entry["last_failure_at"] = timestamp
+ entry["last_http_status"] = http_status
+ entry["retry_after"] = None
+ return
+ else:
+ raise ValueError(f"Unsupported result: {result!r}")
+
+ if result == RATE_LIMITED:
+ retry_at = now + RATE_LIMIT_RETRY_DELAY
+
+ entry["status"] = result
+ entry["last_failure_at"] = timestamp
+ entry["last_http_status"] = http_status
+ entry["retry_after"] = format_timestamp(retry_at) if retry_at is not None
else None
+
+
+def extract_http_status(text: str) -> int | None:
+ labelled_statuses = re.findall(
+
r"(?:http(?:\s+status)?|status(?:\s+code)?|error\s+code)\s*[:=]?\s*([1-5]\d{2})",
Review Comment:
[P1] Recognize Codex's actual refresh-error format
Current Codex reports transient refresh responses as `Failed to refresh
token: 503 Service Unavailable: ...`
([source](https://github.com/openai/codex/blob/e597169e9a783156e50ae9765d891a3dd74df064/codex-rs/login/src/auth/manager.rs#L1356-L1373)),
but this regex only extracts numbers labelled `HTTP`, `status`, or `error
code`. Passing that exact message to this helper returns `fatal` with no
status, so the workflow skips its transient retry path. The generic permanent
refresh message is likewise missed and skips account rotation. Prefer a
structured failure kind/status; otherwise cover Codex's canonical refresh
messages and add upstream-shaped fixtures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
