1fanwang opened a new pull request, #72400: URL: https://github.com/apache/airflow/pull/72400
Synchronous `AnthropicBatchOperator` retries can create another billable Message Batch when a worker dies while polling. The batch ID is pushed to XCom, but retries clear XCom before the next attempt. On Airflow 3.3+, synchronous waits now persist the batch ID through `ResumableJobMixin`. Retries reconnect to active or canceling batches and recover successful terminal batches. Failed, fully canceled, strict partial-error, expired, and missing batches submit fresh. Deferrable execution and `wait_for_completion=False` keep their existing behavior. ## Testing ```text $ uv run --project providers/anthropic pytest providers/anthropic/tests/unit/anthropic/operators/test_batch.py -q 35 passed, 1 warning in 16.86s ``` A local fake Anthropic client drove the real operator flow and `TaskStateStoreAccessor` supervisor messages. The first attempt crashed during polling without requesting cancellation. The same retry then ran against the baseline and changed implementations. <details> <summary>Raw retry proof</summary> ```text $ uv run --project providers/anthropic python anthropic_batch_retry_proof.py REMOTE SUBMIT id=batch_1 WORKER CRASH id=batch_1 STATE GET key=anthropic_batch_id value=None REMOTE SUBMIT id=batch_2 REMOTE COMPLETE id=batch_2 BASELINE RESULT submissions=2 stored_after_crash=None retry_result=batch_2 cancelled=[] STATE GET key=anthropic_batch_id value=None REMOTE SUBMIT id=batch_1 STATE SET key=anthropic_batch_id value=batch_1 WORKER CRASH id=batch_1 STATE GET key=anthropic_batch_id value=batch_1 STATE GET key=anthropic_batch_id value=batch_1 REMOTE GET id=batch_1 status=in_progress Reconnecting to existing job external_id_key=anthropic_batch_id external_id=batch_1 status=in_progress REMOTE COMPLETE id=batch_1 CHANGED RESULT submissions=1 stored_after_crash=batch_1 retry_result=batch_1 cancelled=[] ``` </details> --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes - GitHub Copilot CLI (GPT-5.6 Sol) Generated-by: GitHub Copilot CLI (GPT-5.6 Sol) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
