Ankurdeewan opened a new pull request, #59798:
URL: https://github.com/apache/airflow/pull/59798
This PR adds an opt-in deferrable mode to `AzureBatchOperator`.
**What this changes**
- Adds a `deferrable` flag (default: `False`).
- When `deferrable=True`, the operator:
- submits the Azure Batch pool/job/task
- immediately defers
- moves all waiting and polling to a trigger
- The worker no longer blocks on sleep-based polling in deferrable mode.
**What stays the same**
- Default behavior is unchanged.
- With `deferrable=False`, the operator still runs exactly as before:
- same polling logic
- same exceptions
- same cleanup behavior on both success and failure
- Existing DAGs are not affected unless `deferrable=True` is explicitly
enabled.
**Cleanup behavior**
Cleanup semantics are preserved:
- Cleanup runs on both success and failure, matching historical behavior.
- Cleanup does not run inside `execute_complete()`.
- Cleanup runs exactly once in all paths (sync, deferrable, failure,
timeout).
**Why this is needed**
Previously, the operator blocked a worker while polling Azure Batch using
sleep-based loops.
This change aligns the Azure Batch operator with existing deferrable
patterns and reduces unnecessary worker usage.
Related issue: #59779
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
