1fanwang opened a new pull request, #72399:
URL: https://github.com/apache/airflow/pull/72399
Retries of synchronous `AzureDataFactoryRunPipelineOperator` tasks currently
submit a second Azure pipeline run when the worker exits after submission.
After this change, a retry reconnects to an active run or recovers a succeeded
run. Failed and cancelled runs still submit a new pipeline. Asynchronous and
deferrable execution are unchanged.
The operator now uses `ResumableJobMixin` for synchronous waits. It stores
the Azure run ID before polling, checks that run from a fresh process, and
restores both `self.run_id` and the `run_id` XCom when reconnecting. Airflow
versions without resumable jobs keep the prior behavior and warn if durability
is requested explicitly.
## Testing
```console
AIRFLOW_HOME=<isolated> uv run --project providers/microsoft/azure pytest
providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_data_factory.py::TestAzureDataFactoryRunPipelineOperatorResumable
-q
======================== 18 passed, 1 warning in 13.07s
========================
AIRFLOW_HOME=<isolated> uv run --project providers/microsoft/azure pytest
providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_data_factory.py
-q
======================== 41 passed, 1 warning in 9.88s
=========================
```
The live retry check used the real operator `execute()` path and
`TaskStateStoreAccessor`, with local Azure and supervisor stand-ins because
Azure credentials were unavailable.
```console
uv run --project providers/microsoft/azure python dev/adf_resumable_e2e.py
```
<details>
<summary>Raw retry logs</summary>
```text
upstream/main
[2026-09-01T23:08:24.648622Z] {data_factory.py:173} INFO - Executing the
pipeline pipeline.
[2026-09-01T23:08:24.648672Z] {data_factory.py:192} INFO - Waiting for
pipeline run run-1 to terminate.
attempt_1=crashed
stored_run_id=None
[2026-09-01T23:08:24.648852Z] {data_factory.py:173} INFO - Executing the
pipeline pipeline.
[2026-09-01T23:08:24.648871Z] {data_factory.py:192} INFO - Waiting for
pipeline run run-2 to terminate.
[2026-09-01T23:08:24.648886Z] {data_factory.py:202} INFO - Pipeline run
run-2 has completed successfully.
attempt_2=success
recovered_run_id=run-2
stored_run_id=None
submissions=2
resumable change
[2026-09-01T23:08:24.649111Z] {data_factory.py:216} INFO - Executing the
pipeline pipeline.
[2026-09-01T23:08:24.650514Z] {data_factory.py:292} INFO - Waiting for
pipeline run run-1 to terminate.
attempt_1=crashed
stored_run_id=run-1
[2026-09-01T23:08:24.650856Z] {data_factory.py:216} INFO - Executing the
pipeline pipeline.
[2026-09-01T23:08:24.650931Z] {resumablejobmixin.py:161} INFO - Reconnecting
to existing job external_id_key=azure_data_factory_run_id external_id=run-1
status=InProgress
[2026-09-01T23:08:24.650988Z] {data_factory.py:292} INFO - Waiting for
pipeline run run-1 to terminate.
[2026-09-01T23:08:24.651007Z] {data_factory.py:301} INFO - Pipeline run
run-1 has completed successfully.
attempt_2=success
recovered_run_id=run-1
stored_run_id=run-1
submissions=1
```
</details>
Targeted pre-commit hooks passed. The manual `mypy-providers` check could
not start because Docker was unavailable.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes - GitHub Copilot CLI (GPT-5.6 Sol)
Generated-by: GitHub Copilot CLI (GPT-5.6 Sol) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
