amoghrajesh opened a new pull request, #77162:
URL: https://github.com/apache/airflow/pull/77162
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
### What
When an `AzureDataFactoryRunPipelineOperator` try fails while its pipeline
run is still going, for example because the worker died or the wait timed out,
the retry starts a second pipeline run. The first run keeps going in Azure, so
the same pipeline runs twice and you pay for both.
### Current behaviour
`execute` calls `run_pipeline` on every try. The run ID lives only in XCom,
and Airflow clears XCom before a retry, so the next try can't find the previous
run.
### Proposed change
The operator now uses `ResumableJobMixin`, like the Power BI operator in
this provider. It stores the pipeline run id in the task state store before
waiting. On retry it checks that run first:
| Stored run status | What the retry does |
|---|---|
| `Queued`, `InProgress`, `Canceling` | Waits on the same run (sync or
deferrable) |
| `Succeeded` | Returns without starting a new run |
| `Failed`, `Cancelled`, or the run no longer exists | Starts a new run, as
today |
| Still going, with `wait_for_termination=False` | Returns without waiting,
as a fresh try would |
Also adds a "Durable execution" section to the Data Factory operator docs,
covering the table above and how clearing interacts with the stored run ID.
### Changes of Note
- The old submit and wait code moved into the mixin hooks (`submit_job`,
`poll_until_complete`) unchanged.
- `run_id` is still pushed to XCom once per try, including on reconnect and
when the stored run already succeeded, so the operator extra link and
downstream tasks keep working.
- If Azure no longer knows the stored run (`ResourceNotFoundError`), the
retry starts a new run instead of failing.
- `durable` is a named `__init__` parameter forwarded only when set, as in
the Power BI operator, so `default_args` reaches it.
- The in-memory `TaskStateStore` test helper moved to
`unit/microsoft/azure/test_utils.py`, shared by the Data Factory and Power BI
tests.
### User implications / backcompat
- On Airflow 3.3+, `durable` defaults to `True`. Pass `durable=False` to
keep starting a new run on every try.
- Below Airflow 3.3, nothing changes.
- Clearing the task instance from the UI or REST API discards the stored run
id, so the next try starts a new run. Ticking "keep task state" keeps it, and
the next try acts like a retry.
- Before 3.5, a Dag run clear, marking a task success or failed, and the CLI
clear commands keep the stored run id. A task whose run already succeeded then
returns without running the pipeline again. From 3.5, every clear discards it
by default.
- No common.compat bump: the provider already requires `>=1.23.0`, which
ships the mixin shim.
### Testing
**Setup:** Breeze `airflow standalone` with this branch's operator, against
a real Data Factory. The pipeline `durable_demo` has one Wait activity of 150
seconds. The demo Dags use `retries=1`. A demo subclass gives try 1 a 60 second
timeout, so try 1 fails while the pipeline is still `InProgress`. Try 2 gets
600 seconds.
<details>
<summary>Demo Dag</summary>
```python
from datetime import timedelta
from airflow.providers.microsoft.azure.operators.data_factory import
AzureDataFactoryRunPipelineOperator
from airflow.sdk import DAG
class DemoAdfOperator(AzureDataFactoryRunPipelineOperator):
# Try 1 gives up after 60s against a 150s Wait pipeline, so the retry
finds the run still going.
def execute(self, context):
if context["ti"].try_number > 1:
self.timeout = 600
return super().execute(context)
SCENARIOS = {
"adf_before": {"durable": False, "deferrable": False, "retry_delay": 20},
"adf_after_sync": {"durable": True, "deferrable": False, "retry_delay":
20},
"adf_after_deferrable": {"durable": True, "deferrable": True,
"retry_delay": 20},
"adf_after_succeeded": {"durable": True, "deferrable": False,
"retry_delay": 150},
}
for dag_id, cfg in SCENARIOS.items():
with DAG(
dag_id,
schedule=None,
catchup=False,
default_args={"retries": 1, "retry_delay":
timedelta(seconds=cfg["retry_delay"])},
):
DemoAdfOperator(
task_id="run_pipeline",
azure_data_factory_conn_id="azure_data_factory_default",
pipeline_name="durable_demo",
check_interval=10,
timeout=60,
durable=cfg["durable"],
deferrable=cfg["deferrable"],
)
```
</details>
**Steps:** Triggered all 4 Dags. In each, try 1 timed out with the pipeline
still `InProgress`, then try 2 ran.
**Results:**
| Dag | Try 2 log | Pipeline runs in ADF | Dag run duration |
|---|---|---|---|
| `adf_before` (`durable=False`) | "Executing the durable_demo pipeline",
new run id | 2 | 4m 13s |
| `adf_after_sync` | "Reconnecting to existing job", waits on the same run
id | 1 | 2m 41s |
| `adf_after_deferrable` | "Reconnecting to existing job", trigger reports
the same run "has Succeeded" | 1 | 2m 45s |
| `adf_after_succeeded` | "Job already completed successfully, skipping
resubmission" | 1 | 3m 40s |
**Screenshots:**
ADF Monitor, Pipeline runs: `adf_before` started 2 runs, each durable Dag
started 1.
<img width="1600" height="1000" alt="adf_monitor_pipeline_runs"
src="https://github.com/user-attachments/assets/fc64b594-4352-4498-a5ef-e53ae8a094c4"
/>
`adf_before`, try 1 and try 2 (new run on try 2):
<img width="3200" height="2000" alt="airflow_adf_before_try1"
src="https://github.com/user-attachments/assets/ba69627d-e2d3-4fbc-8a5e-4bfc89b073b2"
/>
<img width="3200" height="2000" alt="airflow_adf_before_try2"
src="https://github.com/user-attachments/assets/51ff266d-f511-43fd-a2b6-a69f8bee9979"
/>
`adf_after_sync`, try 1 and try 2 (reconnects to the same run id):
<img width="3200" height="2000" alt="airflow_adf_after_sync_try1"
src="https://github.com/user-attachments/assets/0ff95a35-c233-40bc-99b9-ccf68a21c9cd"
/>
<img width="3200" height="2000" alt="airflow_adf_after_sync_try2"
src="https://github.com/user-attachments/assets/6940e31f-6b44-4032-9199-6c5f9817f867"
/>
`adf_after_deferrable`, try 1 and try 2 (reconnects and defers):
<img width="3200" height="2000" alt="airflow_adf_after_deferrable_try1"
src="https://github.com/user-attachments/assets/220651ad-e0ce-4795-9a54-501ee869adaa"
/>
<img width="3200" height="2000" alt="airflow_adf_after_deferrable_try2"
src="https://github.com/user-attachments/assets/b2ec2eaa-2c5a-4bf0-9220-e22b9b34e562"
/>
`adf_after_succeeded`, try 1 and try 2 (skips resubmission):
<img width="3200" height="2000" alt="airflow_adf_after_succeeded_try1"
src="https://github.com/user-attachments/assets/ea3d7d44-275e-4fda-8ac4-e0c73310eec4"
/>
<img width="3200" height="2000" alt="airflow_adf_after_succeeded_try2"
src="https://github.com/user-attachments/assets/1b5039d6-e2bc-46ae-8456-38d0c62509a5"
/>
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]