amoghrajesh opened a new pull request, #77162:
URL: https://github.com/apache/airflow/pull/77162

    <!-- SPDX-License-Identifier: Apache-2.0
         https://www.apache.org/licenses/LICENSE-2.0 -->
   
   <!--
   Thank you for contributing!
   
   Please provide above a brief description of the changes made in this pull 
request.
   Write a good git commit message following this guide: 
https://chris.beams.io/posts/git-commit/
   
   Please make sure that your code changes are covered with tests.
   And in case of new features or big changes remember to adjust the 
documentation.
   
   For user-facing UI changes, please attach before/after screenshots (or a 
short
   screen recording) so reviewers can assess the visual impact.
   
   Feel free to ping (in general) for the review if you do not see reaction for 
a few days
   (72 Hours is the minimum reaction time you can expect from volunteers) - we 
sometimes miss notifications.
   
   In case of an existing issue, reference it using one of the following:
   
   * closes: #ISSUE
   * related: #ISSUE
   -->
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   <!--
   If generative AI tooling has been used in the process of authoring this PR, 
please
   change below checkbox to `[X]` followed by the name of the tool, uncomment 
the "Generated-by".
   -->
   
   - [ ] Yes (please specify the tool below)
   
   <!--
   Generated-by: [Tool Name] following [the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   -->
   
   
   ### What
   
   When an `AzureDataFactoryRunPipelineOperator` try fails while its pipeline 
run is still going, for example because the worker died or the wait timed out, 
the retry starts a second pipeline run. The first run keeps going in Azure, so 
the same pipeline runs twice and you pay for both.
   
   ### Current behaviour
   
   `execute` calls `run_pipeline` on every try. The run ID lives only in XCom, 
and Airflow clears XCom before a retry, so the next try can't find the previous 
run.
   
   ### Proposed change
   
   The operator now uses `ResumableJobMixin`, like the Power BI operator in 
this provider. It stores the pipeline run id in the task state store before 
waiting. On retry it checks that run first:
   
   | Stored run status | What the retry does |
   |---|---|
   | `Queued`, `InProgress`, `Canceling` | Waits on the same run (sync or 
deferrable) |
   | `Succeeded` | Returns without starting a new run |
   | `Failed`, `Cancelled`, or the run no longer exists | Starts a new run, as 
today |
   | Still going, with `wait_for_termination=False` | Returns without waiting, 
as a fresh try would |
   
   Also adds a "Durable execution" section to the Data Factory operator docs, 
covering the table above and how clearing interacts with the stored run ID.
   
   ### Changes of Note
   
   - The old submit and wait code moved into the mixin hooks (`submit_job`, 
`poll_until_complete`) unchanged.
   - `run_id` is still pushed to XCom once per try, including on reconnect and 
when the stored run already succeeded, so the operator extra link and 
downstream tasks keep working.
   - If Azure no longer knows the stored run (`ResourceNotFoundError`), the 
retry starts a new run instead of failing.
   - `durable` is a named `__init__` parameter forwarded only when set, as in 
the Power BI operator, so `default_args` reaches it.
   - The in-memory `TaskStateStore` test helper moved to 
`unit/microsoft/azure/test_utils.py`, shared by the Data Factory and Power BI 
tests.
   
   ### User implications / backcompat
   
   - On Airflow 3.3+, `durable` defaults to `True`. Pass `durable=False` to 
keep starting a new run on every try.
   - Below Airflow 3.3, nothing changes.
   - Clearing the task instance from the UI or REST API discards the stored run 
id, so the next try starts a new run. Ticking "keep task state" keeps it, and 
the next try acts like a retry.
   - Before 3.5, a Dag run clear, marking a task success or failed, and the CLI 
clear commands keep the stored run id. A task whose run already succeeded then 
returns without running the pipeline again. From 3.5, every clear discards it 
by default.
   - No common.compat bump: the provider already requires `>=1.23.0`, which 
ships the mixin shim.
   
   ### Testing
   
   **Setup:** Breeze `airflow standalone` with this branch's operator, against 
a real Data Factory. The pipeline `durable_demo` has one Wait activity of 150 
seconds. The demo Dags use `retries=1`. A demo subclass gives try 1 a 60 second 
timeout, so try 1 fails while the pipeline is still `InProgress`. Try 2 gets 
600 seconds.
   
   <details>
   <summary>Demo Dag</summary>
   
   ```python
   from datetime import timedelta
   
   from airflow.providers.microsoft.azure.operators.data_factory import 
AzureDataFactoryRunPipelineOperator
   from airflow.sdk import DAG
   
   
   class DemoAdfOperator(AzureDataFactoryRunPipelineOperator):
       # Try 1 gives up after 60s against a 150s Wait pipeline, so the retry 
finds the run still going.
       def execute(self, context):
           if context["ti"].try_number > 1:
               self.timeout = 600
           return super().execute(context)
   
   
   SCENARIOS = {
       "adf_before": {"durable": False, "deferrable": False, "retry_delay": 20},
       "adf_after_sync": {"durable": True, "deferrable": False, "retry_delay": 
20},
       "adf_after_deferrable": {"durable": True, "deferrable": True, 
"retry_delay": 20},
       "adf_after_succeeded": {"durable": True, "deferrable": False, 
"retry_delay": 150},
   }
   
   for dag_id, cfg in SCENARIOS.items():
       with DAG(
           dag_id,
           schedule=None,
           catchup=False,
           default_args={"retries": 1, "retry_delay": 
timedelta(seconds=cfg["retry_delay"])},
       ):
           DemoAdfOperator(
               task_id="run_pipeline",
               azure_data_factory_conn_id="azure_data_factory_default",
               pipeline_name="durable_demo",
               check_interval=10,
               timeout=60,
               durable=cfg["durable"],
               deferrable=cfg["deferrable"],
           )
   ```
   
   </details>
   
   **Steps:** Triggered all 4 Dags. In each, try 1 timed out with the pipeline 
still `InProgress`, then try 2 ran.
   
   **Results:**
   
   | Dag | Try 2 log | Pipeline runs in ADF | Dag run duration |
   |---|---|---|---|
   | `adf_before` (`durable=False`) | "Executing the durable_demo pipeline", 
new run id | 2 | 4m 13s |
   | `adf_after_sync` | "Reconnecting to existing job", waits on the same run 
id | 1 | 2m 41s |
   | `adf_after_deferrable` | "Reconnecting to existing job", trigger reports 
the same run "has Succeeded" | 1 | 2m 45s |
   | `adf_after_succeeded` | "Job already completed successfully, skipping 
resubmission" | 1 | 3m 40s |
   
   **Screenshots:**
   
   ADF Monitor, Pipeline runs: `adf_before` started 2 runs, each durable Dag 
started 1.
   
   <img width="1600" height="1000" alt="adf_monitor_pipeline_runs" 
src="https://github.com/user-attachments/assets/fc64b594-4352-4498-a5ef-e53ae8a094c4";
 />
   
   
   `adf_before`, try 1 and try 2 (new run on try 2):
   
   <img width="3200" height="2000" alt="airflow_adf_before_try1" 
src="https://github.com/user-attachments/assets/ba69627d-e2d3-4fbc-8a5e-4bfc89b073b2";
 />
   
   <img width="3200" height="2000" alt="airflow_adf_before_try2" 
src="https://github.com/user-attachments/assets/51ff266d-f511-43fd-a2b6-a69f8bee9979";
 />
   
   
   
   `adf_after_sync`, try 1 and try 2 (reconnects to the same run id):
   
   <img width="3200" height="2000" alt="airflow_adf_after_sync_try1" 
src="https://github.com/user-attachments/assets/0ff95a35-c233-40bc-99b9-ccf68a21c9cd";
 />
   
   <img width="3200" height="2000" alt="airflow_adf_after_sync_try2" 
src="https://github.com/user-attachments/assets/6940e31f-6b44-4032-9199-6c5f9817f867";
 />
   
   
   `adf_after_deferrable`, try 1 and try 2 (reconnects and defers):
   
   <img width="3200" height="2000" alt="airflow_adf_after_deferrable_try1" 
src="https://github.com/user-attachments/assets/220651ad-e0ce-4795-9a54-501ee869adaa";
 />
   
   <img width="3200" height="2000" alt="airflow_adf_after_deferrable_try2" 
src="https://github.com/user-attachments/assets/b2ec2eaa-2c5a-4bf0-9220-e22b9b34e562";
 />
   
   
   
   `adf_after_succeeded`, try 1 and try 2 (skips resubmission):
   
   <img width="3200" height="2000" alt="airflow_adf_after_succeeded_try1" 
src="https://github.com/user-attachments/assets/ea3d7d44-275e-4fda-8ac4-e0c73310eec4";
 />
   
   <img width="3200" height="2000" alt="airflow_adf_after_succeeded_try2" 
src="https://github.com/user-attachments/assets/1b5039d6-e2bc-46ae-8456-38d0c62509a5";
 />
   
   
   ---
   
   * Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information. Note: commit author/co-author name and email in commits 
become permanently public when merged.
   * For fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   * When adding dependency, check compliance with the [ASF 3rd Party License 
Policy](https://www.apache.org/legal/resolved.html#category-x).
   * For significant user-facing changes create newsfragment: 
`{pr_number}.significant.rst`, in 
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
 You can add this file in a follow-up commit after the PR is created so you 
know the PR number.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to