dabla commented on code in PR #77162:
URL: https://github.com/apache/airflow/pull/77162#discussion_r4589934548
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/data_factory.py:
##########
@@ -115,8 +120,13 @@ class AzureDataFactoryRunPipelineOperator(BaseOperator):
:param check_interval: Time in seconds to check on a pipeline run's status
for non-asynchronous waits.
Used only if ``wait_for_termination`` is True.
:param deferrable: Run operator in deferrable mode.
+ :param durable: Keep the pipeline run id in the task state store, so that
a retry reconnects to the
Review Comment:
**[warning]** The new default changes what a retry and a Dag run clear do,
and the provider changelog does not say so.
On Airflow 3.3+ every existing `AzureDataFactoryRunPipelineOperator` becomes
durable with this release. The part users will notice is the one the docs page
describes: on 3.3 and 3.4, clearing a whole Dag run keeps the stored run id, so
a task whose pipeline run succeeded returns without running the pipeline again.
For someone who clears a Dag run to re-run the pipeline, that is a silent no-op.
The Power BI change got a `.. note::` under the Changelog header for the
same default
([15.3.0](https://github.com/apache/airflow/blob/e6146e2bac63569c61491b429e831c58451dc560/providers/microsoft/azure/docs/changelog.rst#L57-L65)).
Could this one get the same, naming the Dag run clear case and `durable=False`?
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/data_factory.py:
##########
@@ -181,11 +199,28 @@ def execute(self, context: Context) -> None:
start_from_failure=self.start_from_failure,
parameters=self.parameters,
)
- self.run_id = response.run_id
- # Push the ``run_id`` value to XCom regardless of what happens during
execution. This allows for
- # retrieval the executed pipeline's ``run_id`` for downstream tasks
especially if performing an
- # asynchronous wait.
- context["ti"].xcom_push(key="run_id", value=self.run_id)
+ self._record_run_id(response.run_id, context)
+ return response.run_id
+
+ def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+ try:
+ return self.hook.get_pipeline_run_status(
+ str(external_id), self.resource_group_name, self.factory_name
+ )
+ except ResourceNotFoundError:
+ return "NotFound"
+
+ def is_job_active(self, status: str) -> bool:
+ return self.wait_for_termination and status in
AzureDataFactoryPipelineRunStatus.INTERMEDIATE_STATES
Review Comment:
**[warning]** A stored run in `Canceling` is reconnected to, so the retry
waits for a run that can only end `Cancelled`, and fails.
`INTERMEDIATE_STATES` includes `Canceling`. The retry then waits on it,
`wait_for_pipeline_run_status` returns `False` and the try raises "has failed
or has been cancelled". Only the try after that sees `Cancelled` and starts a
new run, so one cancelled run costs two tries, and with `retries=1` the task
fails without the pipeline being started again.
This is reachable in deferrable mode without the user doing anything: the
trigger cancels the pipeline run on any exception while polling
([triggers/data_factory.py#L246-L258](https://github.com/apache/airflow/blob/e6146e2bac63569c61491b429e831c58451dc560/providers/microsoft/azure/src/airflow/providers/microsoft/azure/triggers/data_factory.py#L246-L258)).
A transient polling error therefore still ends the run, and the retry has
nothing to reconnect to. The Power BI operator turned that off for the durable
case with `cancel_on_error=not self.durable`
([powerbi.py#L230](https://github.com/apache/airflow/blob/e6146e2bac63569c61491b429e831c58451dc560/providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/powerbi.py#L230));
`AzureDataFactoryTrigger` has no such switch yet.
Two things would close it:
- treat `Canceling` like `Cancelled` on the retry (start a new run), or wait
for it to settle and then start one, instead of reconnecting;
- let the trigger keep the run alive on a polling error when the operator is
durable, as Power BI does.
If the second is out of scope here, the docs section should say that a
trigger error in deferrable mode still cancels the run.
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/data_factory.py:
##########
@@ -163,13 +177,17 @@ def __init__(
self.timeout = timeout
self.check_interval = check_interval
self.deferrable = deferrable
+ self.run_id: str | None = None
Review Comment:
**[blocker]** `self.run_id: str | None` makes "MyPy providers checks" fail
with three errors.
`poll_until_complete` passes `self.run_id` to `wait_for_pipeline_run_status`
(line 230), `get_pipeline_run_status` (line 245) and
`AzureDataFactoryTrigger(run_id=...)` (line 252), which all take `str`. mypy
cannot narrow the attribute through `_record_run_id`.
A local in `poll_until_complete` settles all three and keeps the attribute
for `on_kill`:
```python
run_id = str(external_id)
self._record_run_id(run_id, context)
```
then `run_id` instead of `self.run_id` in the rest of the method.
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/data_factory.py:
##########
@@ -181,11 +199,28 @@ def execute(self, context: Context) -> None:
start_from_failure=self.start_from_failure,
parameters=self.parameters,
)
- self.run_id = response.run_id
- # Push the ``run_id`` value to XCom regardless of what happens during
execution. This allows for
- # retrieval the executed pipeline's ``run_id`` for downstream tasks
especially if performing an
- # asynchronous wait.
- context["ti"].xcom_push(key="run_id", value=self.run_id)
+ self._record_run_id(response.run_id, context)
Review Comment:
**[nit]** The XCom push sits between `run_pipeline` and the write to the
task state store.
`execute_resumable` stores the id only after `submit_job` returns, and its
docstring names that gap as the window in which a retry still duplicates the
run. `_record_run_id` here adds a supervisor round trip to the window: if the
push fails, the run is started and never stored.
`poll_until_complete` calls `_record_run_id` right after the store write on
every path (fresh, reconnect, `durable=False`), so `submit_job` only needs
`self.run_id = response.run_id` for `on_kill`, with the push left to the later
call.
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/data_factory.py:
##########
@@ -181,11 +199,28 @@ def execute(self, context: Context) -> None:
start_from_failure=self.start_from_failure,
parameters=self.parameters,
)
- self.run_id = response.run_id
- # Push the ``run_id`` value to XCom regardless of what happens during
execution. This allows for
- # retrieval the executed pipeline's ``run_id`` for downstream tasks
especially if performing an
- # asynchronous wait.
- context["ti"].xcom_push(key="run_id", value=self.run_id)
+ self._record_run_id(response.run_id, context)
+ return response.run_id
+
+ def get_job_status(self, external_id: JsonValue, context: Context) -> str:
+ try:
+ return self.hook.get_pipeline_run_status(
+ str(external_id), self.resource_group_name, self.factory_name
+ )
+ except ResourceNotFoundError:
+ return "NotFound"
+
+ def is_job_active(self, status: str) -> bool:
+ return self.wait_for_termination and status in
AzureDataFactoryPipelineRunStatus.INTERMEDIATE_STATES
+
+ def is_job_succeeded(self, status: str) -> bool:
+ # Without wait_for_termination a retry does not wait for a run in
progress either.
+ return status == AzureDataFactoryPipelineRunStatus.SUCCEEDED or (
+ not self.wait_for_termination and status in
AzureDataFactoryPipelineRunStatus.INTERMEDIATE_STATES
Review Comment:
**[nit]** `is_job_succeeded` answers `True` for a run in progress when
`wait_for_termination=False`.
The mixin then logs "Job already completed successfully, skipping
resubmission" and counts `resumable_job.already_succeeded` for a run that is
`Queued` or `InProgress`.
`poll_until_complete` already returns without waiting when
`wait_for_termination` is `False`, so both predicates can say what the status
is:
```python
def is_job_active(self, status: str) -> bool:
return status in AzureDataFactoryPipelineRunStatus.INTERMEDIATE_STATES
def is_job_succeeded(self, status: str) -> bool:
return status == AzureDataFactoryPipelineRunStatus.SUCCEEDED
```
The retry takes the reconnect branch, pushes the XCom and returns, with the
right log line and metric, and it also emits the `deferrable` has no effect
warning a first try gives.
`test_execute_without_waiting_returns_for_run_in_progress` passes unchanged.
##########
providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_data_factory.py:
##########
@@ -496,3 +501,140 @@ def
test_azure_data_factory_run_pipeline_operator_async_execute_complete_fail(se
context={},
event={"status": "error", "message": "error", "run_id":
AZ_PIPELINE_RUN_ID},
)
+
+
[email protected](not AIRFLOW_V_3_3_PLUS, reason="The task state store
arrived in Airflow 3.3")
+class TestAzureDataFactoryRunPipelineOperatorDurable:
+ CONFIG = {
+ "task_id": TASK_ID,
+ "azure_data_factory_conn_id": AZURE_DATA_FACTORY_CONN_ID,
+ "pipeline_name": PIPELINE_NAME,
+ "resource_group_name": "resource-group-name",
+ "factory_name": "factory-name",
+ "check_interval": 1,
+ "timeout": 3,
+ }
+
+ @pytest.fixture(autouse=True)
+ def setup_mocks(self):
+ with (
+ patch.object(
+ AzureDataFactoryHook,
+ "run_pipeline",
+ autospec=True,
+ return_value=MagicMock(run_id=NEW_RUN_ID),
Review Comment:
**[nit]** Two mocks without a spec: `MagicMock(run_id=NEW_RUN_ID)` here and
`{"ti": MagicMock()}` in `make_context` (line 536).
`AGENTS.md` asks for `spec`/`autospec`. `spec=CreateRunResponse` for the
response and a `spec` of the task instance for `ti` would make a typo in
`xcom_push` fail the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]