yuqian90 commented on issue #7098: [AIRFLOW-4453] Make behavior of 
`none_failed` consistent with documentation
URL: https://github.com/apache/airflow/pull/7098#issuecomment-582219010
 
 
   > @yuqian90 I agree with what @ChrisEverling said 
[here](https://github.com/apache/airflow/pull/4182#discussion_r362862918), and 
with the folks who created the issue, "none failed" includes "all skipped", and 
the documentation backs that up. It took some debugging to find this extra 
condition, and I still don't understand the rationale behind it.
   > 
   > Why not make the current trigger rule behave as documented, and add a new 
one to preserve the unexpected "none failed and not all skipped" behavior?
   
   @jared-martin Sure. I think continuing what @TV4Fun was originally doing in 
this PR is also fine. However, I was pointing out in my previous comments the 
side-effect of making this change when e.g. when join tasks with `none_failed` 
trigger_rule are nested in branching logic.
   
   If people think it's better to proceed with fixing the existing 
`none_failed` to match what the doc says, I would suggest we update the parts 
in the doc about using `none_failed` task to create join for 
`BranchPythonOperator`. It can no longer be a `none_failed` `DummyOperator`. It 
needs to be something like the following, where the join task checks the state 
of its parent `BranchPythonOperator` and skip itself if the parent is skipped. 
I would highly recommend making `create_join` a free function inside 
`airflow/operators/python.py` to make it more convenient for users of 
`BranchPythonOperator` and `ShortCircuitOperator`.
   
   ```python
   from airflow.models import DAG
   from airflow.operators.dummy_operator import DummyOperator
   from airflow.operators.python import BranchPythonOperator, PythonOperator
   from airflow.utils.dates import days_ago
   
   def create_join(branch_task_id, *args, **kwargs):
           """
           :param branch_task_id: The task_id of the parent 
BranchPythonOperator task that created the branching logic.
           :type branch_task_id: str
           """
           def python_callable(ti, **_):
               from airflow.utils.session import create_session
               from airflow.exceptions import AirflowSkipException
               from airflow.utils.state import State
               from airflow.models import TaskInstance
   
               with create_session() as session:
                   branch_ti = 
session.query(TaskInstance).filter(TaskInstance.dag_id == ti.dag_id,
                                                                  
TaskInstance.task_id == branch_task_id,
                                                                  
TaskInstance.execution_date == ti.execution_date
                                                                 ).first()
                   if not branch_ti:
                       return
   
                   if branch_ti.state == State.SKIPPED:
                       raise AirflowSkipException("Skipping {} because parent 
BranchPythonOperator {} is skipped."
                                                  .format(ti.task_id, 
branch_task_id))
   
           return PythonOperator(trigger_rule="none_failed", 
python_callable=python_callable, *args, **kwargs)
   
   
   with DAG(dag_id="example_nested_branch_dag", start_date=days_ago(2), 
schedule_interval="@daily") as dag:
       branch_1 = BranchPythonOperator(task_id="branch_1", 
python_callable=lambda: "false_1")
       true_1 = DummyOperator(task_id="true_1")
       false_1 = DummyOperator(task_id="false_1")
       branch_2 = BranchPythonOperator(task_id="branch_2", 
python_callable=lambda: "true_2")
       true_2 = DummyOperator(task_id="true_2")
       false_2 = DummyOperator(task_id="false_2")
       join_2 = create_join(task_id="join_2", branch_task_id=branch_2.task_id)
       true_3 = DummyOperator(task_id="true_3")
       join_1 = create_join(task_id="join_1", branch_task_id=branch_1.task_id)
   
       branch_1 >> [true_1, false_1]
       true_1 >> branch_2 >> [true_2, false_2] >> join_2 >> true_3 >> join_1
       false_1 >> join_1
   ```
   
   You can see the DAG now behaves the same as what it should have been before 
this PR:
   
   
![image](https://user-images.githubusercontent.com/6637585/73807257-c1675600-4806-11ea-803e-c79ca9276758.png)
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

Reply via email to