[ https://issues.apache.org/jira/browse/AIRFLOW-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929083#comment-16929083 ]
Wesley edited comment on AIRFLOW-2843 at 9/13/19 10:08 AM: ----------------------------------------------------------- Using the operator stumbled upon the case where the `external_dag_id` and `external_task_id` exist but the DAG is disabled, so the ExternalTaskSensor task instance keep poking for a DagRun that does not exist. Perhaps the `check_existence` should check DagRun instead? or the name should be more specific like `check_dag_existence`/ `check_task_existence`? What you guys think about this use case? was (Author: wesleybatista): Using the operator stumbled upon the case where the `external_dag_id` and `external_task_id` exist but the DAG is disabled, so the ExternalTaskSensor task instance keep polling for a DagRun that does not exist. Perhaps the `check_existence` should check DagRun instead? or the name should be more specific like `check_dag_existence`/ `check_task_existence`? What you guys think about this use case? > ExternalTaskSensor: Add option to cease waiting immediately if the external > DAG/task doesn't exist > -------------------------------------------------------------------------------------------------- > > Key: AIRFLOW-2843 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2843 > Project: Apache Airflow > Issue Type: Improvement > Components: operators > Reporter: Xiaodong DENG > Assignee: Xiaodong DENG > Priority: Minor > Fix For: 1.10.3 > > > h2. Background > *ExternalTaskSensor* will keep waiting (given restrictions of retries, > poke_interval, etc), even if the external DAG/task specified doesn't exist at > all. In some cases, this waiting may still make sense as new DAG may backfill. > But it may be good to provide an option to cease waiting immediately if the > external DAG/task specified doesn't exist. > h2. Proposal > Provide an argument "check_existence". Set to *True* to check if the external > DAG/task exists, and immediately cease waiting if the external DAG/task does > not exist. > The default value is set to *False* (no check or ceasing will happen) so it > will not affect any existing DAGs or user expectation. -- This message was sent by Atlassian Jira (v8.3.2#803003)