[
https://issues.apache.org/jira/browse/AIRFLOW-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christophe updated AIRFLOW-3943:
--------------------------------
Attachment: table_A_sensor.log
> ExternalTaskSensor are randomly not being triggered (staying idle)
> ------------------------------------------------------------------
>
> Key: AIRFLOW-3943
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3943
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun, scheduler
> Affects Versions: 1.10.2
> Reporter: Christophe
> Priority: Major
> Attachments: dag_ingestion.png, dag_processing.png, table_A_sensor.log
>
>
> Hello, I don't know how to reproduce the error but with guidance I can go to
> log or somewhere else to search more information if needed.
> Since we upgraded from 1.10.1 to 1.10.2 it appeared that 3 days out of 10
> days — where everything was running fine — a subset of our ExternalTaskSensor
> stay in idle time and are not triggered / runned.
> To explain more, we have daily ingestion jobs that runs a 2' and afterwards
> we have processing jobs with sensors that waits for ingestion to be run.
> If the first Gantt (dag_ingestion.jpg) we see the ingestion part for one
> table saying `A`, this ingestion ends up around 3:23.
> In the second Gantt (dag_processing.jpg) for the processing we see that the
> Sensor is waiting for a long time without doing anything — passing green at
> 9:30 because we manually triggered the DAG with a clear.
> In the logs we see that the task is running once at 2:22 and the nothing is
> happening until the manual trigger at 8:32 (9:32 UTC for us).
> I don't know how to debug this and the issue is happening randomly on
> different sensors.
> When we where on 1.10.1 everything was running smoothly.
> PS : I can't uplaod screen on Jira, I don't know why I have some issues :(
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
