Re: Complex Dag Runs Getting Stuck: Tasks Not Transitioning on Success

2017-01-13 Thread Arthur Purvis
sub dags don't really work well if you that's what you mean by "parent/child DAGs" On Thu, Jan 12, 2017 at 12:31 PM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Is there any error in the logs of the task instance (on the worker) towards > the end? Perhaps the airflow process is unabl

Re: Complex Dag Runs Getting Stuck: Tasks Not Transitioning on Success

2017-01-12 Thread Maxime Beauchemin
Is there any error in the logs of the task instance (on the worker) towards the end? Perhaps the airflow process is unable to communicate with the database in some cases? Maybe network or database being unresponsive? If that is the case there should be a stack trace in the log. In any case, the ta

Complex Dag Runs Getting Stuck: Tasks Not Transitioning on Success

2017-01-12 Thread Ken Edwards
I've successfully been using Airflow the past few months to run relatively simple dags on a regular cadence. However, I've run into stability issues with more complex workflows that consist of parent/child dags, some of which may contain hundreds of tasks. The most common symptom is that state tran