[jira] [Commented] (AIRFLOW-1142) SubDAG Tasks Not Executed Even Though All Dependencies Met
[ https://issues.apache.org/jira/browse/AIRFLOW-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003137#comment-16003137 ] ASF subversion and git services commented on AIRFLOW-1142: -- Commit 5800f565628d11d8ea504468bcc14c4d1c0da10c in incubator-airflow's branch refs/heads/v1-8-stable from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=5800f56 ] [AIRFLOW-1142] Do not reset orphaned state for backfills The scheduler could interfere with backfills when it resets the state of tasks that were considered orphaned. This patch prevents the scheduler from doing so and adds a guard in the backfill. Closes #2260 from bolkedebruin/AIRFLOW-1142 (cherry picked from commit 4e79b830e3261b9d54fdbc7c9dcb510d36565986) Signed-off-by: Bolke de Bruin > SubDAG Tasks Not Executed Even Though All Dependencies Met > -- > > Key: AIRFLOW-1142 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1142 > Project: Apache Airflow > Issue Type: Bug > Components: subdag >Affects Versions: 1.8.1 > Environment: 1.8.1rc1+incubating, Celery >Reporter: Joe Schmid >Priority: Blocker > Fix For: 1.8.1 > > Attachments: 2017-04-24T23-20-38-776547, run3-scheduler-stdout.log, > run3-task.log, SubDAGOperatorTaskLog-DEBUG.txt, Test_Nested_SubDAG_0.png, > Test_Nested_SubDAG_1-Zoomed.png, test_nested_subdag.py > > > Testing on 1.8.1rc1, we noticed that tasks in subdags were not getting > executed even though all dependencies had been met. > We were able to create a simple test DAG that re-creates the issue. Attached > is a test DAG, the log file of the subdag operator that shows it fails to run > even though dependencies are met, and screenshots of what the UI looks like. > This is definitely a regression as we have many similarly constructed DAGs > that have been running successfully on a pre-v1.8 version (a fork of > 1.7.1.3+master) for some time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1142) SubDAG Tasks Not Executed Even Though All Dependencies Met
[ https://issues.apache.org/jira/browse/AIRFLOW-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003138#comment-16003138 ] ASF subversion and git services commented on AIRFLOW-1142: -- Commit 5800f565628d11d8ea504468bcc14c4d1c0da10c in incubator-airflow's branch refs/heads/v1-8-stable from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=5800f56 ] [AIRFLOW-1142] Do not reset orphaned state for backfills The scheduler could interfere with backfills when it resets the state of tasks that were considered orphaned. This patch prevents the scheduler from doing so and adds a guard in the backfill. Closes #2260 from bolkedebruin/AIRFLOW-1142 (cherry picked from commit 4e79b830e3261b9d54fdbc7c9dcb510d36565986) Signed-off-by: Bolke de Bruin > SubDAG Tasks Not Executed Even Though All Dependencies Met > -- > > Key: AIRFLOW-1142 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1142 > Project: Apache Airflow > Issue Type: Bug > Components: subdag >Affects Versions: 1.8.1 > Environment: 1.8.1rc1+incubating, Celery >Reporter: Joe Schmid >Priority: Blocker > Fix For: 1.8.1 > > Attachments: 2017-04-24T23-20-38-776547, run3-scheduler-stdout.log, > run3-task.log, SubDAGOperatorTaskLog-DEBUG.txt, Test_Nested_SubDAG_0.png, > Test_Nested_SubDAG_1-Zoomed.png, test_nested_subdag.py > > > Testing on 1.8.1rc1, we noticed that tasks in subdags were not getting > executed even though all dependencies had been met. > We were able to create a simple test DAG that re-creates the issue. Attached > is a test DAG, the log file of the subdag operator that shows it fails to run > even though dependencies are met, and screenshots of what the UI looks like. > This is definitely a regression as we have many similarly constructed DAGs > that have been running successfully on a pre-v1.8 version (a fork of > 1.7.1.3+master) for some time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1142) SubDAG Tasks Not Executed Even Though All Dependencies Met
[ https://issues.apache.org/jira/browse/AIRFLOW-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987334#comment-15987334 ] ASF subversion and git services commented on AIRFLOW-1142: -- Commit 4e79b830e3261b9d54fdbc7c9dcb510d36565986 in incubator-airflow's branch refs/heads/master from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=4e79b83 ] [AIRFLOW-1142] Do not reset orphaned state for backfills The scheduler could interfere with backfills when it resets the state of tasks that were considered orphaned. This patch prevents the scheduler from doing so and adds a guard in the backfill. Closes #2260 from bolkedebruin/AIRFLOW-1142 > SubDAG Tasks Not Executed Even Though All Dependencies Met > -- > > Key: AIRFLOW-1142 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1142 > Project: Apache Airflow > Issue Type: Bug > Components: subdag >Affects Versions: 1.8.1 > Environment: 1.8.1rc1+incubating, Celery >Reporter: Joe Schmid >Priority: Blocker > Fix For: 1.8.1 > > Attachments: 2017-04-24T23-20-38-776547, run3-scheduler-stdout.log, > run3-task.log, SubDAGOperatorTaskLog-DEBUG.txt, Test_Nested_SubDAG_0.png, > Test_Nested_SubDAG_1-Zoomed.png, test_nested_subdag.py > > > Testing on 1.8.1rc1, we noticed that tasks in subdags were not getting > executed even though all dependencies had been met. > We were able to create a simple test DAG that re-creates the issue. Attached > is a test DAG, the log file of the subdag operator that shows it fails to run > even though dependencies are met, and screenshots of what the UI looks like. > This is definitely a regression as we have many similarly constructed DAGs > that have been running successfully on a pre-v1.8 version (a fork of > 1.7.1.3+master) for some time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1142) SubDAG Tasks Not Executed Even Though All Dependencies Met
[ https://issues.apache.org/jira/browse/AIRFLOW-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987337#comment-15987337 ] ASF subversion and git services commented on AIRFLOW-1142: -- Commit 5800f565628d11d8ea504468bcc14c4d1c0da10c in incubator-airflow's branch refs/heads/v1-8-test from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=5800f56 ] [AIRFLOW-1142] Do not reset orphaned state for backfills The scheduler could interfere with backfills when it resets the state of tasks that were considered orphaned. This patch prevents the scheduler from doing so and adds a guard in the backfill. Closes #2260 from bolkedebruin/AIRFLOW-1142 (cherry picked from commit 4e79b830e3261b9d54fdbc7c9dcb510d36565986) Signed-off-by: Bolke de Bruin > SubDAG Tasks Not Executed Even Though All Dependencies Met > -- > > Key: AIRFLOW-1142 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1142 > Project: Apache Airflow > Issue Type: Bug > Components: subdag >Affects Versions: 1.8.1 > Environment: 1.8.1rc1+incubating, Celery >Reporter: Joe Schmid >Priority: Blocker > Fix For: 1.8.1 > > Attachments: 2017-04-24T23-20-38-776547, run3-scheduler-stdout.log, > run3-task.log, SubDAGOperatorTaskLog-DEBUG.txt, Test_Nested_SubDAG_0.png, > Test_Nested_SubDAG_1-Zoomed.png, test_nested_subdag.py > > > Testing on 1.8.1rc1, we noticed that tasks in subdags were not getting > executed even though all dependencies had been met. > We were able to create a simple test DAG that re-creates the issue. Attached > is a test DAG, the log file of the subdag operator that shows it fails to run > even though dependencies are met, and screenshots of what the UI looks like. > This is definitely a regression as we have many similarly constructed DAGs > that have been running successfully on a pre-v1.8 version (a fork of > 1.7.1.3+master) for some time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1142) SubDAG Tasks Not Executed Even Though All Dependencies Met
[ https://issues.apache.org/jira/browse/AIRFLOW-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987335#comment-15987335 ] ASF subversion and git services commented on AIRFLOW-1142: -- Commit 4e79b830e3261b9d54fdbc7c9dcb510d36565986 in incubator-airflow's branch refs/heads/master from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=4e79b83 ] [AIRFLOW-1142] Do not reset orphaned state for backfills The scheduler could interfere with backfills when it resets the state of tasks that were considered orphaned. This patch prevents the scheduler from doing so and adds a guard in the backfill. Closes #2260 from bolkedebruin/AIRFLOW-1142 > SubDAG Tasks Not Executed Even Though All Dependencies Met > -- > > Key: AIRFLOW-1142 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1142 > Project: Apache Airflow > Issue Type: Bug > Components: subdag >Affects Versions: 1.8.1 > Environment: 1.8.1rc1+incubating, Celery >Reporter: Joe Schmid >Priority: Blocker > Fix For: 1.8.1 > > Attachments: 2017-04-24T23-20-38-776547, run3-scheduler-stdout.log, > run3-task.log, SubDAGOperatorTaskLog-DEBUG.txt, Test_Nested_SubDAG_0.png, > Test_Nested_SubDAG_1-Zoomed.png, test_nested_subdag.py > > > Testing on 1.8.1rc1, we noticed that tasks in subdags were not getting > executed even though all dependencies had been met. > We were able to create a simple test DAG that re-creates the issue. Attached > is a test DAG, the log file of the subdag operator that shows it fails to run > even though dependencies are met, and screenshots of what the UI looks like. > This is definitely a regression as we have many similarly constructed DAGs > that have been running successfully on a pre-v1.8 version (a fork of > 1.7.1.3+master) for some time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1142) SubDAG Tasks Not Executed Even Though All Dependencies Met
[ https://issues.apache.org/jira/browse/AIRFLOW-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987336#comment-15987336 ] ASF subversion and git services commented on AIRFLOW-1142: -- Commit 5800f565628d11d8ea504468bcc14c4d1c0da10c in incubator-airflow's branch refs/heads/v1-8-test from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=5800f56 ] [AIRFLOW-1142] Do not reset orphaned state for backfills The scheduler could interfere with backfills when it resets the state of tasks that were considered orphaned. This patch prevents the scheduler from doing so and adds a guard in the backfill. Closes #2260 from bolkedebruin/AIRFLOW-1142 (cherry picked from commit 4e79b830e3261b9d54fdbc7c9dcb510d36565986) Signed-off-by: Bolke de Bruin > SubDAG Tasks Not Executed Even Though All Dependencies Met > -- > > Key: AIRFLOW-1142 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1142 > Project: Apache Airflow > Issue Type: Bug > Components: subdag >Affects Versions: 1.8.1 > Environment: 1.8.1rc1+incubating, Celery >Reporter: Joe Schmid >Priority: Blocker > Fix For: 1.8.1 > > Attachments: 2017-04-24T23-20-38-776547, run3-scheduler-stdout.log, > run3-task.log, SubDAGOperatorTaskLog-DEBUG.txt, Test_Nested_SubDAG_0.png, > Test_Nested_SubDAG_1-Zoomed.png, test_nested_subdag.py > > > Testing on 1.8.1rc1, we noticed that tasks in subdags were not getting > executed even though all dependencies had been met. > We were able to create a simple test DAG that re-creates the issue. Attached > is a test DAG, the log file of the subdag operator that shows it fails to run > even though dependencies are met, and screenshots of what the UI looks like. > This is definitely a regression as we have many similarly constructed DAGs > that have been running successfully on a pre-v1.8 version (a fork of > 1.7.1.3+master) for some time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1142) SubDAG Tasks Not Executed Even Though All Dependencies Met
[ https://issues.apache.org/jira/browse/AIRFLOW-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985991#comment-15985991 ] Joe Schmid commented on AIRFLOW-1142: - I'm still seeing the issue even when running with v1-8-test + the PR for AIRFLOW-1142: https://github.com/apache/incubator-airflow/pull/2260. In fact, when running with --run-duration 30 I see the issue consistently. I'm attaching two more logs: run3-task.log and run3-scheduler-stdout.log > SubDAG Tasks Not Executed Even Though All Dependencies Met > -- > > Key: AIRFLOW-1142 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1142 > Project: Apache Airflow > Issue Type: Bug > Components: subdag >Affects Versions: 1.8.1 > Environment: 1.8.1rc1+incubating, Celery >Reporter: Joe Schmid >Priority: Blocker > Attachments: 2017-04-24T23-20-38-776547, run3-scheduler-stdout.log, > run3-task.log, SubDAGOperatorTaskLog-DEBUG.txt, Test_Nested_SubDAG_0.png, > Test_Nested_SubDAG_1-Zoomed.png, test_nested_subdag.py > > > Testing on 1.8.1rc1, we noticed that tasks in subdags were not getting > executed even though all dependencies had been met. > We were able to create a simple test DAG that re-creates the issue. Attached > is a test DAG, the log file of the subdag operator that shows it fails to run > even though dependencies are met, and screenshots of what the UI looks like. > This is definitely a regression as we have many similarly constructed DAGs > that have been running successfully on a pre-v1.8 version (a fork of > 1.7.1.3+master) for some time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1142) SubDAG Tasks Not Executed Even Though All Dependencies Met
[
https://issues.apache.org/jira/browse/AIRFLOW-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982876#comment-15982876
]
Joe Schmid commented on AIRFLOW-1142:
-
Bolke, thanks for checking this out. Answers to your questions:
* The two successful runs shown in the UI had a different number of tasks (2)
in the Level2 subdag. The failure happened when I added a third task to the
Level2 subdag. However, the issue does seem to be intermittent and you might
have to trigger that test DAG a few times to observe the issue.
* I can definitely rerun with debug logging on and capture logs
The item that sticks out from the existing log is that dependencies are all met
for the task that never runs and it just logs that over and over:
Dependencies all met for
{models.py:4061} INFO - Updating state for considering 3
task(s)
{jobs.py:1994} INFO - [backfill progress] | finished run 0 of 1 | tasks
waiting: 1 | succeeded: 2 | kicked_off: 0 | failed: 0 | skipped: 0 |
deadlocked: 0 | not ready: 0
{models.py:1126} INFO - Dependencies all met for
{models.py:4061} INFO - Updating state for considering 3
task(s)
{jobs.py:1994} INFO - [backfill progress] | finished run 0 of 1 | tasks
waiting: 1 | succeeded: 2 | kicked_off: 0 | failed: 0 | skipped: 0 |
deadlocked: 0 | not ready: 0
> SubDAG Tasks Not Executed Even Though All Dependencies Met
> --
>
> Key: AIRFLOW-1142
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1142
> Project: Apache Airflow
> Issue Type: Bug
> Components: subdag
>Affects Versions: 1.8.1
> Environment: 1.8.1rc1+incubating, Celery
>Reporter: Joe Schmid
>Priority: Blocker
> Attachments: 2017-04-24T23-20-38-776547, Test_Nested_SubDAG_0.png,
> Test_Nested_SubDAG_1-Zoomed.png, test_nested_subdag.py
>
>
> Testing on 1.8.1rc1, we noticed that tasks in subdags were not getting
> executed even though all dependencies had been met.
> We were able to create a simple test DAG that re-creates the issue. Attached
> is a test DAG, the log file of the subdag operator that shows it fails to run
> even though dependencies are met, and screenshots of what the UI looks like.
> This is definitely a regression as we have many similarly constructed DAGs
> that have been running successfully on a pre-v1.8 version (a fork of
> 1.7.1.3+master) for some time.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1142) SubDAG Tasks Not Executed Even Though All Dependencies Met
[ https://issues.apache.org/jira/browse/AIRFLOW-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982596#comment-15982596 ] Bolke de Bruin commented on AIRFLOW-1142: - - You UI seems to indicate two successful runs. Can you explain why the other two runs seem to succeed? - Can you rerun with debug logging enabled? I have too much missing information At the moment I cannot confirm this with LocalExecutor and by triggering a single dag. > SubDAG Tasks Not Executed Even Though All Dependencies Met > -- > > Key: AIRFLOW-1142 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1142 > Project: Apache Airflow > Issue Type: Bug > Components: subdag >Affects Versions: 1.8.1 > Environment: 1.8.1rc1+incubating, Celery >Reporter: Joe Schmid >Priority: Blocker > Attachments: 2017-04-24T23-20-38-776547, Test_Nested_SubDAG_0.png, > Test_Nested_SubDAG_1-Zoomed.png, test_nested_subdag.py > > > Testing on 1.8.1rc1, we noticed that tasks in subdags were not getting > executed even though all dependencies had been met. > We were able to create a simple test DAG that re-creates the issue. Attached > is a test DAG, the log file of the subdag operator that shows it fails to run > even though dependencies are met, and screenshots of what the UI looks like. > This is definitely a regression as we have many similarly constructed DAGs > that have been running successfully on a pre-v1.8 version (a fork of > 1.7.1.3+master) for some time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1142) SubDAG Tasks Not Executed Even Though All Dependencies Met
[ https://issues.apache.org/jira/browse/AIRFLOW-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982370#comment-15982370 ] Joe Schmid commented on AIRFLOW-1142: - We didn't do extensive testing on 1.8, but I just ran that test DAG successfully 5 times in a row on 1.8 (installed via pip). Based on that, it seems like a bug introduced in 1.8.1. > SubDAG Tasks Not Executed Even Though All Dependencies Met > -- > > Key: AIRFLOW-1142 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1142 > Project: Apache Airflow > Issue Type: Bug > Components: subdag >Affects Versions: 1.8.1 > Environment: 1.8.1rc1+incubating, Celery >Reporter: Joe Schmid >Priority: Blocker > Attachments: 2017-04-24T23-20-38-776547, Test_Nested_SubDAG_0.png, > Test_Nested_SubDAG_1-Zoomed.png, test_nested_subdag.py > > > Testing on 1.8.1rc1, we noticed that tasks in subdags were not getting > executed even though all dependencies had been met. > We were able to create a simple test DAG that re-creates the issue. Attached > is a test DAG, the log file of the subdag operator that shows it fails to run > even though dependencies are met, and screenshots of what the UI looks like. > This is definitely a regression as we have many similarly constructed DAGs > that have been running successfully on a pre-v1.8 version (a fork of > 1.7.1.3+master) for some time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1142) SubDAG Tasks Not Executed Even Though All Dependencies Met
[ https://issues.apache.org/jira/browse/AIRFLOW-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982338#comment-15982338 ] Bolke de Bruin commented on AIRFLOW-1142: - Did it run on 1.8.0? > SubDAG Tasks Not Executed Even Though All Dependencies Met > -- > > Key: AIRFLOW-1142 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1142 > Project: Apache Airflow > Issue Type: Bug > Components: subdag >Affects Versions: 1.8.1 > Environment: 1.8.1rc1+incubating, Celery >Reporter: Joe Schmid >Priority: Blocker > Attachments: 2017-04-24T23-20-38-776547, Test_Nested_SubDAG_0.png, > Test_Nested_SubDAG_1-Zoomed.png, test_nested_subdag.py > > > Testing on 1.8.1rc1, we noticed that tasks in subdags were not getting > executed even though all dependencies had been met. > We were able to create a simple test DAG that re-creates the issue. Attached > is a test DAG, the log file of the subdag operator that shows it fails to run > even though dependencies are met, and screenshots of what the UI looks like. > This is definitely a regression as we have many similarly constructed DAGs > that have been running successfully on a pre-v1.8 version (a fork of > 1.7.1.3+master) for some time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
