[
https://issues.apache.org/jira/browse/AIRFLOW-6769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Boden updated AIRFLOW-6769:
---------------------------------
Description:
After adding a second DAG to Airflow the scheduler spirals into an infinite
loop and is unable to schedule any tasks. The recursion appears to have to do
with logging. I've attached the output from scheduler and my requirements.txt
If I reset Airflow (`airflow resetdb`) and remove all but 1 DAG from the folder
there is no issue. I don't believe it's a specific DAG as I have tried with
various different test ones. I don't even have to schedule/run them, just
having Airflow recognize them causes the loop.
I'm using MySQL and Celery as my store and worker.
I have also attached part of the log file for one of my DAGs outputted by the
scheduler ($AIRFLOW_HOME/logs/scheduler/\{dag_name}/latest/\{dag_name}.log). It
seems `collect_dags` is being called every second by the scheduler.
was:
After adding a second DAG to Airflow the scheduler spirals into an infinite
loop and is unable to schedule any tasks. The recursion appears to have to do
with logging. I've attached the output from scheduler and my requirements.txt
If I reset Airflow (`airflow resetdb`) and remove all but 1 DAG from the folder
there is no issue. I don't believe it's a specific DAG as I have tried with
various different test ones. I don't even have to schedule/run them, just
having Airflow recognize them causes the loop.
I'm using MySQL and Celery as my store and worker.
> Multiple DAGs cause infinite loop in logging
> --------------------------------------------
>
> Key: AIRFLOW-6769
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6769
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 1.10.9
> Environment: macOS 10.15.2
> Python 3.7.5
> Reporter: Chris Boden
> Priority: Major
> Attachments: requirements.txt, scheduler-dag.log, scheduler.log
>
>
> After adding a second DAG to Airflow the scheduler spirals into an infinite
> loop and is unable to schedule any tasks. The recursion appears to have to do
> with logging. I've attached the output from scheduler and my requirements.txt
> If I reset Airflow (`airflow resetdb`) and remove all but 1 DAG from the
> folder there is no issue. I don't believe it's a specific DAG as I have tried
> with various different test ones. I don't even have to schedule/run them,
> just having Airflow recognize them causes the loop.
> I'm using MySQL and Celery as my store and worker.
> I have also attached part of the log file for one of my DAGs outputted by the
> scheduler ($AIRFLOW_HOME/logs/scheduler/\{dag_name}/latest/\{dag_name}.log).
> It seems `collect_dags` is being called every second by the scheduler.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
