[
https://issues.apache.org/jira/browse/AIRFLOW-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Kim updated AIRFLOW-3001:
-------------------------------
Description:
I have created very long term schedule in short interval. (2~3 years as 10 min
interval)
So, dag could be bigger and bigger as scheduling goes on.
Finally, at critical point (I don't know exactly when it is), the allocation of
new task_instances get slow and then almost stop.
I found that in this point, many slow query logs had occurred. (I was using
mysql as meta repository)
queries like this
"SELECT * FROM task_instance WHERE dag_id = 'some_dag_id' AND execution_date =
''2018-09-01 00:00:00"
I could resolve this issue by adding new index consists of dag_id and
execution_date.
So, I wanted 1.10 branch to be modified to create task_instance table with the
index.
Thanks.
was:
I have created very long term schedule in short interval (2~3 years as 10 min
interval)
So, dag could be bigger and bigger as scheduling goes on.
Finally, at critical point (I don't know exactly when it is), the allocation of
new task_instances get slow and then almost stop.
I found that in this point, many slow query logs had occurred. (I was using
mysql as meta repository)
queries like this
"SELECT * FROM task_instance WHERE dag_id = 'some_dag_id' AND execution_date =
''2018-09-01 00:00:00"
I could resolve this issue by adding new index consists of dag_id and
execution_date
So, I wanted 1.10 branch to be modified to create task_instance table with the
index.
Thanks.
> Accumulative tis slow allocation of new schedule
> ------------------------------------------------
>
> Key: AIRFLOW-3001
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3001
> Project: Apache Airflow
> Issue Type: Improvement
> Components: scheduler
> Affects Versions: 1.10.0
> Reporter: Jason Kim
> Assignee: Jason Kim
> Priority: Major
>
> I have created very long term schedule in short interval. (2~3 years as 10
> min interval)
> So, dag could be bigger and bigger as scheduling goes on.
> Finally, at critical point (I don't know exactly when it is), the allocation
> of new task_instances get slow and then almost stop.
> I found that in this point, many slow query logs had occurred. (I was using
> mysql as meta repository)
> queries like this
> "SELECT * FROM task_instance WHERE dag_id = 'some_dag_id' AND execution_date
> = ''2018-09-01 00:00:00"
> I could resolve this issue by adding new index consists of dag_id and
> execution_date.
> So, I wanted 1.10 branch to be modified to create task_instance table with
> the index.
> Thanks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
