[ https://issues.apache.org/jira/browse/AIRFLOW-113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283273#comment-15283273 ]
Chris Riccomini commented on AIRFLOW-113: ----------------------------------------- Also, I wonder if perhaps you want max_active_runs instead of concurrency? >From [FAQ|https://pythonhosted.org/airflow/faq.html]: {quote} concurency defines how many running task instances a DAG is allowed to have, beyond which point things get queued {quote} Whereas: {quote} Is the max_active_runs parameter of your DAG reached? max_active_runs defines how many running concurrent instances of a DAG there are allowed to be. {quote} > DAG concurrency is not honored > ------------------------------ > > Key: AIRFLOW-113 > URL: https://issues.apache.org/jira/browse/AIRFLOW-113 > Project: Apache Airflow > Issue Type: Improvement > Components: celery, executor, scheduler > Affects Versions: Airflow 1.6.2 > Environment: Version of Airflow: 1.6.2 > Airflow configuration: Running a Scheduler with LocalExecutor or > CeleryExecutor > Operating System: 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC > 2015 x86_64 x86_64 x86_64 GNU/Linux > Python Version: 2.7.6 > Reporter: Shenghu Yang > > This is reported in airflow github also: > https://github.com/apache/incubator-airflow/issues/1424 > In our dag, we set the dag_args['concurrency'] = 8, however, when the > scheduler starts to run, we can see this concurrency is not being honored, > airflow scheduler will run up to num of the 'parallelism' (we set as 25) jobs. > What did you expect to happen? > dag_args['concurrency'] = 8 is honored, e.g. only run at most 8 jobs > concurrently. > What happened instead? > when the dag starts to run, we can see the concurrency is not being honored, > airflow scheduler will run up to the 'parallelism' (we set as 25) jobs. > Here is how you can reproduce this issue on your machine: > create a dag which contains nothing but 25 parallelized jobs. > set the dag dag_args['concurrency'] = 8 > set the airflow parallelism to 25 > then run: airflow scheduler > you will see all 25 jobs are scheduled to run, not 8. -- This message was sent by Atlassian JIRA (v6.3.4#6332)