Hey Jason, The problem is max_active_runs_per_dag=1. Set it back to 16. You just need max_active_runs=1 for the individual DAGs. This will allow multiple (different) DAGs to run in parallel, but only one DAG of each type can run at the same type.
Cheers, Chris On Fri, May 27, 2016 at 11:42 PM, Jason Chen <chingchien.c...@gmail.com> wrote: > Hi Chris, > Thanks for your reply. After setting it up, I observed how it works for > couple of days.. > > I tried to to set max_active_runs=1 in the DAG > dag = DAG(...max_active_runs=1...) and it executed fine to avoid two runs > at the same time. > However, I noticed other dags (not the dag that is running) is also > "paused". > My understanding is that "max_active_runs" is basically > "max_active_runs_per_dag". > So, why another dag (different dag name) cannot run at the same time as the > first dag? > I want to have the two dags can be possibly run at the same time and inside > each dag, there is only > one run per dag. > Thanks. > > Jason > > My other settings in airflow.cfg > > max_active_runs_per_dag=1 > parallelism = 32 > dag_concurrency = 16 > > > > On Mon, May 16, 2016 at 8:57 PM, Chris Riccomini <criccom...@apache.org> > wrote: > > > Hey Jason, > > > > For (2), by default, task1 will start running again. You'll have two runs > > going at the same time. If you want to prevent this, you can set > > max_active_runs to 1 in your DAG. > > > > Cheers, > > Chris > > > > On Mon, May 16, 2016 at 1:09 PM, Jason Chen <chingchien.c...@gmail.com> > > wrote: > > > > > I have two questions > > > > > > (1) For the airflow UI: "Tree view", it lists the tasks along with the > > time > > > highlighted in the top (say, 08:30; 09:00, etc). What's the meaning of > > > time? It looks not the UTC time of the task was running. I know in > > > overall, airflow uses UTC time > > > (2) I have a DAG with two tasks: task1 --> task2 > > > Task1 is running hourly and could take longer than one hour to run, > > > sometimes. > > > In such a setup, task1 will be triggered hourly and what happens if the > > > previous task1 is still running ? Will the "new" task1 be queued ? > > > > > > Thanks. > > > Jason > > > > > >
