Chris, Got it. Thanks for your information. I will make the change and try it out!
-Jason On 5/31/16, 9:02 AM, "Chris Riccomini" <criccom...@apache.org> wrote: >Hey Jason, > >The problem is max_active_runs_per_dag=1. Set it back to 16. You just need >max_active_runs=1 for the individual DAGs. This will allow multiple >(different) DAGs to run in parallel, but only one DAG of each type can run >at the same type. > >Cheers, >Chris > >On Fri, May 27, 2016 at 11:42 PM, Jason Chen <chingchien.c...@gmail.com> >wrote: > >> Hi Chris, >> Thanks for your reply. After setting it up, I observed how it works for >> couple of days.. >> >> I tried to to set max_active_runs=1 in the DAG >> dag = DAG(...max_active_runs=1...) and it executed fine to avoid two runs >> at the same time. >> However, I noticed other dags (not the dag that is running) is also >> "paused". >> My understanding is that "max_active_runs" is basically >> "max_active_runs_per_dag". >> So, why another dag (different dag name) cannot run at the same time as the >> first dag? >> I want to have the two dags can be possibly run at the same time and inside >> each dag, there is only >> one run per dag. >> Thanks. >> >> Jason >> >> My other settings in airflow.cfg >> >> max_active_runs_per_dag=1 >> parallelism = 32 >> dag_concurrency = 16 >> >> >> >> On Mon, May 16, 2016 at 8:57 PM, Chris Riccomini <criccom...@apache.org> >> wrote: >> >> > Hey Jason, >> > >> > For (2), by default, task1 will start running again. You'll have two runs >> > going at the same time. If you want to prevent this, you can set >> > max_active_runs to 1 in your DAG. >> > >> > Cheers, >> > Chris >> > >> > On Mon, May 16, 2016 at 1:09 PM, Jason Chen <chingchien.c...@gmail.com> >> > wrote: >> > >> > > I have two questions >> > > >> > > (1) For the airflow UI: "Tree view", it lists the tasks along with the >> > time >> > > highlighted in the top (say, 08:30; 09:00, etc). What's the meaning of >> > > time? It looks not the UTC time of the task was running. I know in >> > > overall, airflow uses UTC time >> > > (2) I have a DAG with two tasks: task1 --> task2 >> > > Task1 is running hourly and could take longer than one hour to run, >> > > sometimes. >> > > In such a setup, task1 will be triggered hourly and what happens if the >> > > previous task1 is still running ? Will the "new" task1 be queued ? >> > > >> > > Thanks. >> > > Jason >> > > >> > >>
