Hi Chris,
I made the changes and tried it out.
It seems not working as expected.
When a dag is running (a particular task inside that dag is taking time),
another task from another dag seems "blocked".
My setting:
(1) airflow.cfg
max_active_runs_per_dag = 16
parallelism = 32
dag_concurrency = 16
(2) A dag (dag1) python file is as below partially. Please note that inside
this DAG, the first task (task1) is a long running task
dag1 = DAG('dag1', schedule_interval=timedelta(minutes=15),
max_active_runs=1, default_args=args)
Then, the tasks are running in the order...
task1 (long running) --> task 2 --> task3
...
(3) In another dag (dag2) python file is as below partially.
dag2 = DAG('dag2', schedule_interval=timedelta(minutes=3),
max_active_runs=1, default_args=args)
...
Then, the tasks are running in the order...
taskA (short running task) --> taskB
(4) Inside the upstart script file. this is the main part how I start
airflow scheduler
env SCHEDULER_RUNS=0
export SCHEDULER_RUNS
script
exec >> ${AIRFLOW_HOME}/scheduler-log/airflow-scheduler.log 2>&1
exec usr/local/bin/airflow scheduler -n ${SCHEDULER_RUNS}
end script
=========================
What I observed are that
(a) task1 (of dag1) is running about 20 mins and during it's running time,
there is no other dag1 triggered. This is as expected.
(b) taskA (of dag2) should be triggered to run every 3 mins. However, it is
NOT triggered if task-1 of dag-1 is running.
taskA seems to be queued/bolcked and not run. It is executed after task-1
(of dag-1) is done. So, it looks like it is dispatched into a "gap" of
task1 and task2 (of dag1). This looks not normal, as it's expected taskA
(of dag 2) should run no matter what happens to another dag (dag-1).
Any suggestions?
Thanks.
Jason
On Tue, May 31, 2016 at 9:02 AM, Chris Riccomini <criccom...@apache.org>
wrote:
> Hey Jason,
>
> The problem is max_active_runs_per_dag=1. Set it back to 16. You just need
> max_active_runs=1 for the individual DAGs. This will allow multiple
> (different) DAGs to run in parallel, but only one DAG of each type can run
> at the same type.
>
> Cheers,
> Chris
>
> On Fri, May 27, 2016 at 11:42 PM, Jason Chen <chingchien.c...@gmail.com>
> wrote:
>
> > Hi Chris,
> > Thanks for your reply. After setting it up, I observed how it works for
> > couple of days..
> >
> > I tried to to set max_active_runs=1 in the DAG
> > dag = DAG(...max_active_runs=1...) and it executed fine to avoid two runs
> > at the same time.
> > However, I noticed other dags (not the dag that is running) is also
> > "paused".
> > My understanding is that "max_active_runs" is basically
> > "max_active_runs_per_dag".
> > So, why another dag (different dag name) cannot run at the same time as
> the
> > first dag?
> > I want to have the two dags can be possibly run at the same time and
> inside
> > each dag, there is only
> > one run per dag.
> > Thanks.
> >
> > Jason
> >
> > My other settings in airflow.cfg
> >
> > max_active_runs_per_dag=1
> > parallelism = 32
> > dag_concurrency = 16
> >
> >
> >
> > On Mon, May 16, 2016 at 8:57 PM, Chris Riccomini <criccom...@apache.org>
> > wrote:
> >
> > > Hey Jason,
> > >
> > > For (2), by default, task1 will start running again. You'll have two
> runs
> > > going at the same time. If you want to prevent this, you can set
> > > max_active_runs to 1 in your DAG.
> > >
> > > Cheers,
> > > Chris
> > >
> > > On Mon, May 16, 2016 at 1:09 PM, Jason Chen <chingchien.c...@gmail.com
> >
> > > wrote:
> > >
> > > > I have two questions
> > > >
> > > > (1) For the airflow UI: "Tree view", it lists the tasks along with
> the
> > > time
> > > > highlighted in the top (say, 08:30; 09:00, etc). What's the meaning
> of
> > > > time? It looks not the UTC time of the task was running. I know in
> > > > overall, airflow uses UTC time
> > > > (2) I have a DAG with two tasks: task1 --> task2
> > > > Task1 is running hourly and could take longer than one hour to run,
> > > > sometimes.
> > > > In such a setup, task1 will be triggered hourly and what happens if
> the
> > > > previous task1 is still running ? Will the "new" task1 be queued ?
> > > >
> > > > Thanks.
> > > > Jason
> > > >
> > >
> >
>
