Hey Jason,
The SequentialExecutor only ever runs one task at a time. It's meant for
debugging purposes. Try switching to the LocalExecutor.
Cheers,
Chris
On Tue, May 31, 2016 at 3:31 PM, Jason Chen <chingchien.c...@gmail.com>
wrote:
> Chris,
> I am running SequentialExecutor.
>
> Thanks.
> Jason
>
>
> On Tue, May 31, 2016 at 1:36 PM, Chris Riccomini <criccom...@apache.org>
> wrote:
>
>> Hey Jason,
>>
>> Are you running the SerialExecutor? This is the default out-of-the-box
>> executor.
>>
>> Cheers,
>> Chris
>>
>> On Tue, May 31, 2016 at 12:59 PM, Jason Chen <chingchien.c...@gmail.com>
>> wrote:
>>
>>> Hi Chris,
>>>
>>> I made the changes and tried it out.
>>> It seems not working as expected.
>>> When a dag is running (a particular task inside that dag is taking
>>> time), another task from another dag seems "blocked".
>>>
>>> My setting:
>>> (1) airflow.cfg
>>> max_active_runs_per_dag = 16
>>> parallelism = 32
>>> dag_concurrency = 16
>>>
>>> (2) A dag (dag1) python file is as below partially. Please note that
>>> inside this DAG, the first task (task1) is a long running task
>>>
>>> dag1 = DAG('dag1', schedule_interval=timedelta(minutes=15),
>>> max_active_runs=1, default_args=args)
>>>
>>> Then, the tasks are running in the order...
>>> task1 (long running) --> task 2 --> task3
>>> ...
>>> (3) In another dag (dag2) python file is as below partially.
>>> dag2 = DAG('dag2', schedule_interval=timedelta(minutes=3),
>>> max_active_runs=1, default_args=args)
>>> ...
>>> Then, the tasks are running in the order...
>>> taskA (short running task) --> taskB
>>>
>>> (4) Inside the upstart script file. this is the main part how I start
>>> airflow scheduler
>>>
>>> env SCHEDULER_RUNS=0
>>> export SCHEDULER_RUNS
>>>
>>> script
>>> exec >> ${AIRFLOW_HOME}/scheduler-log/airflow-scheduler.log 2>&1
>>> exec usr/local/bin/airflow scheduler -n ${SCHEDULER_RUNS}
>>> end script
>>>
>>> =========================
>>>
>>> What I observed are that
>>> (a) task1 (of dag1) is running about 20 mins and during it's running
>>> time, there is no other dag1 triggered. This is as expected.
>>>
>>> (b) taskA (of dag2) should be triggered to run every 3 mins. However, it
>>> is NOT triggered if task-1 of dag-1 is running.
>>> taskA seems to be queued/bolcked and not run. It is executed after
>>> task-1 (of dag-1) is done. So, it looks like it is dispatched into a "gap"
>>> of task1 and task2 (of dag1). This looks not normal, as it's expected taskA
>>> (of dag 2) should run no matter what happens to another dag (dag-1).
>>>
>>>
>>> Any suggestions?
>>> Thanks.
>>> Jason
>>>
>>>
>>> On Tue, May 31, 2016 at 9:02 AM, Chris Riccomini <criccom...@apache.org>
>>> wrote:
>>>
>>>> Hey Jason,
>>>>
>>>> The problem is max_active_runs_per_dag=1. Set it back to 16. You just
>>>> need
>>>> max_active_runs=1 for the individual DAGs. This will allow multiple
>>>> (different) DAGs to run in parallel, but only one DAG of each type can
>>>> run
>>>> at the same type.
>>>>
>>>> Cheers,
>>>> Chris
>>>>
>>>> On Fri, May 27, 2016 at 11:42 PM, Jason Chen <chingchien.c...@gmail.com
>>>> >
>>>> wrote:
>>>>
>>>> > Hi Chris,
>>>> > Thanks for your reply. After setting it up, I observed how it works
>>>> for
>>>> > couple of days..
>>>> >
>>>> > I tried to to set max_active_runs=1 in the DAG
>>>> > dag = DAG(...max_active_runs=1...) and it executed fine to avoid two
>>>> runs
>>>> > at the same time.
>>>> > However, I noticed other dags (not the dag that is running) is also
>>>> > "paused".
>>>> > My understanding is that "max_active_runs" is basically
>>>> > "max_active_runs_per_dag".
>>>> > So, why another dag (different dag name) cannot run at the same time
>>>> as the
>>>> > first dag?
>>>> > I want to have the two dags can be possibly run at the same time and
>>>> inside
>>>> > each dag, there is only
>>>> > one run per dag.
>>>> > Thanks.
>>>> >
>>>> > Jason
>>>> >
>>>> > My other settings in airflow.cfg
>>>> >
>>>> > max_active_runs_per_dag=1
>>>> > parallelism = 32
>>>> > dag_concurrency = 16
>>>> >
>>>> >
>>>> >
>>>> > On Mon, May 16, 2016 at 8:57 PM, Chris Riccomini <
>>>> criccom...@apache.org>
>>>> > wrote:
>>>> >
>>>> > > Hey Jason,
>>>> > >
>>>> > > For (2), by default, task1 will start running again. You'll have
>>>> two runs
>>>> > > going at the same time. If you want to prevent this, you can set
>>>> > > max_active_runs to 1 in your DAG.
>>>> > >
>>>> > > Cheers,
>>>> > > Chris
>>>> > >
>>>> > > On Mon, May 16, 2016 at 1:09 PM, Jason Chen <
>>>> chingchien.c...@gmail.com>
>>>> > > wrote:
>>>> > >
>>>> > > > I have two questions
>>>> > > >
>>>> > > > (1) For the airflow UI: "Tree view", it lists the tasks along
>>>> with the
>>>> > > time
>>>> > > > highlighted in the top (say, 08:30; 09:00, etc). What's the
>>>> meaning of
>>>> > > > time? It looks not the UTC time of the task was running. I know
>>>> in
>>>> > > > overall, airflow uses UTC time
>>>> > > > (2) I have a DAG with two tasks: task1 --> task2
>>>> > > > Task1 is running hourly and could take longer than one hour to
>>>> run,
>>>> > > > sometimes.
>>>> > > > In such a setup, task1 will be triggered hourly and what happens
>>>> if the
>>>> > > > previous task1 is still running ? Will the "new" task1 be queued ?
>>>> > > >
>>>> > > > Thanks.
>>>> > > > Jason
>>>> > > >
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>
