Interesting. Thanks for this solution Lance, gonna try it
> On Jul 6, 2016, at 19:11, Lance Norskog <lance.nors...@gmail.com> wrote:
>
> You could use the XCOM feature to post a semaphore at the start of the task
> and then remove it at the end.
> Another task would see the semaphore and immediately quit.
> If you get a race condition, the third 15-minute task will take care of the
> problem, 15 minutes late.
>
> Lance
>
> On Wed, Jul 6, 2016 at 2:50 PM, Cyril Scetbon <cyril.scet...@free.fr> wrote:
>
>> Hi,
>>
>> I have dags with tasks that use same configuration and same schedule time
>> frequencies.
>>
>> When I have x tasks in a dag they run in //. My dag is scheduled every 15
>> minutes, but sometimes I have at least a task that runs for more than 15
>> minutes and 2 identical tasks should not run at the same time (in my case)
>> but it's what happens.
>>
>> So when the dag is kicked off (every 15 min) I need Airflow to run tasks
>> only for those that are not running, which means :
>>
>> - if at time t1 tasks tk1 is running and tk2, ..., tkn are not running, I
>> need Airflow to run only tk2,.., tkn but not tk1 cause it's already running
>> - if at time t2 tasks tk1, ..., tkn are not running, I need Airflow to run
>> tk1,.., tkn
>>
>> I already tried using depends_on_past=True, however I need failed tasks to
>> be kicked off, cause I can get some temporary issues.
>>
>> I also use an upstream task connected to all tasks in my dag to be able to
>> run manually all tasks if I need to. (I don't use a frequency of 15 minutes
>> in all dags)
>>
>> Any idea ?
>
>
>
>
> --
> Lance Norskog
> lance.nors...@gmail.com
> Redwood City, CA
