Re: Multiple Schedulers - "scheduler_lock"

2019-03-01 Thread Tao Feng
Does the proposal use master-slave architecture(leader scheduler vs slave scheduler)? On Fri, Mar 1, 2019 at 5:32 PM Kevin Yang wrote: > Preventing double-triggering by separating DAG files different schedulers > parse sounds easier and more intuitive. I actually removed one of the > double-trig

Re: Multiple Schedulers - "scheduler_lock"

2019-03-01 Thread Kevin Yang
Preventing double-triggering by separating DAG files different schedulers parse sounds easier and more intuitive. I actually removed one of the double-triggering prevention logic here (expensive) and was re

Re: Multiple Schedulers - "scheduler_lock"

2019-03-01 Thread Maxime Beauchemin
Forgot to mention: the intention was to use the lock, but I never personally got to do the second phase which would consist of skipping the DAG if the lock is on, and expire the lock eventually based on a config setting. Max On Fri, Mar 1, 2019 at 1:57 PM Maxime Beauchemin wrote: > My original

Re: Multiple Schedulers - "scheduler_lock"

2019-03-01 Thread Maxime Beauchemin
My original intention with the lock was preventing "double-triggering" of task (triggering refers to the scheduler putting the message in the queue). Airflow now has good "double-firing-prevention" of tasks (firing happens when the worker receives the message and starts the task), even if the sched

Re: Change Airflow Dag Execution time timezone on Airflow UI

2019-03-01 Thread jiajie zhong
I have serveral featrues to handle in the same time, and I not good at web developer. I let a msg in the JIRA ticket and I will do it when I finish my feature and if ticket still no one assignee From: Tao Feng Sent: Friday, March 1, 2019 14:39 To: dev@airflow.apa

Re: Multiple Schedulers - "scheduler_lock"

2019-03-01 Thread Deng Xiaodong
It’s exactly what my team is doing & what I shared here earlier last year (https://lists.apache.org/thread.html/0e21230e08f07ef6f8e3c59887e9005447d6932639d3ce16a103078f@%3Cdev.airflow.apache.org%3E

Re: Multiple Schedulers - "scheduler_lock"

2019-03-01 Thread Deng Xiaodong
Thanks @Bolke for the inputs! In that case, possibly we can change part of the AIP scope to “thoroughly test if running multiple schedulers causes issue”. And a few thoughts about your inputs: - “Database locking is mostly in place (DagRuns and TaskInstances)”: let’s say we already have DagRun x

Re: Multiple Schedulers - "scheduler_lock"

2019-03-01 Thread Mario Urquizo
We have been running multiple schedulers for about 3 months. We created multiple services to run airflow schedulers. The only difference is that we have each of the schedulers pointed to a directory one level deeper than the DAG home directory that the workers and webapp use. We have seen much be

Re: Multiple Schedulers - "scheduler_lock"

2019-03-01 Thread Bolke de Bruin
I have done quite some work on making it possible to run multiple schedulers at the same time. At the moment I don’t think there are real blockers actually to do so. We just don’t actively test it. Database locking is mostly in place (DagRuns and TaskInstances). And I think the worst that can

Multiple Schedulers - "scheduler_lock"

2019-03-01 Thread Deng Xiaodong
Hi Max, Following https://lists.apache.org/thread.html/0e21230e08f07ef6f8e3c59887e9005447d6932639d3ce16a103078f@%3Cdev.airflow.apache.org%3E , I’m trying to prepare an AI

Re: AIRFLOW-3918 Adding SSH authentication to git-sync for Kubernetes

2019-03-01 Thread Daniel Mateus Pires
Thanks Ash! On Fri, 1 Mar 2019 at 15:01, Ash Berlin-Taylor wrote: > Nice work! Have reviewed this (and Daniel was prompt on updating) so this > is now merged. > > I'll try and get this in 1.10.3 too. Which I need to start working on > preparing a release of. > > -ash > > > On 28 Feb 2019, at 10:

Re: AIRFLOW-3918 Adding SSH authentication to git-sync for Kubernetes

2019-03-01 Thread Ash Berlin-Taylor
Nice work! Have reviewed this (and Daniel was prompt on updating) so this is now merged. I'll try and get this in 1.10.3 too. Which I need to start working on preparing a release of. -ash > On 28 Feb 2019, at 10:24, Daniel Mateus Pires wrote: > > Hi devs! > > I'm trying to get some traction