Re: Airflow scheduler process issue

2019-03-28 Thread Brian Greene
My quick guess - both running with local executor? That runs all af processes in one... so you’d have 2 schedulers. It’s perfectly acceptable to run AF across a couple EC2 instances, but you have to select which parts of the stack run in which one, and support multiple airflow Configs in some

Airflow scheduler process issue

2019-03-28 Thread Adityan MS
Hi all, We have 2 EC2 instances running airflow 1.10.0 apps. As per my understanding there should be only 1 airflow scheduler service running at a given time. However lately we have started to see that mysteriously 2 scheduler processes are running, which ends up causing a deadlock and our Airflo

Re: Hangout: Learn how to hack on Airflow (CHANGE OF DAY)

2019-03-28 Thread Austin Bennett
Would be great if recorded/available later -- have a meeting during the 'live' time. On Thu, Mar 28, 2019 at 9:09 AM Gabriele Di Bernardo wrote: > Hi Ash, > > Are you going to record the session in the end? > > Thank you so much! > > Best, > > > Gabriele > > > On 28 Mar 2019, at 11:35, Ash Berli

Re: Hangout: Learn how to hack on Airflow (CHANGE OF DAY)

2019-03-28 Thread Gabriele Di Bernardo
Hi Ash, Are you going to record the session in the end? Thank you so much! Best, Gabriele > On 28 Mar 2019, at 11:35, Ash Berlin-Taylor wrote: > > Hi all, > > This is today. Hope to see you all there. > > The link below is https://zoom.us/j/458267401 > > My original mail might not have b

Re: Hangout: Learn how to hack on Airflow (CHANGE OF DAY)

2019-03-28 Thread Ash Berlin-Taylor
Hi all, This is today. Hope to see you all there. The link below is https://zoom.us/j/458267401 My original mail might not have been all that clear on the focus of this session: it will be how I work on Airflow itself (rather than using Airflow to build dags) especially around running tests et