Re: [Airflow 2.0] State of the HA Scheduler - AIP-15 and Astronomer/Community plans?

This is really great. It was never my intention to question the viability of the HA scheduler for the 2.0 - if that was misunderstood - true apologies. I certainly appreciate the hard work done on it by the Astronomer team - I just wanted to stress that - unlike most other features of Airflow,

Re: Generic Transfer Operator

I would abstract Beam away from end-users. The goal was minimal to no transforms in the generic operator? So, yes, would have a beam dependency, but airflow-like interaction/wrapper (and then the ability to also extend and generate desired transforms). It sounds like dependency is a concern,

Re: Consider using stale bot for issues

+1 On Fri, Sep 11, 2020 at 7:12 PM Kaxil Naik wrote: > Yeah I agree and looks like Paola and Elad have already volunteered to help > triage. > > Regards, > Kaxil > > On Fri, Sep 11, 2020, 18:09 Tomasz Urbaszek wrote: > > > Hi all, > > > > The main reason I suggested the stale bot was the lack

Re: Consider using stale bot for issues

Yeah I agree and looks like Paola and Elad have already volunteered to help triage. Regards, Kaxil On Fri, Sep 11, 2020, 18:09 Tomasz Urbaszek wrote: > Hi all, > > The main reason I suggested the stale bot was the lack of any > widespread prioritization/reviewing of issues which results in big

Re: Consider using stale bot for issues

Hi all, The main reason I suggested the stale bot was the lack of any widespread prioritization/reviewing of issues which results in big pile of never addresses issues. I think that triage access is a better answer to this problem as engaging more people will help us all. I'm +1 for that.

Re: [Airflow 2.0] State of the HA Scheduler - AIP-15 and Astronomer/Community plans?

Jarek, It has been our intention (as Astronomer) to release the Scheduler HA work directly to open source as part of Airflow 2.0. We realized early on that the Scheduler reliability and performance were highlighted as the key issues from the community as part of the latest survey results from

Re: Consider using stale bot for issues

Hi all, Paola here from Astronomer. I've been working at Astro since our early days in both Customer and Product-centric roles and have of course been closely following all-things Airflow for a long time. I happened to read this conversation around GH Issue management and figure I can step up

Re: [Airflow 2.0] State of the HA Scheduler - AIP-15 and Astronomer/Community plans?

I am personally even super happy if Astronomer provides it to the customers with commercial obligations - until it is merged in 2.1 for example. Including the support - while we are discussing it and merging and releasing it in 2.1 (and likely later supporting migration to the community one

Re: [Airflow 2.0] State of the HA Scheduler - AIP-15 and Astronomer/Community plans?

Hi Jarek, and all. You aren't the only one to have this thought -- it's been on my mind too. Sadly I wasn't able to get the code in a PR-able state before heading off on paternity leave. I have started separating out and submitting the "easy"/preparatory PRs to try to lessen the size of the

Re: [Airflow 2.0] State of the HA Scheduler - AIP-15 and Astronomer/Community plans?

Actually :) the "Valid" link was marked as "Original AIP" in the doc I linked :). I will fix it. On Fri, Sep 11, 2020 at 2:14 PM Jarek Potiuk wrote: > Ah. Thanks! It just popped as the "first" when I looked at it I will mark > it as [ARCHIVED} and move to the archive. > So my point about

Re: [Airflow 2.0] State of the HA Scheduler - AIP-15 and Astronomer/Community plans?

Ah. Thanks! It just popped as the "first" when I looked at it I will mark it as [ARCHIVED} and move to the archive. So my point about updates are not really valid :). It's rather detailed and updated from what I see. J. On Fri, Sep 11, 2020 at 2:08 PM Kaxil Naik wrote: > Just a note here: >

Re: [Airflow 2.0] State of the HA Scheduler - AIP-15 and Astronomer/Community plans?

Just a note here: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651 is the correct link for the AIP proposed for Scheduler HA. The other link was an old proposal from someone else. On Fri, Sep

[Airflow 2.0] State of the HA Scheduler - AIP-15 and Astronomer/Community plans?

I started to feel that we need to clarify statements about the HA Scheduler for our community. Not that I am losing sleep regularly over this but it did keep me away last night when I started to think about it :). I have a feeling that while we already defined some - rather aggressive - timelines

Re: Generic Transfer Operator

I think the biggest downsides were already mentioned by Tomek: more dependency management when using apache-beam (plus possibility of conflicts between dependencies of beam and airflow) and no support for data lineage solutions. Besides, we create a higher entry threshold by creating a necessity