Re: Let's talk Airflow 2.0

6. Prod image is there :) -> alpha quality for now and ready to be tested. And I have plans to add more tests and with Daniel/Greg integrate it with the helm chart and make both - the image and helm the "official" ones. Alpha version of the image will also be released with 1.10.10 so that people ca

Re: Let's talk Airflow 2.0

> > Move (tested) components out of contrib folder > > https://lists.apache.org/thread.html/c880ef89f8cb4a0240c404f9372615b998c4a4eeca342651927d596c@%3Cdev.airflow.apache.org%3E AIP-21 is done. We have a follow-up in the form of system tests and backport packages, but this is not a blocker. We ca

Re: Let's talk Airflow 2.0

4. Improve Scheduler performance and reliability The performance was improved by my work. https://lists.apache.org/thread.html/r4b0a6385d263defaf0fe733a48fb291e3a763da2f172ca1931a652b7%40%3Cdev.airflow.apache.org%3E In Q2, my team will continue to work on performance and reliability. We want to p

Re: Let's talk Airflow 2.0

>>Hi Bin, >> >> I had very similar thoughts about SubDags. End of day DAGs are just graphs and graph joins are pretty easy operations. I’m gonna file a GitHub issue and assign us both and let’s hash it out there :) Sounds great! Looking forward to it. On Thu, Apr 2, 2020 at 1:47 PM Daniel Imberma

Re: Let's talk Airflow 2.0

> I have some thoughts on the Subdag (will open a new thread if necessary). > Instead of having a separate child DAG, would it be better to chain all the > tasks from the child dag to the parent dag and then drop the child dag? > In this way, the whole child dag (actually just the tasks in it) will

Re: Let's talk Airflow 2.0

Hi Bin, I had very similar thoughts about SubDags. End of day DAGs are just graphs and graph joins are pretty easy operations. I’m gonna file a GitHub issue and assign us both and let’s hash it out there :) via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=

Re: Let's talk Airflow 2.0

>> I think it's pretty damn crucial we fix subdags and backfills. I'm on the >> fence about this one. On the one hand it could possibly wait. On the other >> hand it would be embarrasing to release a 2.0 and still have this feature >> broken I would really like to see subdags and backfills being

Re: Let's talk Airflow 2.0

Hello all, I've been reviewing This wiki page and I wanted to discuss which of these features are still relevant/are blockers for 2.0 I figured we should start with the status of the high priority fixes 1. *Knative Executor (h

Re: Let's talk Airflow 2.0

Got it. Thanks Daniel for leading this On Tue, Mar 31, 2020 at 7:40 PM Daniel Imberman wrote: > I think including both is fine as long as the old one contains deprecation > warnings/force a feature flag to allow it (e.g. —allow-deprecated) > > via Newton Mail >

Re: Let's talk Airflow 2.0

I think including both is fine as long as the old one contains deprecation warnings/force a feature flag to allow it (e.g. —allow-deprecated) via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2] On Tue, Mar 31, 2020 at 11:33 AM, Kamil Breguła wr

Re: Let's talk Airflow 2.0

On Sun, Mar 22, 2020 at 9:20 AM Robin Edwards wrote: > Also does the new API need to be feature complete or just enough > functionality to warrant removing the existing experimental one. > I think we should release at least one version that will contain the new and old REST APIs simultaneously.

Re: Let's talk Airflow 2.0

Hi Kaxil, I’m currently working on creating the list of necessary fixes. One thing I’m prioritizing before we shift over is porting or deleting all of the old JIRA tickets. It would be great to start 2.0 with as clean of a slate as possible. However, I will try to get that proposal out sooner s

Re: Let's talk Airflow 2.0

Did we get anywhere on this? What do others think? For context: The code has moved quite a bit between Airflow 1.10 and Airflow 2.0 and it becomes increasingly difficult to backport Core changes without rewriting the PRs. Regards, Kaxil On Tue, Mar 24, 2020 at 2:54 PM Daniel Imberman wrote: >

Re: Let's talk Airflow 2.0

Hi Robin, > > I feel some of the stuff for instance Schedular HA could wait for a point > > release of version 2 (although maybe this a lot further a long than I am > > aware). Like you mentioned Spark did with K8s. > > I agree on this part. The focus would be more on the breaking changes and majo

Re: Let's talk Airflow 2.0

+1 On Sun, Mar 22, 2020 at 8:19 AM Robin Edwards wrote: > I feel some of the stuff for instance Schedular HA could wait for a point > release of version 2 (although maybe this a lot further a long than I am > aware). Like you mentioned Spark did with K8s. > > Also does the new API need to be

Re: Let's talk Airflow 2.0

I feel some of the stuff for instance Schedular HA could wait for a point release of version 2 (although maybe this a lot further a long than I am aware). Like you mentioned Spark did with K8s. Also does the new API need to be feature complete or just enough functionality to warrant removing the

Re: Let's talk Airflow 2.0

Great! Hope to get a few more folx to give +1's but I think we have a good path forward here :) On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk wrote: > > > > > > I agree especially for larger-scale users migrations are a difficult > > process. Perhaps we can adopt something similar to a blockchai

Re: Let's talk Airflow 2.0

> > > I agree especially for larger-scale users migrations are a difficult > process. Perhaps we can adopt something similar to a blockchain fork (e.g. > determine X known airflow using companies, and start the countdown as soon > as Y% of them migrate). I really just want to make sure we don't end

Re: Let's talk Airflow 2.0

Hi Jarek, thank you for your support on this effort! For sure. One comment here. One of the ways we discussed with Kamil > about the approach we want to take is to engage more people in > implementing it - so Kamil would create a basic framework and we might > have many people implementing and te

Re: Let's talk Airflow 2.0

> Hello, fellow Airflowers! > > I wanted to start this discussion in hopes of "kicking the tires" and > agreeing on a timeline for the Airflow 2.0 release. Cool! > > But I want to add _ feature which would be so awesome in a 2.0 press > release! > > We should also > consider that the 2.0 rele

Let's talk Airflow 2.0

Hello, fellow Airflowers! I wanted to start this discussion in hopes of "kicking the tires" and agreeing on a timeline for the Airflow 2.0 release. It seems to me that our 1.10 branch is straying farther and farther from master, and with that, there are two significant consequences. Releases beco