Did we get anywhere on this? What do others think? For context: The code has moved quite a bit between Airflow 1.10 and Airflow 2.0 and it becomes increasingly difficult to backport Core changes without rewriting the PRs.
Regards, Kaxil On Tue, Mar 24, 2020 at 2:54 PM Daniel Imberman <daniel.imber...@gmail.com> wrote: > Hi Robin, > > > > I feel some of the stuff for instance Schedular HA could wait for a > point > > > release of version 2 (although maybe this a lot further a long than I > am > > > aware). Like you mentioned Spark did with K8s. > > > > I agree on this part. The focus would be more on the breaking changes and > major features. @ash has been determining which features can wait. > > > > Also does the new API need to be feature complete or just enough > > > functionality to warrant removing the existing experimental one. > > It needs to be complete enough for us to run full integration tests and > for us to assume that we will not need major revisions in the near future. > It will of course continue to evolve over time (though hopefully mostly > through additions). > > Daniel > On Mar 24, 2020, 6:33 AM -0700, Kaxil Naik <kaxiln...@gmail.com>, wrote: > > +1 > > > > > > > > On Sun, Mar 22, 2020 at 8:19 AM Robin Edwards <r...@bidnamic.com> wrote: > > > > > I feel some of the stuff for instance Schedular HA could wait for a > point > > > release of version 2 (although maybe this a lot further a long than I > am > > > aware). Like you mentioned Spark did with K8s. > > > > > > Also does the new API need to be feature complete or just enough > > > functionality to warrant removing the existing experimental one. > > > > > > Like you said, less sooner is better. > > > > > > R > > > > > > On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <daniel.imber...@gmail.com > > > > > wrote: > > > > > > > Great! Hope to get a few more folx to give +1's but I think we have > a good > > > > path forward here :) > > > > > > > > On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk < > jarek.pot...@polidea.com> > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > I agree especially for larger-scale users migrations are a > difficult > > > > > > process. Perhaps we can adopt something similar to a blockchain > fork > > > > > (e.g. > > > > > > determine X known airflow using companies, and start the > countdown as > > > > > soon > > > > > > as Y% of them migrate). I really just want to make sure we don't > end > > > > up > > > > > > with a python2/3 situation. Even if we continue support it should > > > > only be > > > > > > for bugfixes and we should not add any new features into 1.10. > > > > > > > > > > > > > > > > I think we are in perfect sync - I think feature-migration should > end > > > > > almost immediately after we release 2.0. But bug-fixing should > continue > > > > > for quite some time. On that front - having backport packages will > help > > > > > with releasing "integrations" quite independently from 1.10/2.0 > version > > > > > (which I think is good for those who are - for this or another > reason - > > > > > stuck on 2.0). On the other hand we should make sure that the > important > > > > > stuff for 2.0 that is not "feature" is also backported to 1.10. For > > > > example > > > > > a lot of recent performance improvements that we have now in 2.0 > will > > > > > be possible (and not that complex) to backport to 1.10. Some of > this > > > > effort > > > > > is actually easier to do in 2.0 and then apply to 1.10 in similar > > > > fashion > > > > > as it is easier to understand and reason about the 2.0 code now > when > > > > > we have some refactoring/pylints etc in place. So we should make > sure > > > > > we get the latest 1.10 to a "good" state - before we freeze it for > > > > bugfix > > > > > only. > > > > > I know it might mean that some people will stay with 1.10 for > longer, > > > > but > > > > > that's also OK for them. The reason to migrate to 2.0 should be not > > > > > performance but some important features (like API or HA) that come > > > > > with it. > > > > > > > > > > I couldn't agree more :). If we can start people writing (close > to) 2.0 > > > > > > compliant DAGs before the release of 2.0 that will make the > migration > > > > > > process so much easier :). > > > > > > > > > > > > > > > > Yeah. I even thought that we should write a > > > > > "How good your DAGs are for 2.0" assessment tool. > > > > > > > > > > > > > > > > If there aren't any extra steps or features that we need to add > > > > (beyond > > > > > the > > > > > > ones discussed here), I think a good next step would be to > create an > > > > > > official checklist just so we can see all of these features in > one > > > > place > > > > > > (and hopefully start breaking them down into as small of changes > as > > > > > > possible). > > > > > > > > > > > > Does that sound ok? > > > > > > > > > > > > > > > > Perfectly OK for me! > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Jarek Potiuk > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > >
