Hi Robin, > > I feel some of the stuff for instance Schedular HA could wait for a point > > release of version 2 (although maybe this a lot further a long than I am > > aware). Like you mentioned Spark did with K8s. > > I agree on this part. The focus would be more on the breaking changes and major features. @ash has been determining which features can wait.
> > Also does the new API need to be feature complete or just enough > > functionality to warrant removing the existing experimental one. It needs to be complete enough for us to run full integration tests and for us to assume that we will not need major revisions in the near future. It will of course continue to evolve over time (though hopefully mostly through additions). Daniel On Mar 24, 2020, 6:33 AM -0700, Kaxil Naik <kaxiln...@gmail.com>, wrote: > +1 > > > > On Sun, Mar 22, 2020 at 8:19 AM Robin Edwards <r...@bidnamic.com> wrote: > > > I feel some of the stuff for instance Schedular HA could wait for a point > > release of version 2 (although maybe this a lot further a long than I am > > aware). Like you mentioned Spark did with K8s. > > > > Also does the new API need to be feature complete or just enough > > functionality to warrant removing the existing experimental one. > > > > Like you said, less sooner is better. > > > > R > > > > On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <daniel.imber...@gmail.com> > > wrote: > > > > > Great! Hope to get a few more folx to give +1's but I think we have a good > > > path forward here :) > > > > > > On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk <jarek.pot...@polidea.com> > > > wrote: > > > > > > > > > > > > > > > > > > I agree especially for larger-scale users migrations are a difficult > > > > > process. Perhaps we can adopt something similar to a blockchain fork > > > > (e.g. > > > > > determine X known airflow using companies, and start the countdown as > > > > soon > > > > > as Y% of them migrate). I really just want to make sure we don't end > > > up > > > > > with a python2/3 situation. Even if we continue support it should > > > only be > > > > > for bugfixes and we should not add any new features into 1.10. > > > > > > > > > > > > > I think we are in perfect sync - I think feature-migration should end > > > > almost immediately after we release 2.0. But bug-fixing should continue > > > > for quite some time. On that front - having backport packages will help > > > > with releasing "integrations" quite independently from 1.10/2.0 version > > > > (which I think is good for those who are - for this or another reason - > > > > stuck on 2.0). On the other hand we should make sure that the important > > > > stuff for 2.0 that is not "feature" is also backported to 1.10. For > > > example > > > > a lot of recent performance improvements that we have now in 2.0 will > > > > be possible (and not that complex) to backport to 1.10. Some of this > > > effort > > > > is actually easier to do in 2.0 and then apply to 1.10 in similar > > > fashion > > > > as it is easier to understand and reason about the 2.0 code now when > > > > we have some refactoring/pylints etc in place. So we should make sure > > > > we get the latest 1.10 to a "good" state - before we freeze it for > > > bugfix > > > > only. > > > > I know it might mean that some people will stay with 1.10 for longer, > > > but > > > > that's also OK for them. The reason to migrate to 2.0 should be not > > > > performance but some important features (like API or HA) that come > > > > with it. > > > > > > > > I couldn't agree more :). If we can start people writing (close to) 2.0 > > > > > compliant DAGs before the release of 2.0 that will make the migration > > > > > process so much easier :). > > > > > > > > > > > > > Yeah. I even thought that we should write a > > > > "How good your DAGs are for 2.0" assessment tool. > > > > > > > > > > > > > If there aren't any extra steps or features that we need to add > > > (beyond > > > > the > > > > > ones discussed here), I think a good next step would be to create an > > > > > official checklist just so we can see all of these features in one > > > place > > > > > (and hopefully start breaking them down into as small of changes as > > > > > possible). > > > > > > > > > > Does that sound ok? > > > > > > > > > > > > > Perfectly OK for me! > > > > > > > > > > > > > > > > > > -- > > > > > > > > Jarek Potiuk > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > >
