Hi Kaxil, I’m currently working on creating the list of necessary fixes. One thing I’m prioritizing before we shift over is porting or deleting all of the old JIRA tickets. It would be great to start 2.0 with as clean of a slate as possible. However, I will try to get that proposal out sooner so others can review it in parallel with the ticket migration.
On Tue, Mar 31, 2020 at 11:08 AM, Kaxil Naik <kaxiln...@gmail.com> wrote: Did we get anywhere on this? What do others think? For context: The code has moved quite a bit between Airflow 1.10 and Airflow 2.0 and it becomes increasingly difficult to backport Core changes without rewriting the PRs. Regards, Kaxil On Tue, Mar 24, 2020 at 2:54 PM Daniel Imberman <daniel.imber...@gmail.com> wrote: > Hi Robin, > > > > I feel some of the stuff for instance Schedular HA could wait for a > point > > > release of version 2 (although maybe this a lot further a long than I > am > > > aware). Like you mentioned Spark did with K8s. > > > > I agree on this part. The focus would be more on the breaking changes and > major features. @ash has been determining which features can wait. > > > > Also does the new API need to be feature complete or just enough > > > functionality to warrant removing the existing experimental one. > > It needs to be complete enough for us to run full integration tests and > for us to assume that we will not need major revisions in the near future. > It will of course continue to evolve over time (though hopefully mostly > through additions). > > Daniel > On Mar 24, 2020, 6:33 AM -0700, Kaxil Naik <kaxiln...@gmail.com>, wrote: > > +1 > > > > > > > > On Sun, Mar 22, 2020 at 8:19 AM Robin Edwards <r...@bidnamic.com> wrote: > > > > > I feel some of the stuff for instance Schedular HA could wait for a > point > > > release of version 2 (although maybe this a lot further a long than I > am > > > aware). Like you mentioned Spark did with K8s. > > > > > > Also does the new API need to be feature complete or just enough > > > functionality to warrant removing the existing experimental one. > > > > > > Like you said, less sooner is better. > > > > > > R > > > > > > On Fri, 20 Mar 2020, 20:29 Daniel Imberman, <daniel.imber...@gmail.com > > > > > wrote: > > > > > > > Great! Hope to get a few more folx to give +1's but I think we have > a good > > > > path forward here :) > > > > > > > > On Fri, Mar 20, 2020 at 12:51 PM Jarek Potiuk < > jarek.pot...@polidea.com> > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > I agree especially for larger-scale users migrations are a > difficult > > > > > > process. Perhaps we can adopt something similar to a blockchain > fork > > > > > (e.g. > > > > > > determine X known airflow using companies, and start the > countdown as > > > > > soon > > > > > > as Y% of them migrate). I really just want to make sure we don't > end > > > > up > > > > > > with a python2/3 situation. Even if we continue support it should > > > > only be > > > > > > for bugfixes and we should not add any new features into 1.10. > > > > > > > > > > > > > > > > I think we are in perfect sync - I think feature-migration should > end > > > > > almost immediately after we release 2.0. But bug-fixing should > continue > > > > > for quite some time. On that front - having backport packages will > help > > > > > with releasing "integrations" quite independently from 1.10/2.0 > version > > > > > (which I think is good for those who are - for this or another > reason - > > > > > stuck on 2.0). On the other hand we should make sure that the > important > > > > > stuff for 2.0 that is not "feature" is also backported to 1.10. For > > > > example > > > > > a lot of recent performance improvements that we have now in 2.0 > will > > > > > be possible (and not that complex) to backport to 1.10. Some of > this > > > > effort > > > > > is actually easier to do in 2.0 and then apply to 1.10 in similar > > > > fashion > > > > > as it is easier to understand and reason about the 2.0 code now > when > > > > > we have some refactoring/pylints etc in place. So we should make > sure > > > > > we get the latest 1.10 to a "good" state - before we freeze it for > > > > bugfix > > > > > only. > > > > > I know it might mean that some people will stay with 1.10 for > longer, > > > > but > > > > > that's also OK for them. The reason to migrate to 2.0 should be not > > > > > performance but some important features (like API or HA) that come > > > > > with it. > > > > > > > > > > I couldn't agree more :). If we can start people writing (close > to) 2.0 > > > > > > compliant DAGs before the release of 2.0 that will make the > migration > > > > > > process so much easier :). > > > > > > > > > > > > > > > > Yeah. I even thought that we should write a > > > > > "How good your DAGs are for 2.0" assessment tool. > > > > > > > > > > > > > > > > If there aren't any extra steps or features that we need to add > > > > (beyond > > > > > the > > > > > > ones discussed here), I think a good next step would be to > create an > > > > > > official checklist just so we can see all of these features in > one > > > > place > > > > > > (and hopefully start breaking them down into as small of changes > as > > > > > > possible). > > > > > > > > > > > > Does that sound ok? > > > > > > > > > > > > > > > > Perfectly OK for me! > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Jarek Potiuk > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > >
