Nikolay, I am up for the call. I will try to explain my reasoning in greater detail and will be glad to hear the concerns. Will this Friday, Nov 6th, work?
вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <nizhi...@apache.org>: > Igniters, should we have a call for this topic? > > > 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <ptupit...@apache.org> > написал(а): > > > >> not intend to rewrite everything from scratch > > > >> Every single test from Ignite 2.x should be moved to Ignite 3 > >> regardless of how we choose to proceed. > > > > Alexey, thank you for the explanation, this addresses all of my concerns. > > > > > > > > > > > > On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov < > andrey.mashen...@gmail.com> > > wrote: > > > >> Hi, Igniters. > >> > >> * AFAIU, we need a new repo if we want to apply different restrictions > to > >> pull requests, > >> otherwise I see no difference for myself. > >> E.g. make static analysis (do we have?), compile, styles, and javadoc > >> checks mandatory. > >> > >> I think that relaxed requirements here will lead to bad product quality. > >> > >> * Agree with Pavel, we should 'keep' integrations tests somehow. > >> During active development tests will be broken most of time, so, > >> I'd port them e.g. suite-by-suite once we will have a stable and > featured > >> environment to run them and of course make test's code clear and avoid > >> bad/non-relevant ones. > >> > >> * I like bottom-up approach. > >> With it we could make a better framework. I mean clear component > lifecycle, > >> component wiring mechanics, general methods to approach core components > >> such as exchange/communication > >> to avoid code mess like we have in ExchangeFuture with all these custom > >> callbacks for each component, interfaces like > >> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and > >> a pack of > start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected > >> and so on in various unexpected places. > >> Hope, we will be able to port most of the good code to the new framework > >> version. > >> > >> > >> > >> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk < > >> alexey.goncha...@gmail.com> > >> wrote: > >> > >>> Nikolay, Pavel, > >>> > >>> Thanks for the feedback! First of all, I wanted to stress that I do not > >>> intend to rewrite everything from scratch (I never used this phrase). > >> There > >>> are significant parts of code that would be moved with minimal > >>> modifications. Second, I never said that we will get rid of the old > tests > >>> codebase. Every single test from Ignite 2.x should be moved to Ignite 3 > >>> regardless of how we choose to proceed. > >>> > >>> My point is that for some parts of the code a clean bottom-up > >>> implementation will be cheaper in many ways. Let me give you a few > >> concrete > >>> examples: > >>> > >>> - I think no one can object that we need a cleanly separated > >> persistence > >>> layer for Ignite. There is a very raw draft IEP for this already. On > >> the > >>> other hand, I think we also can agree that we need a split-brain > >>> resistant > >>> replication protocol for caches. There is also an IEP for this. > >> Neither > >>> of > >>> the changes is a good fit for 2.x because they are likely to > introduce > >>> breaking changes in the persistence layer, configuration and > behavior. > >>> Additionally, these components are now tightly coupled, so there is > no > >>> way > >>> these two changes can be implemented in parallel and then merged > >>> together > >>> easily. So what we will end up with is having to implement these > >> changes > >>> sequentially, fixing all existing tests twice, and essentially > >> throwing > >>> away half of the work done because the other part of the change is > >>> re-implemented > >>> - Similar example goes with getting rid of IgniteInternalFuture and > >>> replacing it with CompletableFuture, and any other change that > touches > >>> the > >>> asynchronous part of the code. > >>> > >>> Third, I do not see how this choice affects the UX of Ignite. The end > >> user > >>> experience must be fixed in the IEP regardless of the development > process > >>> and the fact that we have gaps in this area in Ignite 2.x just confirms > >>> that. > >>> > >>> Pavel, agree that a repo/branch is a technicality, I guess if > >> reformulate, > >>> my point is that we might agree to have a single development master > >> branch > >>> with 'disassembled' end-to-end functionality for some period of time to > >>> speed up development, and re-assemble the core features after having > >>> submodules tested independently. > >>> > >>> Nikolay, > >>>> We have many features that have to evolve. > >>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc. > >>> This is not very specific. In the end, resources are limited and we > will > >>> not be able to drive both tracks simultaneously, especially after a > >> couple > >>> of features having been implemented for Ignite 3.0. If there are indeed > >>> some major changes that we want to do in Ignite 2.x instead of putting > >>> effort into 3.0 - let's discuss them. I am just not aware of any, > that's > >>> why I am eager to move forward with Ignite 3.0. > >>> > >>>> We have bugs and issues that can be fixed in 2.x without breaking > >> backward > >>> compatibility. > >>>> We have many users who are happy with the 2.x with all it’s issues. > >>> These changes will be covered by end-to-end tests and migrated to > Ignite > >>> 3.0, so I see no issues here. > >>> > >>> Finally, Anton & Nikolay > >>> I do not have an estimate for this simply because the activity is > >>> community-driven and it depends on the number of people willing to > >>> contribute. With the current pace, I would hope to have an RC of Ignite > >> 3.0 > >>> to be ready by the end of 2021. My gut feeling is that by moving with > >>> incremental changes, we will not be able to implement even half of the > >>> wishlist by that time. > >>> I doubt that releasing several major releases with breaking changes > will > >>> make Ignite users happy either because each upgrade will cost Ignite > >> users > >>> money, so the fewer major versions we release, the better. Thus my wish > >> to > >>> include all breaking changes in one release. > >>> > >>> I'll be now quiet for a while, let's see what other community members > >>> think. > >>> > >>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <ptupit...@apache.org>: > >>> > >>>> 1. Rewriting from scratch is never a good idea. > >>>> We don't want to follow the path of Netscape and lose all our users > >>>> by the time we have a working 3.0 [1] > >>>> > >>>> 2. Not sure about new repo - seems like some pain and no gain, what's > >> the > >>>> problem with a branch? > >>>> > >>>> 3. We should keep existing integration tests when possible. > >>>> We have accumulated a lot of edge case knowledge over the years, > >>>> it is not a good idea to send all of that down the drain. > >>>> Yes, integration tests are slow, but they are the most valuable. > >>>> I think we can move more stuff into nightly runs and have a fast and > >>> modern > >>>> basic suite. > >>>> > >>>> > >>>> Alexey, you are much more familiar with the Ignite core codebase than > >>> most > >>>> of us, > >>>> can you please explain in more detail which particular feature, in > your > >>>> opinion, > >>>> mandates this "start from scratch" approach? > >>>> Is it really not possible at all to follow a less radical way? > >>>> > >>>> > >>>> [1] > >>>> > >>>> > >>> > >> > https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ > >>>> > >>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <nizhi...@apache.org> > >>>> wrote: > >>>> > >>>>> Hello, Alexey. > >>>>> > >>>>> I think that «rewriting from scratch» approach has a high risk to > >> make > >>>> new > >>>>> features unusable. > >>>>> At the time Ignite2 was started no-one wants to do bad UX or bad > >>>> features. > >>>>> Nevertheless, it happen. > >>>>> > >>>>> I think we can avoid it with the Ignite3 and successors if we will > >> move > >>>>> step by step without keeping backward compatibility > >>>>> With the step by step approach, we can focus on each component > >>>> separately. > >>>>> > >>>>>> What new features are we planning to implement for Ignite 2.x? > >>>>> > >>>>> We have many features that have to evolve. > >>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc. > >>>>> We have bugs and issues that can be fixed in 2.x without breaking > >>>> backward > >>>>> compatibility. > >>>>> We have many users who are happy with the 2.x with all it’s issues. > >>>>> > >>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <a...@apache.org> > >>> написал(а): > >>>>>> > >>>>>> Alexey, > >>>>>> > >>>>>> Do we have any estimates of how fast we'll be able to gain > >>>>> production-ready > >>>>>> AI 3.0 in case of a "new repo" choice? > >>>>>> > >>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk < > >>>>> alexey.goncha...@gmail.com> > >>>>>> wrote: > >>>>>> > >>>>>>> Nikolay, > >>>>>>> > >>>>>>> What new features are we planning to implement for Ignite 2.x? I > >>> think > >>>>> once > >>>>>>> we commence working on Ignite 3.0, we should gradually cease the > >>>>> activity > >>>>>>> on Ignite 2.x to mere bugfixes because such parallel development > >>> will > >>>> be > >>>>>>> overwhelming regardless of how we choose to proceed. > >>>>>>> > >>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <nizhi...@apache.org > >>> : > >>>>>>> > >>>>>>>> To be clear: > >>>>>>>> > >>>>>>>>> I would suggest creating a new repository for Ignite 3.0 > >>> (perhaps, a > >>>>>>> new > >>>>>>>> clean branch, but a new repo looks nicer to me) and a new Ignite > >>> 3.0 > >>>>>>>> TeamCity project. > >>>>>>>> > >>>>>>>> +1 for new Team City project. > >>>>>>>> +1 for new branch for Ignite3. > >>>>>>>> -1 for new repo. > >>>>>>>> > >>>>>>>> > >>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov < > >> nizhikov....@gmail.com > >>>> > >>>>>>>> написал(а): > >>>>>>>>> > >>>>>>>>> Hello, Alexey. > >>>>>>>>> > >>>>>>>>> I think it will hurt our project more than help. > >>>>>>>>> Developing new features for 2 separate branches with the > >> different > >>>>> APIs > >>>>>>>> and internal structure is overwhelming > >>>>>>>>> > >>>>>>>>> Maybe we should relax a bit requirements for Ignite3? > >>>>>>>>> Maybe we should move step by step and make Ignite3 with new > >>>>>>>> configuration than Ignite4 with new transactions, etc? > >>>>>>>>> > >>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk < > >>>>>>> alexey.goncha...@gmail.com> > >>>>>>>> написал(а): > >>>>>>>>>> > >>>>>>>>>> Igniters, > >>>>>>>>>> > >>>>>>>>>> I wanted to pitch a rather radical idea regarding the Ignite > >> 3.0 > >>>>>>>>>> development which has occurred to me some time ago. > >>>>>>>>>> > >>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which imply > >>>> major > >>>>>>>>>> changes to the codebase (the change in replication protocol and > >>>> thus > >>>>>>>>>> transactions, change in binary format, updated metastorage, > >> etc). > >>>> We > >>>>>>>> also > >>>>>>>>>> planned significant changes in public APIs: configuration > >> format > >>>>>>> change, > >>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode rework. > >>> The > >>>>>>>> wishlist > >>>>>>>>>> of changes for Ignite 3.0 is huge. > >>>>>>>>>> > >>>>>>>>>> So, I was wondering whether it makes sense to try to change the > >>> old > >>>>>>>>>> codebase, or start with a new baseline and move old pieces of > >>> code > >>>>>>> that > >>>>>>>> do > >>>>>>>>>> not require significant rework. Personally, I would go with the > >>>>> second > >>>>>>>>>> option for the following reasons: > >>>>>>>>>> > >>>>>>>>>> - We have a chance to shift the development paradigm in the > >>> project > >>>>>>> and > >>>>>>>>>> introduce the practice of true unit-tests. In the new baseline > >> at > >>>> the > >>>>>>>>>> beginning there will be no ability to run an end-to-end > >> scenario, > >>>>>>> thus > >>>>>>>> we > >>>>>>>>>> will be forced to write unit-tests. So far, such practice was > >>> hard > >>>> to > >>>>>>>>>> implement because of tight coupling between Ignite components > >> and > >>>>>>>> inability > >>>>>>>>>> to instantiate components without an instance of KernalContext. > >>> For > >>>>>>>>>> example, we should be able to thoroughly test internal > >>> primitives, > >>>>>>>> such as > >>>>>>>>>> replication protocol (without actual communication), > >> distributed > >>>>>>>>>> metastorage contracts, persistence layer, etc. > >>>>>>>>>> - We will significantly reduce the development cycle in the > >>>> beginning > >>>>>>>>>> (right now the RunAll takes two hours of astronomical time with > >>>> empty > >>>>>>>> TC; > >>>>>>>>>> in the new approach developer will be able to run ALL tests > >>> locally > >>>>>>> in > >>>>>>>> a > >>>>>>>>>> matter of minutes) > >>>>>>>>>> - We can get rid of TC bot and enforce green TC by integrating > >> TC > >>>>>>> build > >>>>>>>>>> results with GitHub PRs (the same way Travis is currently > >>>> integrated > >>>>>>>> to PR > >>>>>>>>>> check). We should restrict PR merge without a TC check > >>>>>>>>>> - We will still have to re-write all tests, but only once. If > >> we > >>>> try > >>>>>>> to > >>>>>>>>>> modify the old codebase, we would need to modify all the tests > >>> for > >>>>>>>> every > >>>>>>>>>> major change (public API change, configuration change) > >>>>>>>>>> - We will have fewer conflicts when working together. For > >>> example, > >>>> I > >>>>>>>>>> cannot imagine how one would merge two changes of getting rid > >> of > >>>>>>>>>> IgniteFuture and changes in replication protocol, for example > >>>>>>>>>> > >>>>>>>>>> Technically, I would suggest creating a new repository for > >> Ignite > >>>> 3.0 > >>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer to me) > >>>> and a > >>>>>>>> new > >>>>>>>>>> Ignite 3.0 TeamCity project. > >>>>>>>>>> > >>>>>>>>>> While it may seem quite radical, I do believe that this > >> approach > >>>> will > >>>>>>>> give > >>>>>>>>>> us more benefits than trying to make such major changes in the > >>>>>>> existing > >>>>>>>>>> codebase. If needed, let's schedule a discord chat like before > >> to > >>>>>>>> discuss > >>>>>>>>>> this. > >>>>>>>>>> > >>>>>>>>>> WDYT? > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>> > >>> > >> > >> > >> -- > >> Best regards, > >> Andrey V. Mashenkov > >> > >
