> not intend to rewrite everything from scratch > Every single test from Ignite 2.x should be moved to Ignite 3 > regardless of how we choose to proceed.
Alexey, thank you for the explanation, this addresses all of my concerns. On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <andrey.mashen...@gmail.com> wrote: > Hi, Igniters. > > * AFAIU, we need a new repo if we want to apply different restrictions to > pull requests, > otherwise I see no difference for myself. > E.g. make static analysis (do we have?), compile, styles, and javadoc > checks mandatory. > > I think that relaxed requirements here will lead to bad product quality. > > * Agree with Pavel, we should 'keep' integrations tests somehow. > During active development tests will be broken most of time, so, > I'd port them e.g. suite-by-suite once we will have a stable and featured > environment to run them and of course make test's code clear and avoid > bad/non-relevant ones. > > * I like bottom-up approach. > With it we could make a better framework. I mean clear component lifecycle, > component wiring mechanics, general methods to approach core components > such as exchange/communication > to avoid code mess like we have in ExchangeFuture with all these custom > callbacks for each component, interfaces like > PartitionsExchangeAware, IgniteChangeGlobalStateSupport and > a pack of start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected > and so on in various unexpected places. > Hope, we will be able to port most of the good code to the new framework > version. > > > > On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk < > alexey.goncha...@gmail.com> > wrote: > > > Nikolay, Pavel, > > > > Thanks for the feedback! First of all, I wanted to stress that I do not > > intend to rewrite everything from scratch (I never used this phrase). > There > > are significant parts of code that would be moved with minimal > > modifications. Second, I never said that we will get rid of the old tests > > codebase. Every single test from Ignite 2.x should be moved to Ignite 3 > > regardless of how we choose to proceed. > > > > My point is that for some parts of the code a clean bottom-up > > implementation will be cheaper in many ways. Let me give you a few > concrete > > examples: > > > > - I think no one can object that we need a cleanly separated > persistence > > layer for Ignite. There is a very raw draft IEP for this already. On > the > > other hand, I think we also can agree that we need a split-brain > > resistant > > replication protocol for caches. There is also an IEP for this. > Neither > > of > > the changes is a good fit for 2.x because they are likely to introduce > > breaking changes in the persistence layer, configuration and behavior. > > Additionally, these components are now tightly coupled, so there is no > > way > > these two changes can be implemented in parallel and then merged > > together > > easily. So what we will end up with is having to implement these > changes > > sequentially, fixing all existing tests twice, and essentially > throwing > > away half of the work done because the other part of the change is > > re-implemented > > - Similar example goes with getting rid of IgniteInternalFuture and > > replacing it with CompletableFuture, and any other change that touches > > the > > asynchronous part of the code. > > > > Third, I do not see how this choice affects the UX of Ignite. The end > user > > experience must be fixed in the IEP regardless of the development process > > and the fact that we have gaps in this area in Ignite 2.x just confirms > > that. > > > > Pavel, agree that a repo/branch is a technicality, I guess if > reformulate, > > my point is that we might agree to have a single development master > branch > > with 'disassembled' end-to-end functionality for some period of time to > > speed up development, and re-assemble the core features after having > > submodules tested independently. > > > > Nikolay, > > >We have many features that have to evolve. > > >Snapshots, rebalance, tooling, tracing, zookeeper support, etc. > > This is not very specific. In the end, resources are limited and we will > > not be able to drive both tracks simultaneously, especially after a > couple > > of features having been implemented for Ignite 3.0. If there are indeed > > some major changes that we want to do in Ignite 2.x instead of putting > > effort into 3.0 - let's discuss them. I am just not aware of any, that's > > why I am eager to move forward with Ignite 3.0. > > > > >We have bugs and issues that can be fixed in 2.x without breaking > backward > > compatibility. > > >We have many users who are happy with the 2.x with all it’s issues. > > These changes will be covered by end-to-end tests and migrated to Ignite > > 3.0, so I see no issues here. > > > > Finally, Anton & Nikolay > > I do not have an estimate for this simply because the activity is > > community-driven and it depends on the number of people willing to > > contribute. With the current pace, I would hope to have an RC of Ignite > 3.0 > > to be ready by the end of 2021. My gut feeling is that by moving with > > incremental changes, we will not be able to implement even half of the > > wishlist by that time. > > I doubt that releasing several major releases with breaking changes will > > make Ignite users happy either because each upgrade will cost Ignite > users > > money, so the fewer major versions we release, the better. Thus my wish > to > > include all breaking changes in one release. > > > > I'll be now quiet for a while, let's see what other community members > > think. > > > > пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <ptupit...@apache.org>: > > > > > 1. Rewriting from scratch is never a good idea. > > > We don't want to follow the path of Netscape and lose all our users > > > by the time we have a working 3.0 [1] > > > > > > 2. Not sure about new repo - seems like some pain and no gain, what's > the > > > problem with a branch? > > > > > > 3. We should keep existing integration tests when possible. > > > We have accumulated a lot of edge case knowledge over the years, > > > it is not a good idea to send all of that down the drain. > > > Yes, integration tests are slow, but they are the most valuable. > > > I think we can move more stuff into nightly runs and have a fast and > > modern > > > basic suite. > > > > > > > > > Alexey, you are much more familiar with the Ignite core codebase than > > most > > > of us, > > > can you please explain in more detail which particular feature, in your > > > opinion, > > > mandates this "start from scratch" approach? > > > Is it really not possible at all to follow a less radical way? > > > > > > > > > [1] > > > > > > > > > https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ > > > > > > On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <nizhi...@apache.org> > > > wrote: > > > > > > > Hello, Alexey. > > > > > > > > I think that «rewriting from scratch» approach has a high risk to > make > > > new > > > > features unusable. > > > > At the time Ignite2 was started no-one wants to do bad UX or bad > > > features. > > > > Nevertheless, it happen. > > > > > > > > I think we can avoid it with the Ignite3 and successors if we will > move > > > > step by step without keeping backward compatibility > > > > With the step by step approach, we can focus on each component > > > separately. > > > > > > > > > What new features are we planning to implement for Ignite 2.x? > > > > > > > > We have many features that have to evolve. > > > > Snapshots, rebalance, tooling, tracing, zookeeper support, etc. > > > > We have bugs and issues that can be fixed in 2.x without breaking > > > backward > > > > compatibility. > > > > We have many users who are happy with the 2.x with all it’s issues. > > > > > > > > > 2 нояб. 2020 г., в 14:09, Anton Vinogradov <a...@apache.org> > > написал(а): > > > > > > > > > > Alexey, > > > > > > > > > > Do we have any estimates of how fast we'll be able to gain > > > > production-ready > > > > > AI 3.0 in case of a "new repo" choice? > > > > > > > > > > On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk < > > > > alexey.goncha...@gmail.com> > > > > > wrote: > > > > > > > > > >> Nikolay, > > > > >> > > > > >> What new features are we planning to implement for Ignite 2.x? I > > think > > > > once > > > > >> we commence working on Ignite 3.0, we should gradually cease the > > > > activity > > > > >> on Ignite 2.x to mere bugfixes because such parallel development > > will > > > be > > > > >> overwhelming regardless of how we choose to proceed. > > > > >> > > > > >> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <nizhi...@apache.org > >: > > > > >> > > > > >>> To be clear: > > > > >>> > > > > >>>> I would suggest creating a new repository for Ignite 3.0 > > (perhaps, a > > > > >> new > > > > >>> clean branch, but a new repo looks nicer to me) and a new Ignite > > 3.0 > > > > >>> TeamCity project. > > > > >>> > > > > >>> +1 for new Team City project. > > > > >>> +1 for new branch for Ignite3. > > > > >>> -1 for new repo. > > > > >>> > > > > >>> > > > > >>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov < > nizhikov....@gmail.com > > > > > > > >>> написал(а): > > > > >>>> > > > > >>>> Hello, Alexey. > > > > >>>> > > > > >>>> I think it will hurt our project more than help. > > > > >>>> Developing new features for 2 separate branches with the > different > > > > APIs > > > > >>> and internal structure is overwhelming > > > > >>>> > > > > >>>> Maybe we should relax a bit requirements for Ignite3? > > > > >>>> Maybe we should move step by step and make Ignite3 with new > > > > >>> configuration than Ignite4 with new transactions, etc? > > > > >>>> > > > > >>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk < > > > > >> alexey.goncha...@gmail.com> > > > > >>> написал(а): > > > > >>>>> > > > > >>>>> Igniters, > > > > >>>>> > > > > >>>>> I wanted to pitch a rather radical idea regarding the Ignite > 3.0 > > > > >>>>> development which has occurred to me some time ago. > > > > >>>>> > > > > >>>>> We already have several IEPs targeted to Ignite 3.0 which imply > > > major > > > > >>>>> changes to the codebase (the change in replication protocol and > > > thus > > > > >>>>> transactions, change in binary format, updated metastorage, > etc). > > > We > > > > >>> also > > > > >>>>> planned significant changes in public APIs: configuration > format > > > > >> change, > > > > >>>>> improvements in cache APIs, SQL APIs, transaction mode rework. > > The > > > > >>> wishlist > > > > >>>>> of changes for Ignite 3.0 is huge. > > > > >>>>> > > > > >>>>> So, I was wondering whether it makes sense to try to change the > > old > > > > >>>>> codebase, or start with a new baseline and move old pieces of > > code > > > > >> that > > > > >>> do > > > > >>>>> not require significant rework. Personally, I would go with the > > > > second > > > > >>>>> option for the following reasons: > > > > >>>>> > > > > >>>>> - We have a chance to shift the development paradigm in the > > project > > > > >> and > > > > >>>>> introduce the practice of true unit-tests. In the new baseline > at > > > the > > > > >>>>> beginning there will be no ability to run an end-to-end > scenario, > > > > >> thus > > > > >>> we > > > > >>>>> will be forced to write unit-tests. So far, such practice was > > hard > > > to > > > > >>>>> implement because of tight coupling between Ignite components > and > > > > >>> inability > > > > >>>>> to instantiate components without an instance of KernalContext. > > For > > > > >>>>> example, we should be able to thoroughly test internal > > primitives, > > > > >>> such as > > > > >>>>> replication protocol (without actual communication), > distributed > > > > >>>>> metastorage contracts, persistence layer, etc. > > > > >>>>> - We will significantly reduce the development cycle in the > > > beginning > > > > >>>>> (right now the RunAll takes two hours of astronomical time with > > > empty > > > > >>> TC; > > > > >>>>> in the new approach developer will be able to run ALL tests > > locally > > > > >> in > > > > >>> a > > > > >>>>> matter of minutes) > > > > >>>>> - We can get rid of TC bot and enforce green TC by integrating > TC > > > > >> build > > > > >>>>> results with GitHub PRs (the same way Travis is currently > > > integrated > > > > >>> to PR > > > > >>>>> check). We should restrict PR merge without a TC check > > > > >>>>> - We will still have to re-write all tests, but only once. If > we > > > try > > > > >> to > > > > >>>>> modify the old codebase, we would need to modify all the tests > > for > > > > >>> every > > > > >>>>> major change (public API change, configuration change) > > > > >>>>> - We will have fewer conflicts when working together. For > > example, > > > I > > > > >>>>> cannot imagine how one would merge two changes of getting rid > of > > > > >>>>> IgniteFuture and changes in replication protocol, for example > > > > >>>>> > > > > >>>>> Technically, I would suggest creating a new repository for > Ignite > > > 3.0 > > > > >>>>> (perhaps, a new clean branch, but a new repo looks nicer to me) > > > and a > > > > >>> new > > > > >>>>> Ignite 3.0 TeamCity project. > > > > >>>>> > > > > >>>>> While it may seem quite radical, I do believe that this > approach > > > will > > > > >>> give > > > > >>>>> us more benefits than trying to make such major changes in the > > > > >> existing > > > > >>>>> codebase. If needed, let's schedule a discord chat like before > to > > > > >>> discuss > > > > >>>>> this. > > > > >>>>> > > > > >>>>> WDYT? > > > > >>>> > > > > >>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > > -- > Best regards, > Andrey V. Mashenkov >
