Let's do this community discussion open. Here's the link on zoom call in Russian for Friday 6 PM: https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/
вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <nizhi...@apache.org>: > Time works for me. > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <alexey.goncha...@gmail.com> > написал(а): > > > > Nikolay, > > > > I am up for the call. I will try to explain my reasoning in greater > detail > > and will be glad to hear the concerns. Will this Friday, Nov 6th, work? > > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <nizhi...@apache.org>: > > > >> Igniters, should we have a call for this topic? > >> > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <ptupit...@apache.org> > >> написал(а): > >>> > >>>> not intend to rewrite everything from scratch > >>> > >>>> Every single test from Ignite 2.x should be moved to Ignite 3 > >>>> regardless of how we choose to proceed. > >>> > >>> Alexey, thank you for the explanation, this addresses all of my > concerns. > >>> > >>> > >>> > >>> > >>> > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov < > >> andrey.mashen...@gmail.com> > >>> wrote: > >>> > >>>> Hi, Igniters. > >>>> > >>>> * AFAIU, we need a new repo if we want to apply different restrictions > >> to > >>>> pull requests, > >>>> otherwise I see no difference for myself. > >>>> E.g. make static analysis (do we have?), compile, styles, and javadoc > >>>> checks mandatory. > >>>> > >>>> I think that relaxed requirements here will lead to bad product > quality. > >>>> > >>>> * Agree with Pavel, we should 'keep' integrations tests somehow. > >>>> During active development tests will be broken most of time, so, > >>>> I'd port them e.g. suite-by-suite once we will have a stable and > >> featured > >>>> environment to run them and of course make test's code clear and avoid > >>>> bad/non-relevant ones. > >>>> > >>>> * I like bottom-up approach. > >>>> With it we could make a better framework. I mean clear component > >> lifecycle, > >>>> component wiring mechanics, general methods to approach core > components > >>>> such as exchange/communication > >>>> to avoid code mess like we have in ExchangeFuture with all these > custom > >>>> callbacks for each component, interfaces like > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and > >>>> a pack of > >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected > >>>> and so on in various unexpected places. > >>>> Hope, we will be able to port most of the good code to the new > framework > >>>> version. > >>>> > >>>> > >>>> > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk < > >>>> alexey.goncha...@gmail.com> > >>>> wrote: > >>>> > >>>>> Nikolay, Pavel, > >>>>> > >>>>> Thanks for the feedback! First of all, I wanted to stress that I do > not > >>>>> intend to rewrite everything from scratch (I never used this phrase). > >>>> There > >>>>> are significant parts of code that would be moved with minimal > >>>>> modifications. Second, I never said that we will get rid of the old > >> tests > >>>>> codebase. Every single test from Ignite 2.x should be moved to > Ignite 3 > >>>>> regardless of how we choose to proceed. > >>>>> > >>>>> My point is that for some parts of the code a clean bottom-up > >>>>> implementation will be cheaper in many ways. Let me give you a few > >>>> concrete > >>>>> examples: > >>>>> > >>>>> - I think no one can object that we need a cleanly separated > >>>> persistence > >>>>> layer for Ignite. There is a very raw draft IEP for this already. On > >>>> the > >>>>> other hand, I think we also can agree that we need a split-brain > >>>>> resistant > >>>>> replication protocol for caches. There is also an IEP for this. > >>>> Neither > >>>>> of > >>>>> the changes is a good fit for 2.x because they are likely to > >> introduce > >>>>> breaking changes in the persistence layer, configuration and > >> behavior. > >>>>> Additionally, these components are now tightly coupled, so there is > >> no > >>>>> way > >>>>> these two changes can be implemented in parallel and then merged > >>>>> together > >>>>> easily. So what we will end up with is having to implement these > >>>> changes > >>>>> sequentially, fixing all existing tests twice, and essentially > >>>> throwing > >>>>> away half of the work done because the other part of the change is > >>>>> re-implemented > >>>>> - Similar example goes with getting rid of IgniteInternalFuture and > >>>>> replacing it with CompletableFuture, and any other change that > >> touches > >>>>> the > >>>>> asynchronous part of the code. > >>>>> > >>>>> Third, I do not see how this choice affects the UX of Ignite. The end > >>>> user > >>>>> experience must be fixed in the IEP regardless of the development > >> process > >>>>> and the fact that we have gaps in this area in Ignite 2.x just > confirms > >>>>> that. > >>>>> > >>>>> Pavel, agree that a repo/branch is a technicality, I guess if > >>>> reformulate, > >>>>> my point is that we might agree to have a single development master > >>>> branch > >>>>> with 'disassembled' end-to-end functionality for some period of time > to > >>>>> speed up development, and re-assemble the core features after having > >>>>> submodules tested independently. > >>>>> > >>>>> Nikolay, > >>>>>> We have many features that have to evolve. > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc. > >>>>> This is not very specific. In the end, resources are limited and we > >> will > >>>>> not be able to drive both tracks simultaneously, especially after a > >>>> couple > >>>>> of features having been implemented for Ignite 3.0. If there are > indeed > >>>>> some major changes that we want to do in Ignite 2.x instead of > putting > >>>>> effort into 3.0 - let's discuss them. I am just not aware of any, > >> that's > >>>>> why I am eager to move forward with Ignite 3.0. > >>>>> > >>>>>> We have bugs and issues that can be fixed in 2.x without breaking > >>>> backward > >>>>> compatibility. > >>>>>> We have many users who are happy with the 2.x with all it’s issues. > >>>>> These changes will be covered by end-to-end tests and migrated to > >> Ignite > >>>>> 3.0, so I see no issues here. > >>>>> > >>>>> Finally, Anton & Nikolay > >>>>> I do not have an estimate for this simply because the activity is > >>>>> community-driven and it depends on the number of people willing to > >>>>> contribute. With the current pace, I would hope to have an RC of > Ignite > >>>> 3.0 > >>>>> to be ready by the end of 2021. My gut feeling is that by moving with > >>>>> incremental changes, we will not be able to implement even half of > the > >>>>> wishlist by that time. > >>>>> I doubt that releasing several major releases with breaking changes > >> will > >>>>> make Ignite users happy either because each upgrade will cost Ignite > >>>> users > >>>>> money, so the fewer major versions we release, the better. Thus my > wish > >>>> to > >>>>> include all breaking changes in one release. > >>>>> > >>>>> I'll be now quiet for a while, let's see what other community members > >>>>> think. > >>>>> > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <ptupit...@apache.org>: > >>>>> > >>>>>> 1. Rewriting from scratch is never a good idea. > >>>>>> We don't want to follow the path of Netscape and lose all our users > >>>>>> by the time we have a working 3.0 [1] > >>>>>> > >>>>>> 2. Not sure about new repo - seems like some pain and no gain, > what's > >>>> the > >>>>>> problem with a branch? > >>>>>> > >>>>>> 3. We should keep existing integration tests when possible. > >>>>>> We have accumulated a lot of edge case knowledge over the years, > >>>>>> it is not a good idea to send all of that down the drain. > >>>>>> Yes, integration tests are slow, but they are the most valuable. > >>>>>> I think we can move more stuff into nightly runs and have a fast and > >>>>> modern > >>>>>> basic suite. > >>>>>> > >>>>>> > >>>>>> Alexey, you are much more familiar with the Ignite core codebase > than > >>>>> most > >>>>>> of us, > >>>>>> can you please explain in more detail which particular feature, in > >> your > >>>>>> opinion, > >>>>>> mandates this "start from scratch" approach? > >>>>>> Is it really not possible at all to follow a less radical way? > >>>>>> > >>>>>> > >>>>>> [1] > >>>>>> > >>>>>> > >>>>> > >>>> > >> > https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ > >>>>>> > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <nizhi...@apache.org > > > >>>>>> wrote: > >>>>>> > >>>>>>> Hello, Alexey. > >>>>>>> > >>>>>>> I think that «rewriting from scratch» approach has a high risk to > >>>> make > >>>>>> new > >>>>>>> features unusable. > >>>>>>> At the time Ignite2 was started no-one wants to do bad UX or bad > >>>>>> features. > >>>>>>> Nevertheless, it happen. > >>>>>>> > >>>>>>> I think we can avoid it with the Ignite3 and successors if we will > >>>> move > >>>>>>> step by step without keeping backward compatibility > >>>>>>> With the step by step approach, we can focus on each component > >>>>>> separately. > >>>>>>> > >>>>>>>> What new features are we planning to implement for Ignite 2.x? > >>>>>>> > >>>>>>> We have many features that have to evolve. > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc. > >>>>>>> We have bugs and issues that can be fixed in 2.x without breaking > >>>>>> backward > >>>>>>> compatibility. > >>>>>>> We have many users who are happy with the 2.x with all it’s issues. > >>>>>>> > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <a...@apache.org> > >>>>> написал(а): > >>>>>>>> > >>>>>>>> Alexey, > >>>>>>>> > >>>>>>>> Do we have any estimates of how fast we'll be able to gain > >>>>>>> production-ready > >>>>>>>> AI 3.0 in case of a "new repo" choice? > >>>>>>>> > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk < > >>>>>>> alexey.goncha...@gmail.com> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Nikolay, > >>>>>>>>> > >>>>>>>>> What new features are we planning to implement for Ignite 2.x? I > >>>>> think > >>>>>>> once > >>>>>>>>> we commence working on Ignite 3.0, we should gradually cease the > >>>>>>> activity > >>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel development > >>>>> will > >>>>>> be > >>>>>>>>> overwhelming regardless of how we choose to proceed. > >>>>>>>>> > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov < > nizhi...@apache.org > >>>>> : > >>>>>>>>> > >>>>>>>>>> To be clear: > >>>>>>>>>> > >>>>>>>>>>> I would suggest creating a new repository for Ignite 3.0 > >>>>> (perhaps, a > >>>>>>>>> new > >>>>>>>>>> clean branch, but a new repo looks nicer to me) and a new Ignite > >>>>> 3.0 > >>>>>>>>>> TeamCity project. > >>>>>>>>>> > >>>>>>>>>> +1 for new Team City project. > >>>>>>>>>> +1 for new branch for Ignite3. > >>>>>>>>>> -1 for new repo. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov < > >>>> nizhikov....@gmail.com > >>>>>> > >>>>>>>>>> написал(а): > >>>>>>>>>>> > >>>>>>>>>>> Hello, Alexey. > >>>>>>>>>>> > >>>>>>>>>>> I think it will hurt our project more than help. > >>>>>>>>>>> Developing new features for 2 separate branches with the > >>>> different > >>>>>>> APIs > >>>>>>>>>> and internal structure is overwhelming > >>>>>>>>>>> > >>>>>>>>>>> Maybe we should relax a bit requirements for Ignite3? > >>>>>>>>>>> Maybe we should move step by step and make Ignite3 with new > >>>>>>>>>> configuration than Ignite4 with new transactions, etc? > >>>>>>>>>>> > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk < > >>>>>>>>> alexey.goncha...@gmail.com> > >>>>>>>>>> написал(а): > >>>>>>>>>>>> > >>>>>>>>>>>> Igniters, > >>>>>>>>>>>> > >>>>>>>>>>>> I wanted to pitch a rather radical idea regarding the Ignite > >>>> 3.0 > >>>>>>>>>>>> development which has occurred to me some time ago. > >>>>>>>>>>>> > >>>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which > imply > >>>>>> major > >>>>>>>>>>>> changes to the codebase (the change in replication protocol > and > >>>>>> thus > >>>>>>>>>>>> transactions, change in binary format, updated metastorage, > >>>> etc). > >>>>>> We > >>>>>>>>>> also > >>>>>>>>>>>> planned significant changes in public APIs: configuration > >>>> format > >>>>>>>>> change, > >>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode rework. > >>>>> The > >>>>>>>>>> wishlist > >>>>>>>>>>>> of changes for Ignite 3.0 is huge. > >>>>>>>>>>>> > >>>>>>>>>>>> So, I was wondering whether it makes sense to try to change > the > >>>>> old > >>>>>>>>>>>> codebase, or start with a new baseline and move old pieces of > >>>>> code > >>>>>>>>> that > >>>>>>>>>> do > >>>>>>>>>>>> not require significant rework. Personally, I would go with > the > >>>>>>> second > >>>>>>>>>>>> option for the following reasons: > >>>>>>>>>>>> > >>>>>>>>>>>> - We have a chance to shift the development paradigm in the > >>>>> project > >>>>>>>>> and > >>>>>>>>>>>> introduce the practice of true unit-tests. In the new baseline > >>>> at > >>>>>> the > >>>>>>>>>>>> beginning there will be no ability to run an end-to-end > >>>> scenario, > >>>>>>>>> thus > >>>>>>>>>> we > >>>>>>>>>>>> will be forced to write unit-tests. So far, such practice was > >>>>> hard > >>>>>> to > >>>>>>>>>>>> implement because of tight coupling between Ignite components > >>>> and > >>>>>>>>>> inability > >>>>>>>>>>>> to instantiate components without an instance of > KernalContext. > >>>>> For > >>>>>>>>>>>> example, we should be able to thoroughly test internal > >>>>> primitives, > >>>>>>>>>> such as > >>>>>>>>>>>> replication protocol (without actual communication), > >>>> distributed > >>>>>>>>>>>> metastorage contracts, persistence layer, etc. > >>>>>>>>>>>> - We will significantly reduce the development cycle in the > >>>>>> beginning > >>>>>>>>>>>> (right now the RunAll takes two hours of astronomical time > with > >>>>>> empty > >>>>>>>>>> TC; > >>>>>>>>>>>> in the new approach developer will be able to run ALL tests > >>>>> locally > >>>>>>>>> in > >>>>>>>>>> a > >>>>>>>>>>>> matter of minutes) > >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by integrating > >>>> TC > >>>>>>>>> build > >>>>>>>>>>>> results with GitHub PRs (the same way Travis is currently > >>>>>> integrated > >>>>>>>>>> to PR > >>>>>>>>>>>> check). We should restrict PR merge without a TC check > >>>>>>>>>>>> - We will still have to re-write all tests, but only once. If > >>>> we > >>>>>> try > >>>>>>>>> to > >>>>>>>>>>>> modify the old codebase, we would need to modify all the tests > >>>>> for > >>>>>>>>>> every > >>>>>>>>>>>> major change (public API change, configuration change) > >>>>>>>>>>>> - We will have fewer conflicts when working together. For > >>>>> example, > >>>>>> I > >>>>>>>>>>>> cannot imagine how one would merge two changes of getting rid > >>>> of > >>>>>>>>>>>> IgniteFuture and changes in replication protocol, for example > >>>>>>>>>>>> > >>>>>>>>>>>> Technically, I would suggest creating a new repository for > >>>> Ignite > >>>>>> 3.0 > >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer to > me) > >>>>>> and a > >>>>>>>>>> new > >>>>>>>>>>>> Ignite 3.0 TeamCity project. > >>>>>>>>>>>> > >>>>>>>>>>>> While it may seem quite radical, I do believe that this > >>>> approach > >>>>>> will > >>>>>>>>>> give > >>>>>>>>>>>> us more benefits than trying to make such major changes in the > >>>>>>>>> existing > >>>>>>>>>>>> codebase. If needed, let's schedule a discord chat like before > >>>> to > >>>>>>>>>> discuss > >>>>>>>>>>>> this. > >>>>>>>>>>>> > >>>>>>>>>>>> WDYT? > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>>> > >>>> -- > >>>> Best regards, > >>>> Andrey V. Mashenkov > >>>> > >> > >> > >
