>> b. Implement IEP-61 - Common Replication Infrastructure I suppose, that this is the main cause of the current discussion. I hardly believe that this activity can be done without at least creating a completely new branch.
пт, 13 нояб. 2020 г. в 11:12, Nikolay Izhikov <nizhi...@apache.org>: > My suggestion: > > 1. Reduce Ignite3 scope to the following: > a. Delete all deprecated API and support of obsolete internal > protocols. > b. Implement IEP-61 - Common Replication Infrastructure > c. Implement new Ignite management tool ignitectl as suggested > during Ignite3 discussion. > > 2. Implement and release following improvements like transactions, Calcite > based SQL, etc in the ongoing releases - Ignite 4, 5, 6 > > My concern against separate Ignite 3 repo is the following: > > 1. We spread community to the two very separated part - Ignite3 developers > and Ignite2 maintainers. believe it’s bad for our community. > That can lead to the situation when we don’t fix critical or > blocker issueds «because they will not exists in Ignite3» > That will lead to the solutions never reviewed or reviewed poorly. > > 2. It seems for me that current scope of Ignite3 is too big to be > implemented in any reasonable time. > > > > 13 нояб. 2020 г., в 10:57, Nikolay Izhikov <nizhikov....@gmail.com> > написал(а): > > > > Hello, Valentin. > > > >> Nikolay, Maxim, are you OK with this route? > > > > -1 to have another repo for Ignite3 development. > > > >> 13 нояб. 2020 г., в 03:04, Valentin Kulichenko < > valentin.kuliche...@gmail.com> написал(а): > >> > >> Folks, > >> > >> We already have multiple IEPs for Ignite 3.0, and as far as I know, > there are contributors that would like to work on them (or probably already > started). That said, we should make a decision as soon as possible. > >> > >> At this point, it doesn't seem that there are any strong objections to > the technical side of things. So I would suggest the following: > >> > >> 1. Proceed with Alexey's approach to the development process, as it > seems to be the best (in my opinion - the only) way to address all the > technical concerns and issues expressed in the thread. We'll start by > creating a new repo and a new TC project. > >> 2. Start a separate discussion around transparency. If there are any > changes we need to make to our contributor guidelines, I am happy to talk > them through, but I don't think it's reasonable to delay feature > development because of this. In the short term, I will make sure that > everything that happens within the new repo is as open to the community as > possible. > >> > >> Nikolay, Maxim, are you OK with this route? > >> > >> -Val > >> > >> On Tue, Nov 10, 2020 at 4:55 PM Valentin Kulichenko < > valentin.kuliche...@gmail.com> wrote: > >> Maxim, > >> > >> 2.x and 3.x will have to coexist for some time - I don't see how we can > avoid this considering the set of proposed changes. That said, we > effectively will need to have two "masters" - one for each major version. > Master for 3.x can technically be a branch in the existing repo, but having > a separate repo seems cleaner, simply because it will not be a "branch" in > the traditional sense. > >> > >> Note that the new repo will still be under the Apache org, with the > same set of committers, managed by the community, etc. All the development > happening for 3.0 must follow the rules that we currently have (if > anything, it's an opportunity to improve those rules). > >> > >> As I said during the call on Friday, I strongly believe that if there > is a transparency issue, it will exist regardless of the approach we choose > for 3.0. If community members develop without IEPs or public discussions, > this will happen for both 2.x and 3.x unless we address this separately. I > don't see how this is related to Alexey's suggestion, which targets > *technical* issues with the product more than anything else. This a way to > achieve better modularity, introduce better coverage with unit tests, > reduce conflicts during development, etc. > >> > >> Coming back to transparency, let's identify the issues and fix them. It > probably makes sense to have a separate discussion on this topic. > >> > >> -Val > >> > >> On Tue, Nov 10, 2020 at 1:05 PM Maxim Muzafarov <mmu...@apache.org> > wrote: > >> Sergey, > >> > >> > >> Your summary makes sense to me. > >> > >> However, how we come up from *Development transparency* to *create a > >> separate public repository dedicated for 3.0*? For me *development > >> transparency* is about making changes in the master branch. These > >> changes will definitely be seen by all the Ignite developers. > >> > >> A dedicated public repository is technically public and visible for > >> everyone, but it allows development without IEPs, without public > >> discussion (since all the code changes are not related to the master > >> branch) it also allows a large number of assumptions and deviations > >> (like code-style violations). It also not about *development > >> transparency* since developers which are working on 3.0 is only a > >> subset of all Ignite developers which may continue working on 2.x. For > >> me, this would be a huge step backwards. > >> > >> Ignite veterans should remember how long the branch stabilization took > >> for the 2.x version with the PDS. > >> > >> > >> I think each breaking change should be passed through the master branch. > >> > >> On Tue, 10 Nov 2020 at 22:18, Alexei Scherbakov > >> <alexey.scherbak...@gmail.com> wrote: > >>> > >>> Makes sense to me. > >>> > >>> вт, 10 нояб. 2020 г. в 18:47, Sergey Chugunov < > sergey.chugu...@gmail.com>: > >>> > >>>> Igniters, > >>>> > >>>> I thought over Friday meeting ideas and concerns and summarized them > in > >>>> these three points: > >>>> > >>>> > >>>> 1. *Components design unification approach.* New proposed components > >>>> will be developed by different contributors, but they need to be > unified > >>>> and should integrate with each other easily. To ensure that I > suggest > >>>> calling an architecture group that will create design guidelines > for all > >>>> components and high-level overview of overall architecture. How > code is > >>>> split into components, what are component boundaries, how component > >>>> lifecycle works and what are its interfaces - all these and other > >>>> questions > >>>> should be covered. > >>>> > >>>> 2. *Scope management.* Apache 3.0 should be implemented within a > >>>> reasonable time, so we need some procedure to decide whether a > >>>> particular > >>>> feature should be dropped from the scope of 3.0 and postponed to 3.1 > >>>> release. To do so I suggest to range all features by two parameters: > >>>> criticality for 3.0 and amount of breaking changes. 3.0 scope should > >>>> include features of high criticality AND features with a big amount > of > >>>> breaking changes. All other features can be made optional. > >>>> > >>>> 3. *Development transparency.* Development of all components should > be > >>>> made as transparent for everyone as possible. Any contributor > should be > >>>> able to look over any component at any stage of development. To > achieve > >>>> this I suggest to create a separate public repository dedicated for > 3.0 > >>>> development. It will make the code available for everyone but when > >>>> development of 3.0 is done we won't loose any stars of our current > >>>> repository as we merge dev repo into main one and drop dev. > >>>> > >>>> Do these ideas make sense to you? Are there any concerns not covered > by > >>>> these suggestions? > >>>> > >>>> On Fri, Nov 6, 2020 at 7:36 PM Kseniya Romanova < > romanova.ks....@gmail.com > >>>>> > >>>> wrote: > >>>> > >>>>> Here are the slides from Alexey Goncharuk. Let's think this over and > >>>>> continue on Monday: > >>>>> > >>>>> > >>>> > https://go.gridgain.com/rs/491-TWR-806/images/Ignite_3_Plans_and_development_process.pdf > >>>>> > >>>>> чт, 5 нояб. 2020 г. в 11:13, Anton Vinogradov <a...@apache.org>: > >>>>> > >>>>>> Folks, > >>>>>> > >>>>>> Should we perform cleanup work before (r)evolutional changes? > >>>>>> My huge proposal is to get rid of things which we don't need anyway > >>>>>> - local caches, > >>>>>> - strange tx modes, > >>>>>> - code overcomplexity because of RollingUpgrade feature never > attended > >>>> at > >>>>>> AI, > >>>>>> - etc, > >>>>>> before choosing the way. > >>>>>> > >>>>>> On Tue, Nov 3, 2020 at 3:31 PM Valentin Kulichenko < > >>>>>> valentin.kuliche...@gmail.com> wrote: > >>>>>> > >>>>>>> Ksenia, thanks for scheduling this on such short notice! > >>>>>>> > >>>>>>> As for the original topic, I do support Alexey's idea. We're not > >>>> going > >>>>> to > >>>>>>> rewrite anything from scratch, as most of the components are going > to > >>>>> be > >>>>>>> moved as-is or with minimal modifications. However, the changes > that > >>>>> are > >>>>>>> proposed imply serious rework of the core parts of the code, which > >>>> are > >>>>>> not > >>>>>>> properly decoupled from each other and from other parts. This makes > >>>> the > >>>>>>> incremental approach borderline impossible. Developing in a new > repo, > >>>>>>> however, addresses this concern. As a bonus, we can also refactor > the > >>>>>> code, > >>>>>>> introduce better decoupling, get rid of kernel context, and develop > >>>>> unit > >>>>>>> tests (finally!). > >>>>>>> > >>>>>>> Basically, this proposal only affects the *process*, not the set of > >>>>>> changes > >>>>>>> we had discussed before. Ignite 3.0 is our unique chance to make > >>>> things > >>>>>>> right. > >>>>>>> > >>>>>>> -Val > >>>>>>> > >>>>>>> On Tue, Nov 3, 2020 at 3:06 AM Kseniya Romanova < > >>>>>> romanova.ks....@gmail.com > >>>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Pavel, all the interesting points will be anyway published here in > >>>>>>> English > >>>>>>>> (as the principal "if it's not on devlist it doesn't happened" is > >>>>> still > >>>>>>>> relevant). This is just a quick call for a group of developers. > >>>> Later > >>>>>> we > >>>>>>>> can do a separate presentation of idea and discussion in English > as > >>>>> we > >>>>>>> did > >>>>>>>> for the Ignite 3.0 draft of changes. > >>>>>>>> > >>>>>>>> вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <ptupit...@apache.org > >>>>> : > >>>>>>>> > >>>>>>>>> Kseniya, > >>>>>>>>> > >>>>>>>>> Thanks for scheduling this call. > >>>>>>>>> Do you think we can switch to English if non-Russian speaking > >>>>>> community > >>>>>>>>> members decide to join? > >>>>>>>>> > >>>>>>>>> On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova < > >>>>>>>> romanova.ks....@gmail.com > >>>>>>>>>> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Let's do this community discussion open. Here's the link on > >>>> zoom > >>>>>> call > >>>>>>>> in > >>>>>>>>>> Russian for Friday 6 PM: > >>>>>>>>>> > >>>>>> > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/ > >>>>>>>>>> > >>>>>>>>>> вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov < > >>>>> nizhi...@apache.org > >>>>>>> : > >>>>>>>>>> > >>>>>>>>>>> Time works for me. > >>>>>>>>>>> > >>>>>>>>>>>> 3 нояб. 2020 г., в 12:40, Alexey Goncharuk < > >>>>>>>>> alexey.goncha...@gmail.com > >>>>>>>>>>> > >>>>>>>>>>> написал(а): > >>>>>>>>>>>> > >>>>>>>>>>>> Nikolay, > >>>>>>>>>>>> > >>>>>>>>>>>> I am up for the call. I will try to explain my reasoning in > >>>>>>> greater > >>>>>>>>>>> detail > >>>>>>>>>>>> and will be glad to hear the concerns. Will this Friday, > >>>> Nov > >>>>>> 6th, > >>>>>>>>> work? > >>>>>>>>>>>> > >>>>>>>>>>>> вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov < > >>>>>>> nizhi...@apache.org > >>>>>>>>> : > >>>>>>>>>>>> > >>>>>>>>>>>>> Igniters, should we have a call for this topic? > >>>>>>>>>>>>> > >>>>>>>>>>>>>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn < > >>>>>> ptupit...@apache.org > >>>>>>>> > >>>>>>>>>>>>> написал(а): > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> not intend to rewrite everything from scratch > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Every single test from Ignite 2.x should be moved to > >>>>> Ignite > >>>>>> 3 > >>>>>>>>>>>>>>> regardless of how we choose to proceed. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Alexey, thank you for the explanation, this addresses all > >>>>> of > >>>>>> my > >>>>>>>>>>> concerns. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov < > >>>>>>>>>>>>> andrey.mashen...@gmail.com> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi, Igniters. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> * AFAIU, we need a new repo if we want to apply > >>>> different > >>>>>>>>>> restrictions > >>>>>>>>>>>>> to > >>>>>>>>>>>>>>> pull requests, > >>>>>>>>>>>>>>> otherwise I see no difference for myself. > >>>>>>>>>>>>>>> E.g. make static analysis (do we have?), compile, > >>>> styles, > >>>>>> and > >>>>>>>>>> javadoc > >>>>>>>>>>>>>>> checks mandatory. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I think that relaxed requirements here will lead to bad > >>>>>>> product > >>>>>>>>>>> quality. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> * Agree with Pavel, we should 'keep' integrations tests > >>>>>>> somehow. > >>>>>>>>>>>>>>> During active development tests will be broken most of > >>>>> time, > >>>>>>> so, > >>>>>>>>>>>>>>> I'd port them e.g. suite-by-suite once we will have a > >>>>> stable > >>>>>>> and > >>>>>>>>>>>>> featured > >>>>>>>>>>>>>>> environment to run them and of course make test's code > >>>>> clear > >>>>>>> and > >>>>>>>>>> avoid > >>>>>>>>>>>>>>> bad/non-relevant ones. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> * I like bottom-up approach. > >>>>>>>>>>>>>>> With it we could make a better framework. I mean clear > >>>>>>> component > >>>>>>>>>>>>> lifecycle, > >>>>>>>>>>>>>>> component wiring mechanics, general methods to approach > >>>>> core > >>>>>>>>>>> components > >>>>>>>>>>>>>>> such as exchange/communication > >>>>>>>>>>>>>>> to avoid code mess like we have in ExchangeFuture with > >>>> all > >>>>>>> these > >>>>>>>>>>> custom > >>>>>>>>>>>>>>> callbacks for each component, interfaces like > >>>>>>>>>>>>>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport > >>>>> and > >>>>>>>>>>>>>>> a pack of > >>>>>>>>>>>>> > >>>>>> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected > >>>>>>>>>>>>>>> and so on in various unexpected places. > >>>>>>>>>>>>>>> Hope, we will be able to port most of the good code to > >>>> the > >>>>>> new > >>>>>>>>>>> framework > >>>>>>>>>>>>>>> version. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk < > >>>>>>>>>>>>>>> alexey.goncha...@gmail.com> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Nikolay, Pavel, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks for the feedback! First of all, I wanted to > >>>> stress > >>>>>>> that > >>>>>>>> I > >>>>>>>>> do > >>>>>>>>>>> not > >>>>>>>>>>>>>>>> intend to rewrite everything from scratch (I never used > >>>>>> this > >>>>>>>>>> phrase). > >>>>>>>>>>>>>>> There > >>>>>>>>>>>>>>>> are significant parts of code that would be moved with > >>>>>>> minimal > >>>>>>>>>>>>>>>> modifications. Second, I never said that we will get > >>>> rid > >>>>> of > >>>>>>> the > >>>>>>>>> old > >>>>>>>>>>>>> tests > >>>>>>>>>>>>>>>> codebase. Every single test from Ignite 2.x should be > >>>>> moved > >>>>>>> to > >>>>>>>>>>> Ignite 3 > >>>>>>>>>>>>>>>> regardless of how we choose to proceed. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> My point is that for some parts of the code a clean > >>>>>> bottom-up > >>>>>>>>>>>>>>>> implementation will be cheaper in many ways. Let me > >>>> give > >>>>>> you > >>>>>>> a > >>>>>>>>> few > >>>>>>>>>>>>>>> concrete > >>>>>>>>>>>>>>>> examples: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> - I think no one can object that we need a cleanly > >>>>>> separated > >>>>>>>>>>>>>>> persistence > >>>>>>>>>>>>>>>> layer for Ignite. There is a very raw draft IEP for > >>>> this > >>>>>>>>> already. > >>>>>>>>>> On > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> other hand, I think we also can agree that we need a > >>>>>>>> split-brain > >>>>>>>>>>>>>>>> resistant > >>>>>>>>>>>>>>>> replication protocol for caches. There is also an IEP > >>>>> for > >>>>>>>> this. > >>>>>>>>>>>>>>> Neither > >>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>> the changes is a good fit for 2.x because they are > >>>>> likely > >>>>>> to > >>>>>>>>>>>>> introduce > >>>>>>>>>>>>>>>> breaking changes in the persistence layer, > >>>> configuration > >>>>>> and > >>>>>>>>>>>>> behavior. > >>>>>>>>>>>>>>>> Additionally, these components are now tightly > >>>> coupled, > >>>>> so > >>>>>>>> there > >>>>>>>>>> is > >>>>>>>>>>>>> no > >>>>>>>>>>>>>>>> way > >>>>>>>>>>>>>>>> these two changes can be implemented in parallel and > >>>>> then > >>>>>>>> merged > >>>>>>>>>>>>>>>> together > >>>>>>>>>>>>>>>> easily. So what we will end up with is having to > >>>>> implement > >>>>>>>> these > >>>>>>>>>>>>>>> changes > >>>>>>>>>>>>>>>> sequentially, fixing all existing tests twice, and > >>>>>>> essentially > >>>>>>>>>>>>>>> throwing > >>>>>>>>>>>>>>>> away half of the work done because the other part of > >>>> the > >>>>>>>> change > >>>>>>>>> is > >>>>>>>>>>>>>>>> re-implemented > >>>>>>>>>>>>>>>> - Similar example goes with getting rid of > >>>>>>>> IgniteInternalFuture > >>>>>>>>>> and > >>>>>>>>>>>>>>>> replacing it with CompletableFuture, and any other > >>>>> change > >>>>>>> that > >>>>>>>>>>>>> touches > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> asynchronous part of the code. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Third, I do not see how this choice affects the UX of > >>>>>> Ignite. > >>>>>>>> The > >>>>>>>>>> end > >>>>>>>>>>>>>>> user > >>>>>>>>>>>>>>>> experience must be fixed in the IEP regardless of the > >>>>>>>> development > >>>>>>>>>>>>> process > >>>>>>>>>>>>>>>> and the fact that we have gaps in this area in Ignite > >>>> 2.x > >>>>>>> just > >>>>>>>>>>> confirms > >>>>>>>>>>>>>>>> that. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Pavel, agree that a repo/branch is a technicality, I > >>>>> guess > >>>>>> if > >>>>>>>>>>>>>>> reformulate, > >>>>>>>>>>>>>>>> my point is that we might agree to have a single > >>>>>> development > >>>>>>>>> master > >>>>>>>>>>>>>>> branch > >>>>>>>>>>>>>>>> with 'disassembled' end-to-end functionality for some > >>>>>> period > >>>>>>> of > >>>>>>>>>> time > >>>>>>>>>>> to > >>>>>>>>>>>>>>>> speed up development, and re-assemble the core features > >>>>>> after > >>>>>>>>>> having > >>>>>>>>>>>>>>>> submodules tested independently. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Nikolay, > >>>>>>>>>>>>>>>>> We have many features that have to evolve. > >>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper > >>>>> support, > >>>>>>>> etc. > >>>>>>>>>>>>>>>> This is not very specific. In the end, resources are > >>>>>> limited > >>>>>>>> and > >>>>>>>>> we > >>>>>>>>>>>>> will > >>>>>>>>>>>>>>>> not be able to drive both tracks simultaneously, > >>>>> especially > >>>>>>>>> after a > >>>>>>>>>>>>>>> couple > >>>>>>>>>>>>>>>> of features having been implemented for Ignite 3.0. If > >>>>>> there > >>>>>>>> are > >>>>>>>>>>> indeed > >>>>>>>>>>>>>>>> some major changes that we want to do in Ignite 2.x > >>>>> instead > >>>>>>> of > >>>>>>>>>>> putting > >>>>>>>>>>>>>>>> effort into 3.0 - let's discuss them. I am just not > >>>> aware > >>>>>> of > >>>>>>>> any, > >>>>>>>>>>>>> that's > >>>>>>>>>>>>>>>> why I am eager to move forward with Ignite 3.0. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x > >>>> without > >>>>>>>>> breaking > >>>>>>>>>>>>>>> backward > >>>>>>>>>>>>>>>> compatibility. > >>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with all > >>>>>> it’s > >>>>>>>>>> issues. > >>>>>>>>>>>>>>>> These changes will be covered by end-to-end tests and > >>>>>>> migrated > >>>>>>>> to > >>>>>>>>>>>>> Ignite > >>>>>>>>>>>>>>>> 3.0, so I see no issues here. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Finally, Anton & Nikolay > >>>>>>>>>>>>>>>> I do not have an estimate for this simply because the > >>>>>>> activity > >>>>>>>> is > >>>>>>>>>>>>>>>> community-driven and it depends on the number of people > >>>>>>> willing > >>>>>>>>> to > >>>>>>>>>>>>>>>> contribute. With the current pace, I would hope to have > >>>>> an > >>>>>> RC > >>>>>>>> of > >>>>>>>>>>> Ignite > >>>>>>>>>>>>>>> 3.0 > >>>>>>>>>>>>>>>> to be ready by the end of 2021. My gut feeling is that > >>>> by > >>>>>>>> moving > >>>>>>>>>> with > >>>>>>>>>>>>>>>> incremental changes, we will not be able to implement > >>>>> even > >>>>>>> half > >>>>>>>>> of > >>>>>>>>>>> the > >>>>>>>>>>>>>>>> wishlist by that time. > >>>>>>>>>>>>>>>> I doubt that releasing several major releases with > >>>>> breaking > >>>>>>>>> changes > >>>>>>>>>>>>> will > >>>>>>>>>>>>>>>> make Ignite users happy either because each upgrade > >>>> will > >>>>>> cost > >>>>>>>>>> Ignite > >>>>>>>>>>>>>>> users > >>>>>>>>>>>>>>>> money, so the fewer major versions we release, the > >>>>> better. > >>>>>>> Thus > >>>>>>>>> my > >>>>>>>>>>> wish > >>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>> include all breaking changes in one release. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I'll be now quiet for a while, let's see what other > >>>>>> community > >>>>>>>>>> members > >>>>>>>>>>>>>>>> think. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn < > >>>>>>>>> ptupit...@apache.org > >>>>>>>>>>> : > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 1. Rewriting from scratch is never a good idea. > >>>>>>>>>>>>>>>>> We don't want to follow the path of Netscape and lose > >>>>> all > >>>>>>> our > >>>>>>>>>> users > >>>>>>>>>>>>>>>>> by the time we have a working 3.0 [1] > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 2. Not sure about new repo - seems like some pain and > >>>> no > >>>>>>> gain, > >>>>>>>>>>> what's > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> problem with a branch? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> 3. We should keep existing integration tests when > >>>>>> possible. > >>>>>>>>>>>>>>>>> We have accumulated a lot of edge case knowledge over > >>>>> the > >>>>>>>> years, > >>>>>>>>>>>>>>>>> it is not a good idea to send all of that down the > >>>>> drain. > >>>>>>>>>>>>>>>>> Yes, integration tests are slow, but they are the most > >>>>>>>> valuable. > >>>>>>>>>>>>>>>>> I think we can move more stuff into nightly runs and > >>>>> have > >>>>>> a > >>>>>>>> fast > >>>>>>>>>> and > >>>>>>>>>>>>>>>> modern > >>>>>>>>>>>>>>>>> basic suite. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Alexey, you are much more familiar with the Ignite > >>>> core > >>>>>>>> codebase > >>>>>>>>>>> than > >>>>>>>>>>>>>>>> most > >>>>>>>>>>>>>>>>> of us, > >>>>>>>>>>>>>>>>> can you please explain in more detail which particular > >>>>>>>> feature, > >>>>>>>>> in > >>>>>>>>>>>>> your > >>>>>>>>>>>>>>>>> opinion, > >>>>>>>>>>>>>>>>> mandates this "start from scratch" approach? > >>>>>>>>>>>>>>>>> Is it really not possible at all to follow a less > >>>>> radical > >>>>>>> way? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov < > >>>>>>>>>> nizhi...@apache.org > >>>>>>>>>>>> > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Hello, Alexey. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I think that «rewriting from scratch» approach has a > >>>>> high > >>>>>>>> risk > >>>>>>>>> to > >>>>>>>>>>>>>>> make > >>>>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>>> features unusable. > >>>>>>>>>>>>>>>>>> At the time Ignite2 was started no-one wants to do > >>>> bad > >>>>> UX > >>>>>>> or > >>>>>>>>> bad > >>>>>>>>>>>>>>>>> features. > >>>>>>>>>>>>>>>>>> Nevertheless, it happen. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I think we can avoid it with the Ignite3 and > >>>> successors > >>>>>> if > >>>>>>> we > >>>>>>>>>> will > >>>>>>>>>>>>>>> move > >>>>>>>>>>>>>>>>>> step by step without keeping backward compatibility > >>>>>>>>>>>>>>>>>> With the step by step approach, we can focus on each > >>>>>>>> component > >>>>>>>>>>>>>>>>> separately. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> What new features are we planning to implement for > >>>>>> Ignite > >>>>>>>> 2.x? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> We have many features that have to evolve. > >>>>>>>>>>>>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper > >>>>>> support, > >>>>>>>> etc. > >>>>>>>>>>>>>>>>>> We have bugs and issues that can be fixed in 2.x > >>>>> without > >>>>>>>>> breaking > >>>>>>>>>>>>>>>>> backward > >>>>>>>>>>>>>>>>>> compatibility. > >>>>>>>>>>>>>>>>>> We have many users who are happy with the 2.x with > >>>> all > >>>>>> it’s > >>>>>>>>>> issues. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov < > >>>>>> a...@apache.org > >>>>>>>> > >>>>>>>>>>>>>>>> написал(а): > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Alexey, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Do we have any estimates of how fast we'll be able > >>>> to > >>>>>> gain > >>>>>>>>>>>>>>>>>> production-ready > >>>>>>>>>>>>>>>>>>> AI 3.0 in case of a "new repo" choice? > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk < > >>>>>>>>>>>>>>>>>> alexey.goncha...@gmail.com> > >>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Nikolay, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> What new features are we planning to implement for > >>>>>> Ignite > >>>>>>>>> 2.x? > >>>>>>>>>> I > >>>>>>>>>>>>>>>> think > >>>>>>>>>>>>>>>>>> once > >>>>>>>>>>>>>>>>>>>> we commence working on Ignite 3.0, we should > >>>>> gradually > >>>>>>>> cease > >>>>>>>>>> the > >>>>>>>>>>>>>>>>>> activity > >>>>>>>>>>>>>>>>>>>> on Ignite 2.x to mere bugfixes because such > >>>> parallel > >>>>>>>>>> development > >>>>>>>>>>>>>>>> will > >>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>> overwhelming regardless of how we choose to > >>>> proceed. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov < > >>>>>>>>>>> nizhi...@apache.org > >>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> To be clear: > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> I would suggest creating a new repository for > >>>>> Ignite > >>>>>>> 3.0 > >>>>>>>>>>>>>>>> (perhaps, a > >>>>>>>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>>>>>> clean branch, but a new repo looks nicer to me) > >>>> and > >>>>> a > >>>>>>> new > >>>>>>>>>> Ignite > >>>>>>>>>>>>>>>> 3.0 > >>>>>>>>>>>>>>>>>>>>> TeamCity project. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> +1 for new Team City project. > >>>>>>>>>>>>>>>>>>>>> +1 for new branch for Ignite3. > >>>>>>>>>>>>>>>>>>>>> -1 for new repo. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov < > >>>>>>>>>>>>>>> nizhikov....@gmail.com > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> написал(а): > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Hello, Alexey. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> I think it will hurt our project more than help. > >>>>>>>>>>>>>>>>>>>>>> Developing new features for 2 separate branches > >>>>> with > >>>>>>> the > >>>>>>>>>>>>>>> different > >>>>>>>>>>>>>>>>>> APIs > >>>>>>>>>>>>>>>>>>>>> and internal structure is overwhelming > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Maybe we should relax a bit requirements for > >>>>> Ignite3? > >>>>>>>>>>>>>>>>>>>>>> Maybe we should move step by step and make > >>>> Ignite3 > >>>>>> with > >>>>>>>> new > >>>>>>>>>>>>>>>>>>>>> configuration than Ignite4 with new transactions, > >>>>> etc? > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk < > >>>>>>>>>>>>>>>>>>>> alexey.goncha...@gmail.com> > >>>>>>>>>>>>>>>>>>>>> написал(а): > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Igniters, > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> I wanted to pitch a rather radical idea > >>>> regarding > >>>>>> the > >>>>>>>>> Ignite > >>>>>>>>>>>>>>> 3.0 > >>>>>>>>>>>>>>>>>>>>>>> development which has occurred to me some time > >>>>> ago. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> We already have several IEPs targeted to Ignite > >>>>> 3.0 > >>>>>>>> which > >>>>>>>>>>> imply > >>>>>>>>>>>>>>>>> major > >>>>>>>>>>>>>>>>>>>>>>> changes to the codebase (the change in > >>>> replication > >>>>>>>>> protocol > >>>>>>>>>>> and > >>>>>>>>>>>>>>>>> thus > >>>>>>>>>>>>>>>>>>>>>>> transactions, change in binary format, updated > >>>>>>>>> metastorage, > >>>>>>>>>>>>>>> etc). > >>>>>>>>>>>>>>>>> We > >>>>>>>>>>>>>>>>>>>>> also > >>>>>>>>>>>>>>>>>>>>>>> planned significant changes in public APIs: > >>>>>>>> configuration > >>>>>>>>>>>>>>> format > >>>>>>>>>>>>>>>>>>>> change, > >>>>>>>>>>>>>>>>>>>>>>> improvements in cache APIs, SQL APIs, > >>>> transaction > >>>>>> mode > >>>>>>>>>> rework. > >>>>>>>>>>>>>>>> The > >>>>>>>>>>>>>>>>>>>>> wishlist > >>>>>>>>>>>>>>>>>>>>>>> of changes for Ignite 3.0 is huge. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> So, I was wondering whether it makes sense to > >>>> try > >>>>> to > >>>>>>>>> change > >>>>>>>>>>> the > >>>>>>>>>>>>>>>> old > >>>>>>>>>>>>>>>>>>>>>>> codebase, or start with a new baseline and move > >>>>> old > >>>>>>>> pieces > >>>>>>>>>> of > >>>>>>>>>>>>>>>> code > >>>>>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>>> do > >>>>>>>>>>>>>>>>>>>>>>> not require significant rework. Personally, I > >>>>> would > >>>>>> go > >>>>>>>>> with > >>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> second > >>>>>>>>>>>>>>>>>>>>>>> option for the following reasons: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> - We have a chance to shift the development > >>>>> paradigm > >>>>>>> in > >>>>>>>>> the > >>>>>>>>>>>>>>>> project > >>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>>> introduce the practice of true unit-tests. In > >>>> the > >>>>>> new > >>>>>>>>>> baseline > >>>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>> beginning there will be no ability to run an > >>>>>>> end-to-end > >>>>>>>>>>>>>>> scenario, > >>>>>>>>>>>>>>>>>>>> thus > >>>>>>>>>>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>>>>>> will be forced to write unit-tests. So far, such > >>>>>>>> practice > >>>>>>>>>> was > >>>>>>>>>>>>>>>> hard > >>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>> implement because of tight coupling between > >>>> Ignite > >>>>>>>>>> components > >>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>> inability > >>>>>>>>>>>>>>>>>>>>>>> to instantiate components without an instance of > >>>>>>>>>>> KernalContext. > >>>>>>>>>>>>>>>> For > >>>>>>>>>>>>>>>>>>>>>>> example, we should be able to thoroughly test > >>>>>> internal > >>>>>>>>>>>>>>>> primitives, > >>>>>>>>>>>>>>>>>>>>> such as > >>>>>>>>>>>>>>>>>>>>>>> replication protocol (without actual > >>>>> communication), > >>>>>>>>>>>>>>> distributed > >>>>>>>>>>>>>>>>>>>>>>> metastorage contracts, persistence layer, etc. > >>>>>>>>>>>>>>>>>>>>>>> - We will significantly reduce the development > >>>>> cycle > >>>>>>> in > >>>>>>>>> the > >>>>>>>>>>>>>>>>> beginning > >>>>>>>>>>>>>>>>>>>>>>> (right now the RunAll takes two hours of > >>>>>> astronomical > >>>>>>>> time > >>>>>>>>>>> with > >>>>>>>>>>>>>>>>> empty > >>>>>>>>>>>>>>>>>>>>> TC; > >>>>>>>>>>>>>>>>>>>>>>> in the new approach developer will be able to > >>>> run > >>>>>> ALL > >>>>>>>>> tests > >>>>>>>>>>>>>>>> locally > >>>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>>> matter of minutes) > >>>>>>>>>>>>>>>>>>>>>>> - We can get rid of TC bot and enforce green TC > >>>> by > >>>>>>>>>> integrating > >>>>>>>>>>>>>>> TC > >>>>>>>>>>>>>>>>>>>> build > >>>>>>>>>>>>>>>>>>>>>>> results with GitHub PRs (the same way Travis is > >>>>>>>> currently > >>>>>>>>>>>>>>>>> integrated > >>>>>>>>>>>>>>>>>>>>> to PR > >>>>>>>>>>>>>>>>>>>>>>> check). We should restrict PR merge without a TC > >>>>>> check > >>>>>>>>>>>>>>>>>>>>>>> - We will still have to re-write all tests, but > >>>>> only > >>>>>>>> once. > >>>>>>>>>> If > >>>>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>> try > >>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>> modify the old codebase, we would need to modify > >>>>> all > >>>>>>> the > >>>>>>>>>> tests > >>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>> every > >>>>>>>>>>>>>>>>>>>>>>> major change (public API change, configuration > >>>>>> change) > >>>>>>>>>>>>>>>>>>>>>>> - We will have fewer conflicts when working > >>>>>> together. > >>>>>>>> For > >>>>>>>>>>>>>>>> example, > >>>>>>>>>>>>>>>>> I > >>>>>>>>>>>>>>>>>>>>>>> cannot imagine how one would merge two changes > >>>> of > >>>>>>>> getting > >>>>>>>>>> rid > >>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>> IgniteFuture and changes in replication > >>>> protocol, > >>>>>> for > >>>>>>>>>> example > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Technically, I would suggest creating a new > >>>>>> repository > >>>>>>>> for > >>>>>>>>>>>>>>> Ignite > >>>>>>>>>>>>>>>>> 3.0 > >>>>>>>>>>>>>>>>>>>>>>> (perhaps, a new clean branch, but a new repo > >>>> looks > >>>>>>> nicer > >>>>>>>>> to > >>>>>>>>>>> me) > >>>>>>>>>>>>>>>>> and a > >>>>>>>>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>>>>>>>> Ignite 3.0 TeamCity project. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> While it may seem quite radical, I do believe > >>>> that > >>>>>>> this > >>>>>>>>>>>>>>> approach > >>>>>>>>>>>>>>>>> will > >>>>>>>>>>>>>>>>>>>>> give > >>>>>>>>>>>>>>>>>>>>>>> us more benefits than trying to make such major > >>>>>>> changes > >>>>>>>> in > >>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>> existing > >>>>>>>>>>>>>>>>>>>>>>> codebase. If needed, let's schedule a discord > >>>> chat > >>>>>>> like > >>>>>>>>>> before > >>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>> discuss > >>>>>>>>>>>>>>>>>>>>>>> this. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> WDYT? > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>> Best regards, > >>>>>>>>>>>>>>> Andrey V. Mashenkov > >>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >>> > >>> -- > >>> > >>> Best regards, > >>> Alexei Scherbakov > > > > > -- Sincerely yours, Ivan Daschinskiy
