Hello! In my opinion, what you are actually proposing is writing a new IMDG/distributed database.
I'm not sure why one would assume that this new product will be particularly successful with users. We have some very good developers out there now, but some of the people who actually wrote Ignite 2.x are no longer with us. Maybe it will be a new and good IMDG, maybe not. Time has passed, the bar is now higher than it was before. People might as well consider moving to a different IMDG not called Ignite, than moving to a different, experimental IMDG named Ignite 3.0. Maybe there are some bad parts that we have (such as persistence format/separation, split-brain protocol, etc), but we have 95 out of 100 parts which are very good and battle tested, and I would prefer them to be kept intact instead of cherry-picking them one by one into the new grid. We have very good distributed compute/peer class loading support, people don't realize how good it is, but they will notice if it is gone/rewritten from scratch to "just ok" state instead of "very good". I think that we should just do a deep refactoring/API change in 3.0, and leave the rest of stuff to 3.x or 4.0. I also happen to think that the code we already have is very valuable, and it would be a pity to see it replaced with cleaner, worse offering. Regards, -- Ilya Kasnacheev пн, 2 нояб. 2020 г. в 18:18, Alexey Goncharuk <alexey.goncha...@gmail.com>: > Nikolay, Pavel, > > Thanks for the feedback! First of all, I wanted to stress that I do not > intend to rewrite everything from scratch (I never used this phrase). There > are significant parts of code that would be moved with minimal > modifications. Second, I never said that we will get rid of the old tests > codebase. Every single test from Ignite 2.x should be moved to Ignite 3 > regardless of how we choose to proceed. > > My point is that for some parts of the code a clean bottom-up > implementation will be cheaper in many ways. Let me give you a few concrete > examples: > > - I think no one can object that we need a cleanly separated persistence > layer for Ignite. There is a very raw draft IEP for this already. On the > other hand, I think we also can agree that we need a split-brain > resistant > replication protocol for caches. There is also an IEP for this. Neither > of > the changes is a good fit for 2.x because they are likely to introduce > breaking changes in the persistence layer, configuration and behavior. > Additionally, these components are now tightly coupled, so there is no > way > these two changes can be implemented in parallel and then merged > together > easily. So what we will end up with is having to implement these changes > sequentially, fixing all existing tests twice, and essentially throwing > away half of the work done because the other part of the change is > re-implemented > - Similar example goes with getting rid of IgniteInternalFuture and > replacing it with CompletableFuture, and any other change that touches > the > asynchronous part of the code. > > Third, I do not see how this choice affects the UX of Ignite. The end user > experience must be fixed in the IEP regardless of the development process > and the fact that we have gaps in this area in Ignite 2.x just confirms > that. > > Pavel, agree that a repo/branch is a technicality, I guess if reformulate, > my point is that we might agree to have a single development master branch > with 'disassembled' end-to-end functionality for some period of time to > speed up development, and re-assemble the core features after having > submodules tested independently. > > Nikolay, > >We have many features that have to evolve. > >Snapshots, rebalance, tooling, tracing, zookeeper support, etc. > This is not very specific. In the end, resources are limited and we will > not be able to drive both tracks simultaneously, especially after a couple > of features having been implemented for Ignite 3.0. If there are indeed > some major changes that we want to do in Ignite 2.x instead of putting > effort into 3.0 - let's discuss them. I am just not aware of any, that's > why I am eager to move forward with Ignite 3.0. > > >We have bugs and issues that can be fixed in 2.x without breaking backward > compatibility. > >We have many users who are happy with the 2.x with all it’s issues. > These changes will be covered by end-to-end tests and migrated to Ignite > 3.0, so I see no issues here. > > Finally, Anton & Nikolay > I do not have an estimate for this simply because the activity is > community-driven and it depends on the number of people willing to > contribute. With the current pace, I would hope to have an RC of Ignite 3.0 > to be ready by the end of 2021. My gut feeling is that by moving with > incremental changes, we will not be able to implement even half of the > wishlist by that time. > I doubt that releasing several major releases with breaking changes will > make Ignite users happy either because each upgrade will cost Ignite users > money, so the fewer major versions we release, the better. Thus my wish to > include all breaking changes in one release. > > I'll be now quiet for a while, let's see what other community members > think. > > пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <ptupit...@apache.org>: > > > 1. Rewriting from scratch is never a good idea. > > We don't want to follow the path of Netscape and lose all our users > > by the time we have a working 3.0 [1] > > > > 2. Not sure about new repo - seems like some pain and no gain, what's the > > problem with a branch? > > > > 3. We should keep existing integration tests when possible. > > We have accumulated a lot of edge case knowledge over the years, > > it is not a good idea to send all of that down the drain. > > Yes, integration tests are slow, but they are the most valuable. > > I think we can move more stuff into nightly runs and have a fast and > modern > > basic suite. > > > > > > Alexey, you are much more familiar with the Ignite core codebase than > most > > of us, > > can you please explain in more detail which particular feature, in your > > opinion, > > mandates this "start from scratch" approach? > > Is it really not possible at all to follow a less radical way? > > > > > > [1] > > > > > https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ > > > > On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <nizhi...@apache.org> > > wrote: > > > > > Hello, Alexey. > > > > > > I think that «rewriting from scratch» approach has a high risk to make > > new > > > features unusable. > > > At the time Ignite2 was started no-one wants to do bad UX or bad > > features. > > > Nevertheless, it happen. > > > > > > I think we can avoid it with the Ignite3 and successors if we will move > > > step by step without keeping backward compatibility > > > With the step by step approach, we can focus on each component > > separately. > > > > > > > What new features are we planning to implement for Ignite 2.x? > > > > > > We have many features that have to evolve. > > > Snapshots, rebalance, tooling, tracing, zookeeper support, etc. > > > We have bugs and issues that can be fixed in 2.x without breaking > > backward > > > compatibility. > > > We have many users who are happy with the 2.x with all it’s issues. > > > > > > > 2 нояб. 2020 г., в 14:09, Anton Vinogradov <a...@apache.org> > написал(а): > > > > > > > > Alexey, > > > > > > > > Do we have any estimates of how fast we'll be able to gain > > > production-ready > > > > AI 3.0 in case of a "new repo" choice? > > > > > > > > On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk < > > > alexey.goncha...@gmail.com> > > > > wrote: > > > > > > > >> Nikolay, > > > >> > > > >> What new features are we planning to implement for Ignite 2.x? I > think > > > once > > > >> we commence working on Ignite 3.0, we should gradually cease the > > > activity > > > >> on Ignite 2.x to mere bugfixes because such parallel development > will > > be > > > >> overwhelming regardless of how we choose to proceed. > > > >> > > > >> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <nizhi...@apache.org>: > > > >> > > > >>> To be clear: > > > >>> > > > >>>> I would suggest creating a new repository for Ignite 3.0 > (perhaps, a > > > >> new > > > >>> clean branch, but a new repo looks nicer to me) and a new Ignite > 3.0 > > > >>> TeamCity project. > > > >>> > > > >>> +1 for new Team City project. > > > >>> +1 for new branch for Ignite3. > > > >>> -1 for new repo. > > > >>> > > > >>> > > > >>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <nizhikov....@gmail.com > > > > > >>> написал(а): > > > >>>> > > > >>>> Hello, Alexey. > > > >>>> > > > >>>> I think it will hurt our project more than help. > > > >>>> Developing new features for 2 separate branches with the different > > > APIs > > > >>> and internal structure is overwhelming > > > >>>> > > > >>>> Maybe we should relax a bit requirements for Ignite3? > > > >>>> Maybe we should move step by step and make Ignite3 with new > > > >>> configuration than Ignite4 with new transactions, etc? > > > >>>> > > > >>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk < > > > >> alexey.goncha...@gmail.com> > > > >>> написал(а): > > > >>>>> > > > >>>>> Igniters, > > > >>>>> > > > >>>>> I wanted to pitch a rather radical idea regarding the Ignite 3.0 > > > >>>>> development which has occurred to me some time ago. > > > >>>>> > > > >>>>> We already have several IEPs targeted to Ignite 3.0 which imply > > major > > > >>>>> changes to the codebase (the change in replication protocol and > > thus > > > >>>>> transactions, change in binary format, updated metastorage, etc). > > We > > > >>> also > > > >>>>> planned significant changes in public APIs: configuration format > > > >> change, > > > >>>>> improvements in cache APIs, SQL APIs, transaction mode rework. > The > > > >>> wishlist > > > >>>>> of changes for Ignite 3.0 is huge. > > > >>>>> > > > >>>>> So, I was wondering whether it makes sense to try to change the > old > > > >>>>> codebase, or start with a new baseline and move old pieces of > code > > > >> that > > > >>> do > > > >>>>> not require significant rework. Personally, I would go with the > > > second > > > >>>>> option for the following reasons: > > > >>>>> > > > >>>>> - We have a chance to shift the development paradigm in the > project > > > >> and > > > >>>>> introduce the practice of true unit-tests. In the new baseline at > > the > > > >>>>> beginning there will be no ability to run an end-to-end scenario, > > > >> thus > > > >>> we > > > >>>>> will be forced to write unit-tests. So far, such practice was > hard > > to > > > >>>>> implement because of tight coupling between Ignite components and > > > >>> inability > > > >>>>> to instantiate components without an instance of KernalContext. > For > > > >>>>> example, we should be able to thoroughly test internal > primitives, > > > >>> such as > > > >>>>> replication protocol (without actual communication), distributed > > > >>>>> metastorage contracts, persistence layer, etc. > > > >>>>> - We will significantly reduce the development cycle in the > > beginning > > > >>>>> (right now the RunAll takes two hours of astronomical time with > > empty > > > >>> TC; > > > >>>>> in the new approach developer will be able to run ALL tests > locally > > > >> in > > > >>> a > > > >>>>> matter of minutes) > > > >>>>> - We can get rid of TC bot and enforce green TC by integrating TC > > > >> build > > > >>>>> results with GitHub PRs (the same way Travis is currently > > integrated > > > >>> to PR > > > >>>>> check). We should restrict PR merge without a TC check > > > >>>>> - We will still have to re-write all tests, but only once. If we > > try > > > >> to > > > >>>>> modify the old codebase, we would need to modify all the tests > for > > > >>> every > > > >>>>> major change (public API change, configuration change) > > > >>>>> - We will have fewer conflicts when working together. For > example, > > I > > > >>>>> cannot imagine how one would merge two changes of getting rid of > > > >>>>> IgniteFuture and changes in replication protocol, for example > > > >>>>> > > > >>>>> Technically, I would suggest creating a new repository for Ignite > > 3.0 > > > >>>>> (perhaps, a new clean branch, but a new repo looks nicer to me) > > and a > > > >>> new > > > >>>>> Ignite 3.0 TeamCity project. > > > >>>>> > > > >>>>> While it may seem quite radical, I do believe that this approach > > will > > > >>> give > > > >>>>> us more benefits than trying to make such major changes in the > > > >> existing > > > >>>>> codebase. If needed, let's schedule a discord chat like before to > > > >>> discuss > > > >>>>> this. > > > >>>>> > > > >>>>> WDYT? > > > >>>> > > > >>> > > > >>> > > > >> > > > > > > > > >
