Time works for me.
> 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <alexey.goncha...@gmail.com>
> написал(а):
>
> Nikolay,
>
> I am up for the call. I will try to explain my reasoning in greater detail
> and will be glad to hear the concerns. Will this Friday, Nov 6th, work?
>
> вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <nizhi...@apache.org>:
>
>> Igniters, should we have a call for this topic?
>>
>>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <ptupit...@apache.org>
>> написал(а):
>>>
>>>> not intend to rewrite everything from scratch
>>>
>>>> Every single test from Ignite 2.x should be moved to Ignite 3
>>>> regardless of how we choose to proceed.
>>>
>>> Alexey, thank you for the explanation, this addresses all of my concerns.
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov <
>> andrey.mashen...@gmail.com>
>>> wrote:
>>>
>>>> Hi, Igniters.
>>>>
>>>> * AFAIU, we need a new repo if we want to apply different restrictions
>> to
>>>> pull requests,
>>>> otherwise I see no difference for myself.
>>>> E.g. make static analysis (do we have?), compile, styles, and javadoc
>>>> checks mandatory.
>>>>
>>>> I think that relaxed requirements here will lead to bad product quality.
>>>>
>>>> * Agree with Pavel, we should 'keep' integrations tests somehow.
>>>> During active development tests will be broken most of time, so,
>>>> I'd port them e.g. suite-by-suite once we will have a stable and
>> featured
>>>> environment to run them and of course make test's code clear and avoid
>>>> bad/non-relevant ones.
>>>>
>>>> * I like bottom-up approach.
>>>> With it we could make a better framework. I mean clear component
>> lifecycle,
>>>> component wiring mechanics, general methods to approach core components
>>>> such as exchange/communication
>>>> to avoid code mess like we have in ExchangeFuture with all these custom
>>>> callbacks for each component, interfaces like
>>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and
>>>> a pack of
>> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected
>>>> and so on in various unexpected places.
>>>> Hope, we will be able to port most of the good code to the new framework
>>>> version.
>>>>
>>>>
>>>>
>>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk <
>>>> alexey.goncha...@gmail.com>
>>>> wrote:
>>>>
>>>>> Nikolay, Pavel,
>>>>>
>>>>> Thanks for the feedback! First of all, I wanted to stress that I do not
>>>>> intend to rewrite everything from scratch (I never used this phrase).
>>>> There
>>>>> are significant parts of code that would be moved with minimal
>>>>> modifications. Second, I never said that we will get rid of the old
>> tests
>>>>> codebase. Every single test from Ignite 2.x should be moved to Ignite 3
>>>>> regardless of how we choose to proceed.
>>>>>
>>>>> My point is that for some parts of the code a clean bottom-up
>>>>> implementation will be cheaper in many ways. Let me give you a few
>>>> concrete
>>>>> examples:
>>>>>
>>>>> - I think no one can object that we need a cleanly separated
>>>> persistence
>>>>> layer for Ignite. There is a very raw draft IEP for this already. On
>>>> the
>>>>> other hand, I think we also can agree that we need a split-brain
>>>>> resistant
>>>>> replication protocol for caches. There is also an IEP for this.
>>>> Neither
>>>>> of
>>>>> the changes is a good fit for 2.x because they are likely to
>> introduce
>>>>> breaking changes in the persistence layer, configuration and
>> behavior.
>>>>> Additionally, these components are now tightly coupled, so there is
>> no
>>>>> way
>>>>> these two changes can be implemented in parallel and then merged
>>>>> together
>>>>> easily. So what we will end up with is having to implement these
>>>> changes
>>>>> sequentially, fixing all existing tests twice, and essentially
>>>> throwing
>>>>> away half of the work done because the other part of the change is
>>>>> re-implemented
>>>>> - Similar example goes with getting rid of IgniteInternalFuture and
>>>>> replacing it with CompletableFuture, and any other change that
>> touches
>>>>> the
>>>>> asynchronous part of the code.
>>>>>
>>>>> Third, I do not see how this choice affects the UX of Ignite. The end
>>>> user
>>>>> experience must be fixed in the IEP regardless of the development
>> process
>>>>> and the fact that we have gaps in this area in Ignite 2.x just confirms
>>>>> that.
>>>>>
>>>>> Pavel, agree that a repo/branch is a technicality, I guess if
>>>> reformulate,
>>>>> my point is that we might agree to have a single development master
>>>> branch
>>>>> with 'disassembled' end-to-end functionality for some period of time to
>>>>> speed up development, and re-assemble the core features after having
>>>>> submodules tested independently.
>>>>>
>>>>> Nikolay,
>>>>>> We have many features that have to evolve.
>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
>>>>> This is not very specific. In the end, resources are limited and we
>> will
>>>>> not be able to drive both tracks simultaneously, especially after a
>>>> couple
>>>>> of features having been implemented for Ignite 3.0. If there are indeed
>>>>> some major changes that we want to do in Ignite 2.x instead of putting
>>>>> effort into 3.0 - let's discuss them. I am just not aware of any,
>> that's
>>>>> why I am eager to move forward with Ignite 3.0.
>>>>>
>>>>>> We have bugs and issues that can be fixed in 2.x without breaking
>>>> backward
>>>>> compatibility.
>>>>>> We have many users who are happy with the 2.x with all it’s issues.
>>>>> These changes will be covered by end-to-end tests and migrated to
>> Ignite
>>>>> 3.0, so I see no issues here.
>>>>>
>>>>> Finally, Anton & Nikolay
>>>>> I do not have an estimate for this simply because the activity is
>>>>> community-driven and it depends on the number of people willing to
>>>>> contribute. With the current pace, I would hope to have an RC of Ignite
>>>> 3.0
>>>>> to be ready by the end of 2021. My gut feeling is that by moving with
>>>>> incremental changes, we will not be able to implement even half of the
>>>>> wishlist by that time.
>>>>> I doubt that releasing several major releases with breaking changes
>> will
>>>>> make Ignite users happy either because each upgrade will cost Ignite
>>>> users
>>>>> money, so the fewer major versions we release, the better. Thus my wish
>>>> to
>>>>> include all breaking changes in one release.
>>>>>
>>>>> I'll be now quiet for a while, let's see what other community members
>>>>> think.
>>>>>
>>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <ptupit...@apache.org>:
>>>>>
>>>>>> 1. Rewriting from scratch is never a good idea.
>>>>>> We don't want to follow the path of Netscape and lose all our users
>>>>>> by the time we have a working 3.0 [1]
>>>>>>
>>>>>> 2. Not sure about new repo - seems like some pain and no gain, what's
>>>> the
>>>>>> problem with a branch?
>>>>>>
>>>>>> 3. We should keep existing integration tests when possible.
>>>>>> We have accumulated a lot of edge case knowledge over the years,
>>>>>> it is not a good idea to send all of that down the drain.
>>>>>> Yes, integration tests are slow, but they are the most valuable.
>>>>>> I think we can move more stuff into nightly runs and have a fast and
>>>>> modern
>>>>>> basic suite.
>>>>>>
>>>>>>
>>>>>> Alexey, you are much more familiar with the Ignite core codebase than
>>>>> most
>>>>>> of us,
>>>>>> can you please explain in more detail which particular feature, in
>> your
>>>>>> opinion,
>>>>>> mandates this "start from scratch" approach?
>>>>>> Is it really not possible at all to follow a less radical way?
>>>>>>
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>>
>>>>>
>>>>
>> https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/
>>>>>>
>>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov <nizhi...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello, Alexey.
>>>>>>>
>>>>>>> I think that «rewriting from scratch» approach has a high risk to
>>>> make
>>>>>> new
>>>>>>> features unusable.
>>>>>>> At the time Ignite2 was started no-one wants to do bad UX or bad
>>>>>> features.
>>>>>>> Nevertheless, it happen.
>>>>>>>
>>>>>>> I think we can avoid it with the Ignite3 and successors if we will
>>>> move
>>>>>>> step by step without keeping backward compatibility
>>>>>>> With the step by step approach, we can focus on each component
>>>>>> separately.
>>>>>>>
>>>>>>>> What new features are we planning to implement for Ignite 2.x?
>>>>>>>
>>>>>>> We have many features that have to evolve.
>>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc.
>>>>>>> We have bugs and issues that can be fixed in 2.x without breaking
>>>>>> backward
>>>>>>> compatibility.
>>>>>>> We have many users who are happy with the 2.x with all it’s issues.
>>>>>>>
>>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <a...@apache.org>
>>>>> написал(а):
>>>>>>>>
>>>>>>>> Alexey,
>>>>>>>>
>>>>>>>> Do we have any estimates of how fast we'll be able to gain
>>>>>>> production-ready
>>>>>>>> AI 3.0 in case of a "new repo" choice?
>>>>>>>>
>>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk <
>>>>>>> alexey.goncha...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Nikolay,
>>>>>>>>>
>>>>>>>>> What new features are we planning to implement for Ignite 2.x? I
>>>>> think
>>>>>>> once
>>>>>>>>> we commence working on Ignite 3.0, we should gradually cease the
>>>>>>> activity
>>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel development
>>>>> will
>>>>>> be
>>>>>>>>> overwhelming regardless of how we choose to proceed.
>>>>>>>>>
>>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov <nizhi...@apache.org
>>>>> :
>>>>>>>>>
>>>>>>>>>> To be clear:
>>>>>>>>>>
>>>>>>>>>>> I would suggest creating a new repository for Ignite 3.0
>>>>> (perhaps, a
>>>>>>>>> new
>>>>>>>>>> clean branch, but a new repo looks nicer to me) and a new Ignite
>>>>> 3.0
>>>>>>>>>> TeamCity project.
>>>>>>>>>>
>>>>>>>>>> +1 for new Team City project.
>>>>>>>>>> +1 for new branch for Ignite3.
>>>>>>>>>> -1 for new repo.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov <
>>>> nizhikov....@gmail.com
>>>>>>
>>>>>>>>>> написал(а):
>>>>>>>>>>>
>>>>>>>>>>> Hello, Alexey.
>>>>>>>>>>>
>>>>>>>>>>> I think it will hurt our project more than help.
>>>>>>>>>>> Developing new features for 2 separate branches with the
>>>> different
>>>>>>> APIs
>>>>>>>>>> and internal structure is overwhelming
>>>>>>>>>>>
>>>>>>>>>>> Maybe we should relax a bit requirements for Ignite3?
>>>>>>>>>>> Maybe we should move step by step and make Ignite3 with new
>>>>>>>>>> configuration than Ignite4 with new transactions, etc?
>>>>>>>>>>>
>>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
>>>>>>>>> alexey.goncha...@gmail.com>
>>>>>>>>>> написал(а):
>>>>>>>>>>>>
>>>>>>>>>>>> Igniters,
>>>>>>>>>>>>
>>>>>>>>>>>> I wanted to pitch a rather radical idea regarding the Ignite
>>>> 3.0
>>>>>>>>>>>> development which has occurred to me some time ago.
>>>>>>>>>>>>
>>>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which imply
>>>>>> major
>>>>>>>>>>>> changes to the codebase (the change in replication protocol and
>>>>>> thus
>>>>>>>>>>>> transactions, change in binary format, updated metastorage,
>>>> etc).
>>>>>> We
>>>>>>>>>> also
>>>>>>>>>>>> planned significant changes in public APIs: configuration
>>>> format
>>>>>>>>> change,
>>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode rework.
>>>>> The
>>>>>>>>>> wishlist
>>>>>>>>>>>> of changes for Ignite 3.0 is huge.
>>>>>>>>>>>>
>>>>>>>>>>>> So, I was wondering whether it makes sense to try to change the
>>>>> old
>>>>>>>>>>>> codebase, or start with a new baseline and move old pieces of
>>>>> code
>>>>>>>>> that
>>>>>>>>>> do
>>>>>>>>>>>> not require significant rework. Personally, I would go with the
>>>>>>> second
>>>>>>>>>>>> option for the following reasons:
>>>>>>>>>>>>
>>>>>>>>>>>> - We have a chance to shift the development paradigm in the
>>>>> project
>>>>>>>>> and
>>>>>>>>>>>> introduce the practice of true unit-tests. In the new baseline
>>>> at
>>>>>> the
>>>>>>>>>>>> beginning there will be no ability to run an end-to-end
>>>> scenario,
>>>>>>>>> thus
>>>>>>>>>> we
>>>>>>>>>>>> will be forced to write unit-tests. So far, such practice was
>>>>> hard
>>>>>> to
>>>>>>>>>>>> implement because of tight coupling between Ignite components
>>>> and
>>>>>>>>>> inability
>>>>>>>>>>>> to instantiate components without an instance of KernalContext.
>>>>> For
>>>>>>>>>>>> example, we should be able to thoroughly test internal
>>>>> primitives,
>>>>>>>>>> such as
>>>>>>>>>>>> replication protocol (without actual communication),
>>>> distributed
>>>>>>>>>>>> metastorage contracts, persistence layer, etc.
>>>>>>>>>>>> - We will significantly reduce the development cycle in the
>>>>>> beginning
>>>>>>>>>>>> (right now the RunAll takes two hours of astronomical time with
>>>>>> empty
>>>>>>>>>> TC;
>>>>>>>>>>>> in the new approach developer will be able to run ALL tests
>>>>> locally
>>>>>>>>> in
>>>>>>>>>> a
>>>>>>>>>>>> matter of minutes)
>>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by integrating
>>>> TC
>>>>>>>>> build
>>>>>>>>>>>> results with GitHub PRs (the same way Travis is currently
>>>>>> integrated
>>>>>>>>>> to PR
>>>>>>>>>>>> check). We should restrict PR merge without a TC check
>>>>>>>>>>>> - We will still have to re-write all tests, but only once. If
>>>> we
>>>>>> try
>>>>>>>>> to
>>>>>>>>>>>> modify the old codebase, we would need to modify all the tests
>>>>> for
>>>>>>>>>> every
>>>>>>>>>>>> major change (public API change, configuration change)
>>>>>>>>>>>> - We will have fewer conflicts when working together. For
>>>>> example,
>>>>>> I
>>>>>>>>>>>> cannot imagine how one would merge two changes of getting rid
>>>> of
>>>>>>>>>>>> IgniteFuture and changes in replication protocol, for example
>>>>>>>>>>>>
>>>>>>>>>>>> Technically, I would suggest creating a new repository for
>>>> Ignite
>>>>>> 3.0
>>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer to me)
>>>>>> and a
>>>>>>>>>> new
>>>>>>>>>>>> Ignite 3.0 TeamCity project.
>>>>>>>>>>>>
>>>>>>>>>>>> While it may seem quite radical, I do believe that this
>>>> approach
>>>>>> will
>>>>>>>>>> give
>>>>>>>>>>>> us more benefits than trying to make such major changes in the
>>>>>>>>> existing
>>>>>>>>>>>> codebase. If needed, let's schedule a discord chat like before
>>>> to
>>>>>>>>>> discuss
>>>>>>>>>>>> this.
>>>>>>>>>>>>
>>>>>>>>>>>> WDYT?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Andrey V. Mashenkov
>>>>
>>
>>
