Re: [REVIEW REQUEST] IGNITE-11951 Improvements in JdkMarshaller

2019-07-11 Thread Павлухин Иван
Merged the patch to master [1]. Thank Alex Plekhanov for a review.

[1] https://issues.apache.org/jira/browse/IGNITE-11951

ср, 10 июл. 2019 г. в 08:42, Павлухин Иван :
>
> Hi,
>
> I made some small improvements in JdkMarshaller [1]. I will be happy
> if someone reviews it. Changes are quite simple.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-11951
>
> --
> Best regards,
> Ivan Pavlukhin



-- 
Best regards,
Ivan Pavlukhin


Re: Revisit added java modules in ignite.sh

2019-07-11 Thread Павлухин Иван
I merged the patch [1] by lazy consensus.

[1] https://issues.apache.org/jira/browse/IGNITE-11946

пн, 8 июл. 2019 г. в 20:48, Павлухин Иван :
>
> Igniters,
>
> I present a ticket [1] removing java.transaction and java.corba
> modules from sh/bat scripts for your review. I remind that it fixes
> nodejs, python and php suites when running on Java 9 and 10.
>
> In case there is no activity on this I would like to merge it this week.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-11946
>
> ср, 3 июл. 2019 г. в 21:40, Павлухин Иван :
> >
> > I removed command line arguments enabling java.transaction and
> > java.corba module from TC configuration. All activity in a ticket [1].
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-11946
> >
> > вс, 30 июн. 2019 г. в 07:31, Павлухин Иван :
> > >
> > > I made 2 experiments:
> > > 1. Removed --add-modules=java.transaction from TC configurations. I
> > > have not observed any new test failures after that change.
> > > 2. Removed that option from ignite.sh in PR. In PR problematic thin
> > > client suites are green.
> > >
> > > See the ticket [1] for additional details. If there is no objections I
> > > will cleanup that option from TC configurations and Ignite scripts.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-9687
> > >
> > > пт, 28 июн. 2019 г. в 10:09, Павлухин Иван :
> > > >
> > > > Folks,
> > > >
> > > > During investigation of thin client test suite failures [1] I found
> > > > that it is not clear whether --add-modules=java.transaction in
> > > > ignite.sh/bat is needed (we add this module for Java 9 and 10). In
> > > > thread [1] I already mentioned that launching ignite.sh with Java 9
> > > > and ignite-jta libraries on a classpath leads to an error. You can see
> > > > how it can look in a log [2].
> > > >
> > > > My proposal here is to get rid off --add-modules=java.transaction from
> > > > stratup scripts and parameters used by TC. And I will try to explain
> > > > why.
> > > >
> > > > The story started in a ticket [3]. Where a recipe
> > > > --add-modules=java.transaction
> > > > --patch-module=java.transaction=$HOME/.m2/repository/javax/transaction/javax.transaction-api/1.3/javax.transaction-api-1.3.jar
> > > > allowed to execute JTA tests successfully. After that
> > > > --add-modules=java.transaction was added to launch scripts. But that
> > > > setup allowed to launch tests. And they failed actually due to
> > > > problems with resolving needed classes for test depndencies
> > > > (org.ow2.jotm:jotm-core). Needed classes were not found because the
> > > > dependency required classes from java standard library which were
> > > > moved to optional modules. And the thing is even trickier with a
> > > > --patch-module option because javax.transaction-api-1.3.jar contains a
> > > > superset of classes comparing to java.transaction module. So, I do not
> > > > see a need of adding this module. Moreover that module was removed
> > > > completely starting from Java 11. And even if missed something and it
> > > > has some meaning for Java 9/10, I still think that having uniform
> > > > setup for all Java versions simplifies matters.
> > > >
> > > > Have I missed something? Do we have an example how to check JTA
> > > > integration? How it can be used with a standalone Ignite setup?
> > > >
> > > > P.S. Also I will try to do a similar reseach about java.xml.bind
> > > > module if noone knows why do we need it.
> > > >
> > > > [1] 
> > > > http://apache-ignite-developers.2346864.n4.nabble.com/Thin-client-test-suites-failure-td42388.html
> > > > [2] 
> > > > https://ci.ignite.apache.org/viewLog.html?buildId=4217786=IgniteTests24Java8_ThinClientNodeJs=buildLog_IgniteTests24Java8=pull%2F6644%2Fhead&_focus=423#_state=423
> > > > [3] https://issues.apache.org/jira/browse/IGNITE-9687
> > > > --
> > > > Best regards,
> > > > Ivan Pavlukhin
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
>
>
>
> --
> Best regards,
> Ivan Pavlukhin



-- 
Best regards,
Ivan Pavlukhin


Re: [discussion] using custom build of H2 for Ignite

2019-07-11 Thread Denis Magda
Dmitry,

Let's not flood this discussion with topics unrelated to the subj. Please
find my notes on some of your statements below. If you believe the notes
don't solve your concerns, please start or restart another discussion.

Overall, it seems that a separate vendor-neutral Github repository is a
good solution for our community. Any other thoughts before we proceed?




The notes:


> We already have locked in relation to
> - DEB/RPM package release (I don't know if PMC knows how to prepare it),
> - we have Node.js packages without releasing procedure.
>

Instructions for Node.JS/Python/PHP were shared by contributors over email,
and we forgot to put them on wiki. As both of us know, Igor Sapego agreed
to move everything to wiki. DEB/RPM instructions are to be provided by
another GridGain contributor - Peter Ivanov. I believe the work is in
progress.

Please, for the sake of efficiency and to finish 2.7.5 post-release
procedures, swap your ASF hat with a GridGain for a moment to follow up
with Igor and Peter to expedite the resolution. It might be a matter of a
short verbal conversation and as a result, the Ignite community will have
all the instructions documented.

- we have some gridgain:shmem with unclear reasoning behind,


Please see my previous response on this matter. The source code belongs to
Ignite and anybody can rebuild this library and name it the way the
community likes. With my GridGain hat on for a minute, the company is not
ready to invest its time on artifact renaming just because of "gridgain"
name in it. If other contributors that have no affiliation with GridGain
would like to do that please go ahead:
http://apache-ignite-developers.2346864.n4.nabble.com/Apache-Ignite-ignite-core-dependecies-tp41096p41132.html

-
Denis


On Thu, Jul 11, 2019 at 1:06 AM Dmitriy Pavlov  wrote:

> Denis, Igniters,
>
> I want to add 2 cents to make it as clear as possible
>
> Every contributor and committer (whatever company he is working for) is
> appreciated and I encourage all GridGain'ers to participate.
>
> Everyone participates as an individual. So if someone provides some tool in
> his or her own GitHub - it may be ok in some cases. We're still using
> abbrev plugin hosted in my public individual GitHub repository.
>
> But once we have something which is prepared and released for Ignite
> outside of the project, by any company - we may face a situation that
> - the release is not possible because we need a release of dependency first
> - a company changed its priorities, the company does not consider this
> project is strategic anymore. This happens from time to time in OpenSource.
>
> And if release is not possible - what is the point? what's the point to
> discuss, to contribute code? It won't reach users.  Impossibility to
> release Ignite = shortest way to the Attic.
>
> Ignite PMC members should prevent such outside locks, and protect the
> project from commercial interests influence. This helps to increase
> diversity and community building, as well.
>
> We already have locked in relation to
> - DEB/RPM package release (I don't know if PMC knows how to prepare it),
> - we have some gridgain:shmem with unclear reasoning behind,
> - we have Node.js packages without releasing procedure.
>
> I hope we will overcome it soon and we don't need to start the healing
> process.
>
> Nevertheless, we will not add new (dead)locks. I hope it is enough
> justification for -1.
>
> Sincerely,
> Dmitriy Pavlov
>
> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
>
> чт, 11 июл. 2019 г. в 00:03, Dmitriy Pavlov :
>
> > Denis, I'm not negative in relation to  GridGain, if Sberbank will
> suggest
> > Ignite (or any other Apache project I'm involved too) code to be
> dependent
> > on
> > 'com.sberbank.whatsoever.coollibrary'
> > binary, they can count on my veto, as well.
> >
> > There is no difference in company suggesting this: 'com.microsoft',
> > 'com.intel' - I see no difference there.
> >
> > Disclaimer: Opinions expressed in this email are those of the author,
> > and do not necessarily represent the views of any company the author
> > might be affiliated with at the moment of writing.
> >
> > ср, 10 июл. 2019 г. в 23:13, Denis Magda :
> >
> >> Ok, folks, let's create a separate Github repo for the H2 fork and
> ensure
> >> the binaries of that fork are used by Ignite. As for the discussion with
> >> ASF legal, nobody suggested any alternatives and I'm signing off that
> >> discussion:
> >>
> >>
> http://apache-ignite-developers.2346864.n4.nabble.com/Place-for-MPL-2-0-EPL-1-0-source-code-in-ASF-ecosystem-td42604.html#a42621
> >>
> >> Dmitriy, I don't fully understand your negative attitude towards
> GridGain.
> >> That's just one of the biggest contributors to Ignite because of
> >> historical
> >> reasons - Ignite was donated by this vendor and GridGain is 

[jira] [Created] (IGNITE-11975) Fix Ignite Examples on site - Data Grid feature, how to start

2019-07-11 Thread Dmitriy Pavlov (JIRA)
Dmitriy Pavlov created IGNITE-11975:
---

 Summary: Fix Ignite Examples on site - Data Grid feature, how to 
start
 Key: IGNITE-11975
 URL: https://issues.apache.org/jira/browse/IGNITE-11975
 Project: Ignite
  Issue Type: Improvement
Reporter: Dmitriy Pavlov
Assignee: Dmitriy Pavlov


https://ignite.apache.org/features/datagrid.html

Contains several examples, which won't compile



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


Re: [discussion] using custom build of H2 for Ignite

2019-07-11 Thread Dmitriy Pavlov
Denis, Igniters,

I want to add 2 cents to make it as clear as possible

Every contributor and committer (whatever company he is working for) is
appreciated and I encourage all GridGain'ers to participate.

Everyone participates as an individual. So if someone provides some tool in
his or her own GitHub - it may be ok in some cases. We're still using
abbrev plugin hosted in my public individual GitHub repository.

But once we have something which is prepared and released for Ignite
outside of the project, by any company - we may face a situation that
- the release is not possible because we need a release of dependency first
- a company changed its priorities, the company does not consider this
project is strategic anymore. This happens from time to time in OpenSource.

And if release is not possible - what is the point? what's the point to
discuss, to contribute code? It won't reach users.  Impossibility to
release Ignite = shortest way to the Attic.

Ignite PMC members should prevent such outside locks, and protect the
project from commercial interests influence. This helps to increase
diversity and community building, as well.

We already have locked in relation to
- DEB/RPM package release (I don't know if PMC knows how to prepare it),
- we have some gridgain:shmem with unclear reasoning behind,
- we have Node.js packages without releasing procedure.

I hope we will overcome it soon and we don't need to start the healing
process.

Nevertheless, we will not add new (dead)locks. I hope it is enough
justification for -1.

Sincerely,
Dmitriy Pavlov

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.

чт, 11 июл. 2019 г. в 00:03, Dmitriy Pavlov :

> Denis, I'm not negative in relation to  GridGain, if Sberbank will suggest
> Ignite (or any other Apache project I'm involved too) code to be dependent
> on
> 'com.sberbank.whatsoever.coollibrary'
> binary, they can count on my veto, as well.
>
> There is no difference in company suggesting this: 'com.microsoft',
> 'com.intel' - I see no difference there.
>
> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
>
> ср, 10 июл. 2019 г. в 23:13, Denis Magda :
>
>> Ok, folks, let's create a separate Github repo for the H2 fork and ensure
>> the binaries of that fork are used by Ignite. As for the discussion with
>> ASF legal, nobody suggested any alternatives and I'm signing off that
>> discussion:
>>
>> http://apache-ignite-developers.2346864.n4.nabble.com/Place-for-MPL-2-0-EPL-1-0-source-code-in-ASF-ecosystem-td42604.html#a42621
>>
>> Dmitriy, I don't fully understand your negative attitude towards GridGain.
>> That's just one of the biggest contributors to Ignite because of
>> historical
>> reasons - Ignite was donated by this vendor and GridGain is trying to make
>> this community diversified allocating its own time and resources. I'm
>> pretty sure that many ASF projects are in the same situation - that has to
>> be a first vendor who does most of the technological changes and helps to
>> grow the community. Luckily, these days another vendor supports Ignite
>> which is Sberbank, hopefully, more vendors to come. Personally, I do
>> believe that most of successful open source projects depend on vendors who
>> invest $$$ and let their employees work on open source daily.
>>
>> -
>> Denis
>>
>>
>> On Wed, Jul 10, 2019 at 8:03 AM Павлухин Иван 
>> wrote:
>>
>> > Dmitriy,
>> >
>> > Thank you for sharing your vision.
>> >
>> > Actually I think that an agreement within the community is the most
>> > important thing.
>> >
>> > ср, 10 июл. 2019 г. в 17:44, Dmitriy Pavlov :
>> > >
>> > > Hi Ivan,
>> > >
>> > >
>> > >
>> > > I don’t need a policy to clearly understand that the Apache project
>> stops
>> > > to be healthy once it is controlled by one entity. This is related to
>> > > governance, not to OSI approved license (if a lib is open-source or
>> not).
>> > >
>> > >
>> > >
>> > > Apache mission - Create software for the public good.
>> > >
>> > > Apache project is:
>> > >
>> > > - Open Source, commercial-friendly Licence,
>> > >
>> > > - Open Governance (clear, documented leader election),
>> > >
>> > > - Open Brand (Brand owner is always ASF, it is a non-profit, charity
>> > > organization).
>> > >
>> > >
>> > >
>> > > And once community adds a dependency to any vendor-controlled and
>> > released
>> > > artifact I suppose it is not anymore for the public good, it is for
>> good
>> > of
>> > > vendor. A goal can be to be advertized using this open-source project
>> and
>> > > the Apache brand. Vendor in some cases want just Apache brand, and
>> rarely
>> > > shares Apache principles and Apache Way.
>> > >
>> > >
>> > >
>> > > Maybe Konstantin could explain better than me, why the 

Re: Read Repair (ex. Consistency Check) - review request #2

2019-07-11 Thread Anton Vinogradov
Nikolay,

Initial idea is to count fixes per cache.
In other words, event recording should cause metric increment.
Could you help me with RR metrics implementation as a part of your metrics
journey?

On Thu, Jul 11, 2019 at 10:18 AM Nikolay Izhikov 
wrote:

> Anton,
>
> > - metrics are better for monitoring, but the Event is enough for my wish
> to cover AI with consistency check,
>
> Can you clarify, do you have plans to add metrics of RR events?
>
> I think it should be count of incosistency events per cache(maybe per
> partition).
>
>
> В Чт, 11/07/2019 в 09:53 +0300, Anton Vinogradov пишет:
> > Slava,
> >
> > Thanks for your review first!
> >
> > > > Anyway, I left some comments in your pull-request at github. Please
> take
> >
> > a
> > > > look. The most annoying thing is poorly documented code :(
> >
> > Since your comments mostly about Javadoc (does this mean that my solution
> > is so great that you ask me only to fix Javadocs :) ?),
> > I'd like to propose you to provide fixes as a PR since you have a vision
> of
> > how it should be made. I'll review them and merge shortly.
> >
> > > > By the way, is it required to add test related to fail-over
> scenarios?
> >
> > The best check is to use RR at real code.
> > For example, I'm injecting RR now to the test with concurrent
> modifications
> > and restarts [1].
> >
> > > > I just checked, the IEP page and I still cannot find Jira tickets
> that
> > > > should cover existing limitations/improvements.
> > > > I would suggest creating the following tasks, at least:
> >
> > Mostly agree with you, but
> > - MVCC is not production ready,
> > - not sure near support really required,
> > - metrics are better for monitoring, but the Event is enough for my wish
> to
> > cover AI with consistency check,
> > - do we really need Platforms and Thin Client support?
> > Also, we should not produce stillborn issue.
> > All limitations listed at proxy creation method and they definitely are
> not
> > showstoppers and may be fixed later if someone interested.
> > Сoming back to AI 3.0 discussion, we have A LOT of features and it's
> almost
> > impossible (require much more time that feature's cost) to support them
> all.
> > I will be pretty happy in case someone will do this and provide help if
> > necessary!
> >
> > > > Perhaps, it would be a good idea to think about the recovery too
> >
> > Do you mean per partition check and recovery?
> > That's a good idea, but I found it's not easy to imagine API to for such
> > tool.
> > In case you ready to assist with proper API/design this will definitely
> > help.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-11973
> >
> > On Wed, Jul 10, 2019 at 3:43 PM Вячеслав Коптилин <
> slava.kopti...@gmail.com>
> > wrote:
> >
> > > Perhaps, it would be a good idea to think about the recovery tool/
> > > control-utility command that will allow achieving the same goal.
> > > If I am not mistaken it was already proposed in the email thread.
> > >
> > > Thanks,
> > > S.
> > >
> > > ср, 10 июл. 2019 г. в 15:33, Вячеслав Коптилин <
> slava.kopti...@gmail.com>:
> > >
> > > > Hi Anton,
> > > >
> > > > Well, the ReadRepair feature is finally merged and that is good news
> :)
> > > >
> > > > Unfortunately, I cannot find a consensus about the whole
> functionality in
> > > > any of these topics:
> > > >  -
> > > >
> > >
> > >
> http://apache-ignite-developers.2346864.n4.nabble.com/Consistency-check-and-fix-review-request-td41629.html
> > > >  -
> > > >
> > >
> > >
> http://apache-ignite-developers.2346864.n4.nabble.com/Read-Repair-ex-Consistency-Check-review-request-2-td42421.html
> > > > Also, there are no comments/discussion in JIRA. That makes me sad :(
> > > > especially when a feature is huge, not obvious and involves changing
> > >
> > > public
> > > > API (and that is the case, I think).
> > > >
> > > > Anyway, I left some comments in your pull-request at github. Please
> take
> > >
> > > a
> > > > look. The most annoying thing is poorly documented code :(
> > > > By the way, is it required to add test related to fail-over
> scenarios?
> > > >
> > > > I just checked, the IEP page and I still cannot find Jira tickets
> that
> > > > should cover existing limitations/improvements.
> > > > I would suggest creating the following tasks, at least:
> > > >  - MVCC support
> > > >  - Near caches
> > > >  - Additional metrics (number of violations, number of repaired
> entries
> > > > etc)
> > > >  - Ignite C++ (It looks like, .Net is already aware of that feature)
> > > >  - Thin clients support
> > > >  - Perhaps, it would be useful to support different strategies to
> resolve
> > > > inconsistencies
> > > >
> > > > What do you think?
> > > >
> > > > Thanks,
> > > > S.
> > > >
> > > >
> > > > ср, 10 июл. 2019 г. в 10:16, Anton Vinogradov :
> > > >
> > > > > Folks,
> > > > >
> > > > > Thanks to everyone for tips and reviews.
> > > > > Yardstick checked, no performance drop found.
> > > > > Additional measurement: RR get() is just up 

Re: Read Repair (ex. Consistency Check) - review request #2

2019-07-11 Thread Nikolay Izhikov
Anton,

> - metrics are better for monitoring, but the Event is enough for my wish to 
> cover AI with consistency check,

Can you clarify, do you have plans to add metrics of RR events?

I think it should be count of incosistency events per cache(maybe per 
partition).


В Чт, 11/07/2019 в 09:53 +0300, Anton Vinogradov пишет:
> Slava,
> 
> Thanks for your review first!
> 
> > > Anyway, I left some comments in your pull-request at github. Please take
> 
> a
> > > look. The most annoying thing is poorly documented code :(
> 
> Since your comments mostly about Javadoc (does this mean that my solution
> is so great that you ask me only to fix Javadocs :) ?),
> I'd like to propose you to provide fixes as a PR since you have a vision of
> how it should be made. I'll review them and merge shortly.
> 
> > > By the way, is it required to add test related to fail-over scenarios?
> 
> The best check is to use RR at real code.
> For example, I'm injecting RR now to the test with concurrent modifications
> and restarts [1].
> 
> > > I just checked, the IEP page and I still cannot find Jira tickets that
> > > should cover existing limitations/improvements.
> > > I would suggest creating the following tasks, at least:
> 
> Mostly agree with you, but
> - MVCC is not production ready,
> - not sure near support really required,
> - metrics are better for monitoring, but the Event is enough for my wish to
> cover AI with consistency check,
> - do we really need Platforms and Thin Client support?
> Also, we should not produce stillborn issue.
> All limitations listed at proxy creation method and they definitely are not
> showstoppers and may be fixed later if someone interested.
> Сoming back to AI 3.0 discussion, we have A LOT of features and it's almost
> impossible (require much more time that feature's cost) to support them all.
> I will be pretty happy in case someone will do this and provide help if
> necessary!
> 
> > > Perhaps, it would be a good idea to think about the recovery too
> 
> Do you mean per partition check and recovery?
> That's a good idea, but I found it's not easy to imagine API to for such
> tool.
> In case you ready to assist with proper API/design this will definitely
> help.
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-11973
> 
> On Wed, Jul 10, 2019 at 3:43 PM Вячеслав Коптилин 
> wrote:
> 
> > Perhaps, it would be a good idea to think about the recovery tool/
> > control-utility command that will allow achieving the same goal.
> > If I am not mistaken it was already proposed in the email thread.
> > 
> > Thanks,
> > S.
> > 
> > ср, 10 июл. 2019 г. в 15:33, Вячеслав Коптилин :
> > 
> > > Hi Anton,
> > > 
> > > Well, the ReadRepair feature is finally merged and that is good news :)
> > > 
> > > Unfortunately, I cannot find a consensus about the whole functionality in
> > > any of these topics:
> > >  -
> > > 
> > 
> > http://apache-ignite-developers.2346864.n4.nabble.com/Consistency-check-and-fix-review-request-td41629.html
> > >  -
> > > 
> > 
> > http://apache-ignite-developers.2346864.n4.nabble.com/Read-Repair-ex-Consistency-Check-review-request-2-td42421.html
> > > Also, there are no comments/discussion in JIRA. That makes me sad :(
> > > especially when a feature is huge, not obvious and involves changing
> > 
> > public
> > > API (and that is the case, I think).
> > > 
> > > Anyway, I left some comments in your pull-request at github. Please take
> > 
> > a
> > > look. The most annoying thing is poorly documented code :(
> > > By the way, is it required to add test related to fail-over scenarios?
> > > 
> > > I just checked, the IEP page and I still cannot find Jira tickets that
> > > should cover existing limitations/improvements.
> > > I would suggest creating the following tasks, at least:
> > >  - MVCC support
> > >  - Near caches
> > >  - Additional metrics (number of violations, number of repaired entries
> > > etc)
> > >  - Ignite C++ (It looks like, .Net is already aware of that feature)
> > >  - Thin clients support
> > >  - Perhaps, it would be useful to support different strategies to resolve
> > > inconsistencies
> > > 
> > > What do you think?
> > > 
> > > Thanks,
> > > S.
> > > 
> > > 
> > > ср, 10 июл. 2019 г. в 10:16, Anton Vinogradov :
> > > 
> > > > Folks,
> > > > 
> > > > Thanks to everyone for tips and reviews.
> > > > Yardstick checked, no performance drop found.
> > > > Additional measurement: RR get() is just up to 7% slower than regular
> > 
> > get
> > > > on real benchmarks (8 clients, 4 servers, 3 backups)
> > > > Code merged to the master.
> > > > "Must have" tasks created and attached to the IEP.
> > > > 
> > > > On Thu, Jul 4, 2019 at 12:18 PM Anton Vinogradov  
> > > > wrote:
> > > > 
> > > > > Folks,
> > > > > 
> > > > > Just a minor update.
> > > > > 
> > > > > RunAll [1] with enabled ReadRepair proxy is almost green now (~10
> > 
> > tests
> > > > > left, started with 6k :)).
> > > > > During the analisys, I've found some tests with
> > > > > - 

Re: Read Repair (ex. Consistency Check) - review request #2

2019-07-11 Thread Anton Vinogradov
Slava,

Thanks for your review first!

>> Anyway, I left some comments in your pull-request at github. Please take
a
>> look. The most annoying thing is poorly documented code :(
Since your comments mostly about Javadoc (does this mean that my solution
is so great that you ask me only to fix Javadocs :) ?),
I'd like to propose you to provide fixes as a PR since you have a vision of
how it should be made. I'll review them and merge shortly.

>> By the way, is it required to add test related to fail-over scenarios?
The best check is to use RR at real code.
For example, I'm injecting RR now to the test with concurrent modifications
and restarts [1].

>> I just checked, the IEP page and I still cannot find Jira tickets that
>> should cover existing limitations/improvements.
>> I would suggest creating the following tasks, at least:
Mostly agree with you, but
- MVCC is not production ready,
- not sure near support really required,
- metrics are better for monitoring, but the Event is enough for my wish to
cover AI with consistency check,
- do we really need Platforms and Thin Client support?
Also, we should not produce stillborn issue.
All limitations listed at proxy creation method and they definitely are not
showstoppers and may be fixed later if someone interested.
Сoming back to AI 3.0 discussion, we have A LOT of features and it's almost
impossible (require much more time that feature's cost) to support them all.
I will be pretty happy in case someone will do this and provide help if
necessary!

>> Perhaps, it would be a good idea to think about the recovery too
Do you mean per partition check and recovery?
That's a good idea, but I found it's not easy to imagine API to for such
tool.
In case you ready to assist with proper API/design this will definitely
help.

[1] https://issues.apache.org/jira/browse/IGNITE-11973

On Wed, Jul 10, 2019 at 3:43 PM Вячеслав Коптилин 
wrote:

> Perhaps, it would be a good idea to think about the recovery tool/
> control-utility command that will allow achieving the same goal.
> If I am not mistaken it was already proposed in the email thread.
>
> Thanks,
> S.
>
> ср, 10 июл. 2019 г. в 15:33, Вячеслав Коптилин :
>
> > Hi Anton,
> >
> > Well, the ReadRepair feature is finally merged and that is good news :)
> >
> > Unfortunately, I cannot find a consensus about the whole functionality in
> > any of these topics:
> >  -
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Consistency-check-and-fix-review-request-td41629.html
> >  -
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Read-Repair-ex-Consistency-Check-review-request-2-td42421.html
> > Also, there are no comments/discussion in JIRA. That makes me sad :(
> > especially when a feature is huge, not obvious and involves changing
> public
> > API (and that is the case, I think).
> >
> > Anyway, I left some comments in your pull-request at github. Please take
> a
> > look. The most annoying thing is poorly documented code :(
> > By the way, is it required to add test related to fail-over scenarios?
> >
> > I just checked, the IEP page and I still cannot find Jira tickets that
> > should cover existing limitations/improvements.
> > I would suggest creating the following tasks, at least:
> >  - MVCC support
> >  - Near caches
> >  - Additional metrics (number of violations, number of repaired entries
> > etc)
> >  - Ignite C++ (It looks like, .Net is already aware of that feature)
> >  - Thin clients support
> >  - Perhaps, it would be useful to support different strategies to resolve
> > inconsistencies
> >
> > What do you think?
> >
> > Thanks,
> > S.
> >
> >
> > ср, 10 июл. 2019 г. в 10:16, Anton Vinogradov :
> >
> >> Folks,
> >>
> >> Thanks to everyone for tips and reviews.
> >> Yardstick checked, no performance drop found.
> >> Additional measurement: RR get() is just up to 7% slower than regular
> get
> >> on real benchmarks (8 clients, 4 servers, 3 backups)
> >> Code merged to the master.
> >> "Must have" tasks created and attached to the IEP.
> >>
> >> On Thu, Jul 4, 2019 at 12:18 PM Anton Vinogradov  wrote:
> >>
> >> > Folks,
> >> >
> >> > Just a minor update.
> >> >
> >> > RunAll [1] with enabled ReadRepair proxy is almost green now (~10
> tests
> >> > left, started with 6k :)).
> >> > During the analisys, I've found some tests with
> >> > - unexpected repairs at tx caches
> >> > - inconsistent state after the test finished (different entries across
> >> the
> >> > topology)
> >> > For example,
> >> > - testInvokeAllAppliedOnceOnBinaryTypeRegistration generates obsolete
> >> > versions on backups in case of retry, fixed [2]
> >> > - initial cache load generates not equal versions on backups, fixed
> [3]
> >> > - testAccountTxNodeRestart causes unexpected repairs (entries have
> >> > different versions), to be investigated.
> >> >
> >> > What's next?
> >> >
> >> > 1) Going to merge the solution once "RunAll with ReadRepair enabled"
> >> > becomes fully green.
> >> > 2) Going to add special check