Re: Request for review : IGNITE-3303 Apache Flink Integration - Flink source

[Saikat Maitra]

Hi Alexey,

Thank you for reviewing the changes and sharing feedback, I am updating the
PR. I will share the changes shortly.

Regards,
Saikat

On Tue, Sep 4, 2018 at 10:59 AM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> Hello Saikat,
>
> I see a few fellow Igniters added some comments to your PR (including me).
> I believe the PR can be merged after you address them.
>
> Thanks,
> AG
>
> пт, 31 авг. 2018 г. в 3:11, Saikat Maitra :
>
> > Thank you, Denis
> >
> > Regards,
> > Saikat
> >
> > On Thu, Aug 30, 2018 at 7:01 PM, Denis Magda  wrote:
> >
> > > Hello Saikat,
> > >
> > > Hopefully, someone from the community will review the changes in the
> > > nearest time.
> > >
> > > --
> > > Denis
> > >
> > > On Thu, Aug 30, 2018 at 4:37 PM Saikat Maitra  >
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > The changes for IGNITE-3303 for IgniteSource is complete. This will
> > help
> > > is
> > > > streaming data from Ignite cluster and process, filter, transform and
> > > > publish it back to Ignite using IgniteSink or in any other data sink.
> > > >
> > > > I was hoping if the changes can be approved I can go ahead merge the
> > > > changes.
> > > >
> > > >
> > > > Regards,
> > > > Saikat
> > > >
> > > >
> > > >
> > > > On Tue, Aug 28, 2018 at 12:56 AM, Saikat Maitra <
> > saikat.mai...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Andrew,
> > > > >
> > > > > As discussed I have incorporated the changes. Please review and let
> > me
> > > > > know if any changes required.
> > > > >
> > > > > Regards,
> > > > > Saikat
> > > > >
> > > > > On Mon, Aug 27, 2018 at 1:45 AM, Saikat Maitra <
> > > saikat.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> I have updated the PR with additional tests.
> > > > >>
> > > > >> Please review and share feedback.
> > > > >>
> > > > >> This PR is related to IgniteSink but allows to stream data from
> > > Ignite.
> > > > >>
> > > > >> PR https://github.com/apache/ignite/pull/870/files
> > > > >>
> > > > >> Review https://reviews.ignite.apache.
> org/ignite/review/IGNT-CR-135
> > > > >>
> > > > >> Regards,
> > > > >> Saikat
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Unknown known issue on cache rebalancing delayed

[Roman Shtykh]

Anton, Maxim, thanks for following up! Looks like a good enough trade-off.
Sorry, couldn't catch the conversation because of the different time zone ;)

On Tuesday, September 4, 2018, 7:54:05 p.m. GMT+9, Anton Vinogradov 
 wrote:  
 
 Maxim,

Let's create a branch with 10 checks of Sync and 10 checks of Async.
Then, run it 20 times at TC.
This should be enough I think.

вт, 4 сент. 2018 г. в 13:09, Maxim Muzafarov :

> Anton,
>
> I agree with you 20 time is not enough. I've checked the single run of the
> test class - it consumes ~7min per each execution.
> CacheSuite8 total execution timeout - 210 min, so we can perform only 30
> class execution in this suite. Our strategy here is
> to `20 times within single` and put into the TC queue 50 runs. Total ~7000
> min or 5 days.
>
> Not sure that we should perform exactly 1000 executions, hopefully, we will
> stop adding to the queue new tasks at some point.
>
> On Tue, 4 Sep 2018 at 12:59 Anton Vinogradov  wrote:
>
> > Maxim,
> > 20 is not 1k :)
> > Also, you forgot to check GridCacheRebalancingAsyncSelfTest
> >
> > I'm not sure we should have exactly 1k runs, but 20 is definitely not
> > enough.
> >
> > Roman,
> > I propose to use IDEA "run until failure" feature and perform test
> locally
> > (at your PC) while you're not using PC.
> >
> > вт, 4 сент. 2018 г. в 12:51, Maxim Muzafarov :
> >
> > > Roman, Anton,
> > >
> > > I've already created additional PR [2] all and run it on TC [1].
> > > Please, follow up with the results.
> > >
> > > [1]
> > >
> > >
> >
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_Cache8=buildTypeStatusDiv_IgniteTests24Java8=pull%2F4676%2Fhead
> > > [2] https://github.com/apache/ignite/pull/4676/files
> > >
> > >
> > > On Tue, 4 Sep 2018 at 12:46 Roman Shtykh 
> > > wrote:
> > >
> > > > Anton,
> > > > Thank you. I would like to recheck it. How can this (1_000 runs) be
> > done
> > > > in TC?
> > > >
> > > >
> > > >    On Tuesday, September 4, 2018, 5:42:01 p.m. GMT+9, Anton
> > Vinogradov <
> > > > a...@apache.org> wrote:
> > > >
> > > >  Roman,
> > > >
> > > > I see you uncommented this line.
> > > > I do not remember deadlock detail, but I remember it was the
> extremely
> > > rare
> > > > case.
> > > > I found and "fixed" it some days before merge when I had 24x7 sanity
> > > check
> > > > week :)
> > > >
> > > > So, I propose to have at least 1_000 runs of this tests before
> keeping
> > > this
> > > > uncommented.
> > > >
> > > >
> > > >
> > > > вт, 21 авг. 2018 г. в 11:08, Maxim Muzafarov :
> > > >
> > > > > Roman,
> > > > >
> > > > > I worked recently on rebalance improvements and haven't found any
> > > > problems
> > > > > with delayed cache rebalacne.
> > > > > Agree with you - let's uncomment this and remove scary comment.
> Will
> > > you
> > > > > create a ticket for it?
> > > > >
> > > > > In case of any problems we can easily detec deadlock with newly
> > > > configured
> > > > > `FailureHandler`.
> > > > >
> > > > > On Tue, 21 Aug 2018 at 03:49 Roman Shtykh 
> wrote:
> > > > >
> > > > > > Hi Maxim,
> > > > > >
> > > > > > I have some issues with a cluster with rebalance delay enabled,
> but
> > > > need
> > > > > > to check more -- if I find it's related I'll share.
> > > > > > Just wanted to make sure it's not an issue anymore from someone
> > > working
> > > > > on
> > > > > > rebalancing. We should remove that comment then, it looks scary
> :)
> > > > > >
> > > > > > --
> > > > > > Roman Shtykh
> > > > > >
> > > > > >
> > > > > > On Tuesday, August 21, 2018, 12:49:00 a.m. GMT+9, Maxim
> Muzafarov <
> > > > > > maxmu...@gmail.com> wrote:
> > > > > >
> > > > > >
> > > > > > Hello Roman,
> > > > > >
> > > > > > Did you faced with real issue of delayed rebalance or it's just
> > only
> > > > for
> > > > > > your personal interest?
> > > > > > If yes, please, share details and we will try to help you.
> > > > > >
> > > > > > As for this comment I don't think he is actual. That change was
> in
> > > > 2015.
> > > > > > Much has changed
> > > > > > within rebalance process since that time. I've uncommented it and
> > > > > > rechecked with that
> > > > > > cache configuration and haven't seen any failed tests or issues.
> > > > > >
> > > > > > Probably, that problem was about cache in SYNC mode does not
> start
> > > util
> > > > > it
> > > > > > loads all data
> > > > > > from other nodes. But currently delayed rebalance works the same
> > way
> > > as
> > > > > > IgniteCache#rebalance(),
> > > > > > so you can `setRebalanceDelay` to `-1` and call it manually to
> > check.
> > > > > >
> > > > > > On Mon, 20 Aug 2018 at 11:19 Roman Shtykh
> >  > > >
> > > > > > wrote:
> > > > > >
> > > > > > Igniters,
> > > > > > I have found "Known issue, possible deadlock in case of low
> > priority
> > > > > cache
> > > > > > rebalancing delayed" comment in
> > > > > > GridCacheRebalancingSyncSelfTest#getConfiguration.Can you please
> > > > explain
> > > > > > when using rebalance delay can be an issue 

Re: Compression prototype

[Dmitriy Setrakyan]

In my view, dictionary of 1024 bytes is not going to be nearly enough.

On Tue, Sep 4, 2018 at 8:06 AM, Ilya Kasnacheev 
wrote:

> Hello!
>
> In case of Apache Ignite, most of savings is due to BinaryObject format,
> which encodes types and fields with byte sequences. Any enum/string flags
> will also get in dictionary. And then as it processes a record it fills up
> its individual dictionary.
>
> But, in one cache, most if not all entries have identical BinaryObject
> layout so a tiny dictionary covers that case. Compression algorithms are
> not very keen on large dictionaries, preferring to work with local
> regularities in byte stream.
>
> E.g. if we have large entries in cache with low BinaryObject overhead,
> they're served just fine by "generic" compression.
>
> All of the above is my speculations, actually. I just observe that on a
> large data set, compression ratio is around 0.4 (2.5x) with a dictionary of
> 1024 bytes. The rest is black box.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 4 сент. 2018 г. в 17:16, Dmitriy Setrakyan :
>
> > On Tue, Sep 4, 2018 at 2:55 AM, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com
> > >
> > wrote:
> >
> > > Hello!
> > >
> > > Each node has a local dictionary (per node currently, per cache
> planned).
> > > Dictionary is never shared between nodes. As data patterns shift,
> > > dictionary rotation is also planned.
> > >
> > > With Zstd, the best dictionary size seems to be 1024 bytes. I imagine
> It
> > is
> > > enough to store common BinaryObject boilerplate, and everything else is
> > > compressed on the fly. The source sample is 16k records.
> > >
> > >
> > Thanks, Ilya, understood. I think per-cache is a better idea. However, I
> > have a question about dictionary size. Ignite stores TBs of data. How do
> > you plan the dictionary to fit in 1K bytes?
> >
> > D.
> >
>

Re: Python thin client

[Dmitriy Setrakyan]

Got it, sounds good!

On Tue, Sep 4, 2018 at 10:54 AM, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:

> Dmitriy,
>
> It would be quite messy to implement with Python modular system.
>
> First of all, Python 2 and Python 3 are different languages with a small
> common subset of syntax rules. That's why what we see in a stack trace is a
> syntax error, and not a “missing feature” error.
>
> Second, there is no single entry point in Python code. User is allowed to
> import any name, from any module, in any order. In fact the module is run
> when it first discovered by CPython during any `import` operation, and that
> is how the imported entities are created and initialized: by the chain of
> imports.
>
> It means for us, that implementing even a simple compatibility message in
> Python 2 requires any module in our program (or at least all the modules,
> that represent the public API of our library) to consist entirely of a
> valid Python 2 code.
>
> It can be achieved by writing stubs with a version check, putting said
> stubs in place of real modules, and proxying all the calls through the
> conditional imports. It would take a small effort once, but make code less
> readable and harder to maintain for the rest of its life cycle. Should we,
> for example, provide two testing environments − for the main program and
> for Python 2-compatible stubs?
>
> As far as I know, no Python developer is making such efforts nowadays.
> There are some projects with a long history, that achieve 2/3-compatibility
> through the use of restricted syntax and external packages like `six`, or
> simply support two separate versions. Most of the new projects are creating
> with the latest Python 3, pip and virtualenv in mind.
>
> I took the idea of my `setup.py` solution from the Django project, which
> is dropped Python 2 support not long ago. Its setup relies on `setuptools`
> 9+ option, or otherwise displays a simple banner; the banner is likely to
> be buried under the further cryptic output of old setuptools, but it is
> better than nothing.
>
>
> On 9/5/18 2:21 AM, Dmitriy Setrakyan wrote:
>
>> Dmitriy,
>>
>> setuptools sounds like an installation step. Does it make sense to add a
>> check during startup of a client?
>>
>> D.
>>
>> On Tue, Sep 4, 2018 at 7:25 AM, Dmitry Melnichuk <
>> dmitry.melnic...@nobitlost.com> wrote:
>>
>> Nikolay,
>>>
>>> There is indeed a feature in `setuptools` I just learned about, which
>>> would help in this case (and I believe the case you demonstrated can be
>>> quite typical, thank you for pointing it out). It gives user a clever
>>> message in the end of a stack trace like
>>>
>>> UnsupportedPythonVersion: pyignite requires Python '>=3.4' but the
 running Python is 2.7.15

>>>
>>> This feature is not 100% bullet-proof, and it will not help users who
>>> will
>>> try to install my client other way than with `pip` or `setup.py`.
>>> It will also be less helpful with old versions of `pip` (before 9.0).
>>> However, it should cover most situations.
>>>
>>> On 9/4/18 7:15 PM, Nikolay Izhikov wrote:
>>>
>>> Hello, Dmitry.

 I understand that for experienced Python developer it obvious from
 stack trace I send.

 But can we check python version on startup? And print big fat error
 message "You are using wrong python version".



>>>  From my experience, there are some tickets in Ignite that should be
 implemented in various thin clients. It a very trivial changes, but
 lack of testability makes this task harder then steel.

 I think a .Net DEVNOTES are very good example.

 Please, be gentle with your fellow contributors and make DEVNOTES as
 clear as possible.


>>>
>>
>

Re: MVCC and transactional SQL is merged to master

[Lucas Beeler]

Congratulations!

--
Lucas BEELER
Technical Consultant, Professional Services
GridGain Systems, Inc.
http://www.gridgain.com

> On Aug 30, 2018, at 12:41 PM, Dmitriy Setrakyan  wrote:
> 
> Very nice! Looking forward to seeing this functionality in 2.7 release.
> 
> On Thu, Aug 30, 2018 at 5:22 AM, Yakov Zhdanov  wrote:
> 
>> Great news, Vladimir! Congratulations!
>> 
>> --Yakov
>> 
>> 2018-08-30 15:15 GMT+03:00 Vladimir Ozerov :
>> 
>>> Igniters,
>>> 
>>> I am glad to announce that we finally merged MVCC and transactional SQL
>>> support to master branch.
>>> 
>>> This long journey started more than a year ago with multiple design
>>> brainstorm sessions, conducted by Apache Ignite fellows - Semen Boikov,
>>> Alexey Goncharuk, Sergi Vladykin.
>>> 
>>> As things had became more clear, we gradually switched to active
>>> development phase in November 2017. Since then we implemented new
>>> transactional model based on multi-version approach and snapshot
>> isolation,
>>> and almost fully reworked SQL engine to support transactions.
>>> 
>>> But this is not the end of the story. In Apache Ignite 2.7 we expect to
>>> release transactional SQL as "release candidate". To achieve this we
>> still
>>> need to implement a number of things, such as new transactional protocol
>>> for key-value API, historical rebalance, continuous queries. Between AI
>> 2.7
>>> and AI 2.8 we will work on several not-yet-supported cache operations,
>> and
>>> also will focus on performance and stability.
>>> 
>>> I would like to thank all community members, who worked hard to make MVCC
>>> happen: Igor Seliverstov, Alexander Paschenko, Sergey Kalashnikov, igor
>>> Sapego, Roman Kondakov, Pavel Kuznetsov, Ivan Pavlukihn, Andrey
>> Mashenkov,
>>> and many other contributors who helped us with design, testing and
>>> benchmarking.
>>> 
>>> Release notes and documentation will be prepared by AI 2.7 release.
>>> 
>>> Please feel free to ask any questions about the feature here.
>>> 
>>> Vladimir.
>>> 
>> 

Re: ScanQuery fails with OutOfMemory when iterating

[Denis Magda]

Hi Slava,

Thanks for looking into it. Looks like exactly what happened on the user
side.

--
Denis

On Tue, Sep 4, 2018 at 5:43 AM Вячеслав Коптилин 
wrote:

> Hi Denis,
>
> It looks like a known issue
> https://issues.apache.org/jira/browse/IGNITE-8892
> It is already fixed and will be available in Apache Ignite 2.7
>
> Thanks,
> S.
>
> вс, 2 сент. 2018 г. в 17:40, Denis Magda :
>
> > Igniters,
> >
> > A user reported the issue on SO:
> >
> >
> https://stackoverflow.com/questions/52117160/ignite-consumes-all-memory-and-fails-with-outofmemory-when-iterating-over-cache
> >
> > Does it sound familiar? Were we fixing something like that before?
> >
> > Please scan query and Ignite memory experts join the conversation there.
> >
> > --
> > Denis
> >
>

Re: IGNITE-640: multimap initial implementation

[Denis Magda]

Amir, Anton,

How is dev/review process going? Is there any chance we get this capability
into 2.7?

--
Denis

On Mon, Jul 9, 2018 at 10:27 PM Amir Akhmedov 
wrote:

> Hi Anton,
>
> I checked your last comments in the ticket and left some responses. Please
> check them and let me know
>
> Thanks,
> Amir
>
> P.S. do you mind to have a chat/call through gitter/Skype to discuss the
> details? Sometimes 5 minutes of chat can be more productive than long
> running email chains. Please, do not hesitate to directly email me if you
> mind to have a chat/call.
>
> On Wed, Jun 27, 2018 at 11:26 AM Anton Vinogradov  wrote:
>
> > Sure,
> > Hope it will be tomorrow
> >
> > ср, 27 июн. 2018 г. в 18:11, Amir Akhmedov :
> >
> > > Anton V,
> > > I put some comments into jira ticket. Can you please take a look once
> you
> > > have a chance?
> > >
> > > Thanks,
> > > Amir
> > >
> > > On Mon, Jun 18, 2018, 7:54 AM Anton Vinogradov  wrote:
> > >
> > > > Amir,
> > > >
> > > > Everything is fine, I'll check changes this week.
> > > >
> > > > вс, 17 июн. 2018 г. в 6:09, Amir Akhmedov :
> > > >
> > > > > Anton,
> > > > > I created a news PR [1]. Since it includes the same changes I did
> not
> > > run
> > > > > TC tests on it. Please let me know if you think otherwise.
> > > > >
> > > > > [1]  https://github.com/apache/ignite/pull/4207
> > > > >
> > > > > Thanks,
> > > > > Amir
> > > > >
> > > > >
> > > > > On Wed, Jun 13, 2018 at 8:38 AM Anton Vinogradov 
> > > wrote:
> > > > >
> > > > > > Amir,
> > > > > >
> > > > > > Thanks for attempt.
> > > > > > As far as I can see you have all changes at this commit:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/pull/3926/commits/cd0e50e05d3860788378ebf1a29dc0525460872a
> > > > > >
> > > > > > You can simply apply it to local branch based on master by patch
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/commit/cd0e50e05d3860788378ebf1a29dc0525460872a.patch
> > > > > >
> > > > > > In case you use IDEA, just apply patch from clipboard, and that's
> > > will
> > > > be
> > > > > > you PR.
> > > > > >
> > > > > > BTW, next time you can use easiest way to squash your changes -
> > just
> > > to
> > > > > > pull all changes from existing PR with squash
> > > > > > > git pull https://github.com/apache/ignite.git pull/XXX/head
> > > --squash
> > > > > >
> > > > > >
> > > > > >
> > > > > > вт, 5 июн. 2018 г. в 19:34, Amir Akhmedov <
> amir.akhme...@gmail.com
> > >:
> > > > > >
> > > > > > > Dmitry P., Anton V.,
> > > > > > > I made some changes and updated the ticket. Also as was asked I
> > > tried
> > > > > to
> > > > > > > squash the commits into one but looks like I screwed up
> > everything
> > > > and
> > > > > > the
> > > > > > > PR now looks completely terrible. Since I'm not an advanced git
> > > user,
> > > > > > could
> > > > > > > you please check the PR and let me know if anything could be
> done
> > > > > there?
> > > > > > If
> > > > > > > not I will try to create a new PR.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Amir
> > > > > > >
> > > > > > > On Tue, May 29, 2018 at 10:37 AM, Dmitry Pavlov <
> > > > dpavlov@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Amir,
> > > > > > > >
> > > > > > > > As far as I know, several Igniters provided some feedback in
> > > > ticket.
> > > > > > Are
> > > > > > > > you agree?
> > > > > > > >
> > > > > > > > Sincerely,
> > > > > > > > Dmitriy Pavlov
> > > > > > > >
> > > > > > > > чт, 10 мая 2018 г. в 20:01, Dmitry Pavlov <
> > dpavlov@gmail.com
> > > >:
> > > > > > > >
> > > > > > > > > Hi Amir,
> > > > > > > > >
> > > > > > > > > This is a very necessary contribution, the patch defenetely
> > > will
> > > > > not
> > > > > > be
> > > > > > > > > ignored.
> > > > > > > > >
> > > > > > > > > Igniters,
> > > > > > > > >
> > > > > > > > > Who can make a review from the committers?
> > > > > > > > >
> > > > > > > > > Sincerely,
> > > > > > > > > Dmitriy Pavlov
> > > > > > > > >
> > > > > > > > > вт, 8 мая 2018 г. в 5:52, Amir Akhmedov <
> > > amir.akhme...@gmail.com
> > > > >:
> > > > > > > > >
> > > > > > > > >> Hi Igniters,
> > > > > > > > >>
> > > > > > > > >> Can someone take a look at this PR please?
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> Amir
> > > > > > > > >>
> > > > > > > > >> On Mon, Apr 30, 2018 at 5:28 AM, Pavel Tupitsyn <
> > > > > > ptupit...@apache.org
> > > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi Amir,
> > > > > > > > >> >
> > > > > > > > >> > I have filed [1] for multimap in .NET, it will be done
> > > later.
> > > > > > > > >> > In order to fix IgniteParityTest failures, please add
> the
> > > > > > following
> > > > > > > to
> > > > > > > > >> > MissingMembers array there:
> > > > > > > > >> >
> > > > > > > > >> > "multimap" // IGNITE-8425
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> > Pavel
> > > > > > > > >> >
> > > > 

Re: Hi... trying to write a test case for a pacth ... need help....

[Dmitriy Setrakyan]

Hi Paul, what Jira ticket are you working on?

On Tue, Sep 4, 2018 at 7:02 AM, Paul Anderson  wrote:

> ... The code is done ready to go, but the test suite looks huge, I know the
> two test classes I have to augment... but I am unsure of a couple of
> things.
>
> 1) how do I run a specific test (command line)
> 2) how do I start (make sure is started) an ignite instance
>

[jira] [Created] (IGNITE-9467) Custom SQL function: Hexadecimal string with odd number of characters

[Joe Feise (JIRA)]

Joe Feise created IGNITE-9467:
-

 Summary: Custom SQL function: Hexadecimal string with odd number 
of characters
 Key: IGNITE-9467
 URL: https://issues.apache.org/jira/browse/IGNITE-9467
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.6
Reporter: Joe Feise


I have been trying to use a custom SQL function which expects an array of 
doubles, as explained here: 
[https://stackoverflow.com/questions/33097103/how-to-sum-an-array-of-doubles-with-ignite-data-grid-sql]

I get this error:

select arraysum(value_double) from time_series;
[Hexadecimal string with odd number of characters: "-9.386298778519813E-152"; 
SQL statement:
select arraysum(value_double) from time_series 
[90003-195]|http://192.168.6.247:64909/query.do?jsessionid=456cb3c32fa9887ebfaaf775c4f3beff]
 90003/90003
 
It seems that Ignite tries to read the double value as a hex string.
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4683: Ignite 2.5.1 p12 volatile

[ascherbakoff]

GitHub user ascherbakoff opened a pull request:

https://github.com/apache/ignite/pull/4683

Ignite 2.5.1 p12 volatile



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite 
ignite-2.5.1-p12-volatile

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4683.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4683


commit acfef907db8204ac93fc235770f36bf7f61269c3
Author: Ilya Kasnacheev 
Date:   2018-04-17T16:50:51Z

IGNITE-2766 Fix .net test. - Fixes #3853.

Signed-off-by: dpavlov 

(cherry picked from commit 96cb795)

commit 6cea78e4e13fe43555b78dcd683366f54c6816ff
Author: Andrey Kuznetsov 
Date:   2018-04-17T16:58:43Z

IGNITE-7770 Test testRandomMixedTxConfigurations partialy fixed

Signed-off-by: Andrey Gura 

commit e394693a7389b4daff328827abdb1dcd28783f66
Author: Maxim Muzafarov 
Date:   2018-04-17T18:18:36Z

IGNITE-8301 testReconnectCacheDestroyedAndCreated should excpect recreated 
client cache - Fixes #3856.

Signed-off-by: dpavlov 

(cherry picked from commit 56be24b)

commit 4685ebe5f5dda4023980398806e222fada895e26
Author: Vasiliy Sisko 
Date:   2018-04-18T03:44:44Z

IGNITE-8140 Web Console: Fixed code generation for large numbers in 
configuration params.

(cherry picked from commit eda5fe7)

commit 5efc589fcabffdb29cd6dfe0e7323bc91db47703
Author: Ilya Borisov 
Date:   2018-04-18T04:39:41Z

IGNITE-8294 Web Console: Move "Beta" ribbon to the left.

(cherry picked from commit 69606e4)

commit af46856d7f7ec88aa663865a108900abb8314ffe
Author: oleg-ostanin 
Date:   2018-04-17T17:58:53Z

IGNITE-8274 sqlline.sh script uses JAVA_HOME now

Signed-off-by: Andrey Gura 

(cherry picked from commit c3ff274)

commit ba4e337068ebfcc4bbb93509166f118225fe0cb4
Author: Dmitriy Shabalin 
Date:   2018-04-18T11:43:13Z

IGNITE-8298 Web Console: Fixed tables UI issues.

(cherry picked from commit a050436)

commit 25b25f271bc89d77013e1cda2bad30d941e1c8ad
Author: Pavel Kovalenko 
Date:   2018-04-18T12:57:45Z

IGNITE-8122 Restore partition state from WAL if no checkpoints are done - 
Fixes #3745.

Signed-off-by: Alexey Goncharuk 

commit 4ab86f13cbc7fef68f5fd39354e242fe6cca285a
Author: skalashnikov 
Date:   2018-04-18T13:37:20Z

IGNITE-7512 Check for null before validateKeyAndValue in 
GridDhtAtomicCache.updateWithBatch - Fixes #3429.

Signed-off-by: Alexey Goncharuk 

commit 7f318da5b357cca1a7e483a37d2556fbd56fc2a7
Author: Ilya Lantukh 
Date:   2018-04-18T14:56:56Z

IGNITE-8017 Disable WAL during initial rebalancing

commit b80fdbedc11b48aa89c6f29146363cec6f552abb
Author: Ilya Lantukh 
Date:   2018-04-18T16:04:39Z

IGNITE-8276 Fixed incorrect assertion during initialValue - Fixes #3827.

Signed-off-by: Alexey Goncharuk 

commit 5cd32329fe5f303eaf369519771ee22f2a2cf822
Author: Pavel Kovalenko 
Date:   2018-04-18T16:41:44Z

IGNITE-8116 Fixed historical rebalance from WAL

commit 442d87d5799c27ef7258c7ac36e3e0b312676713
Author: Pavel Kovalenko 
Date:   2018-04-18T18:03:32Z

IGNITE-8243 Fixed possible memory leak. Added latch manager to diagnostic 
messages. - Fixes #3850.

Signed-off-by: dpavlov 

(cherry picked from commit 6c6f094)

commit 5bc9fe9fa4e51c9af677f8120528746f345d990f
Author: dpavlov 
Date:   2018-04-18T18:15:17Z

IGNITE-8302 Reduce test IO consumption in case Direct IO is enabled, fixes 
flakyness of testPageRecoveryAfterFileCorruption - Fixes #3865.

Signed-off-by: dpavlov 

(cherry picked from commit d853ebb)

commit 3696c064f02567f471420ced50e2786a4f83d8c9
Author: dpavlov 
Date:   2018-04-18T18:16:35Z

IGNITE-8302 Licenses fix for Reduce test IO consumption in case Direct IO

(cherry picked from commit e7716f4)

commit 061a5a82c95f4f926c7789764901b70383152df1
Author: shroman 
Date:   2018-04-18T10:38:52Z

IGNITE-8143: Modified fetching EXTERNAL_LIBS for docker container. - Fixes 
#3751.

Signed-off-by: shroman 

commit 83d488b083706e1d04200b4e603db1dc337d8fb4
Author: Dmitriy Shabalin 
Date:   2018-04-19T08:16:18Z

IGNITE-8298 Web Console: Fixed loader under Safari.

(cherry picked from commit 0897309)

commit f9a0ee18f9d88b8f278d27dddc5f7cd3a52ef5c2
Author: Nikita Amelchev 
Date:   2018-04-19T09:57:06Z

IGNITE-: Support Time data type in BinaryObjectImpl.writeFieldByOrder. 
This closes #2878.

commit a095b98e1123f9562cbbc85d1c1c4b4221542c71
Author: Ivan Daschinskiy 
Date:   2018-04-19T12:25:23Z

IGNITE-8021 Fixed tests - Fixes #3864.

Signed-off-by: Alexey Goncharuk 

commit 27385e73183fe4bbc2f366f975facaefbcad2a6e
Author: Alexey Goncharuk 
Date:   2018-04-18T07:59:46Z

IGNITE-8201 Fixed NPE

commit 

Re: Python thin client

[Dmitry Melnichuk]

Dmitriy,

It would be quite messy to implement with Python modular system.

First of all, Python 2 and Python 3 are different languages with a small 
common subset of syntax rules. That's why what we see in a stack trace 
is a syntax error, and not a “missing feature” error.


Second, there is no single entry point in Python code. User is allowed 
to import any name, from any module, in any order. In fact the module is 
run when it first discovered by CPython during any `import` operation, 
and that is how the imported entities are created and initialized: by 
the chain of imports.


It means for us, that implementing even a simple compatibility message 
in Python 2 requires any module in our program (or at least all the 
modules, that represent the public API of our library) to consist 
entirely of a valid Python 2 code.


It can be achieved by writing stubs with a version check, putting said 
stubs in place of real modules, and proxying all the calls through the 
conditional imports. It would take a small effort once, but make code 
less readable and harder to maintain for the rest of its life cycle. 
Should we, for example, provide two testing environments − for the main 
program and for Python 2-compatible stubs?


As far as I know, no Python developer is making such efforts nowadays. 
There are some projects with a long history, that achieve 
2/3-compatibility through the use of restricted syntax and external 
packages like `six`, or simply support two separate versions. Most of 
the new projects are creating with the latest Python 3, pip and 
virtualenv in mind.


I took the idea of my `setup.py` solution from the Django project, which 
is dropped Python 2 support not long ago. Its setup relies on 
`setuptools` 9+ option, or otherwise displays a simple banner; the 
banner is likely to be buried under the further cryptic output of old 
setuptools, but it is better than nothing.


On 9/5/18 2:21 AM, Dmitriy Setrakyan wrote:

Dmitriy,

setuptools sounds like an installation step. Does it make sense to add a
check during startup of a client?

D.

On Tue, Sep 4, 2018 at 7:25 AM, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:


Nikolay,

There is indeed a feature in `setuptools` I just learned about, which
would help in this case (and I believe the case you demonstrated can be
quite typical, thank you for pointing it out). It gives user a clever
message in the end of a stack trace like


UnsupportedPythonVersion: pyignite requires Python '>=3.4' but the
running Python is 2.7.15


This feature is not 100% bullet-proof, and it will not help users who will
try to install my client other way than with `pip` or `setup.py`.
It will also be less helpful with old versions of `pip` (before 9.0).
However, it should cover most situations.

On 9/4/18 7:15 PM, Nikolay Izhikov wrote:


Hello, Dmitry.

I understand that for experienced Python developer it obvious from
stack trace I send.

But can we check python version on startup? And print big fat error
message "You are using wrong python version".





 From my experience, there are some tickets in Ignite that should be
implemented in various thin clients. It a very trivial changes, but
lack of testability makes this task harder then steel.

I think a .Net DEVNOTES are very good example.

Please, be gentle with your fellow contributors and make DEVNOTES as
clear as possible.

[jira] [Created] (IGNITE-9466) AsyncFileIO may not close channel after method close invocation

[Sergey Antonov (JIRA)]

Sergey Antonov created IGNITE-9466:
--

 Summary: AsyncFileIO may not close channel after method close 
invocation
 Key: IGNITE-9466
 URL: https://issues.apache.org/jira/browse/IGNITE-9466
 Project: Ignite
  Issue Type: Bug
Reporter: Sergey Antonov


 
{code:java}
/** {@inheritDoc} */
@Override public void close() throws IOException {
for (ChannelOpFuture asyncFut : asyncFuts) {
try {
asyncFut.getUninterruptibly(); // Ignore interrupts while waiting 
for channel close.
}
catch (IgniteCheckedException e) {
throw new IOException(e);
}
}

ch.close();
}{code}
If `asyncFut.getUninterruptibly()` throw exception, the channel associated with 
AsyncFileIO will not be closed.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Python thin client

[Dmitriy Setrakyan]

Dmitriy,

setuptools sounds like an installation step. Does it make sense to add a
check during startup of a client?

D.

On Tue, Sep 4, 2018 at 7:25 AM, Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:

> Nikolay,
>
> There is indeed a feature in `setuptools` I just learned about, which
> would help in this case (and I believe the case you demonstrated can be
> quite typical, thank you for pointing it out). It gives user a clever
> message in the end of a stack trace like
>
> > UnsupportedPythonVersion: pyignite requires Python '>=3.4' but the
> > running Python is 2.7.15
>
> This feature is not 100% bullet-proof, and it will not help users who will
> try to install my client other way than with `pip` or `setup.py`.
> It will also be less helpful with old versions of `pip` (before 9.0).
> However, it should cover most situations.
>
> On 9/4/18 7:15 PM, Nikolay Izhikov wrote:
>
>> Hello, Dmitry.
>>
>> I understand that for experienced Python developer it obvious from
>> stack trace I send.
>>
>> But can we check python version on startup? And print big fat error
>> message "You are using wrong python version".
>>
> >
>
>> From my experience, there are some tickets in Ignite that should be
>> implemented in various thin clients. It a very trivial changes, but
>> lack of testability makes this task harder then steel.
>>
>> I think a .Net DEVNOTES are very good example.
>>
>> Please, be gentle with your fellow contributors and make DEVNOTES as
>> clear as possible.
>>
>

Re: Failed to activate node components

[wt]

POC5-server.xml

  

Here is my config file



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Failed to activate node components

[wt]

this is how i am starting the service - 

IgniteConfiguration cfg =
Ignition.loadSpringBean("POC5-server.xml","grid.cfg");
Ignite ig =Ignition.start(cfg);
ig.cluster().active(true);


*i should point out that if i start it like this i don't get the error*

Ignite ig = Ignition.start("POC5-server.xml");
ig.cluster().active(true);






--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Table Names in Spark Catalog

[Nikolay Izhikov]

Hello, Stuart.

Sorry for the silence.

I was swamped the last couple of days.

I think you can go forward and implement suggested solution.
I'm -0 with it.
So no block from my side, but I'm still no happy with abstractions :).

В Пн, 03/09/2018 в 09:35 +0100, Stuart Macdonald пишет:
> Nikolay, Val, it would be good if we could reach agreement here so that I
> can make the necessary modifications before the 2.7 cutoff.
> 
> Nikolay - would you be comfortable if I went ahead and made database=schema?
> 
> Stuart.
> 
> On Mon, Aug 27, 2018 at 10:22 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
> 
> > Hi Nikolay,
> > 
> > I think it's actually pretty unfortunate that Spark uses term "database"
> > here, as it essentially refers to a schema in my view. Usually, database is
> > something you create a physical connection to, and connection is bind to
> > that database. To connect to another database you need to create a new
> > connection. In Spark, however, you can switch between "databases" within a
> > single session, which looks really weird to me because it's usually a
> > characteristic of a schema. Having said that, I understand your concern,
> > but I don't think there is an ideal solution.
> > 
> > As for your approach, I still don't understand how it will allow to fully
> > support schemas in catalog.
> > - How will you get a list of tables within a particular schema? In other
> > words, what would listTables() method return?
> > - How will you switch between the schemas?
> > - Etc.
> > 
> > I still think assuming database=schema is the best we can do here, but I
> > would be happy to hear another opinions from other community members.
> > 
> > OPTION_SCHEMA should definitely be introduced though (I thought we already
> > did, no?). CREATE TABLE will be supported with this ticket:
> > https://issues.apache.org/jira/browse/IGNITE-5780. For now we will have to
> > throw an exception if custom schema name is provided when creating a Spark
> > session, but table does not exist yet.
> > 
> > -Val
> > 
> > On Sun, Aug 26, 2018 at 7:56 AM Nikolay Izhikov 
> > wrote:
> > 
> > > Igniters,
> > > 
> > > Personally, I don't like the solution with database == schema name.
> > > 
> > > 1. I think we should try to use the right abstractions.
> > > schema == database doesn't sound right for me.
> > > 
> > > Do you want to answer to all of our users something like that:
> > > 
> > > - "How I can change Ignite SQL schema?"
> > > - "This is obvious, just use setDatabase("MY_SCHEMA_NAME")".
> > > 
> > > 2. I think we restrict whole solution with that decision.
> > > If Ignite will support multiple databases in the future we just don't
> > 
> > have
> > > a place for it.
> > > 
> > > I think we should do the following:
> > > 
> > > 1. IgniteExternalCatalog should be able to return *ALL* tables
> > > within Ignite instance.
> > > We shouldn't restrict tables list by schema by default.
> > > We should return tables with schema name - `schema.table`
> > > 
> > > 2. We should introduce `OPTION_SCHEMA` for a dataframe to specify
> > > a schema.
> > > 
> > > There is an issue with the second step: We can't use schema name
> > > in `CREATE TABLE` clause.
> > > This is restriction of current Ignite SQL.
> > > 
> > > I propose to make the following:
> > > 
> > > 1. For all write modes that requires the creation of table we
> > > should disallow usage of table outside of `SQL_PUBLIC`
> > > or usage of `OPTION_SCHEMA`. We should throw proper exception for
> > > this case.
> > > 
> > > 2. Create a ticket to support `CREATE TABLE` with custom schema
> > > name.
> > > 
> > > 3. After resolving ticket from step 2 we can add full support of
> > > custom schema to Spark integration.
> > > 
> > > 4. We should throw an exception if user try to use setDatabase.
> > > 
> > > Is that makes sense for you?
> > > 
> > > В Вс, 26/08/2018 в 14:09 +0100, Stuart Macdonald пишет:
> > > > I'll go ahead and make the changes to represent the schema name as the
> > > > database name for the purposes of the Spark catalog.
> > > > 
> > > > If anyone knows of an existing way to list all available schemata
> > 
> > within
> > > an
> > > > Ignite instance please let me know, otherwise the first task will be
> > > > creating that mechanism.
> > > > 
> > > > Stuart.
> > > > 
> > > > On Fri, Aug 24, 2018 at 6:23 PM Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com> wrote:
> > > > 
> > > > > Nikolay,
> > > > > 
> > > > > If there are multiple configuration in XML, IgniteContext will always
> > > 
> > > use
> > > > > only one of them. Looks like current approach simply doesn't work. I
> > > > > propose to report schema name as 'database' in Spark. If there are
> > > 
> > > multiple
> > > > > clients, you would create multiple sessions and multiple catalogs.
> > > > > 
> > > > > Makes sense?
> > > > > 
> > > > > -Val
> > > > > 
> > > > > On 

Re: Request for review : IGNITE-3303 Apache Flink Integration - Flink source

[Alexey Goncharuk]

Hello Saikat,

I see a few fellow Igniters added some comments to your PR (including me).
I believe the PR can be merged after you address them.

Thanks,
AG

пт, 31 авг. 2018 г. в 3:11, Saikat Maitra :

> Thank you, Denis
>
> Regards,
> Saikat
>
> On Thu, Aug 30, 2018 at 7:01 PM, Denis Magda  wrote:
>
> > Hello Saikat,
> >
> > Hopefully, someone from the community will review the changes in the
> > nearest time.
> >
> > --
> > Denis
> >
> > On Thu, Aug 30, 2018 at 4:37 PM Saikat Maitra 
> > wrote:
> >
> > > Hello,
> > >
> > > The changes for IGNITE-3303 for IgniteSource is complete. This will
> help
> > is
> > > streaming data from Ignite cluster and process, filter, transform and
> > > publish it back to Ignite using IgniteSink or in any other data sink.
> > >
> > > I was hoping if the changes can be approved I can go ahead merge the
> > > changes.
> > >
> > >
> > > Regards,
> > > Saikat
> > >
> > >
> > >
> > > On Tue, Aug 28, 2018 at 12:56 AM, Saikat Maitra <
> saikat.mai...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Andrew,
> > > >
> > > > As discussed I have incorporated the changes. Please review and let
> me
> > > > know if any changes required.
> > > >
> > > > Regards,
> > > > Saikat
> > > >
> > > > On Mon, Aug 27, 2018 at 1:45 AM, Saikat Maitra <
> > saikat.mai...@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> I have updated the PR with additional tests.
> > > >>
> > > >> Please review and share feedback.
> > > >>
> > > >> This PR is related to IgniteSink but allows to stream data from
> > Ignite.
> > > >>
> > > >> PR https://github.com/apache/ignite/pull/870/files
> > > >>
> > > >> Review https://reviews.ignite.apache.org/ignite/review/IGNT-CR-135
> > > >>
> > > >> Regards,
> > > >> Saikat
> > > >>
> > > >
> > > >
> > >
> >
>

[jira] [Created] (IGNITE-9465) Node.js client: improve complex object flags processing

[Alexey Kosenchuk (JIRA)]

Alexey Kosenchuk created IGNITE-9465:


 Summary: Node.js client: improve complex object flags processing
 Key: IGNITE-9465
 URL: https://issues.apache.org/jira/browse/IGNITE-9465
 Project: Ignite
  Issue Type: Bug
  Components: thin client
Reporter: Alexey Kosenchuk
Assignee: ekaterina.vergizova
 Fix For: 2.7


1) fix the issue in the full schema support

2) do not throw exception when object with HAS_RAW_DATA flag is received

3) support OFFSET_*_BYTE flags/optimizations when writing data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9464) MVCC TX: cache GET supports Mvcc tx mode.

[Andrew Mashenkov (JIRA)]

Andrew Mashenkov created IGNITE-9464:


 Summary: MVCC TX: cache GET supports Mvcc tx mode.
 Key: IGNITE-9464
 URL: https://issues.apache.org/jira/browse/IGNITE-9464
 Project: Ignite
  Issue Type: New Feature
  Components: cache, mvcc
Reporter: Andrew Mashenkov






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4608: IGNITE-9360 Remove SnapTreeMap and tests on it.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4608


---

[jira] [Created] (IGNITE-9463) [ML] Update ML tutorial with new model composition/update features

[Aleksey Zinoviev (JIRA)]

Aleksey Zinoviev created IGNITE-9463:


 Summary: [ML] Update ML tutorial with new model composition/update 
features
 Key: IGNITE-9463
 URL: https://issues.apache.org/jira/browse/IGNITE-9463
 Project: Ignite
  Issue Type: New Feature
  Components: ml
Affects Versions: 2.7
Reporter: Aleksey Zinoviev
Assignee: Aleksey Zinoviev
 Fix For: 2.7






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4682: IGNITE-6454 muted flaky tests

[dgarus]

GitHub user dgarus opened a pull request:

https://github.com/apache/ignite/pull/4682

IGNITE-6454 muted flaky tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dgarus/ignite IGNITE-6454

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4682.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4682


commit 75646c063f93cb32dd4e85a4441ef431f44cf3dd
Author: dgarus 
Date:   2018-09-04T15:32:12Z

IGNITE-6454 muted flaky tests




---

Re: Questions about getAllInternal(...)

[Alexey Goncharuk]

Hello Steve,

A cache entry becomes obsolete once the on-heap object is no longer locked
and is not used by any thread. Since we moved to off-heap-first model in
Ignite 2.0, we must clean on-heap entries as soon as possible to keep the
heap small. Thus both of the places you pointed out.

Hope this helps,
--AG

вт, 4 сент. 2018 г. в 12:04, steve.hostett...@gmail.com <
steve.hostett...@gmail.com>:

> Hello,
>
> in the case of local caches without eviction policy. I have the following
> questions:
>
> 1) I would to understand why, in the method, getAllInternal the method
> entryEx(cacheKey); uses the topology in the case of a local cache.
> Furthermore, it calls the method  map.putEntryIfObsoleteOrAbsent. But the
> entry cannot be obsolete or absent because it is a local cache.
>
>
> 2) Similarly, why is there a touch on the key : ctx.evicts().touch(entry,
> ctx.affinity().affinityTopologyVersion()); when the evicting policy is null
> (never evict)? This puts locks even when the context is lock=false.
>
>
> Thanks a lot for shedding some light on this.
>
> Steve
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

[GitHub] ignite pull request #4592: IGNITE-9054 Avoid using OptimizedMarshaller with ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4592


---

[GitHub] ignite pull request #4659: IGNITE-9387: update interface for trainers

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4659


---

Re: Compression prototype

[Ilya Kasnacheev]

Hello!

In case of Apache Ignite, most of savings is due to BinaryObject format,
which encodes types and fields with byte sequences. Any enum/string flags
will also get in dictionary. And then as it processes a record it fills up
its individual dictionary.

But, in one cache, most if not all entries have identical BinaryObject
layout so a tiny dictionary covers that case. Compression algorithms are
not very keen on large dictionaries, preferring to work with local
regularities in byte stream.

E.g. if we have large entries in cache with low BinaryObject overhead,
they're served just fine by "generic" compression.

All of the above is my speculations, actually. I just observe that on a
large data set, compression ratio is around 0.4 (2.5x) with a dictionary of
1024 bytes. The rest is black box.

Regards,
-- 
Ilya Kasnacheev


вт, 4 сент. 2018 г. в 17:16, Dmitriy Setrakyan :

> On Tue, Sep 4, 2018 at 2:55 AM, Ilya Kasnacheev  >
> wrote:
>
> > Hello!
> >
> > Each node has a local dictionary (per node currently, per cache planned).
> > Dictionary is never shared between nodes. As data patterns shift,
> > dictionary rotation is also planned.
> >
> > With Zstd, the best dictionary size seems to be 1024 bytes. I imagine It
> is
> > enough to store common BinaryObject boilerplate, and everything else is
> > compressed on the fly. The source sample is 16k records.
> >
> >
> Thanks, Ilya, understood. I think per-cache is a better idea. However, I
> have a question about dictionary size. Ignite stores TBs of data. How do
> you plan the dictionary to fit in 1K bytes?
>
> D.
>

[GitHub] akalash commented on a change in pull request #3: IGNITE-9333 Add statistics page

[GitBox]

akalash commented on a change in pull request #3: IGNITE-9333 Add statistics 
page
URL: https://github.com/apache/ignite-teamcity-bot/pull/3#discussion_r214728863
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/tcmodel/result/issues/IssueRef.java
 ##
 @@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.ci.tcmodel.result.issues;
+
+import javax.xml.bind.annotation.XmlAttribute;
+
+/**
+ * Issue short version from list of build's related issues.
+ *
+ * See example of XML, e.g. here
+ * https://ci.ignite.apache.org/app/rest/latest/builds/id:1694977/relatedIssues
+ */
+public class IssueRef {
 
 Review comment:
   We already have IssueRef class but it means other entity and it is very 
awkward. I think this is correct name(because TC have it), but perhaps old 
class should be renamed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] akalash commented on a change in pull request #3: IGNITE-9333 Add statistics page

[GitBox]

akalash commented on a change in pull request #3: IGNITE-9333 Add statistics 
page
URL: https://github.com/apache/ignite-teamcity-bot/pull/3#discussion_r214726814
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/tcmodel/result/RelatedIssuesRef.java
 ##
 @@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.ci.tcmodel.result;
+
+import javax.xml.bind.annotation.XmlAttribute;
+
+/**
+ * Related issues reference.
+ */
+public class RelatedIssuesRef {
 
 Review comment:
   extends AbstractRef like ChangesListRef


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] akalash commented on a change in pull request #3: IGNITE-9333 Add statistics page

[GitBox]

akalash commented on a change in pull request #3: IGNITE-9333 Add statistics 
page
URL: https://github.com/apache/ignite-teamcity-bot/pull/3#discussion_r214727827
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/rest/build/GetBuildTestFailures.java
 ##
 @@ -149,4 +155,69 @@ public TestFailuresSummary getBuildTestFails(
 
 return res;
 }
+
+@GET
+@Path("history")
+public BuildStatisticsSummary[] getBuildsHistory(
+@Nullable @QueryParam("server") String server,
+@Nullable @QueryParam("buildType") String buildType,
+@Nullable @QueryParam("branch") String branch,
+@Nullable @QueryParam("count") Integer count)
+throws ServiceUnauthorizedException {
+
+String srvId = isNullOrEmpty(server) ? "apache" : server;
+String buildTypeId = isNullOrEmpty(buildType) ? 
"IgniteTests24Java8_RunAll" : buildType;
+String branchName = isNullOrEmpty(branch) ? "refs/heads/master" : 
branch;
+int cnt = count == null ? 50 : count;
+
+final BackgroundUpdater updater = 
CtxListener.getBackgroundUpdater(context);
+
+final ITcHelper tcHelper = CtxListener.getTcHelper(context);
+
+final ICredentialsProv prov = ICredentialsProv.get(req);
+
+try (IAnalyticsEnabledTeamcity teamcity = tcHelper.server(srvId, 
prov)) {
+
+int[] finishedBuilds = 
teamcity.getBuildNumbersFromHistory(buildTypeId, branchName, cnt);
+
+BuildStatisticsSummary[] buildsStatistics = new 
BuildStatisticsSummary[finishedBuilds.length];
+
+for (int i = 0; i < finishedBuilds.length; i++) {
+int buildId = finishedBuilds[i];
+
+FullQueryParams param = new FullQueryParams();
+param.setBuildId(buildId);
+param.setBranch(branchName);
+param.setServerId(srvId);
+
+buildsStatistics[finishedBuilds.length - 1 - i] = updater.get(
+BUILDS_STATISTICS_SUMMARY_CACHE_NAME, prov, param,
+(k) -> getBuildStatisticsSummaryNoCache(srvId, buildId), 
false);
+}
+
+return buildsStatistics;
+}
+}
+
+
+@GET
+@Path("historyNoCache")
 
 Review comment:
   Looks like this endpoint is not required. It can be just simple method.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] akalash commented on a change in pull request #3: IGNITE-9333 Add statistics page

[GitBox]

akalash commented on a change in pull request #3: IGNITE-9333 Add statistics 
page
URL: https://github.com/apache/ignite-teamcity-bot/pull/3#discussion_r214714321
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
 ##
 @@ -240,14 +243,14 @@ private IgnitePersistentTeamcity(Ignite ignite, 
IgniteTeamcityHelper teamcity) {
 return loaded;
 }
 
-private  V timedLoadIfAbsentOrMerge(IgniteCache> 
cache, int seconds, K key,
+private  V timedLoadIfAbsentOrMerge(IgniteCache> 
cache, int seconds, long cnt, K key,
 BiFunction loadWithMerge) {
 @Nullable final Expirable persistedBuilds = cache.get(key);
 
 int fields = ObjectInterner.internFields(persistedBuilds);
 
 if (persistedBuilds != null) {
-if (persistedBuilds.isAgeLessThanSecs(seconds))
+if (persistedBuilds.isAgeLessThanSecs(seconds) && 
persistedBuilds.isLastCntGreaterThanCurrentCnt(cnt))
 
 Review comment:
   Method name "isLastCntGreaterThanCurrentCnt" is not clear for me, I can't 
catch which is last cnt and which is current cnt. Maybe better something like 
"hasCounterGreaterThan"?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] akalash commented on a change in pull request #3: IGNITE-9333 Add statistics page

[GitBox]

akalash commented on a change in pull request #3: IGNITE-9333 Add statistics 
page
URL: https://github.com/apache/ignite-teamcity-bot/pull/3#discussion_r214725893
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
 ##
 @@ -290,14 +298,16 @@ private IgnitePersistentTeamcity(Ignite ignite, 
IgniteTeamcityHelper teamcity) {
 
 return mergeByIdToHistoricalOrder(persistedValue, builds);
 });
+
+return buildRefs.stream().skip(cnt < buildRefs.size() ? 
buildRefs.size() - cnt : 0).collect(Collectors.toList());
 
 Review comment:
   In what case it can be happen(cnt < buildRefs.size())?
   
   And maybe better use some formatting for stream, because one line stream is 
too hard to read.
   ```
   buildRefs.stream()
   .skip(cnt < buildRefs.size() ? buildRefs.size() - cnt : 0)
   .collect(Collectors.toList());
   ```
   ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] akalash commented on a change in pull request #3: IGNITE-9333 Add statistics page

[GitBox]

akalash commented on a change in pull request #3: IGNITE-9333 Add statistics 
page
URL: https://github.com/apache/ignite-teamcity-bot/pull/3#discussion_r214729838
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/web/model/current/BuildStatisticsSummary.java
 ##
 @@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.ci.web.model.current;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import javax.annotation.Nonnull;
+import org.apache.ignite.ci.ITeamcity;
+import org.apache.ignite.ci.tcmodel.result.Build;
+import org.apache.ignite.ci.tcmodel.result.issues.IssueRef;
+import org.apache.ignite.ci.tcmodel.result.issues.IssueUsage;
+import org.apache.ignite.ci.tcmodel.result.problems.ProblemOccurrence;
+import org.apache.ignite.ci.util.TimeUtil;
+import org.apache.ignite.ci.web.IBackgroundUpdatable;
+
+/**
+ * Summary of build statistics.
+ */
+public class BuildStatisticsSummary extends UpdateInfo implements 
IBackgroundUpdatable {
+/** Build with test and problems references. */
+public Build build;
+
+/** List of problem occurrences. */
+private List problems;
+
+/** List of related issues. */
+public List relatedIssues;
+
+/** Duration printable. */
+public String durationPrintable;
+
+/** Build run result printable. */
+public String result;
+
+public BuildStatisticsSummary(Build build){
+this.build = build;
+}
+
+/** Initialize build statistics. */
+public void initialize(@Nonnull final ITeamcity teamcity) {
+
+problems = teamcity.getProblems(build).getProblemsNonNull();
+
+relatedIssues = 
teamcity.getIssuesUsagesList(build.relatedIssuesRef.href).getIssuesUsagesNonNull().stream()
+.map(IssueUsage::getIssue).collect(Collectors.toList());
+
+durationPrintable = TimeUtil
+.getDurationPrintable(build.getFinishDate().getTime() - 
build.getStartDate().getTime());
+
+result = getResult();
+}
+
+private long getExecutionTimeoutCount() {
+return 
getProblemsStream().filter(ProblemOccurrence::isExecutionTimeout).count();
+}
+
+private long getJvmCrashProblemCount() {
+return 
getProblemsStream().filter(ProblemOccurrence::isJvmCrash).count();
+}
+
+private long getExitCodeProblemsCount() {
+return 
getProblemsStream().filter(ProblemOccurrence::isExitCode).count();
+}
+
+private long getOomeProblemCount() {
+return getProblemsStream().filter(ProblemOccurrence::isOome).count();
+}
+
+private long getFailedTestsProblemCount() {
+return 
getProblemsStream().filter(ProblemOccurrence::isFailedTests).count();
+}
+
+private long getSnapshotDepProblemCount() {
+return 
getProblemsStream().filter(ProblemOccurrence::isSnapshotDepProblem).count();
+}
+
+private long getOtherProblemCount() {
+return getProblemsStream().filter(p ->
+!p.isFailedTests()
+&& !p.isSnapshotDepProblem()
+&& !p.isExecutionTimeout()
+&& !p.isJvmCrash()
+&& !p.isExitCode()
+&& !p.isOome()).count();
+}
+
+private Stream getProblemsStream() {
+if (problems == null)
+return Stream.empty();
+
+return problems.stream().filter(Objects::nonNull);
+}
+
+/**
+ * Build Run Result (filled if failed).
+ *
+ * @return printable result.
+ */
+private String getResult() {
+StringBuilder res = new StringBuilder();
+
+addKnownProblemCnt(res, "TIMEOUT", getExecutionTimeoutCount());
+addKnownProblemCnt(res, "JVM CRASH", getJvmCrashProblemCount());
+addKnownProblemCnt(res, "OOMe", getOomeProblemCount());
+addKnownProblemCnt(res, "EXIT CODE", getExitCodeProblemsCount());
+addKnownProblemCnt(res, "FAILED TESTS", getFailedTestsProblemCount());
+addKnownProblemCnt(res, "SNAPSHOT DEPENDENCY ERROR", 
getSnapshotDepProblemCount());
+addKnownProblemCnt(res, "OTHER", getOtherProblemCount());
 
 Review comment:
   Looks like 

[GitHub] akalash commented on a change in pull request #3: IGNITE-9333 Add statistics page

[GitBox]

akalash commented on a change in pull request #3: IGNITE-9333 Add statistics 
page
URL: https://github.com/apache/ignite-teamcity-bot/pull/3#discussion_r214726174
 
 

 ##
 File path: 
ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
 ##
 @@ -290,14 +298,16 @@ private IgnitePersistentTeamcity(Ignite ignite, 
IgniteTeamcityHelper teamcity) {
 
 return mergeByIdToHistoricalOrder(persistedValue, builds);
 });
+
+return buildRefs.stream().skip(cnt < buildRefs.size() ? 
buildRefs.size() - cnt : 0).collect(Collectors.toList());
 }
 
 @NotNull
 private List mergeByIdToHistoricalOrder(List 
persistedVal, List mostActualVal) {
 final SortedMap merge = new TreeMap<>();
 
 if (persistedVal != null)
-persistedVal.forEach(b -> merge.put(b.getId(), b));
+persistedVal.stream().forEach(b -> merge.put(b.getId(), b));
 
 Review comment:
   For what? It is same but longer, isn't it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Re: Possible problems with closing AsyncFileIO

[Nikolay Izhikov]

Hello, Sergey.

If some resources are not closed after usage it certainly a bug.
Can you create a ticket and attach a simplest reproducer for desribed issue?

В Вт, 04/09/2018 в 17:44 +0300, Sergey Antonov пишет:
> Hello, Igniters!
> 
>  I found bug in org.apache
> .ignite.internal.processors.cache.persistence.file.AsyncFileIO#close(): If
> one of  async futures (ChannelOpFuture) throw IgniteCheckedException
> AsynchronousFileChannel associated with AsyncFileIO instance will not be
> closed.
> It's correct behaviour or bug?
> 

signature.asc
Description: This is a digitally signed message part

Possible problems with closing AsyncFileIO

[Sergey Antonov]

Hello, Igniters!

 I found bug in org.apache
.ignite.internal.processors.cache.persistence.file.AsyncFileIO#close(): If
one of  async futures (ChannelOpFuture) throw IgniteCheckedException
AsynchronousFileChannel associated with AsyncFileIO instance will not be
closed.
It's correct behaviour or bug?

-- 
BR, Sergey Antonov

Re: Python thin client

[Dmitry Melnichuk]

Nikolay,

There is indeed a feature in `setuptools` I just learned about, which 
would help in this case (and I believe the case you demonstrated can be 
quite typical, thank you for pointing it out). It gives user a clever 
message in the end of a stack trace like


> UnsupportedPythonVersion: pyignite requires Python '>=3.4' but the
> running Python is 2.7.15

This feature is not 100% bullet-proof, and it will not help users who 
will try to install my client other way than with `pip` or `setup.py`.

It will also be less helpful with old versions of `pip` (before 9.0).
However, it should cover most situations.

On 9/4/18 7:15 PM, Nikolay Izhikov wrote:

Hello, Dmitry.

I understand that for experienced Python developer it obvious from
stack trace I send.

But can we check python version on startup? And print big fat error
message "You are using wrong python version".

>

From my experience, there are some tickets in Ignite that should be
implemented in various thin clients. It a very trivial changes, but
lack of testability makes this task harder then steel.

I think a .Net DEVNOTES are very good example.

Please, be gentle with your fellow contributors and make DEVNOTES as
clear as possible.

Re: Compression prototype

[Dmitriy Setrakyan]

On Tue, Sep 4, 2018 at 2:55 AM, Ilya Kasnacheev 
wrote:

> Hello!
>
> Each node has a local dictionary (per node currently, per cache planned).
> Dictionary is never shared between nodes. As data patterns shift,
> dictionary rotation is also planned.
>
> With Zstd, the best dictionary size seems to be 1024 bytes. I imagine It is
> enough to store common BinaryObject boilerplate, and everything else is
> compressed on the fly. The source sample is 16k records.
>
>
Thanks, Ilya, understood. I think per-cache is a better idea. However, I
have a question about dictionary size. Ignite stores TBs of data. How do
you plan the dictionary to fit in 1K bytes?

D.

[GitHub] ignite pull request #4592: IGNITE-9054 Avoid using OptimizedMarshaller with ...

[alamar]

GitHub user alamar reopened a pull request:

https://github.com/apache/ignite/pull/4592

IGNITE-9054 Avoid using OptimizedMarshaller with initial ScanQuery.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-gg-14092

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4592.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4592


commit d5ae876319086dacb1f7ff7fde8aed113892f744
Author: Ilya Kasnacheev 
Date:   2018-08-22T15:17:51Z

IGNITE-9054 Avoid using OptimizedMarshaller with initial ScanQuery.




---

[GitHub] ignite pull request #4592: IGNITE-9054 Avoid using OptimizedMarshaller with ...

[alamar]

Github user alamar closed the pull request at:

https://github.com/apache/ignite/pull/4592


---

Hi... trying to write a test case for a pacth ... need help....

[Paul Anderson]

... The code is done ready to go, but the test suite looks huge, I know the
two test classes I have to augment... but I am unsure of a couple of things.

1) how do I run a specific test (command line)
2) how do I start (make sure is started) an ignite instance

[GitHub] ignite pull request #4664: IGNITE-8987 Ignite hangs during getting of atomic...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4664


---

[GitHub] ignite pull request #4671: Ignite 9448

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4671


---

[GitHub] ignite pull request #4681: IGNITE-9422 Fixed NPE on clients when new binary ...

[ivandasch]

GitHub user ivandasch opened a pull request:

https://github.com/apache/ignite/pull/4681

IGNITE-9422 Fixed NPE on clients when new binary meta from joined nod…

…e arrived.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9422

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4681.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4681


commit 9bdfab6aebac141a2a81df49f51addf84e7dcd26
Author: Ivan Daschinskiy 
Date:   2018-09-04T12:40:41Z

IGNITE-9422 Fixed NPE on clients when new binary meta from joined node 
arrived.




---

[jira] [Created] (IGNITE-9462) ADD originator to EVT_NODE_LEFT

[Yuriy Sergeev (JIRA)]

Yuriy Sergeev created IGNITE-9462:
-

 Summary: ADD originator to EVT_NODE_LEFT
 Key: IGNITE-9462
 URL: https://issues.apache.org/jira/browse/IGNITE-9462
 Project: Ignite
  Issue Type: Task
Reporter: Yuriy Sergeev


Pls add originator to EVT_NODE_LEFT, Thx



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4666: Ignite 1.8.20

[dmekhanikov]

Github user dmekhanikov closed the pull request at:

https://github.com/apache/ignite/pull/4666


---

Re: ScanQuery fails with OutOfMemory when iterating

[Вячеслав Коптилин]

Hi Denis,

It looks like a known issue
https://issues.apache.org/jira/browse/IGNITE-8892
It is already fixed and will be available in Apache Ignite 2.7

Thanks,
S.

вс, 2 сент. 2018 г. в 17:40, Denis Magda :

> Igniters,
>
> A user reported the issue on SO:
>
> https://stackoverflow.com/questions/52117160/ignite-consumes-all-memory-and-fails-with-outofmemory-when-iterating-over-cache
>
> Does it sound familiar? Were we fixing something like that before?
>
> Please scan query and Ignite memory experts join the conversation there.
>
> --
> Denis
>

Re: PHP thin client

[Igor Sapego]

Great,

I'll take a look

Best Regards,
Igor


On Wed, Aug 29, 2018 at 10:37 PM Alexey Kosenchuk <
alexey.kosenc...@nobitlost.com> wrote:

> Hi folks,
>
> PHP thin client is ready for review.
>
> Jira with the scope of work - [1].
>
> Implementation, examples, tests:
>PR - [2],
>repository - [3].
>
> API spec - [4].
>
> Readme (how to for the client, instructions for the examples and tests,
> etc.) - [5].
>
> Regards,
> -Alexey
>
> [1] https://issues.apache.org/jira/browse/IGNITE-7783
> [2] https://github.com/apache/ignite/pull/4649
> [3]
> https://github.com/nobitlost/ignite/tree/ignite-7783/modules/platforms/php
> [4]
>
> https://rawgit.com/nobitlost/ignite/ignite-7783-docs/modules/platforms/php/api_docs/html/index.html
> [5]
>
> https://github.com/nobitlost/ignite/blob/ignite-7783-docs/modules/platforms/php/README.md
>

[GitHub] ignite pull request #4583: IGNITE-8829

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4583


---

[GitHub] ignite pull request #4680: IGNITE-9442

[xtern]

GitHub user xtern opened a pull request:

https://github.com/apache/ignite/pull/4680

IGNITE-9442



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xtern/ignite IGNITE-9442

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4680.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4680


commit fb7f21a41917b0ce1f58343376a5884aa3940bb2
Author: pereslegin-pa 
Date:   2018-09-04T12:06:06Z

IGNITE-9442 Block set on all nodes.




---

[GitHub] ignite pull request #4679: IGNITE-8927

[devozerov]

GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/4679

IGNITE-8927



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8927

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4679.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4679


commit 40fcd40cd713d012e1d9ce87b4612354f6e22bd3
Author: devozerov 
Date:   2018-09-04T09:48:54Z

WIP.

commit 7b6c902ee93fe8f98d000f86afa56571eac57c9a
Author: devozerov 
Date:   2018-09-04T10:23:59Z

WIP.

commit 60e6c3057dbb6fbf598d3a32ad84e7b2890462f0
Author: devozerov 
Date:   2018-09-04T10:51:19Z

WIP on test. Something is wrong with local query.




---

[jira] [Created] (IGNITE-9461) Implement random subspace method and provide an option to combine it with bagging

[Oleg Ignatenko (JIRA)]

Oleg Ignatenko created IGNITE-9461:
--

 Summary: Implement random subspace method and provide an option to 
combine it with bagging
 Key: IGNITE-9461
 URL: https://issues.apache.org/jira/browse/IGNITE-9461
 Project: Ignite
  Issue Type: Task
  Components: ml
Affects Versions: 2.6
Reporter: Oleg Ignatenko


Implement random subspace method (aka attribute bagging or feature bagging) to 
give ML API users more options to address overfitting. Also provide an option 
to combine this method with bagging.

References:

* [Wikipedia article|https://en.wikipedia.org/wiki/Random_subspace_method] 
{quote}Informally, this causes individual learners to not over-focus on 
features that appear highly predictive/descriptive in the training set, but 
fail to be as predictive for points outside that set. For this reason, random 
subspaces are an attractive choice for problems where the number of features is 
much larger than the number of training points, such as learning from fMRI data 
or gene expression data...{quote}
* [Combining Bagging and Random Subspaces to Create Better 
Ensembles|https://pdfs.semanticscholar.org/d38f/979ad85d59fc93058279010efc73a24a712c.pdf]
* [Bagging and the Random Subspace Method for Redundant Feature 
Spaces|https://link.springer.com/chapter/10.1007/3-540-48219-9_1]




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Workflow improvement

[Dmitriy Pavlov]

Hi Dmitrii,

Great news. I love the fact that you did these improvements.

I have limited access to the internet so for now I'm not able to review and
merge.

I can do it next week.

I appreciate your effort.

Sincerely,
Dmitriy Pavlov

пн, 3 сент. 2018 г., 18:20 Dmitrii Ryabov :

> Hi, Dmitriy,
>
> I created a table with possible blockers [1] for the page with the latest
> build analysis. Anton K. has reviewed it. Can you check and merge it?
>
> Also, I created a button to notify PR about analisis results [2]. I used
> GitHub statuses (example [3]), but to work it needs a token with push
> permission. As Apache PMC, can you acquire such token? I think statuses are
> much more notable than comments.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-9376
> [2] https://issues.apache.org/jira/browse/IGNITE-9377
> [3] https://github.com/SomeFire/ignite-teamcity-bot/pulls
>
> 2018-08-25 1:04 GMT+03:00 Dmitriy Pavlov :
>
> > Not sure I understood what bot statistic reduction means.
> >
> > As far as I understood, you also mean locating failure automatically with
> > re-runs on specific git revisions (hashes). I found it will be a good
> > contribution to TC Bot for both flaky and non-flaky failures detection.
> E.g
> > failure occurred after 10 commits merged to master. How to find out bad
> > commit? Automatic location (analog of bisecting, but using TC test) will
> be
> > really helpful. But it is not a simple contribution.
> >
> > пт, 24 авг. 2018 г. в 18:35, Dmitrii Ryabov :
> >
> > > Hi, Dmitriy,
> > >
> > > Good, I'll create tickets for this steps.
> > >
> > > Stable suites are a good idea, but also we need to do some actions
> when a
> > > flaky test will appear in stable suite of master branch run. To find
> out
> > a
> > > guilty commit, we should run previous commits on empty TeamCity's
> queue.
> > > This runs will reduce bot's statistics for the latest commits.
> > >
> > > 2018-08-24 16:32 GMT+03:00 Dmitriy Pavlov :
> > >
> > > > Hi Dmitriy,
> > > >
> > > > Sounds like a plan ;) I totally agree.
> > > > Just one proposal: I would like to avoid hiding any test failures. 2
> > > > separate tables named 'Possible Blockers' and 'Other failures' should
> > be
> > > > completely clear. Comment to PR can contain only first category.
> > > >
> > > > We had a private discussion with Anton K. and he proposed a very
> > > > interesting thing, I would like to bring it here. We can add
> > > configuration
> > > > into TC bot and mark some tests and some suites as 'Known Stable' It
> > > means
> > > > that suite or test failure should be considered as a blocker for
> merge
> > > > every time it fails, even if fail rate is non-zero. Very first list
> of
> > > such
> > > > suites are
> > > >  - Build Apache Ignite
> > > >  - License check
> > > > And tests:
> > > >  - .Net API Parity check
> > > > Please share your vision.
> > > >
> > > > Meanwhile, I've updated the TC bot.
> > > > - Underlying Apache Ignite DB was refactored to use lower partitions
> > > count,
> > > > so restart should be faster.
> > > > - Now 100 builds are saved as our failure rate statistics basis.
> > > Flakiness
> > > > border was not changed, so more test now will be considered as flaky.
> > > > - Now 4 builds required to be failed or timed out in a row before
> > > > notification is sent.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > чт, 23 авг. 2018 г. в 17:09, Dmitrii Ryabov :
> > > >
> > > > > Hi, Dmitriy,
> > > > >
> > > > > I propose the next steps:
> > > > >
> > > > > 1. Show current 0% tests in a separate table at the top of the
> > analysis
> > > > > results page. Thus, we'll see most suspicious (new or very flaky)
> > tests
> > > > > firstly. Or we can hide other tests under "More >>" button, like
> top
> > > long
> > > > > running tests.
> > > > > 2. Create a button by clicking on which the info about 0% tests
> will
> > be
> > > > > written in the PR.
> > > > > 3. Replace button by webhook for TeamCity (for Run All), which will
> > > start
> > > > > analysis on TCH and write results in the PR.
> > > > > 4. ...
> > > > >
> > > > > What do you think?
> > > > >
> > > > >
> > > > > 2018-08-22 8:55 GMT+03:00 Dmitrii Ryabov :
> > > > >
> > > > > > I think we should check not N last runs, but all runs in last N
> > days.
> > > > > >
> > > > > > A simple rule to detect flaky fails by hands - get test history
> > > ordered
> > > > > > by TEST_STATUS_DESC and check its date. As I see, we can get list
> > of
> > > > > > failures from TC. We don't need to check successfull runs,
> because
> > > they
> > > > > are
> > > > > > uninformative for our needs.
> > > > > >
> > > > > > 2018-08-21 20:24 GMT+03:00 Dmitriy Pavlov  >:
> > > > > >
> > > > > >> Hi Dmitriy,
> > > > > >>
> > > > > >> The Bot is able to detect a frequent change of test status, but
> > > > > currently
> > > > > >> only 50 last runs count. Same is true for the failure rate.
> > > > > >>
> > > > > >> This value can be easily changed to 70 or 100, 

Re: Unknown known issue on cache rebalancing delayed

[Anton Vinogradov]

Maxim,

Let's create a branch with 10 checks of Sync and 10 checks of Async.
Then, run it 20 times at TC.
This should be enough I think.

вт, 4 сент. 2018 г. в 13:09, Maxim Muzafarov :

> Anton,
>
> I agree with you 20 time is not enough. I've checked the single run of the
> test class - it consumes ~7min per each execution.
> CacheSuite8 total execution timeout - 210 min, so we can perform only 30
> class execution in this suite. Our strategy here is
> to `20 times within single` and put into the TC queue 50 runs. Total ~7000
> min or 5 days.
>
> Not sure that we should perform exactly 1000 executions, hopefully, we will
> stop adding to the queue new tasks at some point.
>
> On Tue, 4 Sep 2018 at 12:59 Anton Vinogradov  wrote:
>
> > Maxim,
> > 20 is not 1k :)
> > Also, you forgot to check GridCacheRebalancingAsyncSelfTest
> >
> > I'm not sure we should have exactly 1k runs, but 20 is definitely not
> > enough.
> >
> > Roman,
> > I propose to use IDEA "run until failure" feature and perform test
> locally
> > (at your PC) while you're not using PC.
> >
> > вт, 4 сент. 2018 г. в 12:51, Maxim Muzafarov :
> >
> > > Roman, Anton,
> > >
> > > I've already created additional PR [2] all and run it on TC [1].
> > > Please, follow up with the results.
> > >
> > > [1]
> > >
> > >
> >
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_Cache8=buildTypeStatusDiv_IgniteTests24Java8=pull%2F4676%2Fhead
> > > [2] https://github.com/apache/ignite/pull/4676/files
> > >
> > >
> > > On Tue, 4 Sep 2018 at 12:46 Roman Shtykh 
> > > wrote:
> > >
> > > > Anton,
> > > > Thank you. I would like to recheck it. How can this (1_000 runs) be
> > done
> > > > in TC?
> > > >
> > > >
> > > > On Tuesday, September 4, 2018, 5:42:01 p.m. GMT+9, Anton
> > Vinogradov <
> > > > a...@apache.org> wrote:
> > > >
> > > >  Roman,
> > > >
> > > > I see you uncommented this line.
> > > > I do not remember deadlock detail, but I remember it was the
> extremely
> > > rare
> > > > case.
> > > > I found and "fixed" it some days before merge when I had 24x7 sanity
> > > check
> > > > week :)
> > > >
> > > > So, I propose to have at least 1_000 runs of this tests before
> keeping
> > > this
> > > > uncommented.
> > > >
> > > >
> > > >
> > > > вт, 21 авг. 2018 г. в 11:08, Maxim Muzafarov :
> > > >
> > > > > Roman,
> > > > >
> > > > > I worked recently on rebalance improvements and haven't found any
> > > > problems
> > > > > with delayed cache rebalacne.
> > > > > Agree with you - let's uncomment this and remove scary comment.
> Will
> > > you
> > > > > create a ticket for it?
> > > > >
> > > > > In case of any problems we can easily detec deadlock with newly
> > > > configured
> > > > > `FailureHandler`.
> > > > >
> > > > > On Tue, 21 Aug 2018 at 03:49 Roman Shtykh 
> wrote:
> > > > >
> > > > > > Hi Maxim,
> > > > > >
> > > > > > I have some issues with a cluster with rebalance delay enabled,
> but
> > > > need
> > > > > > to check more -- if I find it's related I'll share.
> > > > > > Just wanted to make sure it's not an issue anymore from someone
> > > working
> > > > > on
> > > > > > rebalancing. We should remove that comment then, it looks scary
> :)
> > > > > >
> > > > > > --
> > > > > > Roman Shtykh
> > > > > >
> > > > > >
> > > > > > On Tuesday, August 21, 2018, 12:49:00 a.m. GMT+9, Maxim
> Muzafarov <
> > > > > > maxmu...@gmail.com> wrote:
> > > > > >
> > > > > >
> > > > > > Hello Roman,
> > > > > >
> > > > > > Did you faced with real issue of delayed rebalance or it's just
> > only
> > > > for
> > > > > > your personal interest?
> > > > > > If yes, please, share details and we will try to help you.
> > > > > >
> > > > > > As for this comment I don't think he is actual. That change was
> in
> > > > 2015.
> > > > > > Much has changed
> > > > > > within rebalance process since that time. I've uncommented it and
> > > > > > rechecked with that
> > > > > > cache configuration and haven't seen any failed tests or issues.
> > > > > >
> > > > > > Probably, that problem was about cache in SYNC mode does not
> start
> > > util
> > > > > it
> > > > > > loads all data
> > > > > > from other nodes. But currently delayed rebalance works the same
> > way
> > > as
> > > > > > IgniteCache#rebalance(),
> > > > > > so you can `setRebalanceDelay` to `-1` and call it manually to
> > check.
> > > > > >
> > > > > > On Mon, 20 Aug 2018 at 11:19 Roman Shtykh
> >  > > >
> > > > > > wrote:
> > > > > >
> > > > > > Igniters,
> > > > > > I have found "Known issue, possible deadlock in case of low
> > priority
> > > > > cache
> > > > > > rebalancing delayed" comment in
> > > > > > GridCacheRebalancingSyncSelfTest#getConfiguration.Can you please
> > > > explain
> > > > > > when using rebalance delay can be an issue and why?
> > > > > >
> > > > > > -- Roman
> > > > > >
> > > > > > --
> > > > > > --
> > > > > > Maxim Muzafarov
> > > > > >
> > > > > --
> > > > > --
> > > > > Maxim Muzafarov
> > > > >
> > >
> > > --
> > > --
> > > Maxim Muzafarov
> > >
> 

Re: Apache Ignite 2.7 release

[Roman Shtykh]

Thanks, Ilya!I will check your comments, and discuss it at JIRA.

--
Roman Shtykh 

On Tuesday, September 4, 2018, 7:17:53 p.m. GMT+9, Ilya Kasnacheev 
 wrote:  
 
 Hello!
IGNITE-9408 looks good to me and may be merged right away.
IGNITE-9388 needs more work in my opinion, I have commented the PR. I also 
advice having test for this functionality.

Regards,
-- 
Ilya Kasnacheev


вт, 4 сент. 2018 г. в 6:52, Roman Shtykh :

Igniters,
I would like Mesos integration update be included in the upcoming release.Can 
anyone review prs for the following issues?
IGNITE-9388: mesos IgniteProvider tries to access obsolete ignite.run or 
download from slow archiveIGNITE-9408: Update mesos version

Roman Shtykh 

    On Thursday, August 30, 2018, 9:25:43 p.m. GMT+9, Vyacheslav Daradur 
 wrote:  

 Hi Igniters!

I'm working on the following Service Grid tasks:
- IGNITE-8361 Use discovery messages for service deployment
- IGNITE-8362 Collect service deployment results asynchronously on coordinator
- IGNITE-8363 Handle topology changes during service deployment
- IGNITE-8364 Propagate deployed services to joining nodes
- IGNITE-8365 Introduce service failure events
- IGNITE-3392 Propagate service deployment results from assigned nodes
to initiator

Let's call them *phase 1* because the should be implemented together
(atomically).

I do my best to finish phase 1 for including to 2.7 release.

But I'm not sure that the solution will be fully completed till the
beginning of October.

On Wed, Aug 29, 2018 at 7:18 PM Nikolay Izhikov  wrote:
>
> Hell, Yakov
>
> I'm ok with your proposal.
>
>        * Scope freeze - September 17 - We should have a full list of tickets 
>for 2.7 here.
>        * Code freeze - October 01 - We should merge all 2.7 tickets to master 
>here.
>        * Vote on RC1 - October 11.
>        * Vote on release - October 15.
>
> В Ср, 29/08/2018 в 12:39 +0300, Yakov Zhdanov пишет:
> > Nikolay,
> >
> > I think we should have 2 weeks after code freeze which by the way may
> > include RC1 voting stage. This way I would like us to agree that release
> > candidate should be sent to vote on Oct, 11th and we can release on Oct,
> > 15th.
> >
> > What do you think?
> >
> > --Yakov



-- 
Best Regards, Vyacheslav D.  
  

[jira] [Created] (IGNITE-9460) Update styles on WC top menu

[Alexander Kalinin (JIRA)]

Alexander Kalinin created IGNITE-9460:
-

 Summary: Update styles on WC top menu
 Key: IGNITE-9460
 URL: https://issues.apache.org/jira/browse/IGNITE-9460
 Project: Ignite
  Issue Type: Bug
  Components: UI
Reporter: Alexander Kalinin
Assignee: Alexander Kalinin


1) Remove underline on focus

!https://snag.gy/RBMZyn.jpg!

2) Dropdown menus should look like generic dropdown

!https://snag.gy/peXHDl.jpg!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4678: IGNITE-9361 Remove IgniteInternalCache.isMongo*Ca...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4678


---

[GitHub] ignite pull request #4678: IGNITE-9361 Remove IgniteInternalCache.isMongo*Ca...

[alamar]

GitHub user alamar opened a pull request:

https://github.com/apache/ignite/pull/4678

IGNITE-9361 Remove IgniteInternalCache.isMongo*Cache() and other remn…

…ants.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9361

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4678.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4678


commit 499d300ff74e31f32b5eafcaed06b1fff6c418d3
Author: Ilya Kasnacheev 
Date:   2018-08-23T16:08:48Z

IGNITE-9361 Remove IgniteInternalCache.isMongo*Cache() and other remnants.




---

Re: Apache Ignite 2.7 release

[Ilya Kasnacheev]

Hello!

IGNITE-9408  looks good
to me and may be merged right away.

IGNITE-9388  needs more
work in my opinion, I have commented the PR. I also advice having test for
this functionality.

Regards,
-- 
Ilya Kasnacheev


вт, 4 сент. 2018 г. в 6:52, Roman Shtykh :

> Igniters,
> I would like Mesos integration update be included in the upcoming
> release.Can anyone review prs for the following issues?
> IGNITE-9388: mesos IgniteProvider tries to access obsolete ignite.run or
> download from slow archiveIGNITE-9408: Update mesos version
>
> Roman Shtykh
>
> On Thursday, August 30, 2018, 9:25:43 p.m. GMT+9, Vyacheslav Daradur <
> daradu...@gmail.com> wrote:
>
>  Hi Igniters!
>
> I'm working on the following Service Grid tasks:
> - IGNITE-8361 Use discovery messages for service deployment
> - IGNITE-8362 Collect service deployment results asynchronously on
> coordinator
> - IGNITE-8363 Handle topology changes during service deployment
> - IGNITE-8364 Propagate deployed services to joining nodes
> - IGNITE-8365 Introduce service failure events
> - IGNITE-3392 Propagate service deployment results from assigned nodes
> to initiator
>
> Let's call them *phase 1* because the should be implemented together
> (atomically).
>
> I do my best to finish phase 1 for including to 2.7 release.
>
> But I'm not sure that the solution will be fully completed till the
> beginning of October.
>
> On Wed, Aug 29, 2018 at 7:18 PM Nikolay Izhikov 
> wrote:
> >
> > Hell, Yakov
> >
> > I'm ok with your proposal.
> >
> >* Scope freeze - September 17 - We should have a full list of
> tickets for 2.7 here.
> >* Code freeze - October 01 - We should merge all 2.7 tickets to
> master here.
> >* Vote on RC1 - October 11.
> >* Vote on release - October 15.
> >
> > В Ср, 29/08/2018 в 12:39 +0300, Yakov Zhdanov пишет:
> > > Nikolay,
> > >
> > > I think we should have 2 weeks after code freeze which by the way may
> > > include RC1 voting stage. This way I would like us to agree that
> release
> > > candidate should be sent to vote on Oct, 11th and we can release on
> Oct,
> > > 15th.
> > >
> > > What do you think?
> > >
> > > --Yakov
>
>
>
> --
> Best Regards, Vyacheslav D.

Re: Unknown known issue on cache rebalancing delayed

[Maxim Muzafarov]

Anton,

I agree with you 20 time is not enough. I've checked the single run of the
test class - it consumes ~7min per each execution.
CacheSuite8 total execution timeout - 210 min, so we can perform only 30
class execution in this suite. Our strategy here is
to `20 times within single` and put into the TC queue 50 runs. Total ~7000
min or 5 days.

Not sure that we should perform exactly 1000 executions, hopefully, we will
stop adding to the queue new tasks at some point.

On Tue, 4 Sep 2018 at 12:59 Anton Vinogradov  wrote:

> Maxim,
> 20 is not 1k :)
> Also, you forgot to check GridCacheRebalancingAsyncSelfTest
>
> I'm not sure we should have exactly 1k runs, but 20 is definitely not
> enough.
>
> Roman,
> I propose to use IDEA "run until failure" feature and perform test locally
> (at your PC) while you're not using PC.
>
> вт, 4 сент. 2018 г. в 12:51, Maxim Muzafarov :
>
> > Roman, Anton,
> >
> > I've already created additional PR [2] all and run it on TC [1].
> > Please, follow up with the results.
> >
> > [1]
> >
> >
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_Cache8=buildTypeStatusDiv_IgniteTests24Java8=pull%2F4676%2Fhead
> > [2] https://github.com/apache/ignite/pull/4676/files
> >
> >
> > On Tue, 4 Sep 2018 at 12:46 Roman Shtykh 
> > wrote:
> >
> > > Anton,
> > > Thank you. I would like to recheck it. How can this (1_000 runs) be
> done
> > > in TC?
> > >
> > >
> > > On Tuesday, September 4, 2018, 5:42:01 p.m. GMT+9, Anton
> Vinogradov <
> > > a...@apache.org> wrote:
> > >
> > >  Roman,
> > >
> > > I see you uncommented this line.
> > > I do not remember deadlock detail, but I remember it was the extremely
> > rare
> > > case.
> > > I found and "fixed" it some days before merge when I had 24x7 sanity
> > check
> > > week :)
> > >
> > > So, I propose to have at least 1_000 runs of this tests before keeping
> > this
> > > uncommented.
> > >
> > >
> > >
> > > вт, 21 авг. 2018 г. в 11:08, Maxim Muzafarov :
> > >
> > > > Roman,
> > > >
> > > > I worked recently on rebalance improvements and haven't found any
> > > problems
> > > > with delayed cache rebalacne.
> > > > Agree with you - let's uncomment this and remove scary comment. Will
> > you
> > > > create a ticket for it?
> > > >
> > > > In case of any problems we can easily detec deadlock with newly
> > > configured
> > > > `FailureHandler`.
> > > >
> > > > On Tue, 21 Aug 2018 at 03:49 Roman Shtykh  wrote:
> > > >
> > > > > Hi Maxim,
> > > > >
> > > > > I have some issues with a cluster with rebalance delay enabled, but
> > > need
> > > > > to check more -- if I find it's related I'll share.
> > > > > Just wanted to make sure it's not an issue anymore from someone
> > working
> > > > on
> > > > > rebalancing. We should remove that comment then, it looks scary :)
> > > > >
> > > > > --
> > > > > Roman Shtykh
> > > > >
> > > > >
> > > > > On Tuesday, August 21, 2018, 12:49:00 a.m. GMT+9, Maxim Muzafarov <
> > > > > maxmu...@gmail.com> wrote:
> > > > >
> > > > >
> > > > > Hello Roman,
> > > > >
> > > > > Did you faced with real issue of delayed rebalance or it's just
> only
> > > for
> > > > > your personal interest?
> > > > > If yes, please, share details and we will try to help you.
> > > > >
> > > > > As for this comment I don't think he is actual. That change was in
> > > 2015.
> > > > > Much has changed
> > > > > within rebalance process since that time. I've uncommented it and
> > > > > rechecked with that
> > > > > cache configuration and haven't seen any failed tests or issues.
> > > > >
> > > > > Probably, that problem was about cache in SYNC mode does not start
> > util
> > > > it
> > > > > loads all data
> > > > > from other nodes. But currently delayed rebalance works the same
> way
> > as
> > > > > IgniteCache#rebalance(),
> > > > > so you can `setRebalanceDelay` to `-1` and call it manually to
> check.
> > > > >
> > > > > On Mon, 20 Aug 2018 at 11:19 Roman Shtykh
>  > >
> > > > > wrote:
> > > > >
> > > > > Igniters,
> > > > > I have found "Known issue, possible deadlock in case of low
> priority
> > > > cache
> > > > > rebalancing delayed" comment in
> > > > > GridCacheRebalancingSyncSelfTest#getConfiguration.Can you please
> > > explain
> > > > > when using rebalance delay can be an issue and why?
> > > > >
> > > > > -- Roman
> > > > >
> > > > > --
> > > > > --
> > > > > Maxim Muzafarov
> > > > >
> > > > --
> > > > --
> > > > Maxim Muzafarov
> > > >
> >
> > --
> > --
> > Maxim Muzafarov
> >
>
-- 
--
Maxim Muzafarov

Re: Unknown known issue on cache rebalancing delayed

[Anton Vinogradov]

Maxim,
20 is not 1k :)
Also, you forgot to check GridCacheRebalancingAsyncSelfTest

I'm not sure we should have exactly 1k runs, but 20 is definitely not
enough.

Roman,
I propose to use IDEA "run until failure" feature and perform test locally
(at your PC) while you're not using PC.

вт, 4 сент. 2018 г. в 12:51, Maxim Muzafarov :

> Roman, Anton,
>
> I've already created additional PR [2] all and run it on TC [1].
> Please, follow up with the results.
>
> [1]
>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_Cache8=buildTypeStatusDiv_IgniteTests24Java8=pull%2F4676%2Fhead
> [2] https://github.com/apache/ignite/pull/4676/files
>
>
> On Tue, 4 Sep 2018 at 12:46 Roman Shtykh 
> wrote:
>
> > Anton,
> > Thank you. I would like to recheck it. How can this (1_000 runs) be done
> > in TC?
> >
> >
> > On Tuesday, September 4, 2018, 5:42:01 p.m. GMT+9, Anton Vinogradov <
> > a...@apache.org> wrote:
> >
> >  Roman,
> >
> > I see you uncommented this line.
> > I do not remember deadlock detail, but I remember it was the extremely
> rare
> > case.
> > I found and "fixed" it some days before merge when I had 24x7 sanity
> check
> > week :)
> >
> > So, I propose to have at least 1_000 runs of this tests before keeping
> this
> > uncommented.
> >
> >
> >
> > вт, 21 авг. 2018 г. в 11:08, Maxim Muzafarov :
> >
> > > Roman,
> > >
> > > I worked recently on rebalance improvements and haven't found any
> > problems
> > > with delayed cache rebalacne.
> > > Agree with you - let's uncomment this and remove scary comment. Will
> you
> > > create a ticket for it?
> > >
> > > In case of any problems we can easily detec deadlock with newly
> > configured
> > > `FailureHandler`.
> > >
> > > On Tue, 21 Aug 2018 at 03:49 Roman Shtykh  wrote:
> > >
> > > > Hi Maxim,
> > > >
> > > > I have some issues with a cluster with rebalance delay enabled, but
> > need
> > > > to check more -- if I find it's related I'll share.
> > > > Just wanted to make sure it's not an issue anymore from someone
> working
> > > on
> > > > rebalancing. We should remove that comment then, it looks scary :)
> > > >
> > > > --
> > > > Roman Shtykh
> > > >
> > > >
> > > > On Tuesday, August 21, 2018, 12:49:00 a.m. GMT+9, Maxim Muzafarov <
> > > > maxmu...@gmail.com> wrote:
> > > >
> > > >
> > > > Hello Roman,
> > > >
> > > > Did you faced with real issue of delayed rebalance or it's just only
> > for
> > > > your personal interest?
> > > > If yes, please, share details and we will try to help you.
> > > >
> > > > As for this comment I don't think he is actual. That change was in
> > 2015.
> > > > Much has changed
> > > > within rebalance process since that time. I've uncommented it and
> > > > rechecked with that
> > > > cache configuration and haven't seen any failed tests or issues.
> > > >
> > > > Probably, that problem was about cache in SYNC mode does not start
> util
> > > it
> > > > loads all data
> > > > from other nodes. But currently delayed rebalance works the same way
> as
> > > > IgniteCache#rebalance(),
> > > > so you can `setRebalanceDelay` to `-1` and call it manually to check.
> > > >
> > > > On Mon, 20 Aug 2018 at 11:19 Roman Shtykh  >
> > > > wrote:
> > > >
> > > > Igniters,
> > > > I have found "Known issue, possible deadlock in case of low priority
> > > cache
> > > > rebalancing delayed" comment in
> > > > GridCacheRebalancingSyncSelfTest#getConfiguration.Can you please
> > explain
> > > > when using rebalance delay can be an issue and why?
> > > >
> > > > -- Roman
> > > >
> > > > --
> > > > --
> > > > Maxim Muzafarov
> > > >
> > > --
> > > --
> > > Maxim Muzafarov
> > >
>
> --
> --
> Maxim Muzafarov
>

Re: Compression prototype

[Ilya Kasnacheev]

Hello!

Each node has a local dictionary (per node currently, per cache planned).
Dictionary is never shared between nodes. As data patterns shift,
dictionary rotation is also planned.

With Zstd, the best dictionary size seems to be 1024 bytes. I imagine It is
enough to store common BinaryObject boilerplate, and everything else is
compressed on the fly. The source sample is 16k records.

Regards,
-- 
Ilya Kasnacheev


вт, 4 сент. 2018 г. в 11:49, Dmitriy Setrakyan :

> On Tue, Sep 4, 2018 at 1:16 AM, Ilya Kasnacheev  >
> wrote:
>
> > Hello!
> >
> > The compression is per-binary-object, but dictionary is external, shared
> > between multiple (millions of) entries and stored alongside compressed
> > data.
> >
>
> I was under a different impression. If the dictionary is for the whole data
> set, then it will occupy megabytes (if not gigabytes) of data. What happens
> when a new node joins and has no idea about the dictionary? What happens
> when dictionary between nodes get out-of-sync?
>
> D.
>

Re: Unknown known issue on cache rebalancing delayed

[Maxim Muzafarov]

Roman, Anton,

I've already created additional PR [2] all and run it on TC [1].
Please, follow up with the results.

[1]
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_Cache8=buildTypeStatusDiv_IgniteTests24Java8=pull%2F4676%2Fhead
[2] https://github.com/apache/ignite/pull/4676/files


On Tue, 4 Sep 2018 at 12:46 Roman Shtykh  wrote:

> Anton,
> Thank you. I would like to recheck it. How can this (1_000 runs) be done
> in TC?
>
>
> On Tuesday, September 4, 2018, 5:42:01 p.m. GMT+9, Anton Vinogradov <
> a...@apache.org> wrote:
>
>  Roman,
>
> I see you uncommented this line.
> I do not remember deadlock detail, but I remember it was the extremely rare
> case.
> I found and "fixed" it some days before merge when I had 24x7 sanity check
> week :)
>
> So, I propose to have at least 1_000 runs of this tests before keeping this
> uncommented.
>
>
>
> вт, 21 авг. 2018 г. в 11:08, Maxim Muzafarov :
>
> > Roman,
> >
> > I worked recently on rebalance improvements and haven't found any
> problems
> > with delayed cache rebalacne.
> > Agree with you - let's uncomment this and remove scary comment. Will you
> > create a ticket for it?
> >
> > In case of any problems we can easily detec deadlock with newly
> configured
> > `FailureHandler`.
> >
> > On Tue, 21 Aug 2018 at 03:49 Roman Shtykh  wrote:
> >
> > > Hi Maxim,
> > >
> > > I have some issues with a cluster with rebalance delay enabled, but
> need
> > > to check more -- if I find it's related I'll share.
> > > Just wanted to make sure it's not an issue anymore from someone working
> > on
> > > rebalancing. We should remove that comment then, it looks scary :)
> > >
> > > --
> > > Roman Shtykh
> > >
> > >
> > > On Tuesday, August 21, 2018, 12:49:00 a.m. GMT+9, Maxim Muzafarov <
> > > maxmu...@gmail.com> wrote:
> > >
> > >
> > > Hello Roman,
> > >
> > > Did you faced with real issue of delayed rebalance or it's just only
> for
> > > your personal interest?
> > > If yes, please, share details and we will try to help you.
> > >
> > > As for this comment I don't think he is actual. That change was in
> 2015.
> > > Much has changed
> > > within rebalance process since that time. I've uncommented it and
> > > rechecked with that
> > > cache configuration and haven't seen any failed tests or issues.
> > >
> > > Probably, that problem was about cache in SYNC mode does not start util
> > it
> > > loads all data
> > > from other nodes. But currently delayed rebalance works the same way as
> > > IgniteCache#rebalance(),
> > > so you can `setRebalanceDelay` to `-1` and call it manually to check.
> > >
> > > On Mon, 20 Aug 2018 at 11:19 Roman Shtykh 
> > > wrote:
> > >
> > > Igniters,
> > > I have found "Known issue, possible deadlock in case of low priority
> > cache
> > > rebalancing delayed" comment in
> > > GridCacheRebalancingSyncSelfTest#getConfiguration.Can you please
> explain
> > > when using rebalance delay can be an issue and why?
> > >
> > > -- Roman
> > >
> > > --
> > > --
> > > Maxim Muzafarov
> > >
> > --
> > --
> > Maxim Muzafarov
> >

-- 
--
Maxim Muzafarov

Re: Unknown known issue on cache rebalancing delayed

[Roman Shtykh]

Anton,
Thank you. I would like to recheck it. How can this (1_000 runs) be done in TC?


On Tuesday, September 4, 2018, 5:42:01 p.m. GMT+9, Anton Vinogradov 
 wrote:  
 
 Roman,

I see you uncommented this line.
I do not remember deadlock detail, but I remember it was the extremely rare
case.
I found and "fixed" it some days before merge when I had 24x7 sanity check
week :)

So, I propose to have at least 1_000 runs of this tests before keeping this
uncommented.



вт, 21 авг. 2018 г. в 11:08, Maxim Muzafarov :

> Roman,
>
> I worked recently on rebalance improvements and haven't found any problems
> with delayed cache rebalacne.
> Agree with you - let's uncomment this and remove scary comment. Will you
> create a ticket for it?
>
> In case of any problems we can easily detec deadlock with newly configured
> `FailureHandler`.
>
> On Tue, 21 Aug 2018 at 03:49 Roman Shtykh  wrote:
>
> > Hi Maxim,
> >
> > I have some issues with a cluster with rebalance delay enabled, but need
> > to check more -- if I find it's related I'll share.
> > Just wanted to make sure it's not an issue anymore from someone working
> on
> > rebalancing. We should remove that comment then, it looks scary :)
> >
> > --
> > Roman Shtykh
> >
> >
> > On Tuesday, August 21, 2018, 12:49:00 a.m. GMT+9, Maxim Muzafarov <
> > maxmu...@gmail.com> wrote:
> >
> >
> > Hello Roman,
> >
> > Did you faced with real issue of delayed rebalance or it's just only for
> > your personal interest?
> > If yes, please, share details and we will try to help you.
> >
> > As for this comment I don't think he is actual. That change was in 2015.
> > Much has changed
> > within rebalance process since that time. I've uncommented it and
> > rechecked with that
> > cache configuration and haven't seen any failed tests or issues.
> >
> > Probably, that problem was about cache in SYNC mode does not start util
> it
> > loads all data
> > from other nodes. But currently delayed rebalance works the same way as
> > IgniteCache#rebalance(),
> > so you can `setRebalanceDelay` to `-1` and call it manually to check.
> >
> > On Mon, 20 Aug 2018 at 11:19 Roman Shtykh 
> > wrote:
> >
> > Igniters,
> > I have found "Known issue, possible deadlock in case of low priority
> cache
> > rebalancing delayed" comment in
> > GridCacheRebalancingSyncSelfTest#getConfiguration.Can you please explain
> > when using rebalance delay can be an issue and why?
> >
> > -- Roman
> >
> > --
> > --
> > Maxim Muzafarov
> >
> --
> --
> Maxim Muzafarov
>  

[jira] [Created] (IGNITE-9459) MVCC Cache.size corner cases

[Ivan Pavlukhin (JIRA)]

Ivan Pavlukhin created IGNITE-9459:
--

 Summary: MVCC Cache.size corner cases
 Key: IGNITE-9459
 URL: https://issues.apache.org/jira/browse/IGNITE-9459
 Project: Ignite
  Issue Type: Bug
  Components: mvcc
Reporter: Ivan Pavlukhin


During implementation of Cache.size for MVCC some corner cases were found. 
Attention must be put on:

{{org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManager#mvccRemoveAll}}

{{org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManager#mvccInitialValue}}

{{mvccRemoveAll}} could erroneously decrement size when deleting version which 
was deleted (having {{newVersion}} committed).

{{mvccInitialValue}} could increment value over already existing value from 
history during rebalance.

It worth carefully reproduce both cases before making any fixes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4652: IGNITE-9433 Extracted zip suffix constant

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4652


---

[GitHub] ignite pull request #4677: Ignite 6195

[SGrimstad]

GitHub user SGrimstad opened a pull request:

https://github.com/apache/ignite/pull/4677

Ignite 6195



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite IGNITE-6195

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4677.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4677


commit 26543e10f8143dbc2d313b870081d633baf4cd05
Author: SGrimstad 
Date:   2018-08-14T11:21:13Z

IGNITE-9141  Implemented

commit ee631080782fe17a007e10117a96fc1a72990854
Author: devozerov 
Date:   2018-08-14T12:54:32Z

Merge branch 'master' into ignite-9141

commit 4587bfc24f25045fd7fc2197076797cc6ca54e32
Author: devozerov 
Date:   2018-08-14T13:23:19Z

Review.

commit 1511eb31b644113091be156d872f5adb2daecb84
Author: SGrimstad 
Date:   2018-08-14T11:21:13Z

IGNITE-9141  Implemented

commit 0425a32a3d490165b29bcf9236a49a48f7761666
Author: devozerov 
Date:   2018-08-14T13:23:19Z

Review.

commit a7209f870148f984cdbf3e6437337c7b665defb5
Author: SGrimstad 
Date:   2018-08-20T11:56:11Z

IGNITE-9141 Modified according to review comments. Integration tests added

commit 08e196ad4503d4487fda0da2b61406ef3bd84cbf
Author: SGrimstad 
Date:   2018-08-20T12:28:46Z

IGNITE-9141 javadoc added

commit b006a8dba4ae4319aa78a0d3c6fd6eef47bb1da8
Author: SGrimstad 
Date:   2018-08-20T12:29:46Z

IGNITE-9141 javadoc added

commit 0ed04a7ca655866685841b2ec9c2ed64797bf233
Author: devozerov 
Date:   2018-08-28T08:05:04Z

Merge branch 'master' into ignite-9141

commit 02db267355a8405c64c45d51c81e342df664d612
Author: devozerov 
Date:   2018-08-28T08:45:55Z

Review comments.

commit 0c4301cdc0d6108ed5b51173144e19d3ad450e63
Author: SGrimstad 
Date:   2018-08-28T11:08:47Z

IGNITE-9141 Fixes according to review

commit cef6e572e87503442e5b59fa147669b1f451fa13
Author: SGrimstad 
Date:   2018-08-30T11:28:20Z

ШПТШЕУ-9141 removed fair testing, will be enabled back in IGNITE-8913

commit abac3c53030433a4d47d0ea74f55d46394eda4eb
Author: devozerov 
Date:   2018-08-31T14:32:54Z

Merge branch 'master' into ignite-9141

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/query/h2/twostep/messages/GridQueryNextPageResponse.java
#   
modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/twostep/GridReduceQueryExecutor.java
#   
modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/twostep/ReduceQueryRun.java

commit 038a007c793d8f077509ce33f30d71885ac78672
Author: devozerov 
Date:   2018-08-31T14:47:44Z

Minors.

commit c39bf96daa4df36dfe9599f3a1c9f40320be7921
Author: devozerov 
Date:   2018-08-31T15:11:53Z

Cleanup.

commit b2c8702a2cfa44031eacee7b2f315cd757e5d3e4
Author: devozerov 
Date:   2018-08-31T15:50:10Z

WIP.

commit b12b7317f3e124968070f3fb69926e380d3c4883
Author: devozerov 
Date:   2018-08-31T15:52:30Z

Review.

commit 4cfce62b8f284b78356793d989ea9573d66e5faf
Author: SGrimstad 
Date:   2018-09-03T10:18:14Z

IGNITE-9141 review fix

commit 4076ddf41c0061985e2a8ea38457be0db9d7fe8e
Author: SGrimstad 
Date:   2018-09-04T09:40:38Z

IGNITE-6195 partition proper co-location check




---

Relevance of savepoints support inside of Ignite Transactions

[Dmitrii Ryabov]

Hi, Igniters!

I have an implementation for savepoints support inside of transactions,
which was reviewed by Boikov a year ago.

It allows us to rollback piece of the transaction, not whole tx.

Current restrictions:

* Doesn't work with near caches.

* MVCC should be disabled.



MVCC was merged recently.

So, I want to know – is ticket actual now?



JIRA - https://issues.apache.org/jira/browse/IGNITE-4188

Design -
https://docs.google.com/document/d/1IxTEVrjY44nsyZUUW1f0sGwuzahiPhy5nV-vFS20FZE/edit?usp=sharing


PR - https://github.com/apache/ignite/pull/1815

Upsource - https://reviews.ignite.apache.org/ignite/review/IGNT-CR-186

Discussion -
http://apache-ignite-developers.2346864.n4.nabble.com/TX-savepoints-td12041.html

[GitHub] ignite pull request #4676: Run 20 times rebalancing sync test

[Mmuzaf]

GitHub user Mmuzaf opened a pull request:

https://github.com/apache/ignite/pull/4676

Run 20 times rebalancing sync test



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Mmuzaf/ignite ig-9353-long

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4676.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4676


commit 67709429a708d2f4c692cadce9d72cd2855621b9
Author: Maxim Muzafarov 
Date:   2018-09-04T09:21:00Z

Run 20 times rebalancing sync test




---

Re: Python thin client

[Nikolay Izhikov]

Hello, Dmitry.

I understand that for experienced Python developer it obvious from stack trace 
I send.

But can we check python version on startup?
And print big fat error message "You are using wrong python version".

From my experience, there are some tickets in Ignite that should be implemented 
in various thin clients.
It a very trivial changes, but lack of testability makes this task harder then 
steel.

I think a .Net DEVNOTES are very good example.

Please, be gentle with your fellow contributors and make DEVNOTES as clear as 
possible.

В Пн, 03/09/2018 в 17:07 +1000, Dmitry Melnichuk пишет:
> Hello, Nikolay!
> 
> Thank you for your interest in Python thin client.
> 
> The traceback suggests that you using Python 2.7. Unfortunately, my 
> client supports only Python versions 3.4 and later.
> 
> Since you are using the latest Ubuntu OS, chances are you already have a 
> recent Python 3 installed. Try `pip3`.
> 
> On 9/3/18 4:25 PM, Nikolay Izhikov wrote:
> > Hello, Dmitry.
> > 
> > I tried to build your lib locally and it failed.
> > Error message and pip version are below.
> > 
> > Seems, we have to fix developer instructions.
> > Do we need specific version of pip or something?
> > I tried to use default versions.
> > I using Ubuntu Linux.
> > 
> > dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ pip install -v 
> > -e .
> > Obtaining file:///home/dragon/src/ignite/modules/platforms/python
> >Running setup.py 
> > (path:/home/dragon/src/ignite/modules/platforms/python/setup.py) egg_info 
> > for package from file:///home/dragon/src/ignite/modules/platforms/python
> >  Running command python setup.py egg_info
> >  Traceback (most recent call last):
> >File "", line 1, in 
> >File "/home/dragon/src/ignite/modules/platforms/python/setup.py", 
> > line 20
> >  def is_a_requirement(line: str) -> bool:
> >   ^
> >  SyntaxError: invalid syntax
> > Cleaning up...
> > Command "python setup.py egg_info" failed with error code 1 in 
> > /home/dragon/src/ignite/modules/platforms/python/
> > Exception information:
> > Traceback (most recent call last):
> >File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in 
> > main
> >  status = self.run(options, args)
> >File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 
> > 353, in run
> >  wb.build(autobuilding=True)
> >File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 749, in build
> >  self.requirement_set.prepare_files(self.finder)
> >File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 380, in 
> > prepare_files
> >  ignore_dependencies=self.ignore_dependencies))
> >File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 518, in 
> > _prepare_file
> >  abstract_dist.prep_for_dist()
> >File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 129, in 
> > prep_for_dist
> >  self.req_to_install.run_egg_info()
> >File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 
> > 439, in run_egg_info
> >  command_desc='python setup.py egg_info')
> >File "/usr/lib/python2.7/dist-packages/pip/utils/__init__.py", line 725, 
> > in call_subprocess
> >  % (command_desc, proc.returncode, cwd))
> > InstallationError: Command "python setup.py egg_info" failed with error 
> > code 1 in /home/dragon/src/ignite/modules/platforms/python/
> > dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ apt-cache 
> > policy python-pip
> > python-pip:
> >Установлен: 9.0.1-2.3~ubuntu1
> >Кандидат:   9.0.1-2.3~ubuntu1
> >Таблица версий:
> >   *** 9.0.1-2.3~ubuntu1 500
> >  500 http://ru.archive.ubuntu.com/ubuntu bionic-updates/universe 
> > amd64 Packages
> >  500 http://ru.archive.ubuntu.com/ubuntu bionic-updates/universe 
> > i386 Packages
> >  100 /var/lib/dpkg/status
> >   9.0.1-2 500
> >  500 http://ru.archive.ubuntu.com/ubuntu bionic/universe amd64 
> > Packages
> >  500 http://ru.archive.ubuntu.com/ubuntu bionic/universe i386 
> > Packages
> > dragon:~/src/ignite/modules/platforms/python:[IGNITE-7782]$ uname -a
> > Linux newDragon 4.15.0-33-generic #36-Ubuntu SMP Wed Aug 15 16:00:05 UTC 
> > 2018 x86_64 x86_64 x86_64 GNU/Linux
> > 

signature.asc
Description: This is a digitally signed message part

Questions about getAllInternal(...)

[steve.hostett...@gmail.com]

Hello, 

in the case of local caches without eviction policy. I have the following
questions:

1) I would to understand why, in the method, getAllInternal the method
entryEx(cacheKey); uses the topology in the case of a local cache.
Furthermore, it calls the method  map.putEntryIfObsoleteOrAbsent. But the
entry cannot be obsolete or absent because it is a local cache.


2) Similarly, why is there a touch on the key : ctx.evicts().touch(entry,
ctx.affinity().affinityTopologyVersion()); when the evicting policy is null
(never evict)? This puts locks even when the context is lock=false.


Thanks a lot for shedding some light on this.

Steve



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[GitHub] ignite pull request #4536: IGNITE-9141 Implemented

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4536


---

[GitHub] SomeFire opened a new pull request #5: IGNITE-9377

[GitBox]

SomeFire opened a new pull request #5: IGNITE-9377
URL: https://github.com/apache/ignite-teamcity-bot/pull/5
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Re: Compression prototype

[Dmitriy Setrakyan]

On Tue, Sep 4, 2018 at 1:16 AM, Ilya Kasnacheev 
wrote:

> Hello!
>
> The compression is per-binary-object, but dictionary is external, shared
> between multiple (millions of) entries and stored alongside compressed
> data.
>

I was under a different impression. If the dictionary is for the whole data
set, then it will occupy megabytes (if not gigabytes) of data. What happens
when a new node joins and has no idea about the dictionary? What happens
when dictionary between nodes get out-of-sync?

D.

Re: Unknown known issue on cache rebalancing delayed

[Anton Vinogradov]

Roman,

I see you uncommented this line.
I do not remember deadlock detail, but I remember it was the extremely rare
case.
I found and "fixed" it some days before merge when I had 24x7 sanity check
week :)

So, I propose to have at least 1_000 runs of this tests before keeping this
uncommented.



вт, 21 авг. 2018 г. в 11:08, Maxim Muzafarov :

> Roman,
>
> I worked recently on rebalance improvements and haven't found any problems
> with delayed cache rebalacne.
> Agree with you - let's uncomment this and remove scary comment. Will you
> create a ticket for it?
>
> In case of any problems we can easily detec deadlock with newly configured
> `FailureHandler`.
>
> On Tue, 21 Aug 2018 at 03:49 Roman Shtykh  wrote:
>
> > Hi Maxim,
> >
> > I have some issues with a cluster with rebalance delay enabled, but need
> > to check more -- if I find it's related I'll share.
> > Just wanted to make sure it's not an issue anymore from someone working
> on
> > rebalancing. We should remove that comment then, it looks scary :)
> >
> > --
> > Roman Shtykh
> >
> >
> > On Tuesday, August 21, 2018, 12:49:00 a.m. GMT+9, Maxim Muzafarov <
> > maxmu...@gmail.com> wrote:
> >
> >
> > Hello Roman,
> >
> > Did you faced with real issue of delayed rebalance or it's just only for
> > your personal interest?
> > If yes, please, share details and we will try to help you.
> >
> > As for this comment I don't think he is actual. That change was in 2015.
> > Much has changed
> > within rebalance process since that time. I've uncommented it and
> > rechecked with that
> > cache configuration and haven't seen any failed tests or issues.
> >
> > Probably, that problem was about cache in SYNC mode does not start util
> it
> > loads all data
> > from other nodes. But currently delayed rebalance works the same way as
> > IgniteCache#rebalance(),
> > so you can `setRebalanceDelay` to `-1` and call it manually to check.
> >
> > On Mon, 20 Aug 2018 at 11:19 Roman Shtykh 
> > wrote:
> >
> > Igniters,
> > I have found "Known issue, possible deadlock in case of low priority
> cache
> > rebalancing delayed" comment in
> > GridCacheRebalancingSyncSelfTest#getConfiguration.Can you please explain
> > when using rebalance delay can be an issue and why?
> >
> > -- Roman
> >
> > --
> > --
> > Maxim Muzafarov
> >
> --
> --
> Maxim Muzafarov
>

Re: Compression prototype

[Ilya Kasnacheev]

Hello!

The compression is per-binary-object, but dictionary is external, shared
between multiple (millions of) entries and stored alongside compressed data.

Regards,
-- 
Ilya Kasnacheev


вт, 4 сент. 2018 г. в 2:40, Dmitriy Setrakyan :

> Hi Ilya,
>
> This is very useful. Is the compression going to be per-page, in which case
> the dictionary is going to be kept inside of a page? Or do you have some
> other design in mind?
>
> D.
>
> On Mon, Sep 3, 2018 at 10:36 AM, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com>
> wrote:
>
> > Hello again!
> >
> > I've been running various compression parameters through cod dataset.
> >
> > It looks like the best compression level in terms of speed is either 1 or
> > 2.
> > The default for Zstd seems to be 3 which would almost always perform
> worse.
> > For best performance a dictionary of 1024 is optimal, for better
> > compression
> > one might choose larger dictionaries, 6k looks good but I will also run a
> > few benchmarks on larger dicts. Unfortunately, Zstd crashes if sample
> size
> > is set to more than 16k entries (I guess I should probe the max buffer
> size
> > where problems begin).
> >
> > I'm attaching two charts which show what's we've got. Compression rate
> is a
> > fraction of original records size. Time to run is wall clock time the
> test
> > run. Reasonable compression will increase the run time twofold (of a
> > program
> > that only does text record parsing -> creates objects -> binarylizes them
> > ->
> > compresses -> decompresses). Notation: s{number of bin objects used to
> > train}-d{dictionary length in bytes}-l{compression level}.
> >  > com/file/t374/chart1.png>
> > Second one is basically a zoom in on the first.
> >  > com/file/t374/chart2.png>
> > I think that in additional to dictionary compression we should have
> > dictionary-less compression. On typical data of small records it shows
> > compression rate of 0.8 ~ 0.65, but I can imagine that with larger
> > unstructured records it can be as good as dict-based and much less of a
> > hassle dictionary-processing-wise. WDYT?
> > Sorry for the fine prints. I hope my charts will visible.
> >
> > You can see the updated code as pull request:
> > https://github.com/apache/ignite/pull/4673
> >
> > Regards,
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>

Re: Hello!

[Alexey Goncharuk]

Hello Ivan,

Welcome to the Ignite community! I've added you to the list of
contributors, you should now be able to assign tickets to yourself.

Get familiar with Apache Ignite development process described here:
https://cwiki.apache.org/confluence/display/IGNITE/Development+Process

Instructions on how to contribute can be found here:
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute

Project setup in Intellij IDEA:
https://cwiki.apache.org/confluence/display/IGNITE/Project+Setup

--AG

пн, 3 сент. 2018 г. в 18:27, Ivan Bessonov :

> I'm new to Ignite and I would like to join Apache Ignite development.
> My JIRA's login is ibessonov
>