Re: Annoying extra steps for enabling metrics

[Alex Plehanov]

Due to holidays I can start work on this ticket only after 8 jan 2018

2017-12-30 2:12 GMT+03:00 Denis Magda :

> Good, closed the original ticket.
>
> Alex P, do you have time to work on IGNITE-7346 instead to address the
> issue with the cache events per cache in 2.4 release?
>
> —
> Denis
>
> > On Dec 29, 2017, at 3:10 PM, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
> >
> > Agree.
> >
> > -Val
> >
> > On Fri, Dec 29, 2017 at 3:08 PM, Denis Magda  wrote:
> >
> >> Now I see. Seems I was doing something wrong in my initial reproducer.
> >>
> >> Updated cache metrics readme doc by purging any events related
> parameters
> >> from there:
> >> https://apacheignite.readme.io/v2.3/docs/cache-metrics <
> >> https://apacheignite.readme.io/v2.3/docs/cache-metrics>
> >>
> >> The events readme doc looks good to me. Just updated the javadoc of
> >> IgniteEvents#*Query methods to bring MemoryEventStorageSpi to users
> >> attention.
> >>
> >> As for the cache events, it’s an oversight that there is no parameter
> that
> >> enables/disables the events per cache. Let’s add CacheConfiguration.
> setEventsEnabled
> >> method and have it enabled by default (current behavior). If the user
> >> deploys hundreds of caches or just interested in the events of specific
> >> ones then he can always set this property to ‘false’ in configuration of
> >> the caches of no interest:
> >> https://issues.apache.org/jira/browse/IGNITE-7346 <
> >> https://issues.apache.org/jira/browse/IGNITE-7346>
> >>
> >> Agree?
> >>
> >> —
> >> Denis
> >>
> >>
> >>
> >>> On Dec 29, 2017, at 11:10 AM, Valentin Kulichenko <
> >> valentin.kuliche...@gmail.com> wrote:
> >>>
> >>> Guys,
> >>>
> >>> I'm not sure what issue we're trying to solve. Basically, we have three
> >>> different functionality parts here:
> >>>
> >>> 1. Cache metrics exposed via CacheMetrics interface and MBeans (number
> of
> >>> puts, average put time, this kind of stuff). These are controlled on
> per
> >>> cache level by CacheConfiguration#statisticsEnabled property.
> >>> 2. Listening to cache events. To achieve this you only need to enable
> >>> particular event types and register listeners, this does not depend on
> >>> CacheConfiguration#statisticsEnabled. Here is the test confirming
> this:
> >>> https://gist.github.com/vkulichenko/54043c7b75c69eca5515e7d7692b5276
> >>> 3. Querying cache events via IgniteEvents#*Query methods. This is the
> >> ONLY
> >>> piece that requires MemoryEventStorageSpi to be configured. Since
> >> querying
> >>> is not very frequently used, and event storage introduces significant
> >>> memory overhead, I don't think we should ever implicitly enable it. We
> >>> deliberately made no-op implementation the default one not so long ago.
> >>>
> >>> All three points above seem to work without any issues. The only useful
> >>> addition I see here is ability to enable cache events on per cache
> level.
> >>> But for this we can introduce new property to CacheConfiguration. I
> would
> >>> not mix metrics and events together as these are different APIs
> designed
> >>> for different purposes.
> >>>
> >>> Am I missing something?
> >>>
> >>> -Val
> >>>
> >>> On Fri, Dec 29, 2017 at 8:02 AM, Denis Magda 
> wrote:
> >>>
>  Alexey,
> 
>  I think we should enable memoryEventStorageSPI automatically whenever
>  statisticsEnabled is toggled on. A special message has to be added to
> >> the
>  log pointing out that the automatic activation happened.
> 
>  Does it sound like a good solution?
> 
>  —
>  Denis
> 
> > On Dec 29, 2017, at 3:51 AM, Alexey Plekhanov <
> plehanov.a...@gmail.com
> >>>
>  wrote:
> >
> > Denis, I start working on the issue
> > https://issues.apache.org/jira/browse/IGNITE-6925 and now I don't
>  understand
> > why these properties must be bound to each other?
> >
> > • If we enable statistics on caches, this does not necessarily mean,
>  that
> > we wanted to get some events, we can enable statistics for other
> >> reasons.
> > Conversely, not all events need statistics to be enabled on caches.
> So
> >> we
> > can’t bind statistics flag to events (subscribe to events when
>  statistics is
> > enabled or enable statistics, when subscribing to events)
> > • We can’t set events of interest, when we set not a dummy
>  EventsStorageSpi,
> > because we don’t know which events are interesting.
> > • When we set events of interest, it’s not necessary, that these
> events
>  will
> > be monitored using EventsStorageSpi, we can also subscribe to events
> by
> > events listeners, in this case EventsStorageSpi don’t used.
> >
> > So, there is no general rule (if ... enabled, then ... must be
> enabled
>  too).
> > The only implementation I can propose - is "set MemoryEventStorageSPI
> > instead of NoopEventStorageSPI when includeEventTypes list is not
> >> empty",
> > but even this impl

Re: Spark data frames integration merged

[Nikolay Izhikov]

Thank you, guys.

Val, thanks for all reviews, advices and patience.

Anton, thanks for ignite wisdom you share with me.

Looking forward for next issues :)

P.S Happy New Year for all Ignite community!

В Пт, 29/12/2017 в 13:22 -0800, Valentin Kulichenko пишет:
> Igniters,
> 
> Great news! We completed and merged first part of integration with
> Spark data frames [1]. It contains implementation of Spark data
> source which allows to use DataFrame API to query Ignite data, as
> well as join it with other data frames originated from different
> sources.
> 
> Next planned steps are the following:
> - Implement custom execution strategy to avoid transferring data from
> Ignite to Spark when possible [2]. This should give serious
> performance improvement in cases when only Ignite tables participate
> in a query.
> - Implement ability to save a data frame into Ignite via
> DataFrameWrite API [3].
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-3084
> [2] https://issues.apache.org/jira/browse/IGNITE-7077
> [3] https://issues.apache.org/jira/browse/IGNITE-7337
> 
> Nikolay Izhikov, thanks for the contribution and for all the hard
> work!
> 
> -Val

[GitHub] ignite pull request #3135: IGNITE-6903: Indexes size metric implemented.

[nizhikov]

Github user nizhikov closed the pull request at:

https://github.com/apache/ignite/pull/3135


---

Re: Annoying extra steps for enabling metrics

[Denis Magda]

Good, closed the original ticket. 

Alex P, do you have time to work on IGNITE-7346 instead to address the issue 
with the cache events per cache in 2.4 release?

—
Denis

> On Dec 29, 2017, at 3:10 PM, Valentin Kulichenko 
>  wrote:
> 
> Agree.
> 
> -Val
> 
> On Fri, Dec 29, 2017 at 3:08 PM, Denis Magda  wrote:
> 
>> Now I see. Seems I was doing something wrong in my initial reproducer.
>> 
>> Updated cache metrics readme doc by purging any events related parameters
>> from there:
>> https://apacheignite.readme.io/v2.3/docs/cache-metrics <
>> https://apacheignite.readme.io/v2.3/docs/cache-metrics>
>> 
>> The events readme doc looks good to me. Just updated the javadoc of
>> IgniteEvents#*Query methods to bring MemoryEventStorageSpi to users
>> attention.
>> 
>> As for the cache events, it’s an oversight that there is no parameter that
>> enables/disables the events per cache. Let’s add 
>> CacheConfiguration.setEventsEnabled
>> method and have it enabled by default (current behavior). If the user
>> deploys hundreds of caches or just interested in the events of specific
>> ones then he can always set this property to ‘false’ in configuration of
>> the caches of no interest:
>> https://issues.apache.org/jira/browse/IGNITE-7346 <
>> https://issues.apache.org/jira/browse/IGNITE-7346>
>> 
>> Agree?
>> 
>> —
>> Denis
>> 
>> 
>> 
>>> On Dec 29, 2017, at 11:10 AM, Valentin Kulichenko <
>> valentin.kuliche...@gmail.com> wrote:
>>> 
>>> Guys,
>>> 
>>> I'm not sure what issue we're trying to solve. Basically, we have three
>>> different functionality parts here:
>>> 
>>> 1. Cache metrics exposed via CacheMetrics interface and MBeans (number of
>>> puts, average put time, this kind of stuff). These are controlled on per
>>> cache level by CacheConfiguration#statisticsEnabled property.
>>> 2. Listening to cache events. To achieve this you only need to enable
>>> particular event types and register listeners, this does not depend on
>>> CacheConfiguration#statisticsEnabled. Here is the test confirming this:
>>> https://gist.github.com/vkulichenko/54043c7b75c69eca5515e7d7692b5276
>>> 3. Querying cache events via IgniteEvents#*Query methods. This is the
>> ONLY
>>> piece that requires MemoryEventStorageSpi to be configured. Since
>> querying
>>> is not very frequently used, and event storage introduces significant
>>> memory overhead, I don't think we should ever implicitly enable it. We
>>> deliberately made no-op implementation the default one not so long ago.
>>> 
>>> All three points above seem to work without any issues. The only useful
>>> addition I see here is ability to enable cache events on per cache level.
>>> But for this we can introduce new property to CacheConfiguration. I would
>>> not mix metrics and events together as these are different APIs designed
>>> for different purposes.
>>> 
>>> Am I missing something?
>>> 
>>> -Val
>>> 
>>> On Fri, Dec 29, 2017 at 8:02 AM, Denis Magda  wrote:
>>> 
 Alexey,
 
 I think we should enable memoryEventStorageSPI automatically whenever
 statisticsEnabled is toggled on. A special message has to be added to
>> the
 log pointing out that the automatic activation happened.
 
 Does it sound like a good solution?
 
 —
 Denis
 
> On Dec 29, 2017, at 3:51 AM, Alexey Plekhanov >> 
 wrote:
> 
> Denis, I start working on the issue
> https://issues.apache.org/jira/browse/IGNITE-6925 and now I don't
 understand
> why these properties must be bound to each other?
> 
> • If we enable statistics on caches, this does not necessarily mean,
 that
> we wanted to get some events, we can enable statistics for other
>> reasons.
> Conversely, not all events need statistics to be enabled on caches. So
>> we
> can’t bind statistics flag to events (subscribe to events when
 statistics is
> enabled or enable statistics, when subscribing to events)
> • We can’t set events of interest, when we set not a dummy
 EventsStorageSpi,
> because we don’t know which events are interesting.
> • When we set events of interest, it’s not necessary, that these events
 will
> be monitored using EventsStorageSpi, we can also subscribe to events by
> events listeners, in this case EventsStorageSpi don’t used.
> 
> So, there is no general rule (if ... enabled, then ... must be enabled
 too).
> The only implementation I can propose - is "set MemoryEventStorageSPI
> instead of NoopEventStorageSPI when includeEventTypes list is not
>> empty",
> but even this implementation may be warranted only in some cases.
> 
> 
> Denis Magda-2 wrote
>> Let’s simplifying the metrics as a part of this ticket:
>> https://issues.apache.org/jira/browse/IGNITE-5796
>> ;
>> 
>> Expanded its scope.
>> 
>> —
>> Denis
>> 
>>> On Sep 9, 2017, at 2:44 PM, Valentin Kulichenko <
> 
>>

Re: Annoying extra steps for enabling metrics

[Valentin Kulichenko]

Agree.

-Val

On Fri, Dec 29, 2017 at 3:08 PM, Denis Magda  wrote:

> Now I see. Seems I was doing something wrong in my initial reproducer.
>
> Updated cache metrics readme doc by purging any events related parameters
> from there:
> https://apacheignite.readme.io/v2.3/docs/cache-metrics <
> https://apacheignite.readme.io/v2.3/docs/cache-metrics>
>
> The events readme doc looks good to me. Just updated the javadoc of
> IgniteEvents#*Query methods to bring MemoryEventStorageSpi to users
> attention.
>
> As for the cache events, it’s an oversight that there is no parameter that
> enables/disables the events per cache. Let’s add 
> CacheConfiguration.setEventsEnabled
> method and have it enabled by default (current behavior). If the user
> deploys hundreds of caches or just interested in the events of specific
> ones then he can always set this property to ‘false’ in configuration of
> the caches of no interest:
> https://issues.apache.org/jira/browse/IGNITE-7346 <
> https://issues.apache.org/jira/browse/IGNITE-7346>
>
> Agree?
>
> —
> Denis
>
>
>
> > On Dec 29, 2017, at 11:10 AM, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
> >
> > Guys,
> >
> > I'm not sure what issue we're trying to solve. Basically, we have three
> > different functionality parts here:
> >
> > 1. Cache metrics exposed via CacheMetrics interface and MBeans (number of
> > puts, average put time, this kind of stuff). These are controlled on per
> > cache level by CacheConfiguration#statisticsEnabled property.
> > 2. Listening to cache events. To achieve this you only need to enable
> > particular event types and register listeners, this does not depend on
> > CacheConfiguration#statisticsEnabled. Here is the test confirming this:
> > https://gist.github.com/vkulichenko/54043c7b75c69eca5515e7d7692b5276
> > 3. Querying cache events via IgniteEvents#*Query methods. This is the
> ONLY
> > piece that requires MemoryEventStorageSpi to be configured. Since
> querying
> > is not very frequently used, and event storage introduces significant
> > memory overhead, I don't think we should ever implicitly enable it. We
> > deliberately made no-op implementation the default one not so long ago.
> >
> > All three points above seem to work without any issues. The only useful
> > addition I see here is ability to enable cache events on per cache level.
> > But for this we can introduce new property to CacheConfiguration. I would
> > not mix metrics and events together as these are different APIs designed
> > for different purposes.
> >
> > Am I missing something?
> >
> > -Val
> >
> > On Fri, Dec 29, 2017 at 8:02 AM, Denis Magda  wrote:
> >
> >> Alexey,
> >>
> >> I think we should enable memoryEventStorageSPI automatically whenever
> >> statisticsEnabled is toggled on. A special message has to be added to
> the
> >> log pointing out that the automatic activation happened.
> >>
> >> Does it sound like a good solution?
> >>
> >> —
> >> Denis
> >>
> >>> On Dec 29, 2017, at 3:51 AM, Alexey Plekhanov  >
> >> wrote:
> >>>
> >>> Denis, I start working on the issue
> >>> https://issues.apache.org/jira/browse/IGNITE-6925 and now I don't
> >> understand
> >>> why these properties must be bound to each other?
> >>>
> >>> • If we enable statistics on caches, this does not necessarily mean,
> >> that
> >>> we wanted to get some events, we can enable statistics for other
> reasons.
> >>> Conversely, not all events need statistics to be enabled on caches. So
> we
> >>> can’t bind statistics flag to events (subscribe to events when
> >> statistics is
> >>> enabled or enable statistics, when subscribing to events)
> >>> • We can’t set events of interest, when we set not a dummy
> >> EventsStorageSpi,
> >>> because we don’t know which events are interesting.
> >>> • When we set events of interest, it’s not necessary, that these events
> >> will
> >>> be monitored using EventsStorageSpi, we can also subscribe to events by
> >>> events listeners, in this case EventsStorageSpi don’t used.
> >>>
> >>> So, there is no general rule (if ... enabled, then ... must be enabled
> >> too).
> >>> The only implementation I can propose - is "set MemoryEventStorageSPI
> >>> instead of NoopEventStorageSPI when includeEventTypes list is not
> empty",
> >>> but even this implementation may be warranted only in some cases.
> >>>
> >>>
> >>> Denis Magda-2 wrote
>  Let’s simplifying the metrics as a part of this ticket:
>  https://issues.apache.org/jira/browse/IGNITE-5796
>  ;
> 
>  Expanded its scope.
> 
>  —
>  Denis
> 
> > On Sep 9, 2017, at 2:44 PM, Valentin Kulichenko <
> >>>
>  valentin.kulichenko@
> >>>
>  > wrote:
> >
> > statisticsEnabled property comes from JCache, BTW.
> >
> > -Val
> >
> > On Sat, Sep 9, 2017 at 11:09 AM, Dmitriy Setrakyan <
> >>>
>  dsetrakyan@
> >>>
>  >
> > wrote:
> >
> >> On Sat, Sep 9, 2017 at 8:56 AM, Denis

Re: Annoying extra steps for enabling metrics

[Denis Magda]

Now I see. Seems I was doing something wrong in my initial reproducer.

Updated cache metrics readme doc by purging any events related parameters from 
there:
https://apacheignite.readme.io/v2.3/docs/cache-metrics 


The events readme doc looks good to me. Just updated the javadoc of 
IgniteEvents#*Query methods to bring MemoryEventStorageSpi to users attention.

As for the cache events, it’s an oversight that there is no parameter that 
enables/disables the events per cache. Let’s add 
CacheConfiguration.setEventsEnabled method and have it enabled by default 
(current behavior). If the user deploys hundreds of caches or just interested 
in the events of specific ones then he can always set this property to ‘false’ 
in configuration of the caches of no interest:
https://issues.apache.org/jira/browse/IGNITE-7346 


Agree?

—
Denis



> On Dec 29, 2017, at 11:10 AM, Valentin Kulichenko 
>  wrote:
> 
> Guys,
> 
> I'm not sure what issue we're trying to solve. Basically, we have three
> different functionality parts here:
> 
> 1. Cache metrics exposed via CacheMetrics interface and MBeans (number of
> puts, average put time, this kind of stuff). These are controlled on per
> cache level by CacheConfiguration#statisticsEnabled property.
> 2. Listening to cache events. To achieve this you only need to enable
> particular event types and register listeners, this does not depend on
> CacheConfiguration#statisticsEnabled. Here is the test confirming this:
> https://gist.github.com/vkulichenko/54043c7b75c69eca5515e7d7692b5276
> 3. Querying cache events via IgniteEvents#*Query methods. This is the ONLY
> piece that requires MemoryEventStorageSpi to be configured. Since querying
> is not very frequently used, and event storage introduces significant
> memory overhead, I don't think we should ever implicitly enable it. We
> deliberately made no-op implementation the default one not so long ago.
> 
> All three points above seem to work without any issues. The only useful
> addition I see here is ability to enable cache events on per cache level.
> But for this we can introduce new property to CacheConfiguration. I would
> not mix metrics and events together as these are different APIs designed
> for different purposes.
> 
> Am I missing something?
> 
> -Val
> 
> On Fri, Dec 29, 2017 at 8:02 AM, Denis Magda  wrote:
> 
>> Alexey,
>> 
>> I think we should enable memoryEventStorageSPI automatically whenever
>> statisticsEnabled is toggled on. A special message has to be added to the
>> log pointing out that the automatic activation happened.
>> 
>> Does it sound like a good solution?
>> 
>> —
>> Denis
>> 
>>> On Dec 29, 2017, at 3:51 AM, Alexey Plekhanov 
>> wrote:
>>> 
>>> Denis, I start working on the issue
>>> https://issues.apache.org/jira/browse/IGNITE-6925 and now I don't
>> understand
>>> why these properties must be bound to each other?
>>> 
>>> • If we enable statistics on caches, this does not necessarily mean,
>> that
>>> we wanted to get some events, we can enable statistics for other reasons.
>>> Conversely, not all events need statistics to be enabled on caches. So we
>>> can’t bind statistics flag to events (subscribe to events when
>> statistics is
>>> enabled or enable statistics, when subscribing to events)
>>> • We can’t set events of interest, when we set not a dummy
>> EventsStorageSpi,
>>> because we don’t know which events are interesting.
>>> • When we set events of interest, it’s not necessary, that these events
>> will
>>> be monitored using EventsStorageSpi, we can also subscribe to events by
>>> events listeners, in this case EventsStorageSpi don’t used.
>>> 
>>> So, there is no general rule (if ... enabled, then ... must be enabled
>> too).
>>> The only implementation I can propose - is "set MemoryEventStorageSPI
>>> instead of NoopEventStorageSPI when includeEventTypes list is not empty",
>>> but even this implementation may be warranted only in some cases.
>>> 
>>> 
>>> Denis Magda-2 wrote
 Let’s simplifying the metrics as a part of this ticket:
 https://issues.apache.org/jira/browse/IGNITE-5796
 ;
 
 Expanded its scope.
 
 —
 Denis
 
> On Sep 9, 2017, at 2:44 PM, Valentin Kulichenko <
>>> 
 valentin.kulichenko@
>>> 
 > wrote:
> 
> statisticsEnabled property comes from JCache, BTW.
> 
> -Val
> 
> On Sat, Sep 9, 2017 at 11:09 AM, Dmitriy Setrakyan <
>>> 
 dsetrakyan@
>>> 
 >
> wrote:
> 
>> On Sat, Sep 9, 2017 at 8:56 AM, Denis Magda <
>>> 
 dmagda@
>>> 
 > wrote:
>> 
>>> Surprise!
>>> 
>>> If you want to see cache events then you have to enable one more
>> flag!
>>> 
>>> 
 
>> 
>> 
>> What is the overhead of this statistics collection?
>> 
>> 
>>> Three flags/beans have to be in the config 

[jira] [Created] (IGNITE-7346) Enable Ignite cache events per cache

[Denis Magda (JIRA)]

Denis Magda created IGNITE-7346:
---

 Summary: Enable Ignite cache events per cache
 Key: IGNITE-7346
 URL: https://issues.apache.org/jira/browse/IGNITE-7346
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Reporter: Denis Magda
 Fix For: 2.4


Whenever a cache event from {{EventType.EVTS_CACHE}}, 
{{EventType.EVTS_CACHE_REBALANCE}} or {{EventType.EVTS_CACHE_QUERY}} set is 
enabled and a listener is registered, the cluster will accumulate and send out 
events for all the deployed caches.

Such behavior brings down performance especially if hundreds and thousands of 
caches are used while the user is interested in the monitoring of specific 
caches.

Let's add a configuration property like {{CacheConfiguration.setEventsEnabled}} 
to enable or disable the events per cache. The events will be enabled by 
default to comply with existing behavior.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Annoying extra steps for enabling metrics

[Alex Plehanov]

Valentine, yes, that's exactly what I'm trying to say. I don't see direct
dependencies between these properties (when a property must be set in all
cases another property is set).

2017-12-29 22:10 GMT+03:00 Valentin Kulichenko <
valentin.kuliche...@gmail.com>:

> Guys,
>
> I'm not sure what issue we're trying to solve. Basically, we have three
> different functionality parts here:
>
> 1. Cache metrics exposed via CacheMetrics interface and MBeans (number of
> puts, average put time, this kind of stuff). These are controlled on per
> cache level by CacheConfiguration#statisticsEnabled property.
> 2. Listening to cache events. To achieve this you only need to enable
> particular event types and register listeners, this does not depend on
> CacheConfiguration#statisticsEnabled. Here is the test confirming this:
> https://gist.github.com/vkulichenko/54043c7b75c69eca5515e7d7692b5276
> 3. Querying cache events via IgniteEvents#*Query methods. This is the ONLY
> piece that requires MemoryEventStorageSpi to be configured. Since querying
> is not very frequently used, and event storage introduces significant
> memory overhead, I don't think we should ever implicitly enable it. We
> deliberately made no-op implementation the default one not so long ago.
>
> All three points above seem to work without any issues. The only useful
> addition I see here is ability to enable cache events on per cache level.
> But for this we can introduce new property to CacheConfiguration. I would
> not mix metrics and events together as these are different APIs designed
> for different purposes.
>
> Am I missing something?
>
> -Val
>
> On Fri, Dec 29, 2017 at 8:02 AM, Denis Magda  wrote:
>
> > Alexey,
> >
> > I think we should enable memoryEventStorageSPI automatically whenever
> > statisticsEnabled is toggled on. A special message has to be added to the
> > log pointing out that the automatic activation happened.
> >
> > Does it sound like a good solution?
> >
> > —
> > Denis
> >
> > > On Dec 29, 2017, at 3:51 AM, Alexey Plekhanov  >
> > wrote:
> > >
> > > Denis, I start working on the issue
> > > https://issues.apache.org/jira/browse/IGNITE-6925 and now I don't
> > understand
> > > why these properties must be bound to each other?
> > >
> > > • If we enable statistics on caches, this does not necessarily mean,
> > that
> > > we wanted to get some events, we can enable statistics for other
> reasons.
> > > Conversely, not all events need statistics to be enabled on caches. So
> we
> > > can’t bind statistics flag to events (subscribe to events when
> > statistics is
> > > enabled or enable statistics, when subscribing to events)
> > > • We can’t set events of interest, when we set not a dummy
> > EventsStorageSpi,
> > > because we don’t know which events are interesting.
> > > • When we set events of interest, it’s not necessary, that these events
> > will
> > > be monitored using EventsStorageSpi, we can also subscribe to events by
> > > events listeners, in this case EventsStorageSpi don’t used.
> > >
> > > So, there is no general rule (if ... enabled, then ... must be enabled
> > too).
> > > The only implementation I can propose - is "set MemoryEventStorageSPI
> > > instead of NoopEventStorageSPI when includeEventTypes list is not
> empty",
> > > but even this implementation may be warranted only in some cases.
> > >
> > >
> > > Denis Magda-2 wrote
> > >> Let’s simplifying the metrics as a part of this ticket:
> > >> https://issues.apache.org/jira/browse/IGNITE-5796
> > >> ;
> > >>
> > >> Expanded its scope.
> > >>
> > >> —
> > >> Denis
> > >>
> > >>> On Sep 9, 2017, at 2:44 PM, Valentin Kulichenko <
> > >
> > >> valentin.kulichenko@
> > >
> > >> > wrote:
> > >>>
> > >>> statisticsEnabled property comes from JCache, BTW.
> > >>>
> > >>> -Val
> > >>>
> > >>> On Sat, Sep 9, 2017 at 11:09 AM, Dmitriy Setrakyan <
> > >
> > >> dsetrakyan@
> > >
> > >> >
> > >>> wrote:
> > >>>
> >  On Sat, Sep 9, 2017 at 8:56 AM, Denis Magda <
> > >
> > >> dmagda@
> > >
> > >> > wrote:
> > 
> > > Surprise!
> > >
> > > If you want to see cache events then you have to enable one more
> > flag!
> > >
> > >
> > >> 
> > 
> > 
> >  What is the overhead of this statistics collection?
> > 
> > 
> > > Three flags/beans have to be in the config in total, three! Just to
> > see
> > > cache events. The API is a mess. Let’s contemplate how to fix it.
> > 
> > 
> >  Agree, this is horrible. We need to fix it in 2.3. Is there a
> ticket?
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
> >
>

Re: Spark data frames integration merged

[Denis Magda]

Great news,

Thanks Nikolay and Val!

Nikolay, could you document the feature before the release [1]? I’ve granted 
you required permission.

More on the doc process can be found here [2].

[1] https://issues.apache.org/jira/browse/IGNITE-7345 

[2] https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document 


—
Denis

> On Dec 29, 2017, at 1:22 PM, Valentin Kulichenko 
>  wrote:
> 
> Igniters,
> 
> Great news! We completed and merged first part of integration with Spark data 
> frames [1]. It contains implementation of Spark data source which allows to 
> use DataFrame API to query Ignite data, as well as join it with other data 
> frames originated from different sources.
> 
> Next planned steps are the following:
> - Implement custom execution strategy to avoid transferring data from Ignite 
> to Spark when possible [2]. This should give serious performance improvement 
> in cases when only Ignite tables participate in a query.
> - Implement ability to save a data frame into Ignite via DataFrameWrite API 
> [3].
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-3084 
> 
> [2] https://issues.apache.org/jira/browse/IGNITE-7077 
> 
> [3] https://issues.apache.org/jira/browse/IGNITE-7337 
> 
> 
> Nikolay Izhikov, thanks for the contribution and for all the hard work!
> 
> -Val

[jira] [Created] (IGNITE-7345) Spark Data Frames and Ignite Documentation

[Denis Magda (JIRA)]

Denis Magda created IGNITE-7345:
---

 Summary: Spark Data Frames and Ignite Documentation
 Key: IGNITE-7345
 URL: https://issues.apache.org/jira/browse/IGNITE-7345
 Project: Ignite
  Issue Type: Task
  Components: documentation
Reporter: Denis Magda
Assignee: Nikolay Izhikov
 Fix For: 2.4


Spark Data frames integration [1] needs to be documented in this [2] domain.

[1] 
http://apache-ignite-developers.2346864.n4.nabble.com/Spark-data-frames-integration-merged-td25817.html
[2] https://apacheignite-fs.readme.io/docs



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Spark data frames integration merged

[Valentin Kulichenko]

Igniters,

Great news! We completed and merged first part of integration with Spark
data frames [1]. It contains implementation of Spark data source which
allows to use DataFrame API to query Ignite data, as well as join it with
other data frames originated from different sources.

Next planned steps are the following:
- Implement custom execution strategy to avoid transferring data from
Ignite to Spark when possible [2]. This should give serious performance
improvement in cases when only Ignite tables participate in a query.
- Implement ability to save a data frame into Ignite via DataFrameWrite API
[3].

[1] https://issues.apache.org/jira/browse/IGNITE-3084
[2] https://issues.apache.org/jira/browse/IGNITE-7077
[3] https://issues.apache.org/jira/browse/IGNITE-7337

Nikolay Izhikov, thanks for the contribution and for all the hard work!

-Val

[GitHub] ignite pull request #2742: IGNITE-3084: Implementation of Spark DataFrame fo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2742


---

Re: Annoying extra steps for enabling metrics

[Valentin Kulichenko]

Guys,

I'm not sure what issue we're trying to solve. Basically, we have three
different functionality parts here:

1. Cache metrics exposed via CacheMetrics interface and MBeans (number of
puts, average put time, this kind of stuff). These are controlled on per
cache level by CacheConfiguration#statisticsEnabled property.
2. Listening to cache events. To achieve this you only need to enable
particular event types and register listeners, this does not depend on
CacheConfiguration#statisticsEnabled. Here is the test confirming this:
https://gist.github.com/vkulichenko/54043c7b75c69eca5515e7d7692b5276
3. Querying cache events via IgniteEvents#*Query methods. This is the ONLY
piece that requires MemoryEventStorageSpi to be configured. Since querying
is not very frequently used, and event storage introduces significant
memory overhead, I don't think we should ever implicitly enable it. We
deliberately made no-op implementation the default one not so long ago.

All three points above seem to work without any issues. The only useful
addition I see here is ability to enable cache events on per cache level.
But for this we can introduce new property to CacheConfiguration. I would
not mix metrics and events together as these are different APIs designed
for different purposes.

Am I missing something?

-Val

On Fri, Dec 29, 2017 at 8:02 AM, Denis Magda  wrote:

> Alexey,
>
> I think we should enable memoryEventStorageSPI automatically whenever
> statisticsEnabled is toggled on. A special message has to be added to the
> log pointing out that the automatic activation happened.
>
> Does it sound like a good solution?
>
> —
> Denis
>
> > On Dec 29, 2017, at 3:51 AM, Alexey Plekhanov 
> wrote:
> >
> > Denis, I start working on the issue
> > https://issues.apache.org/jira/browse/IGNITE-6925 and now I don't
> understand
> > why these properties must be bound to each other?
> >
> > • If we enable statistics on caches, this does not necessarily mean,
> that
> > we wanted to get some events, we can enable statistics for other reasons.
> > Conversely, not all events need statistics to be enabled on caches. So we
> > can’t bind statistics flag to events (subscribe to events when
> statistics is
> > enabled or enable statistics, when subscribing to events)
> > • We can’t set events of interest, when we set not a dummy
> EventsStorageSpi,
> > because we don’t know which events are interesting.
> > • When we set events of interest, it’s not necessary, that these events
> will
> > be monitored using EventsStorageSpi, we can also subscribe to events by
> > events listeners, in this case EventsStorageSpi don’t used.
> >
> > So, there is no general rule (if ... enabled, then ... must be enabled
> too).
> > The only implementation I can propose - is "set MemoryEventStorageSPI
> > instead of NoopEventStorageSPI when includeEventTypes list is not empty",
> > but even this implementation may be warranted only in some cases.
> >
> >
> > Denis Magda-2 wrote
> >> Let’s simplifying the metrics as a part of this ticket:
> >> https://issues.apache.org/jira/browse/IGNITE-5796
> >> ;
> >>
> >> Expanded its scope.
> >>
> >> —
> >> Denis
> >>
> >>> On Sep 9, 2017, at 2:44 PM, Valentin Kulichenko <
> >
> >> valentin.kulichenko@
> >
> >> > wrote:
> >>>
> >>> statisticsEnabled property comes from JCache, BTW.
> >>>
> >>> -Val
> >>>
> >>> On Sat, Sep 9, 2017 at 11:09 AM, Dmitriy Setrakyan <
> >
> >> dsetrakyan@
> >
> >> >
> >>> wrote:
> >>>
>  On Sat, Sep 9, 2017 at 8:56 AM, Denis Magda <
> >
> >> dmagda@
> >
> >> > wrote:
> 
> > Surprise!
> >
> > If you want to see cache events then you have to enable one more
> flag!
> >
> >
> >> 
> 
> 
>  What is the overhead of this statistics collection?
> 
> 
> > Three flags/beans have to be in the config in total, three! Just to
> see
> > cache events. The API is a mess. Let’s contemplate how to fix it.
> 
> 
>  Agree, this is horrible. We need to fix it in 2.3. Is there a ticket?
> >
> >
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>
>

[GitHub] ignite pull request #3323: ignite-2.4.1-p7

[agoncharuk]

GitHub user agoncharuk opened a pull request:

https://github.com/apache/ignite/pull/3323

ignite-2.4.1-p7



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2.4.1-p7

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3323.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3323


commit e7ca9b65a68de7752195c8f4d2b5180f3c77d19f
Author: Dmitriy Govorukhin 
Date:   2017-11-13T18:52:47Z

ignite-blt-merge -> ignite-2.4.1

commit cc8168fc184bb7f5e3cc3bbb0743397097f78bfb
Author: Dmitriy Govorukhin 
Date:   2017-11-13T19:13:01Z

merge ignite-pitr-rc1 -> ignite-2.4.1

commit 87e6d74cf6a251c7984f9e68c391f790feccc281
Author: Dmitriy Govorukhin 
Date:   2017-11-14T12:49:33Z

ignite-gg-12877 Compact consistent ID in WAL

commit 9f5a22711baea05bd37ab07c8f928a4837dd83a4
Author: Ilya Lantukh 
Date:   2017-11-14T14:12:28Z

Fixed javadoc.

commit d5af2d78dd8eef8eca8ac5391d31d8c779649bb0
Author: Alexey Kuznetsov 
Date:   2017-11-15T08:09:00Z

IGNITE-6913 Baseline: Added new options to controls.sh for baseline 
manipulations.

commit 713924ce865752b6e99b03bd624136541cea5f9f
Author: Sergey Chugunov 
Date:   2017-11-15T09:03:12Z

IGNITE-5850 failover tests for cache operations during BaselineTopology 
changes

commit b65fd134e748d496f732ec2aa0953a0531f544b8
Author: Ilya Lantukh 
Date:   2017-11-15T12:54:35Z

TX read logging if PITR is enabled.

commit 9b2a567c0e04dc33116b51f88bee75f76e9107d1
Author: Ilya Lantukh 
Date:   2017-11-15T13:45:16Z

TX read logging if PITR is enabled.

commit 993058ccf0b2b8d9e80750c3e45a9ffa31d85dfa
Author: Dmitriy Govorukhin 
Date:   2017-11-15T13:51:54Z

ignite-2.4.1 optimization for store full set node more compacted

commit 1eba521f608d39967aec376b397b7fc800234e54
Author: Dmitriy Govorukhin 
Date:   2017-11-15T13:52:22Z

Merge remote-tracking branch 'professional/ignite-2.4.1' into ignite-2.4.1

commit 564b3fd51f8a7d1d81cb6874df66d0270623049c
Author: Sergey Chugunov 
Date:   2017-11-15T14:00:51Z

IGNITE-5850 fixed issue with initialization of data regions on node 
activation, fixed issue with auto-activation when random node joins inactive 
cluster with existing BLT

commit c6d1fa4da7adfadc80abdc7eaf6452b86a4f6aa4
Author: Sergey Chugunov 
Date:   2017-11-15T16:23:08Z

IGNITE-5850 transitionResult is set earlier when request for changing 
BaselineTopology is sent

commit d65674363163e38a4c5fdd73d1c8d8e1c7610797
Author: Sergey Chugunov 
Date:   2017-11-16T11:59:07Z

IGNITE-5850 new failover tests for changing BaselineTopology up (new node 
added to topology)

commit 20552f3851fe8825191b144179be032965e0b5c6
Author: Sergey Chugunov 
Date:   2017-11-16T12:53:43Z

IGNITE-5850 improved error message when online node is removed from baseline

commit 108bbcae4505ac904a6db774643ad600bfb42c21
Author: Sergey Chugunov 
Date:   2017-11-16T13:45:52Z

IGNITE-5850 BaselineTopology should not change on cluster deactivation

commit deb641ad3bdbf260fa60ad6bf607629652e324bd
Author: Dmitriy Govorukhin 
Date:   2017-11-17T09:45:44Z

ignite-2.4.1 truncate wal and checkpoint history on move/delete snapshot

commit 3c8b06f3659af30d1fd148ccc0f40e216a56c998
Author: Alexey Goncharuk 
Date:   2017-11-17T12:48:12Z

IGNITE-6947 Abandon remap after single map if future is done (fixes NPE)

commit ba2047e5ae7d271a677e0c418375d82d78c4023e
Author: devozerov 
Date:   2017-11-14T12:26:31Z

IGNITE-6901: Fixed assertion during 
IgniteH2Indexing.rebuildIndexesFromHash. This closes #3027.

commit abfc0466d6d61d87255d0fe38cbdf11ad46d4f89
Author: Sergey Chugunov 
Date:   2017-11-17T13:40:57Z

IGNITE-5850 tests for queries in presence of BaselineTopology

commit f4eabaf2a905abacc4c60c01d3ca04f6ca9ec188
Author: Sergey Chugunov 
Date:   2017-11-17T17:23:02Z

IGNITE-5850 implementation for setBaselineTopology(long topVer) migrated 
from wc-251

commit 4edeccd3e0b671aa277f58995df9ff9935baa95a
Author: EdShangGG 
Date:   2017-11-17T18:21:17Z

GG-13074 Multiple snapshot test failures after baseline topology is 
introduced
-adding baseline test to suite
-fixing issues with baseline

commit edae228c8f55990c15ef3044be987dcb00d6c81a
Author: EdShangGG 
Date:   2017-11-18T10:36:41Z

hack with sleep

commit b5bffc7580a4a8ffbcc06f60c282e73979179578
Author: Ilya Lantukh 
Date:   2017-11-18T12:39:19Z

Fixed Ignite.active(true) returning control too early.

commit 1bcdd76aae78665e2bbd49034fb46a1b91ef8389
Author: Ilya Lantukh 
Date:   2017-11-18T13:33:01Z

Fixed baseline topology changes from client/daemon nodes.

commit e3bbecd9f133251818a4b43afa44f46e66dd0325
Author: Alexey Goncharuk 
Date:   2017-11-18T14:16:39Z

Fixed licenses

commit b0d73fe45a8bb89ef82fce561f702095241c0405
Author: Alexey Go

Fwd: Request for contributors permissions

[Denis Magda]

Make sure you are subscribed to both dev and user to receive the email:
https://ignite.apache.org/community/resources.html#mail-lists


> Begin forwarded message:
> 
> From: Denis Magda 
> Subject: Re: Request for contributors permissions
> Date: November 23, 2017 at 7:33:21 AM PST
> To: dev@ignite.apache.org
> Reply-To: dev@ignite.apache.org
> 
> Hell Alexander,
> 
> Welcome, added you to the JIRA contributors list.
> 
> Get familiar with Ignite development process described here:
> https://cwiki.apache.org/confluence/display/IGNITE/Development+Process
> 
> Instructions on how to contribute can be found here:
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
> 
> Project setup in Intellij IDEA:
> https://cwiki.apache.org/confluence/display/IGNITE/Project+Setup
> 
> Once you got familiar and were able to run a few examples, you should pick
> a Jira ticket you would like to start on. Send an email to the dev list 
> sharing your JIRA id, so we can add you as a contributor in Jira.
> 
> These are the easy tickets to start with:
> https://issues.apache.org/jira/browse/IGNITE-4549?jql=project%20%3D%20IGNITE%20AND%20labels%20in%20(newbie)%20and%20status%20%3D%20OPEN
> 
> While those are more advanced but appealing:
> https://ignite.apache.org/community/contribute.html#pick-ticket
> 
> Looking forward to your contributions!
> Denis
> 
>> On Nov 23, 2017, at 2:34 AM, Alexander Kazantsev  
>> wrote:
>> 
>> Hello!
>> I want to fix one of ignite issue, but have not access to take issue.
>> Could you give me access or tell how to do it?
>> My nick is heruv1m in jira and email is a.u.kazant...@gmail.com
> 

Re: hi

[Denis Magda]

Hello and welcome!

Please get familiar with Ignite development process described here:
https://cwiki.apache.org/confluence/display/IGNITE/Development+Process

Subscribe to both dev and user lists:
https://ignite.apache.org/community/resources.html#mail-lists 


Instructions on how to contribute can be found here:
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute

Project setup in Intellij IDEA:
https://cwiki.apache.org/confluence/display/IGNITE/Project+Setup

Once you got familiar and were able to run a few examples, you should pick
a Jira ticket you would like to start on. Send an email to the dev list sharing 
your JIRA id, so we can add you as a contributor in Jira.

These are the easy tickets to start with:
https://issues.apache.org/jira/browse/IGNITE-4549?jql=project%20%3D%20IGNITE%20AND%20labels%20in%20(newbie)%20and%20status%20%3D%20OPEN

While those are more advanced but appealing:
https://ignite.apache.org/community/contribute.html#pick-ticket

Looking forward to your contributions!
Denis

> On Dec 28, 2017, at 10:17 PM, uttam tiwary  wrote:
> 
> Hello all
> 
> I would like to join the dev group and contribute on ignite project.

Re: Annoying extra steps for enabling metrics

[Denis Magda]

Alexey,

I think we should enable memoryEventStorageSPI automatically whenever 
statisticsEnabled is toggled on. A special message has to be added to the log 
pointing out that the automatic activation happened. 

Does it sound like a good solution?

—
Denis

> On Dec 29, 2017, at 3:51 AM, Alexey Plekhanov  wrote:
> 
> Denis, I start working on the issue
> https://issues.apache.org/jira/browse/IGNITE-6925 and now I don't understand
> why these properties must be bound to each other?
> 
> • If we enable statistics on caches, this does not necessarily mean,  that
> we wanted to get some events, we can enable statistics for other reasons.
> Conversely, not all events need statistics to be enabled on caches. So we
> can’t bind statistics flag to events (subscribe to events when statistics is
> enabled or enable statistics, when subscribing to events)
> • We can’t set events of interest, when we set not a dummy EventsStorageSpi,
> because we don’t know which events are interesting.
> • When we set events of interest, it’s not necessary, that these events will
> be monitored using EventsStorageSpi, we can also subscribe to events by
> events listeners, in this case EventsStorageSpi don’t used.
> 
> So, there is no general rule (if ... enabled, then ... must be enabled too).
> The only implementation I can propose - is "set MemoryEventStorageSPI
> instead of NoopEventStorageSPI when includeEventTypes list is not empty",
> but even this implementation may be warranted only in some cases.
> 
> 
> Denis Magda-2 wrote
>> Let’s simplifying the metrics as a part of this ticket:
>> https://issues.apache.org/jira/browse/IGNITE-5796
>> ;
>> 
>> Expanded its scope.
>> 
>> —
>> Denis
>> 
>>> On Sep 9, 2017, at 2:44 PM, Valentin Kulichenko <
> 
>> valentin.kulichenko@
> 
>> > wrote:
>>> 
>>> statisticsEnabled property comes from JCache, BTW.
>>> 
>>> -Val
>>> 
>>> On Sat, Sep 9, 2017 at 11:09 AM, Dmitriy Setrakyan <
> 
>> dsetrakyan@
> 
>> >
>>> wrote:
>>> 
 On Sat, Sep 9, 2017 at 8:56 AM, Denis Magda <
> 
>> dmagda@
> 
>> > wrote:
 
> Surprise!
> 
> If you want to see cache events then you have to enable one more flag!
> 
> 
>> 
 
 
 What is the overhead of this statistics collection?
 
 
> Three flags/beans have to be in the config in total, three! Just to see
> cache events. The API is a mess. Let’s contemplate how to fix it.
 
 
 Agree, this is horrible. We need to fix it in 2.3. Is there a ticket?
> 
> 
> 
> 
> 
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[GitHub] ignite pull request #3314: IGNITE-7214 performance measurement for FCM and K...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3314


---

[jira] [Created] (IGNITE-7344) Fix yardstick devnotes

[Oleg Ostanin (JIRA)]

Oleg Ostanin created IGNITE-7344:


 Summary: Fix yardstick devnotes
 Key: IGNITE-7344
 URL: https://issues.apache.org/jira/browse/IGNITE-7344
 Project: Ignite
  Issue Type: Improvement
Reporter: Oleg Ostanin
Assignee: Oleg Ostanin
Priority: Minor


mvn clean package -Pyardstick -pl modules/yardstick -am -DskipTests

does not build Yardstick module any more

mvn clean package -Pall-java,all-other -pl modules/yardstick -am -DskipTests

does.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Request for contributors permissions

[Alexander Kazantsev]

up


2017-11-23 13:34 GMT+03:00 Alexander Kazantsev :

>
> Hello!
> I want to fix one of ignite issue, but have not access to take issue.
> Could you give me access or tell how to do it?
> My nick is heruv1m in jira and email is a.u.kazant...@gmail.com
>

[jira] [Created] (IGNITE-7343) Add guard to prevent concurrent usage the JDBC thin connection

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-7343:


 Summary: Add guard to prevent concurrent usage the JDBC thin 
connection
 Key: IGNITE-7343
 URL: https://issues.apache.org/jira/browse/IGNITE-7343
 Project: Ignite
  Issue Type: Task
  Components: jdbc
Affects Versions: 2.3
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.4


I propose to add multi-thread guard into {{JdbcThinTcpIo#sendRequest}} method 
to diagnostic concurrency usage of the Ignite JDBC connection. 

The related ticket: IGNITE-7335



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

hi

[uttam tiwary]

Hello all

I would like to join the dev group and contribute on ignite project.

[GitHub] ignite pull request #3322: IGNITE-7335: add multi-tread guard

[tledkov-gridgain]

GitHub user tledkov-gridgain opened a pull request:

https://github.com/apache/ignite/pull/3322

IGNITE-7335: add multi-tread guard



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7335

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3322.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3322


commit 2fe1ca63a08ef3c12f239abfb5549b5a57096447
Author: tledkov-gridgain 
Date:   2017-12-29T12:43:26Z

IGNITE-7335: add multi-tread guard




---

[jira] [Created] (IGNITE-7342) SQL TX: Fix checking whether currently updating row is updated after it pass query filter

[Igor Seliverstov (JIRA)]

Igor Seliverstov created IGNITE-7342:


 Summary: SQL TX: Fix checking whether currently updating row is 
updated after it pass query filter
 Key: IGNITE-7342
 URL: https://issues.apache.org/jira/browse/IGNITE-7342
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Igor Seliverstov
Assignee: Igor Seliverstov
 Fix For: 2.5






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #3241: IGNITE-7097 performance measurement for SparseDis...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3241


---

Re: Annoying extra steps for enabling metrics

[Alexey Plekhanov]

Denis, I start working on the issue
https://issues.apache.org/jira/browse/IGNITE-6925 and now I don't understand
why these properties must be bound to each other?

• If we enable statistics on caches, this does not necessarily mean,  that
we wanted to get some events, we can enable statistics for other reasons.
Conversely, not all events need statistics to be enabled on caches. So we
can’t bind statistics flag to events (subscribe to events when statistics is
enabled or enable statistics, when subscribing to events)
• We can’t set events of interest, when we set not a dummy EventsStorageSpi,
because we don’t know which events are interesting.
• When we set events of interest, it’s not necessary, that these events will
be monitored using EventsStorageSpi, we can also subscribe to events by
events listeners, in this case EventsStorageSpi don’t used.

So, there is no general rule (if ... enabled, then ... must be enabled too).
The only implementation I can propose - is "set MemoryEventStorageSPI
instead of NoopEventStorageSPI when includeEventTypes list is not empty",
but even this implementation may be warranted only in some cases.


Denis Magda-2 wrote
> Let’s simplifying the metrics as a part of this ticket:
> https://issues.apache.org/jira/browse/IGNITE-5796
> ;
> 
> Expanded its scope.
> 
> —
> Denis
> 
>> On Sep 9, 2017, at 2:44 PM, Valentin Kulichenko <

> valentin.kulichenko@

> > wrote:
>> 
>> statisticsEnabled property comes from JCache, BTW.
>> 
>> -Val
>> 
>> On Sat, Sep 9, 2017 at 11:09 AM, Dmitriy Setrakyan <

> dsetrakyan@

> >
>> wrote:
>> 
>>> On Sat, Sep 9, 2017 at 8:56 AM, Denis Magda <

> dmagda@

> > wrote:
>>> 
 Surprise!
 
 If you want to see cache events then you have to enable one more flag!
 
 
> 
>>> 
>>> 
>>> What is the overhead of this statistics collection?
>>> 
>>> 
 Three flags/beans have to be in the config in total, three! Just to see
 cache events. The API is a mess. Let’s contemplate how to fix it.
>>> 
>>> 
>>> Agree, this is horrible. We need to fix it in 2.3. Is there a ticket?





--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[jira] [Created] (IGNITE-7341) Inconsistent data in case of error during rebalance

[Semen Boikov (JIRA)]

Semen Boikov created IGNITE-7341:


 Summary: Inconsistent data in case of error during rebalance 
 Key: IGNITE-7341
 URL: https://issues.apache.org/jira/browse/IGNITE-7341
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 2.0
Reporter: Semen Boikov


Added test reproducing artificial error during rebalance: 
IgniteErrorOnRebalanceTest.
Test scenario:
- start node1, put some data in replicated cache
- start node2, it fails to rebalance entry, stops rebalance and leave partition 
in MOVING state
- stop node1, since there are no others partition owners then node2 becomes 
owner
- start node1, after this node1 and node2 has different data




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #3321: Ignite-2.4-java9

[agura]

GitHub user agura opened a pull request:

https://github.com/apache/ignite/pull/3321

Ignite-2.4-java9



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/agura/incubator-ignite ignite-2.4-java9

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3321.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3321


commit 7b66609de9da4705926fb0a7845d69240eb0dee2
Author: Ivanov Petr 
Date:   2017-12-15T14:35:35Z

IGNITE-7203 Java 8 by default
 * got rid of java-1.8 related profiles
 * remove ml profile and included ml build into main reactor
 * added master properties to maven-compiler-plugin for whole project to be 
compiled only by java-1.8
 * refactored met properties + gathered them in parent module

commit b20f6ab8cb22528d7717229c1a7acc586cda44bc
Author: Ivanov Petr 
Date:   2017-12-18T07:25:19Z

IGNITE-7203 Java 8 by default
IGNITE-7216 Refactor examples project
 * refactored examples module and ml part of other modules: merged java8 
and java packages

commit 3177f28d5651f4ad910e6d54d5e8a225b9664c6b
Author: Ivanov Petr 
Date:   2017-12-18T15:19:02Z

IGNITE-7203 Java 8 by default
IGNITE-7216 Refactor examples project
 * fixed notes from review

commit ab8d197d8ed3c430e7720750db6c00aaf845b249
Author: Pavel Tupitsyn 
Date:   2017-12-26T09:11:41Z

.NET: Fix ExecutableTest

commit f4b0bdc72ba3f62f9125843680ff3ae78098deed
Author: Pavel Tupitsyn 
Date:   2017-12-26T09:26:23Z

.NET: Fix TestStopFromJava

commit b361ac4f6edfaa7d0645a9352145b84bd02745b1
Author: devozerov 
Date:   2017-12-26T11:31:07Z

Attempty to fix AOP failure.

commit 1390677806c9a28aba6720566260b31cdb58b086
Author: Ivanov Petr 
Date:   2017-12-26T12:40:15Z

IGNITE-7203 Java 8 by default
 * replaced aspectj dependency with newest one for attempt to fix AOP tests

commit aa2546aa55a9c22253206add2839917faef47027
Author: Pavel Tupitsyn 
Date:   2017-12-26T14:20:44Z

.NET: Another attempt to fix WindowsServiceTest

commit 017f63b4cb36a62234b8c406b2ebe085549d5628
Author: Pavel Tupitsyn 
Date:   2017-12-27T07:22:22Z

.NET: Another attempt to fix WindowsServiceTest

commit 83b9d21edc71c8f263a093c19be60042dfca36c1
Author: Pavel Tupitsyn 
Date:   2017-12-27T08:24:29Z

.NET: Another attempt to fix WindowsServiceTest

commit a93f2625c08b4c116c5a5df21c0e277f3821ad5f
Author: Pavel Tupitsyn 
Date:   2017-12-27T09:28:30Z

.NET: Another attempt to fix WindowsServiceTest, adding logger, no idea 
what the problem is

commit 2cf50943fb7451f7d5916fc824716b07c911093b
Author: Pavel Tupitsyn 
Date:   2017-12-27T11:35:10Z

.NET: Fix WindowsServiceTest - pass JVM path to the service

commit da9e76a5f093f745c51a0c55b6f8fa13dfc63ae9
Author: Ivanov Petr 
Date:   2017-12-27T15:20:40Z

IGNITE-7203 Java 8 by default
 * further improved AOP test behaviour by moving javaagent call into pom.xml

commit 602830163960b64e370f55b7a1269bb2fab513ba
Author: Ivanov Petr 
Date:   2017-12-28T13:03:40Z

Merge remote-tracking branch 'ignite/master' into ignite-7203

commit a133d5af3618ee5151804766bb1186488855c6e2
Author: Ivanov Petr 
Date:   2017-12-25T13:59:39Z

IGNITE-7144 Java 9: fix build issue with tools.jar not found
 * added profiles to parent and ignite-tools modules to detect java-9 
environment and run parametrised build
 * added macOS compatibility for previous java versions

commit fe35810a6befea9dc7ca12b4496d1d76af442fd7
Author: Ivanov Petr 
Date:   2017-12-15T14:35:35Z

IGNITE-7203 Java 8 by default
 * got rid of java-1.8 related profiles
 * remove ml profile and included ml build into main reactor
 * added master properties to maven-compiler-plugin for whole project to be 
compiled only by java-1.8
 * refactored met properties + gathered them in parent module

commit 0b6ac2147f7d861d5bd525bf863a0d849ea027e0
Author: Ivanov Petr 
Date:   2017-12-18T07:25:19Z

IGNITE-7203 Java 8 by default
IGNITE-7216 Refactor examples project
 * refactored examples module and ml part of other modules: merged java8 
and java packages

commit d678e0fdd5d6867146b1286916890d4ef40f2c6c
Author: Ivanov Petr 
Date:   2017-12-18T15:19:02Z

IGNITE-7203 Java 8 by default
IGNITE-7216 Refactor examples project
 * fixed notes from review

commit 93cc847a2c2501cc477a4c2ade458936ab4ed006
Author: Pavel Tupitsyn 
Date:   2017-12-26T09:11:41Z

.NET: Fix ExecutableTest

commit c9929a89642ded566680b4c1039d9a98cf86a285
Author: Pavel Tupitsyn 
Date:   2017-12-26T09:26:23Z

.NET: Fix TestStopFromJava

commit e0f1922b2b3e3c313b083971352faf76af43acb5
Author: devozerov 
Date:   2017-12-26T11:31:07Z

Attempty to fix AOP failure.

commit e34b1f9ed0ac05a4368fb131ab8219ed8a099ddd
Author: Ivanov Petr 
Date:   2017-12-26T12:40:15Z

IGNIT

RE: Handling slashes in cache names

[Stanislav Lukyanov]

> I would surround such replacements with "_", e.g. "myCacheName_somesymbol_".
Looks nice, will do.

> Here I am confused. I think the cache names should be case insensitive at
> all times. I seriously doubt enforcing this rule would cause problems. If
> we enforce this rule at cache creation time, then we would not have to add
> a hashcode at the end.
I think I would still keep the hashcode. E.g. I’m now also truncating names 
longer than 255 chars, and the truncated names could be equal. There could be 
more edge cases, and adding an imprint of the identity might help to avoid 
them. The names are readable enough with the hashes, but scary enough for users 
not to mess with them manually – I guess that’s a good thing :)
Making cache names always case-insensitive sounds good, but I’d separate it to 
another JIRA issue (it has larger compatibility impact, it affects a different 
part of the code base, etc). Is it OK?

Thanks,
Stan

From: Dmitriy Setrakyan
Sent: 28 декабря 2017 г. 22:33
To: dev@ignite.apache.org
Subject: Re: Handling slashes in cache names

On Thu, Dec 28, 2017 at 9:22 AM, Stanislav Lukyanov 
wrote:

> Hi all ,
>
> I’ve implemented an approach of encoding unsafe characters in the cache
> names for persistent storage directories. You can find it at
> https://github.com/gridgain/apache-ignite/tree/ignite-7264.
> How it works now is: 1) all characters outside of the [a-zA-Z0-9_-] class
> are replaced with their hex value (seems to be the easiest way);


I would surround such replacements with "_", e.g. "myCacheName_somesymbol_".


> 2) a hash of the cache name is added at the end of the name to avoid
> case-insensitive collisions.
> There is still a tiny chance of hitting two cache names that are equal
> ignoring case which also have the same hash, but that’s really unlikely.
>

Here I am confused. I think the cache names should be case insensitive at
all times. I seriously doubt enforcing this rule would cause problems. If
we enforce this rule at cache creation time, then we would not have to add
a hashcode at the end.


>
> It seems that there are no complications with this approach.
> The cache name to directory mapping is like
>   mycache -> cache-mycache-f19fd83d
>   my/cool/cache -> cache-my2fcool2fcache
>

As mentioned above, I would prefer "cache-my_2f_cool_2f_cache"


>   my!@#$%^&()cache -> cache-my21402324255e262829cache-84ba3e99
>
> Turns out the persistence is not the only place that doesn’t like special
> symbols in cache names – I also got an exception from MBean registration
> when creating a cache with ‘*’ or ‘?’. Filed https://issues.apache.org/
> jira/browse/IGNITE-7334 for that.
>
> Please let me know if you have any comments.
>
> Thanks,
> Stan
>
> From: Stanislav Lukyanov
> Sent: 25 декабря 2017 г. 18:09
> To: dev@ignite.apache.org
> Subject: Handling slashes in cache names
>
> Hi all,
>
> I’m looking into https://issues.apache.org/jira/browse/IGNITE-7264, and I
> need some guidance on what’s the best way to approach it.
>
> The problem is that cache names are not restricted, but if persistence is
> enabled the cache needs to have a corresponding directory on the file
> system (“cache-…”) which can’t be created if the cache name contains
> certain characters (or a reserved system name).
>
> A straightforward approach would be to check if a cache name is allowed on
> the local system (e.g. via `Paths.get(name)`) and fail to create cache if
> it isn’t, but I’m a bit concerned with the consistency of the behavior (the
> same cache name be allowed on one system and not on another).
> I think a better way would be to replace special characters (say, all
> non-alphanumeric characters) with underscores in file names (not changing
> the cache configuration). Would this be OK? Are there any risks I’m not
> considering?
>
> WDYT?
>
> Thanks,
> Stan
>
>

[GitHub] ignite pull request #3320: Gg 13139

[slukyano]

GitHub user slukyano opened a pull request:

https://github.com/apache/ignite/pull/3320

Gg 13139



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite gg-13139

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3320.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3320


commit 3668b91817c1da7958cc3d7c4dddf890a2237772
Author: Andrey V. Mashenkov 
Date:   2017-06-27T07:34:42Z

Partially reverted GG-12352.

commit 172f41c489c0ca5b7613163cd325b0c01d5b28b1
Author: sboikov 
Date:   2017-06-27T09:55:11Z

gg-12133 Deadlock for metadata update from GridCacheMapEntry.innerUpdate

(cherry picked from commit 897f4c0)

commit 0521b7780756788d92bfa35ef00f56b5bb01367d
Author: Andrey V. Mashenkov 
Date:   2017-06-27T17:43:07Z

GG-12370: Fixed massive NODE_FAILED events lead to excessive momentary 
memory consumption.

commit 6abe5bf5bd732bf9f79df577e159243520dd5c0b
Author: Andrey V. Mashenkov 
Date:   2017-06-28T06:47:45Z

Fixed compilation.

commit 6b54f991c13efb3630527a3010c4f90105a4fc76
Author: agura 
Date:   2017-06-21T16:09:37Z

ignite-5574 For offheap_tiered cache first try read value from offheap in 
order to avoid redundant synchronization on entry

commit 7d5217260b293b7224340349b5e44792999600f3
Author: Andrey V. Mashenkov 
Date:   2017-06-28T08:47:21Z

Rethrow handshake exceptions as it done for ConnectionException.

commit 81cdea40743131cac9dae49150c1038073595f7e
Author: Andrey V. Mashenkov 
Date:   2017-06-28T10:00:30Z

Fixed tests.

commit 473abcafca568c7c898b0b1ae91fe964084fdf43
Author: agura 
Date:   2017-06-28T11:49:49Z

Tests fixed

commit 4f383ae772631987c3f4ac29bb654b4811fbc407
Author: agura 
Date:   2017-06-28T11:49:49Z

Tests fixed.

commit 4b11bb27d5caeb7ba7bc2929685d7c548f4c327d
Author: Andrey V. Mashenkov 
Date:   2017-06-28T12:45:42Z

Tests fixed.

commit 9b92dac5756619ed5218db269acdae1ef02ef8ae
Author: Andrey V. Mashenkov 
Date:   2017-06-28T15:20:02Z

Tests fixed.

commit 0295a1a7ecb2ef57a917ddc9015ff8b71a6ddb14
Author: mcherkasov 
Date:   2017-06-23T18:00:45Z

IGNITE-5521: Large near caches lead to cluster instability with metrics 
enabled.

commit 16fed552fc8f91de550207eeebd5850e685960a8
Author: AMRepo 
Date:   2017-06-29T07:49:34Z

Fix tests.

commit 65d93e48c264f4bfff0a94856fdfeb83375a3976
Author: mcherkasov 
Date:   2017-06-23T18:00:45Z

IGNITE-5521: Large near caches lead to cluster instability with metrics 
enabled

commit ff1afad1278e939aef71b274e959fd4256904971
Author: Andrey V. Mashenkov 
Date:   2017-06-27T07:34:42Z

Partially reverted GG-12352.

(cherry picked from commit 3668b91)

commit 7fbb95d1698c534957c9f0b7c5558f43985cfe68
Author: Andrey V. Mashenkov 
Date:   2017-06-27T17:43:07Z

GG-12370: Fixed massive NODE_FAILED events lead to excessive momentary 
memory consumption.

(cherry picked from commit 0521b77)

commit 710db327c027d43c0984b7007447cecca71f3cca
Author: Andrey V. Mashenkov 
Date:   2017-06-28T08:47:21Z

Rethrow handshake exceptions as it done for ConnectionException.

(cherry picked from commit 7d52172)

commit 7c619f77009cc6876405839f9f6201d332d94b47
Author: Andrey V. Mashenkov 
Date:   2017-06-28T12:45:42Z

Fixed tests.

(cherry picked from commit 81cdea4)

commit a3717b0cc6d716e3bc966248854334e74b0614e2
Author: Andrey V. Mashenkov 
Date:   2017-06-29T10:06:06Z

Merge remote-tracking branch 'origin/ignite-1.7.12' into ignite-1.7.12

commit 7d41f5da867294befd6cca1de3c945bfd83ae597
Author: Andrey V. Mashenkov 
Date:   2017-06-29T10:15:57Z

Merge branch 'ignite-1.8.8' into ignite-1.9.4

commit 5219a35208b95a2d7dea3825dd4cb4edc74b3542
Author: Slava Koptilin 
Date:   2017-06-26T12:37:26Z

IGNITE-5076: improved multi-threaded start of nodes

commit aba9bf9982c6eba8cb57d64e6b1090914b8d020e
Author: Andrey V. Mashenkov 
Date:   2017-06-30T08:30:20Z

Merge branch 'ignite-1.8.8' into ignite-1.9.4

commit a4fc555b118d91ec0348154b88764f010dbbae52
Author: nikolay_tikhonov 
Date:   2017-06-30T11:38:54Z

Fixed "IGNITE-5424 GridServiceProxy does not unwraps exception message from 
InvocationTargetException." This closes #2168

Signed-off-by: nikolay_tikhonov 

commit 62876fefc8dbd334db9dc6741c2bc91b3409343d
Author: Andrey V. Mashenkov 
Date:   2017-06-30T11:45:18Z

IGNITE-5473 partial fix: Create ignite troubleshooting logger.

commit 75c442a8309d554f6894f30f63ed0b964e7a0aa1
Author: Slava Koptilin 
Date:   2017-06-26T12:37:26Z

Backported IGNITE-5076: improved multi-threaded start of nodes.

commit 551923e1e5139c484fcb716e7f10907fa71021d9
Author: nikolay_tikhonov 
Date:   2017-06-30T11:38:54Z

Backported "IGNITE-5424 GridServiceProxy does not unwraps exception message 
from Invocat

[GitHub] ignite pull request #3319: IGNITE-7340 - Fix flaky GridServiceProcessorMulti...

[dkarachentsev]

GitHub user dkarachentsev opened a pull request:

https://github.com/apache/ignite/pull/3319

IGNITE-7340 - Fix flaky GridServiceProcessorMultiNodeConfigSelfTest#c…

…heckDeployOnEachNodeUpdateTopology

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7340

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3319.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3319


commit ab5622f8c14998908ab36bd22ba6966e36b2e730
Author: dkarachentsev 
Date:   2017-12-29T09:26:02Z

IGNITE-7340 - Fix flaky 
GridServiceProcessorMultiNodeConfigSelfTest#checkDeployOnEachNodeUpdateTopology




---

[jira] [Created] (IGNITE-7340) Fix flaky org.apache.ignite.internal.processors.service.GridServiceProcessorMultiNodeConfigSelfTest#checkDeployOnEachNodeUpdateTopology

[Dmitry Karachentsev (JIRA)]

Dmitry Karachentsev created IGNITE-7340:
---

 Summary: Fix flaky 
org.apache.ignite.internal.processors.service.GridServiceProcessorMultiNodeConfigSelfTest#checkDeployOnEachNodeUpdateTopology
 Key: IGNITE-7340
 URL: https://issues.apache.org/jira/browse/IGNITE-7340
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.3
Reporter: Dmitry Karachentsev
Assignee: Dmitry Karachentsev






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-7339) RENTING partition is not evicted after restore from storage

[Semen Boikov (JIRA)]

Semen Boikov created IGNITE-7339:


 Summary: RENTING partition is not evicted after restore from 
storage
 Key: IGNITE-7339
 URL: https://issues.apache.org/jira/browse/IGNITE-7339
 Project: Ignite
  Issue Type: Bug
Reporter: Semen Boikov
Priority: Critical


If partition was in RENTING state at the moment when node is stopped, then 
after restart it is not evicted.

It seems it is an issue in GridDhtLocalPartition.rent, 'tryEvictAsync' is not 
called is partition was already in RENTING state.

Also there is error in GridDhtPartitionTopologyImpl.checkEvictions: partition 
state is always treated as changed after part.rent call even if part.rent does 
not actually change state.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)