[jira] [Created] (IGNITE-13470) .NET: Add async counterparts to all applicable thin client APIs

[Pavel Tupitsyn (Jira)]

Pavel Tupitsyn created IGNITE-13470:
---

 Summary: .NET: Add async counterparts to all applicable thin 
client APIs
 Key: IGNITE-13470
 URL: https://issues.apache.org/jira/browse/IGNITE-13470
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
 Fix For: 2.10


Add async counterparts to the following .NET Thin Client APIs
* ICacheClient.GetConfiguration
* IClientCluster - all methods
* IClientClusterGroup.GetNodes, GetNode
* IIgniteClient - CreateCache, GetOrCreateCache, DestroyCache



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[MTCGA]: new failures in builds [5607867] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 *Test with high flaky rate in master 
TcpDiscoverySpiFailureTimeoutSelfTest.testConnectionCheckMessage 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-8763687194630085769=%3Cdefault%3E=testDetails
 No changes in the build

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 23:21:51 21-09-2020 

[MTCGA]: new failures in builds [5615580] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 *New Critical Failure in master Cache 7 
https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Cache7?branch=%3Cdefault%3E
 No changes in the build

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 22:36:45 21-09-2020 

Re: [DISCUSSION] Renaming Ignite's product category

[Nikita Ivanov]

My vote is to just call ignite "IgniteDB". That's it. No other additional
explanation is required as no amount of additional verbiage will help.
Every DB is different: from MongoDB, to RedisDB, to CockroachDB, to Oracle
- they all look & act completely different, and they don't go around trying
to explain in one line what they do and how they are different.

"IgniteDB" is clear, concise and gives us the broadest initial acceptance
from the new user perspective.

Thanks,
--
Nikita Ivanov



On Sat, Sep 19, 2020 at 1:10 PM Saikat Maitra 
wrote:

> Hi,
>
> My thoughts are similar to as Denis and Val mentioned like Apache Ignite -
> "A Memory Centric Database".
>
> It aligns to current features of Apache Ignite as mentioned in the below
> post.
>
>
> https://thenewstack.io/memory-centric-architectures-whats-next-for-in-memory-computing
>
> Regards,
> Saikat
>
> On Fri, Sep 18, 2020 at 9:02 AM Carbone, Adam 
> wrote:
>
>> So when I came across Ignite It was described as an In Memory Data Grid
>>
>> So one way to look at this is who do you fashion as Ignite competing
>> against?
>>
>> Are competing against Redis, Aerospike - In Memory Databases
>>
>> Or are you more competing with
>>
>> Gigaspaces - True In memory Compute platform
>>
>> And then you have like of
>>
>> Hazelcast that started as a Distributed Hash and have gained some
>> features...
>>
>> On thing that I think is a differentiator that isn't being highlighted
>> but Is  unique feature to Ignited, and the primary reason we ended up here;
>> The integration with spark and it's distributed/shared Datasets/Dataframes.
>>
>> I don't know for me the In Memory Data Grid I think fits what Ignite
>> is...
>>
>> Regards
>>
>> ~Adam
>>
>> Adam Carbone | Director of Innovation – Intelligent Platform Team |
>> Bottomline Technologies
>> Office: 603-501-6446 | Mobile: 603-570-8418
>> www.bottomline.com
>>
>>
>>
>> On 9/17/20, 11:45 AM, "Glenn Wiebe"  wrote:
>>
>> I agree with Stephen about "database" devaluing what Ignite can do
>> (though
>> it probably hits the majority of existing use cases). I tend to go
>> with
>> "massively distributed storage and compute platform"
>>
>> I know, I didn't take sides, I just have both.
>>
>> Cheers,
>>   Glenn
>>
>> On Thu., Sep. 17, 2020, 7:04 a.m. Stephen Darlington, <
>> stephen.darling...@gridgain.com> wrote:
>>
>> > I think this is a great question. Explaining what Ignite does is
>> always a
>> > challenge, so having a useful “tag line” would be very valuable.
>> >
>> > I’m not sure what the answer is but I think calling it a “database”
>> > devalues all the compute facilities. "Computing platform” may be
>> too vague
>> > but it at least says that we do more than “just” store data.
>> >
>> > On 17 Sep 2020, at 06:29, Valentin Kulichenko <
>> > valentin.kuliche...@gmail.com> wrote:
>> >
>> > My vote is for the "distributed memory-first database". It clearly
>> states
>> > that Ignite is a database (which is true at this point), while still
>> > emphasizing the in-memory computing power endorsed by the platform.
>> >
>> > The "in-memory computing platform" is an ambiguous term and doesn't
>> really
>> > reflect what Ignite is, especially in its current state.
>> >
>> > -Val
>> >
>> > On Wed, Sep 16, 2020 at 3:53 PM Denis Magda 
>> wrote:
>> >
>> >> Igniters,
>> >>
>> >> Throughout the history of our project, we could see how the
>> addition of
>> >> certain features required us to reassess the project's name and
>> category.
>> >>
>> >> Before Ignite joined the ASF, it supported only compute APIs
>> resembling
>> >> the
>> >> MapReduce engine of Hadoop. Those days, it was fair to define
>> Ignite as "a
>> >> distributed in-memory computing engine". Next, at the time of the
>> project
>> >> donation, it already included key-value/SQL/transactional APIs,
>> was used
>> >> as
>> >> a distributed cache, and significantly outgrew the "in-memory
>> computing
>> >> engine" use case. That's how the project transitioned to the
>> product
>> >> category of in-memory caches and we started to name it as an
>> "in-memory
>> >> data grid" or "in-memory computing platform" to differentiate from
>> >> classical caching products such as Memcached and Redis.
>> >>
>> >> Nowadays, the project outgrew its caching use case, and the
>> classification
>> >> of Ignite as an "in-memory data grid" or "in-memory computing
>> platform"
>> >> doesn't sound accurate. We rebuilt our storage engine by replacing
>> a
>> >> typical key-value engine with a B-tree engine that spans across
>> memory and
>> >> disk tiers. And it's not surprising to see more deployments of
>> Ignite as a
>> >> database on its own. So, it feels like we need to reconsider Ignite
>> >> positioning again so that a) application developers can discover
>> it easily
>> 

[jira] [Created] (IGNITE-13469) Adaptation control.sh to picocli framework

[Kirill Tkalenko (Jira)]

Kirill Tkalenko created IGNITE-13469:


 Summary: Adaptation control.sh to picocli framework
 Key: IGNITE-13469
 URL: https://issues.apache.org/jira/browse/IGNITE-13469
 Project: Ignite
  Issue Type: Improvement
  Components: control.sh
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
 Fix For: 2.10


Adding a dependency [1] to module *ignite-control-utility* and adapting code to 
it.

[1] - https://picocli.info/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSSION] Maintenance Mode feature

[Ivan Pavlukhin]

Sergey,

> From  the code complexity perspective I'm trying to design the feature in 
> such a way that all maintenance code is as encapsulated as possible and 
> avoids massive interventions into main workflows of components.

Could please briefly tell what means do you use to achieve
encapsulation? Are Discovery, Communication, Checkpointer and other
components started in a maintenance mode in current design?

2020-09-21 15:19 GMT+03:00, Nikolay Izhikov :
> Hello, Sergey.
>
>> At the moment I'm aware about two use cases for this feature: corrupted
>> PDS cleanup and defragmentation.
>
> AFAIKU There is third use-case for this mode.
>
> Change encryption master key in case node was down during cluster master key
> change.
> In this case, node can’t join to the cluster, because it’s master key
> differs from the cluster.
> To recover node Ignite should locally change master key before join.
>
> Please, take a look into source code [1]
>
> [1]
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/managers/encryption/GridEncryptionManager.java#L710
>
>> 21 сент. 2020 г., в 14:37, Sergey Chugunov 
>> написал(а):
>>
>> Ivan,
>>
>> Sorry for some confusion, MM indeed is not a normal mode. What I was
>> trying
>> to say is that when in MM node still starts and allows the user to
>> perform
>> actions with it like sending commands via control utility/JMX APIs or
>> reading metrics.
>>
>> This is the key point: although the node is not in the cluster but it is
>> still alive can be monitored and supports management to do maintenance.
>>
>> From  the code complexity perspective I'm trying to design the feature in
>> such a way that all maintenance code is as encapsulated as possible and
>> avoids massive interventions into main workflows of components.
>> At the moment I'm aware about two use cases for this feature: corrupted
>> PDS
>> cleanup and defragmentation. As far as I know it won't bring too much
>> complexity in both cases.
>>
>> I cannot say for other components but I believe it will be possible to
>> integrate MM feature into their workflow as well with reasonable amount
>> of
>> refactoring.
>>
>> Does it make sense to you?
>>
>> On Sun, Sep 6, 2020 at 8:08 AM Ivan Pavlukhin 
>> wrote:
>>
>>> Sergey,
>>>
>>> Thank you for your answer!
>>>
>>> Might be I am looking at the subject from a different angle.
>>>
 I think of a node in MM as an almost normal one
>>> I cannot think of such a mode as a normal one, because it apparently
>>> does not perform usual cluster node functions. It is not a part of a
>>> cluster, caches data is not available, Discovery and Communication are
>>> not needed.
>>>
>>> I fear that with "node started in a special mode" approach we will get
>>> an additional flag in the code making the code more complex and
>>> fragile. Should not I worry about it?
>>>
>>> 2020-09-02 10:45 GMT+03:00, Sergey Chugunov :
 Vladislav, Ivan,

 Thank you for your questions and suggestions. Let me answer them.

 Vladislav,

 If I understood you correctly, you're talking about a node performing
>>> some
 automatic actions to fix the problem and then join the cluster as
 usual.

 However the original ticket [1] where we faced the need for Maintenance
 Mode is about exactly the opposite: avoid doing automatic actions and
>>> give
 a user the ability to decide what to do.

 Also the idea of Maintenance Mode is that the node is able to accept
 commands, expose metrics and so on, thus we need all components to be
 initialized (some of them may be partially initialized due to their own
 maintenance).
 To achieve that we need to go through a full cycle of node
 initialization
 including discovery initialization. When discovery is initialized (in
 special isolated mode) I don't think it is easy to switch back to
 normal
 operations without a restart.

 Ivan,

 I think of a node in MM as an almost normal one (maybe with some
>>> components
 skipped some steps of their initialization). Commands are accepted,
 appropriate metrics are exposed e.g. through JMX API and so on.

 So as I see it we'll have special commands for control.{sh|bat} CLI
 allowing user to see reasons why node switched to maintenance mode
 and/or
 trigger actions to fix the problem (I'm still thinking about proper
>>> design
 of these actions though).

 Of course the user should also be able to fix the problem manually e.g.
>>> by
 manually deleting corrupted PDS files when node is down. Ideally
 Maintenance Mode should be smart enough to figure that out and switch
 to
 normal operations without a restart but I'm not sure if it is possible
 without invasive changes of our components' lifecycle.
 So I believe this model (node truly started in Maintenance Mode and new
 commands in control.{sh|bat}) is a good fit for our 

Re: IEP-51: Java Thin Client Async API

[mnk]

Pavel Tupitsyn wrote
>> result of a remote execution is a CompletionStage
> 
> Can you give an example? What is a remote execution? Is this about Compute
> and/or Services?

This is about Compute.

Let's say I'm doing affinityCallAsync for IgniteCallable where R
implements CompletableStage or R is a CompletableFuture.  Then I
wouldn't want to have the CompleteableStage or CompleteableFuture come back
to me actually, rather something that will complete with the T when it's
ready (and it would be nice if I could also cancel it for the
CompleteableFuture case). It could be done as well for affinityCall
(non-async) too, although in my mind the former case is the more important
one. Of course, affinityCall is just an example, it should apply to the
other compute methods as appropriate.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[jira] [Created] (IGNITE-13468) Destroying a caches doesn't clean up cache directory

[Stephen Darlington (Jira)]

Stephen Darlington created IGNITE-13468:
---

 Summary: Destroying a caches doesn't clean up cache directory
 Key: IGNITE-13468
 URL: https://issues.apache.org/jira/browse/IGNITE-13468
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Affects Versions: 2.8.1
Reporter: Stephen Darlington


If I create a cache – either with SQL or with Ignite#getOrCreateCache() – in a 
data region with persistence enabled, destroying the cache removes all the data 
files but not the directory in which they're stored.

For example. Before:

{{$ ls cache*}}
{{ cache-DESTROY_DEMO:}}
{{ cache_data.dat index.bin part-2.bin part-3.bin part-4.bin}}

 

{{cache-ignite-sys-cache:}}
{{ cache_data.dat index.bin}}
{{ $}}

After:

{{$ ls cache*}}
{{ cache-DESTROY_DEMO:}}

 

{{cache-ignite-sys-cache:}}
{{ cache_data.dat index.bin}}
{{ $}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: IEP-51: Java Thin Client Async API

[Pavel Tupitsyn]

> result of a remote execution is a CompletionStage

Can you give an example? What is a remote execution? Is this about Compute
and/or Services?


On Mon, Sep 21, 2020 at 5:11 PM mnk  wrote:

> So one question I have here (and this maybe out of scope) - but what about
> the cases where the result of a remote execution is a CompletionStage? In
> this case, I would want to have a local CompletionStage which "represents"
> the remote one. In other words when the remote one is completed, the value
> should be sent back to the local node for completion the local stage. This
> could even be "doubly asynchronous" - in other words, I executeAsync and
> the
> result of that is a CompletionStage. Note that this differs from what is
> supported today in that the result (the CompletionStage) is immediately
> serialized back to the local node, which of course isn't what's needed.
> More
> advanced would be the case where the result of the remote execution is a
> CompleteableFuture in which case I may want the ability to cancel it
> remotely as well.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Re: IEP-51: Java Thin Client Async API

[mnk]

So one question I have here (and this maybe out of scope) - but what about
the cases where the result of a remote execution is a CompletionStage? In
this case, I would want to have a local CompletionStage which "represents"
the remote one. In other words when the remote one is completed, the value
should be sent back to the local node for completion the local stage. This
could even be "doubly asynchronous" - in other words, I executeAsync and the
result of that is a CompletionStage. Note that this differs from what is
supported today in that the result (the CompletionStage) is immediately
serialized back to the local node, which of course isn't what's needed. More
advanced would be the case where the result of the remote execution is a
CompleteableFuture in which case I may want the ability to cancel it
remotely as well.




--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: IEP-51: Java Thin Client Async API

[mnk]

So one question I have here (and this maybe out of scope) - but what about
the cases where the result of a remote execution is a CompletionStage? In
this case, I would want to have a local CompletionStage which "represents"
the remote one. In other words when the remote one is completed, the value
should be sent back to the local node for completion the local stage. This
could even be "doubly asynchronous" - in other words, I executeAsync and the
result of that is a CompletionStage. Note that this differs from what is
supported today in that the result (the CompletionStage) is immediately
serialized back to the local node, which of course isn't what's needed. More
advanced would be the case where the result of the remote execution is a
CompleteableFuture in which case I may want the ability to cancel it
remotely as well.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[jira] [Created] (IGNITE-13467) Add events for snapshot operation

[Maxim Muzafarov (Jira)]

Maxim Muzafarov created IGNITE-13467:


 Summary: Add events for snapshot operation
 Key: IGNITE-13467
 URL: https://issues.apache.org/jira/browse/IGNITE-13467
 Project: Ignite
  Issue Type: Task
Reporter: Maxim Muzafarov
Assignee: Maxim Muzafarov
 Fix For: 2.10


The snapshot operation states must be exposed to the user by firing Ignite 
events:
- snapshot operation started
- snapshot operation finished



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSSION] Maintenance Mode feature

[Nikolay Izhikov]

Hello, Sergey.

> At the moment I'm aware about two use cases for this feature: corrupted PDS 
> cleanup and defragmentation. 

AFAIKU There is third use-case for this mode.

Change encryption master key in case node was down during cluster master key 
change.
In this case, node can’t join to the cluster, because it’s master key differs 
from the cluster.
To recover node Ignite should locally change master key before join.

Please, take a look into source code [1]

[1] 
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/managers/encryption/GridEncryptionManager.java#L710

> 21 сент. 2020 г., в 14:37, Sergey Chugunov  
> написал(а):
> 
> Ivan,
> 
> Sorry for some confusion, MM indeed is not a normal mode. What I was trying
> to say is that when in MM node still starts and allows the user to perform
> actions with it like sending commands via control utility/JMX APIs or
> reading metrics.
> 
> This is the key point: although the node is not in the cluster but it is
> still alive can be monitored and supports management to do maintenance.
> 
> From  the code complexity perspective I'm trying to design the feature in
> such a way that all maintenance code is as encapsulated as possible and
> avoids massive interventions into main workflows of components.
> At the moment I'm aware about two use cases for this feature: corrupted PDS
> cleanup and defragmentation. As far as I know it won't bring too much
> complexity in both cases.
> 
> I cannot say for other components but I believe it will be possible to
> integrate MM feature into their workflow as well with reasonable amount of
> refactoring.
> 
> Does it make sense to you?
> 
> On Sun, Sep 6, 2020 at 8:08 AM Ivan Pavlukhin  wrote:
> 
>> Sergey,
>> 
>> Thank you for your answer!
>> 
>> Might be I am looking at the subject from a different angle.
>> 
>>> I think of a node in MM as an almost normal one
>> I cannot think of such a mode as a normal one, because it apparently
>> does not perform usual cluster node functions. It is not a part of a
>> cluster, caches data is not available, Discovery and Communication are
>> not needed.
>> 
>> I fear that with "node started in a special mode" approach we will get
>> an additional flag in the code making the code more complex and
>> fragile. Should not I worry about it?
>> 
>> 2020-09-02 10:45 GMT+03:00, Sergey Chugunov :
>>> Vladislav, Ivan,
>>> 
>>> Thank you for your questions and suggestions. Let me answer them.
>>> 
>>> Vladislav,
>>> 
>>> If I understood you correctly, you're talking about a node performing
>> some
>>> automatic actions to fix the problem and then join the cluster as usual.
>>> 
>>> However the original ticket [1] where we faced the need for Maintenance
>>> Mode is about exactly the opposite: avoid doing automatic actions and
>> give
>>> a user the ability to decide what to do.
>>> 
>>> Also the idea of Maintenance Mode is that the node is able to accept
>>> commands, expose metrics and so on, thus we need all components to be
>>> initialized (some of them may be partially initialized due to their own
>>> maintenance).
>>> To achieve that we need to go through a full cycle of node initialization
>>> including discovery initialization. When discovery is initialized (in
>>> special isolated mode) I don't think it is easy to switch back to normal
>>> operations without a restart.
>>> 
>>> Ivan,
>>> 
>>> I think of a node in MM as an almost normal one (maybe with some
>> components
>>> skipped some steps of their initialization). Commands are accepted,
>>> appropriate metrics are exposed e.g. through JMX API and so on.
>>> 
>>> So as I see it we'll have special commands for control.{sh|bat} CLI
>>> allowing user to see reasons why node switched to maintenance mode and/or
>>> trigger actions to fix the problem (I'm still thinking about proper
>> design
>>> of these actions though).
>>> 
>>> Of course the user should also be able to fix the problem manually e.g.
>> by
>>> manually deleting corrupted PDS files when node is down. Ideally
>>> Maintenance Mode should be smart enough to figure that out and switch to
>>> normal operations without a restart but I'm not sure if it is possible
>>> without invasive changes of our components' lifecycle.
>>> So I believe this model (node truly started in Maintenance Mode and new
>>> commands in control.{sh|bat}) is a good fit for our current APIs and ways
>>> to interact with the node.
>>> 
>>> Does it sound reasonable to you?
>>> 
>>> Thank you!
>>> 
>>> [1] https://issues.apache.org/jira/browse/IGNITE-13366
>>> 
>>> On Tue, Sep 1, 2020 at 2:07 PM Ivan Pavlukhin 
>> wrote:
>>> 
 Sergey,
 
 Actually, I missed the point that the discussed mode affects a single
 node but not a whole cluster. Perhaps I mixed terms "mode" and
 "state".
 
 My next thoughts about maintenance routines are about special
 utilities. As far as I remember MySQL provides a bunch of scripts for
 various 

[jira] [Created] (IGNITE-13465) Ignite cluster falls apart if two nodes segmented sequentially

[Aleksey Plekhanov (Jira)]

Aleksey Plekhanov created IGNITE-13465:
--

 Summary: Ignite cluster falls apart if two nodes segmented 
sequentially
 Key: IGNITE-13465
 URL: https://issues.apache.org/jira/browse/IGNITE-13465
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.9
Reporter: Aleksey Plekhanov
 Fix For: 2.9


After ticket IGNITE-13134 sequential nodes segmentation leads to segmentation 
of other nodes in the cluster.

Reproducer attached.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSSION] Maintenance Mode feature

[Sergey Chugunov]

Ivan,

Sorry for some confusion, MM indeed is not a normal mode. What I was trying
to say is that when in MM node still starts and allows the user to perform
actions with it like sending commands via control utility/JMX APIs or
reading metrics.

This is the key point: although the node is not in the cluster but it is
still alive can be monitored and supports management to do maintenance.

>From  the code complexity perspective I'm trying to design the feature in
such a way that all maintenance code is as encapsulated as possible and
avoids massive interventions into main workflows of components.
At the moment I'm aware about two use cases for this feature: corrupted PDS
cleanup and defragmentation. As far as I know it won't bring too much
complexity in both cases.

I cannot say for other components but I believe it will be possible to
integrate MM feature into their workflow as well with reasonable amount of
refactoring.

Does it make sense to you?

On Sun, Sep 6, 2020 at 8:08 AM Ivan Pavlukhin  wrote:

> Sergey,
>
> Thank you for your answer!
>
> Might be I am looking at the subject from a different angle.
>
> > I think of a node in MM as an almost normal one
> I cannot think of such a mode as a normal one, because it apparently
> does not perform usual cluster node functions. It is not a part of a
> cluster, caches data is not available, Discovery and Communication are
> not needed.
>
> I fear that with "node started in a special mode" approach we will get
> an additional flag in the code making the code more complex and
> fragile. Should not I worry about it?
>
> 2020-09-02 10:45 GMT+03:00, Sergey Chugunov :
> > Vladislav, Ivan,
> >
> > Thank you for your questions and suggestions. Let me answer them.
> >
> > Vladislav,
> >
> > If I understood you correctly, you're talking about a node performing
> some
> > automatic actions to fix the problem and then join the cluster as usual.
> >
> > However the original ticket [1] where we faced the need for Maintenance
> > Mode is about exactly the opposite: avoid doing automatic actions and
> give
> > a user the ability to decide what to do.
> >
> > Also the idea of Maintenance Mode is that the node is able to accept
> > commands, expose metrics and so on, thus we need all components to be
> > initialized (some of them may be partially initialized due to their own
> > maintenance).
> > To achieve that we need to go through a full cycle of node initialization
> > including discovery initialization. When discovery is initialized (in
> > special isolated mode) I don't think it is easy to switch back to normal
> > operations without a restart.
> >
> > Ivan,
> >
> > I think of a node in MM as an almost normal one (maybe with some
> components
> > skipped some steps of their initialization). Commands are accepted,
> > appropriate metrics are exposed e.g. through JMX API and so on.
> >
> > So as I see it we'll have special commands for control.{sh|bat} CLI
> > allowing user to see reasons why node switched to maintenance mode and/or
> > trigger actions to fix the problem (I'm still thinking about proper
> design
> > of these actions though).
> >
> > Of course the user should also be able to fix the problem manually e.g.
> by
> > manually deleting corrupted PDS files when node is down. Ideally
> > Maintenance Mode should be smart enough to figure that out and switch to
> > normal operations without a restart but I'm not sure if it is possible
> > without invasive changes of our components' lifecycle.
> > So I believe this model (node truly started in Maintenance Mode and new
> > commands in control.{sh|bat}) is a good fit for our current APIs and ways
> > to interact with the node.
> >
> > Does it sound reasonable to you?
> >
> > Thank you!
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-13366
> >
> > On Tue, Sep 1, 2020 at 2:07 PM Ivan Pavlukhin 
> wrote:
> >
> >> Sergey,
> >>
> >> Actually, I missed the point that the discussed mode affects a single
> >> node but not a whole cluster. Perhaps I mixed terms "mode" and
> >> "state".
> >>
> >> My next thoughts about maintenance routines are about special
> >> utilities. As far as I remember MySQL provides a bunch of scripts for
> >> various maintenance purposes. What user interface for maintenance
> >> tasks execution is assumed? And what do we mean by "starting" a node
> >> in a maintenance mode? Can we do some routines without "starting"
> >> (e.g. try to recover PDS or cleanup)?
> >>
> >> 2020-08-31 23:41 GMT+03:00, Vladislav Pyatkov :
> >> > Hi Sergey.
> >> >
> >> > As I understand any switching from/to MM possible only through manual
> >> > restart a node.
> >> > But in your example that look like a technical actions, that only
> >> possible
> >> > in the case.
> >> > Do you plan to provide a possibility for client where he can make a
> >> > decision without a manual intervention?
> >> >
> >> > For example: Start node and manually agree with an option and after
> >> > automatically resolve conflict and back 

Re: critical security vulnerability for /opt/ignite/apache-ignite/libs/optional/ignite-rest-http/log4j-1.2.17.jar

[Stephen Darlington]

https://issues.apache.org/jira/browse/IGNITE-13464

> On 21 Sep 2020, at 11:02, Ilya Kasnacheev  wrote:
> 
> Hello!
> 
> Good catch! I think you should file a critical level ticket about it.
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> пн, 21 сент. 2020 г. в 12:56, Stephen Darlington 
> mailto:stephen.darling...@gridgain.com>>:
> Actually, this is an interesting one: it’s not the top level ignite-log4j 
> module, but a dependency of ignite-rest-http. Why does the REST API have 
> log4j (and slf4j) dependencies at all?
> 
>> On 21 Sep 2020, at 10:19, Ilya Kasnacheev > > wrote:
>> 
>> Hello!
>> 
>> Log4J 1.x does not have any non-vulnerable releases, and Log4J2 is not 
>> binary compatible.
>> 
>> You can sidestep this by not including ignite-log4j module and instead 
>> resorting to ignite-log4j2.
>> 
>> Regards,
>> -- 
>> Ilya Kasnacheev
>> 
>> 
>> сб, 19 сент. 2020 г. в 01:47, Andrew Story > >:
>> Would it be possible in the next release of Ignite to upgrade the 3rd party
>> component
>> /opt/ignite/apache-ignite/libs/optional/ignite-rest-http/log4j-1.2.17.jar to
>> log4j-core-2.13.3.jar?
>> 
>> This component log4j-1.2.17.jar is flagged as having a critical security
>> vulnerability which is described here:
>> https://nvd.nist.gov/vuln/detail/CVE-2019-17571 
>> 
>> 
>> The latest version of this component appears to be 2.13.3 which should
>> resolve the vulnerability:
>> https://logging.apache.org/log4j/2.x/download.html 
>> .
>> 
>> Thanks,
>> 
>> Andrew Story
>> 
>> 
>> 
>> 
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ 
>> 
> 
> 

[jira] [Created] (IGNITE-13464) Ignite-rest-http includes vulnerable dependencies

[Stephen Darlington (Jira)]

Stephen Darlington created IGNITE-13464:
---

 Summary: Ignite-rest-http includes vulnerable dependencies
 Key: IGNITE-13464
 URL: https://issues.apache.org/jira/browse/IGNITE-13464
 Project: Ignite
  Issue Type: Bug
  Components: rest
Affects Versions: 2.8.1, 2.9
Reporter: Stephen Darlington


The ignite-rest-http module includes a [vulnerable 
version|https://nvd.nist.gov/vuln/detail/CVE-2019-17571] of the log4j library. 
It also appears to include slf4j. Why does the REST API include its own logging 
libraries?

This was spotted in 2.8.1 but still appears to be an issue in master and 2.9.

More here:

http://apache-ignite-users.70518.x6.nabble.com/critical-security-vulnerability-for-opt-ignite-apache-ignite-libs-optional-ignite-rest-http-log4j-1-r-td34031.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13463) Calcite improvements. Rework RootNode rows exchange.

[Stanilovsky Evgeny (Jira)]

Stanilovsky Evgeny created IGNITE-13463:
---

 Summary: Calcite improvements. Rework RootNode rows exchange.
 Key: IGNITE-13463
 URL: https://issues.apache.org/jira/browse/IGNITE-13463
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.8.1
 Environment: In current implementation calcite.exec.rel.RootNode 
generates huge number of context switches, looks like we have a field for 
optimizations here. Need additional investigations and benchmarks.
Reporter: Stanilovsky Evgeny
Assignee: Stanilovsky Evgeny






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]

[Alex Plehanov]

Guys,

During internal testing, we've found a critical bug with discovery (cluster
falls apart if two nodes segmented sequentially). This problem is not
reproduced in 2.8.1. I think we should fix it before release. Under
investigation now. I'll let you know when we get something.

чт, 17 сент. 2020 г. в 00:51, Andrey Gura :

> > So what do you think? Should we proceed with a 'hacked' version of the
> message factory in 2.9 and go for the runtime message generation in later
> release, or keep the code clean and fix the regression in the next releases?
> > Andrey, can you take a look at my change? I think it is fairly
> straightforward and does not change the semantics, just skips the factory
> closures for certain messages.
>
> IMHO 2.5% isn't too much especially because it isn't actual for all
> workloads (I didn't get any significant drops during benchmarking). So
> I prefer the runtime generation in later releases.
>
> On Mon, Sep 14, 2020 at 12:41 PM Alexey Goncharuk
>  wrote:
> >
> > Alexey, Andrey, Igniters,
> >
> > So what do you think? Should we proceed with a 'hacked' version of the
> message factory in 2.9 and go for the runtime message generation in later
> release, or keep the code clean and fix the regression in the next releases?
> > Andrey, can you take a look at my change? I think it is fairly
> straightforward and does not change the semantics, just skips the factory
> closures for certain messages.
> >
> > Personally, I would prefer fixing the regression given that we also
> introduced tracing in this release.
> >
> >
> >
> > пт, 11 сент. 2020 г. в 12:09, Alex Plehanov :
> >>
> >> Alexey,
> >>
> >> We've benchmarked by yardstick commits 130376741bf vs ed52559eb95 and
> the performance of ed52559eb95 is better for about 2.5% on average on our
> environment (about the same results we got benchmarking 65c30ec6 vs
> 0606f03d). We've made 24 runs for each commit of
> IgnitePutTxImplicitBenchmark (we got maximum drop for 2.9 on this
> benchmark), 200 seconds warmup, 300 seconds benchmark, 6 servers, 5 clients
> 50 threads each. Yardstick results for this configuration:
> >> Commit 130376741bf: avg TPS=164096, avg latency=9173464 ns
> >> Commit ed52559eb95: avg TPS=168283, avg latency=8945908 ns
> >>
> >> пт, 11 сент. 2020 г. в 09:51, Artem Budnikov <
> a.budnikov.ign...@gmail.com>:
> >>>
> >>> Hi Everyone,
> >>>
> >>> I posted an instruction on how to publish the docs on
> ignite.apache.org/docs [1]. When you finish with Ignite 2.9, you can
> update the docs by following the instruction. Unfortunately, I won't be
> able to spend any time on this project any longer. You can send your pull
> requests and questions about the documentation to Denis Magda.
> >>>
> >>> -Artem
> >>>
> >>> [1] :
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Document
> >>>
> >>> On Thu, Sep 10, 2020 at 2:45 PM Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
> 
>  Alexey,
> 
>  I've tried to play with message factories locally, but unfortunately,
> I
>  cannot spot the difference between old and new implementation in
>  distributed benchmarks. I pushed an implementation of
> MessageFactoryImpl
>  with the old switch statement to the ignite-2.9-revert-12568 branch
>  (discussed this with Andrey Gura, the change should be compatible
> with the
>  new metrics as we still use the register() mechanics).
> 
>  Can you check if this change makes any difference performance-wise in
> your
>  environment? If yes, we can go with runtime code generation in the
> long
>  term: register classes and generate a dynamic message factory with a
> switch
>  statement once all messages are registered (not in 2.9 though,
> obviously).
> 
>  ср, 9 сент. 2020 г. в 14:53, Alex Plehanov :
> 
>  > Hello guys,
>  >
>  > I've tried to optimize tracing implementation (ticket [1]), it
> reduced the
>  > drop, but not completely removed it.
>  > Ivan Rakov, Alexander Lapin, can you please review the patch?
>  > Ivan Artiukhov, can you please benchmark the patch [2] against 2.8.1
>  > release on your environment?
>  > With this patch on our environment, it's about a 3% drop left, it's
> close
>  > to measurement error and I think such a drop is not a showstopper.
> Guys,
>  > WDYT?
>  >
>  > Also, I found that compatibility is broken for JDBC thin driver
> between 2.8
>  > and 2.9 versions (ticket [3]). I think it's a blocker and should be
>  > fixed in 2.9. I've prepared the patch.
>  > Taras Ledkov, can you please review this patch?
>  >
>  > And one more ticket I propose to include into 2.9 [4] (NIO message
>  > send problem in some circumstances). I will cherry-pick it if there
> is no
>  > objection.
>  >
>  > [1] https://issues.apache.org/jira/browse/IGNITE-13411
>  > [2] https://github.com/apache/ignite/pull/8223
>  > [3] 

[jira] [Created] (IGNITE-13462) .NET: Thin client Dispose hangs when continuous query is active on .NET Core 3.x

[Pavel Tupitsyn (Jira)]

Pavel Tupitsyn created IGNITE-13462:
---

 Summary: .NET: Thin client Dispose hangs when continuous query is 
active on .NET Core 3.x
 Key: IGNITE-13462
 URL: https://issues.apache.org/jira/browse/IGNITE-13462
 Project: Ignite
  Issue Type: Bug
  Components: platforms
Affects Versions: 2.9
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
 Fix For: 2.10


* Switch all projects to netcoreapp3.0 (or 3.1)
* Run Apache.Ignite.Core.Tests.Client.Cache.ContinuousQueryTest - it hangs on 
ClientSocket.Dispose

Looks like Socket.Shutdown call is missing, or we can use Socket.Close with a 
timeout.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)