[jira] [Created] (IGNITE-13107) ODBC: Memory leak in the tests

[Igor Sapego (Jira)]

Igor Sapego created IGNITE-13107:


 Summary: ODBC: Memory leak in the tests
 Key: IGNITE-13107
 URL: https://issues.apache.org/jira/browse/IGNITE-13107
 Project: Ignite
  Issue Type: Improvement
  Components: odbc
Reporter: Igor Sapego
 Fix For: 2.9


The memory leak, which is reproducible on TC Windows debug configuration, have 
place in case of odce-test unit tests executing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: maintenance on repository.apache.org - may have broken an ignite staging repository

[Denis Magda]

Igniters,

Does anybody know what that staging repository is used for? Is it safe to
drop it?

Wes, let's for a couple of days and, in case of no response, you can go
ahead and remove that staging.

-
Denis


On Mon, Jun 1, 2020 at 1:55 PM Wes Wannemacher  wrote:

> Hello ignite folks.
>
> I noticed that there is a staging repo on repository.apache.org, it's url
> is here:
> https://repository.apache.org/content/repositories/orgapacheignite-1249
>
> I had to address some data integrity issues on repository.apache.org and
> that staging repo may have been a casualty. Likely the easiest fix would be
> to drop that repository and stage a new build.
>
> If you have any questions, please let me know.
>
> -Wes
> --
> Wes Wannemacher
>

Re: IEP-46 Thin client - Service invocation

[Alex Plehanov]

Alexey,

Yes, I've got almost the same performance (the difference is about 1%). I
think proxy creation has a relatively low cost compared to 2-4 network hops
required to invoke the service method.

More precisely:
First approach: create a service proxy on server-side each time, when a
service method is invoked by thin client (on each OP_SERVICE_INVOKE
request).
Second approach: create a service proxy by request (OP_SERVICE_GET_PROXY),
store it on server-side in resource map (ConcurrentHashMap), get it from
resource map each time when a service method is invoked (OP_SERVICE_INVOKE)

Only OP_SERVICE_INVOKE was used for benchmarking (OP_SERVICE_GET_PROXY was
used during the warmup phase)

Here are GitHub links to PoC implementations for both approaches ([1],
[2]). Benchmark
class: 
org.apache.ignite.yardstick.thin.service.IgniteThinServiceInvocationBenchmark

[1]: https://github.com/alex-plekhanov/ignite/tree/ignite-13033-bench-a1
[2]: https://github.com/alex-plekhanov/ignite/tree/ignite-13033-bench-a2


вт, 2 июн. 2020 г. в 17:16, Alexey Goncharuk :

> Alexey,
>
> Hello,
> >
> > I've benchmarked 1-operation approach vs 2-operations approach and
> > published benchmark results on IEP page [1]. It looks like performance
> > almost the same, so the single-operation approach should be implemented.
> >
> Do I understand correctly that you suggest to remote
> the OP_SERVICE_GET_PROXY request altogether given that the performance is
> the same?
>
>
> > One more question: do we need some parameter on the server-side to
> disable
> > service invocations by thin client? Do we need to disable it by default?
> > I think the parameter will be useful, but I'm not sure about default
> value.
> > What do you think?
> >
> I do not think we need an additional flag for this, my thought is that it
> should be ruled by appropriate security permissions.
>

Re: [DISCUSS] Ignite process exit code on node stop by failure handler

[Andrey Gura]

>> And why we should try System.exit()?
>> Why we try to stop the node before invoke Runtime#halt()?) By the same
>> reasons we should try to stop the process with shutdown hooks
>> invocation before Runtime#halt().

Once again: we can't be sure that node will be stopped because we
don't manage code of shutdown hook. Of course we can do it using
timeout. But I still don't understand the motivation of such changes.
I believe that there is no real life case for proposed change.

Also I described early why this change is useless for administrator
and why 130 exit code is bad idea.

On Tue, Jun 2, 2020 at 6:20 PM Sergey Antonov  wrote:
>
> Andrey,
>
> > And why we should try System.exit()?
> Why we try to stop the node before invoke Runtime#halt()?) By the same
> reasons we should try to stop the process with shutdown hooks
> invocation before Runtime#halt().
>
> вт, 2 июн. 2020 г. в 17:51, Andrey Gura :
>
> > > Current implementation StopNodeOrHaltFH uses Runtime#halt(). This method
> > > doesn't cause shutdown hooks. Why we don't try to stop the node by
> > > System#exit() before Runtime#halt()?
> >
> > And why we should try System.exit()?
> >
> > We invoke halt() because we don't want to invoke any code in shutdown
> > hook. We don't know what shut down will try to do and we can't sure
> > that node will actually terminated. halt()  gives such guarantee.
> >
> > On Tue, Jun 2, 2020 at 5:35 PM Sergey Antonov 
> > wrote:
> > >
> > > Andrey, Alexey, I can't agree with your position.
> > >
> > > Current implementation StopNodeOrHaltFH uses Runtime#halt(). This method
> > > doesn't cause shutdown hooks. Why we don't try to stop the node by
> > > System#exit() before Runtime#halt()?
> > >
> > >
> > > вт, 2 июн. 2020 г. в 17:18, Andrey Gura :
> > >
> > > > Sergey, Anton, Kirill,
> > > >
> > > > I think we have to do not change anything in FH. At least without
> > > > enough motivation.
> > > >
> > > > Now I see only assumption that it "could be helpful for administration
> > > > purposes". It isn't enough, I believe.
> > > >
> > > > On Tue, Jun 2, 2020 at 4:59 PM ткаленко кирилл 
> > > > wrote:
> > > > >
> > > > > Hello, Alexey!
> > > > >
> > > > > I didn't quite understand about merge.
> > > > >
> > > > > If we use StopNodeOrHaltFailureHandler with tryStop=true, then if we
> > > > don't stop node by timeout, we will terminate jvm.
> > > > >
> > > > > Or do you suggest only stopping the node in StopNodeFailureHandler
> > and
> > > > terminate jvm in StopNodeOrHaltFailureHandler? or leave it as it is?
> > > > >
> > > > > 02.06.2020, 16:46, "Alexey Goncharuk" :
> > > > > >>  > How exactly do you want to change the StopNodeFH?
> > > > > >>  I want to stop JVM with KILL_EXIT_CODE and add an option
> > (constructor
> > > > > >>  argument of JVM option) for disabling JVM termination.
> > > > > >
> > > > > > When the flag is enabled, the behavior is identical to
> > StopNodeOrHaltFH
> > > > > > with tryStop=false. In other words,
> > > > > > StopNodeFH with enabled JVM termination === StopNodeOrHaltFH with
> > > > > > tryStop=false
> > > > > > StopNodeFG with disabled JVM termination ~ StopNodeOrHaltFH with
> > > > > > tryStop=true
> > > > > >
> > > > > > Why have two different failure handlers with identical behavior?
> > > > Perhaps,
> > > > > > there is confusion and we should consider merging these classes
> > into
> > > > one?
> > > >
> > >
> > >
> > > --
> > > BR, Sergey Antonov
> >
>
>
> --
> BR, Sergey Antonov

[MTCGA]: new failures in builds [5348710] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *Test with high flaky rate in master 
BinaryConfigurationTest.TestCodeConfiguration 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=-2134815735276887535&branch=%3Cdefault%3E&tab=testDetails

 *Test with high flaky rate in master 
BinaryConfigurationTest.TestXmlConfiguration 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=4488201393739220401&branch=%3Cdefault%3E&tab=testDetails
 Changes may lead to failure were done by 
 - korlov42  
https://ci.ignite.apache.org/viewModification.html?modId=902491

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 18:31:33 02-06-2020 

Re: [DISCUSS] Ignite process exit code on node stop by failure handler

[Sergey Antonov]

Andrey,

> And why we should try System.exit()?
Why we try to stop the node before invoke Runtime#halt()?) By the same
reasons we should try to stop the process with shutdown hooks
invocation before Runtime#halt().

вт, 2 июн. 2020 г. в 17:51, Andrey Gura :

> > Current implementation StopNodeOrHaltFH uses Runtime#halt(). This method
> > doesn't cause shutdown hooks. Why we don't try to stop the node by
> > System#exit() before Runtime#halt()?
>
> And why we should try System.exit()?
>
> We invoke halt() because we don't want to invoke any code in shutdown
> hook. We don't know what shut down will try to do and we can't sure
> that node will actually terminated. halt()  gives such guarantee.
>
> On Tue, Jun 2, 2020 at 5:35 PM Sergey Antonov 
> wrote:
> >
> > Andrey, Alexey, I can't agree with your position.
> >
> > Current implementation StopNodeOrHaltFH uses Runtime#halt(). This method
> > doesn't cause shutdown hooks. Why we don't try to stop the node by
> > System#exit() before Runtime#halt()?
> >
> >
> > вт, 2 июн. 2020 г. в 17:18, Andrey Gura :
> >
> > > Sergey, Anton, Kirill,
> > >
> > > I think we have to do not change anything in FH. At least without
> > > enough motivation.
> > >
> > > Now I see only assumption that it "could be helpful for administration
> > > purposes". It isn't enough, I believe.
> > >
> > > On Tue, Jun 2, 2020 at 4:59 PM ткаленко кирилл 
> > > wrote:
> > > >
> > > > Hello, Alexey!
> > > >
> > > > I didn't quite understand about merge.
> > > >
> > > > If we use StopNodeOrHaltFailureHandler with tryStop=true, then if we
> > > don't stop node by timeout, we will terminate jvm.
> > > >
> > > > Or do you suggest only stopping the node in StopNodeFailureHandler
> and
> > > terminate jvm in StopNodeOrHaltFailureHandler? or leave it as it is?
> > > >
> > > > 02.06.2020, 16:46, "Alexey Goncharuk" :
> > > > >>  > How exactly do you want to change the StopNodeFH?
> > > > >>  I want to stop JVM with KILL_EXIT_CODE and add an option
> (constructor
> > > > >>  argument of JVM option) for disabling JVM termination.
> > > > >
> > > > > When the flag is enabled, the behavior is identical to
> StopNodeOrHaltFH
> > > > > with tryStop=false. In other words,
> > > > > StopNodeFH with enabled JVM termination === StopNodeOrHaltFH with
> > > > > tryStop=false
> > > > > StopNodeFG with disabled JVM termination ~ StopNodeOrHaltFH with
> > > > > tryStop=true
> > > > >
> > > > > Why have two different failure handlers with identical behavior?
> > > Perhaps,
> > > > > there is confusion and we should consider merging these classes
> into
> > > one?
> > >
> >
> >
> > --
> > BR, Sergey Antonov
>


-- 
BR, Sergey Antonov

Re: [DISCUSS] Ignite process exit code on node stop by failure handler

[Andrey Gura]

> Current implementation StopNodeOrHaltFH uses Runtime#halt(). This method
> doesn't cause shutdown hooks. Why we don't try to stop the node by
> System#exit() before Runtime#halt()?

And why we should try System.exit()?

We invoke halt() because we don't want to invoke any code in shutdown
hook. We don't know what shut down will try to do and we can't sure
that node will actually terminated. halt()  gives such guarantee.

On Tue, Jun 2, 2020 at 5:35 PM Sergey Antonov  wrote:
>
> Andrey, Alexey, I can't agree with your position.
>
> Current implementation StopNodeOrHaltFH uses Runtime#halt(). This method
> doesn't cause shutdown hooks. Why we don't try to stop the node by
> System#exit() before Runtime#halt()?
>
>
> вт, 2 июн. 2020 г. в 17:18, Andrey Gura :
>
> > Sergey, Anton, Kirill,
> >
> > I think we have to do not change anything in FH. At least without
> > enough motivation.
> >
> > Now I see only assumption that it "could be helpful for administration
> > purposes". It isn't enough, I believe.
> >
> > On Tue, Jun 2, 2020 at 4:59 PM ткаленко кирилл 
> > wrote:
> > >
> > > Hello, Alexey!
> > >
> > > I didn't quite understand about merge.
> > >
> > > If we use StopNodeOrHaltFailureHandler with tryStop=true, then if we
> > don't stop node by timeout, we will terminate jvm.
> > >
> > > Or do you suggest only stopping the node in StopNodeFailureHandler and
> > terminate jvm in StopNodeOrHaltFailureHandler? or leave it as it is?
> > >
> > > 02.06.2020, 16:46, "Alexey Goncharuk" :
> > > >>  > How exactly do you want to change the StopNodeFH?
> > > >>  I want to stop JVM with KILL_EXIT_CODE and add an option (constructor
> > > >>  argument of JVM option) for disabling JVM termination.
> > > >
> > > > When the flag is enabled, the behavior is identical to StopNodeOrHaltFH
> > > > with tryStop=false. In other words,
> > > > StopNodeFH with enabled JVM termination === StopNodeOrHaltFH with
> > > > tryStop=false
> > > > StopNodeFG with disabled JVM termination ~ StopNodeOrHaltFH with
> > > > tryStop=true
> > > >
> > > > Why have two different failure handlers with identical behavior?
> > Perhaps,
> > > > there is confusion and we should consider merging these classes into
> > one?
> >
>
>
> --
> BR, Sergey Antonov

[DISCUSS] Make AtomicConfiguration more flexible

[Vladimir Pligin]

Hi Igniters!

It seems that AtomicConfiguration
(https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/AtomicConfiguration.html)
is not flexible enough to cover some reasonable use-cases. It's possible to
configure only cache mode, group name, affinity and backups of the
underlying cache. It would be great to have the ability to inject
full-fledged CacheConfiguration there. For example there's no
straightforward way to configure a node filter, this scenario can be useful
as sometimes it's required to avoid atomic sequence residence on a subset of
nodes.

What do you think about it, are there any pitfalls? 



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: [DISCUSS] Ignite process exit code on node stop by failure handler

[Sergey Antonov]

Andrey, Alexey, I can't agree with your position.

Current implementation StopNodeOrHaltFH uses Runtime#halt(). This method
doesn't cause shutdown hooks. Why we don't try to stop the node by
System#exit() before Runtime#halt()?


вт, 2 июн. 2020 г. в 17:18, Andrey Gura :

> Sergey, Anton, Kirill,
>
> I think we have to do not change anything in FH. At least without
> enough motivation.
>
> Now I see only assumption that it "could be helpful for administration
> purposes". It isn't enough, I believe.
>
> On Tue, Jun 2, 2020 at 4:59 PM ткаленко кирилл 
> wrote:
> >
> > Hello, Alexey!
> >
> > I didn't quite understand about merge.
> >
> > If we use StopNodeOrHaltFailureHandler with tryStop=true, then if we
> don't stop node by timeout, we will terminate jvm.
> >
> > Or do you suggest only stopping the node in StopNodeFailureHandler and
> terminate jvm in StopNodeOrHaltFailureHandler? or leave it as it is?
> >
> > 02.06.2020, 16:46, "Alexey Goncharuk" :
> > >>  > How exactly do you want to change the StopNodeFH?
> > >>  I want to stop JVM with KILL_EXIT_CODE and add an option (constructor
> > >>  argument of JVM option) for disabling JVM termination.
> > >
> > > When the flag is enabled, the behavior is identical to StopNodeOrHaltFH
> > > with tryStop=false. In other words,
> > > StopNodeFH with enabled JVM termination === StopNodeOrHaltFH with
> > > tryStop=false
> > > StopNodeFG with disabled JVM termination ~ StopNodeOrHaltFH with
> > > tryStop=true
> > >
> > > Why have two different failure handlers with identical behavior?
> Perhaps,
> > > there is confusion and we should consider merging these classes into
> one?
>


-- 
BR, Sergey Antonov

Re: [DISCUSS] Ignite process exit code on node stop by failure handler

[Andrey Gura]

Sergey, Anton, Kirill,

I think we have to do not change anything in FH. At least without
enough motivation.

Now I see only assumption that it "could be helpful for administration
purposes". It isn't enough, I believe.

On Tue, Jun 2, 2020 at 4:59 PM ткаленко кирилл  wrote:
>
> Hello, Alexey!
>
> I didn't quite understand about merge.
>
> If we use StopNodeOrHaltFailureHandler with tryStop=true, then if we don't 
> stop node by timeout, we will terminate jvm.
>
> Or do you suggest only stopping the node in StopNodeFailureHandler and 
> terminate jvm in StopNodeOrHaltFailureHandler? or leave it as it is?
>
> 02.06.2020, 16:46, "Alexey Goncharuk" :
> >>  > How exactly do you want to change the StopNodeFH?
> >>  I want to stop JVM with KILL_EXIT_CODE and add an option (constructor
> >>  argument of JVM option) for disabling JVM termination.
> >
> > When the flag is enabled, the behavior is identical to StopNodeOrHaltFH
> > with tryStop=false. In other words,
> > StopNodeFH with enabled JVM termination === StopNodeOrHaltFH with
> > tryStop=false
> > StopNodeFG with disabled JVM termination ~ StopNodeOrHaltFH with
> > tryStop=true
> >
> > Why have two different failure handlers with identical behavior? Perhaps,
> > there is confusion and we should consider merging these classes into one?

Re: IEP-46 Thin client - Service invocation

[Alexey Goncharuk]

Alexey,

Hello,
>
> I've benchmarked 1-operation approach vs 2-operations approach and
> published benchmark results on IEP page [1]. It looks like performance
> almost the same, so the single-operation approach should be implemented.
>
Do I understand correctly that you suggest to remote
the OP_SERVICE_GET_PROXY request altogether given that the performance is
the same?


> One more question: do we need some parameter on the server-side to disable
> service invocations by thin client? Do we need to disable it by default?
> I think the parameter will be useful, but I'm not sure about default value.
> What do you think?
>
I do not think we need an additional flag for this, my thought is that it
should be ruled by appropriate security permissions.

Re: [DISCUSS] Ignite process exit code on node stop by failure handler

[ткаленко кирилл]

Hello, Alexey!

I didn't quite understand about merge. 

If we use StopNodeOrHaltFailureHandler with tryStop=true, then if we don't stop 
node by timeout, we will terminate jvm.

Or do you suggest only stopping the node in StopNodeFailureHandler and 
terminate jvm in StopNodeOrHaltFailureHandler? or leave it as it is?

02.06.2020, 16:46, "Alexey Goncharuk" :
>>  > How exactly do you want to change the StopNodeFH?
>>  I want to stop JVM with KILL_EXIT_CODE and add an option (constructor
>>  argument of JVM option) for disabling JVM termination.
>
> When the flag is enabled, the behavior is identical to StopNodeOrHaltFH
> with tryStop=false. In other words,
> StopNodeFH with enabled JVM termination === StopNodeOrHaltFH with
> tryStop=false
> StopNodeFG with disabled JVM termination ~ StopNodeOrHaltFH with
> tryStop=true
>
> Why have two different failure handlers with identical behavior? Perhaps,
> there is confusion and we should consider merging these classes into one?

Re: IEP-46 Thin client - Service invocation

[Alex Plehanov]

Hello,

I've benchmarked 1-operation approach vs 2-operations approach and
published benchmark results on IEP page [1]. It looks like performance
almost the same, so the single-operation approach should be implemented.

One more question: do we need some parameter on the server-side to disable
service invocations by thin client? Do we need to disable it by default?
I think the parameter will be useful, but I'm not sure about default value.
What do you think?

[1] :
https://cwiki.apache.org/confluence/display/IGNITE/IEP-46%3A+Thin+Client+Service+Invocation

ср, 20 мая 2020 г. в 14:50, Alex Plehanov :

> Pavel,
>
> I will try to benchmark by myself. Thank you.
>
> ср, 20 мая 2020 г. в 13:54, Pavel Tupitsyn :
>
>> Alex,
>>
>> Thank you, the IEP looks complete now.
>>
>> Are you going to do the comparison benchmark of 1-operation vs 2-operation
>> modes?
>> I can do that too, just want to avoid duplicate work.
>>
>> On Wed, May 20, 2020 at 1:17 PM Alex Plehanov 
>> wrote:
>>
>> > Pavel,
>> >
>> > Yes, looks like we can get this information from ServiceDescriptor. I've
>> > removed 'interface name' from requests in IEP. Thank you!
>> >
>> > ср, 20 мая 2020 г. в 12:58, Pavel Tupitsyn :
>> >
>> > > Alex,
>> > >
>> > > IEP looks good to me in general.
>> > >
>> > > One question: what is `Interface name` in the request?
>> > > As I understand, service name is enough to retrieve ServiceDescriptor,
>> > > and then we can get serviceClass from there.
>> > >
>> > > On Wed, May 20, 2020 at 12:42 PM Alex Plehanov <
>> plehanov.a...@gmail.com>
>> > > wrote:
>> > >
>> > > > Pavel,
>> > > >
>> > > > I've moved proposals from this thread to the IEP [1] and described
>> > > changes
>> > > > to protocol, please have a look.
>> > > >
>> > > > [1] :
>> > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-46%3A+Thin+Client+Service+Invocation
>> > > >
>> > > > пн, 18 мая 2020 г. в 15:09, Pavel Tupitsyn :
>> > > >
>> > > > > Hi Alex,
>> > > > >
>> > > > > Thanks for starting this thread.
>> > > > > I've created the IEP some time ago but then got distracted by
>> other
>> > > > things.
>> > > > >
>> > > > > My thoughts:
>> > > > >
>> > > > > *1. Create service proxy on each invoke request*
>> > > > > Single client operation is a lot simpler to implement on both
>> sides
>> > > > (client
>> > > > > and server), so I would prefer this way.
>> > > > > The only concern here is performance. My plan was to benchmark and
>> > then
>> > > > > decide.
>> > > > >
>> > > > > *2. Method resolution*
>> > > > > Taking Python and other dynamically typed languages into account,
>> > > > > current PlatformService implementation seems to be the best fit.
>> > > > >
>> > > > > If we decide to add an optional signature, we should not make it
>> > > > > Java-specific:
>> > > > > use binary type IDs (that are already part of the protocol) to
>> > specify
>> > > > > parameter types.
>> > > > >
>> > > > >
>> > > > > On Mon, May 18, 2020 at 2:27 PM Alex Plehanov <
>> > plehanov.a...@gmail.com
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hello Igniters,
>> > > > > >
>> > > > > > I have plans to implement service invocation in thin clients.
>> I've
>> > > > > already
>> > > > > > implemented a PoC for java thin client and want to discuss some
>> > > > details.
>> > > > > > Also, Pavel Tupitsin has created IEP [1] and the ticket for .Net
>> > thin
>> > > > > > client [2].
>> > > > > >
>> > > > > > To invoke the service method by java thick client we can get
>> > service
>> > > > > proxy
>> > > > > > by:
>> > > > > > IgniteServices.serviceProxy(String name, Class
>> svcItf,
>> > > > boolean
>> > > > > > sticky, long timeout)
>> > > > > > and then invoke the method on this proxy.
>> > > > > >
>> > > > > > Also, we already have service invocation functionality for .Net
>> > thick
>> > > > > > client (see PlatformServices class). For .Net thick client there
>> > are
>> > > > two
>> > > > > > types of operation related to service invocation: create a proxy
>> > and
>> > > > > invoke
>> > > > > > a method (this approach is identical to java thick client).
>> > > > > >
>> > > > > > But I think we can't use two operations for thin clients. If we
>> > > create
>> > > > a
>> > > > > > proxy by a separate operation we should store this resource
>> > somewhere
>> > > > on
>> > > > > > server-side and we should also have an ability to close this
>> > resource
>> > > > > when
>> > > > > > it's not needed anymore. So there is an additional operation on
>> > > > > client-side
>> > > > > > needed to close the proxy explicitly. This is more complex from
>> > users
>> > > > > point
>> > > > > > of view than the current approach for java thick client, so I
>> think
>> > > > it's
>> > > > > > better to use only one operation for thin clients:
>> > OP_SERVICE_INVOKE
>> > > > and
>> > > > > > create a service proxy on each method invocation. In this case,
>> > each
>> > > > > > request should have at least these parameters: service name,
>> > > 

Re: [DISCUSSION] Add autocompletion for commands in control.sh

[ткаленко кирилл]

Maxim I suggested moving control.sh in a separate module, are we talking about 
the same thing?

02.06.2020, 16:15, "Maxim Muzafarov" :
> Folks,
>
> +1
>
> However, AFAIK control.sh is the part of the ignite-core module with
> zero dependencies from external resources.
> Would it be better going the `control.sh` extensions-way?
>
> By the way, according to README.md [1] the picocli is already using by
> the Apache Ignite, right? :-)
>
>>  Picocli is used in the Apache Hadoop Ozone/HDDS command line tools, the 
>> Apache Hive benchmark CLI, has ** Apache [Ignite TensorFlow] **, and Apache 
>> Sling.
>
> [1] https://github.com/remkop/picocli/blame/master/README.md#L199
>
> On Tue, 2 Jun 2020 at 16:09, Ivan Daschinsky  wrote:
>>  +1 But this is not only usability improvement, but also a huge code
>>  improvement. With picocli developers can add custom command without writing
>>  a lot of boilerplate and error prone code to do a trivial task
>>  of parsing CLI arguments. Cleaner code, less bugs also matter.
>>
>>  вт, 2 июн. 2020 г. в 16:02, Sergey Antonov :
>>
>>  > It would be a great usability improvement!
>>  >
>>  > +1 From me.
>>  >
>>  > вт, 2 июн. 2020 г. в 15:54, Zhenya Stanilovsky >  > >:
>>  >
>>  > >
>>  > >
>>  > > good catch ! it`s a little bit pain for now to working with it.
>>  > >
>>  > >
>>  > > >Hi, Igniters!
>>  > > >
>>  > > >At the moment to work with the control.sh we need to know exactly what
>>  > > the name of the command and its options are and so the user can often
>>  > make
>>  > > mistakes when using it. So I think it would be useful to do control.sh
>>  > more
>>  > > user-friendly by adding autocomplete as in modern command-line 
>> utilities.
>>  > > >
>>  > > >For this purpose, I suggest using framework [1] and to do this, take 
>> out
>>  > > control.sh together with its associated classes in a separate module 
>> such
>>  > > as "modules/control-utility".
>>  > > >
>>  > > >Comments, suggestions?
>>  > > >
>>  > > >[1] - https://picocli.info/
>>  > >
>>  > >
>>  > >
>>  > >
>>  >
>>  >
>>  >
>>  > --
>>  > BR, Sergey Antonov
>>  >
>>
>>  --
>>  Sincerely yours, Ivan Daschinskiy

Re: [DISCUSS] Ignite process exit code on node stop by failure handler

[Alexey Goncharuk]

> > How exactly do you want to change the StopNodeFH?
> I want to stop JVM with KILL_EXIT_CODE and add an option (constructor
> argument of JVM option) for disabling JVM termination.
>
When the flag is enabled, the behavior is identical to StopNodeOrHaltFH
with tryStop=false. In other words,
StopNodeFH with enabled JVM termination === StopNodeOrHaltFH with
tryStop=false
StopNodeFG with disabled JVM termination ~ StopNodeOrHaltFH with
tryStop=true

Why have two different failure handlers with identical behavior? Perhaps,
there is confusion and we should consider merging these classes into one?

Re: [DISCUSSION] Add autocompletion for commands in control.sh

[Ivan Daschinsky]

> By the way, according to README.md [1] the picocli is already using bythe
Apache Ignite, right? :-)
Yes, you are right, but a little bit outdated.

> Would it be better going the `control.sh` extensions-way?
Sounds good. Also, it would be great to incorporate in command line
utilities some features from visor-console also. And after developing new
tool, deprecate both (visor-console and control.sh)

вт, 2 июн. 2020 г. в 16:15, Maxim Muzafarov :

> Folks,
>
> +1
>
> However, AFAIK control.sh is the part of the ignite-core module with
> zero dependencies from external resources.
> Would it be better going the `control.sh` extensions-way?
>
>
> By the way, according to README.md [1] the picocli is already using by
> the Apache Ignite, right? :-)
>
> > Picocli is used in the Apache Hadoop Ozone/HDDS command line tools, the
> Apache Hive benchmark CLI, has ** Apache [Ignite TensorFlow] **, and Apache
> Sling.
>
>
> [1] https://github.com/remkop/picocli/blame/master/README.md#L199
>
> On Tue, 2 Jun 2020 at 16:09, Ivan Daschinsky  wrote:
> >
> > +1 But this is not only usability improvement, but also a huge code
> > improvement. With picocli developers can add custom command without
> writing
> > a lot of boilerplate and error prone code to do a trivial task
> > of parsing CLI arguments. Cleaner code, less bugs also matter.
> >
> > вт, 2 июн. 2020 г. в 16:02, Sergey Antonov :
> >
> > > It would be a great usability improvement!
> > >
> > > +1 From me.
> > >
> > > вт, 2 июн. 2020 г. в 15:54, Zhenya Stanilovsky
>  > > >:
> > >
> > > >
> > > >
> > > > good catch ! it`s a little bit pain for now to working with it.
> > > >
> > > >
> > > > >Hi, Igniters!
> > > > >
> > > > >At the moment to work with the control.sh we need to know exactly
> what
> > > > the name of the command and its options are and so the user can often
> > > make
> > > > mistakes when using it. So I think it would be useful to do
> control.sh
> > > more
> > > > user-friendly by adding autocomplete as in modern command-line
> utilities.
> > > > >
> > > > >For this purpose, I suggest using framework [1] and to do this,
> take out
> > > > control.sh together with its associated classes in a separate module
> such
> > > > as "modules/control-utility".
> > > > >
> > > > >Comments, suggestions?
> > > > >
> > > > >[1] -  https://picocli.info/
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > > BR, Sergey Antonov
> > >
> >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSS] Ignite process exit code on node stop by failure handler

[Andrey Gura]

Definitely agree with Alexey Goncharuk. Mentioned FH implementations
don't terminate JVM.

I think returning KILL_EXIT_CODE is bad idea because actually process
wasn't terminated using SIGINT. So it contradicts to motivation
described in proposal.

Also how could it help to administrators. For example node was stopped
by FH. What could script to do? Restart node? It is not good idea
because node was intentionally configured to be stopped.
Sent some notification? It could be done by monitoring tools.

I don't understand the motivation.

On Tue, Jun 2, 2020 at 1:53 PM Sergey Antonov  wrote:
>
> Alexey,
>
> > How exactly do you want to change the StopNodeFH?
> I want to stop JVM with KILL_EXIT_CODE and add an option (constructor
> argument of JVM option) for disabling JVM termination.
>
> вт, 2 июн. 2020 г. в 12:54, Alexey Goncharuk :
>
> > Sergey,
> >
> > How exactly do you want to change the StopNodeFH? The current behavior does
> > not terminate the JVM and its exit code is totally out of our control; one
> > of the use-cases we had in mind for this failure handler is that a user may
> > have other processes running in the same JVM, so we do not want Ignite to
> > affect them.
> >
> > For me, the exit code makes sense only for StopNodeOrHaltFH with
> > tryStop=false (otherwise, the JVM exit is not guaranteed as well), but we
> > already use the KILL_EXIT_CODE there.
> >
> > пн, 1 июн. 2020 г. в 23:19, Sergey Antonov :
> >
> > > Hello, Kirill!
> > >
> > > I'd prefer to don't create a new implementation of a failure handler. We
> > > already have 4 different failure handlers. We will have 6 FH (StopNodeFH
> > > with exit code, StopNodeOrHaltFH with exit code), if we go your way. It
> > > won't make our product simpler and easier.
> > >
> > > I think, we must notify a user if the cluster node had been stopped by a
> > > failure handler. We can't achieve this goal without changing current FH
> > > behavior. So I propose to change it and stop the process with
> > > KILL_EXIT_CODE.
> > > But it would be nice if users will have a flag for avoiding process stop
> > > after a node failure. We can introduce a new JVM option or FH parameter
> > for
> > > that reason. Of course, we must highlight this change in the release
> > notes.
> > >
> > > пн, 1 июн. 2020 г. в 19:07, ткаленко кирилл :
> > >
> > > > I think that [1] and [2] should not be changed, because we can kill
> > > > another client code in this jvm.
> > > > I suggest for these purposes to create a new [3] which will be like [1]
> > > > but with a call [4] after node stop.
> > > > Objections or comments?
> > > >
> > > > [1] - org.apache.ignite.failure.StopNodeFailureHandler
> > > > [2] - org.apache.ignite.failure.StopNodeOrHaltFailureHandler
> > > > [3] - org.apache.ignite.failure.FailureHandler
> > > > [4] - java.lang.System#exit
> > > >
> > > > 25.05.2020, 22:09, "Dmitriy Pavlov" :
> > > > > It seems reasonable to me. Also I would like to propose adding value
> > of
> > > > > Ignition.KILL_EXIT_CODE into javadoc using @value javadoc tag.
> > > > >
> > > > > Dev ops/admins/anyone who admins Ignite may want to know it's value
> > > > without
> > > > > going to Java code.
> > > > >
> > > > > чт, 21 мая 2020 г. в 09:39, Zhenya Stanilovsky
> > > > :
> > > > >
> > > > >>  Thank you Sergey, as for me — very useful proposal huge +1 here.
> > > > >>
> > > > >>  >Четверг, 21 мая 2020, 0:51 +03:00 от Sergey Antonov <
> > > > >>  antonovserge...@gmail.com>:
> > > > >>  >
> > > > >>  >I've created the Ignite ticket for this improvement [1].
> > > > >>  >
> > > > >>  >[1] https://issues.apache.org/jira/browse/IGNITE-13047
> > > > >>  >
> > > > >>  >чт, 21 мая 2020 г. в 00:46, Sergey Antonov <
> > > > antonovserge...@gmail.com >:
> > > > >>  >
> > > > >>  >> Hello, Igniters!
> > > > >>  >>
> > > > >>  >> I'd like to discuss behaviour of Ignite process exit code if the
> > > > node
> > > > >>  was
> > > > >>  >> stopped by failure handler [1][2]. At the moment ignite process
> > > > returns
> > > > >>  >> exit code 0 after the stop in all scenarios, except runtime halt
> > > by
> > > > >>  >> StopNodeOrHaltFH [1]. In this case, the exit code will be 130
> > [3]
> > > > >>  >>
> > > > >>  >> My proposal: always finish Ignite process with code [3], if a
> > node
> > > > was
> > > > >>  >> stopped by FH. It could be helpful for administration purposes,
> > > you
> > > > can
> > > > >>  >> distinguish normal node stop from node stop by FH on OS level.
> > > > >>  >>
> > > > >>  >> WDYT?
> > > > >>  >>
> > > > >>  >> [1]
> > > > >>  >>
> > > > >>
> > > >
> > >
> > https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/failure/StopNodeOrHaltFailureHandler.html
> > > > >>  >>
> > > > >>  >> [2]
> > > > >>  >>
> > > > >>
> > > >
> > >
> > https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/failure/StopNodeFailureHandler.html
> > > > >>  >>
> > > > >>  >> [3]
> > > > >>  >>
> > > > >>
> > > >
> > >
> > https://ignite.apache.org/releases/latest/javadoc

Re: [DISCUSSION] Add autocompletion for commands in control.sh

[Maxim Muzafarov]

Folks,

+1

However, AFAIK control.sh is the part of the ignite-core module with
zero dependencies from external resources.
Would it be better going the `control.sh` extensions-way?


By the way, according to README.md [1] the picocli is already using by
the Apache Ignite, right? :-)

> Picocli is used in the Apache Hadoop Ozone/HDDS command line tools, the 
> Apache Hive benchmark CLI, has ** Apache [Ignite TensorFlow] **, and Apache 
> Sling.


[1] https://github.com/remkop/picocli/blame/master/README.md#L199

On Tue, 2 Jun 2020 at 16:09, Ivan Daschinsky  wrote:
>
> +1 But this is not only usability improvement, but also a huge code
> improvement. With picocli developers can add custom command without writing
> a lot of boilerplate and error prone code to do a trivial task
> of parsing CLI arguments. Cleaner code, less bugs also matter.
>
> вт, 2 июн. 2020 г. в 16:02, Sergey Antonov :
>
> > It would be a great usability improvement!
> >
> > +1 From me.
> >
> > вт, 2 июн. 2020 г. в 15:54, Zhenya Stanilovsky  > >:
> >
> > >
> > >
> > > good catch ! it`s a little bit pain for now to working with it.
> > >
> > >
> > > >Hi, Igniters!
> > > >
> > > >At the moment to work with the control.sh we need to know exactly what
> > > the name of the command and its options are and so the user can often
> > make
> > > mistakes when using it. So I think it would be useful to do control.sh
> > more
> > > user-friendly by adding autocomplete as in modern command-line utilities.
> > > >
> > > >For this purpose, I suggest using framework [1] and to do this, take out
> > > control.sh together with its associated classes in a separate module such
> > > as "modules/control-utility".
> > > >
> > > >Comments, suggestions?
> > > >
> > > >[1] -  https://picocli.info/
> > >
> > >
> > >
> > >
> >
> >
> >
> > --
> > BR, Sergey Antonov
> >
>
>
> --
> Sincerely yours, Ivan Daschinskiy

Re: [DISCUSSION] Add autocompletion for commands in control.sh

[Ivan Daschinsky]

+1 But this is not only usability improvement, but also a huge code
improvement. With picocli developers can add custom command without writing
a lot of boilerplate and error prone code to do a trivial task
of parsing CLI arguments. Cleaner code, less bugs also matter.

вт, 2 июн. 2020 г. в 16:02, Sergey Antonov :

> It would be a great usability improvement!
>
> +1 From me.
>
> вт, 2 июн. 2020 г. в 15:54, Zhenya Stanilovsky  >:
>
> >
> >
> > good catch ! it`s a little bit pain for now to working with it.
> >
> >
> > >Hi, Igniters!
> > >
> > >At the moment to work with the control.sh we need to know exactly what
> > the name of the command and its options are and so the user can often
> make
> > mistakes when using it. So I think it would be useful to do control.sh
> more
> > user-friendly by adding autocomplete as in modern command-line utilities.
> > >
> > >For this purpose, I suggest using framework [1] and to do this, take out
> > control.sh together with its associated classes in a separate module such
> > as "modules/control-utility".
> > >
> > >Comments, suggestions?
> > >
> > >[1] -  https://picocli.info/
> >
> >
> >
> >
>
>
>
> --
> BR, Sergey Antonov
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSSION] Add autocompletion for commands in control.sh

[Sergey Antonov]

It would be a great usability improvement!

+1 From me.

вт, 2 июн. 2020 г. в 15:54, Zhenya Stanilovsky :

>
>
> good catch ! it`s a little bit pain for now to working with it.
>
>
> >Hi, Igniters!
> >
> >At the moment to work with the control.sh we need to know exactly what
> the name of the command and its options are and so the user can often make
> mistakes when using it. So I think it would be useful to do control.sh more
> user-friendly by adding autocomplete as in modern command-line utilities.
> >
> >For this purpose, I suggest using framework [1] and to do this, take out
> control.sh together with its associated classes in a separate module such
> as "modules/control-utility".
> >
> >Comments, suggestions?
> >
> >[1] -  https://picocli.info/
>
>
>
>



-- 
BR, Sergey Antonov

Re: [DISCUSSION] Add autocompletion for commands in control.sh

[Zhenya Stanilovsky]

good catch ! it`s a little bit pain for now to working with it.

 
>Hi, Igniters!
>
>At the moment to work with the control.sh we need to know exactly what the 
>name of the command and its options are and so the user can often make 
>mistakes when using it. So I think it would be useful to do control.sh more 
>user-friendly by adding autocomplete as in modern command-line utilities.
>
>For this purpose, I suggest using framework [1] and to do this, take out 
>control.sh together with its associated classes in a separate module such as 
>"modules/control-utility".
>
>Comments, suggestions?
>
>[1] -  https://picocli.info/ 
 
 
 
 

[DISCUSSION] Add autocompletion for commands in control.sh

[ткаленко кирилл]

Hi, Igniters!

At the moment to work with the control.sh we need to know exactly what the name 
of the command and its options are and so the user can often make mistakes when 
using it. So I think it would be useful to do control.sh more user-friendly by 
adding autocomplete as in modern command-line utilities.

For this purpose, I suggest using framework [1] and to do this, take out 
control.sh together with its associated classes in a separate module such as 
"modules/control-utility".

Comments, suggestions?

[1] - https://picocli.info/

Re: [DISCUSSION] IEP-47 Native persistence defragmentation

[Anton Vinogradov]

Ivan,

Thanks for hints.
Invalidated memory cache was restored :)

On Tue, Jun 2, 2020 at 2:55 PM Ivan Bessonov  wrote:

> Hello Anton,
>
> I'd like to address your last message. First of all, it was already
> partially discussed
> in this thread: [1] To reiterate - expected performance degradation will be
> significant.
> There's no way that we can throttle it because free/reuse lists have to be
> maintained
> sorted all the time. And these are very optimized data structures.
>
> More then that, "dummy" updates clash with data access, this is a very
> dangerous
> thing to do. And these updates don't save you from the situation when last
> pages in
> the file are not data pages, but tree pages, for example. They are much
> harder to
> move. Not only you should update all links to it but also do it
> effectively, without
> blocking the tree too much. I can think of many other examples.
>
> *Easy to implement/understand*
>  - no, it's not easy at all, defragmentation under the load is a very
> challenging thing to
>implement.
>
> *Why we're going to implement distributed system defragmentation in the old
> (offline) way?*
>  - because it's easier and safer, and it won't introduce any performance
> degradation.
>
> [1]
>
> http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html
>
> вт, 2 июн. 2020 г. в 14:17, Anton Vinogradov :
>
> > Folks,
> >
> > Modern OS never ask you to schedule defragmentation and turn your PC off,
> > it performs it while you're browsing.
> > Why we're going to implement distributed system defragmentation in the
> old
> > (offline) way?
> >
> > All you need is to implement free/reuse-list sorting. They should provide
> > pages closest to the file beginning.
> > So, every insert/update will automatically defragment the entry.
> > Also, a special process should iterate over the partitions in a reverse
> way
> > just performing dummy updates.
> > The partition file may be safely truncated after the iterator.
> >
> > Props:
> > - Your system still operating (no downtime)
> > - Defragmentation can be performed partially
> > - Defragmentation can be scheduled to periods of inactivity or performed
> on
> > a regular basis
> > - SQL will not be broken (no reason to recalculate the whole index, it
> will
> > be recalculated in a regular way on every entry update)
> > - Topology changes allowed
> > - Easy to implement/understand
> >
> > Cons:
> > - Performance degradation (solvable by throttling)
> >
> > On Mon, Jun 1, 2020 at 4:04 PM Sergey Chugunov <
> sergey.chugu...@gmail.com>
> > wrote:
> >
> > > Hi Ivan,
> > >
> > > I have an idea about suggested maintenance mode.
> > >
> > > First of all, I agree with your ideas about discovery restrictions:
> node
> > > should not join topology when performing defragmentation.
> > >
> > > At the same time I haven't heard about requests for this mode from
> users,
> > > so we don't know much about possible requirements.
> > > So I suggest to implement it in a pragmatical way: instead of inventing
> > > (unknown in reality) user scenarios lets develop minimal but yet
> > > well-designed functionality that suites our case. If we constrain our
> > > implementation with reasonable set of restrictions that's OK.
> > >
> > > So my idea is the following: to transit a node to maintenance user has
> to
> > > send special command to the node (e.g. with new command in control.sh
> > > utility or via JMX interface). Node saves maintenance request in local
> > > metastorage and waits for restart. User has to manually restart that
> node
> > > in order to finish moving it to maintenance mode.
> > >
> > > When node restarts and finds maintenance request it creates special
> type
> > of
> > > discovery SPI that will not try to join topology at all yet node is
> able
> > to
> > > start all necessary components and APIs like REST processor or JMX
> > > interface.
> > >
> > > When in maintenance, we'll be able to do defragmentation safely and
> > remove
> > > maintenance request from metastorage only when it is completed (with
> all
> > > fault-tolerance logic in mind).
> > >
> > > As we don't have a mechanism (like watcher) to perform a "safe restart"
> > (by
> > > safe I mean Ignite restart without OS-level process restart) we cannot
> > > finish maintenance mode without another manual restart but I think it
> is
> > a
> > > reasonable restriction as maintenance mode shouldn't be an every-day
> > > routine and will be used quite rare.
> > >
> > > What do you think about it?
> > >
> > > On Tue, May 26, 2020 at 5:58 PM Ivan Bessonov 
> > > wrote:
> > >
> > > > Hello Igniters,
> > > >
> > > > I'd like to discuss this new IEP with you: [1]. The main idea is to
> > have
> > > a
> > > > procedure that relocates
> > > > pages to the top of the file as compact as possible which allows us
> to
> > > > trim the file and increase its
> > > > fill-factor. It will be configured manuall

Re: Phani Introduction

[phani adusumilli]

Thanks Ivan.

Thanks

> On Jun 2, 2020, at 5:01 AM, Ivan Pavlukhin  wrote:
> 
> Hi Phani,
> 
> Welcome to Apache Ignite Community!
> 
> I added your account to the contributors list. Now you can assign
> tickets to yourself.
> 
> Please check this page out for commonly asked questions pertaining to
> the contribution process https://ignite.apache.org/community/contribute.html
> 
> Do not hesitate to ask if you need any assistance. Looking forward for
> your contributions!
> 
> 2020-06-02 5:56 GMT+03:00, phani adusumilli :
>> Hello,
>> I am Phani, currently working as an Engineer; interested in distributed
>> systems, caches, persistent stores. I am looking forward to be part of this
>> wonderful Ignite community and contribute. My jira user: psadusumilli.
>> Please let me know if you need any information to add me to the group.
>> 
>> --
>> Thank you
>> Phani S Adusumilli
>> 
> 
> 
> -- 
> 
> Best regards,
> Ivan Pavlukhin

Re: [DISCUSSION] IEP-47 Native persistence defragmentation

[Ivan Bessonov]

Hello Anton,

I'd like to address your last message. First of all, it was already
partially discussed
in this thread: [1] To reiterate - expected performance degradation will be
significant.
There's no way that we can throttle it because free/reuse lists have to be
maintained
sorted all the time. And these are very optimized data structures.

More then that, "dummy" updates clash with data access, this is a very
dangerous
thing to do. And these updates don't save you from the situation when last
pages in
the file are not data pages, but tree pages, for example. They are much
harder to
move. Not only you should update all links to it but also do it
effectively, without
blocking the tree too much. I can think of many other examples.

*Easy to implement/understand*
 - no, it's not easy at all, defragmentation under the load is a very
challenging thing to
   implement.

*Why we're going to implement distributed system defragmentation in the old
(offline) way?*
 - because it's easier and safer, and it won't introduce any performance
degradation.

[1]
http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html

вт, 2 июн. 2020 г. в 14:17, Anton Vinogradov :

> Folks,
>
> Modern OS never ask you to schedule defragmentation and turn your PC off,
> it performs it while you're browsing.
> Why we're going to implement distributed system defragmentation in the old
> (offline) way?
>
> All you need is to implement free/reuse-list sorting. They should provide
> pages closest to the file beginning.
> So, every insert/update will automatically defragment the entry.
> Also, a special process should iterate over the partitions in a reverse way
> just performing dummy updates.
> The partition file may be safely truncated after the iterator.
>
> Props:
> - Your system still operating (no downtime)
> - Defragmentation can be performed partially
> - Defragmentation can be scheduled to periods of inactivity or performed on
> a regular basis
> - SQL will not be broken (no reason to recalculate the whole index, it will
> be recalculated in a regular way on every entry update)
> - Topology changes allowed
> - Easy to implement/understand
>
> Cons:
> - Performance degradation (solvable by throttling)
>
> On Mon, Jun 1, 2020 at 4:04 PM Sergey Chugunov 
> wrote:
>
> > Hi Ivan,
> >
> > I have an idea about suggested maintenance mode.
> >
> > First of all, I agree with your ideas about discovery restrictions: node
> > should not join topology when performing defragmentation.
> >
> > At the same time I haven't heard about requests for this mode from users,
> > so we don't know much about possible requirements.
> > So I suggest to implement it in a pragmatical way: instead of inventing
> > (unknown in reality) user scenarios lets develop minimal but yet
> > well-designed functionality that suites our case. If we constrain our
> > implementation with reasonable set of restrictions that's OK.
> >
> > So my idea is the following: to transit a node to maintenance user has to
> > send special command to the node (e.g. with new command in control.sh
> > utility or via JMX interface). Node saves maintenance request in local
> > metastorage and waits for restart. User has to manually restart that node
> > in order to finish moving it to maintenance mode.
> >
> > When node restarts and finds maintenance request it creates special type
> of
> > discovery SPI that will not try to join topology at all yet node is able
> to
> > start all necessary components and APIs like REST processor or JMX
> > interface.
> >
> > When in maintenance, we'll be able to do defragmentation safely and
> remove
> > maintenance request from metastorage only when it is completed (with all
> > fault-tolerance logic in mind).
> >
> > As we don't have a mechanism (like watcher) to perform a "safe restart"
> (by
> > safe I mean Ignite restart without OS-level process restart) we cannot
> > finish maintenance mode without another manual restart but I think it is
> a
> > reasonable restriction as maintenance mode shouldn't be an every-day
> > routine and will be used quite rare.
> >
> > What do you think about it?
> >
> > On Tue, May 26, 2020 at 5:58 PM Ivan Bessonov 
> > wrote:
> >
> > > Hello Igniters,
> > >
> > > I'd like to discuss this new IEP with you: [1]. The main idea is to
> have
> > a
> > > procedure that relocates
> > > pages to the top of the file as compact as possible which allows us to
> > > trim the file and increase its
> > > fill-factor. It will be configured manually and executed after the
> > restart,
> > > but before node joins
> > > topology (it means any load would be impossible during
> defragmentation).
> > It
> > > is described in detail
> > > in the IEP itself, please read it. This topic was also previously
> > discussed
> > > here on dev-list in [2].
> > >
> > > Here I would like to list a few moments that are not as clear and
> require
> > > your opinion.
> 

Re: Ignite and Java 9+

[steve.hostett...@gmail.com]

Indeed, I misread the code and it already saves "str4e" on 5 bytes and
"str4€" on 7 bytes.
 Sorry for the waste of time.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: [DISCUSSION] IEP-47 Native persistence defragmentation

[Anton Vinogradov]

Folks,

Modern OS never ask you to schedule defragmentation and turn your PC off,
it performs it while you're browsing.
Why we're going to implement distributed system defragmentation in the old
(offline) way?

All you need is to implement free/reuse-list sorting. They should provide
pages closest to the file beginning.
So, every insert/update will automatically defragment the entry.
Also, a special process should iterate over the partitions in a reverse way
just performing dummy updates.
The partition file may be safely truncated after the iterator.

Props:
- Your system still operating (no downtime)
- Defragmentation can be performed partially
- Defragmentation can be scheduled to periods of inactivity or performed on
a regular basis
- SQL will not be broken (no reason to recalculate the whole index, it will
be recalculated in a regular way on every entry update)
- Topology changes allowed
- Easy to implement/understand

Cons:
- Performance degradation (solvable by throttling)

On Mon, Jun 1, 2020 at 4:04 PM Sergey Chugunov 
wrote:

> Hi Ivan,
>
> I have an idea about suggested maintenance mode.
>
> First of all, I agree with your ideas about discovery restrictions: node
> should not join topology when performing defragmentation.
>
> At the same time I haven't heard about requests for this mode from users,
> so we don't know much about possible requirements.
> So I suggest to implement it in a pragmatical way: instead of inventing
> (unknown in reality) user scenarios lets develop minimal but yet
> well-designed functionality that suites our case. If we constrain our
> implementation with reasonable set of restrictions that's OK.
>
> So my idea is the following: to transit a node to maintenance user has to
> send special command to the node (e.g. with new command in control.sh
> utility or via JMX interface). Node saves maintenance request in local
> metastorage and waits for restart. User has to manually restart that node
> in order to finish moving it to maintenance mode.
>
> When node restarts and finds maintenance request it creates special type of
> discovery SPI that will not try to join topology at all yet node is able to
> start all necessary components and APIs like REST processor or JMX
> interface.
>
> When in maintenance, we'll be able to do defragmentation safely and remove
> maintenance request from metastorage only when it is completed (with all
> fault-tolerance logic in mind).
>
> As we don't have a mechanism (like watcher) to perform a "safe restart" (by
> safe I mean Ignite restart without OS-level process restart) we cannot
> finish maintenance mode without another manual restart but I think it is a
> reasonable restriction as maintenance mode shouldn't be an every-day
> routine and will be used quite rare.
>
> What do you think about it?
>
> On Tue, May 26, 2020 at 5:58 PM Ivan Bessonov 
> wrote:
>
> > Hello Igniters,
> >
> > I'd like to discuss this new IEP with you: [1]. The main idea is to have
> a
> > procedure that relocates
> > pages to the top of the file as compact as possible which allows us to
> > trim the file and increase its
> > fill-factor. It will be configured manually and executed after the
> restart,
> > but before node joins
> > topology (it means any load would be impossible during defragmentation).
> It
> > is described in detail
> > in the IEP itself, please read it. This topic was also previously
> discussed
> > here on dev-list in [2].
> >
> > Here I would like to list a few moments that are not as clear and require
> > your opinion.
> >
> >  - what are your overall thoughts on the IEP? Any concerns?
> >
> >  - maintenance mode - how do we communicate with the node that's not in
> > topology? What are
> >the options? As far as I know, we have no current tools like this.
> >
> >  - checkpointer refactoring - these changes will involve intensive
> writing
> > of pages to the storage.
> >If we're going to reuse the offheap page model to perform
> > defragmentation then the
> >checkpointing mechanism will have to be adapted in some form.
> >Are you fine with this? Or we need a separate discussion?
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
> > [2]
> >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html
> >
> >
> > --
> > Sincerely yours,
> > Ivan Bessonov
> >
>

Re: [DISCUSS] Ignite process exit code on node stop by failure handler

[Sergey Antonov]

Alexey,

> How exactly do you want to change the StopNodeFH?
I want to stop JVM with KILL_EXIT_CODE and add an option (constructor
argument of JVM option) for disabling JVM termination.

вт, 2 июн. 2020 г. в 12:54, Alexey Goncharuk :

> Sergey,
>
> How exactly do you want to change the StopNodeFH? The current behavior does
> not terminate the JVM and its exit code is totally out of our control; one
> of the use-cases we had in mind for this failure handler is that a user may
> have other processes running in the same JVM, so we do not want Ignite to
> affect them.
>
> For me, the exit code makes sense only for StopNodeOrHaltFH with
> tryStop=false (otherwise, the JVM exit is not guaranteed as well), but we
> already use the KILL_EXIT_CODE there.
>
> пн, 1 июн. 2020 г. в 23:19, Sergey Antonov :
>
> > Hello, Kirill!
> >
> > I'd prefer to don't create a new implementation of a failure handler. We
> > already have 4 different failure handlers. We will have 6 FH (StopNodeFH
> > with exit code, StopNodeOrHaltFH with exit code), if we go your way. It
> > won't make our product simpler and easier.
> >
> > I think, we must notify a user if the cluster node had been stopped by a
> > failure handler. We can't achieve this goal without changing current FH
> > behavior. So I propose to change it and stop the process with
> > KILL_EXIT_CODE.
> > But it would be nice if users will have a flag for avoiding process stop
> > after a node failure. We can introduce a new JVM option or FH parameter
> for
> > that reason. Of course, we must highlight this change in the release
> notes.
> >
> > пн, 1 июн. 2020 г. в 19:07, ткаленко кирилл :
> >
> > > I think that [1] and [2] should not be changed, because we can kill
> > > another client code in this jvm.
> > > I suggest for these purposes to create a new [3] which will be like [1]
> > > but with a call [4] after node stop.
> > > Objections or comments?
> > >
> > > [1] - org.apache.ignite.failure.StopNodeFailureHandler
> > > [2] - org.apache.ignite.failure.StopNodeOrHaltFailureHandler
> > > [3] - org.apache.ignite.failure.FailureHandler
> > > [4] - java.lang.System#exit
> > >
> > > 25.05.2020, 22:09, "Dmitriy Pavlov" :
> > > > It seems reasonable to me. Also I would like to propose adding value
> of
> > > > Ignition.KILL_EXIT_CODE into javadoc using @value javadoc tag.
> > > >
> > > > Dev ops/admins/anyone who admins Ignite may want to know it's value
> > > without
> > > > going to Java code.
> > > >
> > > > чт, 21 мая 2020 г. в 09:39, Zhenya Stanilovsky
> > > :
> > > >
> > > >>  Thank you Sergey, as for me — very useful proposal huge +1 here.
> > > >>
> > > >>  >Четверг, 21 мая 2020, 0:51 +03:00 от Sergey Antonov <
> > > >>  antonovserge...@gmail.com>:
> > > >>  >
> > > >>  >I've created the Ignite ticket for this improvement [1].
> > > >>  >
> > > >>  >[1] https://issues.apache.org/jira/browse/IGNITE-13047
> > > >>  >
> > > >>  >чт, 21 мая 2020 г. в 00:46, Sergey Antonov <
> > > antonovserge...@gmail.com >:
> > > >>  >
> > > >>  >> Hello, Igniters!
> > > >>  >>
> > > >>  >> I'd like to discuss behaviour of Ignite process exit code if the
> > > node
> > > >>  was
> > > >>  >> stopped by failure handler [1][2]. At the moment ignite process
> > > returns
> > > >>  >> exit code 0 after the stop in all scenarios, except runtime halt
> > by
> > > >>  >> StopNodeOrHaltFH [1]. In this case, the exit code will be 130
> [3]
> > > >>  >>
> > > >>  >> My proposal: always finish Ignite process with code [3], if a
> node
> > > was
> > > >>  >> stopped by FH. It could be helpful for administration purposes,
> > you
> > > can
> > > >>  >> distinguish normal node stop from node stop by FH on OS level.
> > > >>  >>
> > > >>  >> WDYT?
> > > >>  >>
> > > >>  >> [1]
> > > >>  >>
> > > >>
> > >
> >
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/failure/StopNodeOrHaltFailureHandler.html
> > > >>  >>
> > > >>  >> [2]
> > > >>  >>
> > > >>
> > >
> >
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/failure/StopNodeFailureHandler.html
> > > >>  >>
> > > >>  >> [3]
> > > >>  >>
> > > >>
> > >
> >
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/Ignition.html#KILL_EXIT_CODE
> > > >>  >> --
> > > >>  >> BR, Sergey Antonov
> > > >>  >>
> > > >>  >
> > > >>  >--
> > > >>  >BR, Sergey Antonov
> > > >>  >
> > >
> >
> >
> > --
> > BR, Sergey Antonov
> >
>


-- 
BR, Sergey Antonov

Re: Ignite and Java 9+

[Ivan Pavlukhin]

A note. I assumed that we talk about BinaryWriterExImpl.doWriteString.

2020-06-02 12:52 GMT+03:00, Ivan Pavlukhin :
> Hi Steve,
>
> Could you please describe in more details in what cases do you expect
> a profit? AFAIK UTF-8 requires 1 byte for Latin-1 characters.
>
> I did not learn in depth about aforementioned Java 9 compact strings
> but my understanding is that it is about reducing space taken by
> java.lang.String objects in a Java heap memory.
>
> 2020-06-02 11:37 GMT+03:00, steve.hostett...@gmail.com
> :
>> Hello,
>>
>> The method doWriteString(@Nullable String val) writes everything in
>> UTF-8.
>> This is a bit a waste because with  Java 9+
>>   , strings are optimised and if all
>> chars are ISO-LATIN-1 all char are represented with 1 byte instead of
>> two.
>> I
>> would consider creating 2 type of strings : One with type 0x9 that
>> behaves
>> as it does currently and another  0x39 that checks that
>> s.getBytes().length
>> == s.length()
>>
>> what do you think?
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>>
>
>
> --
>
> Best regards,
> Ivan Pavlukhin
>


-- 

Best regards,
Ivan Pavlukhin

Re: [DISCUSS] Ignite process exit code on node stop by failure handler

[Alexey Goncharuk]

Sergey,

How exactly do you want to change the StopNodeFH? The current behavior does
not terminate the JVM and its exit code is totally out of our control; one
of the use-cases we had in mind for this failure handler is that a user may
have other processes running in the same JVM, so we do not want Ignite to
affect them.

For me, the exit code makes sense only for StopNodeOrHaltFH with
tryStop=false (otherwise, the JVM exit is not guaranteed as well), but we
already use the KILL_EXIT_CODE there.

пн, 1 июн. 2020 г. в 23:19, Sergey Antonov :

> Hello, Kirill!
>
> I'd prefer to don't create a new implementation of a failure handler. We
> already have 4 different failure handlers. We will have 6 FH (StopNodeFH
> with exit code, StopNodeOrHaltFH with exit code), if we go your way. It
> won't make our product simpler and easier.
>
> I think, we must notify a user if the cluster node had been stopped by a
> failure handler. We can't achieve this goal without changing current FH
> behavior. So I propose to change it and stop the process with
> KILL_EXIT_CODE.
> But it would be nice if users will have a flag for avoiding process stop
> after a node failure. We can introduce a new JVM option or FH parameter for
> that reason. Of course, we must highlight this change in the release notes.
>
> пн, 1 июн. 2020 г. в 19:07, ткаленко кирилл :
>
> > I think that [1] and [2] should not be changed, because we can kill
> > another client code in this jvm.
> > I suggest for these purposes to create a new [3] which will be like [1]
> > but with a call [4] after node stop.
> > Objections or comments?
> >
> > [1] - org.apache.ignite.failure.StopNodeFailureHandler
> > [2] - org.apache.ignite.failure.StopNodeOrHaltFailureHandler
> > [3] - org.apache.ignite.failure.FailureHandler
> > [4] - java.lang.System#exit
> >
> > 25.05.2020, 22:09, "Dmitriy Pavlov" :
> > > It seems reasonable to me. Also I would like to propose adding value of
> > > Ignition.KILL_EXIT_CODE into javadoc using @value javadoc tag.
> > >
> > > Dev ops/admins/anyone who admins Ignite may want to know it's value
> > without
> > > going to Java code.
> > >
> > > чт, 21 мая 2020 г. в 09:39, Zhenya Stanilovsky
> > :
> > >
> > >>  Thank you Sergey, as for me — very useful proposal huge +1 here.
> > >>
> > >>  >Четверг, 21 мая 2020, 0:51 +03:00 от Sergey Antonov <
> > >>  antonovserge...@gmail.com>:
> > >>  >
> > >>  >I've created the Ignite ticket for this improvement [1].
> > >>  >
> > >>  >[1] https://issues.apache.org/jira/browse/IGNITE-13047
> > >>  >
> > >>  >чт, 21 мая 2020 г. в 00:46, Sergey Antonov <
> > antonovserge...@gmail.com >:
> > >>  >
> > >>  >> Hello, Igniters!
> > >>  >>
> > >>  >> I'd like to discuss behaviour of Ignite process exit code if the
> > node
> > >>  was
> > >>  >> stopped by failure handler [1][2]. At the moment ignite process
> > returns
> > >>  >> exit code 0 after the stop in all scenarios, except runtime halt
> by
> > >>  >> StopNodeOrHaltFH [1]. In this case, the exit code will be 130 [3]
> > >>  >>
> > >>  >> My proposal: always finish Ignite process with code [3], if a node
> > was
> > >>  >> stopped by FH. It could be helpful for administration purposes,
> you
> > can
> > >>  >> distinguish normal node stop from node stop by FH on OS level.
> > >>  >>
> > >>  >> WDYT?
> > >>  >>
> > >>  >> [1]
> > >>  >>
> > >>
> >
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/failure/StopNodeOrHaltFailureHandler.html
> > >>  >>
> > >>  >> [2]
> > >>  >>
> > >>
> >
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/failure/StopNodeFailureHandler.html
> > >>  >>
> > >>  >> [3]
> > >>  >>
> > >>
> >
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/Ignition.html#KILL_EXIT_CODE
> > >>  >> --
> > >>  >> BR, Sergey Antonov
> > >>  >>
> > >>  >
> > >>  >--
> > >>  >BR, Sergey Antonov
> > >>  >
> >
>
>
> --
> BR, Sergey Antonov
>

Re: Ignite and Java 9+

[Ivan Pavlukhin]

Hi Steve,

Could you please describe in more details in what cases do you expect
a profit? AFAIK UTF-8 requires 1 byte for Latin-1 characters.

I did not learn in depth about aforementioned Java 9 compact strings
but my understanding is that it is about reducing space taken by
java.lang.String objects in a Java heap memory.

2020-06-02 11:37 GMT+03:00, steve.hostett...@gmail.com
:
> Hello,
>
> The method doWriteString(@Nullable String val) writes everything in UTF-8.
> This is a bit a waste because with  Java 9+
>   , strings are optimised and if all
> chars are ISO-LATIN-1 all char are represented with 1 byte instead of two.
> I
> would consider creating 2 type of strings : One with type 0x9 that behaves
> as it does currently and another  0x39 that checks that s.getBytes().length
> == s.length()
>
> what do you think?
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>


-- 

Best regards,
Ivan Pavlukhin

Re: Phani Introduction

[Ivan Pavlukhin]

Hi Phani,

Welcome to Apache Ignite Community!

I added your account to the contributors list. Now you can assign
tickets to yourself.

Please check this page out for commonly asked questions pertaining to
the contribution process https://ignite.apache.org/community/contribute.html

Do not hesitate to ask if you need any assistance. Looking forward for
your contributions!

2020-06-02 5:56 GMT+03:00, phani adusumilli :
> Hello,
> I am Phani, currently working as an Engineer; interested in distributed
> systems, caches, persistent stores. I am looking forward to be part of this
> wonderful Ignite community and contribute. My jira user: psadusumilli.
> Please let me know if you need any information to add me to the group.
>
> --
> Thank you
> Phani S Adusumilli
>


-- 

Best regards,
Ivan Pavlukhin

Ignite and Java 9+

[steve.hostett...@gmail.com]

Hello,

The method doWriteString(@Nullable String val) writes everything in UTF-8.
This is a bit a waste because with  Java 9+
  , strings are optimised and if all
chars are ISO-LATIN-1 all char are represented with 1 byte instead of two. I
would consider creating 2 type of strings : One with type 0x9 that behaves
as it does currently and another  0x39 that checks that s.getBytes().length
== s.length()

what do you think?



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/