Re: Threadpools and .WithExecute() for C# clients

[Raymond Wilson]

The source of inbound requests into Server A is from client applications.

Server B is really a cluster of servers that are performing clustered
transformations and computations across a data set.

I originally used IComputeJob and similar functions which work very well
but have the restriction that they return the entire result set from a
Server B node in a single response. These result sets can be large (100's
of megabytes and larger), which makes life pretty hard for Server A if it
has to field multiple incoming responses of this size. So, these types of
requests progressively send responses back (using Ignite messaging) to
Server A using the Ignite messaging fabric. As Server A receives each part
of the overall response it processes it according the business rules
relevant to the request.

The cluster config and numbers of nodes are not really material to this.

Raymond.

On Thu, Jul 18, 2019 at 12:26 AM Alexandr Shapkin  wrote:

> Hi,
>
>
>
> Can you share a more detailed use case, please?
>
>
>
> Right now it’s not clear why do you need a messaging fabric.
>
> If you are interesting in a progress tracking, then you could try a
> CacheAPI or QueryContinious, for example.
>
>
>
> What are the sources of inbound requests? Is it a client requests?
>
>
>
> What is your cluster config? How many nodes do you have for your
> distributed computations?
>
>
>
> *From: *Raymond Wilson 
> *Sent: *Wednesday, July 17, 2019 1:49 PM
> *To: *user 
> *Subject: *Re: Threadpools and .WithExecute() for C# clients
>
>
>
> Hi Alexandr,
>
>
>
> To summarise from the original thread, say I have server A that accepts
> requests. It contacts server B in order to help processing those requests.
> Server B sends in-progress results to server A using the Ignite messaging
> fabric. If the thread pool in server A is saturated with inbound requests,
> then there are no available threads to service the messaging fabric traffic
> from server B to server A resulting in a deadlock condition.
>
>
>
> In the original discussion it was suggested creating a custom thread pool
> to handle the Server B to Server A traffic would resolve it.
>
>
>
> Thanks,
>
> Raymond.
>
>
>
> On Wed, Jul 17, 2019 at 9:48 PM Alexandr Shapkin 
> wrote:
>
> Hi, Raymond!
>
>
>
> As far as I can see, there are no plans for porting custom executors
> configuration in .NET client right now [1].
>
>
>
> Please, remind, why do you need a separate pool instead of a default
> PublicPool?
>
>
>
> [1] - https://issues.apache.org/jira/browse/IGNITE-6566
>
>
>
>
>
>
>
> *From: *Raymond Wilson 
> *Sent: *Wednesday, July 17, 2019 10:58 AM
> *To: *user 
> *Subject: *Threadpools and .WithExecute() for C# clients
>
>
>
> Some time ago I ran into and issue with thread pool exhaustion and
> deadlocking in AI 2.2.
>
>
>
> This is the original thread:
> http://apache-ignite-users.70518.x6.nabble.com/Possible-dead-lock-when-number-of-jobs-exceeds-thread-pool-td17262.html
>
>
>
>
> At the time .WithExecutor() was not implemented in the C# client so there
> was little option but to expand the size of the public thread pool
> sufficiently to prevent the deadlocking.
>
>
>
> We have been revisiting this issue and see that .WithExecutor() is not
> supported in the AI 2.7.5 client.
>
>
>
> Can this be supported in the C# client, or is there a workaround in the
> .Net environment? that does not require this capability?
>
>
>
> Thanks,
>
> Raymond.
>
>
>
>
>
>
>

Re: CacheEntryProcessor causes cluster node to stop

[ihalilaltun]

Hi Vladimir,

here is logs from other node
ignite-3.zip
  



-
İbrahim Halil Altun
Senior Software Engineer @ Segmentify
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Random CorruptedTreeException from Apache Ignite

[ihalilaltun]

Hi Maxim,

we are facing the exact same problem :(

Is it ok/safe to remove cacheGroupName from the code that is already been
created on nodes? If so, when we start our applications from the updated
code will we be still have access to same caches or new caches will be
created on nodes?



-
İbrahim Halil Altun
Senior Software Engineer @ Segmentify
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: CacheEntryProcessor causes cluster node to stop

[Vladimir Pligin]

Hi,

It seems that the root cause of the issue happens on node with
id=fb70df1c-0311-48b4-bb33-88cdf85438ce.
That node becomes unresponsive. Could you please share logs from that node,
it could help?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: Threadpools and .WithExecute() for C# clients

[Alexandr Shapkin]

Hi,

Can you share a more detailed use case, please? 

Right now it’s not clear why do you need a messaging fabric.
If you are interesting in a progress tracking, then you could try a CacheAPI or 
QueryContinious, for example.

What are the sources of inbound requests? Is it a client requests?

What is your cluster config? How many nodes do you have for your distributed 
computations?

From: Raymond Wilson
Sent: Wednesday, July 17, 2019 1:49 PM
To: user
Subject: Re: Threadpools and .WithExecute() for C# clients

Hi Alexandr,

To summarise from the original thread, say I have server A that accepts 
requests. It contacts server B in order to help processing those requests. 
Server B sends in-progress results to server A using the Ignite messaging 
fabric. If the thread pool in server A is saturated with inbound requests, then 
there are no available threads to service the messaging fabric traffic from 
server B to server A resulting in a deadlock condition.

In the original discussion it was suggested creating a custom thread pool to 
handle the Server B to Server A traffic would resolve it.

Thanks,
Raymond.

On Wed, Jul 17, 2019 at 9:48 PM Alexandr Shapkin  wrote:
Hi, Raymond!
 
As far as I can see, there are no plans for porting custom executors 
configuration in .NET client right now [1].
 
Please, remind, why do you need a separate pool instead of a default PublicPool?
 
[1] - https://issues.apache.org/jira/browse/IGNITE-6566
 
 
 
From: Raymond Wilson
Sent: Wednesday, July 17, 2019 10:58 AM
To: user
Subject: Threadpools and .WithExecute() for C# clients
 
Some time ago I ran into and issue with thread pool exhaustion and deadlocking 
in AI 2.2.
 
This is the original thread: 
http://apache-ignite-users.70518.x6.nabble.com/Possible-dead-lock-when-number-of-jobs-exceeds-thread-pool-td17262.html
  
 
At the time .WithExecutor() was not implemented in the C# client so there was 
little option but to expand the size of the public thread pool sufficiently to 
prevent the deadlocking.
 
We have been revisiting this issue and see that .WithExecutor() is not 
supported in the AI 2.7.5 client.
 
Can this be supported in the C# client, or is there a workaround in the .Net 
environment? that does not require this capability?
 
Thanks,
Raymond.
 
 

Re: Metrics for Ignite

[Anton Kurbanov]

Hello,

I would suggest to refer to following metrics and use MX bean:
https://apacheignite.readme.io/docs/cache-metrics, there are average
metrics for get/put/remove.

I'm not entirely sure if you really do want to have some internal
benchmarking for each operation, how big are workloads going to be in your
use-case? Would some generic implementation that is based on cache metrics
MX bean work for you as a solution here? Еhis is necessary to hold timings
for each single operation in the metrics as obviously it will require some
configuration to come up with like how many operations statistics to store,
when they should be dropped, when to reset etc.

The quickest thing to do is to wrap calls to these operations with time
measurements and do with that information any custom things you want to.

And if I've understood it wrong way, you can simply write your own
benchmarks and execute them on your environment within benchmark framework
that Ignite is shipped with:
https://github.com/apache/ignite/tree/master/modules/yardstick

Refer to DEVNOTES.txt file in this module to mvn line to execute and
directory to run and find benchmark config samples, it can warmup, gather
statistics and plot graphs (refer to short yardstick doc:
https://github.com/gridgain/yardstick/blob/master/README.md,
jfreechart-graph-gen.sh script to plot graphs from benchmarks output
directory is available).

пн, 15 июл. 2019 г. в 17:16, Andrei Aleksandrov :

> Hi,
>
> You can try to use the web console for Ignite. It contains the Monitoring
> Dashboard with different cache metrics that were calculated during some
> period of time. Also, it contains different graphics related to cache
> operation throughput and latency:
>
> https://apacheignite-tools.readme.io/docs/ignite-web-console
>
> You can try to use the next "ready to go" installation for testing:
>
> https://console.gridgain.com/monitoring/dashboard
>
> BR,
> Andrei
>
> On 2019/07/15 13:20:09, nikhil dhiman  
> wrote:
> > Hi, I am almost production ready, but I want to plot graphs for>
> > Throughput, latency operation wise[get, put, delete, eviction]. Is>
> > there a way I can produce metrics for time taken by ignite node for>
> > get/put/delete. I can see many metrics via rest module. But i am>
> > unable to find the above metrics.>
> >
> > I am on Ignite 2.7.5 version.>
> >
> > Thanks & Regards,>
> > Nikhil Dhiman>
> >
>

CacheEntryProcessor causes cluster node to stop

[ihalilaltun]

Hi Igniters,

Although class was deployed in SHARED or CONTINUOUS mode node got exception
and halted itself.

log added;
ignite.zip
  


*Ignite version*: 2.7.5 
*Cluster size*: 16 
*Client size*: 22 
*Cluster OS version*: Centos 7 
*Cluster Kernel version*: 4.4.185-1.el7.elrepo.x86_64 
*Java version* : 
java version "1.8.0_201" 
Java(TM) SE Runtime Environment (build 1.8.0_201-b09) 
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode) 



-
İbrahim Halil Altun
Senior Software Engineer @ Segmentify
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Log4j2 problem

[Mikael]

Hi!

I have setup the log4j2 logger and everything works fine until I 
shutdown the application (2.7.5), I run it with ignite.bat and I have my 
own code in a LifeCycleBean, I have some code in beforeNodeStop() and 
some code in afterNodeStop() and it looks like the log4j2 logger stops 
working between beforeNodeStop and afterNodeStop, does Ignite shutdown 
the logger before afterNodeStop is called or something like that ?


I get all logging information from the code in beforeNodeStop() but any 
logging calls in afterNodeStop do nothing at all.

NegativeArraySizeException on cluster rebalance

[ihalilaltun]

Hi Igniters,

We are getting negativearraysizeexception on one of our cacheGroups
rebalancing period. I am adding the details that I could get from logs;

*Ignite version*: 2.7.5 
*Cluster size*: 16 
*Client size*: 22 
*Cluster OS version*: Centos 7 
*Cluster Kernel version*: 4.4.185-1.el7.elrepo.x86_64 
*Java version* : 
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)


Pojo we use;
UserPriceDropRecord.txt

  

Error we got;

[2019-07-17T10:23:13,162][ERROR][sys-#241][GridDhtPartitionSupplier] Failed
to continue supplying [grp=userPriceDropDataCacheGroup,
demander=12d8bad8-62a9-465d-aca4-4afa203d6778,
topVer=AffinityTopologyVersion [topVer=238, minorTopVer=0], topic=1]
java.lang.NegativeArraySizeException: null
at 
org.apache.ignite.internal.pagemem.PageUtils.getBytes(PageUtils.java:63)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFullRow(CacheDataRowAdapter.java:330)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:167)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:108)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.tree.DataRow.(DataRow.java:55)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:92)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:200)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:49)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5512)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5280)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.nextPage(BPlusTree.java:5332)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.next(BPlusTree.java:5566)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$9.onHasNext(IgniteCacheOffheapManagerImpl.java:1185)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteRebalanceIteratorImpl.advance(IgniteRebalanceIteratorImpl.java:79)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteRebalanceIteratorImpl.nextX(IgniteRebalanceIteratorImpl.java:139)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteRebalanceIteratorImpl.next(IgniteRebalanceIteratorImpl.java:185)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteRebalanceIteratorImpl.next(IgniteRebalanceIteratorImpl.java:37)
~[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionSupplier.handleDemandMessage(GridDhtPartitionSupplier.java:333)
[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader.handleDemandMessage(GridDhtPreloader.java:404)
[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:424)
[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$5.apply(GridCachePartitionExchangeManager.java:409)
[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
[ignite-core-2.7.5.jar:2.7.5]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$700(GridCacheIoManager.java:101)
[ignite-core-2.7.5.jar:2.7.5]
at

Re: Threadpools and .WithExecute() for C# clients

[Raymond Wilson]

Hi Alexandr,

To summarise from the original thread, say I have server A that accepts
requests. It contacts server B in order to help processing those requests.
Server B sends in-progress results to server A using the Ignite messaging
fabric. If the thread pool in server A is saturated with inbound requests,
then there are no available threads to service the messaging fabric traffic
from server B to server A resulting in a deadlock condition.

In the original discussion it was suggested creating a custom thread pool
to handle the Server B to Server A traffic would resolve it.

Thanks,
Raymond.

On Wed, Jul 17, 2019 at 9:48 PM Alexandr Shapkin  wrote:

> Hi, Raymond!
>
>
>
> As far as I can see, there are no plans for porting custom executors
> configuration in .NET client right now [1].
>
>
>
> Please, remind, why do you need a separate pool instead of a default
> PublicPool?
>
>
>
> [1] - https://issues.apache.org/jira/browse/IGNITE-6566
>
>
>
>
>
>
>
> *From: *Raymond Wilson 
> *Sent: *Wednesday, July 17, 2019 10:58 AM
> *To: *user 
> *Subject: *Threadpools and .WithExecute() for C# clients
>
>
>
> Some time ago I ran into and issue with thread pool exhaustion and
> deadlocking in AI 2.2.
>
>
>
> This is the original thread:
> http://apache-ignite-users.70518.x6.nabble.com/Possible-dead-lock-when-number-of-jobs-exceeds-thread-pool-td17262.html
>
>
>
>
> At the time .WithExecutor() was not implemented in the C# client so there
> was little option but to expand the size of the public thread pool
> sufficiently to prevent the deadlocking.
>
>
>
> We have been revisiting this issue and see that .WithExecutor() is not
> supported in the AI 2.7.5 client.
>
>
>
> Can this be supported in the C# client, or is there a workaround in the
> .Net environment? that does not require this capability?
>
>
>
> Thanks,
>
> Raymond.
>
>
>
>
>

RE: Threadpools and .WithExecute() for C# clients

[Alexandr Shapkin]

Hi, Raymond!

As far as I can see, there are no plans for porting custom executors 
configuration in .NET client right now [1].

Please, remind, why do you need a separate pool instead of a default PublicPool?

[1] - https://issues.apache.org/jira/browse/IGNITE-6566



From: Raymond Wilson
Sent: Wednesday, July 17, 2019 10:58 AM
To: user
Subject: Threadpools and .WithExecute() for C# clients

Some time ago I ran into and issue with thread pool exhaustion and deadlocking 
in AI 2.2.

This is the original thread: 
http://apache-ignite-users.70518.x6.nabble.com/Possible-dead-lock-when-number-of-jobs-exceeds-thread-pool-td17262.html
  

At the time .WithExecutor() was not implemented in the C# client so there was 
little option but to expand the size of the public thread pool sufficiently to 
prevent the deadlocking.

We have been revisiting this issue and see that .WithExecutor() is not 
supported in the AI 2.7.5 client.

Can this be supported in the C# client, or is there a workaround in the .Net 
environment? that does not require this capability?

Thanks,
Raymond.

Re: Declaring server side CacheEntryListenerin Ignte config

[Andrei Aleksandrov]

Hi,

Could you please provide more details about your case?

Generally, for tracking of the cache updates in Ignite you can use Events:

https://apacheignite.readme.io/docs/events

Continuous Queries:

https://apacheignite.readme.io/docs/continuous-queries

CacheInterceptor:

https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/CacheInterceptor.html

BR,
Andrei

http://apache-ignite-users.70518.x6.nabble.com/Declaring-server-side-CacheEntryListener-in-Ignite-config-td28849.html

On 2019/07/16 13:33:54, Jean-Philippe Laroche  wrote:
> I saw many examples on how to programmatically declare a>
> CacheEntryListenerfrom a client application, but is there a way to 
register>

> by configuration, a CacheEntryListener so it is active on node/cluster>
> startup?>
>

Threadpools and .WithExecute() for C# clients

[Raymond Wilson]

Some time ago I ran into and issue with thread pool exhaustion and
deadlocking in AI 2.2.

This is the original thread:
http://apache-ignite-users.70518.x6.nabble.com/Possible-dead-lock-when-number-of-jobs-exceeds-thread-pool-td17262.html


At the time .WithExecutor() was not implemented in the C# client so there
was little option but to expand the size of the public thread pool
sufficiently to prevent the deadlocking.

We have been revisiting this issue and see that .WithExecutor() is not
supported in the AI 2.7.5 client.

Can this be supported in the C# client, or is there a workaround in the
.Net environment? that does not require this capability?

Thanks,
Raymond.