High cpu on ignite server nodes

2018-06-16 Thread praveeng 
Hi ,

We are observing one issue recently that on ignite servers is 100% CPU 
utilization.
ignite version 1.8, 
server config: RAM 16GB, Cores : 8 
please find a thread dump and cache xml and can you please check and suggest
if there is any issue.

cai003_june14_tdump.zip

  

  ignite-cl-cache-config.xml

  

Thanks,
Praveen 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: High cpu on ignite server nodes

2018-06-16 Thread praveeng 
Hi ,

Following are the high cpu consumed threads.

  PID USER  PR  NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
36708 gmedia20   0 11.031g 6.749g  14720 R 53.5 43.5 267:15.83 java
36709 gmedia20   0 11.031g 6.749g  14720 R 51.5 43.5 266:53.53 java
34433 gmedia20   0 11.031g 6.749g  14720 R 48.8 43.5 268:50.46 java
35687 gmedia20   0 11.031g 6.749g  14720 R 48.8 43.5 270:04.85 java
36706 gmedia20   0 11.031g 6.749g  14720 R 48.8 43.5 266:05.29 java
36712 gmedia20   0 11.031g 6.749g  14720 R 48.5 43.5 268:34.80 java
36713 gmedia20   0 11.031g 6.749g  14720 R 48.5 43.5 270:16.17 java
37366 gmedia20   0 11.031g 6.749g  14720 R 48.5 43.5 269:50.37 java
37367 gmedia20   0 11.031g 6.749g  14720 R 48.5 43.5 267:32.84 java
48957 gmedia20   0 11.031g 6.749g  14720 R 48.2 43.5 266:50.72 java
36707 gmedia20   0 11.031g 6.749g  14720 R 47.8 43.5 268:30.12 java
36714 gmedia20   0 11.031g 6.749g  14720 R 47.5 43.5 266:10.18 java
37811 gmedia20   0 11.031g 6.749g  14720 R 47.5 43.5 267:44.25 java
34438 gmedia20   0 11.031g 6.749g  14720 R 47.2 43.5 269:45.54 java
34439 gmedia20   0 11.031g 6.749g  14720 R 46.8 43.5 268:24.83 java
36710 gmedia20   0 11.031g 6.749g  14720 R 46.5 43.5 269:43.68 java

Thanks,
Praveen



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: How to use Affinity Function to Map a set of Keys to a particularnode in a cluster?

2018-06-16 Thread Stanislav Lukyanov 
Hi,

1. Yes
2. This is controlled by the AffinityFunction::assignPartitions implementation. 
If the default one doesn’t suite you, you have to write another one yourself.

Reading this thread I have a feeling that this is a case of the XY problem 
(https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem).
You’re asking about how to make sure that your keys 1-1 are stored on the 
node 1, but this is not a business problem (this is the Y, not the X).

I believe that 99% of the Ignite’s use cases don’t require to implement the 
AffinityFunction (or even tune the default one).
In these cases, choosing an affinity key for your objects and defining it via 
@AffinityKeyMapped is enough. And a lot of the times you don’t even need the 
affinity key.
The developer normally don’t even need to think about the partition-to-node 
mappings (at least, not from the start).
That’s a part of the beauty of the scalable systems – you may add or remove a 
node and the system will just work, no need to change anything.

Try solving your problem by just defining the @AffinityKeyMapped – it seems 
that this thread has a lot of info on how the affinity collocation works.
If you’re still unsure of how to proceed, I suggest you describe your use case 
in more details.

Thanks,
Stan

From: the_palakkaran
Sent: 13 июня 2018 г. 12:11
To: user@ignite.apache.org
Subject: Re: How to use Affinity Function to Map a set of Keys to a 
particularnode in a cluster?

Thanks for the detailed reply, now everything makes more sense to me.

1.Do I have a control over number of partitions ignite makes? I think this
is done using affinity.setPartitions(5) right? 

2. Can I tell ignite to have a single partition of my cache in each node? [5
nodes => 5 partitions]

Your suggestion method 1 will have worked for me, if I knew in which
partition of which node all entries marked with (int) affinity = 1 --> means
ranging from 1-1. I know about using
CacheConfiguration().getAffinityFunction().partition(Object key), but I
can't use this for each record, I need to know it in prior. And I also need
to know details of partitions against node.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: High cpu on ignite server nodes

2018-06-16 Thread Stanislav Lukyanov 
Hi Praveen,

Is it on all nodes or just one? How many nodes do you have?
Is the cluster fully responsive, or do some operations appear frozen forever?

It looks like a memory/GC issue to me.
jmap shows 2GB of on-heap data. What is the max heap size? Try increasing it. 

Thanks,
Stan

From: praveeng
Sent: 16 июня 2018 г. 17:15
To: user@ignite.apache.org
Subject: Re: High cpu on ignite server nodes

Hi ,

Following are the high cpu consumed threads.

  PID USER  PR  NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
36708 gmedia20   0 11.031g 6.749g  14720 R 53.5 43.5 267:15.83 java
36709 gmedia20   0 11.031g 6.749g  14720 R 51.5 43.5 266:53.53 java
34433 gmedia20   0 11.031g 6.749g  14720 R 48.8 43.5 268:50.46 java
35687 gmedia20   0 11.031g 6.749g  14720 R 48.8 43.5 270:04.85 java
36706 gmedia20   0 11.031g 6.749g  14720 R 48.8 43.5 266:05.29 java
36712 gmedia20   0 11.031g 6.749g  14720 R 48.5 43.5 268:34.80 java
36713 gmedia20   0 11.031g 6.749g  14720 R 48.5 43.5 270:16.17 java
37366 gmedia20   0 11.031g 6.749g  14720 R 48.5 43.5 269:50.37 java
37367 gmedia20   0 11.031g 6.749g  14720 R 48.5 43.5 267:32.84 java
48957 gmedia20   0 11.031g 6.749g  14720 R 48.2 43.5 266:50.72 java
36707 gmedia20   0 11.031g 6.749g  14720 R 47.8 43.5 268:30.12 java
36714 gmedia20   0 11.031g 6.749g  14720 R 47.5 43.5 266:10.18 java
37811 gmedia20   0 11.031g 6.749g  14720 R 47.5 43.5 267:44.25 java
34438 gmedia20   0 11.031g 6.749g  14720 R 47.2 43.5 269:45.54 java
34439 gmedia20   0 11.031g 6.749g  14720 R 46.8 43.5 268:24.83 java
36710 gmedia20   0 11.031g 6.749g  14720 R 46.5 43.5 269:43.68 java

Thanks,
Praveen



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: High cpu on ignite server nodes

2018-06-16 Thread praveeng 
Hi Stan,

The high cpu usage is on all servers.
one doubt , what is the default expiry policy for any cache if we don't set.

Following are the stats of one cache collected from ignitevisor.



Cache 'playerSessionInfoCacheIgnite(@c18)':
+--+
| Name(@) | playerSessionInfoCacheIgnite(@c18) |
| Nodes   | 7  |
| Total size Min/Avg/Max  | 0 / 53201.29 / 151537  |
|   Heap size Min/Avg/Max | 0 / 21857.29 / 50001   |
|   Off-heap size Min/Avg/Max | 0 / 31344.00 / 101536  |
+--+

Nodes for: playerSessionInfoCacheIgnite(@c18)
+=+
|Node ID8(@), IP| CPUs | Heap Used | CPU Load |   Up Time   
|   Size   | Hi/Mi/Rd/Wr |
+=+
| 54F7EA58(@n4), ip.ip.ip.ip1   | 4| 43.81 %   | 2.43 %   | 24:20:08:018
| Total: 1000  | Hi: 0   |
|   |  |   |  | 
|   Heap: 1000 | Mi: 0   |
|   |  |   |  | 
|   Off-Heap: 0| Rd: 0   |
|   |  |   |  | 
|   Off-Heap Memory: 0 | Wr: 0   |
+---+--+---+--+--+--+-+
| D3A97470(@n7), ip.ip.ip.ip2   | 8| 41.88 %   | 0.27 %   | 02:26:29:576
| Total: 151536| Hi: 0   |
|   |  |   |  | 
|   Heap: 5| Mi: 0   |
|   |  |   |  | 
|   Off-Heap: 101536   | Rd: 0   |
|   |  |   |  | 
|   Off-Heap Memory: 100mb | Wr: 0   |
+---+--+---+--+--+--+-+
| 6BA0FEA2(@n5), ip.ip.ip.ip3   | 8| 25.74 %   | 0.30 %   | 02:29:02:915
| Total: 151529| Hi: 0   |
|   |  |   |  | 
|   Heap: 5| Mi: 0   |
|   |  |   |  | 
|   Off-Heap: 101529   | Rd: 0   |
|   |  |   |  | 
|   Off-Heap Memory: 100mb | Wr: 0   |
+---+--+---+--+--+--+-+
| E41C47FD(@n6), ip.ip.ip.ip4   | 8| 38.53 %   | 0.30 %   | 02:27:35:184
| Total: 66344 | Hi: 0   |
|   |  |   |  | 
|   Heap: 50001| Mi: 0   |
|   |  |   |  | 
|   Off-Heap: 16343| Rd: 0   |
|   |  |   |  | 
|   Off-Heap Memory: 16mb  | Wr: 0   |
+---+--+---+--+--+--+-+
| D487DD7A(@n3), ip.ip.ip.ip5   | 4| 36.07 %   | 1.90 %   | 24:27:24:711
| Total: 1000  | Hi: 0   |
|   |  |   |  | 
|   Heap: 1000 | Mi: 0   |
|   |  |   |  | 
|   Off-Heap: 0| Rd: 0   |
|   |  |   |  | 
|   Off-Heap Memory: 0 | Wr: 0   |
+---+--+---+--+--+--+-+
| A30CC6D1(@n2), ip.ip.ip.ip6   | 4| 29.72 %   | 0.50 %   | 24:33:45:581
| Total: 0 | Hi: 0   |
|   |  |   |  | 
|   Heap: 0| Mi: 0   |
|   |  |   |  | 
|   Off-Heap: 0| Rd: 0   |
|   |  |   |  | 
|   Off-Heap Memory: 0 | Wr: 0   |
+---+--+---+--+--+--+-+
| 4FA3F3EB(@n1), ip.ip.ip.ip7   | 4| 28.57 %   | 1.77 %   | 24:38:18:856
| Total: 1000  | Hi: 0   |
|   |  |   |  | 
|   Heap: 1000 | Mi: 0   |
|