Hi Rishi,
seems it's not a good idea to connect ignite repeatedly, I observed a
similar memory issue.
would you mind to share your server configurations (cores, memory)?
http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-td9443i20.html
h
Hi Val,
sorry for this late reply.
yes, we have a client node constantly joining and leaving topology.
does the new versions of Ignite have improvement on this issue?
the reason client node constantly joining and leaving topology is twofold:
1. we used a visor cli called by cron job to check the
>
> we have 2 host hosting 2 client and 2 server for ignite, our conf is bare
> minimal 4 cores 8 GB RAM.
>
> Now with OOME, how often you run the visor command ? we run it every 5
> mins.should you try that ?
>
> On Mon, May 8, 2017 at 9:01 PM, tysli2016 [via Apache Ignit
thanks Rishi, can you share more about that?
what's the version of Ignite? how many Ignite servers? how many CPU/memory?
are you using the Visor in batch mode
(https://apacheignite-tools.readme.io/v1.9/docs/batch-mode)?
or Visor alert?
I have tried Visor batch mode, but it lead to OOME eventually
thanks Denis, can you share what's the Ignite version using? Can the Visor
GUI be started with the alert setting automatically as a background job.
I have tried to use visor in batch mode
(https://apacheignite-tools.readme.io/v1.9/docs/batch-mode), so that a visor
process was created when a cron j
thanks Andrew, can you share more about how to use Nagios\Icinga or Zabbix to
monitor? for what kind of metrics and how to connect/setup?
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/How-to-monitor-and-alert-for-server-counts-tp12533p12555.html
Sent from the A
we have a couple of Ignite servers serve as key-value store and want to get
email notification when any server went down.
anyone are having the same need? what is your solution?
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/How-to-monitor-and-alert-for-server-
reproduced the OOME, the heap dump here
https://drive.google.com/drive/folders/0BwY2dxDlRYhBMEhmckpWeHg1bjg?usp=sharing
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p12529.html
Sent from the
Thanks Andrey, is there an option to monitor the number of server nodes in
the grid?
I found "nc - Total number of nodes in the grid.", seems counting server +
client nodes, correct?
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/OOME-on-2-node-cluster-with-vis
Thank Evgenii!
By running the `${IGNITE_HOME}/bin/ignitevisorcmd.sh -e="'open
-cpath=${IGNITE_HOME}/config/default-config.xml;node'"`, it shows "Ignite
node stopped OK" at the end. Is it an indicator of visor stopped properly?
We use the visor output to check the number of Ignite servers running,
Got "OutOfMemoryError: Java heap space" with 2-node cluster with a `visor`
running repeatedly.
The server nodes are running on CentOS 7 inside Oracle VirtualBox VM with
the same config:
- 2 vCPUs
- 3.5GB memory
- Oracle JDK 2.8.0_121
`default-config.xml` was modified to use non-default multicast
hemanta did you call close() on the Ignite instance?
Ignite ignite = Ignition.start();
// do something with ignite
ignite.close()
I have a similar problem before, and found that Ignite have some threads
running which prevent the jvm from stopping even the main thread ended.
--
Vi
Thanks for your effort in patiently and promptly replies, we are trying to
reproduce the issue, will keep you posted.
Tom
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9669.html
Sent from th
How about this OOME?
http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-GridDhtPartitionMap2-td9504.html
We found it on another client node in the same cluster, however it seems to
exhibit a different pattern of memory leak.
- Tom
--
View this message in con
> Are you sure that all of them are in this map? What is the size of rmtCfgs
map?
This map contains about 1/6 of the total CacheConfigurations, the size of
rmtCfg map is 227,306,792:
Class Name
| Shallow Heap
ic, so the embedded instance is the IgniteKernal object which implement the
Ignite interface.
> So these are instances are not really used by Ignite, but are saved
> somewhere, most likely in your code. Can you use heap dump to trace it?
Yes, there are a whole lot of CacheConfiguration objects c
What do you mean by "embedded instances"? if it's org.apache.ignite.Ignite
object then we have two, one for each .war application deployed.
And what do you mean by "not properly stopped and / or disconnected"? The
application call org.apache.ignite.Ignite.close() only when Tomcat shutdown.
What if
and we have the iptables open on ports 3-5, wondering if it's related
to the symtom.
because it shows `cachesOnDisconnect
org.apache.ignite.internal.processors.cache.GridCacheProcessor` holds the
objects.
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Ig
> How many caches do you have?
there are only 2 caches
> Any idea why you have so many CacheConfiguration objects? Who holds
> references to them?
the shortest paths to the accumulation point for java.util.HashMap @
0x8423c790 shows who holds the references, as you can see below, it's an
org.ap
Hi Val, sorry I m afraid I cannot provide the heap dump because it might
contain some sensitive data.
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9507.html
Sent from the Apache Ignite Users
shortest paths to the accumulation point
Class Name
| Shallow Heap | Retained Heap
OOME found on the same cluster as mentioned in
http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-td9443.html
Recap the setup here:
We have 2 machines (M1, M2)
In M1 running a Ignite server (I1) and a Tomcat server (T1), which hosted 2
Ig
Thx Val for your reply, let me check if I can give you the .hprof file later.
This is the dominator tree showing all 23,850 items from
java.util.HashMap$Node[65535] @ 0xcd657f20, It's too many of them so I just
expand some of them and they are all holding CacheConfiguration objects.
https://drive.
We have 2 machines (M1, M2)
In M1 running a Ignite server (I1) and a Tomcat server (T1), which hosted 2
Ignite clients (C1a, C1b),
similarly in M2 running a Ignite server (I2) and a Tomcat server (T2), which
hosted 2 Ignite clients (C2a, C2b).
OutOfMemoryError were found in both T1 and T2 yesterda
24 matches
Mail list logo
