Hi, What I see from the “top” screenshot is that 64 GB of RAM is already occupied and Linux kernel starts swapping data on disk since allowed virtual memory size is more than 90 GB. All these page ins and page outs can dramatically decrease the performance and seems that kswapd daemon works extensively (kswapd consumes around 54% of CPU cores). If you run instat or dstat you should see that the disk is became a hotspot.
In general you should avoid situations when Linux kernel starts swapping pages on the disk. As a general recommendations please do the following: - it’s not practical to set Java heap size to 40 GB as you did because it will eventually lead to long GC pauses since neither CMS nor G1 will be able to keep up with such huge heaps under the load. My suggestion is to decrease the heap size to a value not bigger than 10 GB. Refer to this [1] JVM tuning recommendations for off-heap caches; - tune Linux kernel to avoid long GC pauses that can be caused by Linux GC “attacks” [2] - if all cache data will not physically fit in RAM on one machine then you have to start one more Ignite node on a separate physical machine, otherwise the swapping is unavoidable. [1] https://apacheignite.readme.io/docs/jvm-and-system-tuning#jvm-tuning-for-clusters-with-off_heap-caches <https://apacheignite.readme.io/docs/jvm-and-system-tuning#jvm-tuning-for-clusters-with-off_heap-caches> [2] https://apacheignite.readme.io/docs/jvm-and-system-tuning#gc-attacks-by-linux Regards, Denis > On May 5, 2016, at 7:17 PM, oskoblya <oskob...@at-consulting.ru> wrote: > > Hi, I’m trying to load tSV file size ~six gigabytes (contains ~90 million > records) into single Apache Ignite node with allocated heap size = 40 > gigabytes. For loading used IgniteDataStreamer. Code and sample (10 > records) of data is here: > https://github.com/olegskoblya/traffic-loader.git > Physical machine has 64 GB of RAM, 8 CPU cores (each has 2,25 GHz). Cache > configured for storing data in unlimited off-heap memory > (trafficCacheConf.setMemoryMode(CacheMemoryMode.OFFHEAP_TIERED); > trafficCacheConf.setOffHeapMaxMemory(0);) > After ~18 min physical memory usage increased to 90% and speed of loading > catastrophically slow down (see picture below – GGVisor used in trial/demo > mode with Apache Ignite nodes deployed on single host :) ): > <http://apache-ignite-users.70518.x6.nabble.com/file/n4798/physical_memory_loading_near_end.png> > > <http://apache-ignite-users.70518.x6.nabble.com/file/n4798/loading_process.png> > > After manually calling GC – nothing changed (in loading speed and memory > usage). Finally file has been loaded but last ~2 million records being > processed the same time as ~88 previous (of course I can’t run sql query on > loaded data because indexes was turned off for better loading performance). > On tab SQL Viewer in column named “Total Memory” placed value 24,7GB for > used Off Heap memory (I mean it is my ~6GB of data “eat” so many?). > <http://apache-ignite-users.70518.x6.nabble.com/file/n4798/off_heap_used.png> > So, it is actually normall that ~6GB of data increases in size four times > after loading into the cache? How being used rest of memory (as 24,7 it is > less then half of 90% used physical memory)? Is it possible to reduce such > memory consuming? And loading 6GB of data to in-memory cache on same machine > in 34 minutes seen as unsatisfactorily. > > I will be grateful for any suggestions. > > > > > -- > View this message in context: > http://apache-ignite-users.70518.x6.nabble.com/Slow-data-loading-and-high-very-memory-usage-issues-tp4798.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com.
