Hi, I’m trying to load tSV file size ~six gigabytes (contains ~90 million records) into single Apache Ignite node with allocated heap size = 40 gigabytes. For loading used IgniteDataStreamer. Code and sample (10 records) of data is here: https://github.com/olegskoblya/traffic-loader.git Physical machine has 64 GB of RAM, 8 CPU cores (each has 2,25 GHz). Cache configured for storing data in unlimited off-heap memory (trafficCacheConf.setMemoryMode(CacheMemoryMode.OFFHEAP_TIERED); trafficCacheConf.setOffHeapMaxMemory(0);) After ~18 min physical memory usage increased to 90% and speed of loading catastrophically slow down (see picture below – GGVisor used in trial/demo mode with Apache Ignite nodes deployed on single host :) ): <http://apache-ignite-users.70518.x6.nabble.com/file/n4798/physical_memory_loading_near_end.png> <http://apache-ignite-users.70518.x6.nabble.com/file/n4798/loading_process.png> After manually calling GC – nothing changed (in loading speed and memory usage). Finally file has been loaded but last ~2 million records being processed the same time as ~88 previous (of course I can’t run sql query on loaded data because indexes was turned off for better loading performance). On tab SQL Viewer in column named “Total Memory” placed value 24,7GB for used Off Heap memory (I mean it is my ~6GB of data “eat” so many?). <http://apache-ignite-users.70518.x6.nabble.com/file/n4798/off_heap_used.png> So, it is actually normall that ~6GB of data increases in size four times after loading into the cache? How being used rest of memory (as 24,7 it is less then half of 90% used physical memory)? Is it possible to reduce such memory consuming? And loading 6GB of data to in-memory cache on same machine in 34 minutes seen as unsatisfactorily.
I will be grateful for any suggestions. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Slow-data-loading-and-high-very-memory-usage-issues-tp4798.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
