Hi, I created a ticket for the issue you found: https://issues.apache.org/jira/browse/IGNITE-7319
Thanks, Mike. 2017-12-18 17:05 GMT+03:00 zbyszek <[email protected]>: > > Dear all, > > I was wondering if this is a know issue which has a chance to be fixed in > future or (I hope) it is me who missed something obvious in working with > Ignite caches. > I have a simple single node test app (built to investigate a memory leak > observed in our PROD deployment), that creates c.a. 20 LOCAL caches per > sec. > with the config below: > > private IgniteCache<String, BinaryObject> createLocalCache(String name) { > CacheConfiguration<String, BinaryObject> cCfg = new > CacheConfiguration<>(); > cCfg.setName(name); > cCfg.setGroupName("localCaches"); // without group leak is much > bigger! > cCfg.setStoreKeepBinary(true); > cCfg.setCacheMode(CacheMode.LOCAL); > cCfg.setOnheapCacheEnabled(false); > cCfg.setCopyOnRead(false); > cCfg.setBackups(0); > cCfg.setWriteBehindEnabled(false); > cCfg.setReadThrough(false); > cCfg.setReadFromBackup(false); > cCfg.setQueryEntities(<some basic query here>); > return ignite.createCache(cCfg).withKeepBinary(); > } > > The caches are placed in the queue and are picked up by the worker thread > which just destorys them after removing from the queue. > This setup seems to generate a memory leak of about 1GB per day. > When looking at heapdump, I see all space is occupied by instances of > java.util.concurrent.ConcurrentSkipListMap$Node: > > (please copy paste table into notepad to see tables correctly) > > Objects by class > +----------------------------------------------------------- > ------------------------------------------------------------ > -------------+--------------------+----------------------+-- > -------------------------+ > | Class > | Objects | Shallow Size | Retained Size | > +----------------------------------------------------------- > ------------------------------------------------------------ > -------------+--------------------+----------------------+-- > -------------------------+ > | java.util.concurrent.ConcurrentSkipListMap$Node > | 4,987,415 13 % | 119,697,960 10 % | ~ 1,204,893,605 100 % | > | > org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$ > CancelableTask > | 4,985,687 13 % | 239,312,976 20 % | ~ 917,361,000 76 % | > | > org.apache.ignite.internal.processors.cache.query.continuous. > CacheContinuousQueryManager$BackupCleaner > | 4,985,680 13 % | 119,656,320 10 % | ~ 558,390,752 46 % | > | org.jsr166.ConcurrentHashMap8 > | 4,990,926 13 % | 199,637,040 17 % | ~ 439,459,352 36 % | > | org.jsr166.LongAdder8 > | 4,992,416 13 % | 159,757,312 13 % | ~ 159,757,312 13 % | > | org.apache.ignite.lang.IgniteUuid > | 4,989,306 13 % | 119,743,344 10 % | ~ 119,745,456 10 % | > | java.util.concurrent.ConcurrentSkipListMap$Index > | 2,488,987 7 % | 59,735,688 5 % | ~ 119,502,384 10 % | > | java.util.concurrent.ConcurrentSkipListMap$HeadIndex > | 49 0 % | 1,568 0 % | ~ 106,991,832 9 % | > | org.jsr166.ConcurrentHashMap8$ValuesView > | 4,985,368 13 % | 79,765,888 7 % | ~ 79,765,888 7 % | > | java.util.HashMap$Node > | 44,335 0 % | 1,418,720 0 % | ~ 79,618,104 7 % | > | java.util.HashMap$Node[] > | 13,093 0 % | 1,098,856 0 % | ~ 68,150,520 6 % | > | java.util.HashMap > | 13,550 0 % | 650,400 0 % | ~ 67,636,112 6 % | > | java.util.concurrent.ConcurrentSkipListMap > | 10 0 % | 480 0 % | ~ 59,830,768 5 % | > > > Merged paths to java.util.concurrent.ConcurrentSkipListMap$Node instances > (first 5 levels) reports no obvious dominator > (at least no dominator from my test namespace): > > > Merged paths > +----------------------------------------------------------- > ---------------------------------------------------------+-- > ------------------+------------------------+--------------+ > | Name > | Objects | Retained Size | Dominators | > +----------------------------------------------------------- > ---------------------------------------------------------+-- > ------------------+------------------------+--------------+ > | +---<All the objects> > | 4,987,415 100 % | 1,037,112,344 100 % | | > | | > | | | | > | +---<Retained from several objects simultaneously (don't have a > dominator)> | 1,245,699 25 % | > 1,037,015,992 99 % | | > | | | > | | | | > | | +---java.util.concurrent.ConcurrentSkipListMap$Index > | | | | > | | | | > | | | | > | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex > | | | | > | | | | > | | | | > | | | +---java.util.concurrent.ConcurrentSkipListMap > | | | | > | | | | | > | | | | > | | | | +---org.apache.ignite.internal.util.GridConcurrentSkipListSet > | | | | > | | | | > | | | | > | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex > | | | | > | | | | > | | | | > | | | +---java.util.concurrent.ConcurrentSkipListMap > | | | | > | | | | > | | | | > | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex > | | | | > | | | > | | | | > | | +---java.util.concurrent.ConcurrentSkipListMap$Node > | | | | > | | | | > | | | | > | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex > | | | | > | | | | > | | | | > | | | +---java.util.concurrent.ConcurrentSkipListMap > | | | | > | | | | > | | | | > | | | +---org.apache.ignite.internal.util.GridConcurrentSkipListSet > | | | | > | | | > | | | | > | | +---org.apache.ignite.thread.IgniteThread > | | | | > | | > | | | | > | +---java.util.concurrent.ConcurrentSkipListMap$Node > | 3,741,257 75 % | 777,983,576 75 % | 3,741,257 | > | | | > | | | | > | | +---<Retained from several objects simultaneously (don't have a > dominator)> | 934,092 19 % | > 777,912,544 75 % | | > | | | | > | | | | > | | | +---java.util.concurrent.ConcurrentSkipListMap$Index > | | | | > | | | | | > | | | | > | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex > | | | | > | | | | | | > | | | | > | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex > | | | | > | | | | | > | | | | > | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index > | | | | > | | | | | > | | | | > | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex > | | | | > | | | | | > | | | | > | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index > | | | | > | | | | > | | | | > | | | +---java.util.concurrent.ConcurrentSkipListMap$Node > | | | | > | | | | | > | | | | > | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex > | | | | > | | | | | | > | | | | > | | | | | +---java.util.concurrent.ConcurrentSkipListMap > | | | | > | | | | | > | | | | > | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index > | | | | > | | | | | > | | | | > | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex > | | | | > | | | | | > | | | | > | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index > | | | | > | | | | > | | | | > | | | +---org.apache.ignite.thread.IgniteThread > | | | | > | | | > | | | | > | | +---java.util.concurrent.ConcurrentSkipListMap$Node > | 2,806,816 56 % | 583,674,344 56 % | 2,806,816 | > | | | | > | | | | > | | | +---<Retained from several objects simultaneously (don't have a > dominator)> | 700,884 14 % | > 583,622,824 56 % | | > | | | | | > | | | | > | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index > | | | | > | | | | | | > | | | | > | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex > | | | | > | | | | | | > | | | | > | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index > | | | | > | | | | | > | | | | > | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node > | | | | > | | | | | | > | | | | > | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex > | | | | > | | | | | | > | | | | > | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index > | | | | > | | | | | | > | | | | > | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node > | | | | > | | | | | > | | | | > | | | | +---org.apache.ignite.thread.IgniteThread > | | | | > | | | | > | | | | > | | | +---java.util.concurrent.ConcurrentSkipListMap$Node > | 2,105,683 42 % | 437,877,768 42 % | 2,105,683 | > | | | | | > | | | | > | | | | +---<Retained from several objects simultaneously (don't have a > dominator)> | 525,764 11 % | > 437,840,192 42 % | | > | | | | | | > | | | | > | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Index > | | | | > | | | | | | > | | | | > | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node > | | | | > | | | | | | > | | | | > | | | | | +---org.apache.ignite.thread.IgniteThread > | | | | > | | | | | > | | | | > | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node > | 1,579,741 32 % | 328,509,568 32 % | 1,579,741 | > | | | | | | > | | | | > | | | | | +---<Retained from several objects simultaneously (don't have > a > dominator)> | 394,396 8 % | > 328,481,960 32 % | | > | | | | | | > | | | | > | | | | | +---java.util.concurrent.ConcurrentSkipListMap$Node > | 1,185,216 24 % | 246,468,448 24 % | 1,185,216 | > | | | | | | > | | | | > | | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex > | 129 0 % | 27,608 0 % | 2 | > | | | | | > | | | | > | | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex > | 178 0 % | 37,576 0 % | 2 | > | | | | | > | | | | > | | | | +---java.util.concurrent.ConcurrentSkipListMap > | 178 0 % | 37,576 0 % | 2 | > | | | | > | | | | > | | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex > | 249 0 % | 51,520 0 % | 2 | > | | | | > | | | | > | | | +---java.util.concurrent.ConcurrentSkipListMap > | 249 0 % | 51,520 0 % | 2 | > | | | | > | | | | > | | | > +---org.apache.ignite.internal.processors.affinity. > GridAffinityAssignmentCache > | 249 0 % | 51,520 0 % | 2 | > | | | > | | | | > | | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex > | 348 0 % | 71,008 0 % | 2 | > | | | | > | | | | > | | | +---java.util.concurrent.ConcurrentSkipListMap > | 348 0 % | 71,008 0 % | 2 | > | | | | > | | | | > | | | > +---org.apache.ignite.internal.processors.affinity. > GridAffinityAssignmentCache > | 348 0 % | 71,008 0 % | 2 | > | | | | > | | | | > | | | +---<Retained from several objects simultaneously (don't have > a > dominator)> | 348 0 % | > 71,008 0 % | | > | | | > | | | | > | | +---java.util.concurrent.ConcurrentSkipListMap$KeyIterator > | 1 0 % | 24 0 % | 1 | > | | | > | | | | > | | +---org.apache.ignite.thread.IgniteThread > | 1 0 % | 24 0 % | 1 | > | | > | | | | > | +---java.util.concurrent.ConcurrentSkipListMap$HeadIndex > | 458 0 % | 96,304 0 % | 11 | > | | | > | | | | > | | +---java.util.concurrent.ConcurrentSkipListMap > | 457 0 % | 96,280 0 % | 10 | > | | | | > | | | | > | | | > +---org.apache.ignite.internal.processors.affinity. > GridAffinityAssignmentCache > | 451 0 % | 96,136 0 % | 2 | > | | | | | > | | | | > | | | | +---<Retained from several objects simultaneously (don't have a > dominator)> | 451 0 % | > 96,136 0 % | | > | | | | | > | | | | > | | | | > +---org.apache.ignite.internal.processors.cache. > CacheAffinitySharedManager$CacheGroupHolder1 > | | | | > | | | | | > | | | | > | | | | > +---org.apache.ignite.internal.processors.cache.CacheGroupContext > | | | | > | | | | | > | | | | > | | | | > +---org.apache.ignite.internal.processors.cache.GridCacheAffinityManager > | | | | > | | | | > | | | | > | | | +---org.apache.ignite.internal.util. > GridBoundedConcurrentOrderedSet > | 4 0 % | 96 0 % | 4 | > | | | | | > | | | | > | | | | > +---org.apache.ignite.internal.processors.cache.query. > GridCacheDistributedQueryManager > | 4 0 % | 96 0 % | 2 | > | | | | | > | | | | > | | | | +---<Retained from several objects simultaneously (don't have > a > dominator)> | 4 0 % | > 96 0 % | | > | | | | > | | | | > | | | +---<Retained from several objects simultaneously (don't have a > dominator)> | 1 0 % | > 24 0 % | | > | | | | | > | | | | > | | | | +---org.apache.ignite.internal.util.GridConcurrentSkipListSet > | | | | > | | | | | | > | | | | > | | | | | > +---org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor > | | | | > | | | | | > | | | | > | | | | +---org.apache.ignite.thread.IgniteThread > | | | | > | | | | > | | | | > | | | > +---org.apache.ignite.internal.processors.cache. > GridCachePartitionExchangeManager > | 1 0 % | 24 0 % | 1 | > | | | | > | | | | > | | | +---<Retained from several objects simultaneously (don't have a > dominator)> | 1 0 % | > 24 0 % | | > | | | | > | | | | > | | | +---java.util.LinkedList$Node > | | | | > | | | | > | | | | > | | | > +---org.apache.ignite.internal.processors.cache. > GridCachePartitionExchangeManager$1 > | | | | > | | | | > | | | | > | | | > +---org.apache.ignite.internal.processors.cache. > GridCachePartitionExchangeManager$2 > | | | | > | | | | > | | | | > | | | > +---org.apache.ignite.internal.processors.cache. > GridCachePartitionExchangeManager$3 > | | | | > | | | | > | | | | > | | | > +---org.apache.ignite.internal.processors.cache. > GridCachePartitionExchangeManager$4 > | | | | > | | | | > | | | | > | | | > +---org.apache.ignite.internal.processors.cache. > GridCachePartitionExchangeManager$5 > | | | | > | | | | > | | | | > | | | > +---org.apache.ignite.internal.processors.cache. > GridCachePartitionExchangeManager$ExchangeWorker > | | | | > | | | | > | | | | > | | | > +---org.apache.ignite.internal.processors.cache.GridCacheSharedContext > | | | | > | | | > | | | | > | | +---org.apache.ignite.internal.util.GridBoundedConcurrentOrderedMap > | 1 0 % | 24 0 % | 1 | > | | | > | | | | > | | > +---org.apache.ignite.internal.processors.cache. > transactions.IgniteTxManager > | 1 0 % | 24 0 % | 1 | > | | | > | | | | > | | +---<Retained from several objects simultaneously (don't have a > dominator)> | 1 0 % | > 24 0 % | | > | | | > | | | | > | | +---java.util.LinkedList$Node > | | | | > | | | > | | | | > | | > +---org.apache.ignite.internal.processors.cache.GridCacheSharedContext > | | | | > | | | > | | | | > | | > +---org.apache.ignite.internal.processors.cache. > transactions.IgniteTxManager$2 > | | | | > | | | > | | | | > | | > +---org.apache.ignite.internal.processors.cache. > transactions.IgniteTxManager$3 > | | | | > | | | > | | | | > | | > +---org.apache.ignite.internal.processors.cache. > transactions.IgniteTxManager$DeadlockDetectionListener > | | | | > | | > | | | | > | +---java.util.concurrent.ConcurrentSkipListMap$KeyIterator > | 1 0 % | 48 0 % | 1 | > | | > | | | | > | +---org.apache.ignite.thread.IgniteThread > | 1 0 % | 48 0 % | 1 | > +----------------------------------------------------------- > ---------------------------------------------------------+-- > ------------------+------------------------+--------------+ > > > > > Also 2 Bigges objects - Dominators are reported as: > > > Objects with biggest retained size > +----------------------------------------------------------- > ------------------------------------------------------------ > -------------------------------------------------+---------------------+ > | > Name > | Retained Size | > +----------------------------------------------------------- > ------------------------------------------------------------ > -------------------------------------------------+---------------------+ > | +---java.util.concurrent.ConcurrentSkipListMap [Stack Local] size = > 4,985,687 > | 59,710,656 5 % | > | | > | | > | > +---org.apache.ignite.internal.processors.cache. > GridCachePartitionExchangeManager > | 24,527,600 2 % | > | | | > | | > | | > +---org.apache.ignite.internal.processors.cache. > GridCachePartitionExchangeManager$ExchangeFutureSet > | 24,525,728 2 % | > | | | > | | > | | +---java.util.LinkedList size = 1,000 > | 24,525,680 2 % | > > > I have also noticed that during the test the most active thread is always > "exchange-worker" with allocation reported by visualvm on the level of > 200MB/sec at the end of the test. > Please let me know if this is a know issue - and if not - if you would be > interested in simple repro app. > > Thank you in advance, > zbyszek. > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > >
