RE: Re: Ignite memory leaks in 2.8.0

[Andrey Davydov]

First scenario cause leak for exactly one instance of ignite. Second scenario cause leak for one instance per distinct key in thread local map (if key is ignite name, as much distinct names used in test, so much links leaked, I don’t know what is key in that map =() .   Ways to GC root in original letter I got using JVisualVM on heap dump I take after following commands:               Ignition.stopAll(true);             TestUtils.pause(5000); //wait for relaxation             System.gc();             TestUtils.pause(1000); //wait for relaxation             System.gc();             TestUtils.pause(1000); //wait for relaxation             System.gc();             TestUtils.pause(1000); //wait for relaxation             System.gc();             TestUtils.pause(1000); //wait for relaxation     Andrey.   От: Ilya Kasnacheev Отправлено: 19 марта 2020 г. в 14:44 Кому: user@ignite.apache.org Тема: Re: Ignite memory leaks in 2.8.0   Hello!   Our test suites start tens of thousands nodes during every suite run.   If there would be any leaks in start-stop scenario, we would surely notice this. I recommend checking why this is a problem in your scenario.   The problem you have mentioned may cause problems with class de-loading, however. Do you bring a new class loader for each test?   Can you file an issue about this so that we code a proper de-allocation?   Regards, -- Ilya Kasnacheev     ср, 18 мар. 2020 г. в 18:37, Andrey Davydov <andrey.davy...@gmail.com>: Hello,   There are at least two way link to IgniteKernal leaks to GC root and makes it unavailable for GC.   The first one:   this     - value: org.apache.ignite.internal.IgniteKernal #1 <- grid     - class: org.apache.ignite.internal.GridKernalContextImpl, value: org.apache.ignite.internal.IgniteKernal #1   <- ctx     - class: org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing, value: org.apache.ignite.internal.GridKernalContextImpl #2    <- this$0     - class: org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$10, value: org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing #2     <- serializer     - class: org.h2.util.JdbcUtils, value: org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$10 #1      <- [5395]     - class: java.lang.Object[], value: org.h2.util.JdbcUtils class JdbcUtils       <- elementData     - class: java.util.Vector, value: java.lang.Object[] #37309        <- classes     - class: sun.misc.Launcher$AppClassLoader, value: java.util.Vector #31         <- contextClassLoader (thread object)     - class: java.lang.Thread, value: sun.misc.Launcher$AppClassLoader #1   org.h2.util.JdbcUtils has static field JavaObjectSerializer serializer, which see IgniteKernal via IgniteH2Indexing. It make closed and stopped IgniteKernal non collectable by GC. If some Ignites run in same JVM, JdbcUtils will always use only one, and it can cause some races.   The second way:   this     - value: org.apache.ignite.internal.IgniteKernal #2 <- grid     - class: org.apache.ignite.internal.GridKernalContextImpl, value: org.apache.ignite.internal.IgniteKernal #2   <- ctx     - class: org.apache.ignite.internal.processors.cache.GridCacheContext, value: org.apache.ignite.internal.GridKernalContextImpl #1    <- cctx     - class: org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry, value: org.apache.ignite.internal.processors.cache.GridCacheContext #24     <- parent     - class: org.apache.ignite.internal.processors.cache.GridCacheMvccCandidate, value: org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry #4      <- [0]     - class: java.lang.Object[], value: org.apache.ignite.internal.processors.cache.GridCacheMvccCandidate #1       <- elements     - class: java.util.ArrayDeque, value: java.lang.Object[] #43259        <- value     - class: java.lang.ThreadLocal$ThreadLocalMap$Entry, value: java.util.ArrayDeque #816         <- [119]     - class: java.lang.ThreadLocal$ThreadLocalMap$Entry[], value: java.lang.ThreadLocal$ThreadLocalMap$Entry #51          <- table     - class: java.lang.ThreadLocal$ThreadLocalMap, value: java.lang.ThreadLocal$ThreadLocalMap$Entry[] #21           <- threadLocals (thread object)     - class: java.lang.Thread, value: java.lang.ThreadLocal$ThreadLocalMap #2   Link to IgniteKernal leaks to ThreadLocal variable, so when we start/stop many instances of Ignite in same jvm during testing, we got many stopped “zomby” ignites on ThreadLocal context of main test thread and it cause OutOfMemory after some dozens of tests.   Andrey.