Alexei, I would follow the Valentin's suggestion in the issue description and would add to Ignite tests a check on thread locals after stopping all grids. And then we will be able to fix thread local cleaning in all places.
Thanks, -- Artem -- On Wed, Apr 6, 2016 at 11:18 AM, Alexei Scherbakov < [email protected]> wrote: > Valentin, > > I did test on demo web app where HttpSession is backed by Ignite cache and > collect some logs. > There are many places in Ignite where thread locals are used. > Then cache methods are accessed in threads from servlet container pool, we > get leaks. > I don't think it will be easy to add thread locals cleaning in all these > places without major code rewrite, but, for sure, it would be ideal > solution. > > > <E0><EF><F0> 05, 2016 4:21:52 PM > org.apache.catalina.loader.WebappClassLoaderBase > checkThreadLocalMapForLeaks > SEVERE: The web application [/app1] created a ThreadLocal with key of type > [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value > [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@5fecc35f]) and a > value of type [java.lang.Integer] (value [0]) but failed to remove it when > the web application was stopped. Threads are going to be renewed over time > to try and avoid a probable memory leak. > <E0><EF><F0> 05, 2016 4:21:52 PM > org.apache.catalina.loader.WebappClassLoaderBase > checkThreadLocalMapForLeaks > SEVERE: The web application [/app1] created a ThreadLocal with key of type > [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1] (value > [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1@4ecdfc04]) > and a value of type [java.util.LinkedList] (value > [[org.apache.ignite.internal.util.tostring.GridToStringThreadLocal@684a2791 > ]]) > but failed to remove it when the web application was stopped. Threads are > going to be renewed over time to try and avoid a probable memory leak. > <E0><EF><F0> 05, 2016 4:21:52 PM > org.apache.catalina.loader.WebappClassLoaderBase > checkThreadLocalMapForLeaks > SEVERE: The web application [/app1] created a ThreadLocal with key of type > [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1] (value > [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1@4e9300e7]) > and a value of type > [org.apache.ignite.internal.binary.BinaryThreadLocalContext] (value > [org.apache.ignite.internal.binary.BinaryThreadLocalContext@7934a88]) but > failed to remove it when the web application was stopped. Threads are going > to be renewed over time to try and avoid a probable memory leak. > <E0><EF><F0> 05, 2016 4:21:52 PM > org.apache.catalina.loader.WebappClassLoaderBase > checkThreadLocalMapForLeaks > SEVERE: The web application [/app1] created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@56502b72]) and a > value of type > [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk] > (value > > [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk@4eab9d2f > ]) > but failed to remove it when the web application was stopped. Threads are > going to be renewed over time to try and avoid a probable memory leak. > <E0><EF><F0> 05, 2016 4:21:52 PM > org.apache.catalina.loader.WebappClassLoaderBase > checkThreadLocalMapForLeaks > SEVERE: The web application [/app1] created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@eb53c9d]) and a > value > of type > > [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry.StreamHolder] > (value > > [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry$StreamHolder@5e2c15f7 > ]) > but failed to remove it when the web application was stopped. Threads are > going to be renewed over time to try and avoid a probable memory leak. > <E0><EF><F0> 05, 2016 4:21:52 PM > org.apache.catalina.loader.WebappClassLoaderBase > checkThreadLocalMapForLeaks > SEVERE: The web application [/app1] created a ThreadLocal with key of type > [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value > [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@55b9da28]) and a > value of type [java.lang.Integer] (value [0]) but failed to remove it when > the web application was stopped. Threads are going to be renewed over time > to try and avoid a probable memory leak. > <E0><EF><F0> 05, 2016 4:21:52 PM > org.apache.catalina.loader.WebappClassLoaderBase > checkThreadLocalMapForLeaks > SEVERE: The web application [/app1] created a ThreadLocal with key of type > [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1] (value > [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1@1ac73431 > ]) > and a value of type [java.lang.Integer] (value [9]) but failed to remove it > when the web application was stopped. Threads are going to be renewed over > time to try and avoid a probable memory leak. > SEVERE: The web application [/app1] created a ThreadLocal with key of type > [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5] (value > [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5@28a1bd99]) > and a value of type > [org.apache.ignite.internal.processors.cache.GridCacheAdapter.FutureHolder] > (value > > [org.apache.ignite.internal.processors.cache.GridCacheAdapter$FutureHolder@6763d70c > ]) > but failed to remove it when the web application was stopped. Threads are > going to be renewed over time to try and avoid a probable memory leak. > > As a solution, I propose to implement servlet filter as a part of Ignite > web integration, which will be responsible for cleaning up all thread > locals after request processing. > Filter must be executed last in filter chain to work correctly. > It is not necessary to use the filter in case of Tomcat, because it > provides own solution to the problem requiring special configuration > parameter for context > User should decide which approach is better to him. > > > 2016-04-06 1:45 GMT+03:00 Valentin Kulichenko < > [email protected] > >: > > > Alexei, > > > > I would start with defining which thread locals cause this and why they > are > > not properly cleaned on node shutdown. Ideally, this should never happen. > > > > -Val > > > > On Tue, Apr 5, 2016 at 10:36 AM, Alexei Scherbakov < > > [email protected]> wrote: > > > > > Hello. > > > > > > Currently I'm working on the IGNITE-967 > > > <https://issues.apache.org/jira/browse/IGNITE-967>. > > > > > > In fact, it is well known, easily reproduceable, ClassLoader memory > leak > > > problem in managed environments, such as servlet containers. > > > See http://wiki.apache.org/tomcat/MemoryLeakProtection to read more on > > > that. > > > In short, the code using thread locals is responsible for cleaning them > > > after request processing > > > to prevent memory leaks related to different lifetime of application > and > > > thread in pool. > > > Also blindly reusing thread from pool can be dangerous, because current > > > ThreadLocal value can be different from expected default value. > > > > > > One posiible way of solving the problem is using post-processing > Filter, > > > which is responsible for cleaning all thread locals on current thread. > > > > > > Another approach would be sticking with container's specific solution > to > > > this problem. > > > On example, tomcat provides the configuration property > > > renewThreadsWhenStoppingContext, > > > which forces it to renew threads involved in servicing requests for > > > destroyed context, thus removing leaks. > > > > > > What do you think? > > > > > > > > > -- > > > > > > Best regards, > > > Alexei Scherbakov > > > > > > > > > -- > > Best regards, > Alexei Scherbakov >
