[Resolved] clearReferencesThreads, Poller SunPKCS11-Solaris and strange context class loader
On 06.05.2010 20:51, Rainer Jung wrote: While doing some testing with 6.0.26 I noticed, that when shutting it down it logs an error about thread "Poller SunPKCS11-Solaris" not being stopped. When I add the more explicit logging from tc6 trunk, it says the thread was started by /manager. The thread sits in java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:681) at java.lang.Thread.run(Thread.java:619) and is started for some version of the JDK (noted on Solaris for e.g. 1.5.0_22 and 1.6.0_3, not for 1.6.0_17) even without having anything related to keystores, https or similar configured in Tomcat (default config). The only non-default is using Log4J instead of Juli. What is strange, is that the check whether the context class loader of the thread is equal to the WebappClassLoader of the context to unload passes. I added an additional output by explicitely printing the contextName of the two loaders and in fact both print "/manager". When deploying two instances of the manager and reloading the original instance with the additional one, I get the same warning. In a heap dump, it seems the context class loader of the thread is the system class loader and not of type WebappClassLoader. Any idea, why the context class loader test fails? Is there any reason why the context class loader of the thread might change when doing the manager reload or shutdown? 1) Why is the class loader in the heap dump not the WebappClassLoader? For producing a heap dump, I added "-Dcom.sun.management.jmxremote=true". As a side effect java.security got initialized much earlier and before Tomcat startup happened. So the poller thread was started earlier and had the system class loader as TCCL. I noticed that by adding "-verbose:class" to track class loading. 2) What triggers startup of that thread in normal Tomcat? When *not* using the jmx system properties, the thread starts during webapp deployment. StandardManager inside start() initializes random numbers by calling generateSessionId(), and that method in ManagerBase calls getDigest(), which calls java.security.MessageDigest.getInstance() which finally initializes the Java Security Providers and starts the poller thread. I didn't actually check it, but I expect the TCCL of the main thread during that stage of webapp deployment is the webapp class loader. If I don't deploy the manager webapp, then the poller is simply started during the ROOT webapp deployment. Whatever context is deployed first starts the thread. 3) Why isn't it there for all Java versions? Sun applied a fix to the SunPKCS11 class for the so-called soft token. The thread is now started only, if the token is "removable" and the soft token is not. The patch is not part e.g. of old Java 6 versions. So for older JDK on Solaris 10, there is always a so-called soft token, and initializing the Java security providers will launch the thread. For older Solaris (e.g. 8) there is no soft token and thus no thread. For newer JDK patch versions, the soft token is identified to not be removable and again no poller thread is started. 4) Do we have to fix anything? If users do not actually use a removable token, upgrading the JDK is enough. Once users start to complain about the leak warning when using a removable token, it would suffice to initialize the Java security providers before deploying the webapps, so that the thread has the system or catalina loader as TCCL. This can be done e.g. by calling java.security.Security.getProviders(). But actually I'm not sure, whether the warning is a false positive. It's quite possible, that the poller thread actually does imply a perm gen leak. Thanks to all who provided valuable pointers. Regards, Rainer - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange context class loader
Hi Konstantin, On 09.05.2010 11:38, Konstantin Kolinko wrote: 2010/5/9 Rainer Jung: On 07.05.2010 20:08, Filip Hanik - Dev Lists wrote: On 05/07/2010 08:03 AM, Rainer Jung wrote: It's there after JDK start with some version of the JDK. It's not there if I only start a plain Java test doing only a sleep using the same JDK. well, if the app uses java.net.URL to a https, then this would make sense that the thread got started with the webapp's class loader Yes, but there is no app. The thread is there even when I remove all webapps (except for the ROOT context I kept). I guess I need to do more systematic testing. Some more research, looking at the source code that I mentioned: 1. "SunPKCS11-Solaris" is not the default name. The default name of the class instance would be "SunPKCS11-Dummy". You may want to look at the "/lib/security/java.security" file in your JRE. It should be configured there. Yes, I knew. PKCS is in there, but it is by default. The file hasn't been touched. Here is some, though failing. example of trying to use this class: http://forums.sun.com/thread.jspa?threadID=5431010 2. sun.security.pkcs11.SunPKCS11 extends java.security.AuthProvider 3. SunPKCS11() constructor calls initToken(slotInfo) and that method will call createPoller() if the token is not present in the slot. (Thus apparently it starts waiting for the token). 4. It is said that PKCS11 "defines a platform-independent API to cryptographic tokens, such as Hardware Security Modules (HSM) and smart cards." The point is: - there is no cryptographic hardware involved in my case - Tomcat default configuration used, HTTPS connector disabled - no tc native - I don't want to use PKCS or any other crypto Nevertheless the threads show up and is spotted by the memory leak detection during manager shutdown/restart as belonging to the manager webapp (by means of the TCCL), although in the dump before the restart its TCCL was the system class loader and the thread also gets started when no manager gets deployed. My first priority is to remove the false positive for the leak detection, only the second prio is getting rid of the thread. Regards, Rainer - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange context class loader
2010/5/9 Rainer Jung : > On 07.05.2010 20:08, Filip Hanik - Dev Lists wrote: >> >> On 05/07/2010 08:03 AM, Rainer Jung wrote: >>> >>> It's there after JDK start with some version of the JDK. It's not >>> there if I only start a plain Java test doing only a sleep using the >>> same JDK. >> >> well, if the app uses java.net.URL to a https, then this would make >> sense that the thread got started with the webapp's class loader > > Yes, but there is no app. The thread is there even when I remove all webapps > (except for the ROOT context I kept). I guess I need to do more systematic > testing. > Some more research, looking at the source code that I mentioned: 1. "SunPKCS11-Solaris" is not the default name. The default name of the class instance would be "SunPKCS11-Dummy". You may want to look at the "/lib/security/java.security" file in your JRE. It should be configured there. Here is some, though failing. example of trying to use this class: http://forums.sun.com/thread.jspa?threadID=5431010 2. sun.security.pkcs11.SunPKCS11 extends java.security.AuthProvider 3. SunPKCS11() constructor calls initToken(slotInfo) and that method will call createPoller() if the token is not present in the slot. (Thus apparently it starts waiting for the token). 4. It is said that PKCS11 "defines a platform-independent API to cryptographic tokens, such as Hardware Security Modules (HSM) and smart cards." Best regards, Konstantin Kolinko - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange context class loader
On 07.05.2010 20:08, Filip Hanik - Dev Lists wrote: On 05/07/2010 08:03 AM, Rainer Jung wrote: It's there after JDK start with some version of the JDK. It's not there if I only start a plain Java test doing only a sleep using the same JDK. well, if the app uses java.net.URL to a https, then this would make sense that the thread got started with the webapp's class loader Yes, but there is no app. The thread is there even when I remove all webapps (except for the ROOT context I kept). I guess I need to do more systematic testing. Regards, Rainer - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange context class loader
On 05/07/2010 08:03 AM, Rainer Jung wrote: On 07.05.2010 14:23, Konstantin Kolinko wrote: 2010/5/7 Rainer Jung: On 07.05.2010 11:00, Konstantin Kolinko wrote: 2010/5/7 Rainer Jung: I'm wondering why the PCKS Token Poller thread was captured by the leak prevention. Since we know the code, it was because its context class loader was equal to the WebappClassLoader of /manager. That's what I don't understand. See my original post. TCCL is inherited from the parent thread. It means that that thread was started either during request processing, or during webapplication startup. But unfortunately that doesn't provide consistent results. If I sart Tomcat without any webapp (but with ROOT context) the thread is started too. The heap dump shows the system cl as the tccl, but during restart or shutdown the leak prevention retrieves the tccl and finds it s equal to the WebappClassLoader of the manager. I'm confused. BTW, 4th result in http://www.google.com/search?q=SunPKCS11+TokenPoller gives source code for that class. I'll see, but I guess the point is more about who starts it when and why, less about what it does. What is this all about, I means this PKCS11 tokens? Is it something that is used in your configuration? No, not at all. There's not HTTPS connector, no keystore no nothing. The PKCS11 provider comes with the JDK but I'm not aware of anything that uses it. Default Tomcat configuration plus log4j. It's there after JDK start with some version of the JDK. It's not there if I only start a plain Java test doing only a sleep using the same JDK. well, if the app uses java.net.URL to a https, then this would make sense that the thread got started with the webapp's class loader I mean, what are those "tokens" that it works with? (for a newbie) Do you use JAAS? Not that I'm aware of. From the code, e.g. SunPKCS11#uninitToken() calls 1) destroyPoller() 2) createPoller() which restarts the thread. So the one at shutdown time might be different from the one that you see when Tomcat starts. Hmmm. Will dig deeper over the weekend. Regards, Rainer - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange context class loader
On 07.05.2010 14:23, Konstantin Kolinko wrote: 2010/5/7 Rainer Jung: On 07.05.2010 11:00, Konstantin Kolinko wrote: 2010/5/7 Rainer Jung: I'm wondering why the PCKS Token Poller thread was captured by the leak prevention. Since we know the code, it was because its context class loader was equal to the WebappClassLoader of /manager. That's what I don't understand. See my original post. TCCL is inherited from the parent thread. It means that that thread was started either during request processing, or during webapplication startup. But unfortunately that doesn't provide consistent results. If I sart Tomcat without any webapp (but with ROOT context) the thread is started too. The heap dump shows the system cl as the tccl, but during restart or shutdown the leak prevention retrieves the tccl and finds it s equal to the WebappClassLoader of the manager. I'm confused. BTW, 4th result in http://www.google.com/search?q=SunPKCS11+TokenPoller gives source code for that class. I'll see, but I guess the point is more about who starts it when and why, less about what it does. What is this all about, I means this PKCS11 tokens? Is it something that is used in your configuration? No, not at all. There's not HTTPS connector, no keystore no nothing. The PKCS11 provider comes with the JDK but I'm not aware of anything that uses it. Default Tomcat configuration plus log4j. It's there after JDK start with some version of the JDK. It's not there if I only start a plain Java test doing only a sleep using the same JDK. I mean, what are those "tokens" that it works with? (for a newbie) Do you use JAAS? Not that I'm aware of. From the code, e.g. SunPKCS11#uninitToken() calls 1) destroyPoller() 2) createPoller() which restarts the thread. So the one at shutdown time might be different from the one that you see when Tomcat starts. Hmmm. Will dig deeper over the weekend. Regards, Rainer - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange context class loader
2010/5/7 Rainer Jung : > On 07.05.2010 11:00, Konstantin Kolinko wrote: >> >> 2010/5/7 Rainer Jung: >>> >>> I'm wondering why the PCKS Token >>> Poller thread was captured by the leak prevention. Since we know the >>> code, >>> it was because its context class loader was equal to the >>> WebappClassLoader >>> of /manager. That's what I don't understand. See my original post. >>> >> >> TCCL is inherited from the parent thread. It means that that thread >> was started either during request processing, or during webapplication >> startup. > > But unfortunately that doesn't provide consistent results. > > If I sart Tomcat without any webapp (but with ROOT context) the thread is > started too. The heap dump shows the system cl as the tccl, but during > restart or shutdown the leak prevention retrieves the tccl and finds it s > equal to the WebappClassLoader of the manager. > > I'm confused. > >> BTW, 4th result in >> http://www.google.com/search?q=SunPKCS11+TokenPoller >> gives source code for that class. > > I'll see, but I guess the point is more about who starts it when and why, > less about what it does. > What is this all about, I means this PKCS11 tokens? Is it something that is used in your configuration? I mean, what are those "tokens" that it works with? (for a newbie) Do you use JAAS? >From the code, e.g. SunPKCS11#uninitToken() calls 1) destroyPoller() 2) createPoller() which restarts the thread. So the one at shutdown time might be different from the one that you see when Tomcat starts. Best regards, Konstantin Kolinko - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange context class loader
On 07.05.2010 11:00, Konstantin Kolinko wrote: 2010/5/7 Rainer Jung: I'm wondering why the PCKS Token Poller thread was captured by the leak prevention. Since we know the code, it was because its context class loader was equal to the WebappClassLoader of /manager. That's what I don't understand. See my original post. TCCL is inherited from the parent thread. It means that that thread was started either during request processing, or during webapplication startup. But unfortunately that doesn't provide consistent results. If I sart Tomcat without any webapp (but with ROOT context) the thread is started too. The heap dump shows the system cl as the tccl, but during restart or shutdown the leak prevention retrieves the tccl and finds it s equal to the WebappClassLoader of the manager. I'm confused. BTW, 4th result in http://www.google.com/search?q=SunPKCS11+TokenPoller gives source code for that class. I'll see, but I guess the point is more about who starts it when and why, less about what it does. Regards, Rainer - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange context class loader
2010/5/7 Rainer Jung : > I'm wondering why the PCKS Token > Poller thread was captured by the leak prevention. Since we know the code, > it was because its context class loader was equal to the WebappClassLoader > of /manager. That's what I don't understand. See my original post. > TCCL is inherited from the parent thread. It means that that thread was started either during request processing, or during webapplication startup. BTW, 4th result in http://www.google.com/search?q=SunPKCS11+TokenPoller gives source code for that class. Best regards, Konstantin Kolinko - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange context class loader
On 06.05.2010 23:24, Sylvain Laurent wrote: When you analyzed the heap dump, what path to GC roots was retaining the classloader in memory ? I did the heap dump before the context restart. Note: I don't want to debug a memory leak. I'm wondering why the PCKS Token Poller thread was captured by the leak prevention. Since we know the code, it was because its context class loader was equal to the WebappClassLoader of /manager. That's what I don't understand. See my original post. Regards, Rainer On 6 mai 2010, at 20:51, Rainer Jung wrote: While doing some testing with 6.0.26 I noticed, that when shutting it down it logs an error about thread "Poller SunPKCS11-Solaris" not being stopped. When I add the more explicit logging from tc6 trunk, it says the thread was started by /manager. The thread sits in java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:681) at java.lang.Thread.run(Thread.java:619) and is started for some version of the JDK (noted on Solaris for e.g. 1.5.0_22 and 1.6.0_3, not for 1.6.0_17) even without having anything related to keystores, https or similar configured in Tomcat (default config). The only non-default is using Log4J instead of Juli. What is strange, is that the check whether the context class loader of the thread is equal to the WebappClassLoader of the context to unload passes. I added an additional output by explicitely printing the contextName of the two loaders and in fact both print "/manager". When deploying two instances of the manager and reloading the original instance with the additional one, I get the same warning. In a heap dump, it seems the context class loader of the thread is the system class loader and not of type WebappClassLoader. Any idea, why the context class loader test fails? Is there any reason why the context class loader of the thread might change when doing the manager reload or shutdown? Regards, Rainer - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: clearReferencesThreads, Poller SunPKCS11-Solaris and strange context class loader
When you analyzed the heap dump, what path to GC roots was retaining the classloader in memory ? On 6 mai 2010, at 20:51, Rainer Jung wrote: > While doing some testing with 6.0.26 I noticed, that when shutting it down it > logs an error about thread "Poller SunPKCS11-Solaris" not being stopped. When > I add the more explicit logging from tc6 trunk, it says the thread was > started by /manager. > > The thread sits in > > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:681) > at java.lang.Thread.run(Thread.java:619) > > and is started for some version of the JDK (noted on Solaris for e.g. > 1.5.0_22 and 1.6.0_3, not for 1.6.0_17) even without having anything related > to keystores, https or similar configured in Tomcat (default config). The > only non-default is using Log4J instead of Juli. > > What is strange, is that the check whether the context class loader of the > thread is equal to the WebappClassLoader of the context to unload passes. I > added an additional output by explicitely printing the contextName of the two > loaders and in fact both print "/manager". > > When deploying two instances of the manager and reloading the original > instance with the additional one, I get the same warning. > > In a heap dump, it seems the context class loader of the thread is the system > class loader and not of type WebappClassLoader. Any idea, why the context > class loader test fails? Is there any reason why the context class loader of > the thread might change when doing the manager reload or shutdown? > > Regards, > > Rainer > > > - > To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org > For additional commands, e-mail: dev-h...@tomcat.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
clearReferencesThreads, Poller SunPKCS11-Solaris and strange context class loader
While doing some testing with 6.0.26 I noticed, that when shutting it down it logs an error about thread "Poller SunPKCS11-Solaris" not being stopped. When I add the more explicit logging from tc6 trunk, it says the thread was started by /manager. The thread sits in java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:681) at java.lang.Thread.run(Thread.java:619) and is started for some version of the JDK (noted on Solaris for e.g. 1.5.0_22 and 1.6.0_3, not for 1.6.0_17) even without having anything related to keystores, https or similar configured in Tomcat (default config). The only non-default is using Log4J instead of Juli. What is strange, is that the check whether the context class loader of the thread is equal to the WebappClassLoader of the context to unload passes. I added an additional output by explicitely printing the contextName of the two loaders and in fact both print "/manager". When deploying two instances of the manager and reloading the original instance with the additional one, I get the same warning. In a heap dump, it seems the context class loader of the thread is the system class loader and not of type WebappClassLoader. Any idea, why the context class loader test fails? Is there any reason why the context class loader of the thread might change when doing the manager reload or shutdown? Regards, Rainer - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
