https://bz.apache.org/bugzilla/show_bug.cgi?id=65553
Bug ID: 65553 Summary: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak Product: Tomcat 9 Version: 9.0.52 Hardware: PC Status: NEW Severity: minor Priority: P2 Component: Catalina Assignee: dev@tomcat.apache.org Reporter: thomas.hoffm...@speed4trade.com Target Milestone: ----- Created attachment 38020 --> https://bz.apache.org/bugzilla/attachment.cgi?id=38020&action=edit Stacktrace on creation of orphaned thread Hello, we are using the org.apache.catalina.realm.JNDIRealm for authentication of users against our windows AD. When undeploying the application, we see the following warning in our logs: WARNING [Catalina-utility-1] org.apache.catalina.loader.Webapp ClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-106] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.base@11.0.3/java.net.SocketInputStream.socketRead0(Native Method) java.base@11.0.3/java.net.SocketInputStream.socketRead(SocketInputStream.java:115) java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:168) java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:140) java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448) java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68) java.base@11.0.3/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1104) java.base@11.0.3/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:823) java.base@11.0.3/java.io.BufferedInputStream.fill(BufferedInputStream.java:252) java.base@11.0.3/java.io.BufferedInputStream.read1(BufferedInputStream.java:292) java.base@11.0.3/java.io.BufferedInputStream.read(BufferedInputStream.java:351) java.naming@11.0.3/com.sun.jndi.ldap.Connection.run(Connection.java:832) java.base@11.0.3/java.lang.Thread.run(Thread.java:834) The warning is not always shown but quite often. Summary of the analysis of the problem: On tomcat startup, the worker-thread is running under the tomcat classloader. But when a reconnect happens, the thread is running with the classloader of the web application and gets thus reported. The details: Digging into the problem via remote debugging showed the reason how this happens: During startup, Tomcat is initializing the JNDIRealm. The open-method of JNDIRealm is switching the classloader between bootstrap-CL and tomcat-lib-CL, depending on the attribute "useContextClassLoader". Afterwards the context-Object is created (createDirContext). Within this LdapCtx, an LdapClient is used to communicate with the AD-Server. This LdapClient uses a com.sun.jndi.ldap.Connection for TCP communication. This connection opens the reported Worker-Thread. This can be seen at https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.naming/share/classes/com/sun/jndi/ldap/Connection.java around line 243 --> worker = Obj.helper.createThread(this); So far, so good. Somehow, the com.sun.jndi.ldap.Connection is sometimes closed and the thread dies. At least, the thread is not visible any more. Maybe because of a timeout on the AD-server side or something else happened. If a new user accesses the site, the JNDIRealm is authenticating the user. This triggers the following chain (path is shortened): JNDIRealm.getUserBySearch --> LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect --> LdapClient.getInstance --> Connection.<init> A detailed stack is attached as an image. This call chain creates a new com.sun.jndi.ldap.Connection and thus a new thread. But this time, the thread is connected to the classloader of the web-application. On undeployment, the thread is thus reported to be orphaned. It was tested with Tomcat 9.0.52, Windows 10, OpenJDK 11.0.12_7. As the authentication is conducted within tomcat, before the application is triggered, thus the application can't handle that error. The flag "useContextClassLoader" is not taken into account during authentication. During this process a re-connect can occur and create the thread with the wrong classloader. Thanks in advance, Thomas -- You are receiving this mail because: You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org