Re: Migration JDK11 - Tomcat 9 - NoClassDefFoundError: java/sql/Statement

[Rob Sargent]

On 3/29/22 10:54, Senguttuvan, Gopalakrishnan (CWM-NR) wrote:

Hi Thomas,

Thanks for the response.
This issue occurred when I have start the Tomcat server in Linux-QA machine.


Regards,
Gopalakrishnan S


Double check your runtime environment - something there is out of whack 
I suspect.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Migration JDK11 - Tomcat 9 - NoClassDefFoundError: java/sql/Statement

[Senguttuvan, Gopalakrishnan (CWM-NR)]

Hi Thomas,

Thanks for the response.
This issue occurred when I have start the Tomcat server in Linux-QA machine.


Regards,
Gopalakrishnan S

-Original Message-


From: Thomas Hoffmann (Speed4Trade GmbH) 
[mailto:thomas.hoffm...@speed4trade.com.INVALID] 
Sent: March 26, 22 1:39 AM
To: Tomcat Users List 
Subject: AW: Migration JDK11 - Tomcat 9 - NoClassDefFoundError: 
java/sql/Statement

[External]

Hello,
does this error show up in IntelliJ ?
Seems like there is a bug in IntelliJ:
https://stackoverflow.com/questions/52981800/getting-noclassfoundexception-java-sql-sqlexception-in-intellij-idea-for-jdk-1

java.sql.statement is shipped with Java 11.

Greetings,
Thomsa

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

> -Ursprüngliche Nachricht-
> Von: Senguttuvan, Gopalakrishnan (CWM-NR) 
> 
> Gesendet: Freitag, 25. März 2022 17:13
> An: users@tomcat.apache.org
> Betreff: Migration JDK11 - Tomcat 9 - NoClassDefFoundError:
> java/sql/Statement
>
> Hi Team,
>
> We are migrating our application from JDK8 to JDK11 (RedHat OpenJDK11).
> Modified the JAVA_HOME to JDK11 path.
> Currently we are using Tomcat version 9. (It is working fine with JDK8).
> Since the JDK11 won't support the JAVA_ENDORSED_DIRS, so I have 
> removed the "-Djava.endorsed.dirs" in catalina.sh.
>
> Once removed endorsed entry, got the below exception while start the
> application:
> java.lang.NoClassDefFoundError: java/sql/Statement
> at
> java.xml/com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.close(U
> TF
> 8Reader.java)
> at
> java.xml/com.sun.org.apache.xerces.internal.impl.XMLEntityManager.clos
> eR
> eaders(XMLEntityManager.java:1452)
> at
> java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration
> .cl
> eanup(XML11Configuration.java:803)
> at
> java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration
> .p
> arse(XML11Configuration.java:844)
> at
> java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XM
> L
> Parser.java:141)
> at
> java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.
> par
> se(AbstractSAXParser.java:1216)
> at
> java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAX
> P
> arser.parse(SAXParserImpl.java:635)
> at 
> org.apache.tomcat.util.digester.Digester.parse(Digester.java:1517)
> at
> org.apache.catalina.startup.Catalina.parseServerXml(Catalina.java:584)
> at 
> org.apache.catalina.startup.Catalina.load(Catalina.java:675)
> at 
> org.apache.catalina.startup.Catalina.load(Catalina.java:712)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> et
> hodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> ega
> tingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at 
> org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:302)
> at
> org.apache.catalina.startup.Bootstrap.mainStatement(Bootstrap.java:472
> ) Exception in thread "Thread-0" java.lang.NoClassDefFoundError:
> java/sql/Statement
> at java.base/java.io.PrintWriter.close(PrintWriter.java)
> at 
> org.apache.juli.FileHandler.closeWriter(FileHandler.java:339)
> at org.apache.juli.FileHandler.close(FileHandler.java:327)
> at
> org.apache.juli.ClassLoaderLogManager.resetLoggers(ClassLoaderLogManag
> er.java:397)
> at
> org.apache.juli.ClassLoaderLogManager.shutdown(ClassLoaderLogManager.j
> ava:376)
> at
> org.apache.juli.ClassLoaderLogManager$Cleaner.run(ClassLoaderLogManage
> r.java:80)
>
>
> Kindly help us to resolve this issue and let me know if you need any 
> further details.
>
>
>
>
> Regards,
> Gopalakrishnan S
>

__ This 
email is intended only for the use of the individual(s) to whom it is addressed 
and may be privileged and confidential. Unauthorised use or disclosure is 
prohibited. If you receive this e-mail in error, please advise immediately and 
delete the original message. This message may have been altered without your or 
our knowledge and the sender does not accept any liability for any errors or 
omissions in the message. Emails are monitored by supervisory personnel in 
jurisdictions where monitoring is permitted. Such communications are retained 
and may be produced to regulatory authorities 

Re: AW: AW: AW: AW: Question to possible memory leak by Threadlocal variable

[Christopher Schultz]

Thomas,

On 3/29/22 02:42, Thomas Hoffmann (Speed4Trade GmbH) wrote:

Hello Mark,


-Ursprüngliche Nachricht-
Von: Mark Eggers 
Gesendet: Montag, 28. März 2022 23:55
An: users@tomcat.apache.org
Betreff: Re: AW: AW: AW: Question to possible memory leak by Threadlocal
variable

Thomas:

On 3/28/2022 2:01 PM, Thomas Hoffmann (Speed4Trade GmbH) wrote:

Hello Chris,


-Ursprüngliche Nachricht-
Von: Christopher Schultz 
Gesendet: Montag, 28. März 2022 18:48
An: users@tomcat.apache.org
Betreff: Re: AW: AW: Question to possible memory leak by Threadlocal
variable

Thomas,

On 3/25/22 16:59, Thomas Hoffmann (Speed4Trade GmbH) wrote:

-Ursprüngliche Nachricht-
Von: Christopher Schultz 
Gesendet: Freitag, 25. März 2022 14:05
An: users@tomcat.apache.org
Betreff: Re: AW: Question to possible memory leak by Threadlocal
variable

Thomas,

On 3/24/22 05:49, Thomas Hoffmann (Speed4Trade GmbH) wrote:




-Ursprüngliche Nachricht-
Von: Mark Thomas 
Gesendet: Donnerstag, 24. März 2022 09:32
An: users@tomcat.apache.org
Betreff: Re: Question to possible memory leak by Threadlocal
variable

On 24/03/2022 07:57, Thomas Hoffmann (Speed4Trade GmbH)

wrote:





Is it correct, that every spawned thread must call tl.remove()
to cleanup all

the references to prevent the logged warning (and not only the
main thread)?

Yes. Or the threads need to exit.


Second question is: How might it cause a memory leak?
The threads are terminated and hold a reference to this static
variable. But

on the other side, that class A is also eligible for garbage
collection after undeployment.

So both, the thread class and the class A are ready to get
garbage collected. Maybe I missed something (?)


It sounds as if the clean-up is happening too late. Tomcat
expects clean-up to be completed once contextDestroyed() has
returned for all ServLetContextListeners. If the clean-up is
happening asynchronously

(e.g.

the call is made to stop the threads but doesn't wait until the
threads have
stopped) you could see this message.

In this case it sounds as if you aren't going to get a memory
leak but Tomcat can't tell that at the point it checks.

Mark

-
--
-- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


Hello Mark,
thanks for the information.
The shutdown of the framework is currently placed within the
destroy()

method of a servlet (with load on startup).

At least the debugger shows that servlet-->destroy() is executed
before

the method checkThreadLocalMapForLeaks() runs.

I will take a look, whether the threads already exited.


Tomcat only checks its own request-processing threads for
ThreadLocals, so any threads created by the application or that
library are unrelated to the warning you are seeing.

Any library which saves ThreadLocals from request-processing
threads is going to have this problem if the objects are of types
loaded by the webapp ClassLoader.

There are a few ways to mitigate this, but they are ugly and it
would be better if the library didn't use ThreadLocal storage, or
if it would use vanilla classes from java.* and not its own types.

You say that those objects are eligible for GC after the library
shuts down, but that's not true: anything you stick in ThreadLocal
storage

is being held ...

by the ThreadLocal storage and won't be GC'd. If an object can't be
collected, the java.lang.Class defining it can't be collected, and
therefore the ClassLoader which loaded it (the webapp
ClassLoader) can't be free'd. We call this a "pinned ClassLoader"
and it still contains all of the java.lang.Class instances that the
ClassLoader ever loaded during its lifetime. If you reload
repeatedly, you'll see un-collectable ClassLoader instances piling
up in memory which is
*definitely* a leak.

The good news for you is that Tomcat has noticed the problem and
will, over time, retire and replace each of the affected Threads in
its request- processing thread pool. As those Thread objects are
garbage-collected, the TheradLocal storage for each is also
collected, etc. and *eventually* your leak will be resolved. But it
would be

better not to have one in the first place.


Why not name the library? Why anonymize the object type if it's
org.apache.something?

-chris


Hello Chris,
I didn't want to blame any library  But as you ask for it, I send
more

details.


Regarding the ThreadLocal thing:
I thought that the threadlocal variables are stored within the
Thread-class in the member variable "ThreadLocal.ThreadLocalMap
threadLocals":
https://github.com/AdoptOpenJDK/openjdk-

jdk11/blob/master/src/java.bas

e/share/classes/java/lang/Thread.java

So I thought, when the thread dies, these variables will also be
released and automatically removed from the ThreadLocal variable /
instance (?)

This is correct, but if the ThreadLocal is being stored in the
request- processing thread, then when 

AW: AW: AW: Question to possible memory leak by Threadlocal variable

[Thomas Hoffmann (Speed4Trade GmbH)]

Hello Chris,

> -Ursprüngliche Nachricht-
> Von: Christopher Schultz 
> Gesendet: Montag, 28. März 2022 18:48
> An: users@tomcat.apache.org
> Betreff: Re: AW: AW: Question to possible memory leak by Threadlocal
> variable
> 
> Thomas,
> 
> On 3/25/22 16:59, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> >> -Ursprüngliche Nachricht-
> >> Von: Christopher Schultz 
> >> Gesendet: Freitag, 25. März 2022 14:05
> >> An: users@tomcat.apache.org
> >> Betreff: Re: AW: Question to possible memory leak by Threadlocal
> >> variable
> >>
> >> Thomas,
> >>
> >> On 3/24/22 05:49, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> >>>
> >>>
>  -Ursprüngliche Nachricht-
>  Von: Mark Thomas 
>  Gesendet: Donnerstag, 24. März 2022 09:32
>  An: users@tomcat.apache.org
>  Betreff: Re: Question to possible memory leak by Threadlocal
>  variable
> 
>  On 24/03/2022 07:57, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> 
>  
> 
> > Is it correct, that every spawned thread must call tl.remove() to
> > cleanup all
>  the references to prevent the logged warning (and not only the main
>  thread)?
> 
>  Yes. Or the threads need to exit.
> 
> > Second question is: How might it cause a memory leak?
> > The threads are terminated and hold a reference to this static
> > variable. But
>  on the other side, that class A is also eligible for garbage
>  collection after undeployment.
> > So both, the thread class and the class A are ready to get garbage
> > collected. Maybe I missed something (?)
> 
>  It sounds as if the clean-up is happening too late. Tomcat expects
>  clean-up to be completed once contextDestroyed() has returned for
>  all ServLetContextListeners. If the clean-up is happening
>  asynchronously
> >> (e.g.
>  the call is made to stop the threads but doesn't wait until the
>  threads have
>  stopped) you could see this message.
> 
>  In this case it sounds as if you aren't going to get a memory leak
>  but Tomcat can't tell that at the point it checks.
> 
>  Mark
> 
>  ---
>  -- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>  For additional commands, e-mail: users-h...@tomcat.apache.org
> >>>
> >>> Hello Mark,
> >>> thanks for the information.
> >>> The shutdown of the framework is currently placed within the
> >>> destroy()
> >> method of a servlet (with load on startup).
> >>> At least the debugger shows that servlet-->destroy() is executed
> >>> before
> >> the method checkThreadLocalMapForLeaks() runs.
> >>> I will take a look, whether the threads already exited.
> >>
> >> Tomcat only checks its own request-processing threads for
> >> ThreadLocals, so any threads created by the application or that
> >> library are unrelated to the warning you are seeing.
> >>
> >> Any library which saves ThreadLocals from request-processing threads
> >> is going to have this problem if the objects are of types loaded by
> >> the webapp ClassLoader.
> >>
> >> There are a few ways to mitigate this, but they are ugly and it would
> >> be better if the library didn't use ThreadLocal storage, or if it
> >> would use vanilla classes from java.* and not its own types.
> >>
> >> You say that those objects are eligible for GC after the library
> >> shuts down, but that's not true: anything you stick in ThreadLocal storage
> is being held ...
> >> by the ThreadLocal storage and won't be GC'd. If an object can't be
> >> collected, the java.lang.Class defining it can't be collected, and
> >> therefore the ClassLoader which loaded it (the webapp
> >> ClassLoader) can't be free'd. We call this a "pinned ClassLoader" and
> >> it still contains all of the java.lang.Class instances that the
> >> ClassLoader ever loaded during its lifetime. If you reload
> >> repeatedly, you'll see un-collectable ClassLoader instances piling up
> >> in memory which is
> >> *definitely* a leak.
> >>
> >> The good news for you is that Tomcat has noticed the problem and
> >> will, over time, retire and replace each of the affected Threads in
> >> its request- processing thread pool. As those Thread objects are
> >> garbage-collected, the TheradLocal storage for each is also
> >> collected, etc. and *eventually* your leak will be resolved. But it would 
> >> be
> better not to have one in the first place.
> >>
> >> Why not name the library? Why anonymize the object type if it's
> >> org.apache.something?
> >>
> >> -chris
> >
> > Hello Chris,
> > I didn't want to blame any library  But as you ask for it, I send more
> details.
> >
> > Regarding the ThreadLocal thing:
> > I thought that the threadlocal variables are stored within the
> > Thread-class in the member variable "ThreadLocal.ThreadLocalMap
> > threadLocals":
> > https://github.com/AdoptOpenJDK/openjdk-
> jdk11/blob/master/src/java.bas
> > 

AW: AW: AW: AW: Question to possible memory leak by Threadlocal variable

[Thomas Hoffmann (Speed4Trade GmbH)]

Hello Mark,

> -Ursprüngliche Nachricht-
> Von: Mark Eggers 
> Gesendet: Montag, 28. März 2022 23:55
> An: users@tomcat.apache.org
> Betreff: Re: AW: AW: AW: Question to possible memory leak by Threadlocal
> variable
> 
> Thomas:
> 
> On 3/28/2022 2:01 PM, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> > Hello Chris,
> >
> >> -Ursprüngliche Nachricht-
> >> Von: Christopher Schultz 
> >> Gesendet: Montag, 28. März 2022 18:48
> >> An: users@tomcat.apache.org
> >> Betreff: Re: AW: AW: Question to possible memory leak by Threadlocal
> >> variable
> >>
> >> Thomas,
> >>
> >> On 3/25/22 16:59, Thomas Hoffmann (Speed4Trade GmbH) wrote:
>  -Ursprüngliche Nachricht-
>  Von: Christopher Schultz 
>  Gesendet: Freitag, 25. März 2022 14:05
>  An: users@tomcat.apache.org
>  Betreff: Re: AW: Question to possible memory leak by Threadlocal
>  variable
> 
>  Thomas,
> 
>  On 3/24/22 05:49, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> >
> >
> >> -Ursprüngliche Nachricht-
> >> Von: Mark Thomas 
> >> Gesendet: Donnerstag, 24. März 2022 09:32
> >> An: users@tomcat.apache.org
> >> Betreff: Re: Question to possible memory leak by Threadlocal
> >> variable
> >>
> >> On 24/03/2022 07:57, Thomas Hoffmann (Speed4Trade GmbH)
> wrote:
> >>
> >> 
> >>
> >>> Is it correct, that every spawned thread must call tl.remove()
> >>> to cleanup all
> >> the references to prevent the logged warning (and not only the
> >> main thread)?
> >>
> >> Yes. Or the threads need to exit.
> >>
> >>> Second question is: How might it cause a memory leak?
> >>> The threads are terminated and hold a reference to this static
> >>> variable. But
> >> on the other side, that class A is also eligible for garbage
> >> collection after undeployment.
> >>> So both, the thread class and the class A are ready to get
> >>> garbage collected. Maybe I missed something (?)
> >>
> >> It sounds as if the clean-up is happening too late. Tomcat
> >> expects clean-up to be completed once contextDestroyed() has
> >> returned for all ServLetContextListeners. If the clean-up is
> >> happening asynchronously
>  (e.g.
> >> the call is made to stop the threads but doesn't wait until the
> >> threads have
> >> stopped) you could see this message.
> >>
> >> In this case it sounds as if you aren't going to get a memory
> >> leak but Tomcat can't tell that at the point it checks.
> >>
> >> Mark
> >>
> >> -
> >> --
> >> -- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >
> > Hello Mark,
> > thanks for the information.
> > The shutdown of the framework is currently placed within the
> > destroy()
>  method of a servlet (with load on startup).
> > At least the debugger shows that servlet-->destroy() is executed
> > before
>  the method checkThreadLocalMapForLeaks() runs.
> > I will take a look, whether the threads already exited.
> 
>  Tomcat only checks its own request-processing threads for
>  ThreadLocals, so any threads created by the application or that
>  library are unrelated to the warning you are seeing.
> 
>  Any library which saves ThreadLocals from request-processing
>  threads is going to have this problem if the objects are of types
>  loaded by the webapp ClassLoader.
> 
>  There are a few ways to mitigate this, but they are ugly and it
>  would be better if the library didn't use ThreadLocal storage, or
>  if it would use vanilla classes from java.* and not its own types.
> 
>  You say that those objects are eligible for GC after the library
>  shuts down, but that's not true: anything you stick in ThreadLocal
>  storage
> >> is being held ...
>  by the ThreadLocal storage and won't be GC'd. If an object can't be
>  collected, the java.lang.Class defining it can't be collected, and
>  therefore the ClassLoader which loaded it (the webapp
>  ClassLoader) can't be free'd. We call this a "pinned ClassLoader"
>  and it still contains all of the java.lang.Class instances that the
>  ClassLoader ever loaded during its lifetime. If you reload
>  repeatedly, you'll see un-collectable ClassLoader instances piling
>  up in memory which is
>  *definitely* a leak.
> 
>  The good news for you is that Tomcat has noticed the problem and
>  will, over time, retire and replace each of the affected Threads in
>  its request- processing thread pool. As those Thread objects are
>  garbage-collected, the TheradLocal storage for each is also
>  collected, etc. and *eventually* your leak will be resolved. But it
>  would be
> >> better not