Those are raw threads. And are pointing to the Thread.contextClassLoader. That's not a place a pinned classloader occurs.
Look for something in your webapp's own classes that has a reference to the WebAppClassLoader. The folks at Eclipse MAT have talked about this approach using their MAT project. https://www.eclipse.org/forums/index.php/t/205797/ Mark Thomas (from the tomcat project that also has this problem with hot deploy) has also documented many Java and Javax classes that have this classloader pinning / memory leak as well. https://github.com/markt-asf/memory-leaks/tree/master/src/org/apache/markt/leaks Joakim Erdfelt / [email protected] On Fri, Jun 26, 2020 at 3:33 PM Kiao Desouzza <[email protected]> wrote: > I fired off profiler. > > After couple of uploads of WAR file: > https://imgur.com/a/1BNM1fr > > One of the threads that shows as GC root for stale web app class loader: > https://imgur.com/a/U9s4Y0V > > ... now i hope i have something useful. > > Regards > > On Fri, Jun 26, 2020 at 3:36 PM Jan Bartel <[email protected]> wrote: > >> I wonder if this is the same issue that was reported a few years ago, >> that we could never get to the bottom of: >> https://github.com/eclipse/jetty.project/issues/1425 >> >> Jan >> >> On Fri, 26 Jun 2020 at 15:06, Joakim Erdfelt <[email protected]> wrote: >> >>> Hot reload on Java has many pitfalls. >>> >>> Most of them come from WebApp ClassLoader Pinning. >>> Where a class being loaded from the WebApp is keeping a reference to the >>> WebApp ClassLoader, preventing it from being recycled. >>> >>> We have many LeakPreventers implemented that prevent the WebApp >>> ClassLoader Pinning, for the common JVM and Servlet/Java EE classes, but >>> there are still more in various other 3rd party libraries. >>> >>> See >>> https://www.eclipse.org/jetty/documentation/current/preventing-memory-leaks.html >>> >>> Give them a try, if it works out for you GREAT. >>> But if you still have this issue, then you'll have to figure out which >>> class is holding the reference to the WebAppClassLoader and perhaps >>> implement a LeakPreventere for it to let you have hot reload success. >>> >>> Joakim Erdfelt / [email protected] >>> >>> >>> On Fri, Jun 26, 2020 at 7:54 AM Kiao Desouzza <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> I am having an issue with file system quota tools not reporting >>>> correctly on the directory where jetty is deployed. After some >>>> investigation I think it's down to left over file descriptors like these >>>> (output from lsof): >>>> >>>> java 39467 sclient 5r REG >>>> 8,1 134017 101365281 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/HikariCP-java7-2.4.13.jar >>>> (deleted) >>>> java 39467 sclient 5r REG >>>> 8,1 134017 101365281 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/HikariCP-java7-2.4.13.jar >>>> (deleted) >>>> java 39467 sclient 45r REG >>>> 8,1 69409 101365257 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/activation-1.1.1.jar >>>> (deleted) >>>> java 39467 sclient 46r REG >>>> 8,1 267827 101365259 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/admin_2.13.jar >>>> (deleted) >>>> java 39467 sclient 47r REG >>>> 8,1 499844 101365280 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/c3p0-0.9.5.4.jar >>>> (deleted) >>>> java 39467 sclient 48r REG >>>> 8,1 246918 101365265 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-beanutils-1.9.4.jar >>>> (deleted) >>>> java 39467 sclient 49r REG >>>> 8,1 347669 101365260 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-codec-1.14.jar >>>> (deleted) >>>> java 39467 sclient 50r REG >>>> 8,1 588337 101365276 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-collections-3.2.2.jar >>>> (deleted) >>>> java 39467 sclient 51r REG >>>> 8,1 208475 101365250 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-dbcp2-2.7.0.jar >>>> (deleted) >>>> java 39467 sclient 52r REG >>>> 8,1 146108 101365282 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-digester-1.8.1.jar >>>> (deleted) >>>> java 39467 sclient 53r REG >>>> 8,1 214788 101365256 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-io-2.6.jar >>>> (deleted) >>>> java 39467 sclient 54r REG >>>> 8,1 61829 101365278 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-logging-1.2.jar >>>> (deleted) >>>> java 39467 sclient 55r REG >>>> 8,1 129592 101365268 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-pool2-2.7.0.jar >>>> (deleted) >>>> java 39467 sclient 56r REG >>>> 8,1 186077 102297951 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/commons-validator-1.6.jar >>>> (deleted) >>>> java 39467 sclient 57r REG >>>> 8,1 580870 101365271 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/flyway-core-6.3.3.jar >>>> (deleted) >>>> java 39467 sclient 58r REG >>>> 8,1 1586948 101365273 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/freemarker-2.3.29.jar >>>> (deleted) >>>> java 39467 sclient 59r REG >>>> 8,1 80511 101365272 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/geny_2.13-0.6.0.jar >>>> (deleted) >>>> java 39467 sclient 60r REG >>>> 8,1 219146 101365253 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/javax.mail-api-1.6.2.jar >>>> (deleted) >>>> java 39467 sclient 61r REG >>>> 8,1 25464 101365283 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/jetty-continuation-9.4.28.v20200408.jar >>>> (deleted) >>>> java 39467 sclient 62r REG >>>> 8,1 624036 101365275 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/mchange-commons-java-0.2.15.jar >>>> (deleted) >>>> java 39467 sclient 63r REG >>>> 8,1 927447 101365252 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/postgresql-42.2.10.jar >>>> (deleted) >>>> java 39467 sclient 64r REG >>>> 8,1 134017 1352185 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/HikariCP-java7-2.4.13.jar >>>> (deleted) >>>> java 39467 sclient 65r REG >>>> 8,1 4250 101365255 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scala-collection-compat_2.13-2.1.6.jar >>>> (deleted) >>>> java 39467 sclient 66r REG >>>> 8,1 5865349 101365264 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scala-library-2.13.2.jar >>>> (deleted) >>>> java 39467 sclient 67r REG >>>> 8,1 225338 101365277 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scala-parser-combinators_2.13-1.1.2.jar >>>> (deleted) >>>> java 39467 sclient 68r REG >>>> 8,1 129113 101365279 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scalaj-http_2.13-2.4.2.jar >>>> (deleted) >>>> java 39467 sclient 69r REG >>>> 8,1 3625052 101365254 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scala-reflect-2.13.2.jar >>>> (deleted) >>>> java 39467 sclient 70r REG >>>> 8,1 1801315 101365266 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scalikejdbc-core_2.13-3.4.2.jar >>>> (deleted) >>>> java 39467 sclient 71r REG >>>> 8,1 6504 102297938 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scalikejdbc-interpolation-macro_2.13-3.4.2.jar >>>> (deleted) >>>> java 39467 sclient 72r REG >>>> 8,1 201322 101365263 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scalikejdbc-interpolation_2.13-3.4.2.jar >>>> (deleted) >>>> java 39467 sclient 73r REG >>>> 8,1 19036 101365261 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/scalikejdbc_2.13-3.4.2.jar >>>> (deleted) >>>> java 39467 sclient 74r REG >>>> 8,1 41472 101365274 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/slf4j-api-1.7.30.jar >>>> (deleted) >>>> java 39467 sclient 75r REG >>>> 8,1 35063 101365284 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/stats_collector_2.13-0.1.0-SNAPSHOT.jar >>>> (deleted) >>>> java 39467 sclient 76r REG >>>> 8,1 187761 101365270 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/ujson_2.13-1.1.0.jar >>>> (deleted) >>>> java 39467 sclient 77r REG >>>> 8,1 114284 101365269 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-1577458225866104653.dir/webapp/WEB-INF/lib/upickle-core_2.13-1.1.0.jar >>>> (deleted) >>>> java 39467 sclient 94r REG >>>> 8,1 69409 1220495 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/activation-1.1.1.jar >>>> (deleted) >>>> java 39467 sclient 95r REG >>>> 8,1 267827 1352165 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/admin_2.13.jar >>>> (deleted) >>>> java 39467 sclient 96r REG >>>> 8,1 499844 1352184 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/c3p0-0.9.5.4.jar >>>> (deleted) >>>> java 39467 sclient 97r REG >>>> 8,1 246918 1352170 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-beanutils-1.9.4.jar >>>> (deleted) >>>> java 39467 sclient 98r REG >>>> 8,1 347669 1352166 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-codec-1.14.jar >>>> (deleted) >>>> java 39467 sclient 99r REG >>>> 8,1 588337 1352180 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-collections-3.2.2.jar >>>> (deleted) >>>> java 39467 sclient 100r REG >>>> 8,1 208475 957623 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-dbcp2-2.7.0.jar >>>> (deleted) >>>> java 39467 sclient 101r REG >>>> 8,1 146108 1352186 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-digester-1.8.1.jar >>>> (deleted) >>>> java 39467 sclient 102r REG >>>> 8,1 214788 1212954 >>>> /home/client/tmp/jetty-0_0_0_0-8443-root_war-_-any-17785782056536538882.dir/webapp/WEB-INF/lib/commons-io-2.6.jar >>>> (deleted) >>>> ... >>>> >>>> The file system tools(du) correctly report used disk space but file >>>> system quota is not seeing space being released after each update of >>>> deployed WAR file (used space grows). After I restart jetty file system >>>> quota is updated correctly. It's only JAR files from what I can tell so >>>> could it be related to the class loader in some way? >>>> >>>> FS is xfs on centos8 with pquota (project quota) >>>> >>>> I am using latest release 9.4.30.v20200611 >>>> >>>> Regards >>>> _______________________________________________ >>>> jetty-users mailing list >>>> [email protected] >>>> To unsubscribe from this list, visit >>>> https://www.eclipse.org/mailman/listinfo/jetty-users >>>> >>> _______________________________________________ >>> jetty-users mailing list >>> [email protected] >>> To unsubscribe from this list, visit >>> https://www.eclipse.org/mailman/listinfo/jetty-users >>> >> >> >> -- >> Jan Bartel <[email protected]> >> www.webtide.com >> *Expert assistance from the creators of Jetty and CometD* >> >> _______________________________________________ >> jetty-users mailing list >> [email protected] >> To unsubscribe from this list, visit >> https://www.eclipse.org/mailman/listinfo/jetty-users >> > _______________________________________________ > jetty-users mailing list > [email protected] > To unsubscribe from this list, visit > https://www.eclipse.org/mailman/listinfo/jetty-users >
_______________________________________________ jetty-users mailing list [email protected] To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jetty-users
