Jakarta EE 11 Release Delayed
Good morning, I received an update from the Jakarta EE Community mailing list this morning that the Jakarta EE 11 final release will be pushed out a quarter to deal with platform TCK migration to Arquillian/Junit 5. The exact target date is still TBD. I am assuming this also pushes Apache Tomcat 11’s final release as well? Regards, William Crowell This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: Virtual Thread with Http11Nio2Protocol
Nicolas, Which JDK version? Regards, William Crowell From: Nicolas BONAMY Date: Thursday, December 7, 2023 at 2:55 PM To: users@tomcat.apache.org Subject: Virtual Thread with Http11Nio2Protocol Hi, I try to use virtual thread on Apache Tomcat 10.1.16 with this configuration on macOS or on Linux: But when I make a request, I'm not on a virtual thread : Thread[#76,Thread-14,5,main] . I profiled my application too but no virtual threads are used. If I use a Http11NioProtocol instead of Http11Nio2Protocol, all requests are on virtual thread : VirtualThread[#65,http-nio-8080-virt-0]/runnable@ForkJoinPool-1-worker-1 Http11Nio2Protocol is not working with virtual threads? Has anyone encountered this problem before? CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe. This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: Thread Pool Question
I should clarify the ask here… I have some long running JDBC queries against Oracle, and I do not want to tie up Tomcat’s web thread pool with them. I would only have between 1-10 threads in this pool. Regards, William Crowell This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Thread Pool Question
If I create a separate thread pool in Tomcat’s server.xml like this: Then how do I get a thread to assign any work to it? Regards, William Crowell This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: Are there any known class loader leaks in Tomcat 9?
Olaf and Sevendu, Thank you for your replies. Correct, I sincerely doubt this is a Tomcat class loading bug. I am using Tomcat’s normal class loader (webapp/WAR) to load the classes into memory, and it is a single class loader. I am going to periodically run: jcmd GC.class_stats I am only having the issue in general operation and not on a redeploy, and I have to restart the server daily. I will find out more details as to how these classes are loaded into memory. Here is what is going on… I have a 16GB Linux instance and one Apache Tomcat 9.0.78 instance running on it. It is running JDK 1.8.0_371-b25. I have min and max JVM heap setting at 8GB. JVM arguments are: -XX:CICompilerCount=3 -XX:ConcGCThreads=1 -XX:+DisableExplicitGC -XX:G1HeapRegionSize=4194304 -XX:GCLogFileSize=3145728 -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=8589934592 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=8589934592 -XX:MaxNewSize=5150605312 -XX:MinHeapDeltaBytes=4194304 -XX:NativeMemoryTracking=detail -XX:NumberOfGCLogFiles=25 -XX:+PrintGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:+UseG1GC -XX:+UseGCLogFileRotation The MaxMetaspaceSize is not set, so this means it is unlimited. After the server comes up and after a short period of time I get a native out of memory condition: # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 8589934592 bytes for committing reserved memory. # Possible reasons: # The system is out of physical RAM or swap space # The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full # Decrease Java heap size (-Xmx/-Xms) # Decrease number of Java threads # Decrease Java thread stack sizes (-Xss) # Set larger code cache with -XX:ReservedCodeCacheSize= # This output file may be truncated or incomplete. # # Out of Memory Error (os_linux.cpp:2798), pid=191803, tid=0x14fff94b7700 # # JRE version: (8.0_371) (build ) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.371-b25 mixed mode linux-amd64 compressed oops) # Core dump written. Default location: /hosting/core or core.191803 # Regards, William Crowell From: Olaf Kock Date: Tuesday, November 7, 2023 at 4:25 AM To: users@tomcat.apache.org Subject: Re: Are there any known class loader leaks in Tomcat 9? On 06.11.23 18:55, William Crowell wrote: > Good afternoon, > > I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am > loading some very large Java classes into Metaspace. I know this is not good > practice, but I inherited this library. These classes have business rules > and are doing some basic primitive and array manipulations, and I am running > out of native memory. I don’t think there are any JNI calls in this code > base. > > Are there anything existing issues with the Tomcat 9 class loader? I doubt > there are but wanted to check. Hi William, when you say "I am loading some very large Java classes into Metaspace. I know this is not good practice" it does not sound like this inherited library is using tomcat's standard way of classloading - please clarify. Also, nothing in that statement points to anything where a leak on tomcat's side would make a difference: Are you experiencing problems after reloading a webapp, or just in general operation? You might simply need to assign more metaspace to the VM for this library to operate properly. So, this boils down to: * How does this library load the classes into memory? * Do you redeploy? * What's your metaspace (and general memory) setting, and what are the conditions under which you run out of memory? * Do you see any log message that hint at classes that won't be garbage collected because of stale references? Tomcat issues those warnings, in case you (for example) start your own background threads that keep holding the memory allocated to them. Also: I second your doubt about Tomcat's classloader being the cause for your problems - unless proven otherwise. Olaf CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe. This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Are there any known class loader leaks in Tomcat 9?
Good afternoon, I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am loading some very large Java classes into Metaspace. I know this is not good practice, but I inherited this library. These classes have business rules and are doing some basic primitive and array manipulations, and I am running out of native memory. I don’t think there are any JNI calls in this code base. Are there anything existing issues with the Tomcat 9 class loader? I doubt there are but wanted to check. Regards, William Crowell This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: Virtual Thread Configuration In Tomcat 11
When I set -Djdk.tracePinnedThreads=short, then I see this: … Thread[#41,ForkJoinPool-1-worker-4,5,CarrierThreads] com.mysql.cj.jdbc.ConnectionImpl.isValid(ConnectionImpl.java:2516) <== monitors:1 Thread[#39,ForkJoinPool-1-worker-2,5,CarrierThreads] com.mysql.cj.jdbc.ConnectionImpl.isValid(ConnectionImpl.java:2516) <== monitors:1 Thread[#41,ForkJoinPool-1-worker-4,5,CarrierThreads] com.mysql.cj.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:2005) <== monitors:1 Thread[#43,ForkJoinPool-1-worker-5,5,CarrierThreads] com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:893) <== monitors:1 com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1061) <== monitors:1 com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1009) <== monitors:1 Thread[#43,ForkJoinPool-1-worker-5,5,CarrierThreads] com.mysql.cj.jdbc.ConnectionImpl.commit(ConnectionImpl.java:791) <== monitors:1 Thread[#43,ForkJoinPool-1-worker-5,5,CarrierThreads] com.mysql.cj.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:2005) <== monitors:1 … I started digging into the MySQL client code: https://github.com/mysql/mysql-connector-j/blob/8.0.33/src/main/user-impl/java/com/mysql/cj/jdbc/ConnectionImpl.java This class is littered with synchronized blocks, so now this makes sense to me why virtual threads would take longer than a regular OS thread. Regards, William Crowell This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: Virtual Thread Configuration In Tomcat 11
Chris, I did set -Djdk.tracePinnedThreads=full and found there were a few pinned threads: Thread[#43,ForkJoinPool-1-worker-4,5,CarrierThreads] java.base/java.lang.VirtualThread$VThreadContinuation.onPinned(VirtualThread.java:185) java.base/jdk.internal.vm.Continuation.onPinned0(Continuation.java:393) java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:631) java.base/java.lang.System$2.parkVirtualThread(System.java:2648) java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67) java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408) … com.mysql.cj.protocol.FullReadInputStream.readFully(FullReadInputStream.java:64) com.mysql.cj.protocol.a.SimplePacketReader.readHeaderLocal(SimplePacketReader.java:81) … com.mysql.cj.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:2005) <== monitors:1 com.zaxxer.hikari.pool.ProxyConnection.setAutoCommit(ProxyConnection.java:401) com.zaxxer.hikari.pool.HikariProxyConnection.setAutoCommit(HikariProxyConnection.java) … Over the 12 test runs I have performed, I found a 53% decrease in throughput when using virtual threads: http://ec2-18-188-185-212.us-east-2.compute.amazonaws.com:8080/web-report/ My example code is located here: https://github.com/wcrowell/rest-mysql-app/ The JMX test I used is located here: https://github.com/wcrowell/rest-mysql-app/blob/main/virtual-thread-test.jmx This test makes a POST request to save a record to MySQL: http://ec2-18-117-196-158.us-east-2.compute.amazonaws.com:8080/vt/save Regards, William Crowell This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: Virtual Thread Configuration In Tomcat 11
Chris, Great post earlier today! This is a super interesting topic to me. You can find the performance testing results located here: http://ec2-18-188-185-212.us-east-2.compute.amazonaws.com:8080/web-report/ I did 10 runs with 1000 threads with a ramp up time of 3 seconds for a duration of 200 seconds. Ignore the 11th run. This is a really simple REST application with Spring. It just makes an insert into a MySQL table. When I get a moment I will put my code into GitHub, but again, there is not much to it. You can find a very similar Spring Boot example with embedded Tomcat here: https://medium.com/@anil.java.story/embracing-virtual-threads-in-spring-boot-4140d3b8a5a But he uses JDK 20 with --enable-preview turned on. The mistake I think I made was that the inserts are not really blocking because they happen so quickly. Maybe because I did not put enough load in the test? I have no synchronized blocks in my code. So, it appeared to me that the inserts were taking twice the amount of time with virtual threads. I think that extra time is just overhead of pinning virtual threads to a platform thread and managing the ForkJoinPool. What is going to happen is that people are going to “flip the switch”…useVirtualThreads=”true”. They will find that performance will not be as good as using an operating system thread and abandon virtual threads because their code does not block. So, I need to come up with an example where my code blocks and redo the perf test and prove this out. I don’t know. Regards, William Crowell This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: Virtual Thread Configuration In Tomcat 11
I did some performance testing with virtual threads on Apache Tomcat 11.0.0-M10 and JDK 21 (21+35-2513). I have a simple REST service using Spring 6.0.11 that does an insert into MySQL 8.0.32. I have 3 separate boxes all running Rocky Linux 9.2 on AWS (t3a.xlarge which is 4 vCPUs and 16GiB RAM): 1) An application server running Tomcat 11.0.0-M10 with JDK 21 2) MySQL 8.0.32 3) JMeter 5.6.2 I know that JDK 21 is nowhere near GA, but I got some interesting results. I did 10 test runs with useVirtualThreads=“true”, and 10 test runs without virtual threads (the default). Each run used 1000 threads in JMeter for about 3 minutes and 20 seconds for a ramp up time of 3 seconds. What I found was using platform threads performed at least twice as fast as virtual threads. Maybe I am interpreting the results wrong, but this is what I found consistently across each test. Again, I realize that performance is not the only goal of virtual threads. Just my observations. Regards, William Crowell This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: Virtual Thread Configuration In Tomcat 11
Mark, Thanks for your reply. Just to clarify…this is all I need in Tomcat 11’s server.xml (as well as JDK21): … … Regards, William Crowell This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Virtual Thread Configuration In Tomcat 11
Hi, To use virtual threads in Tomcat 11, do you need to configure anything with the connectors in server.xml, or does Tomcat 11 use virtual threads by default (with JDK 21)? Is there any documentation as of yet to configure virtual threads? The only samples I have seen are with Spring Boot using the embedded Tomcat server. Thanks in advance, William Crowell This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: Tomcat 11 Release Date
Mark, Thank you sir and will let you know on additional features. Have a great weekend. Regards, Bill Crowell | Enterprise Architect, OpenLogic<https://www.openlogic.com/?utm_leadsource=email-signature_source=outlook-direct-email_medium=email_campaign=2019-common_content=email-signature-link> Perforce Software<http://www.perforce.com/?utm_leadsource=email-signature_source=outlook-direct-email_medium=email_campaign=2019-common_content=email-signature-link> P: +1 720 536 0699 Visit us on: LinkedIn<https://www.linkedin.com/company/perforce?utm_leadsource=email-signature_source=outlook-direct-email_medium=email_campaign=2019-common_content=email-signature-link> | Twitter<https://twitter.com/perforce?utm_leadsource=email-signature_source=outlook-direct-email_medium=email_campaign=2019-common_content=email-signature-link> | Facebook<https://www.facebook.com/perforce/?utm_leadsource=email-signature_source=outlook-direct-email_medium=email_campaign=2019-common_content=email-signature-link> | YouTube<https://www.youtube.com/user/perforcesoftware?utm_leadsource=email-signature_source=outlook-direct-email_medium=email_campaign=2019-common_content=email-signature-link> From: Mark Thomas Date: Friday, June 9, 2023 at 4:36 AM To: users@tomcat.apache.org Subject: Re: Tomcat 11 Release Date On 08/06/2023 21:37, William Crowell wrote: > Mark, > > Thank you for your reply. Any ideas on Tomcat 11 features that will be > included with a final release next year? Virtual thread support was the the first major new feature and that was included in the recent 11.0.0-M7 release. There will probably be more loom related changes as it starts being used and bottlenecks are discovered. We hoped to take advantage of project Panama to simplify OpenSSL integration but that looks like it will remain in preview for Java 21. QUIC is still on the radar but without Panama support it looks unlikely that we'll add QUIC support to Tomcat 11. The Jakarta spec changes look to be mostly about clarifying grey areas and small feature improvements. Other than that, the best thing to do is follow the change log and if there are features you'd like to see, speak up. Mark# > > Regards, > > William Crowell > > From: Mark Thomas > Date: Thursday, June 8, 2023 at 4:05 PM > To: users@tomcat.apache.org > Subject: Re: Tomcat 11 Release Date > On 08/06/2023 19:56, William Crowell wrote: >> Good afternoon, >> >> I know we are in milestone 7 of Apache Tomcat 11, but do any of you know >> when the release date will be for Apache Tomcat 11? It does not have to be >> an exact date, but just a ballpark date. > > The criteria for a stable release are: > - All implemented Jakarta specifications are released > - Tomcat has fully implemented all the specifications > - Committer judgement > > Expanding on those, my best guess for the Jakarta EE specs to be > released is March. Tomcat follows the Jakarta development pretty closely > so I am fairly confident that Tomcat will have fully implemented the > relevant Jakarta EE 11 APIs long before the specs are finally released. > > The committer judgement has changed over the years. Back when the major > Tomcat versions had significantly different code bases, we wanted to > allow time for bugs to be found before declaring a major release stable. > Currently we try and keep the code bases more in-line and are actively > looking for opportunities to reduce differences where we can. The major > benefit of this is a greater degree of confidence in the code. My best > guess in that the committers will declare Tomcat 11 stable a few Tomcat > release cycles after the Jakarta EE 11 release. > > All that adds up to somewhere around June 2024 as a best guess at the > moment. > > HTH, > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > CAUTION: This email originated from outside of the organization. Do not click > on links or open attachments unless you recognize the sender and know the > content is safe. > > > This e-mail may contain information that is privileged or confidential. If > you are not the intended recipient, please delete the e-mail and any > attachments and notify us immediately. > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe. This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: Tomcat 11 Release Date
Mark, Thank you for your reply. Any ideas on Tomcat 11 features that will be included with a final release next year? Regards, William Crowell From: Mark Thomas Date: Thursday, June 8, 2023 at 4:05 PM To: users@tomcat.apache.org Subject: Re: Tomcat 11 Release Date On 08/06/2023 19:56, William Crowell wrote: > Good afternoon, > > I know we are in milestone 7 of Apache Tomcat 11, but do any of you know when > the release date will be for Apache Tomcat 11? It does not have to be an > exact date, but just a ballpark date. The criteria for a stable release are: - All implemented Jakarta specifications are released - Tomcat has fully implemented all the specifications - Committer judgement Expanding on those, my best guess for the Jakarta EE specs to be released is March. Tomcat follows the Jakarta development pretty closely so I am fairly confident that Tomcat will have fully implemented the relevant Jakarta EE 11 APIs long before the specs are finally released. The committer judgement has changed over the years. Back when the major Tomcat versions had significantly different code bases, we wanted to allow time for bugs to be found before declaring a major release stable. Currently we try and keep the code bases more in-line and are actively looking for opportunities to reduce differences where we can. The major benefit of this is a greater degree of confidence in the code. My best guess in that the committers will declare Tomcat 11 stable a few Tomcat release cycles after the Jakarta EE 11 release. All that adds up to somewhere around June 2024 as a best guess at the moment. HTH, Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe. This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Tomcat 11 Release Date
Good afternoon, I know we are in milestone 7 of Apache Tomcat 11, but do any of you know when the release date will be for Apache Tomcat 11? It does not have to be an exact date, but just a ballpark date. Regards, William Crowell This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
