Am 20.03.24 um 18:17 schrieb Christopher Schultz:
Rainer,
Thanks for writing this up.
On 3/20/24 07:22, Rainer Jung wrote:
I wanted to share an observation and I hope the things are correct how
I am describing them. Maybe things have already improved and I am not
aware of it, hints welcome.
Part of JEP 425 (Project Loom, Java virtual threads) discusses how to
handle observability of virtual threads from inside the JVM and tooling.
The final outcome is, that virtual threads are not included in the
typical JVM APIs which one can use to observe threads, like
enumerating them or accessing the stacks of individual threads. As a
consequence, also jstack and "kill -QUIT" do not show virtual threads
at all, not even when they are attached to a native thread and
executing code.
O_O
There is one single method to help with observability of virtual threads
https://docs.oracle.com/en/java/javase/21/docs/api/jdk.management/com/sun/management/HotSpotDiagnosticMXBean.html#dumpThreads(java.lang.String,com.sun.management.HotSpotDiagnosticMXBean.ThreadDumpFormat)
which dumps a full thread dump to a file. Of course that is by no
means appropriate, if you want to do a fine grained observation. At
least you can choose to write a json structure instead of a hard to
parse text format, but that's it.
Boy, that's a real miss from the JVM team. I'm surprised that they have
overlooked this. I don't see a real reason that a Virtual Thread would
be treated any differently than a regular thread.
The JEP hints at "since thre could be millions of virtual threads it
makes no sense to support them in the existing JMX APIs". I don't by
this extremely simplistic reasoning.
Furthermore, the "dump to file" method they provided does not contain
any locking information you usually expect in a thread dump.
For instance I am often using a tool, that inspects our
RequestProcessors to find long running requests and then retrieves the
list of Java threads as ThreadInfo objects to find the one executing
the request and finally retrieves the stack of that request from the
JVM. Such an approach is no longer possible. Almost all of JMX does
not show any info about virtual threads.
It also seems Tomcat no longer uses a pool when a connector is
configured to use virtual threads. So there are no metrics like
currentThreadCount or currentThreadsBusy.
When using virtual threads, the number of requests is the same as the
number of threads, since each request -> new Virtual Thread.... though I
think the request-count is only incremented when the request has
completed, so maybe there are threads running that can't be counted (yet).
But I understand that you were hoping to get some information about
these threads and though maybe Tomcat's metrics could help. I guess not
really.
But thread-busy count is the same as in-flight-request count. And
current thread count is the same as in-flight-request count as well.
I guess this also limits the use of some manager features and probably
also the StuckThreadsDetectionValve when combined with virtual threads.
Most likely. I'm fairly sure that uses the "usual Thread-walker things"
which you are reporting do not work any more with VTs.
What JVM are you using for your investigations?
Zulu 21, but I expect the other OpenJDK variants to behave the same way.
It mostly comes out of the JEP.
Of course Tomcat has solved most of the problems, that virtual threads
want to solve, by doing the hard work of using the NIO and NIO2 APIs.
So virtual threads are probably not that important for us. But we
should be aware of the observability deficiencies whenever we would
discuss whether we could switch from NIO or NIO2 to virtual threads to
simplify connector code in some distant future.
+1
I've been enthusiastically talking with markt about dropping all that
nasty NIO stuff and "going back to BIO with virtual threads" which, at
this point, is mostly a threat and not a promise. But if VTs really
deliver everything they are claiming to deliver, then it may be possible
to go back to BIO as long as servlet-async is retired. And I'm not
holding my breath on that one.
There was an interesting talk at FOSDEM about the current limitations of
the virtual thread implementation. They have several situations where
they switch over to blocking. I think for instance file I/O and also
some types of locks they can not yet handle well.
They have some improvements in the pipeline, but I think one would at
least wait for those. And hopefully they improve the observability in
the meantime.
I wish I had known this type of problem at FOSDEM so I could have
directly talked to the OpenJDK devs there.
Thanks and regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org