Romain,
On 3/20/24 16:25, Romain Manni-Bucau wrote:
Chris, added some comments inline.
Le mer. 20 mars 2024 à 19:41, Christopher Schultz <
ch...@christopherschultz.net> a écrit :
Romain,
On 3/20/24 13:34, Romain Manni-Bucau wrote:
Thread dumps being dump of threads - literally os threads - and virtual
threads not being threads at all - they are runnables in a dedicated
thread
pool - it is quite fair to not make them the same and have their
scheduler
- pool - only in the thread dump but not themselves no?
Maybe.
If you take a thread dump today (with a "real" thread), you only get a
Java stack trace, you get no "native stack trace" or anything like that.
So from that perspective, the "thread" is really the instance of
java.lang.Thread which could just as easily be a Virtual Thread.
You also get no scheduling information, other than what the thread's
"priority" is... but you can't get any real-time data about where it
sits in the scheduling queue, etc.
Well, this does not work like that, as mentionned it is a ForkJoinPool so
not a plain queue - except the design which is not a 1 task = 1 position,
it has multiple queues - so the position in "the queue" does not give you
the information you are mentionned there.
I was using "scheduling queue" as an abstract idea, there. I know it's a
work-stealing queue which is not straightforward to describe.
If you really care about monitoring this pool you can just
instrument java.lang.VirtualThread#DEFAULT_SCHEDULER (plain reflection
works good enough if you open this part but agents too).
This is Rainer's complaint (which I agree with): there are no standard
-- available -- APIs for this kind of thing. Use of reflection for
instrumentation is an anti-pattern IMHO.
The JRE team over the years has learned that instrumentation is a vital
part of application monitoring and has done a really great job of
exposing JRE internals through standard APIs. This one is a sore spot
that really needs to be fixed.
I'm much less interested in what the "native thread" is doing _below_
the Java part. Presumably, it's always running
Thread.cpp::runTheJavaCode and that's not useful information to anybody.
Depends how you instrument/query it, while at code level it does not change
much things and you still get the thread stack context.
... unless you can't get a reference to the threads in the first place,
which is what Rainer is saying.
For raw thread dumps you indeed need the new jcmd command
(Thread.dump_to_file) and I agree there is no real point to not let it go
in the plain jstack with a new toggle you can enable at need (but hopefully
not by default since we'll end up with undownloadable/too big dumps).
I don't know why this is any different or worse than taking a heap dump.
Heap dumps have been available for decades and nobody says "OMG what
will we do with a 16GiB heap dump?!" You only take a heap-dump when you
NEED it, and the same is true for a thread dump. If you want a thread
dump and your system can't log it all because you have 100M threads,
well, then you have made some bad choices.
If there existed an API to query (virtual) threads, your application or
some tool could examine the threads and only log those you care about.
That would solve BOTH problems.
Ultimately using reflection and opening jdk.internal.vm package you can
also just use ThreadContainers.root() and iterate over children()/threads()
but not sure how portable it will be.
There is a negative effect there, before we were often decorating executors
(the Runnable to execute) to add before/after context and potentially track
threads there but it does not work anymore since threads are totally
unbounded.
If the Virtual Thread is not mounted on an OS thread, then it's
"suspended" or "blocked" or whatever-it-is. If it's on an OS thread, it
had better be running: that's the whole point of the scheme. (I suppose
it could be BLOCKED-yet-on-an-OS-thread -- one of the current problems
which hopefully will be less of a problem in the future, but I'd like to
ignore that for now).
I don't know what's wrong with having millions of threads, and still
being able to walk through them using existing APIs. Nobody walks
through threads for no good reason... if you want to see the threads,
you should be able to see the threads.
If I wanted to walk-through every instance of java.lang.String in a JVM
(¡millions of them!), then I should be able to do it even if (a) it's a
weird thing to do and (b) it will take a while. Well, I want to do it,
so let me do it(!).
If you have only one "real" Thread (e.g. the main thread) and everything
else is Virtual... when you ask for a thread dump or walk all the
threads, do you only see "main" and not the mounted-on-OS-threads
Virtual Threads? If the JVM were willing to consider Virtual Threads
"visible" and therefore dumpable, etc. through existing interfaces
_while they were mounted on an OS thread_ I could almost agree that
makes some sense. But if I have a unit of work not making any progress,
I'm gonna want to be able to see that (Virtual) Thread in a thread dump
or any other kind of analysis tool, including its stack trace.
Similarly to why threadlocal are not recommended for virtual thread this
would probably make an useless pressure on the JVM IMHO - why there is an
option to see it but it is mainly for debug purposes.
See virtual threads as continuations (suspendable/resumable "Runnable")
in
a dedicated and not programmatically configurable nor
selectable/proviable
thread pool, they are not in thread dumps and this doesnt bother you ;) -
ultimately if you want it you want all java objects to be monitored and
in
the thread dump which would be weird - but I agree the semantic is
misleading.
I'm not sure why ThreadLocals are not recommended for Virtual Threads.
Honestly, the presence ThreadLocal is a gigantic hack for badly-written
APIs but if you accept that fact, there doesn't seem to be anything
really bad about it. It "doesn't make sense" for Virtual Threads because
it's wasteful: you probably use something once or twice and don't get
the benefit of it when the thread dies -- because you aren't supposed to
use thread pools anymore. But it's no less wasteful and dumb than it was
in the first place.
Guess the issue is they are bound to low level threads and for virtual
threads this consumes too much mem and since they are shared between
standard threads and virtual threads you can't optimize them by backing
them by scopes like that, so let's blame the "history" maybe?
I honestly haven't looked at how ThreadLocal is managed. with Virtual
Threads. Does anybody use multi-megabyte ThreadLocals? If not, it
doesn't matter. The whole point of VTs is that yu fire-off a unit of
work and forget about it forever. The thread eventually dies and you get
your memory back.
If you use too much memory that way, VT isn't the solution to the
problem you are trying to solve.
IIRC, Tomcat uses ThreadLocal to store date-formatting objects for
access logs. That's a great use for ThreadLocal. If the threads are no
longer pooled, it's "wasteful" in that we create and discard a
SimpleDateFormat for every log, but what's the alternative? Use a shared
SimpleDateFormat and synchronize across threads? Yuck. So it's up to
Tomcat to decide if ThreadLocal makes sense in a Virtual Thread world.
DateTimeFormatter solved it some versions ago, a singleton is the best
solution for such a thing.
DateTimeFormatter wasn't available until Java 1.8. In order to make
Tomcat's code maintainable across all supported versions, we have to go
back to Java 7, so we haven't moved to java.time. I suspect we will do
that relatively soon now that 8.5 is reaching EOL.
Anything else - when thread safety can't be reached - will end up in
"pools" - a queue being a cheap pool - which should be sized as the backing
forkjoin pool of virtual threads probably.
Yes, we use pools for other things where each thread doesn't really
needs its own copy. Sometimes performance is more important than memory
reduction. This is why I mention SimpleDateFormat: it's a very very
small class, and one-per-thread is not expensive at all, even if you
decide that 100M threads is a good idea.
(Honestly, we need to start using java.time. Now that Tomcat 8.5 no
longer needs to be supported, we should look at our options for moving-on.)
From my tests virtual threads do their job but they stay slower than a
proper reactive/async impl mainly due to the overhead they add
*everywhere*
compare to reactive programming plus this single thread pool issue which
adds a lot of contention when all the app/lot of tasks is/are done using
it
- vs bulkhead pattern for ex.
But if you come from a plain sync application it can be very interesting
if
it stays compatible and you don't need to add throttling to control the
memory - often in the old style you throttle with the number of threads,
now you need a semaphore or alike.
Will not be a free lunch ;).
Agreed, free lunches never exist.
But.
If an application exists that never bothered to convert to using
non-blocking I/O but uses many threads, it could be very easily changed
(slightly) to use Virtual Threads and get a performance boost out of the
deal. Or at least be able to schedule many more tasks without running
out of stack space.
The price you pay for that is CPU usage. And most of us have CPU to spare.
Hmm, I have more than doubt. I currently have a case for a customer, we
scaled to the desired perf level tuning threads to a high number -
everything is old style and synchronous - but if we move to VT it will just
blow up and explode in mem in ~5s of load - tested a close approach.
Hmm. What is using all that memory? Is it Java-heap, or is it native? Or
are they launching 100M threads and that's why memory is exhausted?
without a bounded thread pool, it's easy to just allow infinite work to
be scheduled. Infinite scheduled-work requires infinite memory.
So I suspect most of blocking apps also rely in their prod deployment on
the implicit thread throttling so VT can just make them blow up so "easy"
in terms of code, hard in terms of tuning since them the actual throttling
was not planned anywhere else in the code I think.
That would make sense.
-chris
On 3/20/24 07:22, Rainer Jung wrote:
I wanted to share an observation and I hope the things are correct how
I
am describing them. Maybe things have already improved and I am not
aware of it, hints welcome.
Part of JEP 425 (Project Loom, Java virtual threads) discusses how to
handle observability of virtual threads from inside the JVM and
tooling.
The final outcome is, that virtual threads are not included in the
typical JVM APIs which one can use to observe threads, like enumerating
them or accessing the stacks of individual threads. As a consequence,
also jstack and "kill -QUIT" do not show virtual threads at all, not
even when they are attached to a native thread and executing code.
O_O
There is one single method to help with observability of virtual
threads
https://docs.oracle.com/en/java/javase/21/docs/api/jdk.management/com/sun/management/HotSpotDiagnosticMXBean.html#dumpThreads(java.lang.String,com.sun.management.HotSpotDiagnosticMXBean.ThreadDumpFormat)
which dumps a full thread dump to a file. Of course that is by no means
appropriate, if you want to do a fine grained observation. At least you
can choose to write a json structure instead of a hard to parse text
format, but that's it.
Boy, that's a real miss from the JVM team. I'm surprised that they have
overlooked this. I don't see a real reason that a Virtual Thread would
be treated any differently than a regular thread.
For instance I am often using a tool, that inspects our
RequestProcessors to find long running requests and then retrieves the
list of Java threads as ThreadInfo objects to find the one executing
the
request and finally retrieves the stack of that request from the JVM.
Such an approach is no longer possible. Almost all of JMX does not show
any info about virtual threads.
It also seems Tomcat no longer uses a pool when a connector is
configured to use virtual threads. So there are no metrics like
currentThreadCount or currentThreadsBusy.
When using virtual threads, the number of requests is the same as the
number of threads, since each request -> new Virtual Thread.... though I
think the request-count is only incremented when the request has
completed, so maybe there are threads running that can't be counted
(yet).
But I understand that you were hoping to get some information about
these threads and though maybe Tomcat's metrics could help. I guess not
really.
But thread-busy count is the same as in-flight-request count. And
current thread count is the same as in-flight-request count as well.
I guess this also limits the use of some manager features and probably
also the StuckThreadsDetectionValve when combined with virtual threads.
Most likely. I'm fairly sure that uses the "usual Thread-walker things"
which you are reporting do not work any more with VTs.
What JVM are you using for your investigations?
Of course Tomcat has solved most of the problems, that virtual threads
want to solve, by doing the hard work of using the NIO and NIO2 APIs.
So
virtual threads are probably not that important for us. But we should
be
aware of the observability deficiencies whenever we would discuss
whether we could switch from NIO or NIO2 to virtual threads to simplify
connector code in some distant future.
+1
I've been enthusiastically talking with markt about dropping all that
nasty NIO stuff and "going back to BIO with virtual threads" which, at
this point, is mostly a threat and not a promise. But if VTs really
deliver everything they are claiming to deliver, then it may be possible
to go back to BIO as long as servlet-async is retired. And I'm not
holding my breath on that one.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
