Hi Xin, Thank you for your explanations. I would like to reply on the RequestStackTrace parts:
First of all, notice that the currently proposed change is in JEP draft: Native Profiler Hook for Unbiased Stack Traces (Experimental) (https://openjdk.org/jeps/8380294), and its implementation is tracked in https://bugs.openjdk.org/browse/JDK-8383923. That is a much-reworked approach from the one that you mentioned. That is basically a JVMTI extension that asks JFR to report a stack-trace in its recording. There is also an extension to this in JEP Draft: Unbiased Stack-Walk JVMTI extension API (https://openjdk.org/jeps/8381322), which is a JVMTI extension that does not depend on JFR at all, and instead reports the stack-trace as JVMTI callbacks. Regarding your 'missing features' and questions: - Supporting _thread_in_vm should be possible, I will attempt that in my PR 31059 and will let you know. Ignoring threads in-VM may just be there for historical reasons. - Native frames support is currently being worked on separately by Andrei Pangin (https://bugs.openjdk.org/browse/JDK-8373791) and will then also work with the JVMTI extensions. - We will *not* make it compatible with ASGCT, at least not directly. ASGCT is inherently not made for the sort of asynchronousness that we need. What would perhaps work for you is the JVMTI-only proposal: https://openjdk.org/jeps/8381322, that would allow you to use JVMTI callback to build your own ASGCT_CallTrace structure if you really need that. - Extending the stack-walker to handle anything Kotlin-specific is most likely not going to happen. I am not familiar with how Kotlin works, but again, perhaps the JVMTI callbacks approach would work for you? Best regards, Roman On Wednesday, October 14th, 2026 at 01:03, Liu Xin <[email protected]> wrote: > Hi serviceability-dev and hotspot-jfr-dev, > > I am looking into CPU profiling support for Virtual Threads and Kotlin > coroutines. Below are our conclusions regarding heap access, our prototyping > experience with RequestStackTrace, and the path forward in GoogleJDK. I > would like to learn opinions and options from the OpenJDK community. > > 1. Background & Technical Motivation > ----------------------------------- > Profiling asynchronous and concurrent execution models—specifically Kotlin > coroutines and Virtual Threads—presents a challenge for ASGCT: > > - Kotlin Coroutines: Implemented via Continuation-Passing Style (CPS). > Suspended coroutines save their logical callers as a linked list of > Continuation objects (not the same as Virtual Thread's Continuation) > directly > on the Java heap. > - Virtual Threads: Unmounted frames and continuation states reside in > heap-allocated StackChunk objects. > > > 2. The Verdict: Unviability of Accessing Heap Objects from ASGCT > ---------------------------------------------------------------- > We explored whether the established profiling model (walking frames inside a > SIGPROF signal handler via an extended ASGCT) could be adapted to traverse > heap-allocated continuation objects. > > Our conclusion is that there is fundamentally no safe way to dereference or > traverse Java heap objects from ASGCT: > > - Concurrent Relocation & Load Barriers (ZGC & Shenandoah): > Modern collectors relocate heap objects concurrently while mutator threads > execute. Accessing heap pointers from ASGCT bypasses load barriers or > invokes > barriers in prohibited contexts (e.g., during active relocation or page > allocation), leading to immediate VM crashes or memory corruption. > > - STW Evacuation Races on Java Threads in Native (_thread_in_native): > When a JavaThread executes native code via JNI/FFM or system calls, its > state > is _thread_in_native and it is considered safepoint-safe; STW collectors > (such as G1) do not wait for it before starting evacuation pauses. If > SIGPROF > interrupts a JavaThread while in native, GC worker threads may be > concurrently > evacuating objects on the Java heap. Dereferencing heap pointers from ASGCT > races directly against object copying and mark word updates. > > - Forwarding Pointers & Compact Object Headers: > During evacuation, GC installs forwarding pointers into the object's > markWord. > With Compact Object Headers (-XX:+UseCompactObjectHeaders), decoding > oop->klass() from a forwarding pointer results in an invalid pointer > dereference and a fatal SIGSEGV. > > - Lack of JIT OopMaps at Arbitrary Signal PCs (Cannot Inspect Locals or > Arguments): > JIT compilers (C1/C2) only generate OopMap records at designated safepoint > PCs (e.g. calls, backedges, safepoint polls). Because SIGPROF interrupts > execution at an arbitrary instruction PC, HotSpot has no OopMap to decode > which CPU registers or stack slots hold live object references (oops) versus > primitive values. Consequently, a stack walker in ASGCT cannot inspect > method arguments or local variables (such as the 1st argument Continuation > passed to BaseContinuationImpl.resumeWith) to locate the active coroutine > root on the stack without intrusive mechanisms (like compiler intrinsics > caching references on JavaThread). > > Based on my estimation, patching G1 GC to be ASGCT-aware would be a > substantial > and invasive change and would still be forced to drop samples whenever race > conditions occur. Therefore, we decided against extending ASGCT. > > > 3. A New Path — RequestStackTrace > --------------------------------- > > Summary of JVMTI RequestStackTrace > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Roman Kennke's PR (#29067: "8373578: JVMTI replacement for AsyncGetCallTrace", > https://github.com/openjdk/jdk/pull/29067) refactors HotSpot's JFR CPU-time > thread sampler infrastructure and exposes JFR cooperative sampling (JEP-518) > as a JVMTI extension function: RequestStackTrace (or RequestJFRStackTrace). > The new API is async-signal-safe: instead of walking stacks synchronously in > an > arbitrary signal context, it enqueues a request into a thread-local queue to > be > processed cooperatively when the target thread hits a safepoint poll in a > valid > VM state. > > Prototype & Experience > ~~~~~~~~~~~~~~~~~~~~~~ > We have experimented with RequestStackTrace in Google's JDK. We collected > empirical profiling data across microbenchmarks and multi-threaded DaCapo > suites, and the results show that it is very promising to replace ASGCT: > > - Safe for G1 GC and Concurrent GCs (ZGC). > - Produces clean, well-formed Java traces without the broken or unrooted > stacks frequently observed under ASGCT. > - Across DaCapo benchmarks, the number of samples collected by > RequestStackTrace > is comparable to ASGCT (~90% of ASGCT samples), and the remaining gaps are > known and addressable. > > Missing Features to Address > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > However, our evaluation highlighted a few missing features needed for > production parity and for our goals with Kotlin: > > - Support _thread_in_vm in StackWalker::request_stack_trace: > StackWalker currently rejects requests when jt->thread_state() == > _thread_in_vm (check_state() returns false), silently dropping requests. In > real-world workloads, mutator threads spend significant CPU time in > _thread_in_vm (e.g., JNI entries, runtime stubs, object allocation slow > paths, monitor contention, and class verification), leading to 30%–50%+ > sample loss in some workloads. We need a way to safely enqueue biased > requests > up to the Last Java Frame (LJF) and drain them when transitioning back to > Java > or native. > > - Support Mixed Stacktrace (Native Frames): > Currently, profilers built on ASGCT plus native unwinding (e.g., > async-profiler) can produce mixed stacktraces today. In RequestStackTrace, > we need support for mixed traces across two main scenarios: > * Native frames before/below the Last Java Frame (LJF): Native code calling > into Java via JNI or FFM upcalls. These native caller frames remain on the > physical execution stack when StackWalker runs, but are currently skipped > because StackWalker only walks Java frames. > * Native frames called by the Last Java Frame (LJF): Java code calling out > to > native C/C++ via JNI or FFM downcalls (e.g., burning CPU in native > functions). Because cooperative sampling defers stack walking to > safepoints > and unwinds starting from the LJF, native frames above the LJF need to be > retained. > > - Be ASGCT-compatible: > The stack walker produces ASGCT_CallTrace rather than a JFR event. > > - Extend the StackWalker to Generate Logical Callers for Kotlin Coroutines: > While Project Loom continuations (StackChunk) are handled at the VM level, > Kotlin coroutines maintain their logical caller hierarchy on the heap. We > prototyped an extension in the cooperative stack walker to traverse the > Continuation chain and interleave logical coroutine caller frames into the > stack trace. > > Questions & Discussion Points for OpenJDK > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > - Status of PR #29067 & Plans for a Public Stacktrace API: > Has OpenJDK reached a consensus on providing a new public stacktrace > interface > (such as JVMTI's RequestStackTrace)? Given Roman's shift to PR #31059 > (https://github.com/openjdk/jdk/pull/31059), is JDK-8380294 intended to > succeed RequestStackTrace? The updated PR offers less flexibility. Because > ASGCT_CallTrace's struct and ABI are widely integrated across our > infrastructure, GoogleJDK plans to adhere to it; if JFR events are produced > instead, we would need to transform them. > > - Community Attitude toward Features: > What is OpenJDK's / Serviceability's attitude toward those features? In > particular, is the JFR CPU profiler team interested in feature 1 > (_thread_in_vm) and feature 2 (mixed stack traces)? If the JFR CPU profiler > would take features 1 and 2 down the road, it would be well worth > refactoring > and sharing the underlying data structures and utility classes between the > JFR > profiler and the JVMTI extension. If not, I think it’s easier to manage our > patches if RequestStackTrace just doesn’t depend on JFR CPU profiler. > > Thanks, > --lx >
