Hi Brendan, I'm not quite seeing the complete context here so a few queries:
1. Which platform are you running Dtrace on? (hotspot has to provide helper d-scripts to allow Dtrace to walk Java stacks)
2. What are perf-events? How do they try to walk stacks? 3. Does "jstack -m" show full stacks for the apps where you see problem? Thanks, David On 17/06/2014 10:14 AM, Brendan Gregg wrote:
Thanks but no, I'm aware of that bug and workarounds (I'm using the LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so workaround, which isn't mentioned in the bug comments, but probably should be). That bug is about missing symbols, but the stacks shown in that bug still go all the way to thread_start. My stacks often don't. For simple programs, the stacks are complete. But something complex (eg, vert.x with event loops), and the stacks are often incomplete, one frame only. Very much like what I see with -fomit-frame-pointer, although this is hotspot, not gcc. Such incomplete stacks are seen using either DTrace or perf_events. It was suggested to me to email the hotspot developers, because this may well be a hotspot optimization they are familiar with. It may also be something really obvious, like that the JVM breaks native stacks due to optimized frames / green threads / etc, and there is absolutely no way around it (no way to disable it). If that's true, it may also mean that the DTrace jstack() action has always had this issue. I'm still reading the source... Brendan On Mon, Jun 16, 2014 at 4:04 AM, Staffan Larsen <staffan.lar...@oracle.com <mailto:staffan.lar...@oracle.com>> wrote: I think this is the bug you are looking at: https://bugs.openjdk.java.net/browse/JDK-7187999, but I’ll defer to someone else to confirm. /Staffan On 16 jun 2014, at 12:47, Roland Westrelin <roland.westre...@oracle.com <mailto:roland.westre...@oracle.com>> wrote:Forwarding to serviceability alias where this question belongs I think. Begin forwarded message:*From: *Brendan Gregg <brendan.d.gr...@gmail.com <mailto:brendan.d.gr...@gmail.com>> *Subject: **system profilers and incomplete stacks* *Date: *June 12, 2014 at 7:15:54 PM GMT+2 *To: *hotspot-compiler-...@openjdk.java.net <mailto:hotspot-compiler-...@openjdk.java.net> G'Day, Is there a way to run hotspot so that a system profiler (eg, DTrace, or Linux perf_events) can measure complete stacks? I often get incomplete, partial stacks, with one or a few frames only. I'm not worried about symbols right now, what I'd like is to walk stacks all the way down to thread start. I've been browsing the hotspot code, but haven't found out how yet. I suspect it's related to Java optimized frames, and has ditched the frame pointer. I was looking for an equivalent -fno-omit-frame-pointer option. Here's an example: # dtrace -n 'profile-99 /execname == "java"/ { @[jstack(100, 8000)] = count(); }' [...] org/mozilla/javascript/ ScriptableObject.createSlot(Ljava/lang/String;II)Lorg/mozilla/javascript/ScriptableObject$Slot;* 0x884acce8200002da 1 sun/nio/ch/SocketChannelImpl.read(Ljava/nio/ByteBuffer;)I* 0xffffffff20007f4b 1 org/mozilla/javascript/ScriptRuntime.newObjectLiteral([Ljava/lang/Object;[Ljava/lang/Object;[ILorg/mozilla/javascript/Context;Lorg/mozilla/javascript/Scriptable;)Lorg/mozilla/javascript/Scriptable;* 0xa20000041 1 [...] I see similar incomplete stacks with Linux perf_events. Oracle JDKs from 6 to 8, and OpenJDK. thanks, Brendan -- http://www.brendangregg.com <http://www.brendangregg.com/>-- http://www.brendangregg.com
