G'Day, I checked the JDK 9 early access releases, but didn't see anything for JDK-6276264. I've also since learned that Twitter has an OpenJDK fork with frame pointers disabled, for the same purpose: stack profiling (using Linux perf_events). Might this be worked on for JDK 9? I can help test. thanks,
Brendan On Mon, Jun 16, 2014 at 11:52 PM, Brendan Gregg <brendan.d.gr...@gmail.com> wrote: > G'Day Serguei, > > On Mon, Jun 16, 2014 at 10:45 PM, serguei.spit...@oracle.com > <serguei.spit...@oracle.com> wrote: >> >> Hi Brendan, >> >> We are aware of these issues and work with the Solaris team to fix them in >> JDK 9. >> One is the frame pointer is used by the server compiler as a general >> purpose register on intel. >> Another is about the virtual (or inlined) frames. >> >> There are a couple of related bugs: >> https://bugs.openjdk.java.net/browse/JDK-6617153 >> https://bugs.openjdk.java.net/browse/JDK-6276264 >> >> There can be more issues filed on this. > > > Ah, thanks, it's JDK-6276264. > > As Tom Rodriguez said at the time (2005): "The server VM uses the frame > pointer as an allocatable register and there's no way to turn that off." I > was really hoping there was a way to turn that off, like > -fno-omit-frame-pointer. > > This also means DTrace jstack() has never worked fully. For the applications > I tried it on, 50% of stacks were incomplete. Perhaps it wasn't that bad in > 2005. I've been getting more mileage today from Java profilers. > >> Please, note, that the jstack action is not implemented on Linux yet. > > > Linux doesn't have DTrace jstack(), no, but its perf_events does has support > for loading an auxiliary file of symbols, which can created via a Java agent > for that purpose (eg, https://github.com/jrudolph/perf-map-agent). But that > hasn't been working fully for the same reason - incomplete stacks. > > Brendan > >> >> Thanks, >> Serguei >> >> >> >> On 6/16/14 5:14 PM, Brendan Gregg wrote: >> >> Thanks but no, I'm aware of that bug and workarounds (I'm using the >> LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so workaround, which >> isn't mentioned in the bug comments, but probably should be). That bug is >> about missing symbols, but the stacks shown in that bug still go all the way >> to thread_start. My stacks often don't. >> >> For simple programs, the stacks are complete. But something complex (eg, >> vert.x with event loops), and the stacks are often incomplete, one frame >> only. Very much like what I see with -fomit-frame-pointer, although this is >> hotspot, not gcc. Such incomplete stacks are seen using either DTrace or >> perf_events. >> >> It was suggested to me to email the hotspot developers, because this may >> well be a hotspot optimization they are familiar with. It may also be >> something really obvious, like that the JVM breaks native stacks due to >> optimized frames / green threads / etc, and there is absolutely no way >> around it (no way to disable it). If that's true, it may also mean that the >> DTrace jstack() action has always had this issue. I'm still reading the >> source... >> >> Brendan >> >> >> >> On Mon, Jun 16, 2014 at 4:04 AM, Staffan Larsen >> <staffan.lar...@oracle.com> wrote: >>> >>> I think this is the bug you are looking at: >>> https://bugs.openjdk.java.net/browse/JDK-7187999, but I’ll defer to someone >>> else to confirm. >>> >>> /Staffan >>> >>> >>> On 16 jun 2014, at 12:47, Roland Westrelin <roland.westre...@oracle.com> >>> wrote: >>> >>> Forwarding to serviceability alias where this question belongs I think. >>> >>> Begin forwarded message: >>> >>> From: Brendan Gregg <brendan.d.gr...@gmail.com> >>> Subject: system profilers and incomplete stacks >>> Date: June 12, 2014 at 7:15:54 PM GMT+2 >>> To: hotspot-compiler-...@openjdk.java.net >>> >>> G'Day, >>> >>> Is there a way to run hotspot so that a system profiler (eg, DTrace, or >>> Linux perf_events) can measure complete stacks? I often get incomplete, >>> partial stacks, with one or a few frames only. I'm not worried about symbols >>> right now, what I'd like is to walk stacks all the way down to thread start. >>> >>> I've been browsing the hotspot code, but haven't found out how yet. I >>> suspect it's related to Java optimized frames, and has ditched the frame >>> pointer. I was looking for an equivalent -fno-omit-frame-pointer option. >>> >>> Here's an example: >>> >>> # dtrace -n 'profile-99 /execname == "java"/ { @[jstack(100, 8000)] = >>> count(); }' >>> [...] >>> org/mozilla/javascript/ >>> >>> ScriptableObject.createSlot(Ljava/lang/String;II)Lorg/mozilla/javascript/ScriptableObject$Slot;* >>> 0x884acce8200002da >>> 1 >>> >>> sun/nio/ch/SocketChannelImpl.read(Ljava/nio/ByteBuffer;)I* >>> 0xffffffff20007f4b >>> 1 >>> >>> >>> org/mozilla/javascript/ScriptRuntime.newObjectLiteral([Ljava/lang/Object;[Ljava/lang/Object;[ILorg/mozilla/javascript/Context;Lorg/mozilla/javascript/Scriptable;)Lorg/mozilla/javascript/Scriptable;* >>> 0xa20000041 >>> 1 >>> [...] >>> >>> I see similar incomplete stacks with Linux perf_events. Oracle JDKs from >>> 6 to 8, and OpenJDK. >>> >>> thanks, >>> >>> Brendan >>> -- >>> http://www.brendangregg.com >>> >>> >>> >> >> >> >> -- >> http://www.brendangregg.com >> >> > > > > -- > http://www.brendangregg.com
