On 22 okt 2014, at 02:10, Brendan Gregg <brendan.d.gr...@gmail.com> wrote:
> G'Day, > > I checked the JDK 9 early access releases, but didn't see anything for > JDK-6276264. That bug has been closed as a duplicate of https://bugs.openjdk.java.net/browse/JDK-6617153, which is still open. > I've also since learned that Twitter has an OpenJDK fork > with frame pointers disabled, for the same purpose: stack profiling > (using Linux perf_events). Might this be worked on for JDK 9? I would welcome that change - perhaps Twitter can contribute it? Thanks, /Staffan > I can > help test. thanks, > > Brendan > > On Mon, Jun 16, 2014 at 11:52 PM, Brendan Gregg > <brendan.d.gr...@gmail.com> wrote: >> G'Day Serguei, >> >> On Mon, Jun 16, 2014 at 10:45 PM, serguei.spit...@oracle.com >> <serguei.spit...@oracle.com> wrote: >>> >>> Hi Brendan, >>> >>> We are aware of these issues and work with the Solaris team to fix them in >>> JDK 9. >>> One is the frame pointer is used by the server compiler as a general >>> purpose register on intel. >>> Another is about the virtual (or inlined) frames. >>> >>> There are a couple of related bugs: >>> https://bugs.openjdk.java.net/browse/JDK-6617153 >>> https://bugs.openjdk.java.net/browse/JDK-6276264 >>> >>> There can be more issues filed on this. >> >> >> Ah, thanks, it's JDK-6276264. >> >> As Tom Rodriguez said at the time (2005): "The server VM uses the frame >> pointer as an allocatable register and there's no way to turn that off." I >> was really hoping there was a way to turn that off, like >> -fno-omit-frame-pointer. >> >> This also means DTrace jstack() has never worked fully. For the applications >> I tried it on, 50% of stacks were incomplete. Perhaps it wasn't that bad in >> 2005. I've been getting more mileage today from Java profilers. >> >>> Please, note, that the jstack action is not implemented on Linux yet. >> >> >> Linux doesn't have DTrace jstack(), no, but its perf_events does has support >> for loading an auxiliary file of symbols, which can created via a Java agent >> for that purpose (eg, https://github.com/jrudolph/perf-map-agent). But that >> hasn't been working fully for the same reason - incomplete stacks. >> >> Brendan >> >>> >>> Thanks, >>> Serguei >>> >>> >>> >>> On 6/16/14 5:14 PM, Brendan Gregg wrote: >>> >>> Thanks but no, I'm aware of that bug and workarounds (I'm using the >>> LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so workaround, which >>> isn't mentioned in the bug comments, but probably should be). That bug is >>> about missing symbols, but the stacks shown in that bug still go all the way >>> to thread_start. My stacks often don't. >>> >>> For simple programs, the stacks are complete. But something complex (eg, >>> vert.x with event loops), and the stacks are often incomplete, one frame >>> only. Very much like what I see with -fomit-frame-pointer, although this is >>> hotspot, not gcc. Such incomplete stacks are seen using either DTrace or >>> perf_events. >>> >>> It was suggested to me to email the hotspot developers, because this may >>> well be a hotspot optimization they are familiar with. It may also be >>> something really obvious, like that the JVM breaks native stacks due to >>> optimized frames / green threads / etc, and there is absolutely no way >>> around it (no way to disable it). If that's true, it may also mean that the >>> DTrace jstack() action has always had this issue. I'm still reading the >>> source... >>> >>> Brendan >>> >>> >>> >>> On Mon, Jun 16, 2014 at 4:04 AM, Staffan Larsen >>> <staffan.lar...@oracle.com> wrote: >>>> >>>> I think this is the bug you are looking at: >>>> https://bugs.openjdk.java.net/browse/JDK-7187999, but I’ll defer to someone >>>> else to confirm. >>>> >>>> /Staffan >>>> >>>> >>>> On 16 jun 2014, at 12:47, Roland Westrelin <roland.westre...@oracle.com> >>>> wrote: >>>> >>>> Forwarding to serviceability alias where this question belongs I think. >>>> >>>> Begin forwarded message: >>>> >>>> From: Brendan Gregg <brendan.d.gr...@gmail.com> >>>> Subject: system profilers and incomplete stacks >>>> Date: June 12, 2014 at 7:15:54 PM GMT+2 >>>> To: hotspot-compiler-...@openjdk.java.net >>>> >>>> G'Day, >>>> >>>> Is there a way to run hotspot so that a system profiler (eg, DTrace, or >>>> Linux perf_events) can measure complete stacks? I often get incomplete, >>>> partial stacks, with one or a few frames only. I'm not worried about >>>> symbols >>>> right now, what I'd like is to walk stacks all the way down to thread >>>> start. >>>> >>>> I've been browsing the hotspot code, but haven't found out how yet. I >>>> suspect it's related to Java optimized frames, and has ditched the frame >>>> pointer. I was looking for an equivalent -fno-omit-frame-pointer option. >>>> >>>> Here's an example: >>>> >>>> # dtrace -n 'profile-99 /execname == "java"/ { @[jstack(100, 8000)] = >>>> count(); }' >>>> [...] >>>> org/mozilla/javascript/ >>>> >>>> ScriptableObject.createSlot(Ljava/lang/String;II)Lorg/mozilla/javascript/ScriptableObject$Slot;* >>>> 0x884acce8200002da >>>> 1 >>>> >>>> sun/nio/ch/SocketChannelImpl.read(Ljava/nio/ByteBuffer;)I* >>>> 0xffffffff20007f4b >>>> 1 >>>> >>>> >>>> org/mozilla/javascript/ScriptRuntime.newObjectLiteral([Ljava/lang/Object;[Ljava/lang/Object;[ILorg/mozilla/javascript/Context;Lorg/mozilla/javascript/Scriptable;)Lorg/mozilla/javascript/Scriptable;* >>>> 0xa20000041 >>>> 1 >>>> [...] >>>> >>>> I see similar incomplete stacks with Linux perf_events. Oracle JDKs from >>>> 6 to 8, and OpenJDK. >>>> >>>> thanks, >>>> >>>> Brendan >>>> -- >>>> http://www.brendangregg.com >>>> >>>> >>>> >>> >>> >>> >>> -- >>> http://www.brendangregg.com >>> >>> >> >> >> >> -- >> http://www.brendangregg.com
