Re: system profilers and incomplete stacks

serguei.spit...@oracle.com Tue, 17 Jun 2014 00:12:32 -0700

Brendan,

Thank you for the details, especially, about the perf_events!
I was not aware about the issue on Linux.


I agree, the Solaris jstack issue was not that bad back in 2005.
It needs to be fixed cooperatively with the OS.
We have an idea how to fix it, this work is at the prototyping stage now.

Thanks,
Serguei

On 6/16/14 11:52 PM, Brendan Gregg wrote:

G'Day Serguei,

On Mon, Jun 16, 2014 at 10:45 PM, serguei.spit...@oracle.com<mailto:serguei.spit...@oracle.com> <serguei.spit...@oracle.com<mailto:serguei.spit...@oracle.com>> wrote:


    Hi Brendan,

    We are aware of these issues and work with the Solaris team to fix
    them in JDK 9.
    One is the frame pointer is used by the server compiler as a
    general purpose register on intel.
    Another is about the virtual (or inlined) frames.

    There are a couple of related bugs:
    https://bugs.openjdk.java.net/browse/JDK-6617153
    https://bugs.openjdk.java.net/browse/JDK-6276264

    There can be more issues filed on this.


Ah, thanks, it's JDK-6276264.

As Tom Rodriguez said at the time (2005): "The server VM uses theframe pointer as an allocatable register and there's no way to turnthat off." I was really hoping there was a way to turn that off, like-fno-omit-frame-pointer.

This also means DTrace jstack() has never worked fully. For theapplications I tried it on, 50% of stacks were incomplete. Perhaps itwasn't that bad in 2005. I've been getting more mileage today fromJava profilers.


    Please, note, that the jstack action is not implemented on Linux yet.

Linux doesn't have DTrace jstack(), no, but its perf_events does hassupport for loading an auxiliary file of symbols, which can createdvia a Java agent for that purpose (eg,https://github.com/jrudolph/perf-map-agent). But that hasn't beenworking fully for the same reason - incomplete stacks.


Brendan


    Thanks,
    Serguei



    On 6/16/14 5:14 PM, Brendan Gregg wrote:

    Thanks but no, I'm aware of that bug and workarounds (I'm using
    the LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so
    workaround, which isn't mentioned in the bug comments, but
    probably should be). That bug is about missing symbols, but the
    stacks shown in that bug still go all the way to thread_start. My
    stacks often don't.

    For simple programs, the stacks are complete. But something
    complex (eg, vert.x with event loops), and the stacks are often
    incomplete, one frame only. Very much like what I see with
    -fomit-frame-pointer, although this is hotspot, not gcc. Such
    incomplete stacks are seen using either DTrace or perf_events.

    It was suggested to me to email the hotspot developers, because
    this may well be a hotspot optimization they are familiar with.
    It may also be something really obvious, like that the JVM breaks
    native stacks due to optimized frames / green threads / etc, and
    there is absolutely no way around it (no way to disable it). If
    that's true, it may also mean that the DTrace jstack() action has
    always had this issue. I'm still reading the source...

    Brendan



    On Mon, Jun 16, 2014 at 4:04 AM, Staffan Larsen
    <staffan.lar...@oracle.com <mailto:staffan.lar...@oracle.com>> wrote:

        I think this is the bug you are looking at:
        https://bugs.openjdk.java.net/browse/JDK-7187999, but I’ll
        defer to someone else to confirm.

        /Staffan


        On 16 jun 2014, at 12:47, Roland Westrelin
        <roland.westre...@oracle.com
        <mailto:roland.westre...@oracle.com>> wrote:

        Forwarding to serviceability alias where this question
        belongs I think.

        Begin forwarded message:

        *From: *Brendan Gregg <brendan.d.gr...@gmail.com
        <mailto:brendan.d.gr...@gmail.com>>
        *Subject: **system profilers and incomplete stacks*
        *Date: *June 12, 2014 at 7:15:54 PM GMT+2
        *To: *hotspot-compiler-...@openjdk.java.net
        <mailto:hotspot-compiler-...@openjdk.java.net>

        G'Day,

        Is there a way to run hotspot so that a system profiler
        (eg, DTrace, or Linux perf_events) can measure complete
        stacks? I often get incomplete, partial stacks, with one or
        a few frames only. I'm not worried about symbols right now,
        what I'd like is to walk stacks all the way down to thread
        start.

        I've been browsing the hotspot code, but haven't found out
        how yet. I suspect it's related to Java optimized frames,
        and has ditched the frame pointer. I was looking for an
        equivalent -fno-omit-frame-pointer option.

        Here's an example:

        # dtrace -n 'profile-99 /execname == "java"/ {
        @[jstack(100, 8000)] = count(); }'
        [...]
        org/mozilla/javascript/
        
ScriptableObject.createSlot(Ljava/lang/String;II)Lorg/mozilla/javascript/ScriptableObject$Slot;*
        0x884acce8200002da
        1

        sun/nio/ch/SocketChannelImpl.read(Ljava/nio/ByteBuffer;)I*
        0xffffffff20007f4b
        1

        
org/mozilla/javascript/ScriptRuntime.newObjectLiteral([Ljava/lang/Object;[Ljava/lang/Object;[ILorg/mozilla/javascript/Context;Lorg/mozilla/javascript/Scriptable;)Lorg/mozilla/javascript/Scriptable;*
        0xa20000041
        1
        [...]

        I see similar incomplete stacks with Linux perf_events.
        Oracle JDKs from 6 to 8, and OpenJDK.

        thanks,

        Brendan

--http://www.brendangregg.com <http://www.brendangregg.com/>

--http://www.brendangregg.com





--
http://www.brendangregg.com

Re: system profilers and incomplete stacks

Reply via email to