G'Day Serguei,
On Mon, Jun 16, 2014 at 10:45 PM, serguei.spit...@oracle.com
<serguei.spit...@oracle.com> wrote:
Hi Brendan,
We are aware of these issues and work with the Solaris team to fix them in
JDK 9.
One is the frame pointer is used by the server compiler as a general
purpose register on intel.
Another is about the virtual (or inlined) frames.
There are a couple of related bugs:
https://bugs.openjdk.java.net/browse/JDK-6617153
https://bugs.openjdk.java.net/browse/JDK-6276264
There can be more issues filed on this.
Ah, thanks, it's JDK-6276264.
As Tom Rodriguez said at the time (2005): "The server VM uses the frame
pointer as an allocatable register and there's no way to turn that off." I
was really hoping there was a way to turn that off, like
-fno-omit-frame-pointer.
This also means DTrace jstack() has never worked fully. For the applications
I tried it on, 50% of stacks were incomplete. Perhaps it wasn't that bad in
2005. I've been getting more mileage today from Java profilers.
Please, note, that the jstack action is not implemented on Linux yet.
Linux doesn't have DTrace jstack(), no, but its perf_events does has support
for loading an auxiliary file of symbols, which can created via a Java agent
for that purpose (eg, https://github.com/jrudolph/perf-map-agent). But that
hasn't been working fully for the same reason - incomplete stacks.
Brendan
Thanks,
Serguei
On 6/16/14 5:14 PM, Brendan Gregg wrote:
Thanks but no, I'm aware of that bug and workarounds (I'm using the
LD_AUDIT_64=/usr/lib/dtrace/64/libdtrace_forceload.so workaround, which
isn't mentioned in the bug comments, but probably should be). That bug is
about missing symbols, but the stacks shown in that bug still go all the way
to thread_start. My stacks often don't.
For simple programs, the stacks are complete. But something complex (eg,
vert.x with event loops), and the stacks are often incomplete, one frame
only. Very much like what I see with -fomit-frame-pointer, although this is
hotspot, not gcc. Such incomplete stacks are seen using either DTrace or
perf_events.
It was suggested to me to email the hotspot developers, because this may
well be a hotspot optimization they are familiar with. It may also be
something really obvious, like that the JVM breaks native stacks due to
optimized frames / green threads / etc, and there is absolutely no way
around it (no way to disable it). If that's true, it may also mean that the
DTrace jstack() action has always had this issue. I'm still reading the
source...
Brendan
On Mon, Jun 16, 2014 at 4:04 AM, Staffan Larsen
<staffan.lar...@oracle.com> wrote:
I think this is the bug you are looking at:
https://bugs.openjdk.java.net/browse/JDK-7187999, but I’ll defer to someone
else to confirm.
/Staffan
On 16 jun 2014, at 12:47, Roland Westrelin <roland.westre...@oracle.com>
wrote:
Forwarding to serviceability alias where this question belongs I think.
Begin forwarded message:
From: Brendan Gregg <brendan.d.gr...@gmail.com>
Subject: system profilers and incomplete stacks
Date: June 12, 2014 at 7:15:54 PM GMT+2
To: hotspot-compiler-...@openjdk.java.net
G'Day,
Is there a way to run hotspot so that a system profiler (eg, DTrace, or
Linux perf_events) can measure complete stacks? I often get incomplete,
partial stacks, with one or a few frames only. I'm not worried about symbols
right now, what I'd like is to walk stacks all the way down to thread start.
I've been browsing the hotspot code, but haven't found out how yet. I
suspect it's related to Java optimized frames, and has ditched the frame
pointer. I was looking for an equivalent -fno-omit-frame-pointer option.
Here's an example:
# dtrace -n 'profile-99 /execname == "java"/ { @[jstack(100, 8000)] =
count(); }'
[...]
org/mozilla/javascript/
ScriptableObject.createSlot(Ljava/lang/String;II)Lorg/mozilla/javascript/ScriptableObject$Slot;*
0x884acce8200002da
1
sun/nio/ch/SocketChannelImpl.read(Ljava/nio/ByteBuffer;)I*
0xffffffff20007f4b
1
org/mozilla/javascript/ScriptRuntime.newObjectLiteral([Ljava/lang/Object;[Ljava/lang/Object;[ILorg/mozilla/javascript/Context;Lorg/mozilla/javascript/Scriptable;)Lorg/mozilla/javascript/Scriptable;*
0xa20000041
1
[...]
I see similar incomplete stacks with Linux perf_events. Oracle JDKs from
6 to 8, and OpenJDK.
thanks,
Brendan
--
http://www.brendangregg.com
--
http://www.brendangregg.com
--
http://www.brendangregg.com
