>From time to time a jboss process would end eating all, the available CPU and
>the load avg would skyrocket.
Once the operators restarted jboss, the system'd be normal again (sometimes for
weeks) until the next incident
Since we moved the app from a v440 running Solaris 10 8/07 to a t2000 running
Solaris 10 5/08 the problem started to happen more frequently (2-3 times a
week). The only other modification (besides hard and OS release) is we put the
app server inside a sparse-zone container with the FSS and a 95% limit on the
CPU usage.
Here's what uptime says:
Mon Nov 24 10:10:00 ARST 2008
10:10am up 10 day(s), 14:40, 6 users, load average: 325.52, 320.07, 318.72
(yes, load avg is almost 320, but the server was still usable)
a little output from mpstat shows this:
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 0 531 259 150 46 27 17 43 0 29 99 1 0 0
1 0 0 3 66 0 124 65 36 10 0 86 99 1 0 0
2 0 0 0 23 0 34 22 17 9 0 8 100 0 0 0
3 0 0 0 13 0 13 12 6 9 0 1 100 0 0 0
4 0 0 0 13 0 14 12 9 9 0 2 100 0 0 0
5 0 0 0 38 0 69 37 17 8 0 23 100 0 0 0
6 0 0 0 25 0 40 24 13 8 0 7 95 5 0 0
7 0 0 0 15 0 17 14 10 9 0 3 100 0 0 0
8 0 0 0 11 0 10 10 6 8 0 0 100 0 0 0
9 0 0 0 12 0 11 11 7 9 0 0 100 0 0 0
10 0 0 0 14 0 13 13 7 9 0 0 100 0 0 0
11 0 0 0 13 0 12 12 6 9 0 0 100 0 0 0
12 0 0 0 25 0 37 24 14 8 0 25 100 0 0 0
13 0 0 0 13 0 13 12 9 9 0 1 100 0 0 0
14 0 0 0 13 0 12 12 4 9 0 0 100 0 0 0
15 0 0 0 13 0 12 12 7 9 0 0 100 0 0 0
16 0 0 0 12 0 11 11 5 7 0 0 100 0 0 0
17 0 0 0 14 0 13 13 8 7 0 0 100 0 0 0
18 0 0 0 13 0 12 12 8 9 0 0 100 0 0 0
19 0 0 1 13 0 12 12 7 8 0 0 100 0 0 0
20 0 0 7 25 0 35 25 13 9 0 3 100 0 0 0
21 0 0 72 45 3 78 41 29 11 0 19 99 1 0 0
22 0 0 3 44 4 71 39 21 11 0 135 100 0 0 0
23 0 0 3 20 5 16 14 7 9 0 2 100 0 0 0
24 0 0 3 17 5 13 12 6 9 0 2 99 1 0 0
25 0 0 1 15 4 10 10 8 8 0 0 100 0 0 0
26 0 0 8 15 0 17 14 5 10 0 0 100 0 0 0
27 0 0 0 12 0 11 11 7 8 0 0 100 0 0 0
28 0 0 1 11 0 10 10 6 8 0 0 100 0 0 0
29 0 0 2 67 0 126 66 30 11 0 48 100 0 0 0
30 0 0 0 19 0 25 18 11 8 0 232 100 0 0 0
31 0 0 1 27 0 47 26 12 10 0 10 100 0 0 0
So, 100% cpu in usr mode looks a lot like some kind of infinite loop
I tried to dig it up using dtrace:
profile-1001us
/(execname=="java") && (pid==$1)/
{
@[jstack()]=count();
}
tick-20s
{
exit(O);
}
But I got a LOT of these arrors:
dtrace: error on enabled probe ID 1 (ID 52070: profile:::Profile-1001us):
invalid address (0x96a7e000) in action #2
and th stack traces are mostly the hex addresses without the names:
0xf886ca64
0xf884cec0
0xf8805d3c
0xf8805874
0xf8805c70
0xf8805c70
0xf8805d3c
0xf88380b8
0xf9509440
0xf88b7b44
0xf8805874
0xf8805874
0xf8805764
0xf8805874
0xf8805874
0xf8805d3c
0xf8805874
0xf8805874
0xf8805874
0xf8805874
0xf8805874
0xf8805874
0xf8805764
0xf8805764
0xf8805764
0xf886993c
0xf9999004
0xf884cec0
0xf8805874
0xf8805d3c
0xf884cc7c
0xf994090c
0xf8838df8
0xf8800218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`JVM_DoPrivileged+0x500
libjava.so`Java_java_security_AccessController_doPrivileged__Ljava_security_PrivilegedExceptionAction_2Ljava_security_AccessControlContext_2+0x14
0xf880e22c
0xf880e1d0
0xf89c6470
0xf9dfc018
0xf99ee350
0xf9d74fd8
0xf8838df8
0xf8800218
libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0x5a0
libjvm.so`JVM_DoPrivileged+0x500
libjava.so`Java_java_security_AccessController_doPrivileged__Ljava_security_PrivilegedExceptionAction_2+0x14
0xf9467cc8
0xf9badd44
9852
So my questions are:
any ideas on what to do next? I'm pretty sure it's a jboss/application problem,
but I need to get more data to show the jboss/devel people
what's causing those dtrace errors?
thanks in advance
_______________________________________________
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
