No, it turned out that the docs for Apache Archiva were incorrect and the 
WaitStrategy was effectively still SleepingWaitStrategy. Using the correct 
config to specify BlockingWaitStrategy solved the issue. (LOG4J2-571)

FYI, the wait strategy determines what the consumer thread does when the queue 
is empty & it's waiting for events. Some specialized apps want to avoid the 
latency of waking up a blocked thread, so there are a number of options with 
different trade-offs, with busy spin on one end of the spectrum and blocking on 
the other. For log4j I reduced the set of available options (no busy spin), and 
choose  SleepingWait as the default. This had the best performance in my 
testing. Until now I hadn't seen any excessive CPU usage. 

Sent from my iPhone

> On 2014/03/20, at 22:10, Matt Sicker <[email protected]> wrote:
> 
> Perhaps lmax disruptor doesn't work properly in the IBM JVM?
> 
>> On Tuesday, 18 March 2014, Chris Graham <[email protected]> wrote:
>> 
>> JStack is a Sun thing. This is the IBM JDK on AIX.
>> I've run the tprof command twice and verified it.
>> 
>> The full work though follows.
>> 
>> The output from topas (same as top, effectively) is:
>> 
>> Topas Monitor for host:    XXXXXXXXXXXXXXX      EVENTS/QUEUES    FILE/TTY
>> Wed Mar 19 14:49:55 2014   Interval:  2         Cswitch    3581
>> Readch      686
>>                                                Syscall    2763  Writech
>> 1378
>> CPU  User%  Kern%  Wait%  Idle%  Physc   Entc   Reads         7
>> Rawin         0
>> ALL   48.8    1.2    0.0   50.1   1.03   51.7   Writes        5
>> Ttyout      643
>>                                                Forks         0
>> Igets         0
>> Network  KBPS   I-Pack  O-Pack   KB-In  KB-Out  Execs         0
>> Namei        81
>> Total     2.8     10.0     7.5     1.1     1.7  Runqueue    1.0
>> Dirblk        0
>>                                                Waitqueue   0.0
>> Disk    Busy%     KBPS     TPS KB-Read KB-Writ                   MEMORY
>> Total     0.0      0.0     0.0     0.0     0.0  PAGING           Real,MB
>> 20480
>>                                                Faults        3  % Comp
>> 44
>> FileSystem        KBPS     TPS KB-Read KB-Writ  Steals        0  % Noncomp
>> 54
>> Total              0.6     1.5    0.6    0.0    PgspIn        0  % Client
>> 54
>>                                                PgspOut       0
>> Name            PID  CPU%  PgSp Owner           PageIn        0  PAGING
>> SPACE
>> java        9437312  48.6 739.1 wasadmin        PageOut       0  Size,MB
>> 15552
>> 
>> Process with a PID of 9437312 is the WebSphere instance that runs Archiva,
>> the problem in question.
>> 
>> We then use a native AIX tool, tprof to examine that process, and see what
>> it's doing:
>> 
>> tprof -j -P 9437312 -skex sleep 60
>> 
>> This generates sleep.prof, and the relevant section is:
>> 
>> Configuration information
>> =========================
>> System: AIX 6.1 Node: au02qap207teax2 Machine: 00C43D204C00
>> Tprof command was:
>>    tprof -j -P 9437312 -skex sleep 60
>> Trace command was:
>>    /usr/bin/trace -ad -M -L 2438933299 -T 500000 -j
>> 00A,001,002,003,38F,005,006,134,210,139,5A2,5A5,465,234,5D8, -o -
>> Total Samples = 24008
>> Traced Time = 60.02s (out of a total execution time of 60.02s)
>> 
>> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
>> 
>> Process                          Freq  Total Kernel   User Shared  Other
>> Java
>> =======                          ====  ===== ======   ==== ======  =====
>> ====
>> wait                                4  64.29  64.29   0.00   0.00   0.00
>> 0.00
>> java                              182  34.66   1.97   0.00  32.65   0.05
>> 0.00
>> /usr/sbin/syncd                     2   0.17   0.17   0.00   0.00   0.00
>> 0.00
>> sshd:                               1   0.14   0.09   0.05   0.01   0.00
>> 0.00
>> /usr/bin/ps                        13   0.12   0.10   0.00   0.03   0.00
>> 0.00
>> /usr/bin/sh                        27   0.12   0.11   0.00   0.00   0.00
>> 0.00
>> httpd                              12   0.09   0.05   0.01   0.03   0.00
>> 0.00
>> /usr/bin/scp                        1   0.09   0.07   0.00   0.01   0.00
>> 0.00
>> /opt/freeware/bin/readlink          6   0.04   0.04   0.00   0.00   0.00
>> 0.00
>> /usr/bin/tprof                      1   0.03   0.00   0.01   0.02   0.00
>> 0.00
>> kulagent                            7   0.03   0.02   0.00   0.00   0.00
>> 0.00
>> swapper                             1   0.02   0.02   0.00   0.00   0.00
>> 0.00
>> /usr/bin/grep                       5   0.02   0.02   0.00   0.00   0.00
>> 0.00
>> /usr/sbin/getty                     1   0.02   0.02   0.00   0.00   0.00
>> 0.00
>> kuxagent                            3   0.02   0.01   0.00   0.00   0.00
>> 0.00
>> lrud                                1   0.02   0.02   0.00   0.00   0.00
>> 0.00
>> srmAIX                              3   0.01   0.01   0.00   0.00   0.00
>> 0.00
>> /opt/tivoli/cit/bin/wscanuse        1   0.01   0.01   0.00   0.00   0.00
>> 0.00
>> psmd                                1   0.01   0.01   0.00   0.00   0.00
>> 0.00
>> xmgc                                1   0.01   0.01   0.00   0.00   0.00
>> 0.00
>> nfs_stat                            2   0.01   0.01   0.00   0.00   0.00
>> 0.00
>> /usr/bin/topasrec                   2   0.01   0.01   0.00   0.00   0.00
>> 0.00
>> /usr/bin/xmtopas                    1   0.01   0.00   0.00   0.00   0.00
>> 0.00
>> kcawd                               1   0.00   0.00   0.00   0.00   0.00
>> 0.00
>> /usr/sbin/snmpd                     1   0.00   0.00   0.00   0.00   0.00
>> 0.00
>> /usr/sbin/snmpmibd                  1   0.00   0.00   0.00   0.00   0.00
>> 0.00
>> /usr/bin/date                       1   0.00   0.00   0.00   0.00   0.00
>> 0.00
>> /usr/bin/entstat                    1   0.00   0.00   0.00   0.00   0.00
>> 0.00
>> gil                                 1   0.00   0.00   0.00   0.00   0.00
>> 0.00
>> /usr/bin/vmstat                     1   0.00   0.00   0.00   0.00   0.00
>> 0.00
>> /usr/bin/ln                         1   0.00   0.00   0.00   0.00   0.00
>> 0.00
>> stat_daemon                         1   0.00   0.00   0.00   0.00   0.00
>> 0.00
>> j2pg                                1   0.00   0.00   0.00   0.00   0.00
>> 0.00
>> /usr/sbin/netstat                   1   0.00   0.00   0.00   0.00   0.00
>> 0.00
>> =======                          ====  ===== ======   ==== ======  =====
>> ====
>> Total                             289 100.00  67.11   0.07  32.77   0.05
>> 0.00
>> 
>> Process               PID      TID  Total Kernel   User Shared  Other
>> Java
>> =======               ===      ===  ===== ======   ==== ======  =====
>> ====
>> java              9437312 46072057  23.76   0.05   0.00  23.70   0.00
>> 0.00
>> 
>> Which gives me the needed TID, 46072057 = 0x2BF00F9.
>> 
>> So to trigger a non-fatal heap dump:
>> 
>> kill -3 9437312
>> 
>> and then we look into the created javacore.20140318.173250.9437312.0002.txt
>> file for references to 0x2BF00F9:
>> 
>> 3XMTHREADINFO      "AsyncLoggerConfig-1" J9VMThread:0x0000000031D14600,
>> j9thread_t:0x00000100137D8BD0, java/lang/Thread:0x000000004301C508,
>> state:CW, prio=5
>> 3XMJAVALTHREAD            (java/lang/Thread getId:0x6A, isDaemon:true)
>> 3XMTHREADINFO1            (native thread ID:0x2BF00F9, native priority:0x5,
>> native policy:UNKNOWN)
>> 3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
>> 3XMTHREADINFO3           Java callstack:
>> 4XESTACKTRACE                at sun/misc/Unsafe.park(Native Method)
>> 4XESTACKTRACE                at
>> java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:332)
>> 4XESTACKTRACE                at
>> 
>> com/lmax/disruptor/SleepingWaitStrategy.applyWaitMethod(SleepingWaitStrategy.java:66)
>> 4XESTACKTRACE                at
>> 
>> com/lmax/disruptor/SleepingWaitStrategy.waitFor(SleepingWaitStrategy.java:39)
>> 4XESTACKTRACE                at
>> 
>> com/lmax/disruptor/ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55)
>> 4XESTACKTRACE                at
>> com/lmax/disruptor/BatchEventProcessor.run(BatchEventProcessor.java:115)
>> 4XESTACKTRACE                at
>> 
>> java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>> 4XESTACKTRACE                at
>> 
>> java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>> 4XESTACKTRACE                at java/lang/Thread.run(Thread.java:773)
>> 3XMTHREADINFO3           Native callstack:
>> 4XENATIVESTACK               _event_wait+0x2b8 (0x09000000007E7D3C
>> [libpthreads.a+0x16d3c])
>> 4XENATIVESTACK               _cond_wait_local+0x4e4 (0x09000000007F5A48
>> [libpthreads.a+0x24a48])
>> 4XENATIVESTACK               _cond_wait+0xbc (0x09000000007F6020
>> [libpthreads.a+0x25020])
>> 4XENATIVESTACK               pthread_cond_wait+0x1a8 (0x09000000007F6C8C
>> [libpthreads.a+0x25c8c])
>> 4XENATIVESTACK               (0x0900000001223014 [libj9thr26.so+0x6014])
>> 4XENATIVESTACK               (0x0900000001222C60 [libj9thr26.so+0x5c60])
>> 4XENATIVESTACK               (0x090000000116AE58 [libj9vm26.so+0xfe58])
>> 4XENATIVESTACK               (0x090000000116B17C [libj9vm26.so+0x1017c])
>> 4XENATIVESTACK               (0x0900000001810528
>> [libjclscar_26.so+0x5c528])
>> 4XENATIVESTACK               (0x0900000001813B98
>> [libjclscar_26.so+0x5fb98])
>> 4XENATIVESTACK               (0x0900000001161764 [libj9vm26.so+0x6764])
>> 4XENATIVESTACK               (0x0900000001239CA0 [libj9prt26.so+0x2ca0])
>> 4XENATIVESTACK               (0x09000000011615D4 [libj9vm26.so+0x65d4])
>> 4XENATIVESTACK               (0x090000000121FAF4 [libj9thr26.so+0x2af4])
>> 4XENATIVESTACK               _pthread_body+0xf0 (0x09000000007D4D34
>> [libpthreads.a+0x3d34])
>> NULL
>> 
>> Would you like me to attach the complete sleep.tprof and javacore.txt file
>> to the Jira ticket that I just created?
>> 
>> https://issues.apache.org/jira/browse/LOG4J2-571
>> 
>> Thanks for looking, this has been driving me nuts.
>> 
>> -Chris
>> 
>> 
>> 
>> On Wed, Mar 19, 2014 at 2:45 PM, Remko Popma <[email protected]>
>> wrote:
>> 
>>> Just to double-check: when using blocking wait, and running jstack, does
>>> BlockingWaitStrategy appear in the stack trace?
>>> Also, it it possible to double-check (perhaps attach VisualVM) that it
>>> definitely is the AsyncLoggerConfig-1 thread that consumes so much CPU?
>>> 
>>> 
>>> Sent from my iPhone
>>> 
>>>> On 2014/03/19, at 12:31, Chris Graham <[email protected]> wrote:
>>>> 
>>>> I have tried both Block and Sleep (the default), but not Yield. No
>>>> discernable difference.
>>>> 
>>>> 
>>>>> On Wed, Mar 19, 2014 at 1:53 PM, Remko Popma <[email protected]>
>>> wrote:
>>>>> 
>>>>> As tweeted, I suggest trying the blocking wait strategy. Can you run a
>>>>> jstack dump (and perhaps attach result to a Jira ticket)? In the
>>> attached
>>>>> stack trace below, the AsyncLoggerConfig-1 thread seems to be parked,
>>>>> waiting for a new log event... Doesn't explain high CPU usage...
>>>>> 
>>>>> Sent from my iPhone
>>>>> 
>>>>>> On 2014/03/19, at 10:27, Chris Graham <[email protected]> wrote:
>>>>>> 
>>>>>> Hello Everyone.
>>>>>> 
>>>>>> In this instance, I'm in indirect used of log4j2 2.0-rc1, as it's in
>>> the
>>>>>> web app that I'm using, Apache Archiva 2.0.1.
>>>>>> 
>>>>>> The issue is that when running under WebSphere 8.5.0.2 (obviously on
>>> the
>>>>>> IBM JDK, 1.6) on AIX 6.1 TL8, Apache Archiva when it's doing nothing,
>>> is
>>>>>> sitting idle on around 50% CPU.
>>>>>> 
>>>>>> Obviosuly, this is not good!
>>>>>> 
>>>>>> I've performed the AIX native analysis, to get the native thread ID,
>>>>> mapped
>>>>>> it to a Java thread it, triggered a heap dump, and I've found this as
>>> the
>>>>>> culprit:
>>>>>> 
>>>>>> 3XMTHREADINFO      "AsyncLoggerConfig-1"
>> J9VMThread:0x0000000031D14600,
>>>>>> j9thread_t:0x00000100137D8BD0, java/lang/Thread:0x000000004301C508,
>>>>>> state:CW, prio=5
>>>>>> 3XMJAVALTHREAD            (java/lang/Thread getId:0x6A,
>> isDaemon:true)
>>>>>> 3XMTHREADINFO1            (native thread ID:0x2BF00F9, native
>>>>> priority:0x5,
>>>>>> native policy:UNKNOWN)
>>>>>> 3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0
>>> (0x0)
>>>>>> 3XMTHREADINFO3           Java callstack:
>>>>>> 4XESTACKTRACE                at sun/misc/Unsafe.park(Native Method)
>>>>>> 4XESTACKTRACE                at
>> java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:332)
>>>>>> 4XESTACKTRACE                at
>> com/lmax/disruptor/SleepingWaitStrategy.applyWaitMethod(SleepingWaitStrategy.java:66)
>>>>>> 4XESTACKTRACE
> 
> 
> 
> -- 
> Matt Sicker <[email protected]>

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to