[jira] [Updated] (FLUME-3082) JVM hang?I extended the functionality of tailDIrSource,but sometimes flume process hang,i do not down know why

[lancerlin (JIRA)]

     [ 
https://issues.apache.org/jira/browse/FLUME-3082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

lancerlin updated FLUME-3082:
-----------------------------
    Summary: JVM hang?I extended the functionality of tailDIrSource,but 
sometimes flume process hang,i do not down know why  (was: I extended the 
functionality of tailDIrSource,but sometimes flume process hang,i do not down 
know why)

> JVM hang?I extended the functionality of tailDIrSource,but sometimes flume 
> process hang,i do not down know why
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: FLUME-3082
>                 URL: https://issues.apache.org/jira/browse/FLUME-3082
>             Project: Flume
>          Issue Type: Question
>          Components: Sinks+Sources
>    Affects Versions: 1.7.0
>         Environment: jdk1.7  flume1.6
>            Reporter: lancerlin
>             Fix For: 1.7.0
>
>
> I extended the functionality of tailDIrSource, so it can tail file like 
> /logs/server_.*/.*.log. but sometimes flume process hang,i do not know why 
> and how to deal with.
> this is the jvm info when it hang.
>   S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT   
>   4.52   0.00 100.00   7.21  59.82  67314  255.106     8    0.493  255.599
>   4.52   0.00 100.00   7.21  59.82  67314  255.106     8    0.493  255.599
>   4.52   0.00 100.00   7.21  59.82  67314  255.106     8    0.493  255.599
>   4.52   0.00 100.00   7.21  59.82  67314  255.106     8    0.493  255.599
>   4.52   0.00 100.00   7.21  59.82  67314  255.106     8    0.493  255.599
> #####################################
> using parallel threads in the new generation.
> using thread-local object allocation.
> Concurrent Mark-Sweep GC
> Heap Configuration:
>    MinHeapFreeRatio = 40
>    MaxHeapFreeRatio = 70
>    MaxHeapSize      = 1073741824 (1024.0MB)
>    NewSize          = 348913664 (332.75MB)
>    MaxNewSize       = 348913664 (332.75MB)
>    OldSize          = 697892864 (665.5625MB)
>    NewRatio         = 2
>    SurvivorRatio    = 8
>    PermSize         = 21757952 (20.75MB)
>    MaxPermSize      = 85983232 (82.0MB)
>    G1HeapRegionSize = 0 (0.0MB)
> Heap Usage:
> New Generation (Eden + 1 Survivor Space):
>    capacity = 314048512 (299.5MB)
>    used     = 280758672 (267.75233459472656MB)
>    free     = 33289840 (31.747665405273438MB)
>    89.39977782795545% used
> Eden Space:
>    capacity = 279183360 (266.25MB)
>    used     = 279183360 (266.25MB)
>    free     = 0 (0.0MB)
>    100.0% used
> From Space:
>    capacity = 34865152 (33.25MB)
>    used     = 1575312 (1.5023345947265625MB)
>    free     = 33289840 (31.747665405273438MB)
>    4.518299533012218% used
> To Space:
>    capacity = 34865152 (33.25MB)
>    used     = 0 (0.0MB)
>    free     = 34865152 (33.25MB)
>    0.0% used
> concurrent mark-sweep generation:
>    capacity = 724828160 (691.25MB)
>    used     = 52230760 (49.811134338378906MB)
>    free     = 672597400 (641.4388656616211MB)
>    7.205950718029499% used
> Perm Generation:
>    capacity = 40906752 (39.01171875MB)
>    used     = 24471232 (23.33758544921875MB)
>    free     = 16435520 (15.67413330078125MB)
>    59.82198733353359% used
> 9304 interned Strings occupying 1033824 bytes.
> #####################################
> 2017-04-06 09:38:39
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode):
> "Attach Listener" daemon prio=10 tid=0x00007f3e50001000 nid=0x108e4 waiting 
> on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "121376138@qtp-767421574-5" prio=10 tid=0x00007f3e1c007800 nid=0x108c6 in 
> Object.wait() [0x00007f3e5558e000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000000cb840000> (a 
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>       at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
>       - locked <0x00000000cb840000> (a 
> org.mortbay.thread.QueuedThreadPool$PoolThread)
> "984367986@qtp-767421574-4" prio=10 tid=0x00007f3e1c006800 nid=0x108c5 in 
> Object.wait() [0x00007f3e5568f000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000000cb8694e0> (a 
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>       at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
>       - locked <0x00000000cb8694e0> (a 
> org.mortbay.thread.QueuedThreadPool$PoolThread)
> "1244193860@qtp-767421574-3" prio=10 tid=0x00007f3e1c001800 nid=0x108c4 in 
> Object.wait() [0x00007f3e55790000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000000cb840070> (a 
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>       at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
>       - locked <0x00000000cb840070> (a 
> org.mortbay.thread.QueuedThreadPool$PoolThread)
> "lifecycleSupervisor-1-9" prio=10 tid=0x00007f3e28002000 nid=0x1a413 waiting 
> on condition [0x00007f3e55992000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "lifecycleSupervisor-1-8" prio=10 tid=0x00007f3e34001000 nid=0x1a355 waiting 
> on condition [0x00007f3e55a93000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "lifecycleSupervisor-1-7" prio=10 tid=0x00007f3e30012800 nid=0x1a318 waiting 
> on condition [0x00007f3e5606d000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "PollableSourceRunner-TaildirSource-r1" prio=10 tid=0x00007f3e30010800 
> nid=0x1a317 waiting on condition [0x00007f3e5616e000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>       at java.lang.Thread.sleep(Native Method)
>       at java.lang.Thread.sleep(Thread.java:340)
>       at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:360)
>       at 
> com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly(Uninterruptibles.java:286)
>       at 
> com.google.common.util.concurrent.RateLimiter$SleepingStopwatch$1.sleepMicrosUninterruptibly(RateLimiter.java:427)
>       at 
> com.google.common.util.concurrent.RateLimiter.acquire(RateLimiter.java:277)
>       at 
> com.google.common.util.concurrent.RateLimiter.acquire(RateLimiter.java:263)
>       at 
> org.apache.flume.source.taildir.TaildirSource.tailFileProcess(TaildirSource.java:503)
>       at 
> org.apache.flume.source.taildir.TaildirSource.process(TaildirSource.java:472)
>       at 
> org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
>       at java.lang.Thread.run(Thread.java:745)
> "fileLinesCountDBSaver" prio=10 tid=0x00007f3e3002c000 nid=0x1a315 waiting on 
> condition [0x00007f3e742ba000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae1628> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "commons-pool-EvictionTimer" daemon prio=10 tid=0x00007f3e3000f800 
> nid=0x1a314 in Object.wait() [0x00007f3e741b9000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000000cfae3bf0> (a java.util.TaskQueue)
>       at java.util.TimerThread.mainLoop(Timer.java:552)
>       - locked <0x00000000cfae3bf0> (a java.util.TaskQueue)
>       at java.util.TimerThread.run(Timer.java:505)
> "positionWriter" prio=10 tid=0x00007f3e3001a000 nid=0x1a311 waiting on 
> condition [0x00007f3e743bb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae1658> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "idleFileChecker" prio=10 tid=0x00007f3e30019000 nid=0x1a310 waiting on 
> condition [0x00007f3e744bc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f58> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "DestroyJavaVM" prio=10 tid=0x00007f3e8800f800 nid=0x19db2 waiting on 
> condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "flumeMetricsJsonRegister" prio=10 tid=0x00007f3e888f4000 nid=0x19f29 waiting 
> on condition [0x00007f3e745bd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae1688> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "flumeHeartBeatRegister" prio=10 tid=0x00007f3e88852800 nid=0x19f27 waiting 
> on condition [0x00007f3e746be000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae3c20> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "Druid-ConnectionPool-Destroy-2140821323" daemon prio=10 
> tid=0x00007f3e8885f800 nid=0x19ece waiting on condition [0x00007f3e747bf000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>       at java.lang.Thread.sleep(Native Method)
>       at 
> com.alibaba.druid.pool.DruidDataSource$DestroyConnectionThread.run(DruidDataSource.java:2076)
> "Druid-ConnectionPool-Create-2140821323" daemon prio=10 
> tid=0x00007f3e888ba800 nid=0x19ecb waiting on condition [0x00007f3e748c0000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f88> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:1984)
> "lifecycleSupervisor-1-6" prio=10 tid=0x00007f3e381a6000 nid=0x19e97 waiting 
> on condition [0x00007f3e749c1000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "SinkRunner-PollingRunner-DefaultSinkProcessor" prio=10 
> tid=0x00007f3e381a4800 nid=0x19e96 waiting on condition [0x00007f3e74ac2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfb5b660> (a 
> java.util.concurrent.Semaphore$NonfairSync)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1033)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326)
>       at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:414)
>       at 
> org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:100)
>       at 
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>       at 
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>       at 
> com.fenqile.flume.sink.kafka.MultiLineKafkaSink.process(MultiLineKafkaSink.java:134)
>       at 
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>       at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>       at java.lang.Thread.run(Thread.java:745)
> "metrics-meter-tick-thread-2" daemon prio=10 tid=0x00007f3e381a0800 
> nid=0x19e7b waiting on condition [0x00007f3e74bc3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4fa0> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "metrics-meter-tick-thread-1" daemon prio=10 tid=0x00007f3e3819d800 
> nid=0x19e76 waiting on condition [0x00007f3e74cc4000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4fa0> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "1451095068@qtp-767421574-1 - Acceptor0 SelectChannelConnector@0.0.0.0:33333" 
> prio=10 tid=0x00007f3e3c085000 nid=0x19e3e runnable [0x00007f3e74dc5000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>       at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>       at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000000cfae3c50> (a sun.nio.ch.Util$2)
>       - locked <0x00000000cfae3c68> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000000cfae7598> (a sun.nio.ch.EPollSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
>       at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>       at 
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>       at 
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>       at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> "2091996413@qtp-767421574-0" prio=10 tid=0x00007f3e3c095000 nid=0x19e3d in 
> Object.wait() [0x00007f3e74ec6000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000000cfaeaea0> (a 
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>       at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
>       - locked <0x00000000cfaeaea0> (a 
> org.mortbay.thread.QueuedThreadPool$PoolThread)
> "Abandoned connection cleanup thread" daemon prio=10 tid=0x00007f3e8880a800 
> nid=0x19e3c in Object.wait() [0x00007f3e74fc7000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000000cfae4fd0> (a java.lang.ref.ReferenceQueue$Lock)
>       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>       - locked <0x00000000cfae4fd0> (a java.lang.ref.ReferenceQueue$Lock)
>       at 
> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)
> "lifecycleSupervisor-1-5" prio=10 tid=0x00007f3e3c04d000 nid=0x19e3b waiting 
> on condition [0x00007f3e750c8000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "lifecycleSupervisor-1-4" prio=10 tid=0x00007f3e3c04c000 nid=0x19e3a waiting 
> on condition [0x00007f3e751c9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "lifecycleSupervisor-1-3" prio=10 tid=0x00007f3e3801a000 nid=0x19e39 waiting 
> on condition [0x00007f3e752ca000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "lifecycleSupervisor-1-2" prio=10 tid=0x00007f3e3c04e800 nid=0x19e36 waiting 
> on condition [0x00007f3e755d3000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "lifecycleSupervisor-1-1" prio=10 tid=0x00007f3e38008800 nid=0x19dd7 waiting 
> on condition [0x00007f3e756d4000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "conf-file-poller-0" prio=10 tid=0x00007f3e38007000 nid=0x19dd6 waiting on 
> condition [0x00007f3e757d5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfb563d0> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "lifecycleSupervisor-1-0" prio=10 tid=0x00007f3e88744000 nid=0x19dd5 waiting 
> on condition [0x00007f3e758d6000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
> "Service Thread" daemon prio=10 tid=0x00007f3e8811b000 nid=0x19dbf runnable 
> [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread1" daemon prio=10 tid=0x00007f3e88118000 nid=0x19dbe 
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread0" daemon prio=10 tid=0x00007f3e88116000 nid=0x19dbd 
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "Signal Dispatcher" daemon prio=10 tid=0x00007f3e88113800 nid=0x19dbc 
> runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "Surrogate Locker Thread (Concurrent GC)" daemon prio=10 
> tid=0x00007f3e88111800 nid=0x19dbb waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
> "Finalizer" daemon prio=10 tid=0x00007f3e880f1000 nid=0x19dba in 
> Object.wait() [0x00007f3e84cf2000]
>    java.lang.Thread.State: WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000000cfae5078> (a java.lang.ref.ReferenceQueue$Lock)
>       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>       - locked <0x00000000cfae5078> (a java.lang.ref.ReferenceQueue$Lock)
>       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
>       at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> "Reference Handler" daemon prio=10 tid=0x00007f3e880ef000 nid=0x19db9 in 
> Object.wait() [0x00007f3e8c1b8000]
>    java.lang.Thread.State: WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000000cfc03960> (a java.lang.ref.Reference$Lock)
>       at java.lang.Object.wait(Object.java:503)
>       at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
>       - locked <0x00000000cfc03960> (a java.lang.ref.Reference$Lock)
> "VM Thread" prio=10 tid=0x00007f3e880ea800 nid=0x19db8 runnable 
> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f3e88021000 
> nid=0x19db3 runnable 
> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f3e88022800 
> nid=0x19db4 runnable 
> "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f3e88024800 
> nid=0x19db5 runnable 
> "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f3e88026800 
> nid=0x19db6 runnable 
> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f3e880a7000 nid=0x19db7 
> runnable 
> "VM Periodic Task Thread" prio=10 tid=0x00007f3e8812e000 nid=0x19dc0 waiting 
> on condition 
> JNI global references: 162
> #####################################
> 2017-04-06 09:38:40
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode):
> "Attach Listener" daemon prio=10 tid=0x00007f3e50001000 nid=0x108e4 waiting 
> on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>    Locked ownable synchronizers:
>       - None
> "121376138@qtp-767421574-5" prio=10 tid=0x00007f3e1c007800 nid=0x108c6 in 
> Object.wait() [0x00007f3e5558e000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000000cda47850> (a 
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>       at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
>       - locked <0x00000000cda47850> (a 
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>    Locked ownable synchronizers:
>       - None
> "984367986@qtp-767421574-4" prio=10 tid=0x00007f3e1c006800 nid=0x108c5 in 
> Object.wait() [0x00007f3e5568f000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000000cd980068> (a 
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>       at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
>       - locked <0x00000000cd980068> (a 
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>    Locked ownable synchronizers:
>       - None
> "1244193860@qtp-767421574-3" prio=10 tid=0x00007f3e1c001800 nid=0x108c4 in 
> Object.wait() [0x00007f3e55790000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000000cd9800d8> (a 
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>       at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
>       - locked <0x00000000cd9800d8> (a 
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>    Locked ownable synchronizers:
>       - None
> "lifecycleSupervisor-1-9" prio=10 tid=0x00007f3e28002000 nid=0x1a413 waiting 
> on condition [0x00007f3e55992000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "lifecycleSupervisor-1-8" prio=10 tid=0x00007f3e34001000 nid=0x1a355 waiting 
> on condition [0x00007f3e55a93000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "lifecycleSupervisor-1-7" prio=10 tid=0x00007f3e30012800 nid=0x1a318 waiting 
> on condition [0x00007f3e5606d000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "PollableSourceRunner-TaildirSource-r1" prio=10 tid=0x00007f3e30010800 
> nid=0x1a317 sleeping[0x00007f3e5616e000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>       at java.lang.Thread.sleep(Native Method)
>       at java.lang.Thread.sleep(Thread.java:340)
>       at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:360)
>       at 
> com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly(Uninterruptibles.java:286)
>       at 
> com.google.common.util.concurrent.RateLimiter$SleepingStopwatch$1.sleepMicrosUninterruptibly(RateLimiter.java:427)
>       at 
> com.google.common.util.concurrent.RateLimiter.acquire(RateLimiter.java:277)
>       at 
> com.google.common.util.concurrent.RateLimiter.acquire(RateLimiter.java:263)
>       at 
> org.apache.flume.source.taildir.TaildirSource.tailFileProcess(TaildirSource.java:503)
>       at 
> org.apache.flume.source.taildir.TaildirSource.process(TaildirSource.java:472)
>       at 
> org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "fileLinesCountDBSaver" prio=10 tid=0x00007f3e3002c000 nid=0x1a315 waiting on 
> condition [0x00007f3e742ba000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae1628> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "commons-pool-EvictionTimer" daemon prio=10 tid=0x00007f3e3000f800 
> nid=0x1a314 in Object.wait() [0x00007f3e741b9000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000000cfae3bf0> (a java.util.TaskQueue)
>       at java.util.TimerThread.mainLoop(Timer.java:552)
>       - locked <0x00000000cfae3bf0> (a java.util.TaskQueue)
>       at java.util.TimerThread.run(Timer.java:505)
>    Locked ownable synchronizers:
>       - None
> "positionWriter" prio=10 tid=0x00007f3e3001a000 nid=0x1a311 waiting on 
> condition [0x00007f3e743bb000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae1658> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "idleFileChecker" prio=10 tid=0x00007f3e30019000 nid=0x1a310 waiting on 
> condition [0x00007f3e744bc000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f58> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "DestroyJavaVM" prio=10 tid=0x00007f3e8800f800 nid=0x19db2 waiting on 
> condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>    Locked ownable synchronizers:
>       - None
> "flumeMetricsJsonRegister" prio=10 tid=0x00007f3e888f4000 nid=0x19f29 waiting 
> on condition [0x00007f3e745bd000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae1688> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "flumeHeartBeatRegister" prio=10 tid=0x00007f3e88852800 nid=0x19f27 waiting 
> on condition [0x00007f3e746be000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae3c20> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "Druid-ConnectionPool-Destroy-2140821323" daemon prio=10 
> tid=0x00007f3e8885f800 nid=0x19ece waiting on condition [0x00007f3e747bf000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>       at java.lang.Thread.sleep(Native Method)
>       at 
> com.alibaba.druid.pool.DruidDataSource$DestroyConnectionThread.run(DruidDataSource.java:2076)
>    Locked ownable synchronizers:
>       - None
> "Druid-ConnectionPool-Create-2140821323" daemon prio=10 
> tid=0x00007f3e888ba800 nid=0x19ecb waiting on condition [0x00007f3e748c0000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f88> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:1984)
>    Locked ownable synchronizers:
>       - None
> "lifecycleSupervisor-1-6" prio=10 tid=0x00007f3e381a6000 nid=0x19e97 waiting 
> on condition [0x00007f3e749c1000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "SinkRunner-PollingRunner-DefaultSinkProcessor" prio=10 
> tid=0x00007f3e381a4800 nid=0x19e96 waiting on condition [0x00007f3e74ac2000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfb5b660> (a 
> java.util.concurrent.Semaphore$NonfairSync)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1033)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326)
>       at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:414)
>       at 
> org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:100)
>       at 
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>       at 
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>       at 
> com.fenqile.flume.sink.kafka.MultiLineKafkaSink.process(MultiLineKafkaSink.java:134)
>       at 
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>       at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "metrics-meter-tick-thread-2" daemon prio=10 tid=0x00007f3e381a0800 
> nid=0x19e7b waiting on condition [0x00007f3e74bc3000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4fa0> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "metrics-meter-tick-thread-1" daemon prio=10 tid=0x00007f3e3819d800 
> nid=0x19e76 waiting on condition [0x00007f3e74cc4000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4fa0> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "1451095068@qtp-767421574-1 - Acceptor0 SelectChannelConnector@0.0.0.0:33333" 
> prio=10 tid=0x00007f3e3c085000 nid=0x19e3e runnable [0x00007f3e74dc5000]
>    java.lang.Thread.State: RUNNABLE
>       at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>       at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
>       at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>       - locked <0x00000000cfae3c50> (a sun.nio.ch.Util$2)
>       - locked <0x00000000cfae3c68> (a java.util.Collections$UnmodifiableSet)
>       - locked <0x00000000cfae7598> (a sun.nio.ch.EPollSelectorImpl)
>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>       at 
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
>       at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>       at 
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>       at 
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>       at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>    Locked ownable synchronizers:
>       - None
> "2091996413@qtp-767421574-0" prio=10 tid=0x00007f3e3c095000 nid=0x19e3d in 
> Object.wait() [0x00007f3e74ec6000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000000cfaeaea0> (a 
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>       at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
>       - locked <0x00000000cfaeaea0> (a 
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>    Locked ownable synchronizers:
>       - None
> "Abandoned connection cleanup thread" daemon prio=10 tid=0x00007f3e8880a800 
> nid=0x19e3c in Object.wait() [0x00007f3e74fc7000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000000cfae4fd0> (a java.lang.ref.ReferenceQueue$Lock)
>       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>       - locked <0x00000000cfae4fd0> (a java.lang.ref.ReferenceQueue$Lock)
>       at 
> com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)
>    Locked ownable synchronizers:
>       - None
> "lifecycleSupervisor-1-5" prio=10 tid=0x00007f3e3c04d000 nid=0x19e3b waiting 
> on condition [0x00007f3e750c8000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "lifecycleSupervisor-1-4" prio=10 tid=0x00007f3e3c04c000 nid=0x19e3a waiting 
> on condition [0x00007f3e751c9000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "lifecycleSupervisor-1-3" prio=10 tid=0x00007f3e3801a000 nid=0x19e39 waiting 
> on condition [0x00007f3e752ca000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "lifecycleSupervisor-1-2" prio=10 tid=0x00007f3e3c04e800 nid=0x19e36 waiting 
> on condition [0x00007f3e755d3000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "lifecycleSupervisor-1-1" prio=10 tid=0x00007f3e38008800 nid=0x19dd7 waiting 
> on condition [0x00007f3e756d4000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "conf-file-poller-0" prio=10 tid=0x00007f3e38007000 nid=0x19dd6 waiting on 
> condition [0x00007f3e757d5000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfb563d0> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at 
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "lifecycleSupervisor-1-0" prio=10 tid=0x00007f3e88744000 nid=0x19dd5 waiting 
> on condition [0x00007f3e758d6000]
>    java.lang.Thread.State: WAITING (parking)
>       at sun.misc.Unsafe.park(Native Method)
>       - parking to wait for  <0x00000000cfae4f40> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>       at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
>       at 
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
>       at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>       at java.lang.Thread.run(Thread.java:745)
>    Locked ownable synchronizers:
>       - None
> "Service Thread" daemon prio=10 tid=0x00007f3e8811b000 nid=0x19dbf runnable 
> [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>    Locked ownable synchronizers:
>       - None
> "C2 CompilerThread1" daemon prio=10 tid=0x00007f3e88118000 nid=0x19dbe 
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>    Locked ownable synchronizers:
>       - None
> "C2 CompilerThread0" daemon prio=10 tid=0x00007f3e88116000 nid=0x19dbd 
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>    Locked ownable synchronizers:
>       - None
> "Signal Dispatcher" daemon prio=10 tid=0x00007f3e88113800 nid=0x19dbc 
> runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>    Locked ownable synchronizers:
>       - None
> "Surrogate Locker Thread (Concurrent GC)" daemon prio=10 
> tid=0x00007f3e88111800 nid=0x19dbb waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>    Locked ownable synchronizers:
>       - None
> "Finalizer" daemon prio=10 tid=0x00007f3e880f1000 nid=0x19dba in 
> Object.wait() [0x00007f3e84cf2000]
>    java.lang.Thread.State: WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000000cfae5078> (a java.lang.ref.ReferenceQueue$Lock)
>       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>       - locked <0x00000000cfae5078> (a java.lang.ref.ReferenceQueue$Lock)
>       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
>       at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>    Locked ownable synchronizers:
>       - None
> "Reference Handler" daemon prio=10 tid=0x00007f3e880ef000 nid=0x19db9 in 
> Object.wait() [0x00007f3e8c1b8000]
>    java.lang.Thread.State: WAITING (on object monitor)
>       at java.lang.Object.wait(Native Method)
>       - waiting on <0x00000000cfc03960> (a java.lang.ref.Reference$Lock)
>       at java.lang.Object.wait(Object.java:503)
>       at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
>       - locked <0x00000000cfc03960> (a java.lang.ref.Reference$Lock)
>    Locked ownable synchronizers:
>       - None
> "VM Thread" prio=10 tid=0x00007f3e880ea800 nid=0x19db8 runnable 
> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f3e88021000 
> nid=0x19db3 runnable 
> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f3e88022800 
> nid=0x19db4 runnable 
> "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f3e88024800 
> nid=0x19db5 runnable 
> "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f3e88026800 
> nid=0x19db6 runnable 
> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f3e880a7000 nid=0x19db7 
> runnable 
> "VM Periodic Task Thread" prio=10 tid=0x00007f3e8812e000 nid=0x19dc0 waiting 
> on condition 
> JNI global references: 162
> #####################################
> Deadlock Detection:
> No deadlocks found.
> Thread 67812: (state = BLOCKED)
> Thread 67782: (state = BLOCKED)
>  - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
>  - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=291, line=626 
> (Interpreted frame)
> Thread 67781: (state = BLOCKED)
>  - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
>  - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=291, line=626 
> (Interpreted frame)
> Thread 67780: (state = BLOCKED)
>  - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
>  - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=291, line=626 
> (Interpreted frame)
> Thread 107539: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, 
> line=186 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() 
> @bci=42, line=2043 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=98, line=1085 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 107349: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, 
> line=186 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() 
> @bci=42, line=2043 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=98, line=1085 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 107288: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, 
> line=186 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() 
> @bci=42, line=2043 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=98, line=1085 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 107287: (state = BLOCKED)
>  - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
>  - java.lang.Thread.sleep(long, int) @bci=57, line=340 (Interpreted frame)
>  - java.util.concurrent.TimeUnit.sleep(long) @bci=23, line=360 (Interpreted 
> frame)
>  - 
> com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly(long, 
> java.util.concurrent.TimeUnit) @bci=22, line=286 (Interpreted frame)
>  - 
> com.google.common.util.concurrent.RateLimiter$SleepingStopwatch$1.sleepMicrosUninterruptibly(long)
>  @bci=10, line=427 (Compiled frame)
>  - com.google.common.util.concurrent.RateLimiter.acquire(int) @bci=11, 
> line=277 (Compiled frame)
>  - com.google.common.util.concurrent.RateLimiter.acquire() @bci=2, line=263 
> (Compiled frame)
>  - 
> org.apache.flume.source.taildir.TaildirSource.tailFileProcess(org.apache.flume.source.taildir.TailFile,
>  boolean) @bci=12, line=503 (Compiled frame)
>  - org.apache.flume.source.taildir.TaildirSource.process() @bci=95, line=472 
> (Compiled frame)
>  - org.apache.flume.source.PollableSourceRunner$PollingRunner.run() @bci=38, 
> line=139 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 107285: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) 
> @bci=20, line=226 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
>  @bci=68, line=2082 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=122, line=1090 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 107284: (state = BLOCKED)
>  - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
>  - java.util.TimerThread.mainLoop() @bci=201, line=552 (Interpreted frame)
>  - java.util.TimerThread.run() @bci=1, line=505 (Interpreted frame)
> Thread 107281: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) 
> @bci=20, line=226 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
>  @bci=68, line=2082 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=122, line=1090 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 107280: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) 
> @bci=20, line=226 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
>  @bci=68, line=2082 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=122, line=1090 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 105906: (state = BLOCKED)
> Thread 106281: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) 
> @bci=20, line=226 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
>  @bci=68, line=2082 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=122, line=1090 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 106279: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) 
> @bci=20, line=226 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
>  @bci=68, line=2082 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=122, line=1090 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 106190: (state = BLOCKED)
>  - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
>  - com.alibaba.druid.pool.DruidDataSource$DestroyConnectionThread.run() 
> @bci=42, line=2076 (Interpreted frame)
> Thread 106187: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, 
> line=186 (Interpreted frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() 
> @bci=42, line=2043 (Interpreted frame)
>  - com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run() 
> @bci=81, line=1984 (Interpreted frame)
> Thread 106135: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, 
> line=186 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() 
> @bci=42, line=2043 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=98, line=1085 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 106134: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) 
> @bci=20, line=226 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(int,
>  long) @bci=122, line=1033 (Compiled frame)
>  - java.util.concurrent.locks.ReentrantLock$Sync.isHeldExclusively() @bci=7, 
> line=168 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.signal()
>  @bci=4, line=1940 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingDeque.unlinkFirst() @bci=67, line=267 
> (Compiled frame)
>  - java.util.concurrent.LinkedBlockingDeque.pollFirst() @bci=10, line=467 
> (Compiled frame)
>  - java.util.concurrent.LinkedBlockingDeque.poll() @bci=1, line=674 (Compiled 
> frame)
>  - org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake() 
> @bci=104, line=105 (Compiled frame)
>  - org.apache.flume.channel.BasicTransactionSemantics.take() @bci=51, 
> line=113 (Compiled frame)
>  - org.apache.flume.channel.BasicChannelSemantics.take() @bci=26, line=95 
> (Compiled frame)
>  - com.fenqile.flume.sink.kafka.MultiLineKafkaSink.process() @bci=102, 
> line=134 (Compiled frame)
>  - org.apache.flume.sink.DefaultSinkProcessor.process() @bci=4, line=68 
> (Compiled frame)
>  - org.apache.flume.SinkRunner$PollingRunner.run() @bci=24, line=147 
> (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 106107: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) 
> @bci=20, line=226 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
>  @bci=68, line=2082 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=122, line=1090 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 106102: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, 
> line=186 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() 
> @bci=42, line=2043 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=98, line=1085 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 106046: (state = IN_NATIVE)
>  - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 
> (Interpreted frame)
>  - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Interpreted 
> frame)
>  - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Interpreted 
> frame)
>  - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 
> (Interpreted frame)
>  - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Interpreted frame)
>  - org.mortbay.io.nio.SelectorManager$SelectSet.doSelect() @bci=689, line=498 
> (Interpreted frame)
>  - org.mortbay.io.nio.SelectorManager.doSelect(int) @bci=24, line=192 
> (Compiled frame)
>  - org.mortbay.jetty.nio.SelectChannelConnector.accept(int) @bci=5, line=124 
> (Compiled frame)
>  - org.mortbay.jetty.AbstractConnector$Acceptor.run() @bci=153, line=708 
> (Compiled frame)
>  - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=25, line=582 
> (Interpreted frame)
> Thread 106045: (state = BLOCKED)
>  - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
>  - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=291, line=626 
> (Interpreted frame)
> Thread 106044: (state = BLOCKED)
>  - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
>  - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 (Compiled 
> frame)
>  - com.mysql.jdbc.AbandonedConnectionCleanupThread.run() @bci=16, line=43 
> (Compiled frame)
> Thread 106043: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, 
> line=186 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() 
> @bci=42, line=2043 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=98, line=1085 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 106042: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) 
> @bci=20, line=226 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
>  @bci=68, line=2082 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=122, line=1090 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 106041: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, 
> line=186 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() 
> @bci=42, line=2043 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=98, line=1085 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 106038: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, 
> line=186 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() 
> @bci=42, line=2043 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=98, line=1085 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 105943: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, 
> line=186 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() 
> @bci=42, line=2043 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=98, line=1085 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 105942: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) 
> @bci=20, line=226 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
>  @bci=68, line=2082 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=122, line=1090 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 105941: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, 
> line=186 (Compiled frame)
>  - 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() 
> @bci=42, line=2043 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=98, line=1085 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() 
> @bci=1, line=807 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 
> (Compiled frame)
>  - 
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>  @bci=26, line=1130 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> Thread 105916: (state = BLOCKED)
> Thread 105915: (state = BLOCKED)
> Thread 105914: (state = BLOCKED)
>  - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
>  - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 (Interpreted 
> frame)
>  - java.lang.ref.ReferenceQueue.remove() @bci=2, line=151 (Compiled frame)
>  - java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=209 (Compiled 
> frame)
> Thread 105913: (state = BLOCKED)
>  - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
>  - java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
>  - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 
> (Interpreted frame)
> #####################################



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)