[
https://issues.apache.org/jira/browse/KAFKA-5721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajini Sivaram resolved KAFKA-5721.
-----------------------------------
Resolution: Fixed
Fix Version/s: 0.11.0.2
The specific deadlock in the stack trace in this JIRA was fixed under
KAFKA-3994 (0.10.2.0). There have been a couple of other deadlocks which have
been fixed since. The two releases that don't have any known deadlocks are
1.0.0 and 0.11.0.2.
> Kafka broker stops after network failure
> ----------------------------------------
>
> Key: KAFKA-5721
> URL: https://issues.apache.org/jira/browse/KAFKA-5721
> Project: Kafka
> Issue Type: Bug
> Reporter: Val Feron
> Fix For: 0.11.0.2
>
> Attachments: thread_dump_kafka.out
>
>
> +Cluster description+
> * 3 brokers
> * version 0.10.1.1
> * running on AWS
> +Description+
> The following will happen at random intervals, on random brokers
> From the logs here is the information I could gather :
> # Shrinking Intra-cluster replication on a random broker (I suppose it could
> be a temporary network failure but couldn't produce evidence of it)
> # System starts showing close to no activity @02:27:20 (note that it's not
> load related as it happens at very quiet times)
> !https://i.stack.imgur.com/g1Pem.png!
> # From there, this kafka broker doesn't process messages which is expected
> IMO as it dropped out of the cluster replication.
> # Now the real issue appears as the number of connections in CLOSE_WAIT is
> constantly increasing until it reaches the configured ulimit of the
> system/process, ending up crashing the kafka process.
> Now, I've been changing limits to see if kafka would eventually join again
> the ISR before crashing but even with a limit that's very high, kafka just
> seems stuck in a weird state and never recovers.
> Note that between the time when the faulty broker is on its own and the time
> it crashes, kafka is listening and kafka producer.
> For this single crash, I could see 320 errors like this from the producers :
> {code}java.util.concurrent.ExecutionException:
> org.springframework.kafka.core.KafkaProducerException: Failed to send; nested
> exception is org.apache.kafka.common.errors.NotLeaderForPartitionException:
> This server is not the leader for that topic-partition.
> {code}
> ----
> The configuration being the default one and the use being quite standard, I'm
> wondering if I missed something.
> I put in place a script that check the number of kafka file descriptors and
> restarts the service when it gets abnormally high, which does the trick for
> now but I still lose messages when it crashes.
> I'm available to make any verification / test you need.
> Could see some similarities with ticket KAFKA-5007 too.
> cc [~junrao]
> Confirming the logs seen at the time of failure :
> {code}
> /var/log/kafka/kafkaServer.out:[2017-08-09 22:13:29,045] WARN
> [ReplicaFetcherThread-0-2], Error in fetch
> kafka.server.ReplicaFetcherThread$FetchRequest@4a63f79
> (kafka.server.ReplicaFetcherThread)
> /var/log/kafka/kafkaServer.out-java.io.IOException: Connection to 2 was
> disconnected before the response was read
> /var/log/kafka/kafkaServer.out- at
> kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:115)
> /var/log/kafka/kafkaServer.out- at
> kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:112)
> /var/log/kafka/kafkaServer.out- at
> scala.Option.foreach(Option.scala:257)
> /var/log/kafka/kafkaServer.out- at
> kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:112)
> /var/log/kafka/kafkaServer.out- at
> kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:108)
> /var/log/kafka/kafkaServer.out- at
> kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:137)
> /var/log/kafka/kafkaServer.out- at
> kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:143)
> /var/log/kafka/kafkaServer.out- at
> kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:108)
> /var/log/kafka/kafkaServer.out- at
> kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:253)
> /var/log/kafka/kafkaServer.out- at
> kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:238)
> /var/log/kafka/kafkaServer.out- at
> kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
> /var/log/kafka/kafkaServer.out- at
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118)
> /var/log/kafka/kafkaServer.out- at
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103)
> /var/log/kafka/kafkaServer.out- at
> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> {code}
> {code}
> /var/log/kafka/kafkaServer.out:[2017-08-09 22:17:39,940] WARN
> [ReplicaFetcherThread-0-2], Error in fetch
> kafka.server.ReplicaFetcherThread$FetchRequest@3a8493a8
> (kafka.server.ReplicaFetcherThread)
> /var/log/kafka/kafkaServer.out-java.io.IOException: Connection
> [masked:xxx.xxx.xxx.xxx]:9092 (id: 2 rack: null) failed
> /var/log/kafka/kafkaServer.out- at
> kafka.utils.NetworkClientBlockingOps$.awaitReady$1(NetworkClientBlockingOps.scala:83)
> /var/log/kafka/kafkaServer.out- at
> kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:93)
> /var/log/kafka/kafkaServer.out- at
> kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:248)
> /var/log/kafka/kafkaServer.out- at
> kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:238)
> /var/log/kafka/kafkaServer.out- at
> kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
> /var/log/kafka/kafkaServer.out- at
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118)
> /var/log/kafka/kafkaServer.out- at
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103)
> /var/log/kafka/kafkaServer.out- at
> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> {code}
> Finally, here is the thread dump of the faulty node at the time of failure
> (taken when the number of connections in any state on port 9092 > 1000) :
> {code}
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode):
> "Attach Listener" #57 daemon prio=9 os_prio=0 tid=0x00007f8d94003000
> nid=0x20fd waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "executor-Produce" #56 prio=5 os_prio=0 tid=0x00007f8d78002800 nid=0x1c02
> waiting on condition [0x00007f8d402fd000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c06d8008> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "executor-Heartbeat" #55 prio=5 os_prio=0 tid=0x00007f8d98040800 nid=0x1bfe
> waiting for monitor entry [0x00007f8d417fe000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> kafka.coordinator.GroupCoordinator.onExpireHeartbeat(GroupCoordinator.scala:739)
> - waiting to lock <0x00000000c8941218> (a
> kafka.coordinator.GroupMetadata)
> at
> kafka.coordinator.DelayedHeartbeat.onExpiration(DelayedHeartbeat.scala:33)
> at kafka.server.DelayedOperation.run(DelayedOperation.scala:107)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-scheduler-9" #54 daemon prio=5 os_prio=0 tid=0x00007f8d58010800
> nid=0x6b09 waiting on condition [0x00007f8ca30f9000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c03239e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-scheduler-8" #53 daemon prio=5 os_prio=0 tid=0x000000000243d000
> nid=0x6b08 waiting on condition [0x00007f8ca3bfa000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c03239e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-scheduler-7" #52 daemon prio=5 os_prio=0 tid=0x00007f8dc003d800
> nid=0x6b07 waiting on condition [0x00007f8ca3cfb000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c03239e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "ReplicaFetcherThread-0-1" #51 prio=5 os_prio=0 tid=0x00007f8dc003a800
> nid=0x6b06 runnable [0x00007f8ca3dfc000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000c009c370> (a sun.nio.ch.Util$3)
> - locked <0x00000000c009c360> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000c009c380> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.kafka.common.network.Selector.select(Selector.java:470)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
> at
> kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:136)
> at
> kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:143)
> at
> kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:108)
> at
> kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:253)
> at
> kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:238)
> at
> kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
> at
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118)
> at
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> "executor-Fetch" #50 prio=5 os_prio=0 tid=0x00007f8d74003000 nid=0x6b05
> waiting on condition [0x00007f8ca3efd000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c068a9c8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "ReplicaFetcherThread-0-3" #49 prio=5 os_prio=0 tid=0x00007f8dc0026800
> nid=0x6b04 runnable [0x00007f8ca3ffe000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000c009c840> (a sun.nio.ch.Util$3)
> - locked <0x00000000c009c830> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000c009c850> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.kafka.common.network.Selector.select(Selector.java:470)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:286)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
> at
> kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:136)
> at
> kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:143)
> at
> kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:108)
> at
> kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:253)
> at
> kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:238)
> at
> kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
> at
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118)
> at
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> "kafka-request-handler-7" #48 daemon prio=5 os_prio=0 tid=0x00007f8df0d57000
> nid=0x6b01 waiting for monitor entry [0x00007f8d5c3fe000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> kafka.coordinator.GroupCoordinator.doCommitOffsets(GroupCoordinator.scala:432)
> - waiting to lock <0x00000000c8941218> (a
> kafka.coordinator.GroupMetadata)
> at
> kafka.coordinator.GroupCoordinator.handleCommitOffsets(GroupCoordinator.scala:420)
> at kafka.server.KafkaApis.handleOffsetCommitRequest(KafkaApis.scala:322)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:86)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-request-handler-6" #47 daemon prio=5 os_prio=0 tid=0x00007f8df0d55000
> nid=0x6b00 waiting for monitor entry [0x00007f8d681eb000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> kafka.server.DelayedOperationPurgatory$Watchers.tryCompleteWatched(DelayedOperation.scala:308)
> - waiting to lock <0x00000000da1804a8> (a java.util.LinkedList)
> at
> kafka.server.DelayedOperationPurgatory.checkAndComplete(DelayedOperation.scala:234)
> at
> kafka.server.ReplicaManager.tryCompleteDelayedProduce(ReplicaManager.scala:199)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:909)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:902)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> kafka.server.ReplicaManager.updateFollowerLogReadResults(ReplicaManager.scala:902)
> at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:475)
> at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:523)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:79)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-request-handler-5" #46 daemon prio=5 os_prio=0 tid=0x00007f8df0d53000
> nid=0x6aff waiting for monitor entry [0x00007f8d682ec000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> kafka.coordinator.GroupCoordinator.doCommitOffsets(GroupCoordinator.scala:432)
> - waiting to lock <0x00000000c8941218> (a
> kafka.coordinator.GroupMetadata)
> at
> kafka.coordinator.GroupCoordinator.handleCommitOffsets(GroupCoordinator.scala:420)
> at kafka.server.KafkaApis.handleOffsetCommitRequest(KafkaApis.scala:322)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:86)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-request-handler-4" #45 daemon prio=5 os_prio=0 tid=0x00007f8df0d51000
> nid=0x6afe waiting for monitor entry [0x00007f8d683ed000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> kafka.coordinator.GroupCoordinator.doCommitOffsets(GroupCoordinator.scala:432)
> - waiting to lock <0x00000000c8941218> (a
> kafka.coordinator.GroupMetadata)
> at
> kafka.coordinator.GroupCoordinator.handleCommitOffsets(GroupCoordinator.scala:420)
> at kafka.server.KafkaApis.handleOffsetCommitRequest(KafkaApis.scala:322)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:86)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-request-handler-3" #44 daemon prio=5 os_prio=0 tid=0x00007f8df0d4f800
> nid=0x6afd waiting for monitor entry [0x00007f8d684ee000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> kafka.server.DelayedOperationPurgatory$Watchers.tryCompleteWatched(DelayedOperation.scala:308)
> - waiting to lock <0x00000000da1804a8> (a java.util.LinkedList)
> at
> kafka.server.DelayedOperationPurgatory.checkAndComplete(DelayedOperation.scala:234)
> at
> kafka.server.ReplicaManager.tryCompleteDelayedProduce(ReplicaManager.scala:199)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:909)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:902)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> kafka.server.ReplicaManager.updateFollowerLogReadResults(ReplicaManager.scala:902)
> at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:475)
> at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:523)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:79)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-request-handler-2" #43 daemon prio=5 os_prio=0 tid=0x00007f8df0d4a800
> nid=0x6afc waiting for monitor entry [0x00007f8d685ef000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> kafka.coordinator.GroupCoordinator.handleHeartbeat(GroupCoordinator.scala:351)
> - waiting to lock <0x00000000c8941218> (a
> kafka.coordinator.GroupMetadata)
> at kafka.server.KafkaApis.handleHeartbeatRequest(KafkaApis.scala:1095)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:90)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-request-handler-1" #42 daemon prio=5 os_prio=0 tid=0x00007f8df0d49000
> nid=0x6afb waiting for monitor entry [0x00007f8d686f0000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> kafka.coordinator.GroupMetadataManager.kafka$coordinator$GroupMetadataManager$$putCacheCallback$2(GroupMetadataManager.scala:301)
> - waiting to lock <0x00000000c8941218> (a
> kafka.coordinator.GroupMetadata)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$prepareStoreOffsets$1.apply(GroupMetadataManager.scala:357)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$prepareStoreOffsets$1.apply(GroupMetadataManager.scala:357)
> at kafka.server.DelayedProduce.onComplete(DelayedProduce.scala:123)
> at
> kafka.server.DelayedOperation.forceComplete(DelayedOperation.scala:70)
> at kafka.server.DelayedProduce.tryComplete(DelayedProduce.scala:105)
> at
> kafka.server.DelayedOperationPurgatory$Watchers.tryCompleteWatched(DelayedOperation.scala:315)
> - locked <0x00000000da180450> (a kafka.server.DelayedProduce)
> - locked <0x00000000da1804a8> (a java.util.LinkedList)
> at
> kafka.server.DelayedOperationPurgatory.checkAndComplete(DelayedOperation.scala:234)
> at
> kafka.server.ReplicaManager.tryCompleteDelayedProduce(ReplicaManager.scala:199)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:909)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:902)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> kafka.server.ReplicaManager.updateFollowerLogReadResults(ReplicaManager.scala:902)
> at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:475)
> at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:523)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:79)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-request-handler-0" #41 daemon prio=5 os_prio=0 tid=0x00007f8df0d46800
> nid=0x6afa waiting for monitor entry [0x00007f8d68a78000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> kafka.server.DelayedOperationPurgatory$Watchers.tryCompleteWatched(DelayedOperation.scala:308)
> - waiting to lock <0x00000000da1804a8> (a java.util.LinkedList)
> at
> kafka.server.DelayedOperationPurgatory.checkAndComplete(DelayedOperation.scala:234)
> at
> kafka.server.ReplicaManager.tryCompleteDelayedProduce(ReplicaManager.scala:199)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:909)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:902)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> kafka.server.ReplicaManager.updateFollowerLogReadResults(ReplicaManager.scala:902)
> at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:475)
> at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:523)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:79)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> "group-metadata-manager-0" #40 daemon prio=5 os_prio=0 tid=0x00007f8df0d2b000
> nid=0x6af9 waiting for monitor entry [0x00007f8d68b79000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> kafka.server.DelayedOperationPurgatory$Watchers.tryCompleteWatched(DelayedOperation.scala:308)
> - waiting to lock <0x00000000da1804a8> (a java.util.LinkedList)
> at
> kafka.server.DelayedOperationPurgatory.checkAndComplete(DelayedOperation.scala:234)
> at
> kafka.server.ReplicaManager.tryCompleteDelayedProduce(ReplicaManager.scala:199)
> at
> kafka.cluster.Partition.tryCompleteDelayedRequests(Partition.scala:374)
> at kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:457)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$cleanupGroupMetadata$1$$anonfun$apply$10.apply(GroupMetadataManager.scala:600)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$cleanupGroupMetadata$1$$anonfun$apply$10.apply(GroupMetadataManager.scala:593)
> at scala.Option.foreach(Option.scala:257)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$cleanupGroupMetadata$1.apply(GroupMetadataManager.scala:593)
> - locked <0x00000000c8941218> (a kafka.coordinator.GroupMetadata)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$cleanupGroupMetadata$1.apply(GroupMetadataManager.scala:579)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at kafka.utils.Pool$$anon$1.foreach(Pool.scala:89)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at kafka.utils.Pool.foreach(Pool.scala:26)
> at
> kafka.coordinator.GroupMetadataManager.cleanupGroupMetadata(GroupMetadataManager.scala:579)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$1.apply$mcV$sp(GroupMetadataManager.scala:101)
> at
> kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110)
> at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:58)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "ExpirationReaper-2" #39 prio=5 os_prio=0 tid=0x00007f8df0d02000 nid=0x6af8
> waiting on condition [0x00007f8d68c7a000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c068c358> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
> at kafka.utils.timer.SystemTimer.advanceClock(Timer.scala:106)
> at
> kafka.server.DelayedOperationPurgatory.advanceClock(DelayedOperation.scala:350)
> at
> kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper.doWork(DelayedOperation.scala:374)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> "ExpirationReaper-2" #38 prio=5 os_prio=0 tid=0x00007f8df0cf0800 nid=0x6af7
> waiting on condition [0x00007f8d68d7b000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c068c5b0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
> at kafka.utils.timer.SystemTimer.advanceClock(Timer.scala:106)
> at
> kafka.server.DelayedOperationPurgatory.advanceClock(DelayedOperation.scala:350)
> at
> kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper.doWork(DelayedOperation.scala:374)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> "ExpirationReaper-2" #37 prio=5 os_prio=0 tid=0x00007f8df0cdc800 nid=0x6af6
> waiting on condition [0x00007f8d68e7c000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c068c808> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
> at kafka.utils.timer.SystemTimer.advanceClock(Timer.scala:106)
> at
> kafka.server.DelayedOperationPurgatory.advanceClock(DelayedOperation.scala:350)
> at
> kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper.doWork(DelayedOperation.scala:374)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> "kafka-scheduler-6" #36 daemon prio=5 os_prio=0 tid=0x00007f8d6007a800
> nid=0x6af5 waiting on condition [0x00007f8d68f7d000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c03239e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-scheduler-5" #35 daemon prio=5 os_prio=0 tid=0x00007f8df0c58000
> nid=0x6af4 waiting on condition [0x00007f8d6907e000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c03239e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-scheduler-4" #34 daemon prio=5 os_prio=0 tid=0x00007f8d58018800
> nid=0x6af3 waiting on condition [0x00007f8d6917f000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c03239e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-scheduler-3" #33 daemon prio=5 os_prio=0 tid=0x00007f8df0c56800
> nid=0x6af2 waiting on condition [0x00007f8d69280000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c03239e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "metrics-meter-tick-thread-2" #32 daemon prio=5 os_prio=0
> tid=0x00007f8df0c54000 nid=0x6af1 waiting on condition [0x00007f8d6a7f3000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c068caf8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "ExpirationReaper-2" #31 prio=5 os_prio=0 tid=0x00007f8df0c4a800 nid=0x6af0
> waiting on condition [0x00007f8d6a8f4000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c068de68> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
> at kafka.utils.timer.SystemTimer.advanceClock(Timer.scala:106)
> at
> kafka.server.DelayedOperationPurgatory.advanceClock(DelayedOperation.scala:350)
> at
> kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper.doWork(DelayedOperation.scala:374)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> "ExpirationReaper-2" #30 prio=5 os_prio=0 tid=0x00007f8df0c3d000 nid=0x6aef
> waiting on condition [0x00007f8d6a9f5000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c068e0c0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
> at kafka.utils.timer.SystemTimer.advanceClock(Timer.scala:106)
> at
> kafka.server.DelayedOperationPurgatory.advanceClock(DelayedOperation.scala:350)
> at
> kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper.doWork(DelayedOperation.scala:374)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> "kafka-socket-acceptor-PLAINTEXT-9092" #29 prio=5 os_prio=0
> tid=0x00007f8df0bde000 nid=0x6aee runnable [0x00007f8d6aaf6000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000c068e348> (a sun.nio.ch.Util$3)
> - locked <0x00000000c068e338> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000c068e358> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at kafka.network.Acceptor.run(SocketServer.scala:260)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-network-thread-2-PLAINTEXT-1" #28 prio=5 os_prio=0
> tid=0x00007f8df0bdc000 nid=0x6aed waiting on condition [0x00007f8d6abf7000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c06ba8d0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:353)
> at kafka.network.RequestChannel.sendRequest(RequestChannel.scala:212)
> at
> kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:493)
> at
> kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:487)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> kafka.network.Processor.processCompletedReceives(SocketServer.scala:487)
> at kafka.network.Processor.run(SocketServer.scala:417)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-network-thread-2-PLAINTEXT-0" #27 prio=5 os_prio=0
> tid=0x00007f8df0bda800 nid=0x6aec waiting on condition [0x00007f8d6acf8000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c06ba8d0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:353)
> at kafka.network.RequestChannel.sendRequest(RequestChannel.scala:212)
> at
> kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:493)
> at
> kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:487)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> kafka.network.Processor.processCompletedReceives(SocketServer.scala:487)
> at kafka.network.Processor.run(SocketServer.scala:417)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-log-cleaner-thread-0" #23 prio=5 os_prio=0 tid=0x00007f8df08d4800
> nid=0x6aeb waiting on condition [0x00007f8d6aff9000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c068eb60> (a
> java.util.concurrent.CountDownLatch$Sync)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:258)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:218)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> "kafka-scheduler-2" #26 daemon prio=5 os_prio=0 tid=0x00007f8df08d2000
> nid=0x6aea waiting on condition [0x00007f8d6b235000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c03239e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-scheduler-1" #25 daemon prio=5 os_prio=0 tid=0x00007f8df08cf800
> nid=0x6ae9 waiting on condition [0x00007f8d6b336000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c03239e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-scheduler-0" #24 daemon prio=5 os_prio=0 tid=0x00007f8df08ce800
> nid=0x6ae8 waiting on condition [0x00007f8db41d2000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c03239e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "metrics-meter-tick-thread-1" #22 daemon prio=5 os_prio=0
> tid=0x00007f8df0b2a800 nid=0x6ae7 waiting on condition [0x00007f8d6bbfe000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c068caf8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "main-EventThread" #19 daemon prio=5 os_prio=0 tid=0x00007f8df0a81800
> nid=0x6a8c waiting on condition [0x00007f8db42d3000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c0172470> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
> "main-SendThread(xxx.xxx.xxx.xxx:2181)" #18 daemon prio=5 os_prio=0
> tid=0x00007f8df0a87800 nid=0x6a8b runnable [0x00007f8db43d4000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x00000000c0172588> (a sun.nio.ch.Util$3)
> - locked <0x00000000c0172598> (a java.util.Collections$UnmodifiableSet)
> - locked <0x00000000c0172540> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
> "ZkClient-EventThread-17-xxx.xxx.22.69:2181,xxx.xxx.37.169:2181,xxx.xxx.47.202:2181"
> #17 daemon prio=5 os_prio=0 tid=0x00007f8df0a71000 nid=0x6a8a waiting on
> condition [0x00007f8db44d5000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c0175d18> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:67)
> "ThrottledRequestReaper-Produce" #16 prio=5 os_prio=0 tid=0x00007f8df0a62800
> nid=0x6a89 waiting on condition [0x00007f8db45d6000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c0175f40> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
> at
> kafka.server.ClientQuotaManager$ThrottledRequestReaper.doWork(ClientQuotaManager.scala:158)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> "ThrottledRequestReaper-Fetch" #15 prio=5 os_prio=0 tid=0x00007f8df0a5d800
> nid=0x6a88 waiting on condition [0x00007f8db46d7000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c0176140> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
> at
> kafka.server.ClientQuotaManager$ThrottledRequestReaper.doWork(ClientQuotaManager.scala:158)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> "SensorExpiryThread" #14 daemon prio=5 os_prio=0 tid=0x00007f8df0a56000
> nid=0x6a87 waiting on condition [0x00007f8db47d8000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c0176188> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "RMI TCP Accept-0" #11 daemon prio=5 os_prio=0 tid=0x00007f8df0811000
> nid=0x6a85 runnable [0x00007f8db4dda000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at
> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
> at java.net.ServerSocket.implAccept(ServerSocket.java:545)
> at java.net.ServerSocket.accept(ServerSocket.java:513)
> at
> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
> at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
> at java.lang.Thread.run(Thread.java:745)
> "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f8df047c800 nid=0x6a84
> runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x00007f8df0471800
> nid=0x6a83 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x00007f8df046f000
> nid=0x6a82 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x00007f8df046d000
> nid=0x6a81 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007f8df046b000
> nid=0x6a80 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
> tid=0x00007f8df0469800 nid=0x6a7f waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f8df0437000 nid=0x6a7e in
> Object.wait() [0x00007f8dc8293000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
> - locked <0x00000000c0177d78> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f8df0432800
> nid=0x6a7d in Object.wait() [0x00007f8dc8394000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> - locked <0x00000000c0179958> (a java.lang.ref.Reference$Lock)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
> "main" #1 prio=5 os_prio=0 tid=0x00007f8df000c800 nid=0x6a70 waiting on
> condition [0x00007f8df6ee2000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c068f298> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
> at kafka.server.KafkaServer.awaitShutdown(KafkaServer.scala:625)
> at
> kafka.server.KafkaServerStartable.awaitShutdown(KafkaServerStartable.scala:70)
> at kafka.Kafka$.main(Kafka.scala:68)
> at kafka.Kafka.main(Kafka.scala)
> "VM Thread" os_prio=0 tid=0x00007f8df042a800 nid=0x6a7c runnable
> "Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007f8df0023800
> nid=0x6a71 runnable
> "Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007f8df0025000
> nid=0x6a72 runnable
> "Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007f8df0027000
> nid=0x6a73 runnable
> "Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007f8df0028800
> nid=0x6a74 runnable
> "G1 Main Concurrent Mark GC Thread" os_prio=0 tid=0x00007f8df0043800
> nid=0x6a7a runnable
> "Gang worker#0 (G1 Parallel Marking Threads)" os_prio=0
> tid=0x00007f8df0045800 nid=0x6a7b runnable
> "G1 Concurrent Refinement Thread#0" os_prio=0 tid=0x00007f8df0032800
> nid=0x6a79 runnable
> "G1 Concurrent Refinement Thread#1" os_prio=0 tid=0x00007f8df0030800
> nid=0x6a78 runnable
> "G1 Concurrent Refinement Thread#2" os_prio=0 tid=0x00007f8df002f000
> nid=0x6a77 runnable
> "G1 Concurrent Refinement Thread#3" os_prio=0 tid=0x00007f8df002d000
> nid=0x6a76 runnable
> "G1 Concurrent Refinement Thread#4" os_prio=0 tid=0x00007f8df002b000
> nid=0x6a75 runnable
> "VM Periodic Task Thread" os_prio=0 tid=0x00007f8df0826800 nid=0x6a86 waiting
> on condition
> JNI global references: 287
> Found one Java-level deadlock:
> =============================
> "executor-Heartbeat":
> waiting to lock monitor 0x00007f8dbc3df928 (object 0x00000000c8941218, a
> kafka.coordinator.GroupMetadata),
> which is held by "group-metadata-manager-0"
> "group-metadata-manager-0":
> waiting to lock monitor 0x00007f8dac031f58 (object 0x00000000da1804a8, a
> java.util.LinkedList),
> which is held by "kafka-request-handler-1"
> "kafka-request-handler-1":
> waiting to lock monitor 0x00007f8dbc3df928 (object 0x00000000c8941218, a
> kafka.coordinator.GroupMetadata),
> which is held by "group-metadata-manager-0"
> Java stack information for the threads listed above:
> ===================================================
> "executor-Heartbeat":
> at
> kafka.coordinator.GroupCoordinator.onExpireHeartbeat(GroupCoordinator.scala:739)
> - waiting to lock <0x00000000c8941218> (a
> kafka.coordinator.GroupMetadata)
> at
> kafka.coordinator.DelayedHeartbeat.onExpiration(DelayedHeartbeat.scala:33)
> at kafka.server.DelayedOperation.run(DelayedOperation.scala:107)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "group-metadata-manager-0":
> at
> kafka.server.DelayedOperationPurgatory$Watchers.tryCompleteWatched(DelayedOperation.scala:308)
> - waiting to lock <0x00000000da1804a8> (a java.util.LinkedList)
> at
> kafka.server.DelayedOperationPurgatory.checkAndComplete(DelayedOperation.scala:234)
> at
> kafka.server.ReplicaManager.tryCompleteDelayedProduce(ReplicaManager.scala:199)
> at
> kafka.cluster.Partition.tryCompleteDelayedRequests(Partition.scala:374)
> at kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:457)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$cleanupGroupMetadata$1$$anonfun$apply$10.apply(GroupMetadataManager.scala:600)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$cleanupGroupMetadata$1$$anonfun$apply$10.apply(GroupMetadataManager.scala:593)
> at scala.Option.foreach(Option.scala:257)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$cleanupGroupMetadata$1.apply(GroupMetadataManager.scala:593)
> - locked <0x00000000c8941218> (a kafka.coordinator.GroupMetadata)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$cleanupGroupMetadata$1.apply(GroupMetadataManager.scala:579)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at kafka.utils.Pool$$anon$1.foreach(Pool.scala:89)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at kafka.utils.Pool.foreach(Pool.scala:26)
> at
> kafka.coordinator.GroupMetadataManager.cleanupGroupMetadata(GroupMetadataManager.scala:579)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$1.apply$mcV$sp(GroupMetadataManager.scala:101)
> at
> kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110)
> at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:58)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-request-handler-1":
> at
> kafka.coordinator.GroupMetadataManager.kafka$coordinator$GroupMetadataManager$$putCacheCallback$2(GroupMetadataManager.scala:301)
> - waiting to lock <0x00000000c8941218> (a
> kafka.coordinator.GroupMetadata)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$prepareStoreOffsets$1.apply(GroupMetadataManager.scala:357)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$prepareStoreOffsets$1.apply(GroupMetadataManager.scala:357)
> at kafka.server.DelayedProduce.onComplete(DelayedProduce.scala:123)
> at
> kafka.server.DelayedOperation.forceComplete(DelayedOperation.scala:70)
> at kafka.server.DelayedProduce.tryComplete(DelayedProduce.scala:105)
> at
> kafka.server.DelayedOperationPurgatory$Watchers.tryCompleteWatched(DelayedOperation.scala:315)
> - locked <0x00000000da180450> (a kafka.server.DelayedProduce)
> - locked <0x00000000da1804a8> (a java.util.LinkedList)
> at
> kafka.server.DelayedOperationPurgatory.checkAndComplete(DelayedOperation.scala:234)
> at
> kafka.server.ReplicaManager.tryCompleteDelayedProduce(ReplicaManager.scala:199)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:909)
> at
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:902)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> kafka.server.ReplicaManager.updateFollowerLogReadResults(ReplicaManager.scala:902)
> at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:475)
> at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:523)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:79)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> Found 1 deadlock.
> {code}
> Also attached as text file if that's easier for you.
> (I've masked the ip addresses)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
