massakam commented on issue #3630: Broker can not respond to client requests URL: https://github.com/apache/pulsar/issues/3630#issuecomment-467081440 Each time a broker hung, one thread named `ForkJoinPool.commonPool-worker-x` always stopped in the following state: ``` "ForkJoinPool.commonPool-worker-30" #1098 daemon prio=5 os_prio=0 tid=0x00007f6104002800 nid=0x267f waiting for monitor entry [0x00007f5a85b51000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.pulsar.broker.service.persistent.PersistentSubscription.addConsumer(PersistentSubscription.java:100) - waiting to lock <0x00007f6373bd6600> (a org.apache.pulsar.broker.service.persistent.PersistentSubscription) at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$9(PersistentTopic.java:517) at org.apache.pulsar.broker.service.persistent.PersistentTopic$$Lambda$229/184274928.accept(Unknown Source) at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656) at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:669) at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:1997) at org.apache.pulsar.broker.service.persistent.PersistentTopic.subscribe(PersistentTopic.java:513) at org.apache.pulsar.broker.service.ServerCnx.lambda$22(ServerCnx.java:609) at org.apache.pulsar.broker.service.ServerCnx$$Lambda$203/2100949060.apply(Unknown Source) at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981) at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124) at org.apache.pulsar.broker.service.ServerCnx.lambda$18(ServerCnx.java:592) at org.apache.pulsar.broker.service.ServerCnx$$Lambda$201/788613052.apply(Unknown Source) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider.lambda$null$0(PulsarAuthorizationProvider.java:129) at org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider$$Lambda$199/1961028585.accept(Unknown Source) at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656) at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:669) at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:1997) at org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider.lambda$canConsumeAsync$1(PulsarAuthorizationProvider.java:128) at org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider$$Lambda$198/467223006.accept(Unknown Source) at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656) at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.pulsar.zookeeper.ZooKeeperDataCache.lambda$0(ZooKeeperDataCache.java:66) at org.apache.pulsar.zookeeper.ZooKeeperDataCache$$Lambda$25/41765385.accept(Unknown Source) at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656) at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$7(ZooKeeperCache.java:347) at org.apache.pulsar.zookeeper.ZooKeeperCache$$Lambda$23/778720569.accept(Unknown Source) at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656) at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$11(ZooKeeperCache.java:328) at org.apache.pulsar.zookeeper.ZooKeeperCache$$Lambda$27/1695511770.run(Unknown Source) at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) ``` This thread seems to be related to the processing of ZK cache. Could this be the cause of this problem?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
