[ https://issues.apache.org/jira/browse/KAFKA-9189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082642#comment-17082642 ]
David Mao commented on KAFKA-9189: ---------------------------------- also observed this issue, took a thread-dump and found that controlledShutdown() can block indefinitely in zkClient.getControllerId {code:java} "kafka-shutdown-hook" #24 prio=5 os_prio=0 cpu=9.96ms elapsed=184687.63s tid=0x00007f3b20786000 nid=0x4e69 waiting on condition [0x00007f3a8559a000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.5/Native Method) - parking to wait for <0x00000006884f6808> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.5/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.5/AbstractQueuedSynchronizer.java:2123) at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:264) at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:258) at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$1(ZooKeeperClient.scala:252) at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:252) at kafka.zk.KafkaZkClient.retryRequestsUntilConnected(KafkaZkClient.scala:1783) at kafka.zk.KafkaZkClient.retryRequestsUntilConnected(KafkaZkClient.scala:1753) at kafka.zk.KafkaZkClient.retryRequestUntilConnected(KafkaZkClient.scala:1748) at kafka.zk.KafkaZkClient.getControllerId(KafkaZkClient.scala:1119) at kafka.server.KafkaServer.doControlledShutdown$1(KafkaServer.scala:688) at kafka.server.KafkaServer.controlledShutdown(KafkaServer.scala:772) at kafka.server.KafkaServer.$anonfun$shutdown$2(KafkaServer.scala:794) at kafka.server.KafkaServer$$Lambda$1583/0x0000000840924c40.apply$mcV$sp(Unknown Source) at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:68) at kafka.server.KafkaServer.shutdown(KafkaServer.scala:794) at kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:54) at kafka.Kafka$.$anonfun$main$3(Kafka.scala:80) at kafka.Kafka$$$Lambda$172/0x00000008402a1440.apply$mcV$sp(Unknown Source) at kafka.utils.Exit$.$anonfun$addShutdownHook$1(Exit.scala:38) at kafka.Kafka$$$Lambda$173/0x00000008402a7840.run(Unknown Source) at java.lang.Thread.run(java.base@11.0.5/Thread.java:834){code} > Shutdown is blocked if connection to Zookeeper is lost > ------------------------------------------------------ > > Key: KAFKA-9189 > URL: https://issues.apache.org/jira/browse/KAFKA-9189 > Project: Kafka > Issue Type: Bug > Components: core > Affects Versions: 2.3.0 > Environment: Linux, Docker 19.03.4 > Reporter: Boris Granveaud > Priority: Minor > > We are using Kafka and Zookeeper in Docker swarm stacks. When we undeploy a > stack, sometimes Kafka doesn't shutdown properly and is finally killed by > Docker (thus leaving potentially corrupted files). > Here are the steps to reproduce (simple Docker, no swarm): > > {code:java} > docker network create test > docker run -d --network test --name zk --rm zookeeper:3.5.6 > docker run --network test --name kf --rm -e "KAFKA_ZOOKEEPER_CONNECT=zk:2181" > -e "KAFKA_ADVERTISED_LISTENERS=INSIDE://:9091" -e > "KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INSIDE:PLAINTEXT" -e > "KAFKA_INTER_BROKER_LISTENER_NAME=INSIDE" confluentinc/cp-kafka:5.3.1 > {code} > > In another shell: > > {code:java} > docker stop zk > docker stop kf > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005)