Hi everybody, we’ve been seeing problems on restart of our kafka brokers,
trying to acquire the lock before shutdown. After about 45 minutes, the restart
is possible but that makes the deployment too long. We are considering adding a
preStep helm job to remove the lock before restarting, but we’re not sure if
there’s a risk of doing this.
Do you have any pointer on what could be causing this or if it’s safe to remove
it before restarting the brokers?
Thank you!
[2022-07-19 21:35:28,567] INFO [ThrottledChannelReaper-Fetch]: Starting
(kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-07-19 21:35:28,567] INFO [ThrottledChannelReaper-Produce]: Starting
(kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-07-19 21:35:28,567] INFO [ThrottledChannelReaper-Request]: Starting
(kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-07-19 21:35:28,569] INFO [ThrottledChannelReaper-ControllerMutation]:
Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-07-19 21:35:28,585] ERROR [KafkaServer id=4] Fatal error during
KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.KafkaException: Failed to acquire lock on file .lock in
/var/lib/kafka. A Kafka instance in another process or thread is using this
directory.
at kafka.log.LogManager.$anonfun$lockLogDirs$1(LogManager.scala:242)
at
scala.collection.StrictOptimizedIterableOps.flatMap(StrictOptimizedIterableOps.scala:118)
at
scala.collection.StrictOptimizedIterableOps.flatMap$(StrictOptimizedIterableOps.scala:105)
at scala.collection.mutable.ArraySeq.flatMap(ArraySeq.scala:37)
at kafka.log.LogManager.lockLogDirs(LogManager.scala:237)
at kafka.log.LogManager.<init>(LogManager.scala:113)
at kafka.log.LogManager$.apply(LogManager.scala:1319)
at kafka.server.KafkaServer.startup(KafkaServer.scala:267)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
[2022-07-19 21:35:28,588] INFO [KafkaServer id=4] shutting down
(kafka.server.KafkaServer)
[2022-07-19 21:35:28,591] INFO [feature-zk-node-event-process-thread]: Shutting
down
(kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2022-07-19 21:35:28,591] INFO [feature-zk-node-event-process-thread]: Shutdown
completed
(kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2022-07-19 21:35:28,591] INFO [ZooKeeperClient Kafka server] Closing.
(kafka.zookeeper.ZooKeeperClient)
[2022-07-19 21:35:28,697] INFO Session: 0x2007215e11f0002 closed
(org.apache.zookeeper.ZooKeeper)
[2022-07-19 21:35:28,697] INFO EventThread shut down for session:
0x2007215e11f0002 (org.apache.zookeeper.ClientCnxn)
[2022-07-19 21:35:28,698] INFO [ZooKeeperClient Kafka server] Closed.
(kafka.zookeeper.ZooKeeperClient)
[2022-07-19 21:35:28,698] INFO [ThrottledChannelReaper-Fetch]: Shutting down
(kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-07-19 21:35:28,700] INFO [ThrottledChannelReaper-Fetch]: Shutdown
completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-07-19 21:35:28,700] INFO [ThrottledChannelReaper-Produce]: Shutting down
(kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-07-19 21:35:28,700] INFO [ThrottledChannelReaper-Produce]: Shutdown
completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-07-19 21:35:28,700] INFO [ThrottledChannelReaper-Request]: Shutting down
(kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-07-19 21:35:28,700] INFO [ThrottledChannelReaper-Fetch]: Stopped
(kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-07-19 21:35:28,700] INFO [ThrottledChannelReaper-Request]: Stopped
(kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-07-19 21:35:28,700] INFO [ThrottledChannelReaper-Request]: Shutdown
completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-07-19 21:35:28,700] INFO [ThrottledChannelReaper-Produce]: Stopped
(kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-07-19 21:35:28,700] INFO [ThrottledChannelReaper-ControllerMutation]:
Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-07-19 21:35:28,701] INFO [ThrottledChannelReaper-ControllerMutation]:
Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-07-19 21:35:28,701] INFO [ThrottledChannelReaper-ControllerMutation]:
Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2022-07-19 21:35:28,701] INFO Metrics scheduler closed
(org.apache.kafka.common.metrics.Metrics)
[2022-07-19 21:35:28,702] INFO Closing reporter
org.apache.kafka.common.metrics.JmxReporter
(org.apache.kafka.common.metrics.Metrics)
[2022-07-19 21:35:28,702] INFO Metrics reporters closed
(org.apache.kafka.common.metrics.Metrics)
[2022-07-19 21:35:28,705] INFO Broker and topic stats closed
(kafka.server.BrokerTopicStats)
[2022-07-19 21:35:28,708] INFO App info kafka.server for 4 unregistered
(org.apache.kafka.common.utils.AppInfoParser)
[2022-07-19 21:35:28,708] INFO [KafkaServer id=4] shut down completed
(kafka.server.KafkaServer)
[2022-07-19 21:35:28,708] ERROR Exiting Kafka. (kafka.Kafka$)
[2022-07-19 21:35:28,709] INFO [KafkaServer id=4] shutting down
(kafka.server.KafkaServer)
