[jira] [Updated] (KAFKA-7464) Fail to shutdown ReplicaManager during broker cleaned shutdown
[ https://issues.apache.org/jira/browse/KAFKA-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-7464: Priority: Critical (was: Blocker) > Fail to shutdown ReplicaManager during broker cleaned shutdown > -- > > Key: KAFKA-7464 > URL: https://issues.apache.org/jira/browse/KAFKA-7464 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Zhanxiang (Patrick) Huang >Assignee: Zhanxiang (Patrick) Huang >Priority: Critical > Fix For: 2.0.1, 2.1.0 > > > In 2.0 deployment, we saw the following log when shutting down the > ReplicaManager in broker cleaned shutdown: > {noformat} > 2018/09/27 08:22:18.699 WARN [CoreUtils$] [Thread-1] [kafka-server] [] null > java.lang.IllegalArgumentException: null > at java.nio.Buffer.position(Buffer.java:244) ~[?:1.8.0_121] > at sun.nio.ch.IOUtil.write(IOUtil.java:68) ~[?:1.8.0_121] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) > ~[?:1.8.0_121] > at > org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:214) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:164) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:806) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:107) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.Selector.doClose(Selector.java:751) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:739) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:701) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:315) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.clients.NetworkClient.close(NetworkClient.java:595) > ~[kafka-clients-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherBlockingSend.close(ReplicaFetcherBlockingSend.scala:107) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherThread.initiateShutdown(ReplicaFetcherThread.scala:108) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:183) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreach(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) > ~[scala-library-2.11.12.jar:?] > at > kafka.server.AbstractFetcherManager.closeAllFetchers(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherManager.shutdown(ReplicaFetcherManager.scala:37) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.ReplicaManager.shutdown(ReplicaManager.scala:1471) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.KafkaServer$$anonfun$shutdown$12.apply$mcV$sp(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:86) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.KafkaServer.shutdown(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > {noformat} > After that, we noticed that some of the replica fetcher thread fail to > shutdown: > {noformat} > 2018/09/27 08:22:46.176 ERROR [LogDirFailureChannel] > [ReplicaFetcherThread-26-13085] [kafka-server] [] Error while rolling log > segment for video-social-gestures-30 in dir /export/content/kafka/i001_caches > java.nio.channels.ClosedChannelException: null > at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) > ~[?:1.8.0_121] > at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:300) > ~[?:1.8.0_121] > at > org.
[jira] [Updated] (KAFKA-7464) Fail to shutdown ReplicaManager during broker cleaned shutdown
[ https://issues.apache.org/jira/browse/KAFKA-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-7464: Fix Version/s: (was: 2.0.1) > Fail to shutdown ReplicaManager during broker cleaned shutdown > -- > > Key: KAFKA-7464 > URL: https://issues.apache.org/jira/browse/KAFKA-7464 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Zhanxiang (Patrick) Huang >Assignee: Zhanxiang (Patrick) Huang >Priority: Critical > Fix For: 2.1.0 > > > In 2.0 deployment, we saw the following log when shutting down the > ReplicaManager in broker cleaned shutdown: > {noformat} > 2018/09/27 08:22:18.699 WARN [CoreUtils$] [Thread-1] [kafka-server] [] null > java.lang.IllegalArgumentException: null > at java.nio.Buffer.position(Buffer.java:244) ~[?:1.8.0_121] > at sun.nio.ch.IOUtil.write(IOUtil.java:68) ~[?:1.8.0_121] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) > ~[?:1.8.0_121] > at > org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:214) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:164) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:806) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:107) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.Selector.doClose(Selector.java:751) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:739) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:701) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:315) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.clients.NetworkClient.close(NetworkClient.java:595) > ~[kafka-clients-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherBlockingSend.close(ReplicaFetcherBlockingSend.scala:107) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherThread.initiateShutdown(ReplicaFetcherThread.scala:108) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:183) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreach(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) > ~[scala-library-2.11.12.jar:?] > at > kafka.server.AbstractFetcherManager.closeAllFetchers(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherManager.shutdown(ReplicaFetcherManager.scala:37) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.ReplicaManager.shutdown(ReplicaManager.scala:1471) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.KafkaServer$$anonfun$shutdown$12.apply$mcV$sp(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:86) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.KafkaServer.shutdown(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > {noformat} > After that, we noticed that some of the replica fetcher thread fail to > shutdown: > {noformat} > 2018/09/27 08:22:46.176 ERROR [LogDirFailureChannel] > [ReplicaFetcherThread-26-13085] [kafka-server] [] Error while rolling log > segment for video-social-gestures-30 in dir /export/content/kafka/i001_caches > java.nio.channels.ClosedChannelException: null > at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) > ~[?:1.8.0_121] > at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:300) > ~[?:1.8.0_121] > at > org.apache.kaf
[jira] [Updated] (KAFKA-7464) Fail to shutdown ReplicaManager during broker cleaned shutdown
[ https://issues.apache.org/jira/browse/KAFKA-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-7464: Fix Version/s: 2.0.1 > Fail to shutdown ReplicaManager during broker cleaned shutdown > -- > > Key: KAFKA-7464 > URL: https://issues.apache.org/jira/browse/KAFKA-7464 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Zhanxiang (Patrick) Huang >Assignee: Zhanxiang (Patrick) Huang >Priority: Critical > Fix For: 2.0.1, 2.1.0 > > > In 2.0 deployment, we saw the following log when shutting down the > ReplicaManager in broker cleaned shutdown: > {noformat} > 2018/09/27 08:22:18.699 WARN [CoreUtils$] [Thread-1] [kafka-server] [] null > java.lang.IllegalArgumentException: null > at java.nio.Buffer.position(Buffer.java:244) ~[?:1.8.0_121] > at sun.nio.ch.IOUtil.write(IOUtil.java:68) ~[?:1.8.0_121] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) > ~[?:1.8.0_121] > at > org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:214) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:164) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:806) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:107) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.Selector.doClose(Selector.java:751) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:739) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:701) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:315) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.clients.NetworkClient.close(NetworkClient.java:595) > ~[kafka-clients-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherBlockingSend.close(ReplicaFetcherBlockingSend.scala:107) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherThread.initiateShutdown(ReplicaFetcherThread.scala:108) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:183) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreach(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) > ~[scala-library-2.11.12.jar:?] > at > kafka.server.AbstractFetcherManager.closeAllFetchers(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherManager.shutdown(ReplicaFetcherManager.scala:37) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.ReplicaManager.shutdown(ReplicaManager.scala:1471) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.KafkaServer$$anonfun$shutdown$12.apply$mcV$sp(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:86) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.KafkaServer.shutdown(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > {noformat} > After that, we noticed that some of the replica fetcher thread fail to > shutdown: > {noformat} > 2018/09/27 08:22:46.176 ERROR [LogDirFailureChannel] > [ReplicaFetcherThread-26-13085] [kafka-server] [] Error while rolling log > segment for video-social-gestures-30 in dir /export/content/kafka/i001_caches > java.nio.channels.ClosedChannelException: null > at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) > ~[?:1.8.0_121] > at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:300) > ~[?:1.8.0_121] > at > org.apache.kafka.c
[jira] [Updated] (KAFKA-7464) Fail to shutdown ReplicaManager during broker cleaned shutdown
[ https://issues.apache.org/jira/browse/KAFKA-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-7464: --- Fix Version/s: 2.1.0 > Fail to shutdown ReplicaManager during broker cleaned shutdown > -- > > Key: KAFKA-7464 > URL: https://issues.apache.org/jira/browse/KAFKA-7464 > Project: Kafka > Issue Type: Bug >Reporter: Zhanxiang (Patrick) Huang >Assignee: Zhanxiang (Patrick) Huang >Priority: Major > Fix For: 2.0.1, 2.1.0 > > > In 2.0 deployment, we saw the following log when shutting down the > ReplicaManager in broker cleaned shutdown: > {noformat} > 2018/09/27 08:22:18.699 WARN [CoreUtils$] [Thread-1] [kafka-server] [] null > java.lang.IllegalArgumentException: null > at java.nio.Buffer.position(Buffer.java:244) ~[?:1.8.0_121] > at sun.nio.ch.IOUtil.write(IOUtil.java:68) ~[?:1.8.0_121] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) > ~[?:1.8.0_121] > at > org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:214) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:164) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:806) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:107) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.Selector.doClose(Selector.java:751) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:739) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:701) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:315) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.clients.NetworkClient.close(NetworkClient.java:595) > ~[kafka-clients-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherBlockingSend.close(ReplicaFetcherBlockingSend.scala:107) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherThread.initiateShutdown(ReplicaFetcherThread.scala:108) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:183) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreach(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) > ~[scala-library-2.11.12.jar:?] > at > kafka.server.AbstractFetcherManager.closeAllFetchers(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherManager.shutdown(ReplicaFetcherManager.scala:37) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.ReplicaManager.shutdown(ReplicaManager.scala:1471) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.KafkaServer$$anonfun$shutdown$12.apply$mcV$sp(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:86) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.KafkaServer.shutdown(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > {noformat} > After that, we noticed that some of the replica fetcher thread fail to > shutdown: > {noformat} > 2018/09/27 08:22:46.176 ERROR [LogDirFailureChannel] > [ReplicaFetcherThread-26-13085] [kafka-server] [] Error while rolling log > segment for video-social-gestures-30 in dir /export/content/kafka/i001_caches > java.nio.channels.ClosedChannelException: null > at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) > ~[?:1.8.0_121] > at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:300) > ~[?:1.8.0_121] > at > org.apache.kafka.common.record.FileRecords.t
[jira] [Updated] (KAFKA-7464) Fail to shutdown ReplicaManager during broker cleaned shutdown
[ https://issues.apache.org/jira/browse/KAFKA-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-7464: --- Priority: Critical (was: Major) > Fail to shutdown ReplicaManager during broker cleaned shutdown > -- > > Key: KAFKA-7464 > URL: https://issues.apache.org/jira/browse/KAFKA-7464 > Project: Kafka > Issue Type: Bug >Reporter: Zhanxiang (Patrick) Huang >Assignee: Zhanxiang (Patrick) Huang >Priority: Critical > Fix For: 2.0.1, 2.1.0 > > > In 2.0 deployment, we saw the following log when shutting down the > ReplicaManager in broker cleaned shutdown: > {noformat} > 2018/09/27 08:22:18.699 WARN [CoreUtils$] [Thread-1] [kafka-server] [] null > java.lang.IllegalArgumentException: null > at java.nio.Buffer.position(Buffer.java:244) ~[?:1.8.0_121] > at sun.nio.ch.IOUtil.write(IOUtil.java:68) ~[?:1.8.0_121] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) > ~[?:1.8.0_121] > at > org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:214) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:164) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:806) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:107) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.Selector.doClose(Selector.java:751) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:739) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:701) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:315) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.clients.NetworkClient.close(NetworkClient.java:595) > ~[kafka-clients-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherBlockingSend.close(ReplicaFetcherBlockingSend.scala:107) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherThread.initiateShutdown(ReplicaFetcherThread.scala:108) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:183) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreach(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) > ~[scala-library-2.11.12.jar:?] > at > kafka.server.AbstractFetcherManager.closeAllFetchers(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherManager.shutdown(ReplicaFetcherManager.scala:37) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.ReplicaManager.shutdown(ReplicaManager.scala:1471) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.KafkaServer$$anonfun$shutdown$12.apply$mcV$sp(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:86) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.KafkaServer.shutdown(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > {noformat} > After that, we noticed that some of the replica fetcher thread fail to > shutdown: > {noformat} > 2018/09/27 08:22:46.176 ERROR [LogDirFailureChannel] > [ReplicaFetcherThread-26-13085] [kafka-server] [] Error while rolling log > segment for video-social-gestures-30 in dir /export/content/kafka/i001_caches > java.nio.channels.ClosedChannelException: null > at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) > ~[?:1.8.0_121] > at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:300) > ~[?:1.8.0_121] > at > org.apache.kafka.common.recor
[jira] [Updated] (KAFKA-7464) Fail to shutdown ReplicaManager during broker cleaned shutdown
[ https://issues.apache.org/jira/browse/KAFKA-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-7464: --- Fix Version/s: 2.0.1 > Fail to shutdown ReplicaManager during broker cleaned shutdown > -- > > Key: KAFKA-7464 > URL: https://issues.apache.org/jira/browse/KAFKA-7464 > Project: Kafka > Issue Type: Bug >Reporter: Zhanxiang (Patrick) Huang >Assignee: Zhanxiang (Patrick) Huang >Priority: Critical > Fix For: 2.0.1, 2.1.0 > > > In 2.0 deployment, we saw the following log when shutting down the > ReplicaManager in broker cleaned shutdown: > {noformat} > 2018/09/27 08:22:18.699 WARN [CoreUtils$] [Thread-1] [kafka-server] [] null > java.lang.IllegalArgumentException: null > at java.nio.Buffer.position(Buffer.java:244) ~[?:1.8.0_121] > at sun.nio.ch.IOUtil.write(IOUtil.java:68) ~[?:1.8.0_121] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) > ~[?:1.8.0_121] > at > org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:214) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:164) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:806) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:107) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.Selector.doClose(Selector.java:751) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:739) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:701) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:315) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.clients.NetworkClient.close(NetworkClient.java:595) > ~[kafka-clients-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherBlockingSend.close(ReplicaFetcherBlockingSend.scala:107) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherThread.initiateShutdown(ReplicaFetcherThread.scala:108) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:183) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreach(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) > ~[scala-library-2.11.12.jar:?] > at > kafka.server.AbstractFetcherManager.closeAllFetchers(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherManager.shutdown(ReplicaFetcherManager.scala:37) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.ReplicaManager.shutdown(ReplicaManager.scala:1471) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.KafkaServer$$anonfun$shutdown$12.apply$mcV$sp(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:86) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.KafkaServer.shutdown(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > {noformat} > After that, we noticed that some of the replica fetcher thread fail to > shutdown: > {noformat} > 2018/09/27 08:22:46.176 ERROR [LogDirFailureChannel] > [ReplicaFetcherThread-26-13085] [kafka-server] [] Error while rolling log > segment for video-social-gestures-30 in dir /export/content/kafka/i001_caches > java.nio.channels.ClosedChannelException: null > at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) > ~[?:1.8.0_121] > at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:300) > ~[?:1.8.0_121] > at > org.apache.kafka.common.record.FileRecord
[jira] [Updated] (KAFKA-7464) Fail to shutdown ReplicaManager during broker cleaned shutdown
[ https://issues.apache.org/jira/browse/KAFKA-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-7464: --- Affects Version/s: 2.0.0 > Fail to shutdown ReplicaManager during broker cleaned shutdown > -- > > Key: KAFKA-7464 > URL: https://issues.apache.org/jira/browse/KAFKA-7464 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Zhanxiang (Patrick) Huang >Assignee: Zhanxiang (Patrick) Huang >Priority: Critical > Fix For: 2.0.1, 2.1.0 > > > In 2.0 deployment, we saw the following log when shutting down the > ReplicaManager in broker cleaned shutdown: > {noformat} > 2018/09/27 08:22:18.699 WARN [CoreUtils$] [Thread-1] [kafka-server] [] null > java.lang.IllegalArgumentException: null > at java.nio.Buffer.position(Buffer.java:244) ~[?:1.8.0_121] > at sun.nio.ch.IOUtil.write(IOUtil.java:68) ~[?:1.8.0_121] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) > ~[?:1.8.0_121] > at > org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:214) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:164) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:806) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:107) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.Selector.doClose(Selector.java:751) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:739) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:701) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:315) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.clients.NetworkClient.close(NetworkClient.java:595) > ~[kafka-clients-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherBlockingSend.close(ReplicaFetcherBlockingSend.scala:107) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherThread.initiateShutdown(ReplicaFetcherThread.scala:108) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:183) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreach(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) > ~[scala-library-2.11.12.jar:?] > at > kafka.server.AbstractFetcherManager.closeAllFetchers(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherManager.shutdown(ReplicaFetcherManager.scala:37) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.ReplicaManager.shutdown(ReplicaManager.scala:1471) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.KafkaServer$$anonfun$shutdown$12.apply$mcV$sp(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:86) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.KafkaServer.shutdown(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > {noformat} > After that, we noticed that some of the replica fetcher thread fail to > shutdown: > {noformat} > 2018/09/27 08:22:46.176 ERROR [LogDirFailureChannel] > [ReplicaFetcherThread-26-13085] [kafka-server] [] Error while rolling log > segment for video-social-gestures-30 in dir /export/content/kafka/i001_caches > java.nio.channels.ClosedChannelException: null > at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) > ~[?:1.8.0_121] > at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:300) > ~[?:1.8.0_121] > at > org.apac
[jira] [Updated] (KAFKA-7464) Fail to shutdown ReplicaManager during broker cleaned shutdown
[ https://issues.apache.org/jira/browse/KAFKA-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-7464: Fix Version/s: (was: 2.1.0) 2.2.0 > Fail to shutdown ReplicaManager during broker cleaned shutdown > -- > > Key: KAFKA-7464 > URL: https://issues.apache.org/jira/browse/KAFKA-7464 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Zhanxiang (Patrick) Huang >Assignee: Zhanxiang (Patrick) Huang >Priority: Critical > Fix For: 2.0.1, 2.2.0 > > > In 2.0 deployment, we saw the following log when shutting down the > ReplicaManager in broker cleaned shutdown: > {noformat} > 2018/09/27 08:22:18.699 WARN [CoreUtils$] [Thread-1] [kafka-server] [] null > java.lang.IllegalArgumentException: null > at java.nio.Buffer.position(Buffer.java:244) ~[?:1.8.0_121] > at sun.nio.ch.IOUtil.write(IOUtil.java:68) ~[?:1.8.0_121] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) > ~[?:1.8.0_121] > at > org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:214) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:164) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:806) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:107) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.Selector.doClose(Selector.java:751) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:739) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:701) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:315) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.clients.NetworkClient.close(NetworkClient.java:595) > ~[kafka-clients-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherBlockingSend.close(ReplicaFetcherBlockingSend.scala:107) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherThread.initiateShutdown(ReplicaFetcherThread.scala:108) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:183) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreach(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) > ~[scala-library-2.11.12.jar:?] > at > kafka.server.AbstractFetcherManager.closeAllFetchers(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherManager.shutdown(ReplicaFetcherManager.scala:37) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.ReplicaManager.shutdown(ReplicaManager.scala:1471) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.KafkaServer$$anonfun$shutdown$12.apply$mcV$sp(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:86) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.KafkaServer.shutdown(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > {noformat} > After that, we noticed that some of the replica fetcher thread fail to > shutdown: > {noformat} > 2018/09/27 08:22:46.176 ERROR [LogDirFailureChannel] > [ReplicaFetcherThread-26-13085] [kafka-server] [] Error while rolling log > segment for video-social-gestures-30 in dir /export/content/kafka/i001_caches > java.nio.channels.ClosedChannelException: null > at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) > ~[?:1.8.0_121] > at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:300) > ~[?:1.8.0_121
[jira] [Updated] (KAFKA-7464) Fail to shutdown ReplicaManager during broker cleaned shutdown
[ https://issues.apache.org/jira/browse/KAFKA-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-7464: Fix Version/s: (was: 2.2.0) 2.1.0 > Fail to shutdown ReplicaManager during broker cleaned shutdown > -- > > Key: KAFKA-7464 > URL: https://issues.apache.org/jira/browse/KAFKA-7464 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Zhanxiang (Patrick) Huang >Assignee: Zhanxiang (Patrick) Huang >Priority: Critical > Fix For: 2.0.1, 2.1.0 > > > In 2.0 deployment, we saw the following log when shutting down the > ReplicaManager in broker cleaned shutdown: > {noformat} > 2018/09/27 08:22:18.699 WARN [CoreUtils$] [Thread-1] [kafka-server] [] null > java.lang.IllegalArgumentException: null > at java.nio.Buffer.position(Buffer.java:244) ~[?:1.8.0_121] > at sun.nio.ch.IOUtil.write(IOUtil.java:68) ~[?:1.8.0_121] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) > ~[?:1.8.0_121] > at > org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:214) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:164) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:806) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:107) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.Selector.doClose(Selector.java:751) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:739) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:701) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:315) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.clients.NetworkClient.close(NetworkClient.java:595) > ~[kafka-clients-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherBlockingSend.close(ReplicaFetcherBlockingSend.scala:107) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherThread.initiateShutdown(ReplicaFetcherThread.scala:108) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:183) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreach(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) > ~[scala-library-2.11.12.jar:?] > at > kafka.server.AbstractFetcherManager.closeAllFetchers(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherManager.shutdown(ReplicaFetcherManager.scala:37) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.ReplicaManager.shutdown(ReplicaManager.scala:1471) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.KafkaServer$$anonfun$shutdown$12.apply$mcV$sp(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:86) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.KafkaServer.shutdown(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > {noformat} > After that, we noticed that some of the replica fetcher thread fail to > shutdown: > {noformat} > 2018/09/27 08:22:46.176 ERROR [LogDirFailureChannel] > [ReplicaFetcherThread-26-13085] [kafka-server] [] Error while rolling log > segment for video-social-gestures-30 in dir /export/content/kafka/i001_caches > java.nio.channels.ClosedChannelException: null > at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) > ~[?:1.8.0_121] > at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:300) > ~[?:1.8.0_121
[jira] [Updated] (KAFKA-7464) Fail to shutdown ReplicaManager during broker cleaned shutdown
[ https://issues.apache.org/jira/browse/KAFKA-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-7464: Priority: Blocker (was: Critical) > Fail to shutdown ReplicaManager during broker cleaned shutdown > -- > > Key: KAFKA-7464 > URL: https://issues.apache.org/jira/browse/KAFKA-7464 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Zhanxiang (Patrick) Huang >Assignee: Zhanxiang (Patrick) Huang >Priority: Blocker > Fix For: 2.0.1, 2.1.0 > > > In 2.0 deployment, we saw the following log when shutting down the > ReplicaManager in broker cleaned shutdown: > {noformat} > 2018/09/27 08:22:18.699 WARN [CoreUtils$] [Thread-1] [kafka-server] [] null > java.lang.IllegalArgumentException: null > at java.nio.Buffer.position(Buffer.java:244) ~[?:1.8.0_121] > at sun.nio.ch.IOUtil.write(IOUtil.java:68) ~[?:1.8.0_121] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) > ~[?:1.8.0_121] > at > org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:214) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:164) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:806) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:107) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.common.network.Selector.doClose(Selector.java:751) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:739) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:701) > ~[kafka-clients-2.0.0.22.jar:?] > at org.apache.kafka.common.network.Selector.close(Selector.java:315) > ~[kafka-clients-2.0.0.22.jar:?] > at > org.apache.kafka.clients.NetworkClient.close(NetworkClient.java:595) > ~[kafka-clients-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherBlockingSend.close(ReplicaFetcherBlockingSend.scala:107) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherThread.initiateShutdown(ReplicaFetcherThread.scala:108) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:183) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) > ~[scala-library-2.11.12.jar:?] > at scala.collection.mutable.HashMap.foreach(HashMap.scala:130) > ~[scala-library-2.11.12.jar:?] > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) > ~[scala-library-2.11.12.jar:?] > at > kafka.server.AbstractFetcherManager.closeAllFetchers(AbstractFetcherManager.scala:182) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.ReplicaFetcherManager.shutdown(ReplicaFetcherManager.scala:37) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.ReplicaManager.shutdown(ReplicaManager.scala:1471) > ~[kafka_2.11-2.0.0.22.jar:?] > at > kafka.server.KafkaServer$$anonfun$shutdown$12.apply$mcV$sp(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:86) > ~[kafka_2.11-2.0.0.22.jar:?] > at kafka.server.KafkaServer.shutdown(KafkaServer.scala:616) > ~[kafka_2.11-2.0.0.22.jar:?] > {noformat} > After that, we noticed that some of the replica fetcher thread fail to > shutdown: > {noformat} > 2018/09/27 08:22:46.176 ERROR [LogDirFailureChannel] > [ReplicaFetcherThread-26-13085] [kafka-server] [] Error while rolling log > segment for video-social-gestures-30 in dir /export/content/kafka/i001_caches > java.nio.channels.ClosedChannelException: null > at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) > ~[?:1.8.0_121] > at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:300) > ~[?:1.8.0_121] > at > org.a