[
https://issues.apache.org/jira/browse/KAFKA-7925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784403#comment-16784403
]
Abhi commented on KAFKA-7925:
-----------------------------
[~rsivaram]
While look at the broker logs, I found multiple java.lang.NoClassDefFoundError
exceptions in one of the server logs. I am not sure if this is related to this
issue or totally separate issue. These exceptions started coming after the
server was running fine for a day and I don't see such exceptions in other
servers.
[2019-03-03 03:25:33,167] ERROR [ReplicaFetcher replicaId=4, leaderId=5,
fetcherId=3] Error due to (kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.KafkaException: Error processing data for partition
fps.pese.desim_se-0 offset 202742
at
kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7(AbstractFetcherThread.scala:338)
at scala.Option.foreach(Option.scala:274)
at
kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6(AbstractFetcherThread.scala:296)
at
kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted(AbstractFetcherThread.scala:295)
at
scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at
kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$5(AbstractFetcherThread.scala:295)
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
at
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:295)
at
kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:132)
at
kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:131)
at scala.Option.foreach(Option.scala:274)
at
kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:131)
at
kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:113)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)
Caused by: java.lang.NoClassDefFoundError: kafka/log/BatchMetadata
at kafka.log.Log.$anonfun$append$2(Log.scala:925)
at kafka.log.Log.maybeHandleIOException(Log.scala:2013)
at kafka.log.Log.append(Log.scala:827)
at kafka.log.Log.appendAsFollower(Log.scala:807)
at
kafka.cluster.Partition.$anonfun$doAppendRecordsToFollowerOrFutureReplica$1(Partition.scala:708)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:259)
at
kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica(Partition.scala:699)
at
kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica(Partition.scala:715)
at
kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:157)
at
kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7(AbstractFetcherThread.scala:307)
... 16 more
Caused by: java.lang.ClassNotFoundException: kafka.log.BatchMetadata
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 27 more
Other such exceptions included java.lang.NoClassDefFoundError:
kafka/common/LongRef, java.lang.NoClassDefFoundError: kafka/log/LogAppendInfo$,
java.lang.NoClassDefFoundError: kafka/api/LeaderAndIsr,
java.lang.NoClassDefFoundError: org/apache/zookeeper/proto/SetWatches etc.
Do you need server logs for all brokers? The size is in >300MB for the logs is
it okay to upload here on JIRA? I can upload client logs as they are small. I
will try to extract out the logs for the test duration and upload here soon.
> Constant 100% cpu usage by all kafka brokers
> --------------------------------------------
>
> Key: KAFKA-7925
> URL: https://issues.apache.org/jira/browse/KAFKA-7925
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 2.1.0, 2.1.1
> Environment: Java 11, Kafka v2.1.0, Kafka v2.1.1
> Reporter: Abhi
> Priority: Critical
> Attachments: threadump20190212.txt
>
>
> Hi,
> I am seeing constant 100% cpu usage on all brokers in our kafka cluster even
> without any clients connected to any broker.
> This is a bug that we have seen multiple times in our kafka setup that is not
> yet open to clients. It is becoming a blocker for our deployment now.
> I am seeing lot of connections to other brokers in CLOSE_WAIT state (see
> below). In thread usage, I am seeing these threads
> 'kafka-network-thread-6-ListenerName(SASL_PLAINTEXT)-SASL_PLAINTEXT-0,kafka-network-thread-6-ListenerName(SASL_PLAINTEXT)-SASL_PLAINTEXT-1,kafka-network-thread-6-ListenerName(SASL_PLAINTEXT)-SASL_PLAINTEXT-2'
> taking up more than 90% of the cpu time in a 60s interval.
> I have attached a thread dump of one of the brokers in the cluster.
> *Java version:*
> openjdk 11.0.2 2019-01-15
> OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)
> *Kafka verison:* v2.1.0
>
> *connections:*
> java 144319 kafkagod 88u IPv4 3063266 0t0 TCP *:35395 (LISTEN)
> java 144319 kafkagod 89u IPv4 3063267 0t0 TCP *:9144 (LISTEN)
> java 144319 kafkagod 104u IPv4 3064219 0t0 TCP
> mwkafka-prod-02.tbd:47292->mwkafka-zk-prod-05.tbd:2181 (ESTABLISHED)
> java 144319 kafkagod 2003u IPv4 3055115 0t0 TCP *:9092 (LISTEN)
> java 144319 kafkagod 2013u IPv4 7220110 0t0 TCP
> mwkafka-prod-02.tbd:60724->mwkafka-zk-prod-04.dr:2181 (ESTABLISHED)
> java 144319 kafkagod 2020u IPv4 30012904 0t0 TCP
> mwkafka-prod-02.tbd:38988->mwkafka-prod-02.nyc:9092 (ESTABLISHED)
> java 144319 kafkagod 2021u IPv4 30012961 0t0 TCP
> mwkafka-prod-02.tbd:58420->mwkafka-prod-01.nyc:9092 (ESTABLISHED)
> java 144319 kafkagod 2027u IPv4 30015723 0t0 TCP
> mwkafka-prod-02.tbd:58398->mwkafka-prod-01.nyc:9092 (ESTABLISHED)
> java 144319 kafkagod 2028u IPv4 30015630 0t0 TCP
> mwkafka-prod-02.tbd:36248->mwkafka-prod-02.dr:9092 (ESTABLISHED)
> java 144319 kafkagod 2030u IPv4 30015726 0t0 TCP
> mwkafka-prod-02.tbd:39012->mwkafka-prod-02.nyc:9092 (ESTABLISHED)
> java 144319 kafkagod 2031u IPv4 30013619 0t0 TCP
> mwkafka-prod-02.tbd:38986->mwkafka-prod-02.nyc:9092 (ESTABLISHED)
> java 144319 kafkagod 2032u IPv4 30015604 0t0 TCP
> mwkafka-prod-02.tbd:36246->mwkafka-prod-02.dr:9092 (ESTABLISHED)
> java 144319 kafkagod 2033u IPv4 30012981 0t0 TCP
> mwkafka-prod-02.tbd:36924->mwkafka-prod-01.dr:9092 (ESTABLISHED)
> java 144319 kafkagod 2034u IPv4 30012967 0t0 TCP
> mwkafka-prod-02.tbd:39036->mwkafka-prod-02.nyc:9092 (ESTABLISHED)
> java 144319 kafkagod 2035u IPv4 30012898 0t0 TCP
> mwkafka-prod-02.tbd:36866->mwkafka-prod-01.dr:9092 (FIN_WAIT2)
> java 144319 kafkagod 2036u IPv4 30004729 0t0 TCP
> mwkafka-prod-02.tbd:36882->mwkafka-prod-01.dr:9092 (ESTABLISHED)
> java 144319 kafkagod 2037u IPv4 30004914 0t0 TCP
> mwkafka-prod-02.tbd:58426->mwkafka-prod-01.nyc:9092 (ESTABLISHED)
> java 144319 kafkagod 2038u IPv4 30015651 0t0 TCP
> mwkafka-prod-02.tbd:36884->mwkafka-prod-01.dr:9092 (ESTABLISHED)
> java 144319 kafkagod 2039u IPv4 30012966 0t0 TCP
> mwkafka-prod-02.tbd:58422->mwkafka-prod-01.nyc:9092 (ESTABLISHED)
> java 144319 kafkagod 2040u IPv4 30005643 0t0 TCP
> mwkafka-prod-02.tbd:36252->mwkafka-prod-02.dr:9092 (ESTABLISHED)
> java 144319 kafkagod 2041u IPv4 30012944 0t0 TCP
> mwkafka-prod-02.tbd:36286->mwkafka-prod-02.dr:9092 (ESTABLISHED)
> java 144319 kafkagod 2042u IPv4 30012973 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.nyc:51924 (ESTABLISHED)
> java 144319 kafkagod 2043u sock 0,7 0t0 30012463 protocol: TCP
> java 144319 kafkagod 2044u IPv4 30012979 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.dr:39994 (ESTABLISHED)
> java 144319 kafkagod 2045u IPv4 30012899 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.nyc:34548 (ESTABLISHED)
> java 144319 kafkagod 2046u sock 0,7 0t0 30003437 protocol: TCP
> java 144319 kafkagod 2047u IPv4 30012980 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.dr:38120 (ESTABLISHED)
> java 144319 kafkagod 2048u sock 0,7 0t0 30012546 protocol: TCP
> java 144319 kafkagod 2049u IPv4 30005418 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.dr:39686 (CLOSE_WAIT)
> java 144319 kafkagod 2050u IPv4 30009977 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.nyc:34552 (ESTABLISHED)
> java 144319 kafkagod 2060u sock 0,7 0t0 30003439 protocol: TCP
> java 144319 kafkagod 2061u IPv4 30012906 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.nyc:51862 (ESTABLISHED)
> java 144319 kafkagod 2069u IPv4 30005642 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.nyc:34570 (ESTABLISHED)
> java 144319 kafkagod 2073u sock 0,7 0t0 30003440 protocol: TCP
> java 144319 kafkagod 2086u IPv4 30005644 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.nyc:51878 (ESTABLISHED)
> java 144319 kafkagod 2090u sock 0,7 0t0 30012553 protocol: TCP
> java 144319 kafkagod 2093u sock 0,7 0t0 30012502 protocol: TCP
> java 144319 kafkagod 2097u sock 0,7 0t0 30012531 protocol: TCP
> java 144319 kafkagod 2104u IPv4 30005670 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.nyc:34646 (ESTABLISHED)
> java 144319 kafkagod 2105u IPv4 30012933 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.dr:38106 (ESTABLISHED)
> java 144319 kafkagod 2106u IPv4 30012565 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.nyc:34366 (CLOSE_WAIT)
> java 144319 kafkagod 2114u IPv4 30012958 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.dr:39970 (ESTABLISHED)
> java 144319 kafkagod 2115u sock 0,7 0t0 30012569 protocol: TCP
> java 144319 kafkagod 2117u sock 0,7 0t0 30012571 protocol: TCP
> java 144319 kafkagod 2118u IPv4 30012959 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.dr:39972 (ESTABLISHED)
> java 144319 kafkagod 2120u IPv4 30012575 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.dr:37868 (CLOSE_WAIT)
> java 144319 kafkagod 2121u IPv4 30012960 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.dr:39974 (ESTABLISHED)
> java 144319 kafkagod 2122u IPv4 30012577 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.dr:39704 (CLOSE_WAIT)
> java 144319 kafkagod 2127u IPv4 29477410 0t0 TCP
> mwkafka-prod-02.tbd:58804->u-sonar-sonarsec.sdlb:8826 (ESTABLISHED)
> java 144319 kafkagod 2128u IPv4 30012579 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.dr:39708 (CLOSE_WAIT)
> java 144319 kafkagod 2129u IPv4 30012962 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.dr:38110 (ESTABLISHED)
> java 144319 kafkagod 2130u IPv4 30012582 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.dr:37872 (CLOSE_WAIT)
> java 144319 kafkagod 2132u IPv4 30012963 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.dr:38112 (ESTABLISHED)
> java 144319 kafkagod 2133u IPv4 30012602 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.nyc:51622 (CLOSE_WAIT)
> java 144319 kafkagod 2135u IPv4 30012964 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.nyc:51916 (ESTABLISHED)
> java 144319 kafkagod 2136u IPv4 30012605 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.nyc:51626 (CLOSE_WAIT)
> java 144319 kafkagod 2139u IPv4 30012965 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.nyc:51918 (ESTABLISHED)
> java 144319 kafkagod 2140u IPv4 30012607 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.dr:39754 (CLOSE_WAIT)
> java 144319 kafkagod 2141u IPv4 30010735 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.dr:37878 (CLOSE_WAIT)
> java 144319 kafkagod 2144u IPv4 30010741 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.nyc:34402 (CLOSE_WAIT)
> java 144319 kafkagod 2145u IPv4 30010742 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.nyc:51648 (CLOSE_WAIT)
> java 144319 kafkagod 2149u IPv4 30012623 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.nyc:51658 (CLOSE_WAIT)
> java 144319 kafkagod 2152u IPv4 30012625 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.nyc:34416 (CLOSE_WAIT)
> java 144319 kafkagod 2155u IPv4 30012635 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.dr:39778 (CLOSE_WAIT)
> java 144319 kafkagod 2157u IPv4 30012636 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.dr:39780 (CLOSE_WAIT)
> java 144319 kafkagod 2162u IPv4 29630161 0t0 TCP
> mwkafka-prod-02.tbd:45254->u-sonar-sonarpri.sdlb:8826 (ESTABLISHED)
> java 144319 kafkagod 2165u IPv4 30012639 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.dr:37916 (CLOSE_WAIT)
> java 144319 kafkagod 2168u IPv4 30012640 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.dr:37918 (CLOSE_WAIT)
> java 144319 kafkagod 2169u sock 0,7 0t0 30006888 protocol: TCP
> java 144319 kafkagod 2172u IPv4 30012656 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.nyc:51714 (CLOSE_WAIT)
> java 144319 kafkagod 2173u IPv4 30012659 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.nyc:51718 (CLOSE_WAIT)
> java 144319 kafkagod 2176u sock 0,7 0t0 30006891 protocol: TCP
> java 144319 kafkagod 2179u sock 0,7 0t0 30012426 protocol: TCP
> java 144319 kafkagod 2180u sock 0,7 0t0 30012427 protocol: TCP
> java 144319 kafkagod 2183u sock 0,7 0t0 30012429 protocol: TCP
> java 144319 kafkagod 2184u sock 0,7 0t0 30012432 protocol: TCP
> java 144319 kafkagod 2186u sock 0,7 0t0 30012437 protocol: TCP
> java 144319 kafkagod 2187u sock 0,7 0t0 30012459 protocol: TCP
> java 144319 kafkagod 2188u sock 0,7 0t0 30012696 protocol: TCP
> java 144319 kafkagod 2189u IPv4 30012718 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.nyc:34436 (CLOSE_WAIT)
> java 144319 kafkagod 2191u sock 0,7 0t0 30012720 protocol: TCP
> java 144319 kafkagod 2192u IPv4 30009662 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.nyc:34456 (CLOSE_WAIT)
> java 144319 kafkagod 2193u sock 0,7 0t0 30009663 protocol: TCP
> java 144319 kafkagod 2195u sock 0,7 0t0 30012723 protocol: TCP
> java 144319 kafkagod 2196u IPv4 30012727 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.dr:37958 (CLOSE_WAIT)
> java 144319 kafkagod 2197u sock 0,7 0t0 30012791 protocol: TCP
> java 144319 kafkagod 2198u IPv4 30012808 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.dr:39818 (CLOSE_WAIT)
> java 144319 kafkagod 2199u IPv4 30012818 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.dr:39838 (CLOSE_WAIT)
> java 144319 kafkagod 2200u IPv4 30012836 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.dr:37980 (CLOSE_WAIT)
> java 144319 kafkagod 2201u IPv4 30012839 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.dr:37986 (CLOSE_WAIT)
> java 144319 kafkagod 2202u IPv4 30012866 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.nyc:51812 (CLOSE_WAIT)
> java 144319 kafkagod 2204u IPv4 30012867 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.nyc:51814 (CLOSE_WAIT)
> java 144319 kafkagod 2205u IPv4 30012872 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.nyc:51846 (CLOSE_WAIT)
> java 144319 kafkagod 2206u IPv4 30012873 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-01.dr:39890 (CLOSE_WAIT)
> java 144319 kafkagod 2207u IPv4 30012894 0t0 TCP
> mwkafka-prod-02.tbd:9092->mwkafka-prod-02.dr:38000 (CLOSE_WAIT)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
