[ https://issues.apache.org/jira/browse/KAFKA-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105606#comment-17105606 ]
Ishwor Gurung commented on KAFKA-9963: -------------------------------------- [~blodsbror] Just out of curiosity, what was the issue? > High CPU during replication of replaced (empty) broker > ------------------------------------------------------ > > Key: KAFKA-9963 > URL: https://issues.apache.org/jira/browse/KAFKA-9963 > Project: Kafka > Issue Type: Bug > Components: core > Affects Versions: 2.4.1 > Reporter: Evan Williams > Priority: Major > > When replacing a broker, with an empty data dir, and the same broker ID - we > are seeing very high CPU usage during replication, generally up to 100% for > some time, on a 4 VCPU (EC2 R5) host. This is a 6 host cluster, with approx > 1000 topics and 3000 partitions. > > There is of course traffic being served as well, as it catches up and becomes > leader of partitions, however due to the high replication CPU usage - > client's start to have connection issue. Normal traffic on this host is is > around 40% CPU, when it's completely replicated. I'm not sure if this is > 'normal', and we just need to throw more resources at the hosts. Or if > replication should not really use this much CPU. > replica.fetchers=2 (not high). And java version is: OpenJDK Runtime > Environment Corretto-8.252.09.1 (build 1.8.0_252-b09) (AWS's own build). > CPU profiling (2 minutes ) during this 'replace' scenario, shows this: > > {code:java} > 54730000000 19.43% 5473 > java.util.TreeMap$PrivateEntryIterator.nextEntry > 49750000000 17.66% 4975 > scala.collection.convert.Wrappers$JIteratorWrapper.hasNext > 44170000000 15.68% 4417 java.util.TreeMap.successor > 17730000000 6.29% 1773 java.util.TreeMap$ValueIterator.next > 17000000000 6.03% 1700 java.util.TreeMap$PrivateEntryIterator.hasNext > 6010000000 2.13% 601 > scala.collection.convert.Wrappers$JIteratorWrapper.next > 5160000000 1.83% 516 writev > --- 38850000000 ns (13.79%), 3885 samples > [ 0] java.util.TreeMap$PrivateEntryIterator.nextEntry > [ 1] java.util.TreeMap$ValueIterator.next > [ 2] scala.collection.convert.Wrappers$JIteratorWrapper.next > [ 3] scala.collection.Iterator.find > [ 4] scala.collection.Iterator.find$ > [ 5] scala.collection.AbstractIterator.find > [ 6] scala.collection.IterableLike.find > [ 7] scala.collection.IterableLike.find$ > [ 8] scala.collection.AbstractIterable.find > [ 9] kafka.log.ProducerStateManager.lastStableOffset > [10] kafka.log.Log.$anonfun$append$12 > [11] kafka.log.Log.$anonfun$append$2 > [12] kafka.log.Log.append > [13] kafka.log.Log.appendAsFollower > [14] > kafka.cluster.Partition.$anonfun$doAppendRecordsToFollowerOrFutureReplica$1 > [15] kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica > [16] kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica > [17] kafka.server.ReplicaFetcherThread.processPartitionData > [18] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7 > [19] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6 > [20] > kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted > [21] kafka.server.AbstractFetcherThread$$Lambda$552.191789933.apply > [22] scala.collection.mutable.ResizableArray.foreach > [23] scala.collection.mutable.ResizableArray.foreach$ > [24] scala.collection.mutable.ArrayBuffer.foreach > [25] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$5 > [26] kafka.server.AbstractFetcherThread.processFetchRequest > [27] kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3 > [28] kafka.server.AbstractFetcherThread.maybeFetch > [29] kafka.server.AbstractFetcherThread.doWork > [30] kafka.utils.ShutdownableThread.run > > {code} > {code:java} > --- 36320000000 ns (12.89%), 3632 samples > [ 0] scala.collection.convert.Wrappers$JIteratorWrapper.hasNext > [ 1] scala.collection.Iterator.find > [ 2] scala.collection.Iterator.find$ > [ 3] scala.collection.AbstractIterator.find > [ 4] scala.collection.IterableLike.find > [ 5] scala.collection.IterableLike.find$ > [ 6] scala.collection.AbstractIterable.find > [ 7] kafka.log.ProducerStateManager.lastStableOffset > [ 8] kafka.log.Log.$anonfun$append$12 > [ 9] kafka.log.Log.$anonfun$append$2 > [10] kafka.log.Log.append > [11] kafka.log.Log.appendAsFollower > [12] > kafka.cluster.Partition.$anonfun$doAppendRecordsToFollowerOrFutureReplica$1 > [13] kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica > [14] kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica > [15] kafka.server.ReplicaFetcherThread.processPartitionData > [16] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7 > [17] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6 > [18] > kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted > [19] kafka.server.AbstractFetcherThread$$Lambda$552.191789933.apply > [20] scala.collection.mutable.ResizableArray.foreach > [21] scala.collection.mutable.ResizableArray.foreach$ > [22] scala.collection.mutable.ArrayBuffer.foreach > [23] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$5 > [24] kafka.server.AbstractFetcherThread.processFetchRequest > [25] kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3 > [26] kafka.server.AbstractFetcherThread.maybeFetch > [27] kafka.server.AbstractFetcherThread.doWork > [28] kafka.utils.ShutdownableThread.run{code} > > {code:java} > --- 32360000000 ns (11.49%), 3236 samples > [ 0] java.util.TreeMap.successor > [ 1] java.util.TreeMap$PrivateEntryIterator.nextEntry > [ 2] java.util.TreeMap$ValueIterator.next > [ 3] scala.collection.convert.Wrappers$JIteratorWrapper.next > [ 4] scala.collection.Iterator.find > [ 5] scala.collection.Iterator.find$ > [ 6] scala.collection.AbstractIterator.find > [ 7] scala.collection.IterableLike.find > [ 8] scala.collection.IterableLike.find$ > [ 9] scala.collection.AbstractIterable.find > [10] kafka.log.ProducerStateManager.lastStableOffset > [11] kafka.log.Log.$anonfun$append$12 > [12] kafka.log.Log.$anonfun$append$2 > [13] kafka.log.Log.append > [14] kafka.log.Log.appendAsFollower > [15] > kafka.cluster.Partition.$anonfun$doAppendRecordsToFollowerOrFutureReplica$1 > [16] kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica > [17] kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica > [18] kafka.server.ReplicaFetcherThread.processPartitionData > [19] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7 > [20] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6 > [21] > kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted > [22] kafka.server.AbstractFetcherThread$$Lambda$552.191789933.apply > [23] scala.collection.mutable.ResizableArray.foreach > [24] scala.collection.mutable.ResizableArray.foreach$ > [25] scala.collection.mutable.ArrayBuffer.foreach > [26] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$5 > [27] kafka.server.AbstractFetcherThread.processFetchRequest > [28] kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3 > [29] kafka.server.AbstractFetcherThread.maybeFetch > [30] kafka.server.AbstractFetcherThread.doWork > [31] kafka.utils.ShutdownableThread.run--- 15090000000 ns (5.36%), 1509 > samples > [ 0] java.util.TreeMap$PrivateEntryIterator.nextEntry > [ 1] java.util.TreeMap$ValueIterator.next > [ 2] scala.collection.convert.Wrappers$JIteratorWrapper.next > [ 3] scala.collection.Iterator.find > [ 4] scala.collection.Iterator.find$ > [ 5] scala.collection.AbstractIterator.find > [ 6] scala.collection.IterableLike.find > [ 7] scala.collection.IterableLike.find$ > [ 8] scala.collection.AbstractIterable.find > [ 9] kafka.log.ProducerStateManager.lastStableOffset > [10] kafka.log.Log.$anonfun$append$12 > [11] kafka.log.Log.$anonfun$append$2 > [12] kafka.log.Log.append > [13] kafka.log.Log.appendAsFollower > [14] > kafka.cluster.Partition.$anonfun$doAppendRecordsToFollowerOrFutureReplica$1 > [15] kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica > [16] kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica--- > 13200000000 ns (4.69%), 1320 samples > [ 0] scala.collection.convert.Wrappers$JIteratorWrapper.hasNext > [ 1] scala.collection.Iterator.find > [ 2] scala.collection.Iterator.find$ > [ 3] scala.collection.AbstractIterator.find > [ 4] scala.collection.IterableLike.find > [ 5] scala.collection.IterableLike.find$ > [ 6] scala.collection.AbstractIterable.find > [ 7] kafka.log.ProducerStateManager.lastStableOffset > [ 8] kafka.log.Log.$anonfun$append$12 > [ 9] kafka.log.Log.$anonfun$append$2 > [10] kafka.log.Log.append > [11] kafka.log.Log.appendAsFollower > [12] > kafka.cluster.Partition.$anonfun$doAppendRecordsToFollowerOrFutureReplica$1 > [13] kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica > [14] kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica--- > 12390000000 ns (4.40%), 1239 samples > [ 0] java.util.TreeMap$ValueIterator.next > [ 1] scala.collection.convert.Wrappers$JIteratorWrapper.hasNext > [ 2] scala.collection.Iterator.find > [ 3] scala.collection.Iterator.find$ > [ 4] scala.collection.AbstractIterator.find > [ 5] scala.collection.IterableLike.find > [ 6] scala.collection.IterableLike.find$ > [ 7] scala.collection.AbstractIterable.find > [ 8] kafka.log.ProducerStateManager.lastStableOffset > [ 9] kafka.log.Log.$anonfun$append$12 > [10] kafka.log.Log.$anonfun$append$2 > [11] kafka.log.Log.append > [12] kafka.log.Log.appendAsFollower > [13] > kafka.cluster.Partition.$anonfun$doAppendRecordsToFollowerOrFutureReplica$1 > [14] kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica > [15] kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica > [16] kafka.server.ReplicaFetcherThread.processPartitionData > [17] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7 > [18] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6 > [19] > kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted > [20] kafka.server.AbstractFetcherThread$$Lambda$552.191789933.apply > [21] scala.collection.mutable.ResizableArray.foreach > [22] scala.collection.mutable.ResizableArray.foreach$ > [23] scala.collection.mutable.ArrayBuffer.foreach > [24] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$5 > [25] kafka.server.AbstractFetcherThread.processFetchRequest > [26] kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3 > [27] kafka.server.AbstractFetcherThread.maybeFetch > [28] kafka.server.AbstractFetcherThread.doWork > [29] kafka.utils.ShutdownableThread.run{code} > {code:java} > --- 11810000000 ns (4.19%), 1181 samples > [ 0] java.util.TreeMap.successor > [ 1] java.util.TreeMap$PrivateEntryIterator.nextEntry > [ 2] java.util.TreeMap$ValueIterator.next > [ 3] scala.collection.convert.Wrappers$JIteratorWrapper.next > [ 4] scala.collection.Iterator.find > [ 5] scala.collection.Iterator.find$ > [ 6] scala.collection.AbstractIterator.find > [ 7] scala.collection.IterableLike.find > [ 8] scala.collection.IterableLike.find$ > [ 9] scala.collection.AbstractIterable.find > [10] kafka.log.ProducerStateManager.lastStableOffset > [11] kafka.log.Log.$anonfun$append$12 > [12] kafka.log.Log.$anonfun$append$2 > [13] kafka.log.Log.append > [14] kafka.log.Log.appendAsFollower > [15] > kafka.cluster.Partition.$anonfun$doAppendRecordsToFollowerOrFutureReplica$1 > [16] kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica > [17] kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)