Evan Williams created KAFKA-9963:
------------------------------------
Summary: High CPU
Key: KAFKA-9963
URL: https://issues.apache.org/jira/browse/KAFKA-9963
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 2.4.1
Reporter: Evan Williams
When replacing a broker, with an empty data dir, and the same broker ID - we
are seeing very high CPU usage during replication, generally up to 100% for
some time, on a 4 VCPU (EC2 R5) host. This is a 6 host cluster, with approx
1000 topics and 3000 partitions.
There is of course traffic being served as well, as it catches up and becomes
leader of partitions, however due to the high replication CPU usage - client's
start to have connection issue.
CPU profiling during this 'replace' scenario, shows this:
{code:java}
54730000000 19.43% 5473 java.util.TreeMap$PrivateEntryIterator.nextEntry
49750000000 17.66% 4975
scala.collection.convert.Wrappers$JIteratorWrapper.hasNext
44170000000 15.68% 4417 java.util.TreeMap.successor
17730000000 6.29% 1773 java.util.TreeMap$ValueIterator.next
17000000000 6.03% 1700 java.util.TreeMap$PrivateEntryIterator.hasNext
6010000000 2.13% 601
scala.collection.convert.Wrappers$JIteratorWrapper.next
5160000000 1.83% 516 writev
--- 38850000000 ns (13.79%), 3885 samples
[ 0] java.util.TreeMap$PrivateEntryIterator.nextEntry
[ 1] java.util.TreeMap$ValueIterator.next
[ 2] scala.collection.convert.Wrappers$JIteratorWrapper.next
[ 3] scala.collection.Iterator.find
[ 4] scala.collection.Iterator.find$
[ 5] scala.collection.AbstractIterator.find
[ 6] scala.collection.IterableLike.find
[ 7] scala.collection.IterableLike.find$
[ 8] scala.collection.AbstractIterable.find
[ 9] kafka.log.ProducerStateManager.lastStableOffset
[10] kafka.log.Log.$anonfun$append$12
[11] kafka.log.Log.$anonfun$append$2
[12] kafka.log.Log.append
[13] kafka.log.Log.appendAsFollower
[14]
kafka.cluster.Partition.$anonfun$doAppendRecordsToFollowerOrFutureReplica$1
[15] kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica
[16] kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica
[17] kafka.server.ReplicaFetcherThread.processPartitionData
[18] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7
[19] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6
[20] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted
[21] kafka.server.AbstractFetcherThread$$Lambda$552.191789933.apply
[22] scala.collection.mutable.ResizableArray.foreach
[23] scala.collection.mutable.ResizableArray.foreach$
[24] scala.collection.mutable.ArrayBuffer.foreach
[25] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$5
[26] kafka.server.AbstractFetcherThread.processFetchRequest
[27] kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3
[28] kafka.server.AbstractFetcherThread.maybeFetch
[29] kafka.server.AbstractFetcherThread.doWork
[30] kafka.utils.ShutdownableThread.run--- 36320000000 ns (12.89%), 3632
samples
[ 0] scala.collection.convert.Wrappers$JIteratorWrapper.hasNext
[ 1] scala.collection.Iterator.find
[ 2] scala.collection.Iterator.find$
[ 3] scala.collection.AbstractIterator.find
[ 4] scala.collection.IterableLike.find
[ 5] scala.collection.IterableLike.find$
[ 6] scala.collection.AbstractIterable.find
[ 7] kafka.log.ProducerStateManager.lastStableOffset
[ 8] kafka.log.Log.$anonfun$append$12
[ 9] kafka.log.Log.$anonfun$append$2
[10] kafka.log.Log.append
[11] kafka.log.Log.appendAsFollower
[12]
kafka.cluster.Partition.$anonfun$doAppendRecordsToFollowerOrFutureReplica$1
[13] kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica
[14] kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica
[15] kafka.server.ReplicaFetcherThread.processPartitionData
[16] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7
[17] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6
[18] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted
[19] kafka.server.AbstractFetcherThread$$Lambda$552.191789933.apply
[20] scala.collection.mutable.ResizableArray.foreach
[21] scala.collection.mutable.ResizableArray.foreach$
[22] scala.collection.mutable.ArrayBuffer.foreach
[23] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$5
[24] kafka.server.AbstractFetcherThread.processFetchRequest
[25] kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3
[26] kafka.server.AbstractFetcherThread.maybeFetch
[27] kafka.server.AbstractFetcherThread.doWork
[28] kafka.utils.ShutdownableThread.run--- 32360000000 ns (11.49%), 3236
samples
[ 0] java.util.TreeMap.successor
[ 1] java.util.TreeMap$PrivateEntryIterator.nextEntry
[ 2] java.util.TreeMap$ValueIterator.next
[ 3] scala.collection.convert.Wrappers$JIteratorWrapper.next
[ 4] scala.collection.Iterator.find
[ 5] scala.collection.Iterator.find$
[ 6] scala.collection.AbstractIterator.find
[ 7] scala.collection.IterableLike.find
[ 8] scala.collection.IterableLike.find$
[ 9] scala.collection.AbstractIterable.find
[10] kafka.log.ProducerStateManager.lastStableOffset
[11] kafka.log.Log.$anonfun$append$12
[12] kafka.log.Log.$anonfun$append$2
[13] kafka.log.Log.append
[14] kafka.log.Log.appendAsFollower
[15]
kafka.cluster.Partition.$anonfun$doAppendRecordsToFollowerOrFutureReplica$1
[16] kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica
[17] kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica
[18] kafka.server.ReplicaFetcherThread.processPartitionData
[19] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7
[20] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6
[21] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted
[22] kafka.server.AbstractFetcherThread$$Lambda$552.191789933.apply
[23] scala.collection.mutable.ResizableArray.foreach
[24] scala.collection.mutable.ResizableArray.foreach$
[25] scala.collection.mutable.ArrayBuffer.foreach
[26] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$5
[27] kafka.server.AbstractFetcherThread.processFetchRequest
[28] kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3
[29] kafka.server.AbstractFetcherThread.maybeFetch
[30] kafka.server.AbstractFetcherThread.doWork
[31] kafka.utils.ShutdownableThread.run--- 15090000000 ns (5.36%), 1509
samples
[ 0] java.util.TreeMap$PrivateEntryIterator.nextEntry
[ 1] java.util.TreeMap$ValueIterator.next
[ 2] scala.collection.convert.Wrappers$JIteratorWrapper.next
[ 3] scala.collection.Iterator.find
[ 4] scala.collection.Iterator.find$
[ 5] scala.collection.AbstractIterator.find
[ 6] scala.collection.IterableLike.find
[ 7] scala.collection.IterableLike.find$
[ 8] scala.collection.AbstractIterable.find
[ 9] kafka.log.ProducerStateManager.lastStableOffset
[10] kafka.log.Log.$anonfun$append$12
[11] kafka.log.Log.$anonfun$append$2
[12] kafka.log.Log.append
[13] kafka.log.Log.appendAsFollower
[14]
kafka.cluster.Partition.$anonfun$doAppendRecordsToFollowerOrFutureReplica$1
[15] kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica
[16] kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica---
13200000000 ns (4.69%), 1320 samples
[ 0] scala.collection.convert.Wrappers$JIteratorWrapper.hasNext
[ 1] scala.collection.Iterator.find
[ 2] scala.collection.Iterator.find$
[ 3] scala.collection.AbstractIterator.find
[ 4] scala.collection.IterableLike.find
[ 5] scala.collection.IterableLike.find$
[ 6] scala.collection.AbstractIterable.find
[ 7] kafka.log.ProducerStateManager.lastStableOffset
[ 8] kafka.log.Log.$anonfun$append$12
[ 9] kafka.log.Log.$anonfun$append$2
[10] kafka.log.Log.append
[11] kafka.log.Log.appendAsFollower
[12]
kafka.cluster.Partition.$anonfun$doAppendRecordsToFollowerOrFutureReplica$1
[13] kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica
[14] kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica---
12390000000 ns (4.40%), 1239 samples
[ 0] java.util.TreeMap$ValueIterator.next
[ 1] scala.collection.convert.Wrappers$JIteratorWrapper.hasNext
[ 2] scala.collection.Iterator.find
[ 3] scala.collection.Iterator.find$
[ 4] scala.collection.AbstractIterator.find
[ 5] scala.collection.IterableLike.find
[ 6] scala.collection.IterableLike.find$
[ 7] scala.collection.AbstractIterable.find
[ 8] kafka.log.ProducerStateManager.lastStableOffset
[ 9] kafka.log.Log.$anonfun$append$12
[10] kafka.log.Log.$anonfun$append$2
[11] kafka.log.Log.append
[12] kafka.log.Log.appendAsFollower
[13]
kafka.cluster.Partition.$anonfun$doAppendRecordsToFollowerOrFutureReplica$1
[14] kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica
[15] kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica
[16] kafka.server.ReplicaFetcherThread.processPartitionData
[17] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7
[18] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6
[19] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted
[20] kafka.server.AbstractFetcherThread$$Lambda$552.191789933.apply
[21] scala.collection.mutable.ResizableArray.foreach
[22] scala.collection.mutable.ResizableArray.foreach$
[23] scala.collection.mutable.ArrayBuffer.foreach
[24] kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$5
[25] kafka.server.AbstractFetcherThread.processFetchRequest
[26] kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3
[27] kafka.server.AbstractFetcherThread.maybeFetch
[28] kafka.server.AbstractFetcherThread.doWork
[29] kafka.utils.ShutdownableThread.run--- 11810000000 ns (4.19%), 1181
samples
[ 0] java.util.TreeMap.successor
[ 1] java.util.TreeMap$PrivateEntryIterator.nextEntry
[ 2] java.util.TreeMap$ValueIterator.next
[ 3] scala.collection.convert.Wrappers$JIteratorWrapper.next
[ 4] scala.collection.Iterator.find
[ 5] scala.collection.Iterator.find$
[ 6] scala.collection.AbstractIterator.find
[ 7] scala.collection.IterableLike.find
[ 8] scala.collection.IterableLike.find$
[ 9] scala.collection.AbstractIterable.find
[10] kafka.log.ProducerStateManager.lastStableOffset
[11] kafka.log.Log.$anonfun$append$12
[12] kafka.log.Log.$anonfun$append$2
[13] kafka.log.Log.append
[14] kafka.log.Log.appendAsFollower
[15]
kafka.cluster.Partition.$anonfun$doAppendRecordsToFollowerOrFutureReplica$1
[16] kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica
[17] kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
