[ https://issues.apache.org/jira/browse/KAFKA-12946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17363805#comment-17363805 ]
Ron Dagostino commented on KAFKA-12946: --------------------------------------- The only one I am familiar with and would recommend is the upgrade. > __consumer_offsets topic with very big partitions > ------------------------------------------------- > > Key: KAFKA-12946 > URL: https://issues.apache.org/jira/browse/KAFKA-12946 > Project: Kafka > Issue Type: Bug > Components: log cleaner > Affects Versions: 2.0.0 > Reporter: Emi > Priority: Critical > > I am using Kafka 2.0.0 with java 8u191 > There is a partitions of the __consumer_offsets topic that is 600 GB with > 6000 segments older than 4 months. Other partitions of that topic are small: > 20-30 MB. > There are 60 consumer groups, 90 topics and 100 partitions per topic. > There aren't errors in the logs. From the log of the logcleaner, I can see > that partition is never touched from the logcleaner thread for the > compaction, but it only add new segments. > How is this possible? > There was another partition with the same problem, but after some months it > has been compacted. Now there is only one partition with this problem, but > this is bigger and keep growing > I have used the kafka-dump-log tool to check these old segments and I can see > many duplicates. So I would assume that is not compacted. > My settings: > {{offsets.commit.required.acks = -1}} > {{[offsets.commit.timeout.ms|http://offsets.commit.timeout.ms/]}} = 5000 > {{offsets.load.buffer.size = 5242880}} > > {{[offsets.retention.check.interval.ms|http://offsets.retention.check.interval.ms/]}} > = 600000 > {{offsets.retention.minutes = 10080}} > {{offsets.topic.compression.codec = 0}} > {{offsets.topic.num.partitions = 50}} > {{offsets.topic.replication.factor = 3}} > {{offsets.topic.segment.bytes = 104857600}} -- This message was sent by Atlassian Jira (v8.3.4#803005)