[ https://issues.apache.org/jira/browse/KAFKA-8547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lerh Chuan Low updated KAFKA-8547: ---------------------------------- Description: It seems like log cleaner doesn't clean {{__consumer_offsets}} very well on the default policy of compact on that topic. We observed a few out of memory errors with our Kafka servers and our theory was due to 2 overly large partitions in {{__consumer_offsets}}. On further digging, it looks like these 2 large partitions have segments dating up to 3 months ago. Also, these old files collectively consumed most of the data from those partitions (About 10G from the partition's 12G). When we tried dumping those old segments, we see: {code:java} 1:40 $ ./kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000161728257775.log --offsets-decoder --print-data-log --deep-iteration Dumping 00000000161728257775.log Starting offset: 161728257775 offset: 161728257904 position: 61 CreateTime: 1553457816168 isvalid: true keysize: 4 valuesize: 6 magic: 2 compresscodec: NONE producerId: 367038 producerEpoch: 3 sequence: -1 isTransactional: true headerKeys: [] endTxnMarker: COMMIT coordinatorEpoch: 746 offset: 161728258098 position: 200 CreateTime: 1553457816230 isvalid: true keysize: 4 valuesize: 6 magic: 2 compresscodec: NONE producerId: 366036 producerEpoch: 3 sequence: -1 isTransactional: true headerKeys: [] endTxnMarker: COMMIT coordinatorEpoch: 761 ...{code} It looks like all those old segments all contain transactional information, and the 2 partitions are 1 for the control message COMMIT, the other for the control message ABORT. (As a side note, we did take a while to figure out that for a segment with the control bit set, the key really is {{endTxnMarker}} and the value is {{coordinatorEpoch}}...otherwise in a non-control batch dump it would have value and payload. We were wondering if seeing what those 2 partitions contained in their keys may give us any clues). Our current workaround is based on this post: https://issues.apache.org/jira/browse/KAFKA-3917?focusedCommentId=16816874&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16816874. We set the cleanup policy to both compact,delete and very quickly the partition was down to below 2G. Not sure if this is something log cleaner should be able to handle normally? There's a related issue here: https://issues.apache.org/jira/browse/KAFKA-3917, just thought it was a little bit outdated/dead. was: There's a related issue here: https://issues.apache.org/jira/browse/KAFKA-3917, just thought it was a little bit outdated/dead. We observed a few out of memory errors with our Kafka servers and our theory was due to 2 overly large partitions in {{__consumer_offsets}}. On further digging, it looks like these 2 large partitions have segments dating up to 3 months ago. Also, these old files collectively consumed most of the data from those partitions (About 10G from the partition's 12G). When we tried dumping those old segments, we see: {code:java} 1:40 $ ./kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000161728257775.log --offsets-decoder --print-data-log --deep-iteration Dumping 00000000161728257775.log Starting offset: 161728257775 offset: 161728257904 position: 61 CreateTime: 1553457816168 isvalid: true keysize: 4 valuesize: 6 magic: 2 compresscodec: NONE producerId: 367038 producerEpoch: 3 sequence: -1 isTransactional: true headerKeys: [] endTxnMarker: COMMIT coordinatorEpoch: 746 offset: 161728258098 position: 200 CreateTime: 1553457816230 isvalid: true keysize: 4 valuesize: 6 magic: 2 compresscodec: NONE producerId: 366036 producerEpoch: 3 sequence: -1 isTransactional: true headerKeys: [] endTxnMarker: COMMIT coordinatorEpoch: 761 ...{code} It looks like all those old segments all contain transactional information, and the 2 partitions are 1 for the control message COMMIT, the other for the control message ABORT. (As a side note, we did take a while to figure out that for a segment with the control bit set, the key really is {{endTxnMarker}} and the value is {{coordinatorEpoch}}...otherwise in a non-control batch dump it would have value and payload. We were wondering if seeing what those 2 partitions contained in their keys may give us any clues). Our current workaround is based on this post: https://issues.apache.org/jira/browse/KAFKA-3917?focusedCommentId=16816874&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16816874. We set the cleanup policy to both compact,delete and very quickly the partition was down to below 2G. Not sure if this is something log cleaner should be able to handle normally? > 2 __consumer_offsets partitions grow very big > --------------------------------------------- > > Key: KAFKA-8547 > URL: https://issues.apache.org/jira/browse/KAFKA-8547 > Project: Kafka > Issue Type: Bug > Components: log cleaner > Affects Versions: 2.1.1 > Environment: Ubuntu 18.04, Kafka 2.1.12-2.1.1, running as systemd > service > Reporter: Lerh Chuan Low > Priority: Major > > It seems like log cleaner doesn't clean {{__consumer_offsets}} very well on > the default policy of compact on that topic. > We observed a few out of memory errors with our Kafka servers and our theory > was due to 2 overly large partitions in {{__consumer_offsets}}. On further > digging, it looks like these 2 large partitions have segments dating up to 3 > months ago. Also, these old files collectively consumed most of the data from > those partitions (About 10G from the partition's 12G). > When we tried dumping those old segments, we see: > > {code:java} > 1:40 $ ./kafka-run-class.sh kafka.tools.DumpLogSegments --files > 00000000161728257775.log --offsets-decoder --print-data-log --deep-iteration > Dumping 00000000161728257775.log > Starting offset: 161728257775 > offset: 161728257904 position: 61 CreateTime: 1553457816168 isvalid: true > keysize: 4 valuesize: 6 magic: 2 compresscodec: NONE producerId: 367038 > producerEpoch: 3 sequence: -1 isTransactional: true headerKeys: [] > endTxnMarker: COMMIT coordinatorEpoch: 746 > offset: 161728258098 position: 200 CreateTime: 1553457816230 isvalid: true > keysize: 4 valuesize: 6 magic: 2 compresscodec: NONE producerId: 366036 > producerEpoch: 3 sequence: -1 isTransactional: true headerKeys: [] > endTxnMarker: COMMIT coordinatorEpoch: 761 > ...{code} > It looks like all those old segments all contain transactional information, > and the 2 partitions are 1 for the control message COMMIT, the other for the > control message ABORT. (As a side note, we did take a while to figure out > that for a segment with the control bit set, the key really is > {{endTxnMarker}} and the value is {{coordinatorEpoch}}...otherwise in a > non-control batch dump it would have value and payload. We were wondering if > seeing what those 2 partitions contained in their keys may give us any > clues). Our current workaround is based on this post: > https://issues.apache.org/jira/browse/KAFKA-3917?focusedCommentId=16816874&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16816874. > We set the cleanup policy to both compact,delete and very quickly the > partition was down to below 2G. Not sure if this is something log cleaner > should be able to handle normally? > There's a related issue here: > https://issues.apache.org/jira/browse/KAFKA-3917, just thought it was a > little bit outdated/dead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)