[
https://issues.apache.org/jira/browse/KAFKA-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159059#comment-17159059
]
zhangzhisheng commented on KAFKA-9973:
--------------------------------------
__consumer_offsets can not clean
kafka version 2.12-2.4.1
jdk version jdk1.8.0_231
{code:java}
//
21G ./data/kafka-logs/topic-2018066952-7
27G ./data/kafka-logs/__consumer_offsets-25
30G ./data/kafka-logs/topic-2018066951-0
30G ./data/kafka-logs/topic-2018066951-1
30G ./data/kafka-logs/topic-2018066951-3
30G ./data/kafka-logs/topic-2018066951-5
30G ./data/kafka-logs/topic-2018066951-6
30G ./data/kafka-logs/topic-2018066951-7
32G ./data/kafka-logs/__consumer_offsets-30
32G ./data/kafka-logs/topic-201911251114-0
32G ./data/kafka-logs/topic-201911251114-2
{code}
> __consumer_offsets record is invalid lead to log clean failed and
> __consumer_offsets grows too big
> --------------------------------------------------------------------------------------------------
>
> Key: KAFKA-9973
> URL: https://issues.apache.org/jira/browse/KAFKA-9973
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
> Affects Versions: 2.1.1
> Reporter: leibo
> Priority: Major
> Attachments: dump.png, log-cleaner.log.1, screenshot-1.png
>
>
> __consumer_offsets-34 grows to 13GB, and can't be compact successf, the error
> log as bellow:
> {code:java}
> //代码占位符
> [2019-12-30 19:21:41,047] INFO [kafka-log-cleaner-thread-6]: Starting
> (kafka.log.LogCleaner)
> [2019-12-30 19:21:41,079] INFO [kafka-log-cleaner-thread-7]: Starting
> (kafka.log.LogCleaner)
> [2019-12-30 19:21:42,825] WARN [kafka-log-cleaner-thread-0]: Unexpected
> exception thrown when cleaning log
> Log(/home/kafka_2.12-2.1.1/data/kafka-logs/__consumer_offsets-34). Marking
> its partition (__consumer_offsets-34) as uncleanable (kafka.log.LogCleaner)
> org.apache.kafka.common.record.InvalidRecordException: Found invalid number
> of record headers -47
> --
> [2019-12-30 20:23:51,340] INFO [kafka-log-cleaner-thread-5]: Starting
> (kafka.log.LogCleaner)
> [2019-12-30 20:23:51,361] INFO Cleaner 0: Building offset map for log
> __consumer_offsets-34 for 52 segments in offset range [26842714, 73423730).
> (kafka.log.LogCleaner)
> [2019-12-30 20:23:51,398] INFO [kafka-log-cleaner-thread-6]: Starting
> (kafka.log.LogCleaner)
> [2019-12-30 20:23:51,493] INFO [kafka-log-cleaner-thread-7]: Starting
> (kafka.log.LogCleaner)
> [2019-12-30 20:23:59,596] WARN [kafka-log-cleaner-thread-0]: Unexpected
> exception thrown when cleaning log
> Log(/home/kafka_2.12-2.1.1/data/kafka-logs/__consumer_offsets-34). Marking
> its partition (__consumer_offsets-34) as uncleanable (kafka.log.LogCleaner)
> org.apache.kafka.common.record.InvalidRecordException: Found invalid number
> of record headers -47
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
