[ https://issues.apache.org/jira/browse/KAFKA-7092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522545#comment-16522545 ]
Jason Gustafson commented on KAFKA-7092: ---------------------------------------- Yes, you are probably hitting one of those bugs. The most likely one is KAFKA-5413 which is fixed in all versions greater than 0.11. The fix for KAFKA-6264 will be in 2.0.0, which is in the process of being released. We will also be releasing 0.10.2.2 shortly, which will contain the fix for KAFKA-5413. > Multiple topics (including __consumer_offsets) have not been cleaned for a > few months > ------------------------------------------------------------------------------------- > > Key: KAFKA-7092 > URL: https://issues.apache.org/jira/browse/KAFKA-7092 > Project: Kafka > Issue Type: Bug > Components: core > Affects Versions: 0.10.2.1 > Environment: linux > kafka 0.10.2.1 > Reporter: Noam Berman > Priority: Critical > > Hi, > We stumbled upon a scenario - our 8 node cluster (0.10.2.1) hasn't been > cleaning logs for many topics since 26/3/2018 (during which the brokers were > restarted). This includes __consumer_offsets, which has since grown to > enormouse proportions. > The cluster is an upgraded cluster from 0.10.1.0. > I believe this is a reproduction of either > https://issues.apache.org/jira/browse/KAFKA-5413 or > https://issues.apache.org/jira/browse/KAFKA-6264, but i haven't found any > workaround for this. > output of one of __consumer_offsets partition data folders: > {noformat} > user@kafka-broker0a:/var/lib/kafka/data/__consumer_offsets-100$ ls -ltr > total 132832 > -rw-r--r-- 1 kafka kafka 0 Mar 26 07:10 00000000000000000000.log > -rw-r--r-- 1 kafka kafka 320648 Mar 26 07:11 00000000004729570626.log > -rw-r--r-- 1 kafka kafka 2697871 Mar 26 07:11 00000000006877054274.log > -rw-r--r-- 1 kafka kafka 3792861 Mar 26 07:11 00000000006877066658.log > -rw-r--r-- 1 kafka kafka 931064 Mar 26 07:11 00000000006877084831.log > -rw-r--r-- 1 kafka kafka 118282 Mar 26 07:11 00000000006877089209.log > -rw-r--r-- 1 kafka kafka 1807647 Mar 26 07:11 00000000006877089458.log > -rw-r--r-- 1 kafka kafka 239104 Mar 26 07:11 00000000006877096136.log > -rw-r--r-- 1 kafka kafka 1835988 Mar 26 07:11 00000000006877097398.log > -rw-r--r-- 1 kafka kafka 575040 Mar 26 07:11 00000000006877104257.log > -rw-r--r-- 1 kafka kafka 390255 Mar 26 07:11 00000000006877106774.log > -rw-r--r-- 1 kafka kafka 3105807 Mar 26 07:11 00000000006877108341.log > -rw-r--r-- 1 kafka kafka 2086353 Mar 26 07:11 00000000006877120354.log > -rw-r--r-- 1 kafka kafka 2434773 Mar 26 07:12 00000000006877128121.log > -rw-r--r-- 1 kafka kafka 1432500 Mar 26 07:12 00000000006877137370.log > -rw-r--r-- 1 kafka kafka 1901742 Mar 26 07:12 00000000006877142384.log > -rw-r--r-- 1 kafka kafka 2310319 Mar 26 07:12 00000000006877149921.log > -rw-r--r-- 1 kafka kafka 2084367 Mar 26 07:12 00000000006877157735.log > -rw-r--r-- 1 kafka kafka 23317 Mar 26 07:12 00000000006877165836.log > -rw-r--r-- 1 kafka kafka 3715029 Mar 26 07:12 00000000006877165946.log > -rw-r--r-- 1 kafka kafka 2217586 Mar 26 07:12 00000000006877179092.log > -rw-r--r-- 1 kafka kafka 1133491 Mar 26 07:12 00000000006877187739.log > -rw-r--r-- 1 kafka kafka 1351694 Mar 26 07:12 00000000006877191615.log > -rw-r--r-- 1 kafka kafka 1397573 Mar 26 19:11 00000000006877195811.log > -rw-r--r-- 1 kafka kafka 1439494 Mar 28 00:37 00000000006877201824.log > -rw-r--r-- 1 kafka kafka 1679347 Mar 29 07:10 00000000006877207157.log > -rw-r--r-- 1 kafka kafka 0 Mar 29 09:35 00000000000000000000.timeindex > -rw-r--r-- 1 kafka kafka 0 Mar 29 09:35 00000000000000000000.index > -rw-r--r-- 1 kafka kafka 12 Mar 29 09:35 00000000006877089458.timeindex > -rw-r--r-- 1 kafka kafka 8 Mar 29 09:35 00000000006877089458.index > -rw-r--r-- 1 kafka kafka 12 Mar 29 09:35 00000000006877089209.timeindex > -rw-r--r-- 1 kafka kafka 0 Mar 29 09:35 00000000006877089209.index > -rw-r--r-- 1 kafka kafka 12 Mar 29 09:35 00000000006877084831.timeindex > -rw-r--r-- 1 kafka kafka 0 Mar 29 09:35 00000000006877084831.index > -rw-r--r-- 1 kafka kafka 36 Mar 29 09:35 00000000006877066658.timeindex > -rw-r--r-- 1 kafka kafka 24 Mar 29 09:35 00000000006877066658.index > ..... > ..... > -rw-r--r-- 1 kafka kafka 10485760 Jun 24 14:52 00000000006877558794.index > -rw-r--r-- 1 kafka kafka 1235857 Jun 24 14:52 > 00000000006877558794.log{noformat} > As you can see, the oldest log file has a 00000..00000 file name. > Is there any version that we can upgrade to that will solve this issue for > us? > > Thanks > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)