[
https://issues.apache.org/jira/browse/KAFKA-7092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522545#comment-16522545
]
Jason Gustafson commented on KAFKA-7092:
----------------------------------------
Yes, you are probably hitting one of those bugs. The most likely one is
KAFKA-5413 which is fixed in all versions greater than 0.11. The fix for
KAFKA-6264 will be in 2.0.0, which is in the process of being released. We will
also be releasing 0.10.2.2 shortly, which will contain the fix for KAFKA-5413.
> Multiple topics (including __consumer_offsets) have not been cleaned for a
> few months
> -------------------------------------------------------------------------------------
>
> Key: KAFKA-7092
> URL: https://issues.apache.org/jira/browse/KAFKA-7092
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.10.2.1
> Environment: linux
> kafka 0.10.2.1
> Reporter: Noam Berman
> Priority: Critical
>
> Hi,
> We stumbled upon a scenario - our 8 node cluster (0.10.2.1) hasn't been
> cleaning logs for many topics since 26/3/2018 (during which the brokers were
> restarted). This includes __consumer_offsets, which has since grown to
> enormouse proportions.
> The cluster is an upgraded cluster from 0.10.1.0.
> I believe this is a reproduction of either
> https://issues.apache.org/jira/browse/KAFKA-5413 or
> https://issues.apache.org/jira/browse/KAFKA-6264, but i haven't found any
> workaround for this.
> output of one of __consumer_offsets partition data folders:
> {noformat}
> user@kafka-broker0a:/var/lib/kafka/data/__consumer_offsets-100$ ls -ltr
> total 132832
> -rw-r--r-- 1 kafka kafka 0 Mar 26 07:10 00000000000000000000.log
> -rw-r--r-- 1 kafka kafka 320648 Mar 26 07:11 00000000004729570626.log
> -rw-r--r-- 1 kafka kafka 2697871 Mar 26 07:11 00000000006877054274.log
> -rw-r--r-- 1 kafka kafka 3792861 Mar 26 07:11 00000000006877066658.log
> -rw-r--r-- 1 kafka kafka 931064 Mar 26 07:11 00000000006877084831.log
> -rw-r--r-- 1 kafka kafka 118282 Mar 26 07:11 00000000006877089209.log
> -rw-r--r-- 1 kafka kafka 1807647 Mar 26 07:11 00000000006877089458.log
> -rw-r--r-- 1 kafka kafka 239104 Mar 26 07:11 00000000006877096136.log
> -rw-r--r-- 1 kafka kafka 1835988 Mar 26 07:11 00000000006877097398.log
> -rw-r--r-- 1 kafka kafka 575040 Mar 26 07:11 00000000006877104257.log
> -rw-r--r-- 1 kafka kafka 390255 Mar 26 07:11 00000000006877106774.log
> -rw-r--r-- 1 kafka kafka 3105807 Mar 26 07:11 00000000006877108341.log
> -rw-r--r-- 1 kafka kafka 2086353 Mar 26 07:11 00000000006877120354.log
> -rw-r--r-- 1 kafka kafka 2434773 Mar 26 07:12 00000000006877128121.log
> -rw-r--r-- 1 kafka kafka 1432500 Mar 26 07:12 00000000006877137370.log
> -rw-r--r-- 1 kafka kafka 1901742 Mar 26 07:12 00000000006877142384.log
> -rw-r--r-- 1 kafka kafka 2310319 Mar 26 07:12 00000000006877149921.log
> -rw-r--r-- 1 kafka kafka 2084367 Mar 26 07:12 00000000006877157735.log
> -rw-r--r-- 1 kafka kafka 23317 Mar 26 07:12 00000000006877165836.log
> -rw-r--r-- 1 kafka kafka 3715029 Mar 26 07:12 00000000006877165946.log
> -rw-r--r-- 1 kafka kafka 2217586 Mar 26 07:12 00000000006877179092.log
> -rw-r--r-- 1 kafka kafka 1133491 Mar 26 07:12 00000000006877187739.log
> -rw-r--r-- 1 kafka kafka 1351694 Mar 26 07:12 00000000006877191615.log
> -rw-r--r-- 1 kafka kafka 1397573 Mar 26 19:11 00000000006877195811.log
> -rw-r--r-- 1 kafka kafka 1439494 Mar 28 00:37 00000000006877201824.log
> -rw-r--r-- 1 kafka kafka 1679347 Mar 29 07:10 00000000006877207157.log
> -rw-r--r-- 1 kafka kafka 0 Mar 29 09:35 00000000000000000000.timeindex
> -rw-r--r-- 1 kafka kafka 0 Mar 29 09:35 00000000000000000000.index
> -rw-r--r-- 1 kafka kafka 12 Mar 29 09:35 00000000006877089458.timeindex
> -rw-r--r-- 1 kafka kafka 8 Mar 29 09:35 00000000006877089458.index
> -rw-r--r-- 1 kafka kafka 12 Mar 29 09:35 00000000006877089209.timeindex
> -rw-r--r-- 1 kafka kafka 0 Mar 29 09:35 00000000006877089209.index
> -rw-r--r-- 1 kafka kafka 12 Mar 29 09:35 00000000006877084831.timeindex
> -rw-r--r-- 1 kafka kafka 0 Mar 29 09:35 00000000006877084831.index
> -rw-r--r-- 1 kafka kafka 36 Mar 29 09:35 00000000006877066658.timeindex
> -rw-r--r-- 1 kafka kafka 24 Mar 29 09:35 00000000006877066658.index
> .....
> .....
> -rw-r--r-- 1 kafka kafka 10485760 Jun 24 14:52 00000000006877558794.index
> -rw-r--r-- 1 kafka kafka 1235857 Jun 24 14:52
> 00000000006877558794.log{noformat}
> As you can see, the oldest log file has a 00000..00000 file name.
> Is there any version that we can upgrade to that will solve this issue for
> us?
>
> Thanks
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
