[jira] [Commented] (KAFKA-3000) __consumer_offsets topic grows indefinitely
[ https://issues.apache.org/jira/browse/KAFKA-3000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15061168#comment-15061168 ] Maxim Vladimirskiy commented on KAFKA-3000: --- After I set log.cleaner.enable=true the size of the __consumer_offsets started to go down rapidly. Thanks a lot Grant. And you should definitely remove this option all together, it does not make sense to have a cleaner disabled. If a user neither has compacted topics nor using offset management it should not be a huge performance hit to have an idle cleaner sticking around, unless you screwed up implementing it a big time, just joking :). Keep up the good work. Kafka is great! > __consumer_offsets topic grows indefinitely > --- > > Key: KAFKA-3000 > URL: https://issues.apache.org/jira/browse/KAFKA-3000 > Project: Kafka > Issue Type: Bug > Components: offset manager >Affects Versions: 0.8.2.1 > Environment: Ubuntu 14.04, 5 node kafka cluster + 5 node ZooKeeper > cluster. ZooKeeper and Kafka coexist on 5 cloud boxes. >Reporter: Maxim Vladimirskiy >Assignee: Grant Henke > > Old segments of the __consumer_offsets topic seem to be never deleted. As of > Dec 16, 2015, there are segments dating back to when we started use Kafka - > Oct 26, 2015. However the idx for the respective oldest segment file is just > about an hour fresh. All offset related settings are default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3000) __consumer_offsets topic grows indefinitely
[ https://issues.apache.org/jira/browse/KAFKA-3000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15061154#comment-15061154 ] Grant Henke commented on KAFKA-3000: [~horkhe] Glad to hear that was it. I will resolve as not a problem. > __consumer_offsets topic grows indefinitely > --- > > Key: KAFKA-3000 > URL: https://issues.apache.org/jira/browse/KAFKA-3000 > Project: Kafka > Issue Type: Bug > Components: offset manager >Affects Versions: 0.8.2.1 > Environment: Ubuntu 14.04, 5 node kafka cluster + 5 node ZooKeeper > cluster. ZooKeeper and Kafka coexist on 5 cloud boxes. >Reporter: Maxim Vladimirskiy > > Old segments of the __consumer_offsets topic seem to be never deleted. As of > Dec 16, 2015, there are segments dating back to when we started use Kafka - > Oct 26, 2015. However the idx for the respective oldest segment file is just > about an hour fresh. All offset related settings are default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3000) __consumer_offsets topic grows indefinitely
[ https://issues.apache.org/jira/browse/KAFKA-3000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15061133#comment-15061133 ] Maxim Vladimirskiy commented on KAFKA-3000: --- It was set to the default value `false`, I am going to change it to `true`. The largest __consumer_offset partition has grown to be 64G, I guess I have one hell of a compaction ahead. > __consumer_offsets topic grows indefinitely > --- > > Key: KAFKA-3000 > URL: https://issues.apache.org/jira/browse/KAFKA-3000 > Project: Kafka > Issue Type: Bug > Components: offset manager >Affects Versions: 0.8.2.1 > Environment: Ubuntu 14.04, 5 node kafka cluster + 5 node ZooKeeper > cluster. ZooKeeper and Kafka coexist on 5 cloud boxes. >Reporter: Maxim Vladimirskiy > > Old segments of the __consumer_offsets topic seem to be never deleted. As of > Dec 16, 2015, there are segments dating back to when we started use Kafka - > Oct 26, 2015. However the idx for the respective oldest segment file is just > about an hour fresh. All offset related settings are default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3000) __consumer_offsets topic grows indefinitely
[ https://issues.apache.org/jira/browse/KAFKA-3000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15061105#comment-15061105 ] Grant Henke commented on KAFKA-3000: Do you have log.cleaner.enable=true configured on your brokers? The default is false. Since __consumer_offsets is a compacted topic old segments aren't supposed to be deleted. Instead the log cleaner, if enabled, should compact the topic. Discussion about changing the default to true, and more related handling can be found in KAFKA-2988 and this email thread in the developer list: http://search-hadoop.com/m/uyzND1XRTEVv2ToT1&subj=Consumer+Offsets+Compaction > __consumer_offsets topic grows indefinitely > --- > > Key: KAFKA-3000 > URL: https://issues.apache.org/jira/browse/KAFKA-3000 > Project: Kafka > Issue Type: Bug > Components: offset manager >Affects Versions: 0.8.2.1 > Environment: Ubuntu 14.04, 5 node kafka cluster + 5 node ZooKeeper > cluster. ZooKeeper and Kafka coexist on 5 cloud boxes. >Reporter: Maxim Vladimirskiy > > Old segments of the __consumer_offsets topic seem to be never deleted. As of > Dec 16, 2015, there are segments dating back to when we started use Kafka - > Oct 26, 2015. However the idx for the respective oldest segment file is just > about an hour fresh. All offset related settings are default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
