Re: Consumer offsets partitions size much bigger than others

2017-07-24 Thread Luciano Afranllie
Thanks James The issue was that the log cleaner was disabled in that cluster due to an old configuration that we have since 0.9.0.0. Regards Luciano On Tue, Jul 18, 2017 at 7:04 PM, James Cheng wrote: > It's possible that the log-cleaning thread has crashed. That is the thread > that implement

Re: Consumer offsets partitions size much bigger than others

2017-07-18 Thread James Cheng
It's possible that the log-cleaning thread has crashed. That is the thread that implements log compaction. Look in the log-cleaner.log file in your kafka debuglog directory to see if there is any indication that it has crashed (error messages, stack traces, etc). What version of kafka are you u

Consumer offsets partitions size much bigger than others

2017-07-18 Thread Luciano Afranllie
Hi One of our Kafka brokers was running out of disk space and when we checked the file size in the kafka log dir we observed the following $ du -h . --max-depth=2 | grep '__consumer_offsets' 4.0K./kafka-logs/__consumer_offsets-16 4.0K./kafka-logs/__consumer_offsets-40 35G ./kafka-logs