Jeff: This was with 0.9.0.1. It has not recurred since upgrading to 0.10.1.0.
On Fri, Oct 28, 2016 at 9:28 PM, Jeff Widman <j...@netskope.com> wrote: > James, > What version did you experience the problem with? > > On Oct 28, 2016 6:26 PM, "James Brown" <jbr...@easypost.com> wrote: > > > I was having this problem with one of my __consumer_offsets partitions; I > > used reassignment to move the large partition onto a different set of > > machines (which forced the cleaner to run through them again) and after > the > > new machines finished replicating, the partition was back down from 41GB > to > > a nice trim 38MB. > > > > On Fri, Oct 28, 2016 at 1:00 PM, Chi Hoang <chi.ho...@zuora.com> wrote: > > > > > Hi, > > > We have a 3-node cluster that is running 0.9.0.1, and recently saw that > > the > > > "__consumer_offsets" topic on one of the nodes seems really skewed with > > > disk usage that looks like: > > > > > > 73G ./__consumer_offsets-10 > > > 0 ./__consumer_offsets-7 > > > 0 ./__consumer_offsets-4 > > > 0 ./__consumer_offsets-1 > > > 0 ./__consumer_offsets-49 > > > 19G ./__consumer_offsets-46 > > > 0 ./__consumer_offsets-43 > > > 0 ./__consumer_offsets-40 > > > > > > > > > > > > This goes on for all 50 partitions. Upon inspection, we saw that a lot > > of > > > the log files were old: > > > > > > ll __consumer_offsets-10 > > > total 76245192 > > > -rw-r--r-- 1 root root 0 Oct 7 20:14 > 00000000000000000000.index > > > -rw-r--r-- 1 root root 901 Oct 7 20:14 00000000000000000000.log > > > -rw-r--r-- 1 root root 157904 Oct 7 22:15 > 00000000000907046457.index > > > -rw-r--r-- 1 root root 104855056 Oct 7 22:15 00000000000907046457.log > > > -rw-r--r-- 1 root root 157904 Oct 7 22:51 > 00000000000909543421.index > > > -rw-r--r-- 1 root root 104853568 Oct 7 22:51 00000000000909543421.log > > > -rw-r--r-- 1 root root 157904 Oct 7 23:27 > 00000000000910806717.index > > > -rw-r--r-- 1 root root 104853568 Oct 7 23:27 00000000000910806717.log > > > > > > > > > We are using default parameters as it pertains to offset management, > and > > > our config output includes the following entries: > > > > > > log.cleaner.enable = true > > > > > > offsets.retention.minutes = 1440 > > > > > > > > > I tried looking through the issues on JIRA but didn't see a reported > > > issue. Does anyone know what's going on, and how I can fix this? > > > > > > Thanks. > > > > > > > > > > > -- > > James Brown > > Engineer > > > -- James Brown Engineer
