Re: Monitoring and Alerting

Dropping in late on this thread, but we just released our Kafka metrics reporter for AppOptics: https://github.com/appoptics/kafka-appoptics-reporter It ties into the Yammer Metrics reporting interface to capture metrics and publish to Appoptics. Just build and drop the Jar into your Kafka libs di

Re: large amount of disk space freed on restart

; That said this issue only impacts xfs and it seems to be fixed by that > > setting jonathan found. I wonder if you could give that a try and see if > it > > works for you too? I feel dealing with closed files does add a lot of > > complexity so if there is an easy fix I would

Re: large amount of disk space freed on restart

> > > > > > > > > > > > > > > > > > On Thu, May 23, 2013 at 7:14 AM, Jun Rao < > jun...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Jason, > > > > > > > > > > > > > > > > > > > > > > Kafka closes the handler of all delete files. > Otherwise, > > > the > > > > > > broker > > > > > > > > > will > > > > > > > > > > > run out of file handler quickly. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, May 22, 2013 at 10:17 PM, Jason Rosenberg < > > > > > > > j...@squareup.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > So, does this indicate kafka (or the jvm itself) is > not > > > > > > > > aggressively > > > > > > > > > > > > closing file handles of deleted files? Is there a > fix > > > for > > > > > > this? > > > > > > > > Or > > > > > > > > > is > > > > > > > > > > > > there not likely anything to be done? What happens > if > > > the > > > > > disk > > > > > > > > fills > > > > > > > > > > up > > > > > > > > > > > > with file handles for phantom deleted files? > > > > > > > > > > > > > > > > > > > > > > > > Jason > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, May 22, 2013 at 9:50 PM, Jonathan Creasy < > > > > j...@box.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > It isn't uncommon if a process has an open file > > handle > > > > on a > > > > > > > file > > > > > > > > > that > > > > > > > > > > > is > > > > > > > > > > > > > deleted, the space is not freed until the handle is > > > > closed. > > > > > > So > > > > > > > > > > > restarting > > > > > > > > > > > > > the process that has a handle on the file would > cause > > > the > > > > > > space > > > > > > > > to > > > > > > > > > be > > > > > > > > > > > > freed > > > > > > > > > > > > > also. > > > > > > > > > > > > > > > > > > > > > > > > > > You can troubleshoot that with lsof. > > > > > > > > > > > > > Normally, I see 2-4 log segments deleted every hour > > in > > > my > > > > > > > > brokers. > > > > > > > > > I > > > > > > > > > > > see > > > > > > > > > > > > > log lines like this: > > > > > > > > > > > > > > > > > > > > > > > > > > 2013-05-23 04:40:06,857 INFO [kafka-logcleaner-0] > > > > > > > > log.LogManager - > > > > > > > > > > > > > Deleting log segment 035434043157.kafka > from > > > > > > > > > > > > > > topic> > > > > > > > > > > > > > > > > > > > > > > > > > > However, it seems like if I restart the broker, a > > > massive > > > > > > > amount > > > > > > > > of > > > > > > > > > > > disk > > > > > > > > > > > > > space is freed (without a corresponding flood of > > these > > > > log > > > > > > > > segment > > > > > > > > > > > > deleted > > > > > > > > > > > > > messages). Is there an explanation for this? Does > > > kafka > > > > > > keep > > > > > > > > > > > reference > > > > > > > > > > > > to > > > > > > > > > > > > > file segments around, and reuse them as needed or > > > > > something? > > > > > > > And > > > > > > > > > > then > > > > > > > > > > > or > > > > > > > > > > > > > restart, the references to those free segment files > > are > > > > > > > dropped? > > > > > > > > > > > > > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > > > > > > > > > > > > > This is with 0.7.2. > > > > > > > > > > > > > > > > > > > > > > > > > > Jason > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- Mike Heffner Librato, Inc.