No, we haven't configured that. We have a few hundred topics but this seems to be the only one affected (I did a quick check, not thorough).
The relevant config params that we have set in server.properties. log.dir=/var/lib/fk-3p-kafka/logs log.flush.interval.messages=10000 log.flush.interval.ms=1000 log.retention.hours=168 log.segment.bytes=536870912 log.cleanup.interval.mins=1 log.retention.hours=336 On Fri, Jul 25, 2014 at 10:11 AM, Jun Rao <jun...@gmail.com> wrote: > Have you configured log.retention.bytes? > > Thanks, > > Jun > > > On Thu, Jul 24, 2014 at 10:04 AM, Kashyap Paidimarri <kashy...@gmail.com> > wrote: > > > We just noticed that one of our topics has been horribly misbehaving. > > > > *retention.ms <http://retention.ms>* for the topic is set to 1209600000 > ms > > > > However, segments are getting schedule for deletetion as soon as a new > one > > is rolled over. And naturally consumers are running into a > > kafka.common.OffsetOutOfRangeException whenever this happens. > > > > Is this a known bug? It is incredibly serious. We seem to have lost about > > 40 million messages on a single topic and are yet to figure out what all > > topics are affected. > > > > I thought of restarting Kafka but figured I'd leave it untouched while I > > figure out what I can capture for finding the root cause. > > > > Meanwhile in order to keep from losing any more data, I have a periodic > job > > that is doing a *'cp -al' *of the partitions into a separate folder. That > > way Kafka goes ahead and deletes the segment but the data is not lost > from > > the filesystem. > > > > If this is a unseen bug, what should I save from the running instance. > > > > By the way, this has affected all partitions and replicas of the topic > and > > not on a specific host. > > > -- “ The difference between ramen and varelse is not in the creature judged, but in the creature judging. When we declare an alien species to be ramen, it does not mean that *they* have passed a threshold of moral maturity. It means that *we* have. —Demosthenes, *Letter to the Framlings* ”
