Paul, I'm trying to understand the 2nd problem you reported. Are you saying that you set the log.retention.bytes=11534336 (11MB) but nevertheless your log grew to 114MB. Which means the config option didn't really work as expected?
Thanks, Neha On Mon, Aug 19, 2013 at 8:46 PM, Jun Rao <[email protected]> wrote: > For the first question, yes, topic C will get the value of 30MB. > > For the second question, log.retention.bytes only controls the segment log > file size, not the index. Typically, index file size is much smaller than > the log file. The index file of the last (active) segment is presized to > the max index size (defaults to 10MB). However, the size is trimmed as soon > as the segment rolls. > > Thanks, > > Jun > > > On Mon, Aug 19, 2013 at 9:22 AM, Paul Christian > <[email protected]>wrote: > > > Hi Jun, > > > > Thank you for your reply. I'm still a little fuzzy on the concept. > > > > Are you saying I can have topic A, B and C and with > > > > log.retention.bytes.per.topic.A = 15MB > > log.retention.bytes.per.topic.B = 20MB > > log.retention.bytes = 30MB > > > > And thus topic C will get the value 30MB? Since it's not defined like the > > others' 'per topic'? > > > > log.retention.bytes is for all topics that are not included in > > log.retention.bytes.per.topic > > (which defines a map of topic -> size). > > > > Otherwise, log.retention.bytes.per.topic and log.retention.bytes seem > very > > similar to me. > > > > Additionally, we've experimented with this value on our test cluster > where > > we set the log.retention.bytes to 11MB as a test. Below is a snippet from > > our server.properties. > > > > # A size-based retention policy for logs. Segments are pruned from the > log > > as long as the remaining > > # segments don't drop below log.retention.bytes. > > log.retention.bytes=11534336 > > > > Here is a ls -lh from one of the topics > > > > -rw-r--r-- 1 kafka service 10M Aug 19 15:45 00000000000007021913.index > > -rw-r--r-- 1 kafka service 114M Aug 19 15:45 00000000000007021913.log > > > > The index file appears to be reflected in the property > > log.index.size.max.bytes, but the log just keeps going. > > >
