Re: Interaction of retention settings for broker and topic plus partitions
To clarify though, is it correct that a per topic limit will always override the default limit of the same type? (e.g. a large per-topic retention hours vs. a small default retention hours)? That's correct. On Thu, Nov 6, 2014 at 9:34 AM, Jason Rosenberg wrote: > Jun, > > To clarify though, is it correct that a per topic limit will always > override the default limit of the same type? (e.g. a large per-topic > retention hours vs. a small default retention hours)? > > Jason > > On Sat, Sep 20, 2014 at 12:28 AM, Jun Rao wrote: > > > That's right. The rule is that a log segment is deleted if either the > size > > or the time limit is reached. Log sizes are per partition. > > > > Thanks, > > > > Jun > > > > On Thu, Sep 18, 2014 at 2:55 PM, Cory Watson wrote: > > > > > Hello all! > > > > > > I'm curious about the interaction of server and topic level retention > > > settings. It's not clear to me the precedence of the follow: > > > > > >- broker's default log.retention.bytes > > >- topic's retention.bytes (which defaults to broker's > > >log.retention.bytes) > > >- broker's log.retention.hours and log.retention.minutes (if both > are > > >specified then it seems to be the lower of the two, since it's when > > >"either" is exceeded) > > > > > > It seems that the rule is that when any of these are violated then the > > log > > > segment is deleted. Is this right? > > > > > > Also, just to be clear: The log sizes in questions are for a single > > > partitions logs? > > > > > > I have a situation where my per-topic retention.bytes is very high, but > > my > > > default log.retention.hours is lower (the default @ 168 hours). It > seems > > > that it's truncating at the log.retention.hours instead of the topic's > > > retention.bytes. > > > > > > Am I understanding this correctly? :) > > > > > > -- > > > Cory Watson > > > Principal Infrastructure Engineer // Keen IO > > > > > >
Re: Interaction of retention settings for broker and topic plus partitions
Jun, To clarify though, is it correct that a per topic limit will always override the default limit of the same type? (e.g. a large per-topic retention hours vs. a small default retention hours)? Jason On Sat, Sep 20, 2014 at 12:28 AM, Jun Rao wrote: > That's right. The rule is that a log segment is deleted if either the size > or the time limit is reached. Log sizes are per partition. > > Thanks, > > Jun > > On Thu, Sep 18, 2014 at 2:55 PM, Cory Watson wrote: > > > Hello all! > > > > I'm curious about the interaction of server and topic level retention > > settings. It's not clear to me the precedence of the follow: > > > >- broker's default log.retention.bytes > >- topic's retention.bytes (which defaults to broker's > >log.retention.bytes) > >- broker's log.retention.hours and log.retention.minutes (if both are > >specified then it seems to be the lower of the two, since it's when > >"either" is exceeded) > > > > It seems that the rule is that when any of these are violated then the > log > > segment is deleted. Is this right? > > > > Also, just to be clear: The log sizes in questions are for a single > > partitions logs? > > > > I have a situation where my per-topic retention.bytes is very high, but > my > > default log.retention.hours is lower (the default @ 168 hours). It seems > > that it's truncating at the log.retention.hours instead of the topic's > > retention.bytes. > > > > Am I understanding this correctly? :) > > > > -- > > Cory Watson > > Principal Infrastructure Engineer // Keen IO > > >
Re: Interaction of retention settings for broker and topic plus partitions
That's right. The rule is that a log segment is deleted if either the size or the time limit is reached. Log sizes are per partition. Thanks, Jun On Thu, Sep 18, 2014 at 2:55 PM, Cory Watson wrote: > Hello all! > > I'm curious about the interaction of server and topic level retention > settings. It's not clear to me the precedence of the follow: > >- broker's default log.retention.bytes >- topic's retention.bytes (which defaults to broker's >log.retention.bytes) >- broker's log.retention.hours and log.retention.minutes (if both are >specified then it seems to be the lower of the two, since it's when >"either" is exceeded) > > It seems that the rule is that when any of these are violated then the log > segment is deleted. Is this right? > > Also, just to be clear: The log sizes in questions are for a single > partitions logs? > > I have a situation where my per-topic retention.bytes is very high, but my > default log.retention.hours is lower (the default @ 168 hours). It seems > that it's truncating at the log.retention.hours instead of the topic's > retention.bytes. > > Am I understanding this correctly? :) > > -- > Cory Watson > Principal Infrastructure Engineer // Keen IO >
Interaction of retention settings for broker and topic plus partitions
Hello all! I'm curious about the interaction of server and topic level retention settings. It's not clear to me the precedence of the follow: - broker's default log.retention.bytes - topic's retention.bytes (which defaults to broker's log.retention.bytes) - broker's log.retention.hours and log.retention.minutes (if both are specified then it seems to be the lower of the two, since it's when "either" is exceeded) It seems that the rule is that when any of these are violated then the log segment is deleted. Is this right? Also, just to be clear: The log sizes in questions are for a single partitions logs? I have a situation where my per-topic retention.bytes is very high, but my default log.retention.hours is lower (the default @ 168 hours). It seems that it's truncating at the log.retention.hours instead of the topic's retention.bytes. Am I understanding this correctly? :) -- Cory Watson Principal Infrastructure Engineer // Keen IO
