All of the information Todd posted is important to know. There was also jira related to this that has been committed trunk: https://issues.apache.org/jira/browse/KAFKA-2436
Before that patch, log.retention.hours was used to calculate KafkaConfig.logRetentionTimeMillis. But it was not used in LogManager to decide when to delete a log. LogManager was only using the log.retention.ms in the broker configuration. Could you try setting log.retention.ms=3600000 instead of using the hours config? On Mon, Sep 21, 2015 at 10:33 PM, Todd Palino <tpal...@gmail.com> wrote: > Retention is going to be based on a combination of both the retention and > segment size settings (as a side note, it's recommended to use > log.retention.ms and log.segment.ms, not the hours config. That's there > for > legacy reasons, but the ms configs are more consistent). As messages are > received by Kafka, they are written to the current open log segment for > each partition. That segment is rotated when either the log.segment.bytes > or the log.segment.ms limit is reached. Once that happens, the log segment > is closed and a new one is opened. Only after a log segment is closed can > it be deleted via the retention settings. Once the log segment is closed > AND either all the messages in the segment are older than log.retention.ms > OR the total partition size is greater than log.retention.bytes, then the > log segment is purged. > > As a note, the default segment limit is 1 gibibyte. So if you've only > written in 1k of messages, you have a long way to go before that segment > gets rotated. This is why the retention is referred to as a minimum time. > You can easily retain much more than you're expecting for slow topics. > > -Todd > > > On Mon, Sep 21, 2015 at 7:28 PM, allen chan <allen.michael.c...@gmail.com> > wrote: > > > I guess that kind of makes sense. > > The following section in the config is what confused me: > > *"# The following configurations control the disposal of log segments. > The > > policy can* > > *# be set to delete segments after a period of time, or after a given > size > > has accumulated.* > > *# A segment will be deleted whenever *either* of these criteria are met. > > Deletion always happens* > > *# from the end of the log."* > > > > That makes it sound like deletion will happen if either of the criteria > is > > met. > > I thought the whole idea of those two settings (time and bytes) is > telling > > the application when it will need to delete. > > > > > > > > On Mon, Sep 21, 2015 at 7:10 PM, noah <iamn...@gmail.com> wrote: > > > > > "minimum age of a log file to be eligible for deletion" Key word is > > > minimum. If you only have 1k logs, Kafka doesn't need to delete > anything. > > > Try to push more data through and when it needs to, it will start > > deleting > > > old logs. > > > > > > On Mon, Sep 21, 2015 at 8:58 PM allen chan < > allen.michael.c...@gmail.com > > > > > > wrote: > > > > > > > Hi, > > > > > > > > Just brought up new kafka cluster for testing. > > > > Was able to use the console producers to send 1k of logs and received > > it > > > on > > > > the console consumer side. > > > > > > > > The one issue that i have right now is that the retention period does > > not > > > > seem to be working. > > > > > > > > *# The minimum age of a log file to be eligible for deletion* > > > > *log.retention.hours=1* > > > > > > > > I have waited for almost 2 hours and the 1k of logs are still in > kafka. > > > > > > > > I did see these messages pop up on the console > > > > *[2015-09-21 17:12:01,236] INFO Scheduling log segment 0 for log > test-1 > > > for > > > > deletion. (kafka.log.Log)* > > > > *[2015-09-21 17:13:01,238] INFO Deleting segment 0 from log test-1. > > > > (kafka.log.Log)* > > > > *[2015-09-21 17:13:01,239] INFO Deleting index > > > > /var/log/kafka/test-1/00000000000000000000.index.deleted > > > > (kafka.log.OffsetIndex)* > > > > > > > > I know the logs are still in there because i am using > > > > the kafka-consumer-offset-checker.sh and it says how many messages > the > > > > logSize is. > > > > > > > > What am i missing in my configuration? > > > > > > > > > > > > > > > > Thanks! > > > > > > > > -- > > > > Allen Michael Chan > > > > > > > > > > > > > > > -- > > Allen Michael Chan > > > -- Grant Henke Software Engineer | Cloudera gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
