I think you are misunderstanding what log.retention.bytes does. It is a limit on the total size in a log directory (per partition). It's not a limit on the total size of all logs on a Kafka broker node.
There are efforts to add the functionality you are looking for. Take a look at KAFKA-1489 <https://issues.apache.org/jira/browse/KAFKA-1489> and KIP-61 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-61%3A+Add+a+log+retention+parameter+for+maximum+disk+space+usage+percentage> for more detail. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Mon, Oct 17, 2016 at 2:58 AM, Kunal Gupta <kunal.gu...@cube26.com> wrote: > log.retention.bytes=5000000000000 > > Does it not mean that when log size reaches to 5TB then it will discard log > > What you have specified is when disk space of a machine left with 5.5TB > then it will discard the data > > *Thanks, Kunal* > *+91-9958189589* > *Data Analyst* > *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798 > <http://dl.acm.org/citation.cfm?id=2790798> * > *Blog:- **http://learnhardwithkunalgupta.blogspot.in > <http://learnhardwithkunalgupta.blogspot.in> * > > > > > > On Mon, Oct 17, 2016 at 3:21 PM, Ben Davison <ben.davi...@7digital.com> > wrote: > > > Here's an example from our server.properties > > > > log.segment.bytes=1073741824 > > message.max.bytes=5242880 > > num.partitions=3 > > log.retention.bytes=5000000000000 > > num.network.threads=6 > > num.io.threads=16 > > replica.fetch.max.bytes=6242880 > > default.replication.factor=3 > > zookeeper.connection.timeout.ms=60000 > > auto.leader.rebalance.enable=true > > delete.topic.enable=true > > > > > > log.retention.bytes=5000000000000 is the setting, we have 5.5TB > available, > > so we start deleting at 5TB space used. > > > > > > > > > > On Mon, Oct 17, 2016 at 10:37 AM, Kunal Gupta <kunal.gu...@cube26.com> > > wrote: > > > > > I didn't get it ... > > > > > > Can you explain me in form of example or something for which you are > > > feasible > > > > > > On Oct 17, 2016 2:08 PM, "Ben Davison" <ben.davi...@7digital.com> > wrote: > > > > > > > We have it setup so that both log ms is set to 7 days and log delete > > > > bytes(can't remember exactly what the setting is called. So we never > > run > > > > out of space (don't set the value to something like 99% of your disk, > > as > > > > the log cleaner thread might not kick in time, we leave it at 90% of > > > disks > > > > space) > > > > > > > > On Monday, 17 October 2016, Kunal Gupta <kunal.gu...@cube26.com> > > wrote: > > > > > > > > > Please help me :( > > > > > > > > > > *Thanks, Kunal* > > > > > *+91-9958189589* > > > > > *Data Analyst* > > > > > *First Paper Publication : **http://dl.acm.org/citation. > > cfm?id=2790798 > > > > > <http://dl.acm.org/citation.cfm?id=2790798> * > > > > > *Blog:- **http://learnhardwithkunalgupta.blogspot.in > > > > > <http://learnhardwithkunalgupta.blogspot.in> * > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Oct 16, 2016 at 11:23 AM, Kunal Gupta < > > kunal.gu...@cube26.com > > > > > <javascript:;>> > > > > > wrote: > > > > > > > > > > > > > > > > > In my organisation I have 3 machine cluster of Kafka and each > topic > > > > > > assigned two machine for storing there data. > > > > > > > > > > > > There is one topic for which I get lot of data from clients thats > > > data > > > > > > exceeds my disk space in one machine because that machine is a > > leader > > > > of > > > > > > that topic, when I look into kafka-logs seeing all topic data > there > > > is > > > > > only > > > > > > one topic whose data has consumed lot of space. > > > > > > > > > > > > In one month I have 36 GB data from that topic. What I have > thought > > > so > > > > > far > > > > > > is kafka logs will be delete after 24 hours and new data will be > > > > retained > > > > > > for 24 hours and my consumer will consume all data with in 24 > > hours, > > > so > > > > > > there will be no problem of disk space. > > > > > > > > > > > > I have enable everything in server.properties to clean kafka-logs > > but > > > > > > still data is there. Below data is my settings in > server.properties > > > > file. > > > > > > > > > > > > *log.retention.minutes=1440* > > > > > > *log.retention.hours=24* > > > > > > *log.retention.ms <http://log.retention.ms>=86400000* > > > > > > *log.cleaner.delete.retention.ms > > > > > > <http://log.cleaner.delete.retention.ms>=24* > > > > > > *log.segment.bytes=1048576* > > > > > > *log.retention.check.interval.ms > > > > > > <http://log.retention.check.interval.ms>=3000* > > > > > > *log.cleaner.enable=true* > > > > > > *zookeeper.connection.timeout.ms > > > > > > <http://zookeeper.connection.timeout.ms>=30000* > > > > > > *delete.topic.enable = true* > > > > > > *auto.create.topics.enable = true* > > > > > > *default.replication.factor=2* > > > > > > *auto.leader.rebalance.enable=true* > > > > > > *controlled.shutdown.enable=true* > > > > > > *controller.socket.timeout.ms <http://controller.socket. > timeout.ms > > > > > >=120000* > > > > > > > > > > > > Please help me in this so that my machine can able to handle > large > > > > number > > > > > > of request and data well using 3 machine cluster. > > > > > > > > > > > > *Thanks, Kunal* > > > > > > *+91-9958189589* > > > > > > *Data Analyst* > > > > > > *First Paper Publication : **http://dl.acm.org/citation. > > > cfm?id=2790798 > > > > > > <http://dl.acm.org/citation.cfm?id=2790798> * > > > > > > *Blog:- **http://learnhardwithkunalgupta.blogspot.in > > > > > > <http://learnhardwithkunalgupta.blogspot.in> * > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > This email, including attachments, is private and confidential. If > you > > > have > > > > received this email in error please notify the sender and delete it > > from > > > > your system. Emails are not secure and may contain viruses. No > > liability > > > > can be accepted for viruses that might be transferred by this email > or > > > any > > > > attachment. Any unauthorised copying of this message or unauthorised > > > > distribution and publication of the information contained herein are > > > > prohibited. > > > > > > > > 7digital Limited. Registered office: 69 Wilson Street, London EC2A > 2BB. > > > > Registered in England and Wales. Registered No. 04843573. > > > > > > > > > > > -- > > > > > > This email, including attachments, is private and confidential. If you > have > > received this email in error please notify the sender and delete it from > > your system. Emails are not secure and may contain viruses. No liability > > can be accepted for viruses that might be transferred by this email or > any > > attachment. Any unauthorised copying of this message or unauthorised > > distribution and publication of the information contained herein are > > prohibited. > > > > 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB. > > Registered in England and Wales. Registered No. 04843573. > > >
