log.retention.bytes=5000000000000 Does it not mean that when log size reaches to 5TB then it will discard log
What you have specified is when disk space of a machine left with 5.5TB then it will discard the data *Thanks, Kunal* *+91-9958189589* *Data Analyst* *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798 <http://dl.acm.org/citation.cfm?id=2790798> * *Blog:- **http://learnhardwithkunalgupta.blogspot.in <http://learnhardwithkunalgupta.blogspot.in> * On Mon, Oct 17, 2016 at 3:21 PM, Ben Davison <ben.davi...@7digital.com> wrote: > Here's an example from our server.properties > > log.segment.bytes=1073741824 > message.max.bytes=5242880 > num.partitions=3 > log.retention.bytes=5000000000000 > num.network.threads=6 > num.io.threads=16 > replica.fetch.max.bytes=6242880 > default.replication.factor=3 > zookeeper.connection.timeout.ms=60000 > auto.leader.rebalance.enable=true > delete.topic.enable=true > > > log.retention.bytes=5000000000000 is the setting, we have 5.5TB available, > so we start deleting at 5TB space used. > > > > > On Mon, Oct 17, 2016 at 10:37 AM, Kunal Gupta <kunal.gu...@cube26.com> > wrote: > > > I didn't get it ... > > > > Can you explain me in form of example or something for which you are > > feasible > > > > On Oct 17, 2016 2:08 PM, "Ben Davison" <ben.davi...@7digital.com> wrote: > > > > > We have it setup so that both log ms is set to 7 days and log delete > > > bytes(can't remember exactly what the setting is called. So we never > run > > > out of space (don't set the value to something like 99% of your disk, > as > > > the log cleaner thread might not kick in time, we leave it at 90% of > > disks > > > space) > > > > > > On Monday, 17 October 2016, Kunal Gupta <kunal.gu...@cube26.com> > wrote: > > > > > > > Please help me :( > > > > > > > > *Thanks, Kunal* > > > > *+91-9958189589* > > > > *Data Analyst* > > > > *First Paper Publication : **http://dl.acm.org/citation. > cfm?id=2790798 > > > > <http://dl.acm.org/citation.cfm?id=2790798> * > > > > *Blog:- **http://learnhardwithkunalgupta.blogspot.in > > > > <http://learnhardwithkunalgupta.blogspot.in> * > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Oct 16, 2016 at 11:23 AM, Kunal Gupta < > kunal.gu...@cube26.com > > > > <javascript:;>> > > > > wrote: > > > > > > > > > > > > > > In my organisation I have 3 machine cluster of Kafka and each topic > > > > > assigned two machine for storing there data. > > > > > > > > > > There is one topic for which I get lot of data from clients thats > > data > > > > > exceeds my disk space in one machine because that machine is a > leader > > > of > > > > > that topic, when I look into kafka-logs seeing all topic data there > > is > > > > only > > > > > one topic whose data has consumed lot of space. > > > > > > > > > > In one month I have 36 GB data from that topic. What I have thought > > so > > > > far > > > > > is kafka logs will be delete after 24 hours and new data will be > > > retained > > > > > for 24 hours and my consumer will consume all data with in 24 > hours, > > so > > > > > there will be no problem of disk space. > > > > > > > > > > I have enable everything in server.properties to clean kafka-logs > but > > > > > still data is there. Below data is my settings in server.properties > > > file. > > > > > > > > > > *log.retention.minutes=1440* > > > > > *log.retention.hours=24* > > > > > *log.retention.ms <http://log.retention.ms>=86400000* > > > > > *log.cleaner.delete.retention.ms > > > > > <http://log.cleaner.delete.retention.ms>=24* > > > > > *log.segment.bytes=1048576* > > > > > *log.retention.check.interval.ms > > > > > <http://log.retention.check.interval.ms>=3000* > > > > > *log.cleaner.enable=true* > > > > > *zookeeper.connection.timeout.ms > > > > > <http://zookeeper.connection.timeout.ms>=30000* > > > > > *delete.topic.enable = true* > > > > > *auto.create.topics.enable = true* > > > > > *default.replication.factor=2* > > > > > *auto.leader.rebalance.enable=true* > > > > > *controlled.shutdown.enable=true* > > > > > *controller.socket.timeout.ms <http://controller.socket.timeout.ms > > > > >=120000* > > > > > > > > > > Please help me in this so that my machine can able to handle large > > > number > > > > > of request and data well using 3 machine cluster. > > > > > > > > > > *Thanks, Kunal* > > > > > *+91-9958189589* > > > > > *Data Analyst* > > > > > *First Paper Publication : **http://dl.acm.org/citation. > > cfm?id=2790798 > > > > > <http://dl.acm.org/citation.cfm?id=2790798> * > > > > > *Blog:- **http://learnhardwithkunalgupta.blogspot.in > > > > > <http://learnhardwithkunalgupta.blogspot.in> * > > > > > > > > > > > > > > > -- > > > > > > > > > This email, including attachments, is private and confidential. If you > > have > > > received this email in error please notify the sender and delete it > from > > > your system. Emails are not secure and may contain viruses. No > liability > > > can be accepted for viruses that might be transferred by this email or > > any > > > attachment. Any unauthorised copying of this message or unauthorised > > > distribution and publication of the information contained herein are > > > prohibited. > > > > > > 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB. > > > Registered in England and Wales. Registered No. 04843573. > > > > > > > -- > > > This email, including attachments, is private and confidential. If you have > received this email in error please notify the sender and delete it from > your system. Emails are not secure and may contain viruses. No liability > can be accepted for viruses that might be transferred by this email or any > attachment. Any unauthorised copying of this message or unauthorised > distribution and publication of the information contained herein are > prohibited. > > 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB. > Registered in England and Wales. Registered No. 04843573. >
