Hi, Thanks Sachin for your great effort. 😃
The last settings you suggested actually did the job for me. I still do not understand why my previous settings did not work. However I learned a lot about log cleaning. Thank you so much for your quick replies and time. Cheers, Regards, Eranga Heshan *Undergraduate* Computer Science & Engineering University of Moratuwa Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> Email: era...@wso2.com <osanda.des...@gmail.com> <https://www.facebook.com/erangaheshan> <https://twitter.com/erangaheshan> <https://www.linkedin.com/in/erangaheshan> On Wed, Nov 23, 2016 at 2:47 PM, Sachin Mittal <sjmit...@gmail.com> wrote: > Maybe because of these settings > log.retention.check.interval.ms=10000 > log.retention.ms=10000 > > Try setting check interval at 1000 ms. I just feel these values are too > low. > > Try setting retention to be 1 hr and keep check interval to default which > is 300 seconds. > > Also just check the permission of that dir or file. Maybe create a new logs > dir within your workspace where you have complete access. > > > On Wed, Nov 23, 2016 at 12:37 PM, Eranga Heshan <eranga....@gmail.com> > wrote: > > > The reason I set retention time to 10 seconds is because I run kafka for > > about 4 minutes each time. What I am running is a microbenchmark and I > take > > throughput and latency values for 4mins of runtime. > > > > However when I run I get these printed on terminal. > > > > [2016-11-23 12:21:54,517] INFO Deleting index > > /tmp/kafka-logs-3/incrementStream1-1/00000000000035778556.index.deleted > > (kafka.log.OffsetIndex) > > [2016-11-23 12:21:54,516] INFO Deleting index > > /tmp/kafka-logs-3/incrementStream1-1/00000000000035550774.index.deleted > > (kafka.log.OffsetIndex) > > [2016-11-23 12:21:54,517] INFO Deleting index > > /tmp/kafka-logs-3/incrementStream1-1/00000000000036690192.index.deleted > > (kafka.log.OffsetIndex) > > [2016-11-23 12:21:54,517] INFO Deleting index > > /tmp/kafka-logs-3/incrementStream1-1/00000000000036918223.index.deleted > > (kafka.log.OffsetIndex) > > [2016-11-23 12:21:54,582] INFO Rolled new log segment for > > 'incrementStream1-1' in 0 ms. (kafka.log.Log) > > [2016-11-23 12:21:54,797] INFO Rolled new log segment for > > 'incrementStream3-0' in 1 ms. (kafka.log.Log) > > [2016-11-23 12:21:55,493] INFO Rolled new log segment for > > 'incrementStream1-1' in 1 ms. (kafka.log.Log) > > [2016-11-23 12:21:55,787] INFO Rolled new log segment for > > 'incrementStream3-0' in 0 ms. (kafka.log.Log) > > [2016-11-23 12:21:56,439] INFO Rolled new log segment for > > 'incrementStream1-1' in 0 ms. (kafka.log.Log) > > > > When I go to kafka-logs directory I can see the segments of 10MB are > > created with an index file. But after the retention time what happens is > > that the log files change their names from "00000000000009117624.log" to > > "00000000000009117624.log.deleted". And the index files change their > names > > from "00000000000009117624.index" to "00000000000009117624.index. > deleted". > > Nothing gets removed from the directory. > > > > Is there any particular reason for this to happen? > > > > Thanks, > > Regards, > > > > > > Eranga Heshan > > *Undergraduate* > > Computer Science & Engineering > > University of Moratuwa > > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > Email: era...@wso2.com <osanda.des...@gmail.com> > > <https://www.facebook.com/erangaheshan> <https://twitter.com/ > > erangaheshan> > > <https://www.linkedin.com/in/erangaheshan> > > > > On Wed, Nov 23, 2016 at 11:50 AM, Sachin Mittal <sjmit...@gmail.com> > > wrote: > > > > > try bin/kafka-server-start.sh ./config/server.properties > > > > > > Also check how many segments have got created in logs dir and size of > > each. > > > May be set the retention to 1 hour or 15 minutes. 10 sec seems very > less. > > > > > > > > > > > > On Wed, Nov 23, 2016 at 11:21 AM, Eranga Heshan <eranga....@gmail.com> > > > wrote: > > > > > > > As you suggested I did these changes to the server.properties file on > > all > > > > nodes. > > > > > > > > log.retention.check.interval.ms=10000 > > > > log.retention.ms=10000 > > > > log.segment.bytes=10490000 > > > > > > > > How I start each server is by, > > > > > > > > /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/bin/kafka- > > > server-start.sh > > > > -daemon > > > > /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/config/ > > > server.properties > > > > > > > > However, I still get about 4GB log files. Did I miss anything? > > > > > > > > Thanks, > > > > Regards, > > > > > > > > > > > > Eranga Heshan > > > > *Undergraduate* > > > > Computer Science & Engineering > > > > University of Moratuwa > > > > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > > > Email: era...@wso2.com <osanda.des...@gmail.com> > > > > <https://www.facebook.com/erangaheshan> <https://twitter.com/ > > > > erangaheshan> > > > > <https://www.linkedin.com/in/erangaheshan> > > > > > > > > On Wed, Nov 23, 2016 at 10:14 AM, Sachin Mittal <sjmit...@gmail.com> > > > > wrote: > > > > > > > > > Check these in server.properties > > > > > ############################# Log Retention Policy > > > > > ############################# > > > > > > > > > > # The following configurations control the disposal of log > segments. > > > The > > > > > policy can > > > > > # be set to delete segments after a period of time, or after a > given > > > size > > > > > has accumulated. > > > > > # A segment will be deleted whenever *either* of these criteria are > > > met. > > > > > Deletion always happens > > > > > # from the end of the log. > > > > > > > > > > # The minimum age of a log file to be eligible for deletion > > > > > log.retention.hours=168 > > > > > > > > > > # A size-based retention policy for logs. Segments are pruned from > > the > > > > log > > > > > as long as the remaining > > > > > # segments don't drop below log.retention.bytes. > > > > > #log.retention.bytes=1073741824 > > > > > > > > > > # The maximum size of a log segment file. When this size is > reached a > > > new > > > > > log segment will be created. > > > > > log.segment.bytes=1073741824 > > > > > > > > > > Try by setting hours as 1 and set segment bytes as say 10 MB. You > > > should > > > > be > > > > > able to see multiple segments created of size 20 MB under you kafka > > > logs > > > > > dir > > > > > say: tmp\kafka-logs\<topic>\*.log > > > > > > > > > > As the files age > 1 hour it will be marked for deletion and be > > deleted > > > > > next time delete thread runs. By default the delete thread check > is 5 > > > > > minutes. > > > > > > > > > > Also make sure when you start server you specify this property as > the > > > > > argument. > > > > > > > > > > bin\windows\kafka-server-start.bat config\server.properties > > > > > > > > > > > > > > > > > > > > On Wed, Nov 23, 2016 at 9:50 AM, Eranga Heshan < > eranga....@gmail.com > > > > > > > > wrote: > > > > > > > > > > > This is the exact parameter I added to server.config file on all > > the > > > > > nodes. > > > > > > > > > > > > log.retention.ms=1000 > > > > > > > > > > > > Thanks, > > > > > > Regards, > > > > > > > > > > > > > > > > > > Eranga Heshan > > > > > > *Undergraduate* > > > > > > Computer Science & Engineering > > > > > > University of Moratuwa > > > > > > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > > > > > Email: era...@wso2.com <osanda.des...@gmail.com> > > > > > > <https://www.facebook.com/erangaheshan> <https://twitter.com/ > > > > > > erangaheshan> > > > > > > <https://www.linkedin.com/in/erangaheshan> > > > > > > > > > > > > On Wed, Nov 23, 2016 at 9:43 AM, Eranga Heshan < > > eranga....@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > I have 4 node cluster and node1 has zookeeper running. I added > > > those > > > > > > > parameters on each of my server.properties files in kafka > nodes. > > > > > > > > > > > > > > > > > > > > > Eranga Heshan > > > > > > > *Undergraduate* > > > > > > > Computer Science & Engineering > > > > > > > University of Moratuwa > > > > > > > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > > > > > > Email: era...@wso2.com <osanda.des...@gmail.com> > > > > > > > <https://www.facebook.com/erangaheshan> > > > > > > > <https://twitter.com/erangaheshan> > > > > > > > <https://www.linkedin.com/in/erangaheshan> > > > > > > > > > > > > > > On Wed, Nov 23, 2016 at 9:36 AM, Sachin Mittal < > > sjmit...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > >> Kafka logs are not kept based on data consumption. They are > > > deleted > > > > > > after > > > > > > >> it reaches a particular size and retention time. > > > > > > >> Even if they are consumed multiple times in that duration, > they > > > are > > > > > > still > > > > > > >> kept. > > > > > > >> > > > > > > >> Where have you added those params and how are you starting > your > > > > > > zookeeper > > > > > > >> and server. > > > > > > >> > > > > > > >> > > > > > > >> On Wed, Nov 23, 2016 at 8:00 AM, Eranga Heshan < > > > > eranga....@gmail.com> > > > > > > >> wrote: > > > > > > >> > > > > > > >> > Sorry for repeatedly asking the same question, but although > I > > > > tried > > > > > > all > > > > > > >> > those parameters; my log file still at large about 2.5GB. Is > > it > > > > due > > > > > to > > > > > > >> the > > > > > > >> > data inside my topic is flushed to disk? I meant that there > > > might > > > > be > > > > > > >> data > > > > > > >> > inside topic which is not consumed yet or not been deleted > > even > > > > > after > > > > > > >> > consumed. > > > > > > >> > > > > > > > >> > Thanks, > > > > > > >> > Regards, > > > > > > >> > > > > > > > >> > > > > > > > >> > Eranga Heshan > > > > > > >> > *Undergraduate* > > > > > > >> > Computer Science & Engineering > > > > > > >> > University of Moratuwa > > > > > > >> > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > > > > > >> > Email: era...@wso2.com <osanda.des...@gmail.com> > > > > > > >> > <https://www.facebook.com/erangaheshan> < > > https://twitter.com/ > > > > > > >> > erangaheshan> > > > > > > >> > <https://www.linkedin.com/in/erangaheshan> > > > > > > >> > > > > > > > >> > On Tue, Nov 22, 2016 at 2:15 PM, Sachin Mittal < > > > > sjmit...@gmail.com> > > > > > > >> wrote: > > > > > > >> > > > > > > > >> > > Set some value to log.retention.ms which is ms time to > > retain > > > > log > > > > > > and > > > > > > >> > also > > > > > > >> > > log.segment.bytes which is size of a single log file. > > > > > > >> > > delete.topic.enable is needed if you want to delete the > > topic > > > > > > itself. > > > > > > >> > > > > > > > > >> > > Thanks > > > > > > >> > > Sachin > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > On Tue, Nov 22, 2016 at 1:52 PM, Eranga Heshan < > > > > > > eranga....@gmail.com> > > > > > > >> > > wrote: > > > > > > >> > > > > > > > > >> > > > I tried log.retention.bytes=1049000 which is about 10MB. > > But > > > > it > > > > > > did > > > > > > >> not > > > > > > >> > > > work. Is it because I had not given any value to > > > > > > >> delete.topic.enable? > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > Eranga Heshan > > > > > > >> > > > *Undergraduate* > > > > > > >> > > > Computer Science & Engineering > > > > > > >> > > > University of Moratuwa > > > > > > >> > > > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > > > > > >> > > > Email: era...@wso2.com <osanda.des...@gmail.com> > > > > > > >> > > > <https://www.facebook.com/erangaheshan> < > > > > https://twitter.com/ > > > > > > >> > > > erangaheshan> > > > > > > >> > > > <https://www.linkedin.com/in/erangaheshan> > > > > > > >> > > > > > > > > > >> > > > On Tue, Nov 22, 2016 at 1:47 PM, Sachin Mittal < > > > > > > sjmit...@gmail.com> > > > > > > >> > > wrote: > > > > > > >> > > > > > > > > > >> > > > > Check > > > > > > >> > > > > http://kafka.apache.org/documentation.html# > > brokerconfigs > > > > > > >> > > > > log.retention.bytes > > > > > > >> > > > > log.retention.ms > > > > > > >> > > > > log.segment.bytes > > > > > > >> > > > > delete.topic.enable > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > On Tue, Nov 22, 2016 at 1:36 PM, Eranga Heshan < > > > > > > >> eranga....@gmail.com > > > > > > >> > > > > > > > > >> > > > > wrote: > > > > > > >> > > > > > > > > > > >> > > > > > Hi all, > > > > > > >> > > > > > > > > > > > >> > > > > > I want to keep the size of kafka-logs reduced as > much > > as > > > > > > >> possible. > > > > > > >> > To > > > > > > >> > > > do > > > > > > >> > > > > it > > > > > > >> > > > > > I need to set it to delete its unwanted logs > > frequently. > > > > Is > > > > > > >> there a > > > > > > >> > > way > > > > > > >> > > > > to > > > > > > >> > > > > > do it? > > > > > > >> > > > > > > > > > > > >> > > > > > Thanks, > > > > > > >> > > > > > Regards, > > > > > > >> > > > > > > > > > > > >> > > > > > Eranga Heshan > > > > > > >> > > > > > *Undergraduate* > > > > > > >> > > > > > Computer Science & Engineering > > > > > > >> > > > > > University of Moratuwa > > > > > > >> > > > > > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > > > > > >> > > > > > Email: era...@wso2.com <osanda.des...@gmail.com> > > > > > > >> > > > > > <https://www.facebook.com/erangaheshan> < > > > > > > https://twitter.com/ > > > > > > >> > > > > > erangaheshan> > > > > > > >> > > > > > <https://www.linkedin.com/in/erangaheshan> > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
