Check these in server.properties ############################# Log Retention Policy #############################
# The following configurations control the disposal of log segments. The policy can # be set to delete segments after a period of time, or after a given size has accumulated. # A segment will be deleted whenever *either* of these criteria are met. Deletion always happens # from the end of the log. # The minimum age of a log file to be eligible for deletion log.retention.hours=168 # A size-based retention policy for logs. Segments are pruned from the log as long as the remaining # segments don't drop below log.retention.bytes. #log.retention.bytes=1073741824 # The maximum size of a log segment file. When this size is reached a new log segment will be created. log.segment.bytes=1073741824 Try by setting hours as 1 and set segment bytes as say 10 MB. You should be able to see multiple segments created of size 20 MB under you kafka logs dir say: tmp\kafka-logs\<topic>\*.log As the files age > 1 hour it will be marked for deletion and be deleted next time delete thread runs. By default the delete thread check is 5 minutes. Also make sure when you start server you specify this property as the argument. bin\windows\kafka-server-start.bat config\server.properties On Wed, Nov 23, 2016 at 9:50 AM, Eranga Heshan <eranga....@gmail.com> wrote: > This is the exact parameter I added to server.config file on all the nodes. > > log.retention.ms=1000 > > Thanks, > Regards, > > > Eranga Heshan > *Undergraduate* > Computer Science & Engineering > University of Moratuwa > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > Email: era...@wso2.com <osanda.des...@gmail.com> > <https://www.facebook.com/erangaheshan> <https://twitter.com/ > erangaheshan> > <https://www.linkedin.com/in/erangaheshan> > > On Wed, Nov 23, 2016 at 9:43 AM, Eranga Heshan <eranga....@gmail.com> > wrote: > > > I have 4 node cluster and node1 has zookeeper running. I added those > > parameters on each of my server.properties files in kafka nodes. > > > > > > Eranga Heshan > > *Undergraduate* > > Computer Science & Engineering > > University of Moratuwa > > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > > Email: era...@wso2.com <osanda.des...@gmail.com> > > <https://www.facebook.com/erangaheshan> > > <https://twitter.com/erangaheshan> > > <https://www.linkedin.com/in/erangaheshan> > > > > On Wed, Nov 23, 2016 at 9:36 AM, Sachin Mittal <sjmit...@gmail.com> > wrote: > > > >> Kafka logs are not kept based on data consumption. They are deleted > after > >> it reaches a particular size and retention time. > >> Even if they are consumed multiple times in that duration, they are > still > >> kept. > >> > >> Where have you added those params and how are you starting your > zookeeper > >> and server. > >> > >> > >> On Wed, Nov 23, 2016 at 8:00 AM, Eranga Heshan <eranga....@gmail.com> > >> wrote: > >> > >> > Sorry for repeatedly asking the same question, but although I tried > all > >> > those parameters; my log file still at large about 2.5GB. Is it due to > >> the > >> > data inside my topic is flushed to disk? I meant that there might be > >> data > >> > inside topic which is not consumed yet or not been deleted even after > >> > consumed. > >> > > >> > Thanks, > >> > Regards, > >> > > >> > > >> > Eranga Heshan > >> > *Undergraduate* > >> > Computer Science & Engineering > >> > University of Moratuwa > >> > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > >> > Email: era...@wso2.com <osanda.des...@gmail.com> > >> > <https://www.facebook.com/erangaheshan> <https://twitter.com/ > >> > erangaheshan> > >> > <https://www.linkedin.com/in/erangaheshan> > >> > > >> > On Tue, Nov 22, 2016 at 2:15 PM, Sachin Mittal <sjmit...@gmail.com> > >> wrote: > >> > > >> > > Set some value to log.retention.ms which is ms time to retain log > and > >> > also > >> > > log.segment.bytes which is size of a single log file. > >> > > delete.topic.enable is needed if you want to delete the topic > itself. > >> > > > >> > > Thanks > >> > > Sachin > >> > > > >> > > > >> > > On Tue, Nov 22, 2016 at 1:52 PM, Eranga Heshan < > eranga....@gmail.com> > >> > > wrote: > >> > > > >> > > > I tried log.retention.bytes=1049000 which is about 10MB. But it > did > >> not > >> > > > work. Is it because I had not given any value to > >> delete.topic.enable? > >> > > > > >> > > > > >> > > > Eranga Heshan > >> > > > *Undergraduate* > >> > > > Computer Science & Engineering > >> > > > University of Moratuwa > >> > > > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > >> > > > Email: era...@wso2.com <osanda.des...@gmail.com> > >> > > > <https://www.facebook.com/erangaheshan> <https://twitter.com/ > >> > > > erangaheshan> > >> > > > <https://www.linkedin.com/in/erangaheshan> > >> > > > > >> > > > On Tue, Nov 22, 2016 at 1:47 PM, Sachin Mittal < > sjmit...@gmail.com> > >> > > wrote: > >> > > > > >> > > > > Check > >> > > > > http://kafka.apache.org/documentation.html#brokerconfigs > >> > > > > log.retention.bytes > >> > > > > log.retention.ms > >> > > > > log.segment.bytes > >> > > > > delete.topic.enable > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > On Tue, Nov 22, 2016 at 1:36 PM, Eranga Heshan < > >> eranga....@gmail.com > >> > > > >> > > > > wrote: > >> > > > > > >> > > > > > Hi all, > >> > > > > > > >> > > > > > I want to keep the size of kafka-logs reduced as much as > >> possible. > >> > To > >> > > > do > >> > > > > it > >> > > > > > I need to set it to delete its unwanted logs frequently. Is > >> there a > >> > > way > >> > > > > to > >> > > > > > do it? > >> > > > > > > >> > > > > > Thanks, > >> > > > > > Regards, > >> > > > > > > >> > > > > > Eranga Heshan > >> > > > > > *Undergraduate* > >> > > > > > Computer Science & Engineering > >> > > > > > University of Moratuwa > >> > > > > > Mobile: +94 71 138 2686 <%2B94%2071%20552%202087> > >> > > > > > Email: era...@wso2.com <osanda.des...@gmail.com> > >> > > > > > <https://www.facebook.com/erangaheshan> < > https://twitter.com/ > >> > > > > > erangaheshan> > >> > > > > > <https://www.linkedin.com/in/erangaheshan> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > >
