Re: Delete "kafka-logs"

2016-11-24 Thread Eranga Heshan 
Hi,

Thanks Sachin for your great effort. 😃

The last settings you suggested actually did the job for me. I still do not
understand why my previous settings did not work. However I learned a lot
about log cleaning.

Thank you so much for your quick replies and time.

Cheers,
Regards,


Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
Email: era...@wso2.com 
   
   

On Wed, Nov 23, 2016 at 2:47 PM, Sachin Mittal  wrote:

> Maybe because of these settings
> log.retention.check.interval.ms=1
> log.retention.ms=1
>
> Try setting check interval at 1000 ms. I just feel these values are too
> low.
>
> Try setting retention to be 1 hr and keep check interval to default which
> is 300 seconds.
>
> Also just check the permission of that dir or file. Maybe create a new logs
> dir within your workspace where you have complete access.
>
>
> On Wed, Nov 23, 2016 at 12:37 PM, Eranga Heshan 
> wrote:
>
> > The reason I set retention time to 10 seconds is because I run kafka for
> > about 4 minutes each time. What I am running is a microbenchmark and I
> take
> > throughput and latency values for 4mins of runtime.
> >
> > However when I run I get these printed on terminal.
> >
> > [2016-11-23 12:21:54,517] INFO Deleting index
> > /tmp/kafka-logs-3/incrementStream1-1/35778556.index.deleted
> > (kafka.log.OffsetIndex)
> > [2016-11-23 12:21:54,516] INFO Deleting index
> > /tmp/kafka-logs-3/incrementStream1-1/35550774.index.deleted
> > (kafka.log.OffsetIndex)
> > [2016-11-23 12:21:54,517] INFO Deleting index
> > /tmp/kafka-logs-3/incrementStream1-1/36690192.index.deleted
> > (kafka.log.OffsetIndex)
> > [2016-11-23 12:21:54,517] INFO Deleting index
> > /tmp/kafka-logs-3/incrementStream1-1/36918223.index.deleted
> > (kafka.log.OffsetIndex)
> > [2016-11-23 12:21:54,582] INFO Rolled new log segment for
> > 'incrementStream1-1' in 0 ms. (kafka.log.Log)
> > [2016-11-23 12:21:54,797] INFO Rolled new log segment for
> > 'incrementStream3-0' in 1 ms. (kafka.log.Log)
> > [2016-11-23 12:21:55,493] INFO Rolled new log segment for
> > 'incrementStream1-1' in 1 ms. (kafka.log.Log)
> > [2016-11-23 12:21:55,787] INFO Rolled new log segment for
> > 'incrementStream3-0' in 0 ms. (kafka.log.Log)
> > [2016-11-23 12:21:56,439] INFO Rolled new log segment for
> > 'incrementStream1-1' in 0 ms. (kafka.log.Log)
> >
> > When I go to kafka-logs directory I can see the segments of 10MB are
> > created with an index file. But after the retention time what happens is
> > that the log files change their names from "09117624.log" to
> > "09117624.log.deleted". And the index files change their
> names
> > from "09117624.index" to "09117624.index.
> deleted".
> > Nothing gets removed from the directory.
> >
> > Is there any particular reason for this to happen?
> >
> > Thanks,
> > Regards,
> >
> >
> > Eranga Heshan
> > *Undergraduate*
> > Computer Science & Engineering
> > University of Moratuwa
> > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > Email: era...@wso2.com 
> >     > erangaheshan>
> >
> >
> > On Wed, Nov 23, 2016 at 11:50 AM, Sachin Mittal 
> > wrote:
> >
> > > try bin/kafka-server-start.sh ./config/server.properties
> > >
> > > Also check how many segments have got created in logs dir and size of
> > each.
> > > May be set the retention to 1 hour or 15 minutes. 10 sec seems very
> less.
> > >
> > >
> > >
> > > On Wed, Nov 23, 2016 at 11:21 AM, Eranga Heshan 
> > > wrote:
> > >
> > > > As you suggested I did these changes to the server.properties file on
> > all
> > > > nodes.
> > > >
> > > > log.retention.check.interval.ms=1
> > > > log.retention.ms=1
> > > > log.segment.bytes=1049
> > > >
> > > > How I start each server is by,
> > > >
> > > > /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/bin/kafka-
> > > server-start.sh
> > > > -daemon
> > > > /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/config/
> > > server.properties
> > > >
> > > > However, I still get about 4GB log files. Did I miss anything?
> > > >
> > > > Thanks,
> > > > Regards,
> > > >
> > > >
> > > > Eranga Heshan
> > > > *Undergraduate*
> > > > Computer Science & Engineering
> > > > University of Moratuwa
> > > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > > Email: era...@wso2.com 
> > > >     > > > erangaheshan>
> > > >
> > > >
> > > > On Wed, Nov 23, 2016 at 10:14 AM, Sachin Mittal 
> > > > wrote:
> > > >
> > > > > Check these in server.properties
> > > > > # Log Retention Policy
> > > > > ###

Re: Delete "kafka-logs"

2016-11-23 Thread Sachin Mittal 
Maybe because of these settings
log.retention.check.interval.ms=1
log.retention.ms=1

Try setting check interval at 1000 ms. I just feel these values are too low.

Try setting retention to be 1 hr and keep check interval to default which
is 300 seconds.

Also just check the permission of that dir or file. Maybe create a new logs
dir within your workspace where you have complete access.


On Wed, Nov 23, 2016 at 12:37 PM, Eranga Heshan 
wrote:

> The reason I set retention time to 10 seconds is because I run kafka for
> about 4 minutes each time. What I am running is a microbenchmark and I take
> throughput and latency values for 4mins of runtime.
>
> However when I run I get these printed on terminal.
>
> [2016-11-23 12:21:54,517] INFO Deleting index
> /tmp/kafka-logs-3/incrementStream1-1/35778556.index.deleted
> (kafka.log.OffsetIndex)
> [2016-11-23 12:21:54,516] INFO Deleting index
> /tmp/kafka-logs-3/incrementStream1-1/35550774.index.deleted
> (kafka.log.OffsetIndex)
> [2016-11-23 12:21:54,517] INFO Deleting index
> /tmp/kafka-logs-3/incrementStream1-1/36690192.index.deleted
> (kafka.log.OffsetIndex)
> [2016-11-23 12:21:54,517] INFO Deleting index
> /tmp/kafka-logs-3/incrementStream1-1/36918223.index.deleted
> (kafka.log.OffsetIndex)
> [2016-11-23 12:21:54,582] INFO Rolled new log segment for
> 'incrementStream1-1' in 0 ms. (kafka.log.Log)
> [2016-11-23 12:21:54,797] INFO Rolled new log segment for
> 'incrementStream3-0' in 1 ms. (kafka.log.Log)
> [2016-11-23 12:21:55,493] INFO Rolled new log segment for
> 'incrementStream1-1' in 1 ms. (kafka.log.Log)
> [2016-11-23 12:21:55,787] INFO Rolled new log segment for
> 'incrementStream3-0' in 0 ms. (kafka.log.Log)
> [2016-11-23 12:21:56,439] INFO Rolled new log segment for
> 'incrementStream1-1' in 0 ms. (kafka.log.Log)
>
> When I go to kafka-logs directory I can see the segments of 10MB are
> created with an index file. But after the retention time what happens is
> that the log files change their names from "09117624.log" to
> "09117624.log.deleted". And the index files change their names
> from "09117624.index" to "09117624.index.deleted".
> Nothing gets removed from the directory.
>
> Is there any particular reason for this to happen?
>
> Thanks,
> Regards,
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: era...@wso2.com 
>     erangaheshan>
>
>
> On Wed, Nov 23, 2016 at 11:50 AM, Sachin Mittal 
> wrote:
>
> > try bin/kafka-server-start.sh ./config/server.properties
> >
> > Also check how many segments have got created in logs dir and size of
> each.
> > May be set the retention to 1 hour or 15 minutes. 10 sec seems very less.
> >
> >
> >
> > On Wed, Nov 23, 2016 at 11:21 AM, Eranga Heshan 
> > wrote:
> >
> > > As you suggested I did these changes to the server.properties file on
> all
> > > nodes.
> > >
> > > log.retention.check.interval.ms=1
> > > log.retention.ms=1
> > > log.segment.bytes=1049
> > >
> > > How I start each server is by,
> > >
> > > /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/bin/kafka-
> > server-start.sh
> > > -daemon
> > > /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/config/
> > server.properties
> > >
> > > However, I still get about 4GB log files. Did I miss anything?
> > >
> > > Thanks,
> > > Regards,
> > >
> > >
> > > Eranga Heshan
> > > *Undergraduate*
> > > Computer Science & Engineering
> > > University of Moratuwa
> > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > Email: era...@wso2.com 
> > >     > > erangaheshan>
> > >
> > >
> > > On Wed, Nov 23, 2016 at 10:14 AM, Sachin Mittal 
> > > wrote:
> > >
> > > > Check these in server.properties
> > > > # Log Retention Policy
> > > > #
> > > >
> > > > # The following configurations control the disposal of log segments.
> > The
> > > > policy can
> > > > # be set to delete segments after a period of time, or after a given
> > size
> > > > has accumulated.
> > > > # A segment will be deleted whenever *either* of these criteria are
> > met.
> > > > Deletion always happens
> > > > # from the end of the log.
> > > >
> > > > # The minimum age of a log file to be eligible for deletion
> > > > log.retention.hours=168
> > > >
> > > > # A size-based retention policy for logs. Segments are pruned from
> the
> > > log
> > > > as long as the remaining
> > > > # segments don't drop below log.retention.bytes.
> > > > #log.retention.bytes=1073741824
> > > >
> > > > # The maximum size of a log segment file. When this size is reached a
> > new
> > > > log segment will be created.
> > > > log.s

Re: Delete "kafka-logs"

2016-11-22 Thread Eranga Heshan 
The reason I set retention time to 10 seconds is because I run kafka for
about 4 minutes each time. What I am running is a microbenchmark and I take
throughput and latency values for 4mins of runtime.

However when I run I get these printed on terminal.

[2016-11-23 12:21:54,517] INFO Deleting index
/tmp/kafka-logs-3/incrementStream1-1/35778556.index.deleted
(kafka.log.OffsetIndex)
[2016-11-23 12:21:54,516] INFO Deleting index
/tmp/kafka-logs-3/incrementStream1-1/35550774.index.deleted
(kafka.log.OffsetIndex)
[2016-11-23 12:21:54,517] INFO Deleting index
/tmp/kafka-logs-3/incrementStream1-1/36690192.index.deleted
(kafka.log.OffsetIndex)
[2016-11-23 12:21:54,517] INFO Deleting index
/tmp/kafka-logs-3/incrementStream1-1/36918223.index.deleted
(kafka.log.OffsetIndex)
[2016-11-23 12:21:54,582] INFO Rolled new log segment for
'incrementStream1-1' in 0 ms. (kafka.log.Log)
[2016-11-23 12:21:54,797] INFO Rolled new log segment for
'incrementStream3-0' in 1 ms. (kafka.log.Log)
[2016-11-23 12:21:55,493] INFO Rolled new log segment for
'incrementStream1-1' in 1 ms. (kafka.log.Log)
[2016-11-23 12:21:55,787] INFO Rolled new log segment for
'incrementStream3-0' in 0 ms. (kafka.log.Log)
[2016-11-23 12:21:56,439] INFO Rolled new log segment for
'incrementStream1-1' in 0 ms. (kafka.log.Log)

When I go to kafka-logs directory I can see the segments of 10MB are
created with an index file. But after the retention time what happens is
that the log files change their names from "09117624.log" to
"09117624.log.deleted". And the index files change their names
from "09117624.index" to "09117624.index.deleted".
Nothing gets removed from the directory.

Is there any particular reason for this to happen?

Thanks,
Regards,


Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
Email: era...@wso2.com 
   
   

On Wed, Nov 23, 2016 at 11:50 AM, Sachin Mittal  wrote:

> try bin/kafka-server-start.sh ./config/server.properties
>
> Also check how many segments have got created in logs dir and size of each.
> May be set the retention to 1 hour or 15 minutes. 10 sec seems very less.
>
>
>
> On Wed, Nov 23, 2016 at 11:21 AM, Eranga Heshan 
> wrote:
>
> > As you suggested I did these changes to the server.properties file on all
> > nodes.
> >
> > log.retention.check.interval.ms=1
> > log.retention.ms=1
> > log.segment.bytes=1049
> >
> > How I start each server is by,
> >
> > /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/bin/kafka-
> server-start.sh
> > -daemon
> > /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/config/
> server.properties
> >
> > However, I still get about 4GB log files. Did I miss anything?
> >
> > Thanks,
> > Regards,
> >
> >
> > Eranga Heshan
> > *Undergraduate*
> > Computer Science & Engineering
> > University of Moratuwa
> > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > Email: era...@wso2.com 
> >     > erangaheshan>
> >
> >
> > On Wed, Nov 23, 2016 at 10:14 AM, Sachin Mittal 
> > wrote:
> >
> > > Check these in server.properties
> > > # Log Retention Policy
> > > #
> > >
> > > # The following configurations control the disposal of log segments.
> The
> > > policy can
> > > # be set to delete segments after a period of time, or after a given
> size
> > > has accumulated.
> > > # A segment will be deleted whenever *either* of these criteria are
> met.
> > > Deletion always happens
> > > # from the end of the log.
> > >
> > > # The minimum age of a log file to be eligible for deletion
> > > log.retention.hours=168
> > >
> > > # A size-based retention policy for logs. Segments are pruned from the
> > log
> > > as long as the remaining
> > > # segments don't drop below log.retention.bytes.
> > > #log.retention.bytes=1073741824
> > >
> > > # The maximum size of a log segment file. When this size is reached a
> new
> > > log segment will be created.
> > > log.segment.bytes=1073741824
> > >
> > > Try by setting hours as 1 and set segment bytes as say 10 MB. You
> should
> > be
> > > able to see multiple segments created of size 20 MB under you kafka
> logs
> > > dir
> > > say: tmp\kafka-logs\\*.log
> > >
> > > As the files age > 1 hour it will be marked for deletion and be deleted
> > > next time delete thread runs. By default the delete thread check is 5
> > > minutes.
> > >
> > > Also make sure when you start server you specify this property as the
> > > argument.
> > >
> > > bin\windows\kafka-server-start.bat config\server.properties
> > >
> > >
> > >
> > > On Wed, Nov 23, 2016 at 9:50 AM, Eranga Heshan 
> > > wrote:
> > >
> > > > This is the exact para

Re: Delete "kafka-logs"

2016-11-22 Thread Sachin Mittal 
try bin/kafka-server-start.sh ./config/server.properties

Also check how many segments have got created in logs dir and size of each.
May be set the retention to 1 hour or 15 minutes. 10 sec seems very less.



On Wed, Nov 23, 2016 at 11:21 AM, Eranga Heshan 
wrote:

> As you suggested I did these changes to the server.properties file on all
> nodes.
>
> log.retention.check.interval.ms=1
> log.retention.ms=1
> log.segment.bytes=1049
>
> How I start each server is by,
>
> /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/bin/kafka-server-start.sh
> -daemon
> /home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/config/server.properties
>
> However, I still get about 4GB log files. Did I miss anything?
>
> Thanks,
> Regards,
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: era...@wso2.com 
>     erangaheshan>
>
>
> On Wed, Nov 23, 2016 at 10:14 AM, Sachin Mittal 
> wrote:
>
> > Check these in server.properties
> > # Log Retention Policy
> > #
> >
> > # The following configurations control the disposal of log segments. The
> > policy can
> > # be set to delete segments after a period of time, or after a given size
> > has accumulated.
> > # A segment will be deleted whenever *either* of these criteria are met.
> > Deletion always happens
> > # from the end of the log.
> >
> > # The minimum age of a log file to be eligible for deletion
> > log.retention.hours=168
> >
> > # A size-based retention policy for logs. Segments are pruned from the
> log
> > as long as the remaining
> > # segments don't drop below log.retention.bytes.
> > #log.retention.bytes=1073741824
> >
> > # The maximum size of a log segment file. When this size is reached a new
> > log segment will be created.
> > log.segment.bytes=1073741824
> >
> > Try by setting hours as 1 and set segment bytes as say 10 MB. You should
> be
> > able to see multiple segments created of size 20 MB under you kafka logs
> > dir
> > say: tmp\kafka-logs\\*.log
> >
> > As the files age > 1 hour it will be marked for deletion and be deleted
> > next time delete thread runs. By default the delete thread check is 5
> > minutes.
> >
> > Also make sure when you start server you specify this property as the
> > argument.
> >
> > bin\windows\kafka-server-start.bat config\server.properties
> >
> >
> >
> > On Wed, Nov 23, 2016 at 9:50 AM, Eranga Heshan 
> > wrote:
> >
> > > This is the exact parameter I added to server.config file on all the
> > nodes.
> > >
> > > log.retention.ms=1000
> > >
> > > Thanks,
> > > Regards,
> > >
> > >
> > > Eranga Heshan
> > > *Undergraduate*
> > > Computer Science & Engineering
> > > University of Moratuwa
> > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > Email: era...@wso2.com 
> > >     > > erangaheshan>
> > >
> > >
> > > On Wed, Nov 23, 2016 at 9:43 AM, Eranga Heshan 
> > > wrote:
> > >
> > > > I have 4 node cluster and node1 has zookeeper running. I added those
> > > > parameters on each of my server.properties files in kafka nodes.
> > > >
> > > >
> > > > Eranga Heshan
> > > > *Undergraduate*
> > > > Computer Science & Engineering
> > > > University of Moratuwa
> > > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > > Email: era...@wso2.com 
> > > > 
> > > > 
> > > > 
> > > >
> > > > On Wed, Nov 23, 2016 at 9:36 AM, Sachin Mittal 
> > > wrote:
> > > >
> > > >> Kafka logs are not kept based on data consumption. They are deleted
> > > after
> > > >> it reaches a particular size and retention time.
> > > >> Even if they are consumed multiple times in that duration, they are
> > > still
> > > >> kept.
> > > >>
> > > >> Where have you added those params and how are you starting your
> > > zookeeper
> > > >> and server.
> > > >>
> > > >>
> > > >> On Wed, Nov 23, 2016 at 8:00 AM, Eranga Heshan <
> eranga@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Sorry for repeatedly asking the same question, but although I
> tried
> > > all
> > > >> > those parameters; my log file still at large about 2.5GB. Is it
> due
> > to
> > > >> the
> > > >> > data inside my topic is flushed to disk? I meant that there might
> be
> > > >> data
> > > >> > inside topic which is not consumed yet or not been deleted even
> > after
> > > >> > consumed.
> > > >> >
> > > >> > Thanks,
> > > >> > Regards,
> > > >> >
> > > >> >
> > > >> > Eranga Heshan
> > > >> > *Undergraduate*
> > > >> > Computer Science & Engineering
> > > >> > University of Moratuwa
> > > >> > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > >> > Email: era...@wso2.com 
> > > >> >

Re: Delete "kafka-logs"

2016-11-22 Thread Eranga Heshan 
As you suggested I did these changes to the server.properties file on all
nodes.

log.retention.check.interval.ms=1
log.retention.ms=1
log.segment.bytes=1049

How I start each server is by,

/home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/bin/kafka-server-start.sh
-daemon
/home/ubuntu/eranga/software/kafka_2.11-0.10.0.1/config/server.properties

However, I still get about 4GB log files. Did I miss anything?

Thanks,
Regards,


Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
Email: era...@wso2.com 
   
   

On Wed, Nov 23, 2016 at 10:14 AM, Sachin Mittal  wrote:

> Check these in server.properties
> # Log Retention Policy
> #
>
> # The following configurations control the disposal of log segments. The
> policy can
> # be set to delete segments after a period of time, or after a given size
> has accumulated.
> # A segment will be deleted whenever *either* of these criteria are met.
> Deletion always happens
> # from the end of the log.
>
> # The minimum age of a log file to be eligible for deletion
> log.retention.hours=168
>
> # A size-based retention policy for logs. Segments are pruned from the log
> as long as the remaining
> # segments don't drop below log.retention.bytes.
> #log.retention.bytes=1073741824
>
> # The maximum size of a log segment file. When this size is reached a new
> log segment will be created.
> log.segment.bytes=1073741824
>
> Try by setting hours as 1 and set segment bytes as say 10 MB. You should be
> able to see multiple segments created of size 20 MB under you kafka logs
> dir
> say: tmp\kafka-logs\\*.log
>
> As the files age > 1 hour it will be marked for deletion and be deleted
> next time delete thread runs. By default the delete thread check is 5
> minutes.
>
> Also make sure when you start server you specify this property as the
> argument.
>
> bin\windows\kafka-server-start.bat config\server.properties
>
>
>
> On Wed, Nov 23, 2016 at 9:50 AM, Eranga Heshan 
> wrote:
>
> > This is the exact parameter I added to server.config file on all the
> nodes.
> >
> > log.retention.ms=1000
> >
> > Thanks,
> > Regards,
> >
> >
> > Eranga Heshan
> > *Undergraduate*
> > Computer Science & Engineering
> > University of Moratuwa
> > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > Email: era...@wso2.com 
> >     > erangaheshan>
> >
> >
> > On Wed, Nov 23, 2016 at 9:43 AM, Eranga Heshan 
> > wrote:
> >
> > > I have 4 node cluster and node1 has zookeeper running. I added those
> > > parameters on each of my server.properties files in kafka nodes.
> > >
> > >
> > > Eranga Heshan
> > > *Undergraduate*
> > > Computer Science & Engineering
> > > University of Moratuwa
> > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > Email: era...@wso2.com 
> > > 
> > > 
> > > 
> > >
> > > On Wed, Nov 23, 2016 at 9:36 AM, Sachin Mittal 
> > wrote:
> > >
> > >> Kafka logs are not kept based on data consumption. They are deleted
> > after
> > >> it reaches a particular size and retention time.
> > >> Even if they are consumed multiple times in that duration, they are
> > still
> > >> kept.
> > >>
> > >> Where have you added those params and how are you starting your
> > zookeeper
> > >> and server.
> > >>
> > >>
> > >> On Wed, Nov 23, 2016 at 8:00 AM, Eranga Heshan 
> > >> wrote:
> > >>
> > >> > Sorry for repeatedly asking the same question, but although I tried
> > all
> > >> > those parameters; my log file still at large about 2.5GB. Is it due
> to
> > >> the
> > >> > data inside my topic is flushed to disk? I meant that there might be
> > >> data
> > >> > inside topic which is not consumed yet or not been deleted even
> after
> > >> > consumed.
> > >> >
> > >> > Thanks,
> > >> > Regards,
> > >> >
> > >> >
> > >> > Eranga Heshan
> > >> > *Undergraduate*
> > >> > Computer Science & Engineering
> > >> > University of Moratuwa
> > >> > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > >> > Email: era...@wso2.com 
> > >> >     > >> > erangaheshan>
> > >> >
> > >> >
> > >> > On Tue, Nov 22, 2016 at 2:15 PM, Sachin Mittal 
> > >> wrote:
> > >> >
> > >> > > Set some value to log.retention.ms which is ms time to retain log
> > and
> > >> > also
> > >> > > log.segment.bytes which is size of a single log file.
> > >> > > delete.topic.enable is needed if you want to delete the topic
> > itself.
> > >> > >
> > >> > > Thanks
> > >> > > Sachin
> > >> > >
> > >> > >
> > >> > > On Tue, Nov 22, 2016 at 1:52 PM, Era

Re: Delete "kafka-logs"

2016-11-22 Thread Sachin Mittal 
Check these in server.properties
# Log Retention Policy
#

# The following configurations control the disposal of log segments. The
policy can
# be set to delete segments after a period of time, or after a given size
has accumulated.
# A segment will be deleted whenever *either* of these criteria are met.
Deletion always happens
# from the end of the log.

# The minimum age of a log file to be eligible for deletion
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log
as long as the remaining
# segments don't drop below log.retention.bytes.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new
log segment will be created.
log.segment.bytes=1073741824

Try by setting hours as 1 and set segment bytes as say 10 MB. You should be
able to see multiple segments created of size 20 MB under you kafka logs dir
say: tmp\kafka-logs\\*.log

As the files age > 1 hour it will be marked for deletion and be deleted
next time delete thread runs. By default the delete thread check is 5
minutes.

Also make sure when you start server you specify this property as the
argument.

bin\windows\kafka-server-start.bat config\server.properties



On Wed, Nov 23, 2016 at 9:50 AM, Eranga Heshan  wrote:

> This is the exact parameter I added to server.config file on all the nodes.
>
> log.retention.ms=1000
>
> Thanks,
> Regards,
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: era...@wso2.com 
>     erangaheshan>
>
>
> On Wed, Nov 23, 2016 at 9:43 AM, Eranga Heshan 
> wrote:
>
> > I have 4 node cluster and node1 has zookeeper running. I added those
> > parameters on each of my server.properties files in kafka nodes.
> >
> >
> > Eranga Heshan
> > *Undergraduate*
> > Computer Science & Engineering
> > University of Moratuwa
> > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > Email: era...@wso2.com 
> > 
> > 
> > 
> >
> > On Wed, Nov 23, 2016 at 9:36 AM, Sachin Mittal 
> wrote:
> >
> >> Kafka logs are not kept based on data consumption. They are deleted
> after
> >> it reaches a particular size and retention time.
> >> Even if they are consumed multiple times in that duration, they are
> still
> >> kept.
> >>
> >> Where have you added those params and how are you starting your
> zookeeper
> >> and server.
> >>
> >>
> >> On Wed, Nov 23, 2016 at 8:00 AM, Eranga Heshan 
> >> wrote:
> >>
> >> > Sorry for repeatedly asking the same question, but although I tried
> all
> >> > those parameters; my log file still at large about 2.5GB. Is it due to
> >> the
> >> > data inside my topic is flushed to disk? I meant that there might be
> >> data
> >> > inside topic which is not consumed yet or not been deleted even after
> >> > consumed.
> >> >
> >> > Thanks,
> >> > Regards,
> >> >
> >> >
> >> > Eranga Heshan
> >> > *Undergraduate*
> >> > Computer Science & Engineering
> >> > University of Moratuwa
> >> > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> >> > Email: era...@wso2.com 
> >> >     >> > erangaheshan>
> >> >
> >> >
> >> > On Tue, Nov 22, 2016 at 2:15 PM, Sachin Mittal 
> >> wrote:
> >> >
> >> > > Set some value to log.retention.ms which is ms time to retain log
> and
> >> > also
> >> > > log.segment.bytes which is size of a single log file.
> >> > > delete.topic.enable is needed if you want to delete the topic
> itself.
> >> > >
> >> > > Thanks
> >> > > Sachin
> >> > >
> >> > >
> >> > > On Tue, Nov 22, 2016 at 1:52 PM, Eranga Heshan <
> eranga@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > I tried log.retention.bytes=1049000 which is about 10MB. But it
> did
> >> not
> >> > > > work. Is it because I had not given any value to
> >> delete.topic.enable?
> >> > > >
> >> > > >
> >> > > > Eranga Heshan
> >> > > > *Undergraduate*
> >> > > > Computer Science & Engineering
> >> > > > University of Moratuwa
> >> > > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> >> > > > Email: era...@wso2.com 
> >> > > >     >> > > > erangaheshan>
> >> > > >
> >> > > >
> >> > > > On Tue, Nov 22, 2016 at 1:47 PM, Sachin Mittal <
> sjmit...@gmail.com>
> >> > > wrote:
> >> > > >
> >> > > > > Check
> >> > > > > http://kafka.apache.org/documentation.html#brokerconfigs
> >> > > > > log.retention.bytes
> >> > > > > log.retention.ms
> >> > > > > log.segment.bytes
> >> > > > > delete.topic.enable
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On

Re: Delete "kafka-logs"

2016-11-22 Thread Eranga Heshan 
This is the exact parameter I added to server.config file on all the nodes.

log.retention.ms=1000

Thanks,
Regards,


Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
Email: era...@wso2.com 
   
   

On Wed, Nov 23, 2016 at 9:43 AM, Eranga Heshan  wrote:

> I have 4 node cluster and node1 has zookeeper running. I added those
> parameters on each of my server.properties files in kafka nodes.
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: era...@wso2.com 
> 
> 
> 
>
> On Wed, Nov 23, 2016 at 9:36 AM, Sachin Mittal  wrote:
>
>> Kafka logs are not kept based on data consumption. They are deleted after
>> it reaches a particular size and retention time.
>> Even if they are consumed multiple times in that duration, they are still
>> kept.
>>
>> Where have you added those params and how are you starting your zookeeper
>> and server.
>>
>>
>> On Wed, Nov 23, 2016 at 8:00 AM, Eranga Heshan 
>> wrote:
>>
>> > Sorry for repeatedly asking the same question, but although I tried all
>> > those parameters; my log file still at large about 2.5GB. Is it due to
>> the
>> > data inside my topic is flushed to disk? I meant that there might be
>> data
>> > inside topic which is not consumed yet or not been deleted even after
>> > consumed.
>> >
>> > Thanks,
>> > Regards,
>> >
>> >
>> > Eranga Heshan
>> > *Undergraduate*
>> > Computer Science & Engineering
>> > University of Moratuwa
>> > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
>> > Email: era...@wso2.com 
>> >    > > erangaheshan>
>> >
>> >
>> > On Tue, Nov 22, 2016 at 2:15 PM, Sachin Mittal 
>> wrote:
>> >
>> > > Set some value to log.retention.ms which is ms time to retain log and
>> > also
>> > > log.segment.bytes which is size of a single log file.
>> > > delete.topic.enable is needed if you want to delete the topic itself.
>> > >
>> > > Thanks
>> > > Sachin
>> > >
>> > >
>> > > On Tue, Nov 22, 2016 at 1:52 PM, Eranga Heshan 
>> > > wrote:
>> > >
>> > > > I tried log.retention.bytes=1049000 which is about 10MB. But it did
>> not
>> > > > work. Is it because I had not given any value to
>> delete.topic.enable?
>> > > >
>> > > >
>> > > > Eranga Heshan
>> > > > *Undergraduate*
>> > > > Computer Science & Engineering
>> > > > University of Moratuwa
>> > > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
>> > > > Email: era...@wso2.com 
>> > > >    > > > > erangaheshan>
>> > > >
>> > > >
>> > > > On Tue, Nov 22, 2016 at 1:47 PM, Sachin Mittal 
>> > > wrote:
>> > > >
>> > > > > Check
>> > > > > http://kafka.apache.org/documentation.html#brokerconfigs
>> > > > > log.retention.bytes
>> > > > > log.retention.ms
>> > > > > log.segment.bytes
>> > > > > delete.topic.enable
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Tue, Nov 22, 2016 at 1:36 PM, Eranga Heshan <
>> eranga@gmail.com
>> > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > I want to keep the size of kafka-logs reduced as much as
>> possible.
>> > To
>> > > > do
>> > > > > it
>> > > > > > I need to set it to delete its unwanted logs frequently. Is
>> there a
>> > > way
>> > > > > to
>> > > > > > do it?
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Regards,
>> > > > > >
>> > > > > > Eranga Heshan
>> > > > > > *Undergraduate*
>> > > > > > Computer Science & Engineering
>> > > > > > University of Moratuwa
>> > > > > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
>> > > > > > Email: era...@wso2.com 
>> > > > > >    > > > > > > erangaheshan>
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: Delete "kafka-logs"

2016-11-22 Thread Eranga Heshan 
I have 4 node cluster and node1 has zookeeper running. I added those
parameters on each of my server.properties files in kafka nodes.


Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
Email: era...@wso2.com 
   
   

On Wed, Nov 23, 2016 at 9:36 AM, Sachin Mittal  wrote:

> Kafka logs are not kept based on data consumption. They are deleted after
> it reaches a particular size and retention time.
> Even if they are consumed multiple times in that duration, they are still
> kept.
>
> Where have you added those params and how are you starting your zookeeper
> and server.
>
>
> On Wed, Nov 23, 2016 at 8:00 AM, Eranga Heshan 
> wrote:
>
> > Sorry for repeatedly asking the same question, but although I tried all
> > those parameters; my log file still at large about 2.5GB. Is it due to
> the
> > data inside my topic is flushed to disk? I meant that there might be data
> > inside topic which is not consumed yet or not been deleted even after
> > consumed.
> >
> > Thanks,
> > Regards,
> >
> >
> > Eranga Heshan
> > *Undergraduate*
> > Computer Science & Engineering
> > University of Moratuwa
> > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > Email: era...@wso2.com 
> >     > erangaheshan>
> >
> >
> > On Tue, Nov 22, 2016 at 2:15 PM, Sachin Mittal 
> wrote:
> >
> > > Set some value to log.retention.ms which is ms time to retain log and
> > also
> > > log.segment.bytes which is size of a single log file.
> > > delete.topic.enable is needed if you want to delete the topic itself.
> > >
> > > Thanks
> > > Sachin
> > >
> > >
> > > On Tue, Nov 22, 2016 at 1:52 PM, Eranga Heshan 
> > > wrote:
> > >
> > > > I tried log.retention.bytes=1049000 which is about 10MB. But it did
> not
> > > > work. Is it because I had not given any value to delete.topic.enable?
> > > >
> > > >
> > > > Eranga Heshan
> > > > *Undergraduate*
> > > > Computer Science & Engineering
> > > > University of Moratuwa
> > > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > > Email: era...@wso2.com 
> > > >     > > > erangaheshan>
> > > >
> > > >
> > > > On Tue, Nov 22, 2016 at 1:47 PM, Sachin Mittal 
> > > wrote:
> > > >
> > > > > Check
> > > > > http://kafka.apache.org/documentation.html#brokerconfigs
> > > > > log.retention.bytes
> > > > > log.retention.ms
> > > > > log.segment.bytes
> > > > > delete.topic.enable
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Nov 22, 2016 at 1:36 PM, Eranga Heshan <
> eranga@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I want to keep the size of kafka-logs reduced as much as
> possible.
> > To
> > > > do
> > > > > it
> > > > > > I need to set it to delete its unwanted logs frequently. Is
> there a
> > > way
> > > > > to
> > > > > > do it?
> > > > > >
> > > > > > Thanks,
> > > > > > Regards,
> > > > > >
> > > > > > Eranga Heshan
> > > > > > *Undergraduate*
> > > > > > Computer Science & Engineering
> > > > > > University of Moratuwa
> > > > > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > > > > Email: era...@wso2.com 
> > > > > >     > > > > > erangaheshan>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Delete "kafka-logs"

2016-11-22 Thread Sachin Mittal 
Kafka logs are not kept based on data consumption. They are deleted after
it reaches a particular size and retention time.
Even if they are consumed multiple times in that duration, they are still
kept.

Where have you added those params and how are you starting your zookeeper
and server.


On Wed, Nov 23, 2016 at 8:00 AM, Eranga Heshan  wrote:

> Sorry for repeatedly asking the same question, but although I tried all
> those parameters; my log file still at large about 2.5GB. Is it due to the
> data inside my topic is flushed to disk? I meant that there might be data
> inside topic which is not consumed yet or not been deleted even after
> consumed.
>
> Thanks,
> Regards,
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: era...@wso2.com 
>     erangaheshan>
>
>
> On Tue, Nov 22, 2016 at 2:15 PM, Sachin Mittal  wrote:
>
> > Set some value to log.retention.ms which is ms time to retain log and
> also
> > log.segment.bytes which is size of a single log file.
> > delete.topic.enable is needed if you want to delete the topic itself.
> >
> > Thanks
> > Sachin
> >
> >
> > On Tue, Nov 22, 2016 at 1:52 PM, Eranga Heshan 
> > wrote:
> >
> > > I tried log.retention.bytes=1049000 which is about 10MB. But it did not
> > > work. Is it because I had not given any value to delete.topic.enable?
> > >
> > >
> > > Eranga Heshan
> > > *Undergraduate*
> > > Computer Science & Engineering
> > > University of Moratuwa
> > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > Email: era...@wso2.com 
> > >     > > erangaheshan>
> > >
> > >
> > > On Tue, Nov 22, 2016 at 1:47 PM, Sachin Mittal 
> > wrote:
> > >
> > > > Check
> > > > http://kafka.apache.org/documentation.html#brokerconfigs
> > > > log.retention.bytes
> > > > log.retention.ms
> > > > log.segment.bytes
> > > > delete.topic.enable
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Nov 22, 2016 at 1:36 PM, Eranga Heshan  >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I want to keep the size of kafka-logs reduced as much as possible.
> To
> > > do
> > > > it
> > > > > I need to set it to delete its unwanted logs frequently. Is there a
> > way
> > > > to
> > > > > do it?
> > > > >
> > > > > Thanks,
> > > > > Regards,
> > > > >
> > > > > Eranga Heshan
> > > > > *Undergraduate*
> > > > > Computer Science & Engineering
> > > > > University of Moratuwa
> > > > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > > > Email: era...@wso2.com 
> > > > >     > > > > erangaheshan>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Delete "kafka-logs"

2016-11-22 Thread Eranga Heshan 
Sorry for repeatedly asking the same question, but although I tried all
those parameters; my log file still at large about 2.5GB. Is it due to the
data inside my topic is flushed to disk? I meant that there might be data
inside topic which is not consumed yet or not been deleted even after
consumed.

Thanks,
Regards,


Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
Email: era...@wso2.com 
   
   

On Tue, Nov 22, 2016 at 2:15 PM, Sachin Mittal  wrote:

> Set some value to log.retention.ms which is ms time to retain log and also
> log.segment.bytes which is size of a single log file.
> delete.topic.enable is needed if you want to delete the topic itself.
>
> Thanks
> Sachin
>
>
> On Tue, Nov 22, 2016 at 1:52 PM, Eranga Heshan 
> wrote:
>
> > I tried log.retention.bytes=1049000 which is about 10MB. But it did not
> > work. Is it because I had not given any value to delete.topic.enable?
> >
> >
> > Eranga Heshan
> > *Undergraduate*
> > Computer Science & Engineering
> > University of Moratuwa
> > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > Email: era...@wso2.com 
> >     > erangaheshan>
> >
> >
> > On Tue, Nov 22, 2016 at 1:47 PM, Sachin Mittal 
> wrote:
> >
> > > Check
> > > http://kafka.apache.org/documentation.html#brokerconfigs
> > > log.retention.bytes
> > > log.retention.ms
> > > log.segment.bytes
> > > delete.topic.enable
> > >
> > >
> > >
> > >
> > > On Tue, Nov 22, 2016 at 1:36 PM, Eranga Heshan 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I want to keep the size of kafka-logs reduced as much as possible. To
> > do
> > > it
> > > > I need to set it to delete its unwanted logs frequently. Is there a
> way
> > > to
> > > > do it?
> > > >
> > > > Thanks,
> > > > Regards,
> > > >
> > > > Eranga Heshan
> > > > *Undergraduate*
> > > > Computer Science & Engineering
> > > > University of Moratuwa
> > > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > > Email: era...@wso2.com 
> > > >     > > > erangaheshan>
> > > >
> > > >
> > >
> >
>

Re: Delete "kafka-logs"

2016-11-22 Thread Sachin Mittal 
Set some value to log.retention.ms which is ms time to retain log and also
log.segment.bytes which is size of a single log file.
delete.topic.enable is needed if you want to delete the topic itself.

Thanks
Sachin


On Tue, Nov 22, 2016 at 1:52 PM, Eranga Heshan  wrote:

> I tried log.retention.bytes=1049000 which is about 10MB. But it did not
> work. Is it because I had not given any value to delete.topic.enable?
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: era...@wso2.com 
>     erangaheshan>
>
>
> On Tue, Nov 22, 2016 at 1:47 PM, Sachin Mittal  wrote:
>
> > Check
> > http://kafka.apache.org/documentation.html#brokerconfigs
> > log.retention.bytes
> > log.retention.ms
> > log.segment.bytes
> > delete.topic.enable
> >
> >
> >
> >
> > On Tue, Nov 22, 2016 at 1:36 PM, Eranga Heshan 
> > wrote:
> >
> > > Hi all,
> > >
> > > I want to keep the size of kafka-logs reduced as much as possible. To
> do
> > it
> > > I need to set it to delete its unwanted logs frequently. Is there a way
> > to
> > > do it?
> > >
> > > Thanks,
> > > Regards,
> > >
> > > Eranga Heshan
> > > *Undergraduate*
> > > Computer Science & Engineering
> > > University of Moratuwa
> > > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > > Email: era...@wso2.com 
> > >     > > erangaheshan>
> > >
> > >
> >
>

Re: Delete "kafka-logs"

2016-11-22 Thread Eranga Heshan 
I tried log.retention.bytes=1049000 which is about 10MB. But it did not
work. Is it because I had not given any value to delete.topic.enable?


Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
Email: era...@wso2.com 
   
   

On Tue, Nov 22, 2016 at 1:47 PM, Sachin Mittal  wrote:

> Check
> http://kafka.apache.org/documentation.html#brokerconfigs
> log.retention.bytes
> log.retention.ms
> log.segment.bytes
> delete.topic.enable
>
>
>
>
> On Tue, Nov 22, 2016 at 1:36 PM, Eranga Heshan 
> wrote:
>
> > Hi all,
> >
> > I want to keep the size of kafka-logs reduced as much as possible. To do
> it
> > I need to set it to delete its unwanted logs frequently. Is there a way
> to
> > do it?
> >
> > Thanks,
> > Regards,
> >
> > Eranga Heshan
> > *Undergraduate*
> > Computer Science & Engineering
> > University of Moratuwa
> > Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> > Email: era...@wso2.com 
> >     > erangaheshan>
> >
> >
>

Re: Delete "kafka-logs"

2016-11-22 Thread Sachin Mittal 
Check
http://kafka.apache.org/documentation.html#brokerconfigs
log.retention.bytes
log.retention.ms
log.segment.bytes
delete.topic.enable




On Tue, Nov 22, 2016 at 1:36 PM, Eranga Heshan  wrote:

> Hi all,
>
> I want to keep the size of kafka-logs reduced as much as possible. To do it
> I need to set it to delete its unwanted logs frequently. Is there a way to
> do it?
>
> Thanks,
> Regards,
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: era...@wso2.com 
>     erangaheshan>
>
>