Re: Failure to reset consumer offsets for specific topics

[Dan Markhasin]

For beats (the topic where timestamps are OK) the producers are Metricbeat
/ Logstash which I assume use a relatively recent producer version.
For the topics with missing timestamps the producers are based on
librdkafka though I'm not sure which version exactly - I wouldn't be
surprised if it's pretty old and doesn't set timestamp on the messages it
produces.

On 26 October 2017 at 08:17, Elyahou Ittah  wrote:

> Which driver is used yo produce these messages ?
>
> On Thu, Oct 26, 2017 at 8:14 AM, Dan Markhasin 
> wrote:
>
> > Furthermore, when looking at messages produced to the data1_log topic
> with
> > print.timestamp=true they all have CreateTime:-1 whereas messages
> produced
> > to the beats topic have valid timestamps. The producers that are sending
> to
> > data1_log are older than the producers that are sending to beats - if
> this
> > is where the broker takes the timestamps from, it explains why they are
> all
> > empty for data1_log.
> >
> > On 26 October 2017 at 08:07, Dan Markhasin  wrote:
> >
> > > After a bit more checking it seems that Kafka isn't writing timestamps
> at
> > > all in the .timeindex file for the topics where offset rewind is not
> > > working.
> > >
> > > The following output is from * a different 0.11.0.0 * which also has a
> > > topic called data1_log (this cluster has not experienced any issues
> > lately):
> > >
> > > bash-4.2$ ls -ltr | tail
> > > -rw-rw-r-- 1 ibiuser users 1073739867 Oct 24 20:42
> > 000723010092.log
> > > -rw-rw-r-- 1 ibiuser users 926216 Oct 24 20:42
> > > 000723010092.index
> > > -rw-rw-r-- 1 ibiuser users 16 Oct 25 05:09
> > leader-epoch-checkpoint
> > > -rw-rw-r-- 1 ibiuser users 1073741239 Oct 25 09:52
> > 000725186047.log
> > > -rw-rw-r-- 1 ibiuser users   10485756 Oct 25 09:52
> > > 000727395414.timeindex
> > > -rw-rw-r-- 1 ibiuser users 10 Oct 25 09:52
> > > 000727395414.snapshot
> > > -rw-rw-r-- 1 ibiuser users  0 Oct 25 09:52
> > > 000725186047.timeindex
> > > -rw-rw-r-- 1 ibiuser users 961144 Oct 25 09:52
> > > 000725186047.index
> > > -rw-rw-r-- 1 ibiuser users   10485760 Oct 25 21:59
> > > 000727395414.index
> > > -rw-rw-r-- 1 ibiuser users 1028809456 Oct 25 21:59
> > 000727395414.log
> > >
> > >
> > > bash-4.2$ /kafka/latest/bin/kafka-run-class.sh
> > > kafka.tools.DumpLogSegments --files 000727395414.timeindex >
> > > /tmp/timestamps 2>&1
> > >
> > > The output looks like this:
> > >
> > > timestamp: 0 offset: 727395414
> > > timestamp: 0 offset: 727395414
> > > timestamp: 0 offset: 727395414
> > > timestamp: 0 offset: 727395414
> > > timestamp: 0 offset: 727395414
> > > timestamp: 0 offset: 727395414
> > > timestamp: 0 offset: 727395414
> > >
> > >
> > > The offset is the same on all of the records and not a single line has
> a
> > > timestamp greater than zero.
> > >
> > > bash-4.2$ cat /tmp/timestamps | grep timestamp | awk '{print $2}' |
> grep
> > > -v 0
> > > bash-4.2$
> > >
> > > For comparison, the .timeindex for the beats topic on the same cluster
> > > looks like this:
> > >
> > > Dumping 000254410745.timeindex
> > > timestamp: 1508978010544 offset: 254410759
> > > timestamp: 1508978011084 offset: 254410763
> > > timestamp: 1508978012080 offset: 254410770
> > > timestamp: 1508978012789 offset: 254410796
> > > timestamp: 1508978013981 offset: 254410808
> > > timestamp: 1508978014526 offset: 254410812
> > > timestamp: 1508978014698 offset: 254410828
> > > timestamp: 1508978014959 offset: 254410834
> > > timestamp: 1508978015677 offset: 254410854
> > > timestamp: 1508978016980 offset: 254410870
> > > timestamp: 1508978017212 offset: 254410883
> > >
> > >
> > > On 26 October 2017 at 07:26, Elyahou Ittah 
> wrote:
> > >
> > >> Which driver is used yo produce these messages ?
> > >>
> > >> On Oct 26, 2017 07:11, "Dan Markhasin"  wrote:
> > >>
> > >> > No, that flag doesn't affect which offsets are returned, only
> executes
> > >> the
> > >> > action (and resets the consumer to latest offset when used,
> regardless
> > >> of
> > >> > datetime value I provide).
> > >> >
> > >> > On 25 October 2017 at 23:44, Hans Jespersen 
> > wrote:
> > >> >
> > >> > > I think you are just missing the —execute flag.
> > >> > >
> > >> > > -hans
> > >> > >
> > >> > > > On Oct 25, 2017, at 1:24 PM, Ted Yu 
> wrote:
> > >> > > >
> > >> > > > I wonder if you have hit KAFKA-5600.
> > >> > > >
> > >> > > > Is it possible that you try out 0.11.0.1 ?
> > >> > > >
> > >> > > > Thanks
> > >> > > >
> > >> > > >> On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin <
> > >> minimi...@gmail.com>
> > >> > > wrote:
> > >> > > >>
> > >> > > >> I am using 0.11.0.0.
> > >> > > >>
> > >> > > >> There is no difference configuration-wise - both have 10
> > partitions
> > >> > and
> > >> > > 2
> > >> > > >> replicas. There are no errors in the logs, but looking in the
> > data
> > >> > > folder
> > >> > > >> it seems like Kafka is not updating the timeindex f

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

[Onur Karaman]

Looks like Jan technically made his KIP wiki page first so I'll just change
my KIP number.

On Wed, Oct 25, 2017 at 4:59 PM, Matthias J. Sax 
wrote:

> Thanks a lot for the KIP. Can we please move the discussion to the dev
> list?
>
> Thus, after fixing the KIP collision, just start a new DISCUSS thread.
>
> Thx.
>
>
> -Matthias
>
> On 10/25/17 4:20 PM, Ted Yu wrote:
> > Have you seen the email a moment ago from Onur which uses the same KIP
> > number ?
> >
> > Looks like there was race condition in modifying wiki.
> >
> > Please consider bumping the KIP number.
> >
> > Thanks
> >
> > On Wed, Oct 25, 2017 at 4:14 PM, Jan Filipiak 
> > wrote:
> >
> >> Hello Kafka-users,
> >>
> >> I want to continue with the development of KAFKA-3705, which allows the
> >> Streams DSL to perform KTableKTable-Joins when the KTables have a
> >> one-to-many relationship.
> >> To make sure we cover the requirements of as many users as possible and
> >> have a good solution afterwards I invite everyone to read through the
> KIP I
> >> put together and
> >> discuss it here in this Thread.
> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+
> >> Support+non-key+joining+in+KTable
> >> https://issues.apache.org/jira/browse/KAFKA-3705
> >> https://github.com/apache/kafka/pull/3720
> >>
> >> I think a public discussion and vote on a solution is exactly what is
> >> needed to bring this feauture into kafka-streams. I am looking forward
> to
> >> everyones opinion!
> >>
> >> Please keep the discussion on the mailing list rather than commenting on
> >> the wiki (wiki discussions get unwieldy fast).
> >>
> >> Best
> >> Jan
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
>
>

Re: Failure to reset consumer offsets for specific topics

[Elyahou Ittah]

Which driver is used yo produce these messages ?

On Thu, Oct 26, 2017 at 8:14 AM, Dan Markhasin  wrote:

> Furthermore, when looking at messages produced to the data1_log topic with
> print.timestamp=true they all have CreateTime:-1 whereas messages produced
> to the beats topic have valid timestamps. The producers that are sending to
> data1_log are older than the producers that are sending to beats - if this
> is where the broker takes the timestamps from, it explains why they are all
> empty for data1_log.
>
> On 26 October 2017 at 08:07, Dan Markhasin  wrote:
>
> > After a bit more checking it seems that Kafka isn't writing timestamps at
> > all in the .timeindex file for the topics where offset rewind is not
> > working.
> >
> > The following output is from * a different 0.11.0.0 * which also has a
> > topic called data1_log (this cluster has not experienced any issues
> lately):
> >
> > bash-4.2$ ls -ltr | tail
> > -rw-rw-r-- 1 ibiuser users 1073739867 Oct 24 20:42
> 000723010092.log
> > -rw-rw-r-- 1 ibiuser users 926216 Oct 24 20:42
> > 000723010092.index
> > -rw-rw-r-- 1 ibiuser users 16 Oct 25 05:09
> leader-epoch-checkpoint
> > -rw-rw-r-- 1 ibiuser users 1073741239 Oct 25 09:52
> 000725186047.log
> > -rw-rw-r-- 1 ibiuser users   10485756 Oct 25 09:52
> > 000727395414.timeindex
> > -rw-rw-r-- 1 ibiuser users 10 Oct 25 09:52
> > 000727395414.snapshot
> > -rw-rw-r-- 1 ibiuser users  0 Oct 25 09:52
> > 000725186047.timeindex
> > -rw-rw-r-- 1 ibiuser users 961144 Oct 25 09:52
> > 000725186047.index
> > -rw-rw-r-- 1 ibiuser users   10485760 Oct 25 21:59
> > 000727395414.index
> > -rw-rw-r-- 1 ibiuser users 1028809456 Oct 25 21:59
> 000727395414.log
> >
> >
> > bash-4.2$ /kafka/latest/bin/kafka-run-class.sh
> > kafka.tools.DumpLogSegments --files 000727395414.timeindex >
> > /tmp/timestamps 2>&1
> >
> > The output looks like this:
> >
> > timestamp: 0 offset: 727395414
> > timestamp: 0 offset: 727395414
> > timestamp: 0 offset: 727395414
> > timestamp: 0 offset: 727395414
> > timestamp: 0 offset: 727395414
> > timestamp: 0 offset: 727395414
> > timestamp: 0 offset: 727395414
> >
> >
> > The offset is the same on all of the records and not a single line has a
> > timestamp greater than zero.
> >
> > bash-4.2$ cat /tmp/timestamps | grep timestamp | awk '{print $2}' | grep
> > -v 0
> > bash-4.2$
> >
> > For comparison, the .timeindex for the beats topic on the same cluster
> > looks like this:
> >
> > Dumping 000254410745.timeindex
> > timestamp: 1508978010544 offset: 254410759
> > timestamp: 1508978011084 offset: 254410763
> > timestamp: 1508978012080 offset: 254410770
> > timestamp: 1508978012789 offset: 254410796
> > timestamp: 1508978013981 offset: 254410808
> > timestamp: 1508978014526 offset: 254410812
> > timestamp: 1508978014698 offset: 254410828
> > timestamp: 1508978014959 offset: 254410834
> > timestamp: 1508978015677 offset: 254410854
> > timestamp: 1508978016980 offset: 254410870
> > timestamp: 1508978017212 offset: 254410883
> >
> >
> > On 26 October 2017 at 07:26, Elyahou Ittah  wrote:
> >
> >> Which driver is used yo produce these messages ?
> >>
> >> On Oct 26, 2017 07:11, "Dan Markhasin"  wrote:
> >>
> >> > No, that flag doesn't affect which offsets are returned, only executes
> >> the
> >> > action (and resets the consumer to latest offset when used, regardless
> >> of
> >> > datetime value I provide).
> >> >
> >> > On 25 October 2017 at 23:44, Hans Jespersen 
> wrote:
> >> >
> >> > > I think you are just missing the —execute flag.
> >> > >
> >> > > -hans
> >> > >
> >> > > > On Oct 25, 2017, at 1:24 PM, Ted Yu  wrote:
> >> > > >
> >> > > > I wonder if you have hit KAFKA-5600.
> >> > > >
> >> > > > Is it possible that you try out 0.11.0.1 ?
> >> > > >
> >> > > > Thanks
> >> > > >
> >> > > >> On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin <
> >> minimi...@gmail.com>
> >> > > wrote:
> >> > > >>
> >> > > >> I am using 0.11.0.0.
> >> > > >>
> >> > > >> There is no difference configuration-wise - both have 10
> partitions
> >> > and
> >> > > 2
> >> > > >> replicas. There are no errors in the logs, but looking in the
> data
> >> > > folder
> >> > > >> it seems like Kafka is not updating the timeindex file for
> >> data1_log -
> >> > > >> notice how the timeindex file for the current log segment is not
> >> being
> >> > > >> updated.
> >> > > >>
> >> > > >> bash-4.2$ pwd
> >> > > >> /kafka/data/data1_log-1
> >> > > >> bash-4.2$ ls -ltr | tail
> >> > > >> -rw-rw-r-- 1 ibiuser it 1073731573 Oct 25 01:21
> >> > 000337554984.log
> >> > > >> -rw-rw-r-- 1 ibiuser it 943616 Oct 25 01:21
> >> > > 000337554984.index
> >> > > >> -rw-rw-r-- 1 ibiuser it 1073734199 Oct 25 13:38
> >> > 000339816017.log
> >> > > >> -rw-rw-r-- 1 ibiuser it   10485756 Oct 25 13:38
> >> > > >> 000341934289.timeindex
> >> > > >> -rw-rw-r-- 1 ibiuser it 10 Oct 25 13:38

Re: Failure to reset consumer offsets for specific topics

[Dan Markhasin]

Furthermore, when looking at messages produced to the data1_log topic with
print.timestamp=true they all have CreateTime:-1 whereas messages produced
to the beats topic have valid timestamps. The producers that are sending to
data1_log are older than the producers that are sending to beats - if this
is where the broker takes the timestamps from, it explains why they are all
empty for data1_log.

On 26 October 2017 at 08:07, Dan Markhasin  wrote:

> After a bit more checking it seems that Kafka isn't writing timestamps at
> all in the .timeindex file for the topics where offset rewind is not
> working.
>
> The following output is from * a different 0.11.0.0 * which also has a
> topic called data1_log (this cluster has not experienced any issues lately):
>
> bash-4.2$ ls -ltr | tail
> -rw-rw-r-- 1 ibiuser users 1073739867 Oct 24 20:42 000723010092.log
> -rw-rw-r-- 1 ibiuser users 926216 Oct 24 20:42
> 000723010092.index
> -rw-rw-r-- 1 ibiuser users 16 Oct 25 05:09 leader-epoch-checkpoint
> -rw-rw-r-- 1 ibiuser users 1073741239 Oct 25 09:52 000725186047.log
> -rw-rw-r-- 1 ibiuser users   10485756 Oct 25 09:52
> 000727395414.timeindex
> -rw-rw-r-- 1 ibiuser users 10 Oct 25 09:52
> 000727395414.snapshot
> -rw-rw-r-- 1 ibiuser users  0 Oct 25 09:52
> 000725186047.timeindex
> -rw-rw-r-- 1 ibiuser users 961144 Oct 25 09:52
> 000725186047.index
> -rw-rw-r-- 1 ibiuser users   10485760 Oct 25 21:59
> 000727395414.index
> -rw-rw-r-- 1 ibiuser users 1028809456 Oct 25 21:59 000727395414.log
>
>
> bash-4.2$ /kafka/latest/bin/kafka-run-class.sh
> kafka.tools.DumpLogSegments --files 000727395414.timeindex >
> /tmp/timestamps 2>&1
>
> The output looks like this:
>
> timestamp: 0 offset: 727395414
> timestamp: 0 offset: 727395414
> timestamp: 0 offset: 727395414
> timestamp: 0 offset: 727395414
> timestamp: 0 offset: 727395414
> timestamp: 0 offset: 727395414
> timestamp: 0 offset: 727395414
>
>
> The offset is the same on all of the records and not a single line has a
> timestamp greater than zero.
>
> bash-4.2$ cat /tmp/timestamps | grep timestamp | awk '{print $2}' | grep
> -v 0
> bash-4.2$
>
> For comparison, the .timeindex for the beats topic on the same cluster
> looks like this:
>
> Dumping 000254410745.timeindex
> timestamp: 1508978010544 offset: 254410759
> timestamp: 1508978011084 offset: 254410763
> timestamp: 1508978012080 offset: 254410770
> timestamp: 1508978012789 offset: 254410796
> timestamp: 1508978013981 offset: 254410808
> timestamp: 1508978014526 offset: 254410812
> timestamp: 1508978014698 offset: 254410828
> timestamp: 1508978014959 offset: 254410834
> timestamp: 1508978015677 offset: 254410854
> timestamp: 1508978016980 offset: 254410870
> timestamp: 1508978017212 offset: 254410883
>
>
> On 26 October 2017 at 07:26, Elyahou Ittah  wrote:
>
>> Which driver is used yo produce these messages ?
>>
>> On Oct 26, 2017 07:11, "Dan Markhasin"  wrote:
>>
>> > No, that flag doesn't affect which offsets are returned, only executes
>> the
>> > action (and resets the consumer to latest offset when used, regardless
>> of
>> > datetime value I provide).
>> >
>> > On 25 October 2017 at 23:44, Hans Jespersen  wrote:
>> >
>> > > I think you are just missing the —execute flag.
>> > >
>> > > -hans
>> > >
>> > > > On Oct 25, 2017, at 1:24 PM, Ted Yu  wrote:
>> > > >
>> > > > I wonder if you have hit KAFKA-5600.
>> > > >
>> > > > Is it possible that you try out 0.11.0.1 ?
>> > > >
>> > > > Thanks
>> > > >
>> > > >> On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin <
>> minimi...@gmail.com>
>> > > wrote:
>> > > >>
>> > > >> I am using 0.11.0.0.
>> > > >>
>> > > >> There is no difference configuration-wise - both have 10 partitions
>> > and
>> > > 2
>> > > >> replicas. There are no errors in the logs, but looking in the data
>> > > folder
>> > > >> it seems like Kafka is not updating the timeindex file for
>> data1_log -
>> > > >> notice how the timeindex file for the current log segment is not
>> being
>> > > >> updated.
>> > > >>
>> > > >> bash-4.2$ pwd
>> > > >> /kafka/data/data1_log-1
>> > > >> bash-4.2$ ls -ltr | tail
>> > > >> -rw-rw-r-- 1 ibiuser it 1073731573 Oct 25 01:21
>> > 000337554984.log
>> > > >> -rw-rw-r-- 1 ibiuser it 943616 Oct 25 01:21
>> > > 000337554984.index
>> > > >> -rw-rw-r-- 1 ibiuser it 1073734199 Oct 25 13:38
>> > 000339816017.log
>> > > >> -rw-rw-r-- 1 ibiuser it   10485756 Oct 25 13:38
>> > > >> 000341934289.timeindex
>> > > >> -rw-rw-r-- 1 ibiuser it 10 Oct 25 13:38
>> > > >> 000341934289.snapshot
>> > > >> -rw-rw-r-- 1 ibiuser it  0 Oct 25 13:38
>> > > >> 000339816017.timeindex
>> > > >> -rw-rw-r-- 1 ibiuser it 566712 Oct 25 13:38
>> > > 000339816017.index
>> > > >> -rw-rw-r-- 1 ibiuser it 17 Oct 25 20:23
>> > leader-epoch-checkpoint
>> > > >> -rw-rw-r-- 1 ibiuser it   10485760 Oct 25 23:03
>

Re: Failure to reset consumer offsets for specific topics

[Dan Markhasin]

After a bit more checking it seems that Kafka isn't writing timestamps at
all in the .timeindex file for the topics where offset rewind is not
working.

The following output is from * a different 0.11.0.0 * which also has a
topic called data1_log (this cluster has not experienced any issues lately):

bash-4.2$ ls -ltr | tail
-rw-rw-r-- 1 ibiuser users 1073739867 Oct 24 20:42 000723010092.log
-rw-rw-r-- 1 ibiuser users 926216 Oct 24 20:42
000723010092.index
-rw-rw-r-- 1 ibiuser users 16 Oct 25 05:09 leader-epoch-checkpoint
-rw-rw-r-- 1 ibiuser users 1073741239 Oct 25 09:52 000725186047.log
-rw-rw-r-- 1 ibiuser users   10485756 Oct 25 09:52
000727395414.timeindex
-rw-rw-r-- 1 ibiuser users 10 Oct 25 09:52
000727395414.snapshot
-rw-rw-r-- 1 ibiuser users  0 Oct 25 09:52
000725186047.timeindex
-rw-rw-r-- 1 ibiuser users 961144 Oct 25 09:52
000725186047.index
-rw-rw-r-- 1 ibiuser users   10485760 Oct 25 21:59
000727395414.index
-rw-rw-r-- 1 ibiuser users 1028809456 Oct 25 21:59 000727395414.log


bash-4.2$ /kafka/latest/bin/kafka-run-class.sh kafka.tools.DumpLogSegments
--files 000727395414.timeindex > /tmp/timestamps 2>&1

The output looks like this:

timestamp: 0 offset: 727395414
timestamp: 0 offset: 727395414
timestamp: 0 offset: 727395414
timestamp: 0 offset: 727395414
timestamp: 0 offset: 727395414
timestamp: 0 offset: 727395414
timestamp: 0 offset: 727395414


The offset is the same on all of the records and not a single line has a
timestamp greater than zero.

bash-4.2$ cat /tmp/timestamps | grep timestamp | awk '{print $2}' | grep -v
0
bash-4.2$

For comparison, the .timeindex for the beats topic on the same cluster
looks like this:

Dumping 000254410745.timeindex
timestamp: 1508978010544 offset: 254410759
timestamp: 1508978011084 offset: 254410763
timestamp: 1508978012080 offset: 254410770
timestamp: 1508978012789 offset: 254410796
timestamp: 1508978013981 offset: 254410808
timestamp: 1508978014526 offset: 254410812
timestamp: 1508978014698 offset: 254410828
timestamp: 1508978014959 offset: 254410834
timestamp: 1508978015677 offset: 254410854
timestamp: 1508978016980 offset: 254410870
timestamp: 1508978017212 offset: 254410883


On 26 October 2017 at 07:26, Elyahou Ittah  wrote:

> Which driver is used yo produce these messages ?
>
> On Oct 26, 2017 07:11, "Dan Markhasin"  wrote:
>
> > No, that flag doesn't affect which offsets are returned, only executes
> the
> > action (and resets the consumer to latest offset when used, regardless of
> > datetime value I provide).
> >
> > On 25 October 2017 at 23:44, Hans Jespersen  wrote:
> >
> > > I think you are just missing the —execute flag.
> > >
> > > -hans
> > >
> > > > On Oct 25, 2017, at 1:24 PM, Ted Yu  wrote:
> > > >
> > > > I wonder if you have hit KAFKA-5600.
> > > >
> > > > Is it possible that you try out 0.11.0.1 ?
> > > >
> > > > Thanks
> > > >
> > > >> On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin  >
> > > wrote:
> > > >>
> > > >> I am using 0.11.0.0.
> > > >>
> > > >> There is no difference configuration-wise - both have 10 partitions
> > and
> > > 2
> > > >> replicas. There are no errors in the logs, but looking in the data
> > > folder
> > > >> it seems like Kafka is not updating the timeindex file for
> data1_log -
> > > >> notice how the timeindex file for the current log segment is not
> being
> > > >> updated.
> > > >>
> > > >> bash-4.2$ pwd
> > > >> /kafka/data/data1_log-1
> > > >> bash-4.2$ ls -ltr | tail
> > > >> -rw-rw-r-- 1 ibiuser it 1073731573 Oct 25 01:21
> > 000337554984.log
> > > >> -rw-rw-r-- 1 ibiuser it 943616 Oct 25 01:21
> > > 000337554984.index
> > > >> -rw-rw-r-- 1 ibiuser it 1073734199 Oct 25 13:38
> > 000339816017.log
> > > >> -rw-rw-r-- 1 ibiuser it   10485756 Oct 25 13:38
> > > >> 000341934289.timeindex
> > > >> -rw-rw-r-- 1 ibiuser it 10 Oct 25 13:38
> > > >> 000341934289.snapshot
> > > >> -rw-rw-r-- 1 ibiuser it  0 Oct 25 13:38
> > > >> 000339816017.timeindex
> > > >> -rw-rw-r-- 1 ibiuser it 566712 Oct 25 13:38
> > > 000339816017.index
> > > >> -rw-rw-r-- 1 ibiuser it 17 Oct 25 20:23
> > leader-epoch-checkpoint
> > > >> -rw-rw-r-- 1 ibiuser it   10485760 Oct 25 23:03
> > > 000341934289.index
> > > >> -rw-rw-r-- 1 ibiuser it  461590419 Oct 25 23:04
> > 000341934289.log
> > > >>
> > > >> For comparison, the beats topic:
> > > >>
> > > >> bash-4.2$ cd ../beats-1
> > > >> bash-4.2$ ls -ltr
> > > >> total 3212088
> > > >> -rw-rw-r-- 1 ibiuser it 17 Oct 25 00:23
> > leader-epoch-checkpoint
> > > >> -rw-rw-r-- 1 ibiuser it 10 Oct 25 20:04
> > > >> 000188672034.snapshot
> > > >> -rw-rw-r-- 1 ibiuser it2773008 Oct 25 20:04
> > > >> 000185224087.timeindex
> > > >> -rw-rw-r-- 1 ibiuser it 1073741779 Oct 25 20:04
> > 000185224087.log
> > > >> -rw-rw-r-- 1 ibiuser i

Re: Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior

[Eric Lalonde]

> On Oct 24, 2017, at 8:53 PM, Matthias J. Sax  wrote:
> 
> Might be worth a try with 1.0.0 RC3 -- even if I doubt that much changes.
> 
> Can you provide debug logs for your Kafka streams applications as well
> as brokers? This would help to dig into this.
> 

I searched, but it wasn’t clear to me — would a Kstream 1.0.0 RC3 client be 
backwards compatible with a broker from the 0.10.x line?

Re: Failure to reset consumer offsets for specific topics

[Kelly Shkuratoff]

Any idea why I started receiving mails from this list as of 2:43 am today?
I didn't make any changes or subscribe to anything. I even clicked
unsubscribe earlier today and am still receiving mails.
Maybe there's a misconfiguration in your email list?

Kelly

On Wed, Oct 25, 2017 at 1:44 PM, Hans Jespersen  wrote:

> I think you are just missing the —execute flag.
>
> -hans
>
> > On Oct 25, 2017, at 1:24 PM, Ted Yu  wrote:
> >
> > I wonder if you have hit KAFKA-5600.
> >
> > Is it possible that you try out 0.11.0.1 ?
> >
> > Thanks
> >
> >> On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin 
> wrote:
> >>
> >> I am using 0.11.0.0.
> >>
> >> There is no difference configuration-wise - both have 10 partitions and
> 2
> >> replicas. There are no errors in the logs, but looking in the data
> folder
> >> it seems like Kafka is not updating the timeindex file for data1_log -
> >> notice how the timeindex file for the current log segment is not being
> >> updated.
> >>
> >> bash-4.2$ pwd
> >> /kafka/data/data1_log-1
> >> bash-4.2$ ls -ltr | tail
> >> -rw-rw-r-- 1 ibiuser it 1073731573 Oct 25 01:21 000337554984.log
> >> -rw-rw-r-- 1 ibiuser it 943616 Oct 25 01:21
> 000337554984.index
> >> -rw-rw-r-- 1 ibiuser it 1073734199 Oct 25 13:38 000339816017.log
> >> -rw-rw-r-- 1 ibiuser it   10485756 Oct 25 13:38
> >> 000341934289.timeindex
> >> -rw-rw-r-- 1 ibiuser it 10 Oct 25 13:38
> >> 000341934289.snapshot
> >> -rw-rw-r-- 1 ibiuser it  0 Oct 25 13:38
> >> 000339816017.timeindex
> >> -rw-rw-r-- 1 ibiuser it 566712 Oct 25 13:38
> 000339816017.index
> >> -rw-rw-r-- 1 ibiuser it 17 Oct 25 20:23 leader-epoch-checkpoint
> >> -rw-rw-r-- 1 ibiuser it   10485760 Oct 25 23:03
> 000341934289.index
> >> -rw-rw-r-- 1 ibiuser it  461590419 Oct 25 23:04 000341934289.log
> >>
> >> For comparison, the beats topic:
> >>
> >> bash-4.2$ cd ../beats-1
> >> bash-4.2$ ls -ltr
> >> total 3212088
> >> -rw-rw-r-- 1 ibiuser it 17 Oct 25 00:23 leader-epoch-checkpoint
> >> -rw-rw-r-- 1 ibiuser it 10 Oct 25 20:04
> >> 000188672034.snapshot
> >> -rw-rw-r-- 1 ibiuser it2773008 Oct 25 20:04
> >> 000185224087.timeindex
> >> -rw-rw-r-- 1 ibiuser it 1073741779 Oct 25 20:04 000185224087.log
> >> -rw-rw-r-- 1 ibiuser it1967440 Oct 25 20:04
> 000185224087.index
> >> -rw-rw-r-- 1 ibiuser it   10485760 Oct 25 23:03
> 000188672034.index
> >> -rw-rw-r-- 1 ibiuser it   10485756 Oct 25 23:04
> >> 000188672034.timeindex
> >> -rw-rw-r-- 1 ibiuser it   50166645 Oct 25 23:04 000188672034.log
> >>
> >>
> >> To give some context to why I'm even trying to reset the offsets, we had
> >> encountered a strange situation earlier today:
> >>
> >> 1) One of the brokers had a hardware failure, and had to be rebuilt from
> >> scratch (data partition was gone)
> >> 2) When it went down, we noticed a spike in lag in one particular
> consumer
> >> group - it seems to have reset its offset to an earlier point in time
> (but
> >> not the earliest offset of the topic); I have read other messages on
> this
> >> mailing list of users who experienced the same behavior with 0.11.0.0
> >> 3) The broker was reinstalled and rejoined the cluster with the same
> >> broker.id (but with no data on it) - it rebalanced and eventually all
> >> replicas became synced and the cluster was functioning normally.
> >> 4) I then decided to bounce the same broker again to see if I can
> reproduce
> >> the issue I saw in #2 - and as soon as the broker was restarted, the
> exact
> >> same consumer group had its offset reset again and was lagging with
> >> millions of records behind the current offset.
> >> 5) I then tried to manually reset the consumer group's offset to a few
> >> minutes before I restarted the broker, only to discover this strange
> >> behavior where no matter which datetime value I provided, it kept
> resetting
> >> to the latest offset.
> >>
> >>
> >>> On 25 October 2017 at 22:48, Ted Yu  wrote:
> >>>
> >>> Do you mind providing a bit more information ?
> >>>
> >>> Release of Kafka you use
> >>>
> >>> Any difference between data1_log and the other, normal topic ?
> >>>
> >>> Probably check the broker log where data1_log is hosted - see if there
> is
> >>> some clue.
> >>>
> >>> Thanks
> >>>
> >>> On Wed, Oct 25, 2017 at 12:11 PM, Dan Markhasin 
> >>> wrote:
> >>>
>  I'm trying to use the kafka-consumer-groups.sh tool in order to rewind
> >> a
>  consumer group's offset, however it seems to be returning the latest
> >>> offset
>  regarding of the requested offset.
> 
>  You can see in the below example that two consecutive commands to
> reset
> >>> the
>  offset to a specific point in time return different (increasing)
> >> offsets,
>  which are actually the latest offsets for the topic.
> 
>  - The consumer group ("test_consumer") is a console consumer that was
>  started

Re: Failure to reset consumer offsets for specific topics

[Elyahou Ittah]

Which driver is used yo produce these messages ?

On Oct 26, 2017 07:11, "Dan Markhasin"  wrote:

> No, that flag doesn't affect which offsets are returned, only executes the
> action (and resets the consumer to latest offset when used, regardless of
> datetime value I provide).
>
> On 25 October 2017 at 23:44, Hans Jespersen  wrote:
>
> > I think you are just missing the —execute flag.
> >
> > -hans
> >
> > > On Oct 25, 2017, at 1:24 PM, Ted Yu  wrote:
> > >
> > > I wonder if you have hit KAFKA-5600.
> > >
> > > Is it possible that you try out 0.11.0.1 ?
> > >
> > > Thanks
> > >
> > >> On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin 
> > wrote:
> > >>
> > >> I am using 0.11.0.0.
> > >>
> > >> There is no difference configuration-wise - both have 10 partitions
> and
> > 2
> > >> replicas. There are no errors in the logs, but looking in the data
> > folder
> > >> it seems like Kafka is not updating the timeindex file for data1_log -
> > >> notice how the timeindex file for the current log segment is not being
> > >> updated.
> > >>
> > >> bash-4.2$ pwd
> > >> /kafka/data/data1_log-1
> > >> bash-4.2$ ls -ltr | tail
> > >> -rw-rw-r-- 1 ibiuser it 1073731573 Oct 25 01:21
> 000337554984.log
> > >> -rw-rw-r-- 1 ibiuser it 943616 Oct 25 01:21
> > 000337554984.index
> > >> -rw-rw-r-- 1 ibiuser it 1073734199 Oct 25 13:38
> 000339816017.log
> > >> -rw-rw-r-- 1 ibiuser it   10485756 Oct 25 13:38
> > >> 000341934289.timeindex
> > >> -rw-rw-r-- 1 ibiuser it 10 Oct 25 13:38
> > >> 000341934289.snapshot
> > >> -rw-rw-r-- 1 ibiuser it  0 Oct 25 13:38
> > >> 000339816017.timeindex
> > >> -rw-rw-r-- 1 ibiuser it 566712 Oct 25 13:38
> > 000339816017.index
> > >> -rw-rw-r-- 1 ibiuser it 17 Oct 25 20:23
> leader-epoch-checkpoint
> > >> -rw-rw-r-- 1 ibiuser it   10485760 Oct 25 23:03
> > 000341934289.index
> > >> -rw-rw-r-- 1 ibiuser it  461590419 Oct 25 23:04
> 000341934289.log
> > >>
> > >> For comparison, the beats topic:
> > >>
> > >> bash-4.2$ cd ../beats-1
> > >> bash-4.2$ ls -ltr
> > >> total 3212088
> > >> -rw-rw-r-- 1 ibiuser it 17 Oct 25 00:23
> leader-epoch-checkpoint
> > >> -rw-rw-r-- 1 ibiuser it 10 Oct 25 20:04
> > >> 000188672034.snapshot
> > >> -rw-rw-r-- 1 ibiuser it2773008 Oct 25 20:04
> > >> 000185224087.timeindex
> > >> -rw-rw-r-- 1 ibiuser it 1073741779 Oct 25 20:04
> 000185224087.log
> > >> -rw-rw-r-- 1 ibiuser it1967440 Oct 25 20:04
> > 000185224087.index
> > >> -rw-rw-r-- 1 ibiuser it   10485760 Oct 25 23:03
> > 000188672034.index
> > >> -rw-rw-r-- 1 ibiuser it   10485756 Oct 25 23:04
> > >> 000188672034.timeindex
> > >> -rw-rw-r-- 1 ibiuser it   50166645 Oct 25 23:04
> 000188672034.log
> > >>
> > >>
> > >> To give some context to why I'm even trying to reset the offsets, we
> had
> > >> encountered a strange situation earlier today:
> > >>
> > >> 1) One of the brokers had a hardware failure, and had to be rebuilt
> from
> > >> scratch (data partition was gone)
> > >> 2) When it went down, we noticed a spike in lag in one particular
> > consumer
> > >> group - it seems to have reset its offset to an earlier point in time
> > (but
> > >> not the earliest offset of the topic); I have read other messages on
> > this
> > >> mailing list of users who experienced the same behavior with 0.11.0.0
> > >> 3) The broker was reinstalled and rejoined the cluster with the same
> > >> broker.id (but with no data on it) - it rebalanced and eventually all
> > >> replicas became synced and the cluster was functioning normally.
> > >> 4) I then decided to bounce the same broker again to see if I can
> > reproduce
> > >> the issue I saw in #2 - and as soon as the broker was restarted, the
> > exact
> > >> same consumer group had its offset reset again and was lagging with
> > >> millions of records behind the current offset.
> > >> 5) I then tried to manually reset the consumer group's offset to a few
> > >> minutes before I restarted the broker, only to discover this strange
> > >> behavior where no matter which datetime value I provided, it kept
> > resetting
> > >> to the latest offset.
> > >>
> > >>
> > >>> On 25 October 2017 at 22:48, Ted Yu  wrote:
> > >>>
> > >>> Do you mind providing a bit more information ?
> > >>>
> > >>> Release of Kafka you use
> > >>>
> > >>> Any difference between data1_log and the other, normal topic ?
> > >>>
> > >>> Probably check the broker log where data1_log is hosted - see if
> there
> > is
> > >>> some clue.
> > >>>
> > >>> Thanks
> > >>>
> > >>> On Wed, Oct 25, 2017 at 12:11 PM, Dan Markhasin  >
> > >>> wrote:
> > >>>
> >  I'm trying to use the kafka-consumer-groups.sh tool in order to
> rewind
> > >> a
> >  consumer group's offset, however it seems to be returning the latest
> > >>> offset
> >  regarding of the requested offset.
> > 
> >  You can see in the below example that two cons

Re: Failure to reset consumer offsets for specific topics

[Dan Markhasin]

No, that flag doesn't affect which offsets are returned, only executes the
action (and resets the consumer to latest offset when used, regardless of
datetime value I provide).

On 25 October 2017 at 23:44, Hans Jespersen  wrote:

> I think you are just missing the —execute flag.
>
> -hans
>
> > On Oct 25, 2017, at 1:24 PM, Ted Yu  wrote:
> >
> > I wonder if you have hit KAFKA-5600.
> >
> > Is it possible that you try out 0.11.0.1 ?
> >
> > Thanks
> >
> >> On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin 
> wrote:
> >>
> >> I am using 0.11.0.0.
> >>
> >> There is no difference configuration-wise - both have 10 partitions and
> 2
> >> replicas. There are no errors in the logs, but looking in the data
> folder
> >> it seems like Kafka is not updating the timeindex file for data1_log -
> >> notice how the timeindex file for the current log segment is not being
> >> updated.
> >>
> >> bash-4.2$ pwd
> >> /kafka/data/data1_log-1
> >> bash-4.2$ ls -ltr | tail
> >> -rw-rw-r-- 1 ibiuser it 1073731573 Oct 25 01:21 000337554984.log
> >> -rw-rw-r-- 1 ibiuser it 943616 Oct 25 01:21
> 000337554984.index
> >> -rw-rw-r-- 1 ibiuser it 1073734199 Oct 25 13:38 000339816017.log
> >> -rw-rw-r-- 1 ibiuser it   10485756 Oct 25 13:38
> >> 000341934289.timeindex
> >> -rw-rw-r-- 1 ibiuser it 10 Oct 25 13:38
> >> 000341934289.snapshot
> >> -rw-rw-r-- 1 ibiuser it  0 Oct 25 13:38
> >> 000339816017.timeindex
> >> -rw-rw-r-- 1 ibiuser it 566712 Oct 25 13:38
> 000339816017.index
> >> -rw-rw-r-- 1 ibiuser it 17 Oct 25 20:23 leader-epoch-checkpoint
> >> -rw-rw-r-- 1 ibiuser it   10485760 Oct 25 23:03
> 000341934289.index
> >> -rw-rw-r-- 1 ibiuser it  461590419 Oct 25 23:04 000341934289.log
> >>
> >> For comparison, the beats topic:
> >>
> >> bash-4.2$ cd ../beats-1
> >> bash-4.2$ ls -ltr
> >> total 3212088
> >> -rw-rw-r-- 1 ibiuser it 17 Oct 25 00:23 leader-epoch-checkpoint
> >> -rw-rw-r-- 1 ibiuser it 10 Oct 25 20:04
> >> 000188672034.snapshot
> >> -rw-rw-r-- 1 ibiuser it2773008 Oct 25 20:04
> >> 000185224087.timeindex
> >> -rw-rw-r-- 1 ibiuser it 1073741779 Oct 25 20:04 000185224087.log
> >> -rw-rw-r-- 1 ibiuser it1967440 Oct 25 20:04
> 000185224087.index
> >> -rw-rw-r-- 1 ibiuser it   10485760 Oct 25 23:03
> 000188672034.index
> >> -rw-rw-r-- 1 ibiuser it   10485756 Oct 25 23:04
> >> 000188672034.timeindex
> >> -rw-rw-r-- 1 ibiuser it   50166645 Oct 25 23:04 000188672034.log
> >>
> >>
> >> To give some context to why I'm even trying to reset the offsets, we had
> >> encountered a strange situation earlier today:
> >>
> >> 1) One of the brokers had a hardware failure, and had to be rebuilt from
> >> scratch (data partition was gone)
> >> 2) When it went down, we noticed a spike in lag in one particular
> consumer
> >> group - it seems to have reset its offset to an earlier point in time
> (but
> >> not the earliest offset of the topic); I have read other messages on
> this
> >> mailing list of users who experienced the same behavior with 0.11.0.0
> >> 3) The broker was reinstalled and rejoined the cluster with the same
> >> broker.id (but with no data on it) - it rebalanced and eventually all
> >> replicas became synced and the cluster was functioning normally.
> >> 4) I then decided to bounce the same broker again to see if I can
> reproduce
> >> the issue I saw in #2 - and as soon as the broker was restarted, the
> exact
> >> same consumer group had its offset reset again and was lagging with
> >> millions of records behind the current offset.
> >> 5) I then tried to manually reset the consumer group's offset to a few
> >> minutes before I restarted the broker, only to discover this strange
> >> behavior where no matter which datetime value I provided, it kept
> resetting
> >> to the latest offset.
> >>
> >>
> >>> On 25 October 2017 at 22:48, Ted Yu  wrote:
> >>>
> >>> Do you mind providing a bit more information ?
> >>>
> >>> Release of Kafka you use
> >>>
> >>> Any difference between data1_log and the other, normal topic ?
> >>>
> >>> Probably check the broker log where data1_log is hosted - see if there
> is
> >>> some clue.
> >>>
> >>> Thanks
> >>>
> >>> On Wed, Oct 25, 2017 at 12:11 PM, Dan Markhasin 
> >>> wrote:
> >>>
>  I'm trying to use the kafka-consumer-groups.sh tool in order to rewind
> >> a
>  consumer group's offset, however it seems to be returning the latest
> >>> offset
>  regarding of the requested offset.
> 
>  You can see in the below example that two consecutive commands to
> reset
> >>> the
>  offset to a specific point in time return different (increasing)
> >> offsets,
>  which are actually the latest offsets for the topic.
> 
>  - The consumer group ("test_consumer") is a console consumer that was
>  started with --from-beginning and terminated after a few seconds, just
>  enough for it t

Re: Consumer poll returning 0 results

[Konstantine Karantasis]

Are you producing any records after you start the consumer?

By default, Kafka consumer starts with auto.offset.reset == latest (
https://kafka.apache.org/documentation/#newconsumerconfigs), which means
that if the consumer doesn't find a previous offset for its consumer group
(e.g. the first time the consumer runs) it will start consuming from the
latest offset and on. Therefore, if there are no new records produced to
this Kafka topic after the consumer is started (specifically after has a
partitions assigned to it), the consumer won't return anything.

To try the above snippet, you have two easy options:

1) Produce records after you start the consumer (e.g. with
kafka-console-producer)
2) Set auto.offset.reset to earliest for this consumer (e.g. in you code
above, props.put("auto.offset.reset", "earliest"); ). Test equivalent
behavior with a command such as:
bin/kafka-console-consumer --bootstrap-server localhost:9092 --topic test
--from-beginning

Omitting "--from-beginning" will be equivalent to what you observe above.

Konstantine

On Wed, Oct 25, 2017 at 6:48 PM, Ted Yu  wrote:

> Can you provide a bit more information ?
>
> Release of Kafka
> Java / Scala version
>
> Thanks
>
> On Wed, Oct 25, 2017 at 6:40 PM, Susheel Kumar 
> wrote:
>
> > Hello Kafka Users,
> >
> > I am trying to run below sample code mentioned in Kafka documentation
> under
> > Automatic Offset Committing for a topic with 1 partition  (tried with 3
> and
> > more partition as well). Create command as follows
> >
> > bin/kafka-topics.sh --create --zookeeper :2181 --replication-factor 3
> > --partitions 1 --topic test --config cleanup.policy=compact,delete
> >
> > but the sample code always returns 0 records unless I provide a custom
> > ConsumerRebalanceListener (below) which sets consumer to beginning.
> >
> > I wonder if the sample code given at Kafka documentation is wrong or am I
> > missing something?
> >
> > https://kafka.apache.org/0101/javadoc/index.html?org/apache/
> > kafka/clients/consumer/KafkaConsumer.html
> >
> >
> > *Automatic Offset Committing*
> >
> > This example demonstrates a simple usage of Kafka's consumer api that
> > relying on automatic offset committing.
> >
> >  Properties props = new Properties();
> >  props.put("bootstrap.servers", "localhost:9092");
> >  props.put("group.id", "test");
> >  props.put("enable.auto.commit", "true");
> >  props.put("auto.commit.interval.ms", "1000");
> >  props.put("key.deserializer",
> > "org.apache.kafka.common.serialization.StringDeserializer");
> >  props.put("value.deserializer",
> > "org.apache.kafka.common.serialization.StringDeserializer");
> >  KafkaConsumer consumer = new KafkaConsumer<>(props);
> >  consumer.subscribe(Arrays.asList("foo", "bar"));
> >  while (true) {
> >  ConsumerRecords records = consumer.poll(100);
> >  for (ConsumerRecord record : records)
> >  System.out.printf("offset = %d, key = %s, value = %s%n",
> > record.offset(), record.key(), record.value());
> >  }
> >
> >
> >
> > 
> >
> > public class SeekToBeginingConsumerRebalancerListener implements
> > org.apache.kafka.clients.consumer.ConsumerRebalanceListener {
> >
> >  private Consumer consumer;
> >  public SeekToBeginingConsumerRebalancerListener(KafkaConsumer<
> String,
> > String> consumer2) {
> >  this.consumer = consumer2;
> >  }
> >  public void onPartitionsRevoked(Collection
> > partitions) {
> >  for (TopicPartition partition : partitions) {
> >
> > //offsetManager.saveOffsetInExternalStore(partition.topic(),
> > partition.partition(),consumer.position(partition));
> >  }
> >  }
> >  public void onPartitionsAssigned(Collection
> > partitions) {
> > /* for (TopicPartition partition : partitions) {
> >  consumer.seek(partition,seekTo));
> >  }*/
> >  consumer.seekToBeginning(partitions);
> >  }
> > }
> >
>

Re: Consumer poll returning 0 results

[Ted Yu]

Can you provide a bit more information ?

Release of Kafka
Java / Scala version

Thanks

On Wed, Oct 25, 2017 at 6:40 PM, Susheel Kumar 
wrote:

> Hello Kafka Users,
>
> I am trying to run below sample code mentioned in Kafka documentation under
> Automatic Offset Committing for a topic with 1 partition  (tried with 3 and
> more partition as well). Create command as follows
>
> bin/kafka-topics.sh --create --zookeeper :2181 --replication-factor 3
> --partitions 1 --topic test --config cleanup.policy=compact,delete
>
> but the sample code always returns 0 records unless I provide a custom
> ConsumerRebalanceListener (below) which sets consumer to beginning.
>
> I wonder if the sample code given at Kafka documentation is wrong or am I
> missing something?
>
> https://kafka.apache.org/0101/javadoc/index.html?org/apache/
> kafka/clients/consumer/KafkaConsumer.html
>
>
> *Automatic Offset Committing*
>
> This example demonstrates a simple usage of Kafka's consumer api that
> relying on automatic offset committing.
>
>  Properties props = new Properties();
>  props.put("bootstrap.servers", "localhost:9092");
>  props.put("group.id", "test");
>  props.put("enable.auto.commit", "true");
>  props.put("auto.commit.interval.ms", "1000");
>  props.put("key.deserializer",
> "org.apache.kafka.common.serialization.StringDeserializer");
>  props.put("value.deserializer",
> "org.apache.kafka.common.serialization.StringDeserializer");
>  KafkaConsumer consumer = new KafkaConsumer<>(props);
>  consumer.subscribe(Arrays.asList("foo", "bar"));
>  while (true) {
>  ConsumerRecords records = consumer.poll(100);
>  for (ConsumerRecord record : records)
>  System.out.printf("offset = %d, key = %s, value = %s%n",
> record.offset(), record.key(), record.value());
>  }
>
>
>
> 
>
> public class SeekToBeginingConsumerRebalancerListener implements
> org.apache.kafka.clients.consumer.ConsumerRebalanceListener {
>
>  private Consumer consumer;
>  public SeekToBeginingConsumerRebalancerListener(KafkaConsumer String> consumer2) {
>  this.consumer = consumer2;
>  }
>  public void onPartitionsRevoked(Collection
> partitions) {
>  for (TopicPartition partition : partitions) {
>
> //offsetManager.saveOffsetInExternalStore(partition.topic(),
> partition.partition(),consumer.position(partition));
>  }
>  }
>  public void onPartitionsAssigned(Collection
> partitions) {
> /* for (TopicPartition partition : partitions) {
>  consumer.seek(partition,seekTo));
>  }*/
>  consumer.seekToBeginning(partitions);
>  }
> }
>

Consumer poll returning 0 results

[Susheel Kumar]

Hello Kafka Users,

I am trying to run below sample code mentioned in Kafka documentation under
Automatic Offset Committing for a topic with 1 partition  (tried with 3 and
more partition as well). Create command as follows

bin/kafka-topics.sh --create --zookeeper :2181 --replication-factor 3
--partitions 1 --topic test --config cleanup.policy=compact,delete

but the sample code always returns 0 records unless I provide a custom
ConsumerRebalanceListener (below) which sets consumer to beginning.

I wonder if the sample code given at Kafka documentation is wrong or am I
missing something?

https://kafka.apache.org/0101/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html


*Automatic Offset Committing*

This example demonstrates a simple usage of Kafka's consumer api that
relying on automatic offset committing.

 Properties props = new Properties();
 props.put("bootstrap.servers", "localhost:9092");
 props.put("group.id", "test");
 props.put("enable.auto.commit", "true");
 props.put("auto.commit.interval.ms", "1000");
 props.put("key.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
 props.put("value.deserializer",
"org.apache.kafka.common.serialization.StringDeserializer");
 KafkaConsumer consumer = new KafkaConsumer<>(props);
 consumer.subscribe(Arrays.asList("foo", "bar"));
 while (true) {
 ConsumerRecords records = consumer.poll(100);
 for (ConsumerRecord record : records)
 System.out.printf("offset = %d, key = %s, value = %s%n",
record.offset(), record.key(), record.value());
 }





public class SeekToBeginingConsumerRebalancerListener implements
org.apache.kafka.clients.consumer.ConsumerRebalanceListener {

 private Consumer consumer;
 public SeekToBeginingConsumerRebalancerListener(KafkaConsumer consumer2) {
 this.consumer = consumer2;
 }
 public void onPartitionsRevoked(Collection partitions) {
 for (TopicPartition partition : partitions) {

//offsetManager.saveOffsetInExternalStore(partition.topic(),
partition.partition(),consumer.position(partition));
 }
 }
 public void onPartitionsAssigned(Collection partitions) {
/* for (TopicPartition partition : partitions) {
 consumer.seek(partition,seekTo));
 }*/
 consumer.seekToBeginning(partitions);
 }
}

Re: [VOTE] 1.0.0 RC3

[Guozhang Wang]

Hello Vahid,

As we have discussed on the PR of KAFKA-6075, Windows is not a supported
platform for the Kafka broker at this point.

As for KAFKA-6100, I'll try to fix it by upgrading RocksDB to 5.7.3 in the
new RC (see below).


Folks,

We just found another blocker issue on the exactly-once feature (
https://issues.apache.org/jira/browse/KAFKA-6119) and a PR fix is
undergoing. After that fix and KAFKA-6100 has been merged we will quickly
roll out another RC.


Guozhang



On Wed, Oct 25, 2017 at 1:50 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Guozhang,
>
> +1 for Ubuntu and Mac: successfully built jars and tested quickstarts with
> this RC (using Java 8).
>
> -1 for Windows: because of KAFKA-6075 and KAFKA-6100. To me these two
> issues (which have simple workarounds) sound like "blockers" - unless
> Kafka does not officially support Windows.
>
> Thanks.
> --Vahid
>
>
>
> From:   Guozhang Wang 
> To: "d...@kafka.apache.org" ,
> "users@kafka.apache.org" , kafka-clients
> 
> Date:   10/23/2017 06:01 PM
> Subject:[VOTE] 1.0.0 RC3
>
>
>
> Hello Kafka users, developers and client-developers,
>
> This is the third candidate for release of Apache Kafka 1.0.0. The main
> PRs
> that gets merged in after RC1 are the following:
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_apache_kafka_commit_dc6bfa553e73ffccd1e604963e076c
> &d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> kjJc7uSVcviKUc&m=IXzIAtGVhimjespiK39c6G1WgcbBmQRf7y_fY5znRvE&s=
> gFHxZDDyvXlxWKAopousiqwYxn6CC1VCaN6vNljV0Jg&e=
>
> 78d8ddcd69
>
> It's worth noting that starting in this version we are using a different
> version protocol with three digits: *major.minor.bug-fix*
>
> Any and all testing is welcome, but the following areas are worth
> highlighting:
>
> 1. Client developers should verify that their clients can produce/consume
> to/from 1.0.0 brokers (ideally with compressed and uncompressed data).
> 2. Performance and stress testing. Heroku and LinkedIn have helped with
> this in the past (and issues have been found and fixed).
> 3. End users can verify that their apps work correctly with the new
> release.
>
> This is a major version release of Apache Kafka. It includes 29 new KIPs.
> See the release notes and release plan
> (*https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_
> confluence_pages_viewpage.action-3FpageId-3D71764913&d=
> DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> kjJc7uSVcviKUc&m=IXzIAtGVhimjespiK39c6G1WgcbBmQ
> Rf7y_fY5znRvE&s=GkBK7f7aFQ0q7_c5RYolZiLXrX4vyW5OkvE1As3To74&e=
> <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> apache.org_confluence_pages_viewpage.action-3FpageId-
> 3D71764913&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_
> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=IXzIAtGVhimjespiK39c6G1WgcbBmQ
> Rf7y_fY5znRvE&s=GkBK7f7aFQ0q7_c5RYolZiLXrX4vyW5OkvE1As3To74&e=
> >*)
> for more details. A few feature highlights:
>
> * Java 9 support with significantly faster TLS and CRC32C implementations
> * JBOD improvements: disk failure only disables failed disk but not the
> broker (KIP-112/KIP-113 part I)
> * Controller improvements: reduced logging change to greatly accelerate
> admin request handling.
> * Newly added metrics across all the modules (KIP-164, KIP-168, KIP-187,
> KIP-188, KIP-196)
> * Kafka Streams API improvements (KIP-120 / 130 / 138 / 150 / 160 / 161),
> and drop compatibility "Evolving" annotations
>
> Release notes for the 1.0.0 release:
> *https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-
> 7Eguozhang_kafka-2D1.0.0-2Drc3_RELEASE-5FNOTES.html&d=
> DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> kjJc7uSVcviKUc&m=IXzIAtGVhimjespiK39c6G1WgcbBmQ
> Rf7y_fY5znRvE&s=w97ukcWA-on6o_E7Z-cXqTpDkRrXueAbdPXFDGR7iuE&e=
> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.
> apache.org_-7Eguozhang_kafka-2D1.0.0-2Drc3_RELEASE-5FNOTES.
> html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> kjJc7uSVcviKUc&m=IXzIAtGVhimjespiK39c6G1WgcbBmQ
> Rf7y_fY5znRvE&s=w97ukcWA-on6o_E7Z-cXqTpDkRrXueAbdPXFDGR7iuE&e=
> >*
>
>
>
> *** Please download, test and vote by Friday, October 20, 8pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.
> apache.org_KEYS&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_
> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=IXzIAtGVhimjespiK39c6G1WgcbBmQ
> Rf7y_fY5znRvE&s=nnpGYoHO1DAO7pumwuwaTnFfYSJJUaD9230r0tXvtMA&e=
>
>
> * Release artifacts to be voted upon (source and binary):
> *https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-
> 7Eguozhang_kafka-2D1.0.0-2Drc3_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_
> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=IXzIAtGVhimjespiK39c6G1WgcbBmQ
> Rf7y_fY5znRvE&s=htdv00t1c_eD1U0it0rlNQFYSn6iGrnH6eTZFonRrrg&e=
> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.
> apache.org_-7Eguozha

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

[Matthias J. Sax]

Thanks a lot for the KIP. Can we please move the discussion to the dev list?

Thus, after fixing the KIP collision, just start a new DISCUSS thread.

Thx.


-Matthias

On 10/25/17 4:20 PM, Ted Yu wrote:
> Have you seen the email a moment ago from Onur which uses the same KIP
> number ?
> 
> Looks like there was race condition in modifying wiki.
> 
> Please consider bumping the KIP number.
> 
> Thanks
> 
> On Wed, Oct 25, 2017 at 4:14 PM, Jan Filipiak 
> wrote:
> 
>> Hello Kafka-users,
>>
>> I want to continue with the development of KAFKA-3705, which allows the
>> Streams DSL to perform KTableKTable-Joins when the KTables have a
>> one-to-many relationship.
>> To make sure we cover the requirements of as many users as possible and
>> have a good solution afterwards I invite everyone to read through the KIP I
>> put together and
>> discuss it here in this Thread.
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+
>> Support+non-key+joining+in+KTable
>> https://issues.apache.org/jira/browse/KAFKA-3705
>> https://github.com/apache/kafka/pull/3720
>>
>> I think a public discussion and vote on a solution is exactly what is
>> needed to bring this feauture into kafka-streams. I am looking forward to
>> everyones opinion!
>>
>> Please keep the discussion on the mailing list rather than commenting on
>> the wiki (wiki discussions get unwieldy fast).
>>
>> Best
>> Jan
>>
>>
>>
>>
>>
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

[Ted Yu]

Have you seen the email a moment ago from Onur which uses the same KIP
number ?

Looks like there was race condition in modifying wiki.

Please consider bumping the KIP number.

Thanks

On Wed, Oct 25, 2017 at 4:14 PM, Jan Filipiak 
wrote:

> Hello Kafka-users,
>
> I want to continue with the development of KAFKA-3705, which allows the
> Streams DSL to perform KTableKTable-Joins when the KTables have a
> one-to-many relationship.
> To make sure we cover the requirements of as many users as possible and
> have a good solution afterwards I invite everyone to read through the KIP I
> put together and
> discuss it here in this Thread.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+
> Support+non-key+joining+in+KTable
> https://issues.apache.org/jira/browse/KAFKA-3705
> https://github.com/apache/kafka/pull/3720
>
> I think a public discussion and vote on a solution is exactly what is
> needed to bring this feauture into kafka-streams. I am looking forward to
> everyones opinion!
>
> Please keep the discussion on the mailing list rather than commenting on
> the wiki (wiki discussions get unwieldy fast).
>
> Best
> Jan
>
>
>
>
>
>
>

[DISCUSS] KIP-213 Support non-key joining in KTable

[Jan Filipiak]

Hello Kafka-users,

I want to continue with the development of KAFKA-3705, which allows the 
Streams DSL to perform KTableKTable-Joins when the KTables have a 
one-to-many relationship.
To make sure we cover the requirements of as many users as possible and 
have a good solution afterwards I invite everyone to read through the 
KIP I put together and

discuss it here in this Thread.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+Support+non-key+joining+in+KTable
https://issues.apache.org/jira/browse/KAFKA-3705
https://github.com/apache/kafka/pull/3720

I think a public discussion and vote on a solution is exactly what is 
needed to bring this feauture into kafka-streams. I am looking forward 
to everyones opinion!


Please keep the discussion on the mailing list rather than commenting on 
the wiki (wiki discussions get unwieldy fast).


Best
Jan

Re: Failure to reset consumer offsets for specific topics

[Ted Yu]

Clarification: my most recent reply was w.r.t. the strange situation Dan
described, not the offset resetting.

On Wed, Oct 25, 2017 at 1:24 PM, Ted Yu  wrote:

> I wonder if you have hit KAFKA-5600.
>
> Is it possible that you try out 0.11.0.1 ?
>
> Thanks
>
> On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin 
> wrote:
>
>> I am using 0.11.0.0.
>>
>> There is no difference configuration-wise - both have 10 partitions and 2
>> replicas. There are no errors in the logs, but looking in the data folder
>> it seems like Kafka is not updating the timeindex file for data1_log -
>> notice how the timeindex file for the current log segment is not being
>> updated.
>>
>> bash-4.2$ pwd
>> /kafka/data/data1_log-1
>> bash-4.2$ ls -ltr | tail
>> -rw-rw-r-- 1 ibiuser it 1073731573 Oct 25 01:21 000337554984.log
>> -rw-rw-r-- 1 ibiuser it 943616 Oct 25 01:21 000337554984.index
>> -rw-rw-r-- 1 ibiuser it 1073734199 Oct 25 13:38 000339816017.log
>> -rw-rw-r-- 1 ibiuser it   10485756 Oct 25 13:38
>> 000341934289.timeindex
>> -rw-rw-r-- 1 ibiuser it 10 Oct 25 13:38
>> 000341934289.snapshot
>> -rw-rw-r-- 1 ibiuser it  0 Oct 25 13:38
>> 000339816017.timeindex
>> -rw-rw-r-- 1 ibiuser it 566712 Oct 25 13:38 000339816017.index
>> -rw-rw-r-- 1 ibiuser it 17 Oct 25 20:23 leader-epoch-checkpoint
>> -rw-rw-r-- 1 ibiuser it   10485760 Oct 25 23:03 000341934289.index
>> -rw-rw-r-- 1 ibiuser it  461590419 Oct 25 23:04 000341934289.log
>>
>> For comparison, the beats topic:
>>
>> bash-4.2$ cd ../beats-1
>> bash-4.2$ ls -ltr
>> total 3212088
>> -rw-rw-r-- 1 ibiuser it 17 Oct 25 00:23 leader-epoch-checkpoint
>> -rw-rw-r-- 1 ibiuser it 10 Oct 25 20:04
>> 000188672034.snapshot
>> -rw-rw-r-- 1 ibiuser it2773008 Oct 25 20:04
>> 000185224087.timeindex
>> -rw-rw-r-- 1 ibiuser it 1073741779 Oct 25 20:04 000185224087.log
>> -rw-rw-r-- 1 ibiuser it1967440 Oct 25 20:04 000185224087.index
>> -rw-rw-r-- 1 ibiuser it   10485760 Oct 25 23:03 000188672034.index
>> -rw-rw-r-- 1 ibiuser it   10485756 Oct 25 23:04
>> 000188672034.timeindex
>> -rw-rw-r-- 1 ibiuser it   50166645 Oct 25 23:04 000188672034.log
>>
>>
>> To give some context to why I'm even trying to reset the offsets, we had
>> encountered a strange situation earlier today:
>>
>> 1) One of the brokers had a hardware failure, and had to be rebuilt from
>> scratch (data partition was gone)
>> 2) When it went down, we noticed a spike in lag in one particular consumer
>> group - it seems to have reset its offset to an earlier point in time (but
>> not the earliest offset of the topic); I have read other messages on this
>> mailing list of users who experienced the same behavior with 0.11.0.0
>> 3) The broker was reinstalled and rejoined the cluster with the same
>> broker.id (but with no data on it) - it rebalanced and eventually all
>> replicas became synced and the cluster was functioning normally.
>> 4) I then decided to bounce the same broker again to see if I can
>> reproduce
>> the issue I saw in #2 - and as soon as the broker was restarted, the exact
>> same consumer group had its offset reset again and was lagging with
>> millions of records behind the current offset.
>> 5) I then tried to manually reset the consumer group's offset to a few
>> minutes before I restarted the broker, only to discover this strange
>> behavior where no matter which datetime value I provided, it kept
>> resetting
>> to the latest offset.
>>
>>
>> On 25 October 2017 at 22:48, Ted Yu  wrote:
>>
>> > Do you mind providing a bit more information ?
>> >
>> > Release of Kafka you use
>> >
>> > Any difference between data1_log and the other, normal topic ?
>> >
>> > Probably check the broker log where data1_log is hosted - see if there
>> is
>> > some clue.
>> >
>> > Thanks
>> >
>> > On Wed, Oct 25, 2017 at 12:11 PM, Dan Markhasin 
>> > wrote:
>> >
>> > > I'm trying to use the kafka-consumer-groups.sh tool in order to
>> rewind a
>> > > consumer group's offset, however it seems to be returning the latest
>> > offset
>> > > regarding of the requested offset.
>> > >
>> > > You can see in the below example that two consecutive commands to
>> reset
>> > the
>> > > offset to a specific point in time return different (increasing)
>> offsets,
>> > > which are actually the latest offsets for the topic.
>> > >
>> > > - The consumer group ("test_consumer") is a console consumer that was
>> > > started with --from-beginning and terminated after a few seconds, just
>> > > enough for it to commit its offsets.
>> > > - The topic data1_log is very busy with thousands of incoming messages
>> > per
>> > > second
>> > > - The datetime value provided is approx. 5 hours earlier than the
>> current
>> > > UTC time
>> > >
>> > > [admin@broker01] ~> /kafka/latest/bin/kafka-consumer-groups.sh
>> > > --bootstrap-server localhost:9092 --reset-offsets --group
>> test_con

Re: Failure to reset consumer offsets for specific topics

[Hans Jespersen]

I think you are just missing the —execute flag.

-hans

> On Oct 25, 2017, at 1:24 PM, Ted Yu  wrote:
> 
> I wonder if you have hit KAFKA-5600.
> 
> Is it possible that you try out 0.11.0.1 ?
> 
> Thanks
> 
>> On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin  wrote:
>> 
>> I am using 0.11.0.0.
>> 
>> There is no difference configuration-wise - both have 10 partitions and 2
>> replicas. There are no errors in the logs, but looking in the data folder
>> it seems like Kafka is not updating the timeindex file for data1_log -
>> notice how the timeindex file for the current log segment is not being
>> updated.
>> 
>> bash-4.2$ pwd
>> /kafka/data/data1_log-1
>> bash-4.2$ ls -ltr | tail
>> -rw-rw-r-- 1 ibiuser it 1073731573 Oct 25 01:21 000337554984.log
>> -rw-rw-r-- 1 ibiuser it 943616 Oct 25 01:21 000337554984.index
>> -rw-rw-r-- 1 ibiuser it 1073734199 Oct 25 13:38 000339816017.log
>> -rw-rw-r-- 1 ibiuser it   10485756 Oct 25 13:38
>> 000341934289.timeindex
>> -rw-rw-r-- 1 ibiuser it 10 Oct 25 13:38
>> 000341934289.snapshot
>> -rw-rw-r-- 1 ibiuser it  0 Oct 25 13:38
>> 000339816017.timeindex
>> -rw-rw-r-- 1 ibiuser it 566712 Oct 25 13:38 000339816017.index
>> -rw-rw-r-- 1 ibiuser it 17 Oct 25 20:23 leader-epoch-checkpoint
>> -rw-rw-r-- 1 ibiuser it   10485760 Oct 25 23:03 000341934289.index
>> -rw-rw-r-- 1 ibiuser it  461590419 Oct 25 23:04 000341934289.log
>> 
>> For comparison, the beats topic:
>> 
>> bash-4.2$ cd ../beats-1
>> bash-4.2$ ls -ltr
>> total 3212088
>> -rw-rw-r-- 1 ibiuser it 17 Oct 25 00:23 leader-epoch-checkpoint
>> -rw-rw-r-- 1 ibiuser it 10 Oct 25 20:04
>> 000188672034.snapshot
>> -rw-rw-r-- 1 ibiuser it2773008 Oct 25 20:04
>> 000185224087.timeindex
>> -rw-rw-r-- 1 ibiuser it 1073741779 Oct 25 20:04 000185224087.log
>> -rw-rw-r-- 1 ibiuser it1967440 Oct 25 20:04 000185224087.index
>> -rw-rw-r-- 1 ibiuser it   10485760 Oct 25 23:03 000188672034.index
>> -rw-rw-r-- 1 ibiuser it   10485756 Oct 25 23:04
>> 000188672034.timeindex
>> -rw-rw-r-- 1 ibiuser it   50166645 Oct 25 23:04 000188672034.log
>> 
>> 
>> To give some context to why I'm even trying to reset the offsets, we had
>> encountered a strange situation earlier today:
>> 
>> 1) One of the brokers had a hardware failure, and had to be rebuilt from
>> scratch (data partition was gone)
>> 2) When it went down, we noticed a spike in lag in one particular consumer
>> group - it seems to have reset its offset to an earlier point in time (but
>> not the earliest offset of the topic); I have read other messages on this
>> mailing list of users who experienced the same behavior with 0.11.0.0
>> 3) The broker was reinstalled and rejoined the cluster with the same
>> broker.id (but with no data on it) - it rebalanced and eventually all
>> replicas became synced and the cluster was functioning normally.
>> 4) I then decided to bounce the same broker again to see if I can reproduce
>> the issue I saw in #2 - and as soon as the broker was restarted, the exact
>> same consumer group had its offset reset again and was lagging with
>> millions of records behind the current offset.
>> 5) I then tried to manually reset the consumer group's offset to a few
>> minutes before I restarted the broker, only to discover this strange
>> behavior where no matter which datetime value I provided, it kept resetting
>> to the latest offset.
>> 
>> 
>>> On 25 October 2017 at 22:48, Ted Yu  wrote:
>>> 
>>> Do you mind providing a bit more information ?
>>> 
>>> Release of Kafka you use
>>> 
>>> Any difference between data1_log and the other, normal topic ?
>>> 
>>> Probably check the broker log where data1_log is hosted - see if there is
>>> some clue.
>>> 
>>> Thanks
>>> 
>>> On Wed, Oct 25, 2017 at 12:11 PM, Dan Markhasin 
>>> wrote:
>>> 
 I'm trying to use the kafka-consumer-groups.sh tool in order to rewind
>> a
 consumer group's offset, however it seems to be returning the latest
>>> offset
 regarding of the requested offset.
 
 You can see in the below example that two consecutive commands to reset
>>> the
 offset to a specific point in time return different (increasing)
>> offsets,
 which are actually the latest offsets for the topic.
 
 - The consumer group ("test_consumer") is a console consumer that was
 started with --from-beginning and terminated after a few seconds, just
 enough for it to commit its offsets.
 - The topic data1_log is very busy with thousands of incoming messages
>>> per
 second
 - The datetime value provided is approx. 5 hours earlier than the
>> current
 UTC time
 
 [admin@broker01] ~> /kafka/latest/bin/kafka-consumer-groups.sh
 --bootstrap-server localhost:9092 --reset-offsets --group test_consumer
 --topic data1_log --to-datetime '2017-10-25T13:40:00.000'
 Note: This will only sho

Re: [VOTE] 1.0.0 RC3

[Vahid S Hashemian]

Hi Guozhang,

+1 for Ubuntu and Mac: successfully built jars and tested quickstarts with 
this RC (using Java 8).

-1 for Windows: because of KAFKA-6075 and KAFKA-6100. To me these two 
issues (which have simple workarounds) sound like "blockers" - unless 
Kafka does not officially support Windows.

Thanks.
--Vahid



From:   Guozhang Wang 
To: "d...@kafka.apache.org" , 
"users@kafka.apache.org" , kafka-clients 

Date:   10/23/2017 06:01 PM
Subject:[VOTE] 1.0.0 RC3



Hello Kafka users, developers and client-developers,

This is the third candidate for release of Apache Kafka 1.0.0. The main 
PRs
that gets merged in after RC1 are the following:

https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_kafka_commit_dc6bfa553e73ffccd1e604963e076c&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=IXzIAtGVhimjespiK39c6G1WgcbBmQRf7y_fY5znRvE&s=gFHxZDDyvXlxWKAopousiqwYxn6CC1VCaN6vNljV0Jg&e=

78d8ddcd69

It's worth noting that starting in this version we are using a different
version protocol with three digits: *major.minor.bug-fix*

Any and all testing is welcome, but the following areas are worth
highlighting:

1. Client developers should verify that their clients can produce/consume
to/from 1.0.0 brokers (ideally with compressed and uncompressed data).
2. Performance and stress testing. Heroku and LinkedIn have helped with
this in the past (and issues have been found and fixed).
3. End users can verify that their apps work correctly with the new 
release.

This is a major version release of Apache Kafka. It includes 29 new KIPs.
See the release notes and release plan
(*https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_pages_viewpage.action-3FpageId-3D71764913&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=IXzIAtGVhimjespiK39c6G1WgcbBmQRf7y_fY5znRvE&s=GkBK7f7aFQ0q7_c5RYolZiLXrX4vyW5OkvE1As3To74&e=
<
https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_pages_viewpage.action-3FpageId-3D71764913&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=IXzIAtGVhimjespiK39c6G1WgcbBmQRf7y_fY5znRvE&s=GkBK7f7aFQ0q7_c5RYolZiLXrX4vyW5OkvE1As3To74&e=
>*)
for more details. A few feature highlights:

* Java 9 support with significantly faster TLS and CRC32C implementations
* JBOD improvements: disk failure only disables failed disk but not the
broker (KIP-112/KIP-113 part I)
* Controller improvements: reduced logging change to greatly accelerate
admin request handling.
* Newly added metrics across all the modules (KIP-164, KIP-168, KIP-187,
KIP-188, KIP-196)
* Kafka Streams API improvements (KIP-120 / 130 / 138 / 150 / 160 / 161),
and drop compatibility "Evolving" annotations

Release notes for the 1.0.0 release:
*https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Eguozhang_kafka-2D1.0.0-2Drc3_RELEASE-5FNOTES.html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=IXzIAtGVhimjespiK39c6G1WgcbBmQRf7y_fY5znRvE&s=w97ukcWA-on6o_E7Z-cXqTpDkRrXueAbdPXFDGR7iuE&e=
<
https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Eguozhang_kafka-2D1.0.0-2Drc3_RELEASE-5FNOTES.html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=IXzIAtGVhimjespiK39c6G1WgcbBmQRf7y_fY5znRvE&s=w97ukcWA-on6o_E7Z-cXqTpDkRrXueAbdPXFDGR7iuE&e=
>*



*** Please download, test and vote by Friday, October 20, 8pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.apache.org_KEYS&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=IXzIAtGVhimjespiK39c6G1WgcbBmQRf7y_fY5znRvE&s=nnpGYoHO1DAO7pumwuwaTnFfYSJJUaD9230r0tXvtMA&e=


* Release artifacts to be voted upon (source and binary):
*https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Eguozhang_kafka-2D1.0.0-2Drc3_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=IXzIAtGVhimjespiK39c6G1WgcbBmQRf7y_fY5znRvE&s=htdv00t1c_eD1U0it0rlNQFYSn6iGrnH6eTZFonRrrg&e=
<
https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Eguozhang_kafka-2D1.0.0-2Drc3_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=IXzIAtGVhimjespiK39c6G1WgcbBmQRf7y_fY5znRvE&s=htdv00t1c_eD1U0it0rlNQFYSn6iGrnH6eTZFonRrrg&e=
>*

* Maven artifacts to be voted upon:
https://urldefense.proofpoint.com/v2/url?u=https-3A__repository.apache.org_content_groups_staging_org_apache_kafka_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=IXzIAtGVhimjespiK39c6G1WgcbBmQRf7y_fY5znRvE&s=fMHq5kB4xP2FiFgdXT3FYM8ylac9FAr80joIjY4oL-s&e=


* Javadoc:
*https://urldefense.proofpoint.com/v2/url?u=http-3A__home.apache.org_-7Eguozhang_kafka-2D1.0.0-2Drc3_javadoc_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=IXzIAtGVhimjespiK39c6G1WgcbBmQRf7y_fY5znRvE&s=

Re: Failure to reset consumer offsets for specific topics

[Ted Yu]

I wonder if you have hit KAFKA-5600.

Is it possible that you try out 0.11.0.1 ?

Thanks

On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin  wrote:

> I am using 0.11.0.0.
>
> There is no difference configuration-wise - both have 10 partitions and 2
> replicas. There are no errors in the logs, but looking in the data folder
> it seems like Kafka is not updating the timeindex file for data1_log -
> notice how the timeindex file for the current log segment is not being
> updated.
>
> bash-4.2$ pwd
> /kafka/data/data1_log-1
> bash-4.2$ ls -ltr | tail
> -rw-rw-r-- 1 ibiuser it 1073731573 Oct 25 01:21 000337554984.log
> -rw-rw-r-- 1 ibiuser it 943616 Oct 25 01:21 000337554984.index
> -rw-rw-r-- 1 ibiuser it 1073734199 Oct 25 13:38 000339816017.log
> -rw-rw-r-- 1 ibiuser it   10485756 Oct 25 13:38
> 000341934289.timeindex
> -rw-rw-r-- 1 ibiuser it 10 Oct 25 13:38
> 000341934289.snapshot
> -rw-rw-r-- 1 ibiuser it  0 Oct 25 13:38
> 000339816017.timeindex
> -rw-rw-r-- 1 ibiuser it 566712 Oct 25 13:38 000339816017.index
> -rw-rw-r-- 1 ibiuser it 17 Oct 25 20:23 leader-epoch-checkpoint
> -rw-rw-r-- 1 ibiuser it   10485760 Oct 25 23:03 000341934289.index
> -rw-rw-r-- 1 ibiuser it  461590419 Oct 25 23:04 000341934289.log
>
> For comparison, the beats topic:
>
> bash-4.2$ cd ../beats-1
> bash-4.2$ ls -ltr
> total 3212088
> -rw-rw-r-- 1 ibiuser it 17 Oct 25 00:23 leader-epoch-checkpoint
> -rw-rw-r-- 1 ibiuser it 10 Oct 25 20:04
> 000188672034.snapshot
> -rw-rw-r-- 1 ibiuser it2773008 Oct 25 20:04
> 000185224087.timeindex
> -rw-rw-r-- 1 ibiuser it 1073741779 Oct 25 20:04 000185224087.log
> -rw-rw-r-- 1 ibiuser it1967440 Oct 25 20:04 000185224087.index
> -rw-rw-r-- 1 ibiuser it   10485760 Oct 25 23:03 000188672034.index
> -rw-rw-r-- 1 ibiuser it   10485756 Oct 25 23:04
> 000188672034.timeindex
> -rw-rw-r-- 1 ibiuser it   50166645 Oct 25 23:04 000188672034.log
>
>
> To give some context to why I'm even trying to reset the offsets, we had
> encountered a strange situation earlier today:
>
> 1) One of the brokers had a hardware failure, and had to be rebuilt from
> scratch (data partition was gone)
> 2) When it went down, we noticed a spike in lag in one particular consumer
> group - it seems to have reset its offset to an earlier point in time (but
> not the earliest offset of the topic); I have read other messages on this
> mailing list of users who experienced the same behavior with 0.11.0.0
> 3) The broker was reinstalled and rejoined the cluster with the same
> broker.id (but with no data on it) - it rebalanced and eventually all
> replicas became synced and the cluster was functioning normally.
> 4) I then decided to bounce the same broker again to see if I can reproduce
> the issue I saw in #2 - and as soon as the broker was restarted, the exact
> same consumer group had its offset reset again and was lagging with
> millions of records behind the current offset.
> 5) I then tried to manually reset the consumer group's offset to a few
> minutes before I restarted the broker, only to discover this strange
> behavior where no matter which datetime value I provided, it kept resetting
> to the latest offset.
>
>
> On 25 October 2017 at 22:48, Ted Yu  wrote:
>
> > Do you mind providing a bit more information ?
> >
> > Release of Kafka you use
> >
> > Any difference between data1_log and the other, normal topic ?
> >
> > Probably check the broker log where data1_log is hosted - see if there is
> > some clue.
> >
> > Thanks
> >
> > On Wed, Oct 25, 2017 at 12:11 PM, Dan Markhasin 
> > wrote:
> >
> > > I'm trying to use the kafka-consumer-groups.sh tool in order to rewind
> a
> > > consumer group's offset, however it seems to be returning the latest
> > offset
> > > regarding of the requested offset.
> > >
> > > You can see in the below example that two consecutive commands to reset
> > the
> > > offset to a specific point in time return different (increasing)
> offsets,
> > > which are actually the latest offsets for the topic.
> > >
> > > - The consumer group ("test_consumer") is a console consumer that was
> > > started with --from-beginning and terminated after a few seconds, just
> > > enough for it to commit its offsets.
> > > - The topic data1_log is very busy with thousands of incoming messages
> > per
> > > second
> > > - The datetime value provided is approx. 5 hours earlier than the
> current
> > > UTC time
> > >
> > > [admin@broker01] ~> /kafka/latest/bin/kafka-consumer-groups.sh
> > > --bootstrap-server localhost:9092 --reset-offsets --group test_consumer
> > > --topic data1_log --to-datetime '2017-10-25T13:40:00.000'
> > > Note: This will only show information about consumers that use the Java
> > > consumer API (non-ZooKeeper-based consumers).
> > >
> > >
> > > TOPIC  PARTITION  NEW-OFFSET
> > > data1_log

Re: Failure to reset consumer offsets for specific topics

[Dan Markhasin]

I am using 0.11.0.0.

There is no difference configuration-wise - both have 10 partitions and 2
replicas. There are no errors in the logs, but looking in the data folder
it seems like Kafka is not updating the timeindex file for data1_log -
notice how the timeindex file for the current log segment is not being
updated.

bash-4.2$ pwd
/kafka/data/data1_log-1
bash-4.2$ ls -ltr | tail
-rw-rw-r-- 1 ibiuser it 1073731573 Oct 25 01:21 000337554984.log
-rw-rw-r-- 1 ibiuser it 943616 Oct 25 01:21 000337554984.index
-rw-rw-r-- 1 ibiuser it 1073734199 Oct 25 13:38 000339816017.log
-rw-rw-r-- 1 ibiuser it   10485756 Oct 25 13:38
000341934289.timeindex
-rw-rw-r-- 1 ibiuser it 10 Oct 25 13:38
000341934289.snapshot
-rw-rw-r-- 1 ibiuser it  0 Oct 25 13:38
000339816017.timeindex
-rw-rw-r-- 1 ibiuser it 566712 Oct 25 13:38 000339816017.index
-rw-rw-r-- 1 ibiuser it 17 Oct 25 20:23 leader-epoch-checkpoint
-rw-rw-r-- 1 ibiuser it   10485760 Oct 25 23:03 000341934289.index
-rw-rw-r-- 1 ibiuser it  461590419 Oct 25 23:04 000341934289.log

For comparison, the beats topic:

bash-4.2$ cd ../beats-1
bash-4.2$ ls -ltr
total 3212088
-rw-rw-r-- 1 ibiuser it 17 Oct 25 00:23 leader-epoch-checkpoint
-rw-rw-r-- 1 ibiuser it 10 Oct 25 20:04
000188672034.snapshot
-rw-rw-r-- 1 ibiuser it2773008 Oct 25 20:04
000185224087.timeindex
-rw-rw-r-- 1 ibiuser it 1073741779 Oct 25 20:04 000185224087.log
-rw-rw-r-- 1 ibiuser it1967440 Oct 25 20:04 000185224087.index
-rw-rw-r-- 1 ibiuser it   10485760 Oct 25 23:03 000188672034.index
-rw-rw-r-- 1 ibiuser it   10485756 Oct 25 23:04
000188672034.timeindex
-rw-rw-r-- 1 ibiuser it   50166645 Oct 25 23:04 000188672034.log


To give some context to why I'm even trying to reset the offsets, we had
encountered a strange situation earlier today:

1) One of the brokers had a hardware failure, and had to be rebuilt from
scratch (data partition was gone)
2) When it went down, we noticed a spike in lag in one particular consumer
group - it seems to have reset its offset to an earlier point in time (but
not the earliest offset of the topic); I have read other messages on this
mailing list of users who experienced the same behavior with 0.11.0.0
3) The broker was reinstalled and rejoined the cluster with the same
broker.id (but with no data on it) - it rebalanced and eventually all
replicas became synced and the cluster was functioning normally.
4) I then decided to bounce the same broker again to see if I can reproduce
the issue I saw in #2 - and as soon as the broker was restarted, the exact
same consumer group had its offset reset again and was lagging with
millions of records behind the current offset.
5) I then tried to manually reset the consumer group's offset to a few
minutes before I restarted the broker, only to discover this strange
behavior where no matter which datetime value I provided, it kept resetting
to the latest offset.


On 25 October 2017 at 22:48, Ted Yu  wrote:

> Do you mind providing a bit more information ?
>
> Release of Kafka you use
>
> Any difference between data1_log and the other, normal topic ?
>
> Probably check the broker log where data1_log is hosted - see if there is
> some clue.
>
> Thanks
>
> On Wed, Oct 25, 2017 at 12:11 PM, Dan Markhasin 
> wrote:
>
> > I'm trying to use the kafka-consumer-groups.sh tool in order to rewind a
> > consumer group's offset, however it seems to be returning the latest
> offset
> > regarding of the requested offset.
> >
> > You can see in the below example that two consecutive commands to reset
> the
> > offset to a specific point in time return different (increasing) offsets,
> > which are actually the latest offsets for the topic.
> >
> > - The consumer group ("test_consumer") is a console consumer that was
> > started with --from-beginning and terminated after a few seconds, just
> > enough for it to commit its offsets.
> > - The topic data1_log is very busy with thousands of incoming messages
> per
> > second
> > - The datetime value provided is approx. 5 hours earlier than the current
> > UTC time
> >
> > [admin@broker01] ~> /kafka/latest/bin/kafka-consumer-groups.sh
> > --bootstrap-server localhost:9092 --reset-offsets --group test_consumer
> > --topic data1_log --to-datetime '2017-10-25T13:40:00.000'
> > Note: This will only show information about consumers that use the Java
> > consumer API (non-ZooKeeper-based consumers).
> >
> >
> > TOPIC  PARTITION  NEW-OFFSET
> > data1_log  8  301485420
> > data1_log  1  342788637
> > data1_log  7  287621428
> > data1_log  3  268612266
> > data1_log  0  201860717
> > data1_log  9  202749553
> > data1_log  4 

Re: Failure to reset consumer offsets for specific topics

[Ted Yu]

Do you mind providing a bit more information ?

Release of Kafka you use

Any difference between data1_log and the other, normal topic ?

Probably check the broker log where data1_log is hosted - see if there is
some clue.

Thanks

On Wed, Oct 25, 2017 at 12:11 PM, Dan Markhasin  wrote:

> I'm trying to use the kafka-consumer-groups.sh tool in order to rewind a
> consumer group's offset, however it seems to be returning the latest offset
> regarding of the requested offset.
>
> You can see in the below example that two consecutive commands to reset the
> offset to a specific point in time return different (increasing) offsets,
> which are actually the latest offsets for the topic.
>
> - The consumer group ("test_consumer") is a console consumer that was
> started with --from-beginning and terminated after a few seconds, just
> enough for it to commit its offsets.
> - The topic data1_log is very busy with thousands of incoming messages per
> second
> - The datetime value provided is approx. 5 hours earlier than the current
> UTC time
>
> [admin@broker01] ~> /kafka/latest/bin/kafka-consumer-groups.sh
> --bootstrap-server localhost:9092 --reset-offsets --group test_consumer
> --topic data1_log --to-datetime '2017-10-25T13:40:00.000'
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
>
>
> TOPIC  PARTITION  NEW-OFFSET
> data1_log  8  301485420
> data1_log  1  342788637
> data1_log  7  287621428
> data1_log  3  268612266
> data1_log  0  201860717
> data1_log  9  202749553
> data1_log  4  188974032
> data1_log  6  234308481
> data1_log  2  263507741
> data1_log  5  232707238
>
> [admin@broker01] ~> /kafka/latest/bin/kafka-consumer-groups.sh
> --bootstrap-server localhost:9092 --reset-offsets --group test_consumer
> --topic data1_log --to-datetime '2017-10-25T13:40:00.000'
> Note: This will only show information about consumers that use the Java
> consumer API (non-ZooKeeper-based consumers).
>
>
> TOPIC  PARTITION  NEW-OFFSET
> data1_log  8  301485491
> data1_log  1  342788779
> data1_log  7  287621534
> data1_log  3  268612364
> data1_log  0  201860796
> data1_log  9  202749620
> data1_log  4  188974068
> data1_log  6  234308564
> data1_log  2  263507823
> data1_log  5  232707293
>
> This issue seems to be topic-specific - there is a different topic (also
> very active) where the same command consistently returns the correct
> offsets fixed in the time for the requested datetime.
>
> What could be the issue here?
>
> Thanks,
> Dan
>

Re: "Persistence strategy" clarification

[Andrea Giordano]

Ok, it has sense.
So I have to think that the Kafka write strategy (writing directly to the disk 
if I understood) is “overwritten” by the Linux page-caching strategy? 
Anyway this invalid everything because it store data on the RAM before using 
disk space... 

> On 25 Oct 2017, at 20:51, Fabio Yamada  wrote:
> 
> Hi Andrea,
> 
> Although I'm not experienced in Kafka I could find reference in the docs to
> explain the behavior you experienced in the graph analysis. Following an
> excerpt from the official documentation:
> 
> Understanding Linux OS Flush Behavior
> 
> 
> In Linux, data written to the filesystem is maintained in pagecache until
> it must be written out to disk (due to an application-level fsync or the
> OS's own flush policy). The flushing of data is done by a set of background
> threads called pdflush (or in post 2.6.32 kernels "flusher threads").
> Pdflush has a configurable policy that controls how much dirty data can be
> maintained in cache and for how long before it must be written back to
> disk. This policy is described here. When Pdflush cannot keep up with the
> rate of data being written it will eventually cause the writing process to
> block incurring latency in the writes to slow down the accumulation of data.
> 
> Basically the flat area you are seeing in the graph is the pagecache
> building up by the OS. Then the data is flushed to disk at once and the
> cycle repeats.
> 
> Regards,
> Yamada
> 
> On Wed, Oct 25, 2017 at 1:44 PM, Andrea Giordano <
> andrea.giordano@gmail.com> wrote:
> 
>> Hi,
>> I’m looking at Kafka documentation, in particular at Persistence section:
>> 
>> https://kafka.apache.org/documentation/#persistence
>> 
>> If I understood it says that Kafka writes on disks data as they arrive
>> instead of use RAM. It sounds really strange to me (writes on disks are not
>> heavy operations?) but clearly I trust kafka developers. I would like to
>> have a confirm of that.
>> 
>> Assuming it and to verify it I executed a simple task with a data stream
>> of 500kb/s for some minutes on a  machine with 4GB-200GB and I printed
>> graphs of ram memory usage(%) and disk space usage (MB) you can find here:
>> 
>> 
>> RAM : https://ibb.co/mzYD5m
>> 
>> DISK SPACE: https://ibb.co/coAMrR
>> 
>> The stream is ingested at second 125 and finish at second around 870.
>> 
>> Accordingly to what I understood, I expected to see a linear decreasing
>> graph (due to the gradually occupation of space as data arrive) about disk
>> space usage, instead I’m not able to explain why there are these plain
>> regions which indicate no other space is occupied in these seconds.
>> 
>> Thank you,
>> Andrea
>> 
>> 
>> 
>> 

Failure to reset consumer offsets for specific topics

[Dan Markhasin]

I'm trying to use the kafka-consumer-groups.sh tool in order to rewind a
consumer group's offset, however it seems to be returning the latest offset
regarding of the requested offset.

You can see in the below example that two consecutive commands to reset the
offset to a specific point in time return different (increasing) offsets,
which are actually the latest offsets for the topic.

- The consumer group ("test_consumer") is a console consumer that was
started with --from-beginning and terminated after a few seconds, just
enough for it to commit its offsets.
- The topic data1_log is very busy with thousands of incoming messages per
second
- The datetime value provided is approx. 5 hours earlier than the current
UTC time

[admin@broker01] ~> /kafka/latest/bin/kafka-consumer-groups.sh
--bootstrap-server localhost:9092 --reset-offsets --group test_consumer
--topic data1_log --to-datetime '2017-10-25T13:40:00.000'
Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).


TOPIC  PARTITION  NEW-OFFSET
data1_log  8  301485420
data1_log  1  342788637
data1_log  7  287621428
data1_log  3  268612266
data1_log  0  201860717
data1_log  9  202749553
data1_log  4  188974032
data1_log  6  234308481
data1_log  2  263507741
data1_log  5  232707238

[admin@broker01] ~> /kafka/latest/bin/kafka-consumer-groups.sh
--bootstrap-server localhost:9092 --reset-offsets --group test_consumer
--topic data1_log --to-datetime '2017-10-25T13:40:00.000'
Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).


TOPIC  PARTITION  NEW-OFFSET
data1_log  8  301485491
data1_log  1  342788779
data1_log  7  287621534
data1_log  3  268612364
data1_log  0  201860796
data1_log  9  202749620
data1_log  4  188974068
data1_log  6  234308564
data1_log  2  263507823
data1_log  5  232707293

This issue seems to be topic-specific - there is a different topic (also
very active) where the same command consistently returns the correct
offsets fixed in the time for the requested datetime.

What could be the issue here?

Thanks,
Dan

Re: "Persistence strategy" clarification

[Fabio Yamada]

Hi Andrea,

Although I'm not experienced in Kafka I could find reference in the docs to
explain the behavior you experienced in the graph analysis. Following an
excerpt from the official documentation:

Understanding Linux OS Flush Behavior


In Linux, data written to the filesystem is maintained in pagecache until
it must be written out to disk (due to an application-level fsync or the
OS's own flush policy). The flushing of data is done by a set of background
threads called pdflush (or in post 2.6.32 kernels "flusher threads").
Pdflush has a configurable policy that controls how much dirty data can be
maintained in cache and for how long before it must be written back to
disk. This policy is described here. When Pdflush cannot keep up with the
rate of data being written it will eventually cause the writing process to
block incurring latency in the writes to slow down the accumulation of data.

Basically the flat area you are seeing in the graph is the pagecache
building up by the OS. Then the data is flushed to disk at once and the
cycle repeats.

Regards,
Yamada

On Wed, Oct 25, 2017 at 1:44 PM, Andrea Giordano <
andrea.giordano@gmail.com> wrote:

> Hi,
> I’m looking at Kafka documentation, in particular at Persistence section:
>
> https://kafka.apache.org/documentation/#persistence
>
> If I understood it says that Kafka writes on disks data as they arrive
> instead of use RAM. It sounds really strange to me (writes on disks are not
> heavy operations?) but clearly I trust kafka developers. I would like to
> have a confirm of that.
>
> Assuming it and to verify it I executed a simple task with a data stream
> of 500kb/s for some minutes on a  machine with 4GB-200GB and I printed
> graphs of ram memory usage(%) and disk space usage (MB) you can find here:
>
>
> RAM : https://ibb.co/mzYD5m
>
> DISK SPACE: https://ibb.co/coAMrR
>
> The stream is ingested at second 125 and finish at second around 870.
>
> Accordingly to what I understood, I expected to see a linear decreasing
> graph (due to the gradually occupation of space as data arrive) about disk
> space usage, instead I’m not able to explain why there are these plain
> regions which indicate no other space is occupied in these seconds.
>
> Thank you,
> Andrea
>
>
>
>

"Persistence strategy" clarification

[Andrea Giordano]

Hi,
I’m looking at Kafka documentation, in particular at Persistence section:

https://kafka.apache.org/documentation/#persistence

If I understood it says that Kafka writes on disks data as they arrive instead 
of use RAM. It sounds really strange to me (writes on disks are not heavy 
operations?) but clearly I trust kafka developers. I would like to have a 
confirm of that.

Assuming it and to verify it I executed a simple task with a data stream of 
500kb/s for some minutes on a  machine with 4GB-200GB and I printed graphs of 
ram memory usage(%) and disk space usage (MB) you can find here:


RAM : https://ibb.co/mzYD5m 

DISK SPACE: https://ibb.co/coAMrR 

The stream is ingested at second 125 and finish at second around 870.

Accordingly to what I understood, I expected to see a linear decreasing graph 
(due to the gradually occupation of space as data arrive) about disk space 
usage, instead I’m not able to explain why there are these plain regions which 
indicate no other space is occupied in these seconds. 

Thank you,
Andrea

Re: Log Compaction Not Picking up Topic [solved]

[Elmar Weber]

Hello Xin, hello Jan,

worked perfectly. I did a build of an image based on 0.11.0.1 and 
applied the missing patch, cleaning went through and resulted in the 
expected size.


Thanks a lot for the quick help,
Elmar


On 10/25/2017 1:03 PM, Xin Li wrote:

Hey Elmar,
The only thing you need to do is upgrade,
Kafka track cleaned offset using cleaner-offset-checkpoint file.

Best,
Xin

Xin Li Data EngineeringXin.Li@ trivago.com 
www.trivago.com F +49 (0) 211 540 
65 115We're hiring! Check out our vacancies http://company.trivago.com/jobs/Court of 
registration: Amtsgericht Düsseldorf, HRB 51842
Managing directors: Rolf Schrömgens · Malte Siewert · Peter Vinnemeier · Andrej 
Lehnert · Johannes Thomas
trivago GmbH · Bennigsen-Platz 1 · D – 40474 Düsseldorf
* This email message may contain legally privileged and/or confidential 
information.
You are hereby notified that any disclosure, copying, distribution, or use of 
this email message is strictly prohibited.

On 25.10.17, 12:34, "Elmar Weber"  wrote:

 Hi,
 
 thanks, I'll give it a try, we run on Kubernetes so it's not a big issue

 to replicate the whole env including data.
 
 One question I'd have left:

 - How can I force a re-compaction over the whole topic? Because I guess
the Log Cleaner market everything so far as not able to clean, how
will it recheck the whole log?
 
 Best,

 Elmar
 
 
 
 
 On 10/25/2017 12:29 PM, Jan Filipiak wrote:

 > Hi,
 >
 > unfortunatly there is nothing trivial you could do here.
 > Without upgrading your kafkas you can only bounce the partition back and
 > forth
 > between brokers so they compact while its still small.
 >
 > With upgrading you could also just cherrypick this very commit or put a
 > logstatement to verify.
 >
 > Given the Logsizes your dealing with, I am very confident that this is
 > your issue.
 >
 > Best Jan
 >
 >
 > On 25.10.2017 12:21, Elmar Weber wrote:
 >> Hi,
 >>
 >> On 10/25/2017 12:15 PM, Xin Li wrote:
 >> > I think that is a bug, and  should be fixed in this task
 >> https://issues.apache.org/jira/browse/KAFKA-6030.
 >> > We experience that in our kafka cluster, we just check out the
 >> 11.0.2 version, build it ourselves.
 >>
 >> thanks for the hint, as it looks like a calculation issue, would it be
 >> possible to verify this by manually changing the clean ratio or some
 >> other settings?
 >>
 >> Best,
 >> Elmar
 >>
 >
 >
 
 

Re: Log Compaction Not Picking up Topic

[Xin Li]

Hey Elmar,
The only thing you need to do is upgrade, 
Kafka track cleaned offset using cleaner-offset-checkpoint file.

Best,
Xin

Xin Li Data EngineeringXin.Li@ trivago.com 
www.trivago.com F +49 (0) 211 
540 65 115We're hiring! Check out our vacancies 
http://company.trivago.com/jobs/Court of registration: Amtsgericht Düsseldorf, 
HRB 51842
Managing directors: Rolf Schrömgens · Malte Siewert · Peter Vinnemeier · Andrej 
Lehnert · Johannes Thomas
trivago GmbH · Bennigsen-Platz 1 · D – 40474 Düsseldorf
* This email message may contain legally privileged and/or confidential 
information.
You are hereby notified that any disclosure, copying, distribution, or use of 
this email message is strictly prohibited.

On 25.10.17, 12:34, "Elmar Weber"  wrote:

Hi,

thanks, I'll give it a try, we run on Kubernetes so it's not a big issue 
to replicate the whole env including data.

One question I'd have left:
- How can I force a re-compaction over the whole topic? Because I guess
   the Log Cleaner market everything so far as not able to clean, how
   will it recheck the whole log?

Best,
Elmar




On 10/25/2017 12:29 PM, Jan Filipiak wrote:
> Hi,
> 
> unfortunatly there is nothing trivial you could do here.
> Without upgrading your kafkas you can only bounce the partition back and 
> forth
> between brokers so they compact while its still small.
> 
> With upgrading you could also just cherrypick this very commit or put a 
> logstatement to verify.
> 
> Given the Logsizes your dealing with, I am very confident that this is 
> your issue.
> 
> Best Jan
> 
> 
> On 25.10.2017 12:21, Elmar Weber wrote:
>> Hi,
>>
>> On 10/25/2017 12:15 PM, Xin Li wrote:
>> > I think that is a bug, and  should be fixed in this task 
>> https://issues.apache.org/jira/browse/KAFKA-6030.
>> > We experience that in our kafka cluster, we just check out the 
>> 11.0.2 version, build it ourselves.
>>
>> thanks for the hint, as it looks like a calculation issue, would it be 
>> possible to verify this by manually changing the clean ratio or some 
>> other settings?
>>
>> Best,
>> Elmar
>>
> 
> 

Re: Log Compaction Not Picking up Topic

[Xin Li]

Hey, 
Because of the overflow the calculation for dirty ratios is minus, and I guess 
upgrade is the one time for good fix.

And we running that for quite a while, so far so good.

Best,
Xin

Xin Li Data EngineeringXin.Li@ trivago.com 
www.trivago.com F +49 (0) 211 
540 65 115We're hiring! Check out our vacancies 
http://company.trivago.com/jobs/Court of registration: Amtsgericht Düsseldorf, 
HRB 51842
Managing directors: Rolf Schrömgens · Malte Siewert · Peter Vinnemeier · Andrej 
Lehnert · Johannes Thomas
trivago GmbH · Bennigsen-Platz 1 · D – 40474 Düsseldorf
* This email message may contain legally privileged and/or confidential 
information.
You are hereby notified that any disclosure, copying, distribution, or use of 
this email message is strictly prohibited.

On 25.10.17, 12:21, "Elmar Weber"  wrote:

Hi,

On 10/25/2017 12:15 PM, Xin Li wrote:
 > I think that is a bug, and  should be fixed in this task 
https://issues.apache.org/jira/browse/KAFKA-6030.
 > We experience that in our kafka cluster, we just check out the 11.0.2 
version, build it ourselves.

thanks for the hint, as it looks like a calculation issue, would it be 
possible to verify this by manually changing the clean ratio or some 
other settings?

Best,
Elmar

Re: Log Compaction Not Picking up Topic

[Elmar Weber]

Hi,

thanks, I'll give it a try, we run on Kubernetes so it's not a big issue 
to replicate the whole env including data.


One question I'd have left:
- How can I force a re-compaction over the whole topic? Because I guess
  the Log Cleaner market everything so far as not able to clean, how
  will it recheck the whole log?

Best,
Elmar




On 10/25/2017 12:29 PM, Jan Filipiak wrote:

Hi,

unfortunatly there is nothing trivial you could do here.
Without upgrading your kafkas you can only bounce the partition back and 
forth

between brokers so they compact while its still small.

With upgrading you could also just cherrypick this very commit or put a 
logstatement to verify.


Given the Logsizes your dealing with, I am very confident that this is 
your issue.


Best Jan


On 25.10.2017 12:21, Elmar Weber wrote:

Hi,

On 10/25/2017 12:15 PM, Xin Li wrote:
> I think that is a bug, and  should be fixed in this task 
https://issues.apache.org/jira/browse/KAFKA-6030.
> We experience that in our kafka cluster, we just check out the 
11.0.2 version, build it ourselves.


thanks for the hint, as it looks like a calculation issue, would it be 
possible to verify this by manually changing the clean ratio or some 
other settings?


Best,
Elmar

Re: Log Compaction Not Picking up Topic

[Jan Filipiak]

Hi,

unfortunatly there is nothing trivial you could do here.
Without upgrading your kafkas you can only bounce the partition back and 
forth

between brokers so they compact while its still small.

With upgrading you could also just cherrypick this very commit or put a 
logstatement to verify.


Given the Logsizes your dealing with, I am very confident that this is 
your issue.


Best Jan


On 25.10.2017 12:21, Elmar Weber wrote:

Hi,

On 10/25/2017 12:15 PM, Xin Li wrote:
> I think that is a bug, and  should be fixed in this task 
https://issues.apache.org/jira/browse/KAFKA-6030.
> We experience that in our kafka cluster, we just check out the 
11.0.2 version, build it ourselves.


thanks for the hint, as it looks like a calculation issue, would it be 
possible to verify this by manually changing the clean ratio or some 
other settings?


Best,
Elmar

Re: Log Compaction Not Picking up Topic

[Elmar Weber]

Hi,

On 10/25/2017 12:15 PM, Xin Li wrote:
> I think that is a bug, and  should be fixed in this task 
https://issues.apache.org/jira/browse/KAFKA-6030.
> We experience that in our kafka cluster, we just check out the 11.0.2 
version, build it ourselves.


thanks for the hint, as it looks like a calculation issue, would it be 
possible to verify this by manually changing the clean ratio or some 
other settings?


Best,
Elmar

Re: Log Compaction Not Picking up Topic

[Xin Li]

Hey,
I think that is a bug, and  should be fixed in this task 
https://issues.apache.org/jira/browse/KAFKA-6030.
We experience that in our kafka cluster, we just check out the 11.0.2 version, 
build it ourselves.

Best,
Xin

Xin Li Data EngineeringXin.Li@ trivago.com 
www.trivago.com F +49 (0) 211 
540 65 115We're hiring! Check out our vacancies 
http://company.trivago.com/jobs/Court of registration: Amtsgericht Düsseldorf, 
HRB 51842
Managing directors: Rolf Schrömgens · Malte Siewert · Peter Vinnemeier · Andrej 
Lehnert · Johannes Thomas
trivago GmbH · Bennigsen-Platz 1 · D – 40474 Düsseldorf
* This email message may contain legally privileged and/or confidential 
information.
You are hereby notified that any disclosure, copying, distribution, or use of 
this email message is strictly prohibited.

On 25.10.17, 12:04, "Manikumar"  wrote:

any errors in log cleaner logs?

On Wed, Oct 25, 2017 at 3:12 PM, Elmar Weber  wrote:

> Hello,
>
> I'm having trouble getting Kafka to compact a topic. It's over 300GB and
> has enough segments to warrant cleaning. It should only be about 40 GB
> (there is a copy in a db that is unique on the key). Below are the configs
> we have (default broker) and topic override.
>
>
> Is there something I'm missing on which setting is overriding which one or
> something still wrongly?
>
> retention.ms and delete.retentions.ms I set manually after creation on
> the topic and some segments have been created already.
>
> Kafka version 0.11
>
> Server Defaults for new segments of the topic:
>
> The settings used when a new log was created for the topic:
>
> {compression.type -> producer, message.format.version -> 0.11.0-IV2,
> file.delete.delay.ms -> 6, max.message.bytes -> 2097152,
> min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime,
> min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false,
> min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096,
> unclean.leader.election.enable -> false, retention.bytes -> -1,
> delete.retention.ms -> 8640, cleanup.policy -> compact, flush.ms ->
> 9223372036854775807, segment.ms -> 60480, segment.bytes ->
> 1073741824, retention.ms -> -1, message.timestamp.difference.max.ms ->
> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages ->
> 9223372036854775807}
>
> Topic Overrides (overridden after creation).
>
> {retention.ms=360, delete.retention.ms=360,
> max.message.bytes=10485760, cleanup.policy=compact}
>
>
>
> The full server startup config:
>
> advertised.host.name = null
> advertised.listeners = null
> advertised.port = null
> alter.config.policy.class.name = null
> authorizer.class.name =
> auto.create.topics.enable = false
> auto.leader.rebalance.enable = true
> background.threads = 10
> broker.id = 1
> broker.id.generation.enable = true
> broker.rack = europe-west1-c
> compression.type = producer
> connections.max.idle.ms = 60
> controlled.shutdown.enable = true
> controlled.shutdown.max.retries = 3
> controlled.shutdown.retry.backoff.ms = 5000
> controller.socket.timeout.ms = 3
> create.topic.policy.class.name = null
> default.replication.factor = 1
> delete.records.purgatory.purge.interval.requests = 1
> delete.topic.enable = true
> fetch.purgatory.purge.interval.requests = 1000
> group.initial.rebalance.delay.ms = 0
> group.max.session.timeout.ms = 30
> group.min.session.timeout.ms = 6000
> host.name =
> inter.broker.listener.name = null
> inter.broker.protocol.version = 0.11.0-IV2
> leader.imbalance.check.interval.seconds = 300
> leader.imbalance.per.broker.percentage = 10
> listener.security.protocol.map = SSL:SSL,SASL_PLAINTEXT:SASL_PL
> AINTEXT,TRACE:TRACE,SASL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT
> listeners = null
> log.cleaner.backoff.ms = 15000
> log.cleaner.dedupe.buffer.size = 134217728
> log.cleaner.delete.retention.ms = 8640
> log.cleaner.enable = true
> log.cleaner.io.buffer.load.factor = 0.9
> log.cleaner.io.buffer.size = 524288
> log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
> log.cleaner.min.cleanable.ratio = 0.5
> log.cleaner.min.compaction.lag.ms = 0
> log.cleaner.threads = 1
> log.cleanup.policy = [delete]
> log.dir = /tmp/kafka-logs
> log.dirs = /var/lib/kafka/data/topics
> log.flush.interval.messages = 9223372036854775807
> log.flush.interval.ms = null
> log.flush.offset.checkpoint.interval.ms = 6
> log.flush.scheduler.interval.ms = 9223372036854775807
> log.flush.start.offset.checkpoint.interval.ms = 6
> log.index.interv

Re: Log Compaction Not Picking up Topic

[Elmar Weber]

On 10/25/2017 12:03 PM, Manikumar wrote:

any errors in log cleaner logs?


Not as far as I can see

root@kafka-1:/opt/kafka/logs# cat log-cleaner.log* | grep -i error
(empty)

However, I've seen that it actually did cleaning of the whole topic 
(excerpts below), but it didn't seem to find anything worth cleaning 
(size stayed the same).


Are there any global settings that could affect this?
I'm running a default config from Kafka, the only things changed are 
message size, topic creation/deletion and the defaults for retention. 
Which are all overwritten for this topic (see original post)


I'm writing a simple script to read and confirm the key distributions to 
debug further, but as the same data (without duplicates) is also written 
to a DB I'm pretty sure that the size is too big for not having gotten 
compacted.



root@kafka-1:/opt/kafka/logs# cat log-cleaner.log* | grep 
events.lg.aggregated
[2017-10-24 11:08:43,462] INFO Cleaner 0: Beginning cleaning of log 
events.lg.aggregated-0. (kafka.log.LogCleaner)
[2017-10-24 11:08:43,462] INFO Cleaner 0: Building offset map for 
events.lg.aggregated-0... (kafka.log.LogCleaner)
[2017-10-24 11:08:43,688] INFO Cleaner 0: Building offset map for log 
events.lg.aggregated-0 for 1 segments in offset range [0, 81308). 
(kafka.log.LogCleaner)
[2017-10-24 11:08:44,163] INFO Cleaner 0: Offset map for log 
events.lg.aggregated-0 complete. (kafka.log.LogCleaner)
[2017-10-24 11:08:44,165] INFO Cleaner 0: Cleaning log 
events.lg.aggregated-0 (cleaning prior to Tue Oct 24 11:08:30 UTC 2017, 
discarding tombstones prior to Thu Jan 01 00:00:00 UTC 1970)... 
(kafka.log.LogCleaner)
[2017-10-24 11:08:44,166] INFO Cleaner 0: Cleaning segment 0 in log 
events.lg.aggregated-0 (largest timestamp Tue Oct 24 11:08:30 UTC 2017) 
into 0, retaining deletes. (kafka.log.LogCleaner)
[2017-10-24 11:08:47,865] INFO Cleaner 0: Swapping in cleaned segment 0 
for segment(s) 0 in log events.lg.aggregated-0. (kafka.log.LogCleaner)
Log cleaner thread 0 cleaned log events.lg.aggregated-0 (dirty 
section = [0, 0])
[2017-10-24 11:10:47,875] INFO Cleaner 0: Beginning cleaning of log 
events.lg.aggregated-0. (kafka.log.LogCleaner)
[2017-10-24 11:10:47,875] INFO Cleaner 0: Building offset map for 
events.lg.aggregated-0... (kafka.log.LogCleaner)
[2017-10-24 11:10:47,910] INFO Cleaner 0: Building offset map for log 
events.lg.aggregated-0 for 1 segments in offset range [81308, 154902). 
(kafka.log.LogCleaner)
[2017-10-24 11:10:48,410] INFO Cleaner 0: Offset map for log 
events.lg.aggregated-0 complete. (kafka.log.LogCleaner)
[2017-10-24 11:10:48,411] INFO Cleaner 0: Cleaning log 
events.lg.aggregated-0 (cleaning prior to Tue Oct 24 11:10:32 UTC 2017, 
discarding tombstones prior to Mon Oct 23 11:08:30 UTC 2017)... 
(kafka.log.LogCleaner)
[2017-10-24 11:10:48,411] INFO Cleaner 0: Cleaning segment 0 in log 
events.lg.aggregated-0 (largest timestamp Tue Oct 24 11:08:30 UTC 2017) 
into 0, retaining deletes. (kafka.log.LogCleaner)
[2017-10-24 11:10:50,308] INFO Cleaner 0: Swapping in cleaned segment 0 
for segment(s) 0 in log events.lg.aggregated-0. (kafka.log.LogCleaner)
[2017-10-24 11:10:50,309] INFO Cleaner 0: Cleaning segment 81308 in log 
events.lg.aggregated-0 (largest timestamp Tue Oct 24 11:10:32 UTC 2017) 
into 81308, retaining deletes. (kafka.log.LogCleaner)
[2017-10-24 11:10:53,389] INFO Cleaner 0: Swapping in cleaned segment 
81308 for segment(s) 81308 in log events.lg.aggregated-0. 
(kafka.log.LogCleaner)
Log cleaner thread 0 cleaned log events.lg.aggregated-0 (dirty 
section = [81308, 81308])

Re: Log Compaction Not Picking up Topic

[Manikumar]

any errors in log cleaner logs?

On Wed, Oct 25, 2017 at 3:12 PM, Elmar Weber  wrote:

> Hello,
>
> I'm having trouble getting Kafka to compact a topic. It's over 300GB and
> has enough segments to warrant cleaning. It should only be about 40 GB
> (there is a copy in a db that is unique on the key). Below are the configs
> we have (default broker) and topic override.
>
>
> Is there something I'm missing on which setting is overriding which one or
> something still wrongly?
>
> retention.ms and delete.retentions.ms I set manually after creation on
> the topic and some segments have been created already.
>
> Kafka version 0.11
>
> Server Defaults for new segments of the topic:
>
> The settings used when a new log was created for the topic:
>
> {compression.type -> producer, message.format.version -> 0.11.0-IV2,
> file.delete.delay.ms -> 6, max.message.bytes -> 2097152,
> min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime,
> min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false,
> min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096,
> unclean.leader.election.enable -> false, retention.bytes -> -1,
> delete.retention.ms -> 8640, cleanup.policy -> compact, flush.ms ->
> 9223372036854775807, segment.ms -> 60480, segment.bytes ->
> 1073741824, retention.ms -> -1, message.timestamp.difference.max.ms ->
> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages ->
> 9223372036854775807}
>
> Topic Overrides (overridden after creation).
>
> {retention.ms=360, delete.retention.ms=360,
> max.message.bytes=10485760, cleanup.policy=compact}
>
>
>
> The full server startup config:
>
> advertised.host.name = null
> advertised.listeners = null
> advertised.port = null
> alter.config.policy.class.name = null
> authorizer.class.name =
> auto.create.topics.enable = false
> auto.leader.rebalance.enable = true
> background.threads = 10
> broker.id = 1
> broker.id.generation.enable = true
> broker.rack = europe-west1-c
> compression.type = producer
> connections.max.idle.ms = 60
> controlled.shutdown.enable = true
> controlled.shutdown.max.retries = 3
> controlled.shutdown.retry.backoff.ms = 5000
> controller.socket.timeout.ms = 3
> create.topic.policy.class.name = null
> default.replication.factor = 1
> delete.records.purgatory.purge.interval.requests = 1
> delete.topic.enable = true
> fetch.purgatory.purge.interval.requests = 1000
> group.initial.rebalance.delay.ms = 0
> group.max.session.timeout.ms = 30
> group.min.session.timeout.ms = 6000
> host.name =
> inter.broker.listener.name = null
> inter.broker.protocol.version = 0.11.0-IV2
> leader.imbalance.check.interval.seconds = 300
> leader.imbalance.per.broker.percentage = 10
> listener.security.protocol.map = SSL:SSL,SASL_PLAINTEXT:SASL_PL
> AINTEXT,TRACE:TRACE,SASL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT
> listeners = null
> log.cleaner.backoff.ms = 15000
> log.cleaner.dedupe.buffer.size = 134217728
> log.cleaner.delete.retention.ms = 8640
> log.cleaner.enable = true
> log.cleaner.io.buffer.load.factor = 0.9
> log.cleaner.io.buffer.size = 524288
> log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
> log.cleaner.min.cleanable.ratio = 0.5
> log.cleaner.min.compaction.lag.ms = 0
> log.cleaner.threads = 1
> log.cleanup.policy = [delete]
> log.dir = /tmp/kafka-logs
> log.dirs = /var/lib/kafka/data/topics
> log.flush.interval.messages = 9223372036854775807
> log.flush.interval.ms = null
> log.flush.offset.checkpoint.interval.ms = 6
> log.flush.scheduler.interval.ms = 9223372036854775807
> log.flush.start.offset.checkpoint.interval.ms = 6
> log.index.interval.bytes = 4096
> log.index.size.max.bytes = 10485760
> log.message.format.version = 0.11.0-IV2
> log.message.timestamp.difference.max.ms = 9223372036854775807
> log.message.timestamp.type = CreateTime
> log.preallocate = false
> log.retention.bytes = -1
> log.retention.check.interval.ms = 30
> log.retention.hours = -1
> log.retention.minutes = null
> log.retention.ms = null
> log.roll.hours = 168
> log.roll.jitter.hours = 0
> log.roll.jitter.ms = null
> log.roll.ms = null
> log.segment.bytes = 1073741824
> log.segment.delete.delay.ms = 6
> max.connections.per.ip = 2147483647
> max.connections.per.ip.overrides =
> message.max.bytes = 2097152
> metric.reporters = []
> metrics.num.samples = 2
> metrics.recording.level = INFO
> metrics.sample.window.ms = 3
> min.insync.replicas = 1
> num.io.threads = 8
> num.network.threads = 3
> num.partitions = 1
> num.recovery.threads.per.data.dir = 1
> num.replica.fetchers = 1
> offset.metadata.max.bytes = 4096
> offsets.commit.required.acks = -1
> offsets.commit.timeout.ms = 5000
> offsets.load.buffer.size = 5242880
> offsets.retention.check.interval.ms = 60
> offsets.retention.minutes = 1440
> offsets.topic.compression.codec = 0
> offsets.topic.num.partitions = 50
> offsets.topic.replication.factor = 1
> offsets.topic.segment.bytes = 104857600
> port = 9092
> principal.builder.c

Log Compaction Not Picking up Topic

[Elmar Weber]

Hello,

I'm having trouble getting Kafka to compact a topic. It's over 300GB and 
has enough segments to warrant cleaning. It should only be about 40 GB 
(there is a copy in a db that is unique on the key). Below are the 
configs we have (default broker) and topic override.



Is there something I'm missing on which setting is overriding which one 
or something still wrongly?


retention.ms and delete.retentions.ms I set manually after creation on 
the topic and some segments have been created already.


Kafka version 0.11

Server Defaults for new segments of the topic:

The settings used when a new log was created for the topic:

{compression.type -> producer, message.format.version -> 0.11.0-IV2, 
file.delete.delay.ms -> 6, max.message.bytes -> 2097152, 
min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, 
min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, 
min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, 
unclean.leader.election.enable -> false, retention.bytes -> -1, 
delete.retention.ms -> 8640, cleanup.policy -> compact, flush.ms -> 
9223372036854775807, segment.ms -> 60480, segment.bytes -> 
1073741824, retention.ms -> -1, message.timestamp.difference.max.ms -> 
9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 
9223372036854775807}


Topic Overrides (overridden after creation).

{retention.ms=360, delete.retention.ms=360, 
max.message.bytes=10485760, cleanup.policy=compact}




The full server startup config:

advertised.host.name = null
advertised.listeners = null
advertised.port = null
alter.config.policy.class.name = null
authorizer.class.name =
auto.create.topics.enable = false
auto.leader.rebalance.enable = true
background.threads = 10
broker.id = 1
broker.id.generation.enable = true
broker.rack = europe-west1-c
compression.type = producer
connections.max.idle.ms = 60
controlled.shutdown.enable = true
controlled.shutdown.max.retries = 3
controlled.shutdown.retry.backoff.ms = 5000
controller.socket.timeout.ms = 3
create.topic.policy.class.name = null
default.replication.factor = 1
delete.records.purgatory.purge.interval.requests = 1
delete.topic.enable = true
fetch.purgatory.purge.interval.requests = 1000
group.initial.rebalance.delay.ms = 0
group.max.session.timeout.ms = 30
group.min.session.timeout.ms = 6000
host.name =
inter.broker.listener.name = null
inter.broker.protocol.version = 0.11.0-IV2
leader.imbalance.check.interval.seconds = 300
leader.imbalance.per.broker.percentage = 10
listener.security.protocol.map = 
SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,TRACE:TRACE,SASL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT

listeners = null
log.cleaner.backoff.ms = 15000
log.cleaner.dedupe.buffer.size = 134217728
log.cleaner.delete.retention.ms = 8640
log.cleaner.enable = true
log.cleaner.io.buffer.load.factor = 0.9
log.cleaner.io.buffer.size = 524288
log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
log.cleaner.min.cleanable.ratio = 0.5
log.cleaner.min.compaction.lag.ms = 0
log.cleaner.threads = 1
log.cleanup.policy = [delete]
log.dir = /tmp/kafka-logs
log.dirs = /var/lib/kafka/data/topics
log.flush.interval.messages = 9223372036854775807
log.flush.interval.ms = null
log.flush.offset.checkpoint.interval.ms = 6
log.flush.scheduler.interval.ms = 9223372036854775807
log.flush.start.offset.checkpoint.interval.ms = 6
log.index.interval.bytes = 4096
log.index.size.max.bytes = 10485760
log.message.format.version = 0.11.0-IV2
log.message.timestamp.difference.max.ms = 9223372036854775807
log.message.timestamp.type = CreateTime
log.preallocate = false
log.retention.bytes = -1
log.retention.check.interval.ms = 30
log.retention.hours = -1
log.retention.minutes = null
log.retention.ms = null
log.roll.hours = 168
log.roll.jitter.hours = 0
log.roll.jitter.ms = null
log.roll.ms = null
log.segment.bytes = 1073741824
log.segment.delete.delay.ms = 6
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides =
message.max.bytes = 2097152
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 3
min.insync.replicas = 1
num.io.threads = 8
num.network.threads = 3
num.partitions = 1
num.recovery.threads.per.data.dir = 1
num.replica.fetchers = 1
offset.metadata.max.bytes = 4096
offsets.commit.required.acks = -1
offsets.commit.timeout.ms = 5000
offsets.load.buffer.size = 5242880
offsets.retention.check.interval.ms = 60
offsets.retention.minutes = 1440
offsets.topic.compression.codec = 0
offsets.topic.num.partitions = 50
offsets.topic.replication.factor = 1
offsets.topic.segment.bytes = 104857600
port = 9092
principal.builder.class = class 
org.apache.kafka.common.security.auth.DefaultPrincipalBuilder

producer.purgatory.purge.interval.requests = 1000
queued.max.requests = 500
quota.consumer.default = 9223372036854775807
quota.producer.default = 9223372036854775807
quota.window.num = 11
quota.window.size.seconds = 1
replica.fetch.backoff.m