Re: [VOTE] 1.1.1 RC1

[Dong Lin]

Thank you for testing and voting the release!

I noticed that the date for 1.1.1-rc1 is wrong. Please kindly test and vote
by Tuesday, June 26, 12 pm PT.

Thanks,
Dong

On Fri, Jun 22, 2018 at 10:09 AM, Dong Lin  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 1.1.1.
>
> Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was first
> released with 1.1.0 about 3 months ago. We have fixed about 25 issues since
> that release. A few of the more significant fixes include:
>
> KAFKA-6925  - Fix
> memory leak in StreamsMetricsThreadImpl
> KAFKA-6937  - In-sync
> replica delayed during fetch if replica throttle is exceeded
> KAFKA-6917  - Process
> txn completion asynchronously to avoid deadlock
> KAFKA-6893  - Create
> processors before starting acceptor to avoid ArithmeticException
> KAFKA-6870  -
> Fix ConcurrentModificationException in SampledStat
> KAFKA-6878  - Fix
> NullPointerException when querying global state store
> KAFKA-6879  - Invoke
> session init callbacks outside lock to avoid Controller deadlock
> KAFKA-6857  - Prevent
> follower from truncating to the wrong offset if undefined leader epoch is
> requested
> KAFKA-6854  - Log
> cleaner fails with transaction markers that are deleted during clean
> KAFKA-6747  - Check
> whether there is in-flight transaction before aborting transaction
> KAFKA-6748  - Double
> check before scheduling a new task after the punctuate call
> KAFKA-6739  -
> Fix IllegalArgumentException when down-converting from V2 to V0/V1
> KAFKA-6728  -
> Fix NullPointerException when instantiating the HeaderConverter
>
> Kafka 1.1.1 release plan:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
>
> Release notes for the 1.1.1 release:
> http://home.apache.org/~lindong/kafka-1.1.1-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Thursday, Jun 22, 12pm PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~lindong/kafka-1.1.1-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~lindong/kafka-1.1.1-rc1/javadoc/
>
> * Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc1 tag:
> https://github.com/apache/kafka/tree/1.1.1-rc1
>
> * Documentation:
> http://kafka.apache.org/11/documentation.html
>
> * Protocol:
> http://kafka.apache.org/11/protocol.html
>
> * Successful Jenkins builds for the 1.1 branch:
> Unit/integration tests: *https://builds.apache.org/job/kafka-1.1-jdk7/152/
> *
> System tests: https://jenkins.confluent.io/job/system-test-kafka-br
> anch-builder/1817
>
>
> Please test and verify the release artifacts and submit a vote for this RC,
> or report any issues so we can fix them and get a new RC out ASAP. Although
> this release vote requires PMC votes to pass, testing, votes, and bug
> reports are valuable and appreciated from everyone.
>
> Cheers,
> Dong
>
>
>

Re: [VOTE] 1.1.1 RC1

[Harsha]

+1 . 
 1. Ran unit tests. 
 2. Ran few tests on 3-node cluster

Thanks,
Harsha

On Fri, Jun 22nd, 2018 at 2:41 PM Jakob Homan wrote:

> 
> 
> 
> +1 (binding)
> 
> * verified sigs, NOTICE, LICENSE
> * ran unit tests
> * spot checked headers
> 
> -Jakob
> 
> 
> 
> On 22 June 2018 at 13:19, Ted Yu < yuzhih...@gmail.com > wrote:
> > +1
> >
> > Ran test suite.
> > Checked signatures.
> >
> > On Fri, Jun 22, 2018 at 12:14 PM, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com > wrote:
> >
> >> +1 (non-binding)
> >>
> >> Built from source and ran quickstart successfully on Ubuntu (with Java
> 8).
> >>
> >> Thanks Dong!
> >> --Vahid
> >>
> >>
> >>
> >> From: Dong Lin < lindon...@gmail.com >
> >> To: d...@kafka.apache.org , users@kafka.apache.org ,
> >> kafka-clie...@googlegroups.com
> >> Date: 06/22/2018 10:10 AM
> >> Subject: [VOTE] 1.1.1 RC1
> >>
> >>
> >>
> >> Hello Kafka users, developers and client-developers,
> >>
> >> This is the second candidate for release of Apache Kafka 1.1.1.
> >>
> >> Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was
> first
> >> released with 1.1.0 about 3 months ago. We have fixed about 25 issues
> >> since
> >> that release. A few of the more significant fixes include:
> >>
> >> KAFKA-6925 <
> >> https://issues.apache.org/jira/browse/KAFKA-6925
> >> > - Fix memory
> >> leak in StreamsMetricsThreadImpl
> >> KAFKA-6937 <
> >> https://issues.apache.org/jira/browse/KAFKA-6937
> >> > - In-sync
> >> replica delayed during fetch if replica throttle is exceeded
> >> KAFKA-6917 <
> >> https://issues.apache.org/jira/browse/KAFKA-6917
> >> > - Process txn
> >> completion asynchronously to avoid deadlock
> >> KAFKA-6893 <
> >> https://issues.apache.org/jira/browse/KAFKA-6893
> >> > - Create
> >> processors before starting acceptor to avoid ArithmeticException
> >> KAFKA-6870 <
> >> https://issues.apache.org/jira/browse/KAFKA-6870
> >> > -
> >> Fix ConcurrentModificationException in SampledStat
> >> KAFKA-6878 <
> >> https://issues.apache.org/jira/browse/KAFKA-6878
> >> > - Fix
> >> NullPointerException when querying global state store
> >> KAFKA-6879 <
> >> https://issues.apache.org/jira/browse/KAFKA-6879
> >> > - Invoke
> >> session init callbacks outside lock to avoid Controller deadlock
> >> KAFKA-6857 <
> >> https://issues.apache.org/jira/browse/KAFKA-6857
> >> > - Prevent
> >> follower from truncating to the wrong offset if undefined leader epoch
> is
> >> requested
> >> KAFKA-6854 <
> >> https://issues.apache.org/jira/browse/KAFKA-6854
> >> > - Log cleaner
> >> fails with transaction markers that are deleted during clean
> >> KAFKA-6747 <
> >> https://issues.apache.org/jira/browse/KAFKA-6747
> >> > - Check
> >> whether there is in-flight transaction before aborting transaction
> >> KAFKA-6748 <
> >> https://issues.apache.org/jira/browse/KAFKA-6748
> >> > - Double
> >> check before scheduling a new task after the punctuate call
> >> KAFKA-6739 <
> >> https://issues.apache.org/jira/browse/KAFKA-6739
> >> > -
> >> Fix IllegalArgumentException when down-converting from V2 to V0/V1
> >> KAFKA-6728 <
> >> https://issues.apache.org/jira/browse/KAFKA-6728
> >> > -
> >> Fix NullPointerException when instantiating the HeaderConverter
> >>
> >> Kafka 1.1.1 release plan:
> >> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
> >>
> >>
> >> Release notes for the 1.1.1 release:
> >> http://home.apache.org/~lindong/kafka-1.1.1-rc1/RELEASE_NOTES.html
> >>
> >>
> >> *** Please download, test and vote by Thursday, Jun 22, 12pm PT ***
> >>
> >> Kafka's KEYS file containing PGP keys we use to sign the release:
> >> http://kafka.apache.org/KEYS
> >>
> >>
> >> * Release artifacts to be voted upon (source and binary):
> >> http://home.apache.org/~lindong/kafka-1.1.1-rc1/
> >>
> >>
> >> * Maven artifacts to be voted upon:
> >> https://repository.apache.org/content/groups/staging/
> >>
> >>
> >> * Javadoc:
> >> http://home.apache.org/~lindong/kafka-1.1.1-rc1/javadoc/
> >>
> >>
> >> * Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc1 tag:
> >> https://github.com/apache/kafka/tree/1.1.1-rc1
> >>
> >>
> >> * Documentation:
> >> http://kafka.apache.org/11/documentation.html
> >>
> >>
> >> * Protocol:
> >> http://kafka.apache.org/11/protocol.html
> >>
> >>
> >> * Successful Jenkins builds for the 1.1 branch:
> >> Unit/integration tests:
> >> * https://builds.apache.org/job/kafka-1.1-jdk7/152/
> >> <
> >> https://builds.apache.org/job/kafka-1.1-jdk7/152/
> >> >*
> >> System tests:
> >> https://jenkins.confluent.io/job/system-test-
> >>
> >> kafka-branch-builder/1817
> >>
> >>
> >> Please test and verify the release artifacts and submit a vote for this
> 
> >> RC,
> >> or report any issues so we can fix them and get a new RC out ASAP.
> >> Although
> >> this release vote requires PMC votes to pass, testing, votes, and bug
> >> reports are valuable and appreciated from everyone.
> >>
> >> Cheers,
> >> Dong
> >>
> >>
> >>
> >>
> >>
> 
> 
> 
> 
> 
>

Re: Some Total and Rate metrics are not consistent

[Guozhang Wang]

Hello Sam,

We have just back ported this fix into the 1.1 and 1.0 branch now:
https://github.com/apache/kafka/pull/5277

The up coming 1.1.1 and 1.0.2 release should have this issue fixed.


Guozhang


On Thu, Jun 21, 2018 at 2:36 PM, Sam Lendle  wrote:

> Yep that’s the change I saw. It does look like the issue should be fixed
> since the implementation of update for Count always adds 1 regardless of
> the value.
>
> Thanks for the followup.
>
> On 6/21/18, 1:15 PM, "John Roesler"  wrote:
>
> Hi Sam,
>
> This sounds like a condition I fixed in
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_apache_kafka_commit_ed51b2cdf5bdac210a6904bead1a2ca6e8411406-23diff-
> 2D8b364ed2d0abd8e8ae21f5d322db6564R221=DwIFaQ=
> gFTBenQ7Vj71sUi1A4CkFnmPzqwDo07QsHw-JRepxyw=
> BNCekDhngyXB6C2Ag7PIfHotiuqjAVwLOZLQHB7fyOM=q-lWaxuP-yHvVqew6oPVJiQPM_
> xal1kSHyqFZKeXFe4=fHXuRWDvMLDx75JEh_h4Y50zooJ9Z03d77TnX4tnxzE=
> . I realized that the prior code creates a new Meter, which uses a
> Total
> metric instead of a Count. But that would total all the values of the
> metric, when instead what we want is to "total" the number of
> measurements
> (aka count them).
>
> I just peeked at the 1.1 branch, and it seems this change made it in
> after
> the 1.1 branch cut, so it would only be fixed in 2.0.
>
> Thanks,
> -John
>
> On Wed, Jun 20, 2018 at 8:44 PM Guozhang Wang 
> wrote:
>
> > Thanks for reporting this Sam, could you check and confirm if this
> issue is
> > fixed in trunk? If not, we should file a JIRA.
> >
> >
> > Guozhang
> >
> > On Wed, Jun 20, 2018 at 6:41 PM, Sam Lendle 
> wrote:
> >
> > > It looks like there is indeed a bug in kafka-streams 1.1.0. I
> think what
> > > was happening was the time spent processing each record in ns was
> being
> > > added to the total metric instead of incrementing by 1 for each
> record.
> > > Looks like the implementation has been changed in trunk. I don't
> see any
> > > commit messages mentioning this particular issue, but hopefully the
> > change
> > > fixes it.
> > > --
> > > *From:* Sam Lendle
> > > *Sent:* Wednesday, June 20, 2018 6:10:03 PM
> > > *To:* users@kafka.apache.org
> > > *Subject:* Some Total and Rate metrics are not consistent
> > >
> > >
> > > I’m trying to use the total metrics introduced in KIP-187 (
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> apache.org_confluence_display_KAFKA_KIP-2D=DwIFaQ=
> gFTBenQ7Vj71sUi1A4CkFnmPzqwDo07QsHw-JRepxyw=
> BNCekDhngyXB6C2Ag7PIfHotiuqjAVwLOZLQHB7fyOM=q-lWaxuP-yHvVqew6oPVJiQPM_
> xal1kSHyqFZKeXFe4=jYa-lA5HwT7eOoFDIhz215N6PbZerIuO9AQDqLpNYlQ=
> > > 187+-+Add+cumulative+count+metric+for+all+Kafka+rate+metrics)
> > >
> > >
> > >
> > > For some metrics, the total and rates are not consistent. In
> particular,
> > > for stream-processor-node-metrics, I’m seeing about 500-800
> operations
> > per
> > > second in a particular streams thread/processor node as reported
> by the
> > > process-rate metric, but the process-total metric is increasing by
> about
> > > 100 million per second. See attached screenshot from VisualVM.
> > >
> > >
> > >
> > > Other metrics seem fine, for example forward-rate and forward-total
> > > metrics under stream-processor-node-metrics are consistent.
> > >
> > >
> > >
> > > Am I misunderstanding the interpretation of the –total metrics? If
> this
> > is
> > > a bug, can I do anything in addition to this email to report it?
> File a
> > > JIRA?
> > >
> > >
> > > Best,
> > > Sam
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>


-- 
-- Guozhang

Re: [VOTE] 2.0.0 RC0

[Vahid S Hashemian]

+1 (non-binding)

Built from source and ran quickstart successfully on Ubuntu (with Java 8 
and Java 9).

Thanks Rajini!
--Vahid

[VOTE] 0.11.0.3 RC0

[Matthias J. Sax]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 0.11.0.3.

This is a bug fix release closing 27 tickets:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.11.0.3

Release notes for the 0.11.0.3 release:
http://home.apache.org/~mjsax/kafka-0.11.0.3-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Tuesday, 6/26/18 end-of-day, so we
can close the vote on Wednesday.

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~mjsax/kafka-0.11.0.3-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~mjsax/kafka-0.11.0.3-rc0/javadoc/

* Tag to be voted upon (off 0.11.0 branch) is the 0.11.0.3 tag:
https://github.com/apache/kafka/releases/tag/0.11.0.3-rc0

* Documentation:
http://kafka.apache.org/0110/documentation.html

* Protocol:
http://kafka.apache.org/0110/protocol.html

* Successful Jenkins builds for the 0.11.0 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-0.11.0-jdk7/385/
System tests:
https://jenkins.confluent.io/job/system-test-kafka/job/0.11.0/217/

/**

Thanks,
  -Matthias



signature.asc
Description: OpenPGP digital signature

Re: [VOTE] 1.1.1 RC1

[Jakob Homan]

+1 (binding)

* verified sigs, NOTICE, LICENSE
* ran unit tests
* spot checked headers

-Jakob

On 22 June 2018 at 13:19, Ted Yu  wrote:
> +1
>
> Ran test suite.
> Checked signatures.
>
> On Fri, Jun 22, 2018 at 12:14 PM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
>> +1 (non-binding)
>>
>> Built from source and ran quickstart successfully on Ubuntu (with Java 8).
>>
>> Thanks Dong!
>> --Vahid
>>
>>
>>
>> From:   Dong Lin 
>> To: d...@kafka.apache.org, users@kafka.apache.org,
>> kafka-clie...@googlegroups.com
>> Date:   06/22/2018 10:10 AM
>> Subject:[VOTE] 1.1.1 RC1
>>
>>
>>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the second candidate for release of Apache Kafka 1.1.1.
>>
>> Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was first
>> released with 1.1.0 about 3 months ago. We have fixed about 25 issues
>> since
>> that release. A few of the more significant fixes include:
>>
>> KAFKA-6925 <
>> https://issues.apache.org/jira/browse/KAFKA-6925
>> > - Fix memory
>> leak in StreamsMetricsThreadImpl
>> KAFKA-6937 <
>> https://issues.apache.org/jira/browse/KAFKA-6937
>> > - In-sync
>> replica delayed during fetch if replica throttle is exceeded
>> KAFKA-6917 <
>> https://issues.apache.org/jira/browse/KAFKA-6917
>> > - Process txn
>> completion asynchronously to avoid deadlock
>> KAFKA-6893 <
>> https://issues.apache.org/jira/browse/KAFKA-6893
>> > - Create
>> processors before starting acceptor to avoid ArithmeticException
>> KAFKA-6870 <
>> https://issues.apache.org/jira/browse/KAFKA-6870
>> > -
>> Fix ConcurrentModificationException in SampledStat
>> KAFKA-6878 <
>> https://issues.apache.org/jira/browse/KAFKA-6878
>> > - Fix
>> NullPointerException when querying global state store
>> KAFKA-6879 <
>> https://issues.apache.org/jira/browse/KAFKA-6879
>> > - Invoke
>> session init callbacks outside lock to avoid Controller deadlock
>> KAFKA-6857 <
>> https://issues.apache.org/jira/browse/KAFKA-6857
>> > - Prevent
>> follower from truncating to the wrong offset if undefined leader epoch is
>> requested
>> KAFKA-6854 <
>> https://issues.apache.org/jira/browse/KAFKA-6854
>> > - Log cleaner
>> fails with transaction markers that are deleted during clean
>> KAFKA-6747 <
>> https://issues.apache.org/jira/browse/KAFKA-6747
>> > - Check
>> whether there is in-flight transaction before aborting transaction
>> KAFKA-6748 <
>> https://issues.apache.org/jira/browse/KAFKA-6748
>> > - Double
>> check before scheduling a new task after the punctuate call
>> KAFKA-6739 <
>> https://issues.apache.org/jira/browse/KAFKA-6739
>> > -
>> Fix IllegalArgumentException when down-converting from V2 to V0/V1
>> KAFKA-6728 <
>> https://issues.apache.org/jira/browse/KAFKA-6728
>> > -
>> Fix NullPointerException when instantiating the HeaderConverter
>>
>> Kafka 1.1.1 release plan:
>> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
>>
>>
>> Release notes for the 1.1.1 release:
>> http://home.apache.org/~lindong/kafka-1.1.1-rc1/RELEASE_NOTES.html
>>
>>
>> *** Please download, test and vote by Thursday, Jun 22, 12pm PT ***
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~lindong/kafka-1.1.1-rc1/
>>
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/
>>
>>
>> * Javadoc:
>> http://home.apache.org/~lindong/kafka-1.1.1-rc1/javadoc/
>>
>>
>> * Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc1 tag:
>> https://github.com/apache/kafka/tree/1.1.1-rc1
>>
>>
>> * Documentation:
>> http://kafka.apache.org/11/documentation.html
>>
>>
>> * Protocol:
>> http://kafka.apache.org/11/protocol.html
>>
>>
>> * Successful Jenkins builds for the 1.1 branch:
>> Unit/integration tests:
>> *https://builds.apache.org/job/kafka-1.1-jdk7/152/
>> <
>> https://builds.apache.org/job/kafka-1.1-jdk7/152/
>> >*
>> System tests:
>> https://jenkins.confluent.io/job/system-test-
>>
>> kafka-branch-builder/1817
>>
>>
>> Please test and verify the release artifacts and submit a vote for this
>> RC,
>> or report any issues so we can fix them and get a new RC out ASAP.
>> Although
>> this release vote requires PMC votes to pass, testing, votes, and bug
>> reports are valuable and appreciated from everyone.
>>
>> Cheers,
>> Dong
>>
>>
>>
>>
>>

Re: [VOTE] 1.1.1 RC1

[Ted Yu]

+1

Ran test suite.
Checked signatures.

On Fri, Jun 22, 2018 at 12:14 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> +1 (non-binding)
>
> Built from source and ran quickstart successfully on Ubuntu (with Java 8).
>
> Thanks Dong!
> --Vahid
>
>
>
> From:   Dong Lin 
> To: d...@kafka.apache.org, users@kafka.apache.org,
> kafka-clie...@googlegroups.com
> Date:   06/22/2018 10:10 AM
> Subject:[VOTE] 1.1.1 RC1
>
>
>
> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 1.1.1.
>
> Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was first
> released with 1.1.0 about 3 months ago. We have fixed about 25 issues
> since
> that release. A few of the more significant fixes include:
>
> KAFKA-6925 <
> https://issues.apache.org/jira/browse/KAFKA-6925
> > - Fix memory
> leak in StreamsMetricsThreadImpl
> KAFKA-6937 <
> https://issues.apache.org/jira/browse/KAFKA-6937
> > - In-sync
> replica delayed during fetch if replica throttle is exceeded
> KAFKA-6917 <
> https://issues.apache.org/jira/browse/KAFKA-6917
> > - Process txn
> completion asynchronously to avoid deadlock
> KAFKA-6893 <
> https://issues.apache.org/jira/browse/KAFKA-6893
> > - Create
> processors before starting acceptor to avoid ArithmeticException
> KAFKA-6870 <
> https://issues.apache.org/jira/browse/KAFKA-6870
> > -
> Fix ConcurrentModificationException in SampledStat
> KAFKA-6878 <
> https://issues.apache.org/jira/browse/KAFKA-6878
> > - Fix
> NullPointerException when querying global state store
> KAFKA-6879 <
> https://issues.apache.org/jira/browse/KAFKA-6879
> > - Invoke
> session init callbacks outside lock to avoid Controller deadlock
> KAFKA-6857 <
> https://issues.apache.org/jira/browse/KAFKA-6857
> > - Prevent
> follower from truncating to the wrong offset if undefined leader epoch is
> requested
> KAFKA-6854 <
> https://issues.apache.org/jira/browse/KAFKA-6854
> > - Log cleaner
> fails with transaction markers that are deleted during clean
> KAFKA-6747 <
> https://issues.apache.org/jira/browse/KAFKA-6747
> > - Check
> whether there is in-flight transaction before aborting transaction
> KAFKA-6748 <
> https://issues.apache.org/jira/browse/KAFKA-6748
> > - Double
> check before scheduling a new task after the punctuate call
> KAFKA-6739 <
> https://issues.apache.org/jira/browse/KAFKA-6739
> > -
> Fix IllegalArgumentException when down-converting from V2 to V0/V1
> KAFKA-6728 <
> https://issues.apache.org/jira/browse/KAFKA-6728
> > -
> Fix NullPointerException when instantiating the HeaderConverter
>
> Kafka 1.1.1 release plan:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1
>
>
> Release notes for the 1.1.1 release:
> http://home.apache.org/~lindong/kafka-1.1.1-rc1/RELEASE_NOTES.html
>
>
> *** Please download, test and vote by Thursday, Jun 22, 12pm PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~lindong/kafka-1.1.1-rc1/
>
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
>
> * Javadoc:
> http://home.apache.org/~lindong/kafka-1.1.1-rc1/javadoc/
>
>
> * Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc1 tag:
> https://github.com/apache/kafka/tree/1.1.1-rc1
>
>
> * Documentation:
> http://kafka.apache.org/11/documentation.html
>
>
> * Protocol:
> http://kafka.apache.org/11/protocol.html
>
>
> * Successful Jenkins builds for the 1.1 branch:
> Unit/integration tests:
> *https://builds.apache.org/job/kafka-1.1-jdk7/152/
> <
> https://builds.apache.org/job/kafka-1.1-jdk7/152/
> >*
> System tests:
> https://jenkins.confluent.io/job/system-test-
>
> kafka-branch-builder/1817
>
>
> Please test and verify the release artifacts and submit a vote for this
> RC,
> or report any issues so we can fix them and get a new RC out ASAP.
> Although
> this release vote requires PMC votes to pass, testing, votes, and bug
> reports are valuable and appreciated from everyone.
>
> Cheers,
> Dong
>
>
>
>
>

Re: [VOTE] 1.1.1 RC1

[Vahid S Hashemian]

+1 (non-binding)

Built from source and ran quickstart successfully on Ubuntu (with Java 8).

Thanks Dong!
--Vahid



From:   Dong Lin 
To: d...@kafka.apache.org, users@kafka.apache.org, 
kafka-clie...@googlegroups.com
Date:   06/22/2018 10:10 AM
Subject:[VOTE] 1.1.1 RC1



Hello Kafka users, developers and client-developers,

This is the second candidate for release of Apache Kafka 1.1.1.

Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was first
released with 1.1.0 about 3 months ago. We have fixed about 25 issues 
since
that release. A few of the more significant fixes include:

KAFKA-6925 <
https://issues.apache.org/jira/browse/KAFKA-6925
> - Fix memory
leak in StreamsMetricsThreadImpl
KAFKA-6937 <
https://issues.apache.org/jira/browse/KAFKA-6937
> - In-sync
replica delayed during fetch if replica throttle is exceeded
KAFKA-6917 <
https://issues.apache.org/jira/browse/KAFKA-6917
> - Process txn
completion asynchronously to avoid deadlock
KAFKA-6893 <
https://issues.apache.org/jira/browse/KAFKA-6893
> - Create
processors before starting acceptor to avoid ArithmeticException
KAFKA-6870 <
https://issues.apache.org/jira/browse/KAFKA-6870
> -
Fix ConcurrentModificationException in SampledStat
KAFKA-6878 <
https://issues.apache.org/jira/browse/KAFKA-6878
> - Fix
NullPointerException when querying global state store
KAFKA-6879 <
https://issues.apache.org/jira/browse/KAFKA-6879
> - Invoke
session init callbacks outside lock to avoid Controller deadlock
KAFKA-6857 <
https://issues.apache.org/jira/browse/KAFKA-6857
> - Prevent
follower from truncating to the wrong offset if undefined leader epoch is
requested
KAFKA-6854 <
https://issues.apache.org/jira/browse/KAFKA-6854
> - Log cleaner
fails with transaction markers that are deleted during clean
KAFKA-6747 <
https://issues.apache.org/jira/browse/KAFKA-6747
> - Check
whether there is in-flight transaction before aborting transaction
KAFKA-6748 <
https://issues.apache.org/jira/browse/KAFKA-6748
> - Double
check before scheduling a new task after the punctuate call
KAFKA-6739 <
https://issues.apache.org/jira/browse/KAFKA-6739
> -
Fix IllegalArgumentException when down-converting from V2 to V0/V1
KAFKA-6728 <
https://issues.apache.org/jira/browse/KAFKA-6728
> -
Fix NullPointerException when instantiating the HeaderConverter

Kafka 1.1.1 release plan:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1


Release notes for the 1.1.1 release:
http://home.apache.org/~lindong/kafka-1.1.1-rc1/RELEASE_NOTES.html


*** Please download, test and vote by Thursday, Jun 22, 12pm PT ***

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS


* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~lindong/kafka-1.1.1-rc1/


* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/


* Javadoc:
http://home.apache.org/~lindong/kafka-1.1.1-rc1/javadoc/


* Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc1 tag:
https://github.com/apache/kafka/tree/1.1.1-rc1


* Documentation:
http://kafka.apache.org/11/documentation.html


* Protocol:
http://kafka.apache.org/11/protocol.html


* Successful Jenkins builds for the 1.1 branch:
Unit/integration tests: 
*https://builds.apache.org/job/kafka-1.1-jdk7/152/
<
https://builds.apache.org/job/kafka-1.1-jdk7/152/
>*
System tests: 
https://jenkins.confluent.io/job/system-test-

kafka-branch-builder/1817


Please test and verify the release artifacts and submit a vote for this 
RC,
or report any issues so we can fix them and get a new RC out ASAP. 
Although
this release vote requires PMC votes to pass, testing, votes, and bug
reports are valuable and appreciated from everyone.

Cheers,
Dong

Kafka Streams Session store fetch latency very high with caching turned on

[Sam Lendle]

I am using a session store in a kafka streams application. With caching turned 
on, average fetch latency was very high, about 200 ms after running for about 1 
hour. With caching turned off, it was about 100 μs. We seem to be running fine 
without caching, but I am very curious as to why caching performance is so bad 
in our case. Any insight into what might be going on would be helpful.


Setup/config

  *   I'm using a custom transformer, but the implementation is almost 
identical to this section of KStreamSessionWindowAggregate 
https://github.com/apache/kafka/blob/fdcf75ea326b8e07d8d7c4f5cc14f1e70451bcd4/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSessionWindowAggregate.java#L94-L113
 The main difference is I'm forwarding something other than the updated session 
downstream
  *   Logging is turned off, so updates are not pushed to a change log topic. 
The store starts empty whenever streams is initialized
  *   I don't think I'm setting any potentially related configs. Rocksdb config 
is the default.

We're receiving about 1000 messages/second in a topic w/ five partitions. With 
caching turned on, this custom transformer is the bottleneck and processing 
rate is much lower, 100-200 ops per second. With caching turned off the volume 
is no problem. There are about 500k unique keys per hour.

Using a sampling profiler I saw that most time was spent in TreeMap operations. 
Unfortunately I don't have a copy of the profile data anymore, but I think the 
map in question is the `cache` field in the NamedCache class.

If I look at a plot of fetch latency vs time since starting, it looks to me 
like latency is about O(log(time)). I think what's going on is the size of the 
map is increasing linearly in time, particularly for the first few minutes that 
streams is running, because almost all keys will be unique. So the latency is 
almost entirely spent in TreeMap#get.

Questions:
1) Does my theory make sense?
2) Could the issue be related to the fact that I'm using a state store with the 
transformer/processor API vs the dsl? I know that caching is turned on by 
default for state stores in the dsl but not in the processor API, but I don't 
understand why.
3) My understanding is that streams side state store caching is an optimization 
to reduce the number of writes to the underlying rocksdb store. In that case, 
because I have so many unique keys, and the same keys usually show up a few 
minutes apart, it makes sense that caching wouldn't do much for me. Is that 
correct?
4) Given that things seem to work fine with caching turned off, could there be 
any advantage to having it on and configured differently? If so, what 
configuration changes should I try?

If there's any additional context I can provide please let me know.

Thanks,
Sam

Re: Configuring Kerberos behind an ELB

[Tyler Monahan]

Martin,

I think I tried that already. I setup a user in ad and assigned the shared
SPN record for the ELB to that user. I then added the user to the keytab
file for the kafka servers and had the kafka servers use the SPN for the
user with the dns record. That worked fine for authing through the ELB but
it broke inter broker communication. Since kafka talks to the other nodes
directly and doesn't go through the elb it would fail on the direct
connection as it doesn't have valid credentials as mentioned. I don't know
a solution to the interbroker communication unless there is some way I can
have it use different information for inter broker then it does for
incoming consumer/producer connections.

Tyler Monahan

On Fri, Jun 22, 2018 at 10:27 AM, Martin Gainty  wrote:

> it appears you want:
> common-principal name with common-key distributed to all subdomains
>
>
> Use only one common Service Principal Name:
>
> One of the solutions is to create a new Service Principal  in the
> KDC for the name HTTP/all.ipa@ipa.dom
> then generate a keytab and
> distribute it (keytab) to all servers.
> The servers will use no other key, and they  will identify
> themselves with the common  name,
> so if a client tries to contact them using their individual
>  name, then authentication will fail,
> as the KDC will not have a principal for the other  names
> and the services themselves are not configure to use their hostname only
> the common  name.
>
> assuming you generated common SPN in the KDC
> assuming you generated keytab and distributed to all subdomains
> does this not work for you?
> M-
>
>
>
>
> --
> *From:* Tyler Monahan 
> *Sent:* Friday, June 22, 2018 1:09 PM
> *To:* Ben Wood
> *Cc:* users@kafka.apache.org; Martin Gainty
> *Subject:* Re: Configuring Kerberos behind an ELB
>
> Ben,
>
> Yes. I want to be able to provide consumers/producers with a single
> address they can use to connect to the cluster. Having it behind an elb
> lets us scale up and replace nodes with out needing to mess with
> consumer/producer configurations. I have considered setting up individual
> dns records for each broker and feeding in a list of instances to connect
> to but this is not as flexible as using an elb and does not match our
> general strategy for infrastructure. If at all possible I would like to get
> kafka working behind and elb with kerberos.
>
> Tyler Monahan
>
> On Fri, Jun 22, 2018 at 9:44 AM, Ben Wood  wrote:
>
> Hey Tyler,
>
> What is your end goal? To have a single publicly / internally available
> address to be able to provide to consumers / producers to connect to the
> Kerberized Kafka?
>
> On Fri, Jun 22, 2018 at 9:20 AM, Tyler Monahan 
> wrote:
>
> Martin,
>
> I have read that stack overflow post but it doesn't help with my specific
> problem. An ELB will work if I am not using kerberos just fine. The issue
> started happening when I added kerberos auth to the cluster. The auth has
> to happen before the meta data request so it never gets to the point where
> it is by passing the load balancer. Because I am connecting with the load
> balancers dns record I don't have a valid spn on the brokers for the load
> balancers dns record. This blog post has some work arounds for kerberos
> with a load balancer and details the problem but I haven't been able to get
> any of them to work with kafka because it gets traffic through and ELB but
> also talks to the other brokers directly in my setup.
> https://ssimo.org/blog/id_019.html
>
> Tyler Monahan
>
> On Fri, Jun 22, 2018 at 5:36 AM, Martin Gainty 
> wrote:
>
> > MG>quoting stackoverflow below
> >
> > "You can use an ELB as the bootstrap.servers,
> > *The ELB will be used for the initial metadata request the client makes
> to
> > figure out which topic partitions are on which brokers, *
> > but after (the initial metadata request)
> > the
> > *brokers still need to be directly accessible to the client. *that it'll
> > use the hostname of the server (or advertised.listeners setting if you
> > need to customize it,
> > which, e.g. might be necessary on EC2 instances to get the public IP of a
> > server).
> > If you were trying to use an ELB to make a Kafka cluster publicly
> > available,
> > you'd need to make sure the advertised.listeners for each broker also
> > makes it publicly accessible. "
> >
> > MG> initial metadata request you will see elb
> > MG>after metadata request and topic partition locations are determined
> > MG>elb drops out and client will talk to directly to broker
> > MG>use healthcheck algorithm to determine assigned server/port assigned
> to
> > broker from /brokers/id/$id
> > MG>echo dump | nc localhost 2181 | grep brokers
> >
> >
> > https://stackoverflow.com/questions/38666795/does-kafka-
> > support-elb-in-front-of-broker-cluster
> >
> >  port-elb-in-front-of-broker-cluster>
> > Does Kafka support ELB in front of broker cluster? - Stack ...
> > 

Re: [VOTE] 1.0.2 RC0

[Vahid S Hashemian]

+1 (non-binding)

Built from source and ran quickstart successfully on Ubuntu (with Java 8).

Thanks for running the release Matthias!
--Vahid




From:   "Matthias J. Sax" 
To: d...@kafka.apache.org, users@kafka.apache.org, 
kafka-clie...@googlegroups.com
Date:   06/22/2018 10:42 AM
Subject:[VOTE] 1.0.2 RC0



Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 1.0.2.

This is a bug fix release closing 26 tickets:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.0.2

Release notes for the 1.0.2 release:
http://home.apache.org/~mjsax/kafka-1.0.2-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Tuesday, 6/26/18 end-of-day, so we
can close the vote on Wednesday.

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~mjsax/kafka-1.0.2-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~mjsax/kafka-1.0.2-rc0/javadoc/

* Tag to be voted upon (off 1.0 branch) is the 1.0.2 tag:
https://github.com/apache/kafka/releases/tag/1.0.2-rc0

* Documentation:
http://kafka.apache.org/10/documentation.html

* Protocol:
http://kafka.apache.org/10/protocol.html

* Successful Jenkins builds for the 1.0 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-1.0-jdk7/211/
System tests:
https://jenkins.confluent.io/job/system-test-kafka/job/1.0/217/

/**

Thanks,
  -Matthias

[attachment "signature.asc" deleted by Vahid S Hashemian/Silicon 
Valley/IBM] 

[VOTE] 1.0.2 RC0

[Matthias J. Sax]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 1.0.2.

This is a bug fix release closing 26 tickets:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.0.2

Release notes for the 1.0.2 release:
http://home.apache.org/~mjsax/kafka-1.0.2-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Tuesday, 6/26/18 end-of-day, so we
can close the vote on Wednesday.

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~mjsax/kafka-1.0.2-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~mjsax/kafka-1.0.2-rc0/javadoc/

* Tag to be voted upon (off 1.0 branch) is the 1.0.2 tag:
https://github.com/apache/kafka/releases/tag/1.0.2-rc0

* Documentation:
http://kafka.apache.org/10/documentation.html

* Protocol:
http://kafka.apache.org/10/protocol.html

* Successful Jenkins builds for the 1.0 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-1.0-jdk7/211/
System tests:
https://jenkins.confluent.io/job/system-test-kafka/job/1.0/217/

/**

Thanks,
  -Matthias



signature.asc
Description: OpenPGP digital signature

Re: Configuring Kerberos behind an ELB

[Martin Gainty]

it appears you want:
common-principal name with common-key distributed to all subdomains

Use only one common Service Principal Name:


One of the solutions is to create a new Service Principal  in the KDC for 
the name HTTP/all.ipa@ipa.dom
then generate a keytab and
distribute it (keytab) to all servers.
The servers will use no other key, and they  will identify 
themselves with the common  name,
so if a client tries to contact them using their individual  name, 
then authentication will fail,
as the KDC will not have a principal for the other  names
and the services themselves are not configure to use their hostname only the 
common  name.

assuming you generated common SPN in the KDC
assuming you generated keytab and distributed to all subdomains
does this not work for you?
M-





From: Tyler Monahan 
Sent: Friday, June 22, 2018 1:09 PM
To: Ben Wood
Cc: users@kafka.apache.org; Martin Gainty
Subject: Re: Configuring Kerberos behind an ELB

Ben,

Yes. I want to be able to provide consumers/producers with a single address 
they can use to connect to the cluster. Having it behind an elb lets us scale 
up and replace nodes with out needing to mess with consumer/producer 
configurations. I have considered setting up individual dns records for each 
broker and feeding in a list of instances to connect to but this is not as 
flexible as using an elb and does not match our general strategy for 
infrastructure. If at all possible I would like to get kafka working behind and 
elb with kerberos.

Tyler Monahan

On Fri, Jun 22, 2018 at 9:44 AM, Ben Wood 
mailto:bw...@mesosphere.io>> wrote:
Hey Tyler,

What is your end goal? To have a single publicly / internally available address 
to be able to provide to consumers / producers to connect to the Kerberized 
Kafka?

On Fri, Jun 22, 2018 at 9:20 AM, Tyler Monahan 
mailto:tjmonah...@gmail.com>> wrote:
Martin,

I have read that stack overflow post but it doesn't help with my specific
problem. An ELB will work if I am not using kerberos just fine. The issue
started happening when I added kerberos auth to the cluster. The auth has
to happen before the meta data request so it never gets to the point where
it is by passing the load balancer. Because I am connecting with the load
balancers dns record I don't have a valid spn on the brokers for the load
balancers dns record. This blog post has some work arounds for kerberos
with a load balancer and details the problem but I haven't been able to get
any of them to work with kafka because it gets traffic through and ELB but
also talks to the other brokers directly in my setup.
https://ssimo.org/blog/id_019.html

Tyler Monahan

On Fri, Jun 22, 2018 at 5:36 AM, Martin Gainty 
mailto:mgai...@hotmail.com>> wrote:

> MG>quoting stackoverflow below
>
> "You can use an ELB as the bootstrap.servers,
> *The ELB will be used for the initial metadata request the client makes to
> figure out which topic partitions are on which brokers, *
> but after (the initial metadata request)
> the
> *brokers still need to be directly accessible to the client. *that it'll
> use the hostname of the server (or advertised.listeners setting if you
> need to customize it,
> which, e.g. might be necessary on EC2 instances to get the public IP of a
> server).
> If you were trying to use an ELB to make a Kafka cluster publicly
> available,
> you'd need to make sure the advertised.listeners for each broker also
> makes it publicly accessible. "
>
> MG> initial metadata request you will see elb
> MG>after metadata request and topic partition locations are determined
> MG>elb drops out and client will talk to directly to broker
> MG>use healthcheck algorithm to determine assigned server/port assigned to
> broker from /brokers/id/$id
> MG>echo dump | nc localhost 2181 | grep brokers
>
>
> https://stackoverflow.com/questions/38666795/does-kafka-
> support-elb-in-front-of-broker-cluster
>
> 
> Does Kafka support ELB in front of broker cluster? - Stack ...
> 
> stackoverflow.com
> I had a question regarding Kafka broker clusters on AWS. Right now there
> is an AWS ELB sitting in front of the cluster, but when I set the
> "bootstrap.servers" property of my producer or consumer to...
>
> does this help?
>
> Martin
> __
>
>
>
> --
> *From:* Tyler Monahan mailto:tjmonah...@gmail.com>>
> *Sent:* Thursday, June 21, 2018 6:17 PM
> *To:* users@kafka.apache.org
> *Subject:* Configuring Kerberos behind an ELB
>
> Hello,
>
> I have setup kafka using kerberos successfully however if I try and reach
> kafka through an elb the kerberos authentication fails. The kafka brokers
> are each using their unique hostname for kerberos and 

Re: Kafka Streams - Expiring Records By Process Time

[Sicheng Liu]

Hi John,

I think the situation you describe is reasonable but we can still do some
tricks there. For example, filter out the aggregated metrics with the
aggregated "count" (sum of total number of metrics in the aggregation) less
than some threshold. So to make things simpler, we can assume that this is
for backfill, and we always backfill whole hour of metrics while doing
hourly aggregation. Retention based on process time would work fine in this
case.

Thanks,
Sicheng

On Thu, Jun 21, 2018 at 5:28 PM, John Roesler  wrote:

> Hi Sicheng,
>
> I'm also curious about the details.
>
> Let's say you are doing a simple count aggregation with 24-hour windows.
> You got three events with key "A" on 2017-06-21, one year ago, so the
> windowed key (A,2017-06-21) has a value of 3.
>
> Fast-forward a year later. We get one late event, also for 2017-06-21. The
> correct value of (A,2017-06-21) is now 4.
>
> If you set your retention time to account for such lateness, say 2 years,
> there's no problem. Streams still has the state "(A,2017-06-21): 3" in
> storage, so it can update the value and emit "(A,2017-06-21): 4".
>
> But if you have your retention shorter, let's say 1 month, then Streams
> deleted that state long ago and no longer knows about those three prior
> events. If it processes that event, it will incorrectly report
> "(A,2017-06-21):1"
> as the value for that windowed key.
>
> So, to preserve correctness, you must either discard new events for expired
> windows or set the retention higher than any lateness you'll observe.
>
> On the other hand, you can use processing time, instead of event time, in
> which case our new event actually belongs to today's window, 2018-06-21,
> which is still retained.
>
> But this whole thing only works out if you use the same notion of time,
> event or processing, for *both* window assignment and expiration,
> otherwise, you get incorrect results. Specifically, you can "trick" Streams
> into processing that event for the expired window and get the incorrect
> result of "(A,2017-06-21): 1".
>
> Is that an accurate depiction of the situation, or have I missed something?
>
> Thanks,
> -John
>
> On Thu, Jun 21, 2018 at 5:52 PM Matthias J. Sax 
> wrote:
>
> > Can't you increase retention time accordingly to make sure that "old"
> > metrics are not dropped?
> >
> > -Matthias
> >
> > On 6/21/18 2:07 PM, Sicheng Liu wrote:
> > > Because we might get very "old" metrics (the timestamp on the metric is
> > > very old, even though the metric is just delivered, for example,
> > > backfill.). If you use event-time for retention, these old metrics
> could
> > be
> > > dropped and won't be aggregated. If we use process-time, at least it
> will
> > > stay in state-store for some time for aggregation.
> > >
> > > On Thu, Jun 21, 2018 at 1:24 PM, Matthias J. Sax <
> matth...@confluent.io>
> > > wrote:
> > >
> > >> I don't understand why event-time retention time cannot be used?
> Cannot
> > >> elaborate?
> > >>
> > >> -Matthias
> > >>
> > >> On 6/21/18 10:59 AM, Sicheng Liu wrote:
> > >>> Hi All,
> > >>>
> > >>> We have a use case that we aggregate some metrics with its event-time
> > >>> (timestamp on the metric itself) using the simplest tumbling window.
> > The
> > >>> window itself can be set a retention but since we are aggregating
> with
> > >>> event-time the retention has to be based on event-time too. However,
> in
> > >> our
> > >>> scenario, we have some late arrival metrics (up to one year) and we
> > hope
> > >>> the window retention can be based on process-time so that we can hold
> > the
> > >>> late arrival metrics for some time and expire them after some hours
> > even
> > >>> without new metrics of the same aggregation key coming.
> > >>>
> > >>> We have tried:
> > >>> 1. Set TTL on RocksDB but it is disabled in Kafka Streams.
> > >>> 2. Using low level processor API but scanning the statestore and
> delete
> > >> one
> > >>> by one significantly drops the performance.
> > >>>
> > >>> Please let us know if it is possible to aggregate by event-time but
> > >> setting
> > >>> the window retention based on its process-time.
> > >>>
> > >>> Thanks,
> > >>> Sicheng
> > >>>
> > >>
> > >>
> > >
> >
> >
>

[VOTE] 1.1.1 RC1

[Dong Lin]

Hello Kafka users, developers and client-developers,

This is the second candidate for release of Apache Kafka 1.1.1.

Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was first
released with 1.1.0 about 3 months ago. We have fixed about 25 issues since
that release. A few of the more significant fixes include:

KAFKA-6925  - Fix memory
leak in StreamsMetricsThreadImpl
KAFKA-6937  - In-sync
replica delayed during fetch if replica throttle is exceeded
KAFKA-6917  - Process txn
completion asynchronously to avoid deadlock
KAFKA-6893  - Create
processors before starting acceptor to avoid ArithmeticException
KAFKA-6870  -
Fix ConcurrentModificationException in SampledStat
KAFKA-6878  - Fix
NullPointerException when querying global state store
KAFKA-6879  - Invoke
session init callbacks outside lock to avoid Controller deadlock
KAFKA-6857  - Prevent
follower from truncating to the wrong offset if undefined leader epoch is
requested
KAFKA-6854  - Log cleaner
fails with transaction markers that are deleted during clean
KAFKA-6747  - Check
whether there is in-flight transaction before aborting transaction
KAFKA-6748  - Double
check before scheduling a new task after the punctuate call
KAFKA-6739  -
Fix IllegalArgumentException when down-converting from V2 to V0/V1
KAFKA-6728  -
Fix NullPointerException when instantiating the HeaderConverter

Kafka 1.1.1 release plan:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+1.1.1

Release notes for the 1.1.1 release:
http://home.apache.org/~lindong/kafka-1.1.1-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Thursday, Jun 22, 12pm PT ***

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~lindong/kafka-1.1.1-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/

* Javadoc:
http://home.apache.org/~lindong/kafka-1.1.1-rc1/javadoc/

* Tag to be voted upon (off 1.1 branch) is the 1.1.1-rc1 tag:
https://github.com/apache/kafka/tree/1.1.1-rc1

* Documentation:
http://kafka.apache.org/11/documentation.html

* Protocol:
http://kafka.apache.org/11/protocol.html

* Successful Jenkins builds for the 1.1 branch:
Unit/integration tests: *https://builds.apache.org/job/kafka-1.1-jdk7/152/
*
System tests: https://jenkins.confluent.io/job/system-test-
kafka-branch-builder/1817


Please test and verify the release artifacts and submit a vote for this RC,
or report any issues so we can fix them and get a new RC out ASAP. Although
this release vote requires PMC votes to pass, testing, votes, and bug
reports are valuable and appreciated from everyone.

Cheers,
Dong

Re: Configuring Kerberos behind an ELB

[Tyler Monahan]

Ben,

Yes. I want to be able to provide consumers/producers with a single address
they can use to connect to the cluster. Having it behind an elb lets us
scale up and replace nodes with out needing to mess with consumer/producer
configurations. I have considered setting up individual dns records for
each broker and feeding in a list of instances to connect to but this is
not as flexible as using an elb and does not match our general strategy for
infrastructure. If at all possible I would like to get kafka working behind
and elb with kerberos.

Tyler Monahan

On Fri, Jun 22, 2018 at 9:44 AM, Ben Wood  wrote:

> Hey Tyler,
>
> What is your end goal? To have a single publicly / internally available
> address to be able to provide to consumers / producers to connect to the
> Kerberized Kafka?
>
> On Fri, Jun 22, 2018 at 9:20 AM, Tyler Monahan 
> wrote:
>
>> Martin,
>>
>> I have read that stack overflow post but it doesn't help with my specific
>> problem. An ELB will work if I am not using kerberos just fine. The issue
>> started happening when I added kerberos auth to the cluster. The auth has
>> to happen before the meta data request so it never gets to the point where
>> it is by passing the load balancer. Because I am connecting with the load
>> balancers dns record I don't have a valid spn on the brokers for the load
>> balancers dns record. This blog post has some work arounds for kerberos
>> with a load balancer and details the problem but I haven't been able to
>> get
>> any of them to work with kafka because it gets traffic through and ELB but
>> also talks to the other brokers directly in my setup.
>> https://ssimo.org/blog/id_019.html
>>
>> Tyler Monahan
>>
>> On Fri, Jun 22, 2018 at 5:36 AM, Martin Gainty 
>> wrote:
>>
>> > MG>quoting stackoverflow below
>> >
>> > "You can use an ELB as the bootstrap.servers,
>> > *The ELB will be used for the initial metadata request the client makes
>> to
>> > figure out which topic partitions are on which brokers, *
>> > but after (the initial metadata request)
>> > the
>> > *brokers still need to be directly accessible to the client. *that it'll
>> > use the hostname of the server (or advertised.listeners setting if you
>> > need to customize it,
>> > which, e.g. might be necessary on EC2 instances to get the public IP of
>> a
>> > server).
>> > If you were trying to use an ELB to make a Kafka cluster publicly
>> > available,
>> > you'd need to make sure the advertised.listeners for each broker also
>> > makes it publicly accessible. "
>> >
>> > MG> initial metadata request you will see elb
>> > MG>after metadata request and topic partition locations are determined
>> > MG>elb drops out and client will talk to directly to broker
>> > MG>use healthcheck algorithm to determine assigned server/port assigned
>> to
>> > broker from /brokers/id/$id
>> > MG>echo dump | nc localhost 2181 | grep brokers
>> >
>> >
>> > https://stackoverflow.com/questions/38666795/does-kafka-
>> > support-elb-in-front-of-broker-cluster
>> >
>> > > port-elb-in-front-of-broker-cluster>
>> > Does Kafka support ELB in front of broker cluster? - Stack ...
>> > > port-elb-in-front-of-broker-cluster>
>> > stackoverflow.com
>> > I had a question regarding Kafka broker clusters on AWS. Right now there
>> > is an AWS ELB sitting in front of the cluster, but when I set the
>> > "bootstrap.servers" property of my producer or consumer to...
>> >
>> > does this help?
>> >
>> > Martin
>> > __
>> >
>> >
>> >
>> > --
>> > *From:* Tyler Monahan 
>> > *Sent:* Thursday, June 21, 2018 6:17 PM
>> > *To:* users@kafka.apache.org
>> > *Subject:* Configuring Kerberos behind an ELB
>> >
>> > Hello,
>> >
>> > I have setup kafka using kerberos successfully however if I try and
>> reach
>> > kafka through an elb the kerberos authentication fails. The kafka
>> brokers
>> > are each using their unique hostname for kerberos and when going
>> through an
>> > elb the consumer/producer only sees the elb's dns record which doesn't
>> have
>> > kerberos setup for it causing auth to fail. If I try to setup a service
>> > principle name for that dns record I can only associate it with one of
>> the
>> > brokers behind the elb so the other ones fail.
>> >
>> > I have tried setting up a service account and having the kafka brokers
>> use
>> > that which works when a consumer/producer is talking to the instances
>> > through the elb however inter broker communication which is also over
>> > kerberos fails at that point because it is going directly to the other
>> > nodes instead of through the elb. I am not sure where to go from here as
>> > there doesn't appear to be a way to configure the inter broker
>> > communication to work differently then the incoming consumer
>> communication
>> > short of getting rid of kerberos.
>> >
>> > Any 

Re: Configuring Kerberos behind an ELB

[Ben Wood]

Hey Tyler,

What is your end goal? To have a single publicly / internally available
address to be able to provide to consumers / producers to connect to the
Kerberized Kafka?

On Fri, Jun 22, 2018 at 9:20 AM, Tyler Monahan  wrote:

> Martin,
>
> I have read that stack overflow post but it doesn't help with my specific
> problem. An ELB will work if I am not using kerberos just fine. The issue
> started happening when I added kerberos auth to the cluster. The auth has
> to happen before the meta data request so it never gets to the point where
> it is by passing the load balancer. Because I am connecting with the load
> balancers dns record I don't have a valid spn on the brokers for the load
> balancers dns record. This blog post has some work arounds for kerberos
> with a load balancer and details the problem but I haven't been able to get
> any of them to work with kafka because it gets traffic through and ELB but
> also talks to the other brokers directly in my setup.
> https://ssimo.org/blog/id_019.html
>
> Tyler Monahan
>
> On Fri, Jun 22, 2018 at 5:36 AM, Martin Gainty 
> wrote:
>
> > MG>quoting stackoverflow below
> >
> > "You can use an ELB as the bootstrap.servers,
> > *The ELB will be used for the initial metadata request the client makes
> to
> > figure out which topic partitions are on which brokers, *
> > but after (the initial metadata request)
> > the
> > *brokers still need to be directly accessible to the client. *that it'll
> > use the hostname of the server (or advertised.listeners setting if you
> > need to customize it,
> > which, e.g. might be necessary on EC2 instances to get the public IP of a
> > server).
> > If you were trying to use an ELB to make a Kafka cluster publicly
> > available,
> > you'd need to make sure the advertised.listeners for each broker also
> > makes it publicly accessible. "
> >
> > MG> initial metadata request you will see elb
> > MG>after metadata request and topic partition locations are determined
> > MG>elb drops out and client will talk to directly to broker
> > MG>use healthcheck algorithm to determine assigned server/port assigned
> to
> > broker from /brokers/id/$id
> > MG>echo dump | nc localhost 2181 | grep brokers
> >
> >
> > https://stackoverflow.com/questions/38666795/does-kafka-
> > support-elb-in-front-of-broker-cluster
> >
> >  support-elb-in-front-of-broker-cluster>
> > Does Kafka support ELB in front of broker cluster? - Stack ...
> >  support-elb-in-front-of-broker-cluster>
> > stackoverflow.com
> > I had a question regarding Kafka broker clusters on AWS. Right now there
> > is an AWS ELB sitting in front of the cluster, but when I set the
> > "bootstrap.servers" property of my producer or consumer to...
> >
> > does this help?
> >
> > Martin
> > __
> >
> >
> >
> > --
> > *From:* Tyler Monahan 
> > *Sent:* Thursday, June 21, 2018 6:17 PM
> > *To:* users@kafka.apache.org
> > *Subject:* Configuring Kerberos behind an ELB
> >
> > Hello,
> >
> > I have setup kafka using kerberos successfully however if I try and reach
> > kafka through an elb the kerberos authentication fails. The kafka brokers
> > are each using their unique hostname for kerberos and when going through
> an
> > elb the consumer/producer only sees the elb's dns record which doesn't
> have
> > kerberos setup for it causing auth to fail. If I try to setup a service
> > principle name for that dns record I can only associate it with one of
> the
> > brokers behind the elb so the other ones fail.
> >
> > I have tried setting up a service account and having the kafka brokers
> use
> > that which works when a consumer/producer is talking to the instances
> > through the elb however inter broker communication which is also over
> > kerberos fails at that point because it is going directly to the other
> > nodes instead of through the elb. I am not sure where to go from here as
> > there doesn't appear to be a way to configure the inter broker
> > communication to work differently then the incoming consumer
> communication
> > short of getting rid of kerberos.
> >
> > Any advice would be greatly appreciated.
> >
> > Tyler Monahan
> >
>



-- 
Ben Wood
Software Engineer - Data Agility
Mesosphere

Re: Configuring Kerberos behind an ELB

[Tyler Monahan]

Martin,

I have read that stack overflow post but it doesn't help with my specific
problem. An ELB will work if I am not using kerberos just fine. The issue
started happening when I added kerberos auth to the cluster. The auth has
to happen before the meta data request so it never gets to the point where
it is by passing the load balancer. Because I am connecting with the load
balancers dns record I don't have a valid spn on the brokers for the load
balancers dns record. This blog post has some work arounds for kerberos
with a load balancer and details the problem but I haven't been able to get
any of them to work with kafka because it gets traffic through and ELB but
also talks to the other brokers directly in my setup.
https://ssimo.org/blog/id_019.html

Tyler Monahan

On Fri, Jun 22, 2018 at 5:36 AM, Martin Gainty  wrote:

> MG>quoting stackoverflow below
>
> "You can use an ELB as the bootstrap.servers,
> *The ELB will be used for the initial metadata request the client makes to
> figure out which topic partitions are on which brokers, *
> but after (the initial metadata request)
> the
> *brokers still need to be directly accessible to the client. *that it'll
> use the hostname of the server (or advertised.listeners setting if you
> need to customize it,
> which, e.g. might be necessary on EC2 instances to get the public IP of a
> server).
> If you were trying to use an ELB to make a Kafka cluster publicly
> available,
> you'd need to make sure the advertised.listeners for each broker also
> makes it publicly accessible. "
>
> MG> initial metadata request you will see elb
> MG>after metadata request and topic partition locations are determined
> MG>elb drops out and client will talk to directly to broker
> MG>use healthcheck algorithm to determine assigned server/port assigned to
> broker from /brokers/id/$id
> MG>echo dump | nc localhost 2181 | grep brokers
>
>
> https://stackoverflow.com/questions/38666795/does-kafka-
> support-elb-in-front-of-broker-cluster
>
> 
> Does Kafka support ELB in front of broker cluster? - Stack ...
> 
> stackoverflow.com
> I had a question regarding Kafka broker clusters on AWS. Right now there
> is an AWS ELB sitting in front of the cluster, but when I set the
> "bootstrap.servers" property of my producer or consumer to...
>
> does this help?
>
> Martin
> __
>
>
>
> --
> *From:* Tyler Monahan 
> *Sent:* Thursday, June 21, 2018 6:17 PM
> *To:* users@kafka.apache.org
> *Subject:* Configuring Kerberos behind an ELB
>
> Hello,
>
> I have setup kafka using kerberos successfully however if I try and reach
> kafka through an elb the kerberos authentication fails. The kafka brokers
> are each using their unique hostname for kerberos and when going through an
> elb the consumer/producer only sees the elb's dns record which doesn't have
> kerberos setup for it causing auth to fail. If I try to setup a service
> principle name for that dns record I can only associate it with one of the
> brokers behind the elb so the other ones fail.
>
> I have tried setting up a service account and having the kafka brokers use
> that which works when a consumer/producer is talking to the instances
> through the elb however inter broker communication which is also over
> kerberos fails at that point because it is going directly to the other
> nodes instead of through the elb. I am not sure where to go from here as
> there doesn't appear to be a way to configure the inter broker
> communication to work differently then the incoming consumer communication
> short of getting rid of kerberos.
>
> Any advice would be greatly appreciated.
>
> Tyler Monahan
>

Re: [VOTE] 2.0.0 RC0

[Rajini Sivaram]

Any and all testing is welcome, but testing in the following areas would be
particularly helpful:

   1. Performance and stress testing. Heroku and LinkedIn have helped with this
   in the past (and issues have been found and fixed).
   2. Client developers can verify that their clients can produce/consume
compressed/uncompressed data to/from 2.0.0 brokers
   3. End users can verify that their apps work correctly with the new
   release.

Thank you!

Rajini

On Thu, Jun 21, 2018 at 12:24 PM, Rajini Sivaram 
wrote:

> Sorry, the documentation does go live with the RC (thanks to Ismael for
> pointing this out), so here are the links:
>
> * Documentation:
>
> http://kafka.apache.org/20/documentation.html
>
>
> * Protocol:
>
> http://kafka.apache.org/20/protocol.html
>
>
>
> Regards,
>
>
> Rajini
>
>
> On Wed, Jun 20, 2018 at 9:08 PM, Rajini Sivaram 
> wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>>
>> This is the first candidate for release of Apache Kafka 2.0.0.
>>
>>
>> This is a major version release of Apache Kafka. It includes 40 new  KIPs
>> and
>>
>> several critical bug fixes. Please see the 2.0.0 release plan for more
>> details:
>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820
>>
>>
>> A few notable highlights:
>>
>>- Prefixed wildcard ACLs (KIP-290), Fine grained ACLs for
>>CreateTopics (KIP-277)
>>- SASL/OAUTHBEARER implementation (KIP-255)
>>- Improved quota communication and customization of quotas (KIP-219,
>>KIP-257)
>>- Efficient memory usage for down conversion (KIP-283)
>>- Fix log divergence between leader and follower during fast leader
>>failover (KIP-279)
>>- Drop support for Java 7 and remove deprecated code including old
>>scala clients
>>- Connect REST extension plugin, support for externalizing secrets
>>and improved error handling (KIP-285, KIP-297, KIP-298 etc.)
>>- Scala API for Kafka Streams and other Streams API improvements
>>(KIP-270, KIP-150, KIP-245, KIP-251 etc.)
>>
>>
>> Release notes for the 2.0.0 release:
>>
>> http://home.apache.org/~rsivaram/kafka-2.0.0-rc0/RELEASE_NOTES.html
>>
>>
>> *** Please download, test and vote by Monday, June 25, 4pm PT
>>
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>>
>> http://kafka.apache.org/KEYS
>>
>>
>> * Release artifacts to be voted upon (source and binary):
>>
>> http://home.apache.org/~rsivaram/kafka-2.0.0-rc0/
>>
>>
>> * Maven artifacts to be voted upon:
>>
>> https://repository.apache.org/content/groups/staging/
>>
>>
>> * Javadoc:
>>
>> http://home.apache.org/~rsivaram/kafka-2.0.0-rc0/javadoc/
>>
>>
>> * Tag to be voted upon (off 2.0 branch) is the 2.0.0 tag:
>>
>> https://github.com/apache/kafka/tree/2.0.0-rc0
>>
>>
>> * Documentation:
>>
>> http://home.apache.org/~rsivaram/kafka-2.0.0-rc0/kafka_2.11-
>> 2.0.0-site-docs.tgz
>>
>> (Since documentation cannot go live until 2.0.0 is released, please
>> download and verify)
>>
>>
>> * Successful Jenkins builds for the 2.0 branch:
>>
>> Unit/integration tests: https://builds.apache.org/job/kafka-2.0-jdk8/48/
>>
>> System tests: https://jenkins.confluent.io/job/system-test-kafka/jo
>> b/2.0/6/ (2 failures are known flaky tests)
>>
>>
>>
>> Please test and verify the release artifacts and submit a vote for this RC
>> or report any issues so that we can fix them and roll out a new RC ASAP!
>>
>> Although this release vote requires PMC votes to pass, testing, votes,
>> and bug
>> reports are valuable and appreciated from everyone.
>>
>>
>> Thanks,
>>
>>
>> Rajini
>>
>>
>>
>

Re: Configuring Kerberos behind an ELB

[Martin Gainty]

MG>quoting stackoverflow below

"You can use an ELB as the bootstrap.servers,
The ELB will be used for the initial metadata request the client makes to 
figure out which topic partitions are on which brokers,
but after (the initial metadata request)
the brokers still need to be directly accessible to the client.
that it'll use the hostname of the server (or advertised.listeners setting if 
you need to customize it,
which, e.g. might be necessary on EC2 instances to get the public IP of a 
server).
If you were trying to use an ELB to make a Kafka cluster publicly available,
you'd need to make sure the advertised.listeners for each broker also makes it 
publicly accessible. "

MG> initial metadata request you will see elb
MG>after metadata request and topic partition locations are determined
MG>elb drops out and client will talk to directly to broker
MG>use healthcheck algorithm to determine assigned server/port assigned to 
broker from /brokers/id/$id
MG>echo dump | nc localhost 2181 | grep brokers


https://stackoverflow.com/questions/38666795/does-kafka-support-elb-in-front-of-broker-cluster

[https://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-i...@2.png?v=73d79a89bded]

Does Kafka support ELB in front of broker cluster? - Stack 
...
stackoverflow.com
I had a question regarding Kafka broker clusters on AWS. Right now there is an 
AWS ELB sitting in front of the cluster, but when I set the "bootstrap.servers" 
property of my producer or consumer to...


does this help?

Martin
__





From: Tyler Monahan 
Sent: Thursday, June 21, 2018 6:17 PM
To: users@kafka.apache.org
Subject: Configuring Kerberos behind an ELB

Hello,

I have setup kafka using kerberos successfully however if I try and reach
kafka through an elb the kerberos authentication fails. The kafka brokers
are each using their unique hostname for kerberos and when going through an
elb the consumer/producer only sees the elb's dns record which doesn't have
kerberos setup for it causing auth to fail. If I try to setup a service
principle name for that dns record I can only associate it with one of the
brokers behind the elb so the other ones fail.

I have tried setting up a service account and having the kafka brokers use
that which works when a consumer/producer is talking to the instances
through the elb however inter broker communication which is also over
kerberos fails at that point because it is going directly to the other
nodes instead of through the elb. I am not sure where to go from here as
there doesn't appear to be a way to configure the inter broker
communication to work differently then the incoming consumer communication
short of getting rid of kerberos.

Any advice would be greatly appreciated.

Tyler Monahan

ConsumerConnector and 0.10 message format

[Jeff Pollard]

Hello all,

We are in the process of upgrading all our consumers to version 0.10+. One
of our pre-0.10 consumers still uses the deprecated ConsumerConnector API
(I believe colloquially called the "old consumer").

We're unfortunately not able to upgrade this consumer to the new consumer
API, as the consumer is baked into a larger piece of software we did not
write, and they have not offered a version of it with the new consumer.
They have, however, released a version using Kafka 0.10, which we have
upgraded to.

We also have upgraded our *brokers* to 0.10+, but per the potential
performance impact

noted
in documentation have set the log.message.format.version to 0.8.2. This has
worked well for us, as we've slowly been upgrading our consumers to 0.10+.

My question is about using the old ConsumerConnector API with Kafka 0.10+
client and a 0.10+ broker. From my reading of the documentation, it seems
the performance impact is translating the message format by the broker, and
is independent of the kind (old or new) consumer. I believe both the old
and new consumer both use the same fetch API, but I hadn't verified that
yet.

We just wanted to make sure this was correct, as to avoid any unexpected
performance impacts when we finish our upgrade and change the on-broker
message format to the 0.10+ one.

Thanks again. Happy to answer or clarify any points of this email as needed.