Re: [VOTE] 2.0.1 RC0

[Ismael Juma]

Manikumar, looks like you have the necessary votes. :)

Ismael

On Wed, Nov 7, 2018, 10:45 AM Rajini Sivaram  +1 (binding)
>
> Checked source build and unit tests. Ran quickstart with source and binary.
>
> Thank you for managing the release, Manikumar!
>
> Regards,
>
> Rajini
>
> On Wed, Nov 7, 2018 at 6:18 PM Gwen Shapira  wrote:
>
> > +1 (binding)
> >
> > Checked signatures, build and quickstart.
> >
> > Thank you for managing the release, Mani!
> >
> >
> > On Thu, Oct 25, 2018 at 7:29 PM Manikumar 
> > wrote:
> > >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the first candidate for release of Apache Kafka 2.0.1.
> > >
> > > This is a bug fix release closing 49 tickets:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1
> > >
> > > Release notes for the 2.0.1 release:
> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by  Tuesday, October 30, end of day
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/
> > >
> > > * Javadoc:
> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/javadoc/
> > >
> > > * Tag to be voted upon (off 2.0 branch) is the 2.0.1 tag:
> > > https://github.com/apache/kafka/releases/tag/2.0.1-rc0
> > >
> > > * Documentation:
> > > http://kafka.apache.org/20/documentation.html
> > >
> > > * Protocol:
> > > http://kafka.apache.org/20/protocol.html
> > >
> > > * Successful Jenkins builds for the 2.0 branch:
> > > Unit/integration tests:
> > https://builds.apache.org/job/kafka-2.0-jdk8/177/
> > >
> > > /**
> > >
> > > Thanks,
> > > Manikumar
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>

[RESULTS] [VOTE] Release Kafka version 2.0.1

[Manikumar]

This vote passes with 12 +1 votes (3 bindings) and no 0 or -1 votes.

+1 votes
PMC Members:
* Jun Rao
* Gwen Shapira
* Rajini Sivaram

Committers:
* Harsha
* Ewen Cheslack-Postava
* Manikumar

Community:
* Magnus Edenhill
* Eno Thereska
* Vahid Hashemian
* Jakub Scholz
* Satish Duggana
* Mickael Maison

0 votes
* No votes

-1 votes
* No votes

Vote thread:
https://markmail.org/thread/3gy4a6fmjv2xuoou

Thank you all for the votes.
I'll continue with the release process and the release announcement will
follow in the next few days.

Manikumar

Re: Kafka Streams Session store performance degradation from 0.10.2.1 to 0.11.0.3

[Matthias J. Sax]

Thanks for verifying.

>> From our perspective, it appears something happened after 0.10.2.1 that made 
>> the LRU Cache much slower for our use case. 

That is what I try to figure out. I went over the 0.10.2.2 to 0.11.0.3
Jiras but found nothing I could point out. There are couple of
SessionStore related tickets, but none of them should have an effect
like this.

To narrow it down, it would be helpful to test with other versions, too.
Maybe 0.10.2.2 and 0.11.0.0 to see when the issue was introduced.

Can you also profile v0.10.2.1 so we can compare?

> What would you recommend for our next steps? 

Not sure. If you could help us to track down the issue, that would be
most helpful so get a fix (and you could run from a SNAPSHOT version to
get the fix -- not sure if this would be an option for you).


-Matthias



On 11/7/18 3:47 PM, jonathangor...@newrelic.com wrote:
> Hi Matthias,
> 
> I upgraded to 2.0.0 and we're experiencing the same problem. I've posted a 
> new screengrab of a thread profile:
> 
> https://imgur.com/a/2wncPHw
> 
> From our perspective, it appears something happened after 0.10.2.1 that made 
> the LRU Cache much slower for our use case. What would you recommend for our 
> next steps? 
> 
> Jonathan
> 
> On 2018/11/06 19:22:16, "Matthias J. Sax"  wrote: 
>> Not sure atm why you see a performance degradation. Would need to dig
>> into the details.
>>
>> However, did you consider to upgrade to 2.0 instead or 0.11?
>>
>> Also note that we added a new operator `suppress()` in upcoming 2.1
>> release, that allows you to do rate control without caching:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables
>>
>> Hope this helps.
>>
>>
>> -Matthias
>>
>> On 11/6/18 9:49 AM, Jonathan Gordon wrote:
>>> I have a Kafka Streams app that I'm trying to upgrade from 0.10.2.1 to
>>> 0.11.0.3 but when I do I notice that CPU goes way up and consumption goes
>>> down. A thread profile indicates that the most expensive task is during our
>>> aggregation, fetching from the cache.
>>>
>>> Thread profile with caching:
>>> https://imgur.com/l5VEsC2
>>>
>>> If I disable the cache both performance and consumption are good but we are
>>> producing every single aggregation modification, which is not what we want.
>>>
>>> Thread profile without caching:
>>> https://imgur.com/a/JK3nkou
>>>
>>> I read this thread, which seems relevant e
>>>
>>> https://lists.apache.org/thread.html/2b44e74eaec7172b107bcff96861cf8b4837f55a44714f69d033cc2e@%3Cusers.kafka.apache.org%3E
>>>
>>> Notably: "Note, that caching was _not_ introduced to reduce the writes to
>>> RocksDB, but to reduce the write the the changelog topic and to reduce the
>>> number of records send downstream."
>>>
>>> So how can we reduce the number of records sent downstream while
>>> maintaining the same performance characteristics that we have with caching
>>> turned off? Or put another way, how can I upgrade my app without taking a
>>> hit in performance or behavior?
>>>
>>> Thanks!
>>>
>>
>>



signature.asc
Description: OpenPGP digital signature

Re: [kafka-clients] Re: [VOTE] 2.0.1 RC0

[Manikumar]

Yes, I was about to send closing mail.

On Thu, Nov 8, 2018 at 9:39 PM Ismael Juma  wrote:

> Manikumar, looks like you have the necessary votes. :)
>
> Ismael
>
> On Wed, Nov 7, 2018, 10:45 AM Rajini Sivaram  wrote:
>
>> +1 (binding)
>>
>> Checked source build and unit tests. Ran quickstart with source and
>> binary.
>>
>> Thank you for managing the release, Manikumar!
>>
>> Regards,
>>
>> Rajini
>>
>> On Wed, Nov 7, 2018 at 6:18 PM Gwen Shapira  wrote:
>>
>> > +1 (binding)
>> >
>> > Checked signatures, build and quickstart.
>> >
>> > Thank you for managing the release, Mani!
>> >
>> >
>> > On Thu, Oct 25, 2018 at 7:29 PM Manikumar 
>> > wrote:
>> > >
>> > > Hello Kafka users, developers and client-developers,
>> > >
>> > > This is the first candidate for release of Apache Kafka 2.0.1.
>> > >
>> > > This is a bug fix release closing 49 tickets:
>> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1
>> > >
>> > > Release notes for the 2.0.1 release:
>> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/RELEASE_NOTES.html
>> > >
>> > > *** Please download, test and vote by  Tuesday, October 30, end of day
>> > >
>> > > Kafka's KEYS file containing PGP keys we use to sign the release:
>> > > http://kafka.apache.org/KEYS
>> > >
>> > > * Release artifacts to be voted upon (source and binary):
>> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/
>> > >
>> > > * Maven artifacts to be voted upon:
>> > > https://repository.apache.org/content/groups/staging/
>> > >
>> > > * Javadoc:
>> > > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/javadoc/
>> > >
>> > > * Tag to be voted upon (off 2.0 branch) is the 2.0.1 tag:
>> > > https://github.com/apache/kafka/releases/tag/2.0.1-rc0
>> > >
>> > > * Documentation:
>> > > http://kafka.apache.org/20/documentation.html
>> > >
>> > > * Protocol:
>> > > http://kafka.apache.org/20/protocol.html
>> > >
>> > > * Successful Jenkins builds for the 2.0 branch:
>> > > Unit/integration tests:
>> > https://builds.apache.org/job/kafka-2.0-jdk8/177/
>> > >
>> > > /**
>> > >
>> > > Thanks,
>> > > Manikumar
>> >
>> >
>> >
>> > --
>> > Gwen Shapira
>> > Product Manager | Confluent
>> > 650.450.2760 | @gwenshap
>> > Follow us: Twitter | blog
>> >
>>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAD5tkZYww8XbFFwx5Os4pOjiANGz-Lq81xcSGiTXs-G%2BwpG%3Duw%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

Re: Kafka Streams Session store performance degradation from 0.10.2.1 to 0.11.0.3

[Jonathan Gordon]

Hi Matthias,

I upgraded to 2.0.0 and we're experiencing the same problem. I've posted a
new screengrab of a threadprofile:

https://imgur.com/a/2wncPHw

>From our perspective, it appears something happened after 0.10.2.1 that
made the LRU Cache much slower for our use case. What would you recommend
for our next steps?

Jonathan

On 2018/11/06 19:22:16, "Matthias J. Sax"  wrote:
> Not sure atm why you see a performance degradation. Would need to dig>
> into the details.>
>
> However, did you consider to upgrade to 2.0 instead or 0.11?>
>
> Also note that we added a new operator `suppress()` in upcoming 2.1>
> release, that allows you to do rate control without caching:>
>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables>

>
> Hope this helps.>
>
>
> -Matthias>
>
> On 11/6/18 9:49 AM, Jonathan Gordon wrote:>
> > I have a Kafka Streams app that I'm trying to upgrade from 0.10.2.1 to>
> > 0.11.0.3 but when I do I notice that CPU goes way up and consumption
goes>
> > down. A thread profile indicates that the most expensive task is during
our>
> > aggregation, fetching from the cache.>
> > >
> > Thread profile with caching:>
> > https://imgur.com/l5VEsC2>
> > >
> > If I disable the cache both performance and consumption are good but we
are>
> > producing every single aggregation modification, which is not what we
want.>
> > >
> > Thread profile without caching:>
> > https://imgur.com/a/JK3nkou>
> > >
> > I read this thread, which seems relevant e>
> > >
> >
https://lists.apache.org/thread.html/2b44e74eaec7172b107bcff96861cf8b4837f55a44714f69d033cc2e@%3Cusers.kafka.apache.org%3E>

> > >
> > Notably: "Note, that caching was _not_ introduced to reduce the writes
to>
> > RocksDB, but to reduce the write the the changelog topic and to reduce
the>
> > number of records send downstream.">
> > >
> > So how can we reduce the number of records sent downstream while>
> > maintaining the same performance characteristics that we have with
caching>
> > turned off? Or put another way, how can I upgrade my app without taking
a>
> > hit in performance or behavior?>
> > >
> > Thanks!>
> > >
>
>

Re: [kafka-clients] Re: [VOTE] 2.0.1 RC0

[Manikumar]

+1 (binding)
Built from src and ran tests, verified the artifacts, signatures, Ran core
quick start.

Thanks,
Manikumar

On Thu, Nov 8, 2018 at 12:15 AM Rajini Sivaram 
wrote:

> +1 (binding)
>
> Checked source build and unit tests. Ran quickstart with source and binary.
>
> Thank you for managing the release, Manikumar!
>
> Regards,
>
> Rajini
>
> On Wed, Nov 7, 2018 at 6:18 PM Gwen Shapira  wrote:
>
>> +1 (binding)
>>
>> Checked signatures, build and quickstart.
>>
>> Thank you for managing the release, Mani!
>>
>>
>> On Thu, Oct 25, 2018 at 7:29 PM Manikumar 
>> wrote:
>> >
>> > Hello Kafka users, developers and client-developers,
>> >
>> > This is the first candidate for release of Apache Kafka 2.0.1.
>> >
>> > This is a bug fix release closing 49 tickets:
>> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.0.1
>> >
>> > Release notes for the 2.0.1 release:
>> > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/RELEASE_NOTES.html
>> >
>> > *** Please download, test and vote by  Tuesday, October 30, end of day
>> >
>> > Kafka's KEYS file containing PGP keys we use to sign the release:
>> > http://kafka.apache.org/KEYS
>> >
>> > * Release artifacts to be voted upon (source and binary):
>> > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/
>> >
>> > * Maven artifacts to be voted upon:
>> > https://repository.apache.org/content/groups/staging/
>> >
>> > * Javadoc:
>> > http://home.apache.org/~manikumar/kafka-2.0.1-rc0/javadoc/
>> >
>> > * Tag to be voted upon (off 2.0 branch) is the 2.0.1 tag:
>> > https://github.com/apache/kafka/releases/tag/2.0.1-rc0
>> >
>> > * Documentation:
>> > http://kafka.apache.org/20/documentation.html
>> >
>> > * Protocol:
>> > http://kafka.apache.org/20/protocol.html
>> >
>> > * Successful Jenkins builds for the 2.0 branch:
>> > Unit/integration tests:
>> https://builds.apache.org/job/kafka-2.0-jdk8/177/
>> >
>> > /**
>> >
>> > Thanks,
>> > Manikumar
>>
>>
>>
>> --
>> Gwen Shapira
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAOJcB38LDf1Fgkr8HMG2b2%2B1tA5usQ%3DeeRv0Vjcj63Tq2zoYZQ%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

Marking the coordinator dead and stopped commit offsets

[hacker win7]

Hi,

In kafka 0.10, if kafka consumer log “Marking the coordinator dead ..”  “auto 
commit offset failed ..” then the consumer will not commit the offset any more. 
is there any issues about this, more details in this: 

http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-kafka-consumer-stopped-committing-offsets-td20624.html

Re: Kafka ingestion data not equally distribute among brokers

[Shiuh Rong Yong]

Thank you,

Let me have a look into it.



On 2018/11/07 21:51:21, Manoj Khangaonkar  wrote: 
> Hi> 
> 
> That is part of the Producer API.> 
> 
> For Java see> 
> https://kafka.apache.org/20/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html>
>  
> ProducerRecord> 
> 
> regards> 
> 
> 
> 
> On Tue, Nov 6, 2018 at 5:46 PM Shiuh Rong Yong <> 
> shiuhrongy...@cirosolution.com> wrote:> 
> 
> > Hi Manoj,> 
> >> 
> > I am new on Kafka, is it something mentioned on this site?> 
> > https://kafka.apache.org/0100/javadoc/org/apache/kafka/streams/state/KeyValueStore.html>
> >  
> >> 
> > Where can we set the key and value in Kafka?> 
> >> 
> > Thanks!> 
> >> 
> >> 
> > - Original Message -> 
> > From: "Manoj Khangaonkar" > 
> > To: "users" > 
> > Sent: Tuesday, November 6, 2018 4:36:40 AM> 
> > Subject: Re: Kafka ingestion data not equally distribute among brokers> 
> >> 
> > Hi> 
> >> 
> > In Kafka topic data is split into partitions. Partitions are assigned to> 
> > brokers.> 
> >> 
> > I assume what you are trying to says is that the distribution of messages> 
> > across partitions is not balanced.> 
> >> 
> > Messages are written to topics as key,value. The messages are distributed> 
> > across partitions based on key.> 
> >> 
> > Perhaps you need to pick a better key that gives balanced distribution.> 
> >> 
> > regards> 
> >> 
> >> 
> >> 
> > On Mon, Nov 5, 2018 at 9:22 AM yong  wrote:> 
> >> 
> > > Hi Kafka experts,> 
> > >> 
> > > We have a 3 nodes kafka setup with data streams from mysql binlog, we> 
> > > noticed that from grafana monitoring in and out, some broker nodes are> 
> > > having lower data ingested in and out, is there anyway to balance it? Or> 
> > it> 
> > > is nature of Kafka?> 
> > >> 
> > > Thanks!> 
> > >> 
> >> 
> >> 
> > --> 
> > http://khangaonkar.blogspot.com/> 
> >> 
> 
> 
> -- > 
> http://khangaonkar.blogspot.com/> 
>