Re: [VOTE] 0.11.0.0 RC2

[Ismael Juma]

This vote passes with 7 +1 votes (3 binding) and no 0 or -1 votes.

+1 votes
PMC Members: Jun, Gwen, Guozhang
Committers: Jason, Ismael
Community: Tom Crayford, Jeff Chao

0 votes: none

-1 votes: none

Thanks to everyone who tested, voted and/or contributed to the release.

I'll continue with the release process and the release announcement will follow
in the next few days.

Ismael

P.S. Guozhang's vote was in a separate thread due to some delivery issues
when replying to this thread.

On Fri, Jun 23, 2017 at 2:16 AM, Ismael Juma  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the third candidate for release of Apache Kafka 0.11.0.0.
>
> This is a major version release of Apache Kafka. It includes 32 new KIPs.
> See the release notes and release plan (https://cwiki.apache.org/
> confluence/display/KAFKA/Release+Plan+0.11.0.0) for more details. A few
> feature highlights:
>
> * Exactly-once delivery and transactional messaging
> * Streams exactly-once semantics
> * Admin client with support for topic, ACLs and config management
> * Record headers
> * Request rate quotas
> * Improved resiliency: replication protocol improvement and
> single-threaded controller
> * Richer and more efficient message format
>
> Release notes for the 0.11.0.0 release:
> http://home.apache.org/~ijuma/kafka-0.11.0.0-rc2/RELEASE_NOTES.html
>
> *** Please download, test and vote by Tuesday, June 27, 6pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~ijuma/kafka-0.11.0.0-rc2/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> http://home.apache.org/~ijuma/kafka-0.11.0.0-rc2/javadoc/
>
> * Tag to be voted upon (off 0.11.0 branch) is the 0.11.0.0 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> 8698fa1f41102f1664b05baa4d6953fc9564d91e
>
> * Documentation:
> http://kafka.apache.org/0110/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0110/protocol.html
>
> * Successful Jenkins builds for the 0.11.0 branch:
> Unit/integration tests: https://builds.apache.org/job/
> kafka-0.11.0-jdk7/187/
> System tests: pending (will send an update tomorrow)
>
> /**
>
> Thanks,
> Ismael
>

0.11.0.0 RC2 Vote

[Guozhang Wang]

My vote on the original VOTE thread somehow cannot show up, I have tried
several ways and nothing works. So I am sending it as a new email which
should belong to this thread:

http://mail-archives.apache.org/mod_mbox/kafka-dev/201706.mbox/%3CCAD5tkZYmayKLXV%3Dzut0mMF-KaTNF4QrQehncXXDHmjCmkKgTAg%40mail.gmail.com%3E

---

On Mon, Jun 26, 2017 at 9:36 PM, Guozhang Wang  wrote:

> +1
>
> Verified 0110 web docs and java docs; verified quick start with 2.11 /
> 2.12 scala versions.
>
> One minor observation: on the web docs we show the cmd for 2.11 scala
> version; we'd better make it templated with the version number.
>
> On Mon, Jun 26, 2017 at 3:53 PM, Ismael Juma  wrote:
>
>> Hi Vahid,
>>
>> There are a few known issues when running Kafka on Windows. A PR with some
>> fixes is: https://github.com/apache/kafka/pull/3283. The fact that the
>> index cannot be accessed indicates that it may be a similar issue. I
>> suggest we move this discussion to the relevant JIRAs instead of the
>> release thread.
>>
>> Ismael
>>
>

-- Guozhang

Re: Weird broker lock-up causing an almost global downtime

[Bill Bejeck]

Thanks Vincent.

That's a good start for now.

If you get a chance to forward some logs that would be great.

-Bill

On Tue, Jun 27, 2017 at 6:10 PM, Vincent Rischmann  wrote:

> Sure.
>
> The application reads a topic of keyless events and based on some
> criteria of the event, it creates a new key and uses that for
> `selectKey`.
> Then I groupByKey and count with 3 differents windows. Each count is
> then stored in a database.
>
> The three windows are tumbling windows:
>   - 1 minute window, 1 day retention
>   - 1 hour window, 7 day retention
>   - 1 day window, 15 days retention
> That's basically it for the structure.
> The input topic has 64 partitions.
>
> Tomorrow I can get some logs from Kafka/Zookeeper if that would help.
>
> On Tue, Jun 27, 2017, at 11:41 PM, Bill Bejeck wrote:
> > Hi Vincent,
> >
> > Thanks for reporting this issue.  Could you give us some more details
> > (number topics, partitions per topic and the structure of your Kafka
> > Streams application) so we attempt to reproduce and diagnose the issue?
> >
> > Thanks!
> > Bill
> >
> > On 2017-06-27 14:46 (-0400), Vincent Rischmann  wrote:
> > > Hello. so I had a weird problem this afternoon. I was deploying a
> > > streams application and wanted to delete already existing internal
> > > states data so I ran kafka-streams-application-reset.sh to do it, as
> > > recommended. it wasn't the first time I ran it and it had always worked
> > > before, in staging or in production.
> > > Anyway, I run the command and around 2/3 minutes later we realize a lot
> > > of stuff using the cluster is basically down, unable to fetch or
> > > produce. After investigating logs from the producers and the brokers I
> > > saw that one broker was not responding, despite the process being up.
> It
> > > kept spewing `UnknownTopicOrPartitionException` in the logs, other
> > > brokers were spewing `NotLeaderForPartitionException` regularly. A
> > > zookeeper node logged a lot of this:
> > > > 2017-06-27 15:51:32,897 [myid:2] - INFO  [ProcessThread(sid:2 cport:-
> > > > 1)::PrepRequestProcessor@649] - Got user-level KeeperException when
> > > > processing sessionid:0x159cadf860e0089 type:setData cxid:0x249af08
> > > > zxid:0xb06b3722e txntype:-1 reqpath:n/a Error
> Path:/brokers/topics/event-counter-per-day-store-
> > > > repartition/partitions/4/state Error:KeeperErrorCode = BadVersion for
> > > > /brokers/topics/event-counter-per-day-store-
> > > > repartition/partitions/4/state
> > > So from my point of view it looked like that one broker was "down", not
> > > responding to user requests but yet it was still seen as up by the
> > > cluster and nobody could produce or fetch for the partitions it was
> > > previously a leader. Running kafka-topics.sh --describe I also saw the
> > > leader being -1 for a bunch of partitions.
> > >  As soon as I `kill -9` the process, the cluster stabilized and
> > >  everything went back to normal pretty much in seconds, producers were
> > >  working again as well as consumers. After I restarted the broker, it
> > >  joined the cluster, proceeded to actually do the topic deletions and
> > >  rejoined correctly too.
> > > I'm not sure what exactly happened but that was pretty scary. Has it
> > > happened to anyone else ? My completely uneducated guess is that
> > > somehow, using kafka-streams-application-reset.sh on an application
> with
> > > 5 internal topics caused too many deletions at once and thus caused a
> > > broker to end up with a wrong zookeeper state ? I have no idea if
> that's
> > > a plausible explanation.
> > > Anyway, right now I think I'm going to stop using
> kafka-streams-application-
> > > reset.sh and delete the topics one by one
> > >
>

Re: Weird broker lock-up causing an almost global downtime

[Vincent Rischmann]

Sure.

The application reads a topic of keyless events and based on some
criteria of the event, it creates a new key and uses that for
`selectKey`.
Then I groupByKey and count with 3 differents windows. Each count is
then stored in a database.

The three windows are tumbling windows:
  - 1 minute window, 1 day retention
  - 1 hour window, 7 day retention
  - 1 day window, 15 days retention
That's basically it for the structure.
The input topic has 64 partitions.

Tomorrow I can get some logs from Kafka/Zookeeper if that would help. 

On Tue, Jun 27, 2017, at 11:41 PM, Bill Bejeck wrote:
> Hi Vincent,
> 
> Thanks for reporting this issue.  Could you give us some more details
> (number topics, partitions per topic and the structure of your Kafka
> Streams application) so we attempt to reproduce and diagnose the issue?
> 
> Thanks!
> Bill
> 
> On 2017-06-27 14:46 (-0400), Vincent Rischmann  wrote: 
> > Hello. so I had a weird problem this afternoon. I was deploying a
> > streams application and wanted to delete already existing internal
> > states data so I ran kafka-streams-application-reset.sh to do it, as
> > recommended. it wasn't the first time I ran it and it had always worked
> > before, in staging or in production.
> > Anyway, I run the command and around 2/3 minutes later we realize a lot
> > of stuff using the cluster is basically down, unable to fetch or
> > produce. After investigating logs from the producers and the brokers I
> > saw that one broker was not responding, despite the process being up. It
> > kept spewing `UnknownTopicOrPartitionException` in the logs, other
> > brokers were spewing `NotLeaderForPartitionException` regularly. A
> > zookeeper node logged a lot of this:
> > > 2017-06-27 15:51:32,897 [myid:2] - INFO  [ProcessThread(sid:2 cport:-
> > > 1)::PrepRequestProcessor@649] - Got user-level KeeperException when
> > > processing sessionid:0x159cadf860e0089 type:setData cxid:0x249af08
> > > zxid:0xb06b3722e txntype:-1 reqpath:n/a Error 
> > > Path:/brokers/topics/event-counter-per-day-store-
> > > repartition/partitions/4/state Error:KeeperErrorCode = BadVersion for
> > > /brokers/topics/event-counter-per-day-store-
> > > repartition/partitions/4/state
> > So from my point of view it looked like that one broker was "down", not
> > responding to user requests but yet it was still seen as up by the
> > cluster and nobody could produce or fetch for the partitions it was
> > previously a leader. Running kafka-topics.sh --describe I also saw the
> > leader being -1 for a bunch of partitions.
> >  As soon as I `kill -9` the process, the cluster stabilized and
> >  everything went back to normal pretty much in seconds, producers were
> >  working again as well as consumers. After I restarted the broker, it
> >  joined the cluster, proceeded to actually do the topic deletions and
> >  rejoined correctly too.
> > I'm not sure what exactly happened but that was pretty scary. Has it
> > happened to anyone else ? My completely uneducated guess is that
> > somehow, using kafka-streams-application-reset.sh on an application with
> > 5 internal topics caused too many deletions at once and thus caused a
> > broker to end up with a wrong zookeeper state ? I have no idea if that's
> > a plausible explanation.
> > Anyway, right now I think I'm going to stop using kafka-streams-application-
> > reset.sh and delete the topics one by one
> > 

Re: Weird broker lock-up causing an almost global downtime

[Bill Bejeck]

Hi Vincent,

Thanks for reporting this. 

Could you give some details on your setup (topics, partitions and the structure 
of your streams application) so I can attempt to reproduce the issue?

Thanks!

On 2017-06-27 14:46 (-0400), Vincent Rischmann  wrote: 
> Hello. so I had a weird problem this afternoon. I was deploying a
> streams application and wanted to delete already existing internal
> states data so I ran kafka-streams-application-reset.sh to do it, as
> recommended. it wasn't the first time I ran it and it had always worked
> before, in staging or in production.
> Anyway, I run the command and around 2/3 minutes later we realize a lot
> of stuff using the cluster is basically down, unable to fetch or
> produce. After investigating logs from the producers and the brokers I
> saw that one broker was not responding, despite the process being up. It
> kept spewing `UnknownTopicOrPartitionException` in the logs, other
> brokers were spewing `NotLeaderForPartitionException` regularly. A
> zookeeper node logged a lot of this:
> > 2017-06-27 15:51:32,897 [myid:2] - INFO  [ProcessThread(sid:2 cport:-
> > 1)::PrepRequestProcessor@649] - Got user-level KeeperException when
> > processing sessionid:0x159cadf860e0089 type:setData cxid:0x249af08
> > zxid:0xb06b3722e txntype:-1 reqpath:n/a Error 
> > Path:/brokers/topics/event-counter-per-day-store-
> > repartition/partitions/4/state Error:KeeperErrorCode = BadVersion for
> > /brokers/topics/event-counter-per-day-store-
> > repartition/partitions/4/state
> So from my point of view it looked like that one broker was "down", not
> responding to user requests but yet it was still seen as up by the
> cluster and nobody could produce or fetch for the partitions it was
> previously a leader. Running kafka-topics.sh --describe I also saw the
> leader being -1 for a bunch of partitions.
>  As soon as I `kill -9` the process, the cluster stabilized and
>  everything went back to normal pretty much in seconds, producers were
>  working again as well as consumers. After I restarted the broker, it
>  joined the cluster, proceeded to actually do the topic deletions and
>  rejoined correctly too.
> I'm not sure what exactly happened but that was pretty scary. Has it
> happened to anyone else ? My completely uneducated guess is that
> somehow, using kafka-streams-application-reset.sh on an application with
> 5 internal topics caused too many deletions at once and thus caused a
> broker to end up with a wrong zookeeper state ? I have no idea if that's
> a plausible explanation.
> Anyway, right now I think I'm going to stop using kafka-streams-application-
> reset.sh and delete the topics one by one
> 

Re: Weird broker lock-up causing an almost global downtime

[Bill Bejeck]

Hi Vincent,

Thanks for reporting this issue.  Could you give us some more details (number 
topics, partitions per topic and the structure of your Kafka Streams 
application) so we attempt to reproduce and diagnose the issue?

Thanks!
Bill

On 2017-06-27 14:46 (-0400), Vincent Rischmann  wrote: 
> Hello. so I had a weird problem this afternoon. I was deploying a
> streams application and wanted to delete already existing internal
> states data so I ran kafka-streams-application-reset.sh to do it, as
> recommended. it wasn't the first time I ran it and it had always worked
> before, in staging or in production.
> Anyway, I run the command and around 2/3 minutes later we realize a lot
> of stuff using the cluster is basically down, unable to fetch or
> produce. After investigating logs from the producers and the brokers I
> saw that one broker was not responding, despite the process being up. It
> kept spewing `UnknownTopicOrPartitionException` in the logs, other
> brokers were spewing `NotLeaderForPartitionException` regularly. A
> zookeeper node logged a lot of this:
> > 2017-06-27 15:51:32,897 [myid:2] - INFO  [ProcessThread(sid:2 cport:-
> > 1)::PrepRequestProcessor@649] - Got user-level KeeperException when
> > processing sessionid:0x159cadf860e0089 type:setData cxid:0x249af08
> > zxid:0xb06b3722e txntype:-1 reqpath:n/a Error 
> > Path:/brokers/topics/event-counter-per-day-store-
> > repartition/partitions/4/state Error:KeeperErrorCode = BadVersion for
> > /brokers/topics/event-counter-per-day-store-
> > repartition/partitions/4/state
> So from my point of view it looked like that one broker was "down", not
> responding to user requests but yet it was still seen as up by the
> cluster and nobody could produce or fetch for the partitions it was
> previously a leader. Running kafka-topics.sh --describe I also saw the
> leader being -1 for a bunch of partitions.
>  As soon as I `kill -9` the process, the cluster stabilized and
>  everything went back to normal pretty much in seconds, producers were
>  working again as well as consumers. After I restarted the broker, it
>  joined the cluster, proceeded to actually do the topic deletions and
>  rejoined correctly too.
> I'm not sure what exactly happened but that was pretty scary. Has it
> happened to anyone else ? My completely uneducated guess is that
> somehow, using kafka-streams-application-reset.sh on an application with
> 5 internal topics caused too many deletions at once and thus caused a
> broker to end up with a wrong zookeeper state ? I have no idea if that's
> a plausible explanation.
> Anyway, right now I think I'm going to stop using kafka-streams-application-
> reset.sh and delete the topics one by one
> 

Re: Weird broker lock-up causing an almost global downtime

[Bill Bejeck]

On 2017-06-27 14:46 (-0400), Vincent Rischmann  wrote: 
> Hello. so I had a weird problem this afternoon. I was deploying a
> streams application and wanted to delete already existing internal
> states data so I ran kafka-streams-application-reset.sh to do it, as
> recommended. it wasn't the first time I ran it and it had always worked
> before, in staging or in production.
> Anyway, I run the command and around 2/3 minutes later we realize a lot
> of stuff using the cluster is basically down, unable to fetch or
> produce. After investigating logs from the producers and the brokers I
> saw that one broker was not responding, despite the process being up. It
> kept spewing `UnknownTopicOrPartitionException` in the logs, other
> brokers were spewing `NotLeaderForPartitionException` regularly. A
> zookeeper node logged a lot of this:
> > 2017-06-27 15:51:32,897 [myid:2] - INFO  [ProcessThread(sid:2 cport:-
> > 1)::PrepRequestProcessor@649] - Got user-level KeeperException when
> > processing sessionid:0x159cadf860e0089 type:setData cxid:0x249af08
> > zxid:0xb06b3722e txntype:-1 reqpath:n/a Error 
> > Path:/brokers/topics/event-counter-per-day-store-
> > repartition/partitions/4/state Error:KeeperErrorCode = BadVersion for
> > /brokers/topics/event-counter-per-day-store-
> > repartition/partitions/4/state
> So from my point of view it looked like that one broker was "down", not
> responding to user requests but yet it was still seen as up by the
> cluster and nobody could produce or fetch for the partitions it was
> previously a leader. Running kafka-topics.sh --describe I also saw the
> leader being -1 for a bunch of partitions.
>  As soon as I `kill -9` the process, the cluster stabilized and
>  everything went back to normal pretty much in seconds, producers were
>  working again as well as consumers. After I restarted the broker, it
>  joined the cluster, proceeded to actually do the topic deletions and
>  rejoined correctly too.
> I'm not sure what exactly happened but that was pretty scary. Has it
> happened to anyone else ? My completely uneducated guess is that
> somehow, using kafka-streams-application-reset.sh on an application with
> 5 internal topics caused too many deletions at once and thus caused a
> broker to end up with a wrong zookeeper state ? I have no idea if that's
> a plausible explanation.
> Anyway, right now I think I'm going to stop using kafka-streams-application-
> reset.sh and delete the topics one by one
> 

Re: [ERICSSON] - Trade Compliance: ECCN code for Apache Items

[jan]

I appreciate there may be a loss of subtlety traversing languages, but
this doesn't come over to politely.

I can't help you, the best I can find is
. This *may* be more helpful
than posting here although it covers none of the software you mention,
sorry, but maybe it's worth a look through that page.

I have to admit I'd never heard of ECCN classifications and am
surprised it even exists.

cheers

jan


On 27/06/2017, Axelle Margot  wrote:
> Hello,
>
> You were contacted as part of a new project in France.
>
> For all products you offer, HW and / or SW, we need, as usual, you provide
> some information about your products in order to better prepare the orders.
> So can you send us now the following information about your products:
>
>   *   EU ECCN Code: who define if the product is dual use or not.
>
>  This code is in the format:  Digit - Letter- Digit - Digit - Digit + an
> extension of Letter - Digit - Letter
>
> Example: 5D001.d.2.a to the SW or HW for 5A002.d.2.a
>
> Nota: Ericsson France needs the European ECCN Code, not the US ECCN Code.
>
>
>
>   *   HST code or TARIC code: corresponding to the complete description of
> the property and to define the customs taxes
>
>
>
> If you can't find the ECCN Product code:
> - If you are a reseller, you must contact your supplier as soon as possible
> to send us the information quickly.
> - If it's your equipment, the responsibility of the classification is yours.
> You can refer to Regulation (EC) No 428/2009 of 5 May 2009, or for France
> office, you can also have a look on SBDU website (Service Of Dual-use)
> http://www.entreprises.gouv.fr/biens-double -usage / Home
>
>
>
> We need the EU ECCN Code and HST code for the following family product:
>
>
> Apache
>
> Kafka 0.10.2.1
>
> Zookeper 3.4.9
>
> Puppet client
>
>
>
>
> Regarding the ECCN Code, is this is a mass market product, thanks to precise
> us.
>
>
>
> Please find attached some file who can helps you.
>
> I remind you that in our internal data system we can't record your items
> without the EU ECCN Code. This one is mandatory to valid the order.
>
>
>
> We need these information for the end of next week, the 7th of July.
>
> For Further information, please contact us.
> Best regards,
>
> Axelle MARGOT
> Trade Compliance Adviser / Controle Export
> ERICSSON FRANCE & ERICSSON MAGHREB
> 25, Avenue Carnot
> 91348 MASSY Cedex, France
> Phone : +33 1 81 87 44 11
> Mobile : +33 6 60 14 34 28
> Email : axelle.mar...@ericsson.com
> www.ericsson.com
>
> [Description: Ericsson]
>
>
>
> Remember: a dual-use purpose:
> According to the usual definition, fall into this category "property,
> equipment - including technology, software, know-how immaterial or
> intangible - that could have both civilian and military uses or may - wholly
> or partly - contribute to the development, production, handling, operation,
> maintenance, storage, detection, identification, dissemination of weapons of
> mass destruction '' (WMD - nuclear, biological, chemical , etc.).
>
>
>
>
>
> From: Celine Heerdt
> Sent: lundi 26 juin 2017 15:36
> To: users@kafka.apache.org
> Cc: Axelle Margot 
> Subject: ECCN code for Kafka 0.10.2.1
>
> Hi,
> We in Ericsson are interested in the SW Kafka 0.10.2.1 from Apache. In order
> to begin the trade compliance process, could you please send us the ECCN
> codes for that SW.
> Best regards
> Céline Heerdt
>
>
> [Ericsson]
> Céline Heerdt
> Project Manager
>
> Ericsson
> 25 Avenue Carnot
> 91348 Massy, France
> Mobile +33 6 71 53 92 02
> celine.hee...@ericsson.com
> www.ericsson.com
>
>

Weird broker lock-up causing an almost global downtime

[Vincent Rischmann]

Hello. so I had a weird problem this afternoon. I was deploying a
streams application and wanted to delete already existing internal
states data so I ran kafka-streams-application-reset.sh to do it, as
recommended. it wasn't the first time I ran it and it had always worked
before, in staging or in production.
Anyway, I run the command and around 2/3 minutes later we realize a lot
of stuff using the cluster is basically down, unable to fetch or
produce. After investigating logs from the producers and the brokers I
saw that one broker was not responding, despite the process being up. It
kept spewing `UnknownTopicOrPartitionException` in the logs, other
brokers were spewing `NotLeaderForPartitionException` regularly. A
zookeeper node logged a lot of this:
> 2017-06-27 15:51:32,897 [myid:2] - INFO  [ProcessThread(sid:2 cport:-
> 1)::PrepRequestProcessor@649] - Got user-level KeeperException when
> processing sessionid:0x159cadf860e0089 type:setData cxid:0x249af08
> zxid:0xb06b3722e txntype:-1 reqpath:n/a Error 
> Path:/brokers/topics/event-counter-per-day-store-
> repartition/partitions/4/state Error:KeeperErrorCode = BadVersion for
> /brokers/topics/event-counter-per-day-store-
> repartition/partitions/4/state
So from my point of view it looked like that one broker was "down", not
responding to user requests but yet it was still seen as up by the
cluster and nobody could produce or fetch for the partitions it was
previously a leader. Running kafka-topics.sh --describe I also saw the
leader being -1 for a bunch of partitions.
 As soon as I `kill -9` the process, the cluster stabilized and
 everything went back to normal pretty much in seconds, producers were
 working again as well as consumers. After I restarted the broker, it
 joined the cluster, proceeded to actually do the topic deletions and
 rejoined correctly too.
I'm not sure what exactly happened but that was pretty scary. Has it
happened to anyone else ? My completely uneducated guess is that
somehow, using kafka-streams-application-reset.sh on an application with
5 internal topics caused too many deletions at once and thus caused a
broker to end up with a wrong zookeeper state ? I have no idea if that's
a plausible explanation.
Anyway, right now I think I'm going to stop using kafka-streams-application-
reset.sh and delete the topics one by one

Re: question about document

[Hans Jespersen]

Correct. The use of the word "server" in that sentence is meant as broker (or 
KafkaServer as it shows up in the 'jps' command) not as a physical or virtual 
machine.

-hans

> On Jun 27, 2017, at 1:22 AM, James <896066...@qq.com> wrote:
> 
> Hello,
>At https://kafka.apache.org/intro, I found a sentence:
>Each partition has one server which acts as the "leader" and zero or more 
> servers which act as "followers". 
>In my opinion, it is possible that more than one brokers exist on a 
> machine with different ports and different broker.id, so I think the below is 
> more appropriate:
>  Each partition has one broker which acts as the "leader" and zero or 
> more brokers which act as "followers". 
> Thank you!

Re: [VOTE] 0.11.0.0 RC2

[Ismael Juma]

Hi Gwen,

Thanks for verifying the release candidate.

Regarding KAFKA-4815, the thought did occur to me. However, I was not keen
on moving all the subtasks that are not completed to another issue. Since
you asked, I have done so though. :)

Ismael

On Tue, Jun 27, 2017 at 6:40 AM, Gwen Shapira  wrote:

> Hi,
>
> One super minor issue (that can be fixed without a new RC): The big
> exactly-once stuff (KIP-98) doesn't actually show up as new features in the
> release notes. Most chunks appear as sub-tasks, but the new feature itself
> (KAFKA-4815) is marked as 0.11.1.0 so this is missing. I get that this is
> cosmetic, but having the biggest feature of the release missing from the
> release notes seems like a big deal to me :)
>
> Other than that...
> Validated signatures, ran quickstart, ran tests and everything looks good.
>
> +1 (binding).
>
>
> On Mon, Jun 26, 2017 at 6:54 PM Ismael Juma  wrote:
>
> > Hi Vahid,
> >
> > There are a few known issues when running Kafka on Windows. A PR with
> some
> > fixes is: https://github.com/apache/kafka/pull/3283. The fact that the
> > index cannot be accessed indicates that it may be a similar issue. I
> > suggest we move this discussion to the relevant JIRAs instead of the
> > release thread.
> >
> > Ismael
> >
> > On Mon, Jun 26, 2017 at 11:25 PM, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com> wrote:
> >
> > > Hi Ismael,
> > >
> > > This is the output of core tests from the start until the first failed
> > > test.
> > >
> > > kafka.admin.AdminRackAwareTest >
> > testAssignmentWithRackAwareWithUnevenRacks
> > > PASSED
> > >
> > > kafka.admin.AdminRackAwareTest > testAssignmentWith2ReplicasRackAware
> > > PASSED
> > >
> > > kafka.admin.AdminRackAwareTest >
> > testAssignmentWithRackAwareWithUnevenReplicas
> > > PASSED
> > >
> > > kafka.admin.AdminRackAwareTest > testSkipBrokerWithReplicaAlrea
> dyAssigned
> > > PASSED
> > >
> > > kafka.admin.AdminRackAwareTest > testAssignmentWithRackAware PASSED
> > >
> > > kafka.admin.AdminRackAwareTest > testRackAwareExpansion PASSED
> > >
> > > kafka.admin.AdminRackAwareTest >
> > testAssignmentWith2ReplicasRackAwareWith6Partitions
> > > PASSED
> > >
> > > kafka.admin.AdminRackAwareTest > testAssignmentWith2ReplicasRac
> > > kAwareWith6PartitionsAnd3Brokers PASSED
> > >
> > > kafka.admin.AdminRackAwareTest >
> > testGetRackAlternatedBrokerListAndAssignReplicasToBrokers
> > > PASSED
> > >
> > > kafka.admin.AdminRackAwareTest > testMoreReplicasThanRacks PASSED
> > >
> > > kafka.admin.AdminRackAwareTest > testSingleRack PASSED
> > >
> > > kafka.admin.AdminRackAwareTest >
> > testAssignmentWithRackAwareWithRandomStartIndex
> > > PASSED
> > >
> > > kafka.admin.AdminRackAwareTest > testLargeNumberPartitionsAssignment
> > > PASSED
> > >
> > > kafka.admin.AdminRackAwareTest > testLessReplicasThanRacks PASSED
> > >
> > > kafka.admin.AclCommandTest > testInvalidAuthorizerProperty PASSED
> > >
> > > kafka.admin.ConfigCommandTest > testScramCredentials PASSED
> > >
> > > kafka.admin.ConfigCommandTest > shouldParseArgumentsForTopicsE
> ntityType
> > > PASSED
> > >
> > > kafka.admin.ConfigCommandTest > testUserClientQuotaOpts PASSED
> > >
> > > kafka.admin.ConfigCommandTest > shouldAddTopicConfig PASSED
> > >
> > > kafka.admin.ConfigCommandTest > shouldAddClientConfig PASSED
> > >
> > > kafka.admin.ConfigCommandTest > shouldDeleteBrokerConfig PASSED
> > >
> > > kafka.admin.DeleteConsumerGroupTest >
> > testGroupWideDeleteInZKDoesNothingForActiveConsumerGroup
> > > PASSED
> > >
> > > kafka.admin.ConfigCommandTest > testQuotaConfigEntity PASSED
> > >
> > > kafka.admin.ConfigCommandTest >
> > shouldNotUpdateBrokerConfigIfMalformedBracketConfig
> > > PASSED
> > >
> > > kafka.admin.ConfigCommandTest > shouldFailIfUnrecognisedEntityType
> PASSED
> > >
> > > kafka.admin.AdminTest > testBasicPreferredReplicaElection PASSED
> > >
> > > kafka.admin.ConfigCommandTest >
> > shouldNotUpdateBrokerConfigIfNonExistingConfigIsDeleted
> > > PASSED
> > >
> > > kafka.admin.AdminTest > testPreferredReplicaJsonData PASSED
> > >
> > > kafka.admin.BrokerApiVersionsCommandTest >
> > checkBrokerApiVersionCommandOutput
> > > PASSED
> > >
> > > kafka.admin.ReassignPartitionsCommandTest >
> > shouldRemoveThrottleReplicaListBasedOnProposedAssignment
> > > PASSED
> > >
> > > kafka.admin.ReassignPartitionsCommandTest >
> > shouldFindMovingReplicasMultipleTopics
> > > PASSED
> > >
> > > kafka.admin.ReassignPartitionsCommandTest >
> > shouldNotOverwriteExistingPropertiesWhenLimitIsAdded
> > > PASSED
> > >
> > > kafka.admin.ReassignPartitionsCommandTest >
> > shouldFindMovingReplicasMultipleTopicsAndPartitions
> > > PASSED
> > >
> > > kafka.admin.ReassignPartitionsCommandTest >
> > shouldRemoveThrottleLimitFromAllBrokers
> > > PASSED
> > >
> > > kafka.admin.ReassignPartitionsCommandTest > shouldFindMovingReplicas
> > > PASSED
> > >
> > > kafka.admin.ReassignPartitionsCommandTest >
> > shouldFindMovingReplicasMultiplePartitions
> > > PASSED
> > >
> > > kafka.admin.ReassignP

question about document

[James]

Hello,
At https://kafka.apache.org/intro, I found a sentence:
Each partition has one server which acts as the "leader" and zero or more 
servers which act as "followers". 
In my opinion, it is possible that more than one brokers exist on a machine 
with different ports and different broker.id, so I think the below is more 
appropriate:
  Each partition has one broker which acts as the "leader" and zero or more 
brokers which act as "followers". 
Thank you!

Re: Kafka Stream invalid partitions

[D Stephan]

Thanks Eno, for the info!.  I will try your suggestion.

2017-06-27 14:04 GMT+02:00 Eno Thereska :

> Thanks. I believe we’ve addressed this issue in 0.10.2.1, any chance you
> could try that?
>
> Thanks
> Eno
> > On Jun 27, 2017, at 11:14 AM, D Stephan  wrote:
> >
> > Hello,
> >
> > Thanks for your reply.
> >
> > I use Kafka & KafkaStream version 0.10.2.0.
> > Between the runs, the number of partitions are not intentionally changed
> > programmatically or manually.
> >
> > This topic:  "external-batch-request-store-repartition" is an internally
> > generated topic from this KafkaStream DSL
> > "aggregate"
> > https://kafka.apache.org/0102/javadoc/org/apache/kafka/streams/kstream/
> KGroupedStream.html#aggregate(org.apache.kafka.streams.
> kstream.Initializer,%20org.apache.kafka.streams.kstream.
> Aggregator,%20org.apache.kafka.streams.kstream.Windows,
> %20org.apache.kafka.common.serialization.Serde,%20java.lang.String)
> >
> >
> >
> > I use this API as follows:
> >
> > ...
> > .groupByKey()
> > .aggregate(...)
> > .toStream(...);
> >
> >
> > Please let me know if you need addiotional information.
> >
> > Thanks,
> >
> >
> > 2017-06-27 11:39 GMT+02:00 Eno Thereska :
> >
> >> Hi there,
> >>
> >> Thanks for the report. What version of Kafka are you using? Also,
> between
> >> runs do you change the number of partitions for your topics? I’m trying
> to
> >> figure out how this problem happens, any information on what is
> changing in
> >> between runs is appreciated.
> >>
> >> Thanks,
> >> Eno
> >>
> >>> On Jun 27, 2017, at 8:52 AM, D Stephan  wrote:
> >>>
> >>> Hello,
> >>>
> >>> When I use KafkaStreams DSL GroupByKey and Aggregate APIs, I have
> >> randomly
> >>> & frequently below exceptions:
> >>> In my opinion, it is not practical to clean up the invalid partitions
> >>> everydays.  For your information, this partition is an internal
> partition
> >>> that automatically created by KafkaStream Aggregate API.
> >>> Dou you have any idea or workarounds to mitigate this exception?
> >>>
> >>>
> >>>
> >>>
> >>> 2017-06-21T06:48:31.488210812Z 2017-06-21 06:48:31.487 WARN 1 --- [
> >>> StreamThread-4] o.a.k.s.p.i.InternalTopicManager :
> >>> Could not create internal topics: Existing internal topic
> >>> external-batch-request-store-repartition has invalid partitions.
> >>> Expected: 20 Actual: 1. Use 'kafka.tools.StreamsResetter' tool to clean
> >> up
> >>> invalid topics before processing. Retry #4
> >>>
> >>> 2017-06-21T06:48:31.491071442Z Exception in thread "StreamThread-4"
> >>> org.apache.kafka.streams.errors.StreamsException: Could not create
> >> internal
> >>> topics.
> >>> 2017-06-21T06:48:31.491087557Z at
> >>> org.apache.kafka.streams.processor.internals.InternalTopicManager.
> >> makeReady(InternalTopicManager.java:70)
> >>> 2017-06-21T06:48:31.491091661Z at
> >>> org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.
> >> prepareTopic(StreamPartitionAssignor.java:618)
> >>> 2017-06-21T06:48:31.491096794Z at
> >>> org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.
> >> assign(StreamPartitionAssignor.java:372)
> >>> 2017-06-21T06:48:31.491368662Z at
> >>> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.
> >> performAssignment(ConsumerCoordinator.java:339)
> >>> 2017-06-21T06:48:31.491390576Z at
> >>> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.
> >> onJoinLeader(AbstractCoordinator.java:488)
> >>> 2017-06-21T06:48:31.491397476Z at
> >>> org.apache.kafka.clients.consumer.internals.
> AbstractCoordinator.access$
> >> 1100(AbstractCoordinator.java:89)
> >>> 2017-06-21T06:48:31.491403757Z at
> >>> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$
> >> JoinGroupResponseHandler.handle(AbstractCoordinator.java:438)
> >>> 2017-06-21T06:48:31.491408328Z at
> >>> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$
> >> JoinGroupResponseHandler.handle(AbstractCoordinator.java:420)
> >>> 2017-06-21T06:48:31.491413053Z at
> >>> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$
> >> CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:764)
> >>
> >>
>
>

Re: [VOTE] 0.11.0.0 RC2

[Ismael Juma]

Thanks Jeff, that's helpful. To be clear, this should not affect the Java
client (max.request.size = 1 MB) or librdkafka-based clients
(message.max.bytes = 1MB) with the default settings. I'm a bit surprised
that Sarama doesn't have a similar mechanism. Seems like we'll have to live
with that.

Ismael

On Mon, Jun 26, 2017 at 9:51 PM, Jeff Chao  wrote:

> Hi,
>
> Heroku has been doing additional performance testing on (1) log compaction
> and, separately (2) Go clients with older message format against 0.11-rc2
> brokers with new message format.
>
> For log compaction, we've tested with messages using a single key, messages
> using unique keys, and messages with a bounded key range. There were no
> notable negative performance impacts.
>
> For client testing with old format vs new format, we had Sarama Go async
> producer clients speaking their older client protocol versions and had
> messages producing in a tight loop. This resulted in a high percentage of
> errors, though some messages went through:
>
> Failed to produce message kafka: Failed to produce message to topic
> rc2-topic: kafka server: Message was too large, server rejected it to avoid
> allocation error.
>
> Although this is to be expected as mentioned in the docs (
> http://kafka.apache.org/0110/documentation.html#upgrade_11_message_format)
> where in aggregate messages may become larger than max.message.bytes from
> the broker, we'd like to point out that this might be confusing for users
> running older clients against 0.11. That said, users can however work
> around this issue by tuning their request size to be less than
> max.message.bytes.
>
> This, along with the testing previously mentioned by Tom wraps up our
> performance testing. Overall, we're a +1 (non-binding) for this release,
> but wanted to point out the client issue above.
>
> Thanks,
> Jeff
>
> On Mon, Jun 26, 2017 at 12:41 PM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > Hi Ismael,
> >
> > To answer your questions:
> >
> > 1. Yes, the issues exists in trunk too.
> >
> > 2. I haven't checked with Cygwin, but I can give it a try.
> >
> > And thanks for addressing this issue. I can confirm with your PR I no
> > longer see it.
> > But now that the tests progress I see quite a few errors like this in
> > core:
> >
> > kafka.server.ReplicaFetchTest > classMethod FAILED
> > java.lang.AssertionError: Found unexpected threads,
> > allThreads=Set(ZkClient-EventThread-268-127.0.0.1:56565,
> > ProcessThread(sid:0 cport:56565):, metrics-mete
> > r-tick-thread-2, SessionTracker, Signal Dispatcher, main, Reference
> > Handler, ForkJoinPool-1-worker-1, Attach Listener, ProcessThread(sid:0
> > cport:59720):, ZkClie
> > nt-EventThread-1347-127.0.0.1:59720, kafka-producer-network-thread |
> > producer-1, Test worker-SendThread(127.0.0.1:56565), /127.0.0.1:54942 to
> > /127.0.0.1:54926 w
> > orkers Thread 2, Test worker, SyncThread:0,
> > NIOServerCxn.Factory:/127.0.0.1:0, Test worker-EventThread, Test
> > worker-SendThread(127.0.0.1:59720), /127.0.0.1:5494
> > 2 to /127.0.0.1:54926 workers Thread 3,
> > ZkClient-EventThread-22-127.0.0.1:54976, ProcessThread(sid:0
> > cport:54976):, Test worker-SendThread(127.0.0.1:54976), Fin
> > alizer, metrics-meter-tick-thread-1)
> >
> > I tested on a VM and a physical machine, and both give me a lot of errors
> > like this.
> >
> > Thanks.
> > --Vahid
> >
> >
> >
> >
> > From:   Ismael Juma 
> > To: Vahid S Hashemian 
> > Cc: d...@kafka.apache.org, kafka-clients
> > , Kafka Users 
> > Date:   06/26/2017 03:53 AM
> > Subject:Re: [VOTE] 0.11.0.0 RC2
> >
> >
> >
> > Hi Vahid,
> >
> > Sorry for not replying to the previous email, I had missed it. A couple
> of
> > questions:
> >
> > 1. Is this also happening in trunk? Seems like it should be the case for
> > months and seemingly no-one reported it until the RC stage.
> > 2. Is it correct that this only happens when compiling on Windows without
> > Cygwin?
> >
> > I believe the following PR should fix it, please verify:
> >
> > https://github.com/apache/kafka/pull/3431
> >
> > Ismael
> >
> > On Fri, Jun 23, 2017 at 8:25 PM, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com> wrote:
> >
> > > Hi Ismael,
> > >
> > > Not sure if my response on RC1 was lost or this issue is not a
> > > show-stopper:
> > >
> > > I checked again and with RC2, tests still fail in my Windown 64 bit
> > > environment.
> > >
> > > :clients:checkstyleMain
> > > [ant:checkstyle] [ERROR] C:\Users\User\Downloads\kafka-
> > >
> > 0.11.0.0-src\clients\src\main\java\org\apache\kafka\common\
> > protocol\Errors.java:89:1:
> > > Class Data Abstraction Coupling is 57 (max allowed is 20) classes
> > > [ApiExceptionBuilder, BrokerNotAvailableException,
> > > ClusterAuthorizationException, ConcurrentTransactionsException,
> > > ControllerMovedException, CoordinatorLoadInProgressException,
> > > CoordinatorNotAvailableException, CorruptRecordException,
> > > DuplicateSequenceNumberException, GroupAuthorizationExce

Re: Kafka Stream invalid partitions

[Eno Thereska]

Thanks. I believe we’ve addressed this issue in 0.10.2.1, any chance you could 
try that?

Thanks
Eno
> On Jun 27, 2017, at 11:14 AM, D Stephan  wrote:
> 
> Hello,
> 
> Thanks for your reply.
> 
> I use Kafka & KafkaStream version 0.10.2.0.
> Between the runs, the number of partitions are not intentionally changed
> programmatically or manually.
> 
> This topic:  "external-batch-request-store-repartition" is an internally
> generated topic from this KafkaStream DSL
> "aggregate"
> https://kafka.apache.org/0102/javadoc/org/apache/kafka/streams/kstream/KGroupedStream.html#aggregate(org.apache.kafka.streams.kstream.Initializer,%20org.apache.kafka.streams.kstream.Aggregator,%20org.apache.kafka.streams.kstream.Windows,%20org.apache.kafka.common.serialization.Serde,%20java.lang.String)
> 
> 
> 
> I use this API as follows:
> 
> ...
> .groupByKey()
> .aggregate(...)
> .toStream(...);
> 
> 
> Please let me know if you need addiotional information.
> 
> Thanks,
> 
> 
> 2017-06-27 11:39 GMT+02:00 Eno Thereska :
> 
>> Hi there,
>> 
>> Thanks for the report. What version of Kafka are you using? Also, between
>> runs do you change the number of partitions for your topics? I’m trying to
>> figure out how this problem happens, any information on what is changing in
>> between runs is appreciated.
>> 
>> Thanks,
>> Eno
>> 
>>> On Jun 27, 2017, at 8:52 AM, D Stephan  wrote:
>>> 
>>> Hello,
>>> 
>>> When I use KafkaStreams DSL GroupByKey and Aggregate APIs, I have
>> randomly
>>> & frequently below exceptions:
>>> In my opinion, it is not practical to clean up the invalid partitions
>>> everydays.  For your information, this partition is an internal partition
>>> that automatically created by KafkaStream Aggregate API.
>>> Dou you have any idea or workarounds to mitigate this exception?
>>> 
>>> 
>>> 
>>> 
>>> 2017-06-21T06:48:31.488210812Z 2017-06-21 06:48:31.487 WARN 1 --- [
>>> StreamThread-4] o.a.k.s.p.i.InternalTopicManager :
>>> Could not create internal topics: Existing internal topic
>>> external-batch-request-store-repartition has invalid partitions.
>>> Expected: 20 Actual: 1. Use 'kafka.tools.StreamsResetter' tool to clean
>> up
>>> invalid topics before processing. Retry #4
>>> 
>>> 2017-06-21T06:48:31.491071442Z Exception in thread "StreamThread-4"
>>> org.apache.kafka.streams.errors.StreamsException: Could not create
>> internal
>>> topics.
>>> 2017-06-21T06:48:31.491087557Z at
>>> org.apache.kafka.streams.processor.internals.InternalTopicManager.
>> makeReady(InternalTopicManager.java:70)
>>> 2017-06-21T06:48:31.491091661Z at
>>> org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.
>> prepareTopic(StreamPartitionAssignor.java:618)
>>> 2017-06-21T06:48:31.491096794Z at
>>> org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.
>> assign(StreamPartitionAssignor.java:372)
>>> 2017-06-21T06:48:31.491368662Z at
>>> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.
>> performAssignment(ConsumerCoordinator.java:339)
>>> 2017-06-21T06:48:31.491390576Z at
>>> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.
>> onJoinLeader(AbstractCoordinator.java:488)
>>> 2017-06-21T06:48:31.491397476Z at
>>> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$
>> 1100(AbstractCoordinator.java:89)
>>> 2017-06-21T06:48:31.491403757Z at
>>> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$
>> JoinGroupResponseHandler.handle(AbstractCoordinator.java:438)
>>> 2017-06-21T06:48:31.491408328Z at
>>> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$
>> JoinGroupResponseHandler.handle(AbstractCoordinator.java:420)
>>> 2017-06-21T06:48:31.491413053Z at
>>> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$
>> CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:764)
>> 
>> 

Re: Kafka Stream invalid partitions

[D Stephan]

Hello,

Thanks for your reply.

I use Kafka & KafkaStream version 0.10.2.0.
Between the runs, the number of partitions are not intentionally changed
programmatically or manually.

This topic:  "external-batch-request-store-repartition" is an internally
generated topic from this KafkaStream DSL
"aggregate"
https://kafka.apache.org/0102/javadoc/org/apache/kafka/streams/kstream/KGroupedStream.html#aggregate(org.apache.kafka.streams.kstream.Initializer,%20org.apache.kafka.streams.kstream.Aggregator,%20org.apache.kafka.streams.kstream.Windows,%20org.apache.kafka.common.serialization.Serde,%20java.lang.String)



I use this API as follows:

...
.groupByKey()
.aggregate(...)
.toStream(...);


Please let me know if you need addiotional information.

Thanks,


2017-06-27 11:39 GMT+02:00 Eno Thereska :

> Hi there,
>
> Thanks for the report. What version of Kafka are you using? Also, between
> runs do you change the number of partitions for your topics? I’m trying to
> figure out how this problem happens, any information on what is changing in
> between runs is appreciated.
>
> Thanks,
> Eno
>
> > On Jun 27, 2017, at 8:52 AM, D Stephan  wrote:
> >
> > Hello,
> >
> > When I use KafkaStreams DSL GroupByKey and Aggregate APIs, I have
> randomly
> > & frequently below exceptions:
> > In my opinion, it is not practical to clean up the invalid partitions
> > everydays.  For your information, this partition is an internal partition
> > that automatically created by KafkaStream Aggregate API.
> > Dou you have any idea or workarounds to mitigate this exception?
> >
> >
> >
> >
> > 2017-06-21T06:48:31.488210812Z 2017-06-21 06:48:31.487 WARN 1 --- [
> > StreamThread-4] o.a.k.s.p.i.InternalTopicManager :
> > Could not create internal topics: Existing internal topic
> > external-batch-request-store-repartition has invalid partitions.
> > Expected: 20 Actual: 1. Use 'kafka.tools.StreamsResetter' tool to clean
> up
> > invalid topics before processing. Retry #4
> >
> > 2017-06-21T06:48:31.491071442Z Exception in thread "StreamThread-4"
> > org.apache.kafka.streams.errors.StreamsException: Could not create
> internal
> > topics.
> > 2017-06-21T06:48:31.491087557Z at
> > org.apache.kafka.streams.processor.internals.InternalTopicManager.
> makeReady(InternalTopicManager.java:70)
> > 2017-06-21T06:48:31.491091661Z at
> > org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.
> prepareTopic(StreamPartitionAssignor.java:618)
> > 2017-06-21T06:48:31.491096794Z at
> > org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.
> assign(StreamPartitionAssignor.java:372)
> > 2017-06-21T06:48:31.491368662Z at
> > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.
> performAssignment(ConsumerCoordinator.java:339)
> > 2017-06-21T06:48:31.491390576Z at
> > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.
> onJoinLeader(AbstractCoordinator.java:488)
> > 2017-06-21T06:48:31.491397476Z at
> > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$
> 1100(AbstractCoordinator.java:89)
> > 2017-06-21T06:48:31.491403757Z at
> > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$
> JoinGroupResponseHandler.handle(AbstractCoordinator.java:438)
> > 2017-06-21T06:48:31.491408328Z at
> > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$
> JoinGroupResponseHandler.handle(AbstractCoordinator.java:420)
> > 2017-06-21T06:48:31.491413053Z at
> > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$
> CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:764)
>
>

Re: Kafka Stream invalid partitions

[Eno Thereska]

Hi there,

Thanks for the report. What version of Kafka are you using? Also, between runs 
do you change the number of partitions for your topics? I’m trying to figure 
out how this problem happens, any information on what is changing in between 
runs is appreciated.

Thanks,
Eno

> On Jun 27, 2017, at 8:52 AM, D Stephan  wrote:
> 
> Hello,
> 
> When I use KafkaStreams DSL GroupByKey and Aggregate APIs, I have randomly
> & frequently below exceptions:
> In my opinion, it is not practical to clean up the invalid partitions
> everydays.  For your information, this partition is an internal partition
> that automatically created by KafkaStream Aggregate API.
> Dou you have any idea or workarounds to mitigate this exception?
> 
> 
> 
> 
> 2017-06-21T06:48:31.488210812Z 2017-06-21 06:48:31.487 WARN 1 --- [
> StreamThread-4] o.a.k.s.p.i.InternalTopicManager :
> Could not create internal topics: Existing internal topic
> external-batch-request-store-repartition has invalid partitions.
> Expected: 20 Actual: 1. Use 'kafka.tools.StreamsResetter' tool to clean up
> invalid topics before processing. Retry #4
> 
> 2017-06-21T06:48:31.491071442Z Exception in thread "StreamThread-4"
> org.apache.kafka.streams.errors.StreamsException: Could not create internal
> topics.
> 2017-06-21T06:48:31.491087557Z at
> org.apache.kafka.streams.processor.internals.InternalTopicManager.makeReady(InternalTopicManager.java:70)
> 2017-06-21T06:48:31.491091661Z at
> org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.prepareTopic(StreamPartitionAssignor.java:618)
> 2017-06-21T06:48:31.491096794Z at
> org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.assign(StreamPartitionAssignor.java:372)
> 2017-06-21T06:48:31.491368662Z at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:339)
> 2017-06-21T06:48:31.491390576Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:488)
> 2017-06-21T06:48:31.491397476Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1100(AbstractCoordinator.java:89)
> 2017-06-21T06:48:31.491403757Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:438)
> 2017-06-21T06:48:31.491408328Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:420)
> 2017-06-21T06:48:31.491413053Z at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:764)

kafka producer latency

[Tom Dearman]

Hi,
I have a problem with the latency of my kafka producer under some 
circumstances.  We are running three kafka brokers on version 0.10.2.0 and 3 
zookeepers on version 3.4.8.  The server properties are below, the main 
producer property that we change is that we require acks=all, so at least 2 
will acknowledge our producer requests as we have min.insync.replicas=2.  We 
have It all runs on our own servers but in an OpenShift environment.  The 
zookeeper pods write to local storage, but the kafka broker pods write to Ceph 
storage in such a way that a kafka brokers data is kept and re-assigned to the 
same broker on restart. I am including a link of kafka producer metrics that 
highlights the problem (link is only valid for next 7 days):


https://snapshot.raintank.io/dashboard/snapshot/fjfBMC09aBQcWWzj54uCunqMYhNt4ggO
 


This link has quite a lot of metrics, but the top two are about latency, 
request latency and queue time (I assume that the request latency does not 
include the time spent in the queue). 
@09:29:20, a kafka pod was restarted, the pod was the one which was the overall 
zookeeper leader elector.  This caused very large latency times for our 
messages - average is high, but we are particularly interested in the max 
latency, there was also very high queue time which is just as important to us.
@09:31:00 I had to restart our test client which is causing the load to go to 
the producer as all 14 threads had stopped since they had waited more than 5 
seconds for a producer send.
@09:34:40 I ran a manual rebalance - this hardly causes a blip in the latency.
@09:38:20 a kafka pod was restored, but this time not the one which was the 
overall zookeeper leader elector.  This caused a large latency for the requests 
and queue time.
@09:40:30 I ran a manual rebalance - again it hardly caused a blip.


What I find strange about this is that the rebalance itself seems fine, with a 
controlled shut down, the broker is supposed to do a rebalance before shutting 
down, so I would have thought everything would be off the closing broker and 
the latency of a controlled shut down would be no worse than when I do a manual 
rebalance.


Please can someone help.

Tom


Our server.properties is:

broker.id=-1
listeners=PLAINTEXT://:9092 
num.network.threads=3   
num.io.threads=8
socket.send.buffer.bytes=102400 
socket.receive.buffer.bytes=102400  
socket.request.max.bytes=104857600  
log.dirs=/mnt/data/logs 
num.partitions=20   
num.recovery.threads.per.data.dir=1 
log.retention.hours=168 
log.segment.bytes=1073741824
log.retention.check.interval.ms=30  
zookeeper.connect=zookeeper-0:2181,zookeeper-1:2181,zookeeper-2:2181
zookeeper.connection.timeout.ms=6000
advertised.listeners=PLAINTEXT://kafka-0:9092   
default.replication.factor=3
compression.type=gzip   
delete.topic.enable=true
offsets.retention.minutes=10080 
unclean.leader.election.enable=false
min.insync.replicas=2
auto.leader.rebalance.enable=false  
leader.imbalance.check.interval.seconds=300 
leader.imbalance.per.broker.percentage=10   


inter.broker.protocol.version=0.10.2


log.message.format.version=0.10.2 

Re: Exposing Kafka Topic to External Parties

[Joe San]

So is it in general a good idea to ask my clients who are out of my IT
infrastructure to directly write to my Topic? I'm seeing this as an
anti-pattern. What do you guys think?

On Mon, Jun 26, 2017 at 8:15 PM, Samuel Taylor 
wrote:

> Hi Joe,
>
> For #2, if brokers and clients trust a certain certificate authority (CA),
> you should be able to just sign a new certificate with that CA (without
> having to explicitly share said cert with all parties).
>
> - Samuel
>
> On Fri, Jun 23, 2017 at 3:10 AM, Joe San  wrote:
>
> > Dear Kafka Users,
> >
> > Would you consider it a good practice to expose the Kafka topic directly
> to
> > a 3rd party application? While doing this, I need to satisfy the
> following:
> >
> > 1. I will have say 10 topics and I would need to make sure that only
> > authorized parties are able to write into the Topic
> >
> > 2. If I use certificates (2 way trust), would this mean that when I add
> new
> > broker nodes, I need to make sure that the new certificates are shared
> with
> > all the 3rd parties and their certificates being installed on my new
> broker
> > node?
> >
> > 3. Since I'm exposing my topic directly, a naughty 3rd party could play
> > around and might eventually case a DoS attack?
> >
> > Thanks,
> > Joe
> >
>

Kafka Stream invalid partitions

[D Stephan]

Hello,

When I use KafkaStreams DSL GroupByKey and Aggregate APIs, I have randomly
& frequently below exceptions:
In my opinion, it is not practical to clean up the invalid partitions
everydays.  For your information, this partition is an internal partition
that automatically created by KafkaStream Aggregate API.
Dou you have any idea or workarounds to mitigate this exception?




2017-06-21T06:48:31.488210812Z 2017-06-21 06:48:31.487 WARN 1 --- [
StreamThread-4] o.a.k.s.p.i.InternalTopicManager :
Could not create internal topics: Existing internal topic
external-batch-request-store-repartition has invalid partitions.
Expected: 20 Actual: 1. Use 'kafka.tools.StreamsResetter' tool to clean up
invalid topics before processing. Retry #4

2017-06-21T06:48:31.491071442Z Exception in thread "StreamThread-4"
org.apache.kafka.streams.errors.StreamsException: Could not create internal
topics.
2017-06-21T06:48:31.491087557Z at
org.apache.kafka.streams.processor.internals.InternalTopicManager.makeReady(InternalTopicManager.java:70)
2017-06-21T06:48:31.491091661Z at
org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.prepareTopic(StreamPartitionAssignor.java:618)
2017-06-21T06:48:31.491096794Z at
org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.assign(StreamPartitionAssignor.java:372)
2017-06-21T06:48:31.491368662Z at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:339)
2017-06-21T06:48:31.491390576Z at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:488)
2017-06-21T06:48:31.491397476Z at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1100(AbstractCoordinator.java:89)
2017-06-21T06:48:31.491403757Z at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:438)
2017-06-21T06:48:31.491408328Z at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:420)
2017-06-21T06:48:31.491413053Z at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:764)