Optimal value for leader.imbalance.check.interval.seconds on a large cluster?

We have a 50 node Kafka cluster that manages thousands of partitions. We’re interested in optimizing the auto unbalanced leadership election during rolling restarts of the cluster. We have an automated process that performs broker restarts one by one. After each restart the process verifies t

Rename consumer group?

Hi all, Is there a good way to rename a consumer group? My current plan is to stop the existing consumer and seek a new consumer to the same offsets as the existing consumer, but I'd love to hear any suggestions or experiences people have with this! Thanks, Samuel

Re: Kafka source connector for avro files

We added a Byte Converter which essentially does no conversion. Is this what you are looking for? https://issues.apache.org/jira/browse/KAFKA-4783 On Tue, Apr 25, 2017 at 11:54 AM, Stas Chizhov wrote: > Hi, > > I have a kafka topic with avro messages + schema registry, which is being > backed u

Can I specify replication quota for a topic?

I know in kafka, quotas are used to control client traffic. Recently they proposed something called replication quota which controls replication traffic rate. I've read about them here But I still have few more questions

Kafka source connector for avro files

Hi, I have a kafka topic with avro messages + schema registry, which is being backed up into s3 as a set of avro files. I need to be able to restore a subset of those files into a new topic in the original format with schemas published into a schema registry. Am I right that at the moment there is

Re: the problems of parititons assigned to consumers

For problem 1, you could implement your own PartitionAssignor class to do static assignments. The downside of that is that if one consumer goes offline (due to hardware failure or a bug) then you aren't consuming from the respective partition. For problem 2, if you are finding that the throughput

Re: the problems of parititons assigned to consumers

For Problem 1, you will probably have to either use the low-level API, and/or do manual partition assignment. For problem 2, you can simply re-publish the messages to a new topic with more partitions…or, as in the first problem, just use the low level API. You can also create more consumer gro

Re: Stream applications dying on broker ISR change

Hi Ian, Any chance you could share the full log? Feel free to send it to me directly if you don't want to broadcast it everywhere. Thanks Eno > On 25 Apr 2017, at 17:36, Ian Duffy wrote: > > Thanks again for the quick response Eno. > > We just left the application running in the hope it wou

Re: Stream applications dying on broker ISR change

Thanks again for the quick response Eno. We just left the application running in the hope it would recover; After ~1hour it's still just continuously spilling out the same exception and not managing to continue processing. On 25 April 2017 at 16:24, Eno Thereska wrote: > Hi Ian, > > Retries are

Re: Stream applications dying on broker ISR change

Hi Ian, Retries are sometimes expected and don't always indicate a problem. We should probably adjust the printing of the messages to not print this warning frequently. Are you seeing any crash or does the app proceed? Thanks Eno On 25 Apr 2017 4:02 p.m., "Ian Duffy" wrote: Upgraded a handful

Re: Stream applications dying on broker ISR change

Upgraded a handful of our streams applications to 0.10.2.1 as suggested. Seeing much less issues and much smoother performance. They withstood ISR changes. Seen the following when more consumers were added to a consumer group: 2017-04-25 14:57:37,200 - [WARN] - [1.1.0-11] - [StreamThread-2] o.a.k

Re: [ANNOUNCE] New committer: Rajini Sivaram

Congrats! Eno > On Apr 25, 2017, at 12:17 PM, Rajini Sivaram wrote: > > Thanks everyone! > > It has been a pleasure working with all of you in the Kafka community. Many > thanks to the PMC for this exciting opportunity. > > Regards, > > Rajini > > On Tue, Apr 25, 2017 at 10:51 AM, Damian Guy

Re: [ANNOUNCE] New committer: Rajini Sivaram

Thanks everyone! It has been a pleasure working with all of you in the Kafka community. Many thanks to the PMC for this exciting opportunity. Regards, Rajini On Tue, Apr 25, 2017 at 10:51 AM, Damian Guy wrote: > Congrats > On Tue, 25 Apr 2017 at 09:57, Mickael Maison > wrote: > > > Congratul

Re: [ANNOUNCE] New committer: Rajini Sivaram

Congrats On Tue, 25 Apr 2017 at 09:57, Mickael Maison wrote: > Congratulation Rajini ! > Great news > > On Tue, Apr 25, 2017 at 8:54 AM, Edoardo Comar wrote: > > Congratulations Rajini !!! > > Well deserved > > -- > > Edoardo Comar > > IBM MessageH

Re: [ANNOUNCE] New committer: Rajini Sivaram

Congratulation Rajini ! Great news On Tue, Apr 25, 2017 at 8:54 AM, Edoardo Comar wrote: > Congratulations Rajini !!! > Well deserved > -- > Edoardo Comar > IBM MessageHub > eco...@uk.ibm.com > IBM UK Ltd, Hursley Park, SO21 2JN > > IBM United Kingd

Re: Consumer with another group.id conflicts with streams()

Hi Matthias, thank you for your response. Here are some more details: - my input and output topics have 6 partitions each. - my application instances run from docker images so the is no state left over from a previous run - I have a single docker container running kafka + zookeeper (spotify/kafk

Re: How does replication affect kafka quota?

Thanks for the reply. I have few more questions regarding replication quota a) Will replication quota be valid even when partitions do not move? b) Can I apply replication quota for a particular client-id? If yes, how can I do that? c) Can I apply replication quota for a particular topic? If yes,