Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

Some more details about my use-case When i add the new Kafka Broker, it will not be included in the ISR of the Kafka Topic automatically. We'll need to do Kafka rebalancing - so the load gets balanced across all the Brokers, including the newly added Broker. Can this be done while the Producer

Re: Slow Consumer Group Startup

Bryan, This sounds related to https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance and https://issues.apache.org/jira/browse/KAFKA-4925. -James > On Jun 13, 2017, at 7:02 AM, Bryan Baugher wrote: > > The topics already exist

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

Hello, clarifying the question -> Within the same Kafka cluster, if i want to add Broker & rebalance topic (say topic t1, so it starts using newly added Broker) - can i do that w/o stopping the existing load on topic t1 ? I guess - if i can do that, i should be able to use MirrorMaker to

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

So, mirrormaker let's you use one data center available for live traffic and the other for repartitioning? On Tue, Jun 13, 2017 at 3:14 PM, Mohammed Manna wrote: > You should plan this ahead and make sure that your topic partition and > replication factors are updated

Re: Re-Balancing Kafka topics - Best practices

A few things here… 1) auto.leader.rebalance.enable can have serious performance impacts on larger clusters. It’s currently in need of some development work to enable it to batch leader elections into smaller groups and back off between them, as well as have a better backoff after broker startup.

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

You should plan this ahead and make sure that your topic partition and replication factors are updated correctly using kafka topics admin or a batch job setup for Kafka Mirror Maker. Apache wiki explains how Mirror maker can guarantee a failsafe mirroring. I cannot quite remember how deep the

Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

Hello All, I've streaming data flowing into Kafka topics. What are the best practices wrt. adding or removing Kafka Brokers while data is flowing into Kafka ? for eg. Can I add Broker & re-balance the topics while the data flows into Kafka topic ? Pls. advise. Thanks!

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

Hi Michal, Thanks a lot for your feedback. Your statement about Heartbeat is fair and makes sense. I'll update the KIP accordingly. --Vahid From: Michal Borowiecki To: users@kafka.apache.org, Vahid S Hashemian ,

Re-Balancing Kafka topics - Best practices

Hi All, Fpr Re-balancing Kafka partitions, we can set property -> *auto.leader.rebalance.enable = true in server.properties file.* *Is that the recommended way or is it better to reBalance the kafka partitions manually ?(using *scripts - *kafka-preferred-replica-election.sh, *

Re: KafkaStreams reports RUNNING even though all StreamThreads has crashed

I did create https://issues.apache.org/jira/browse/KAFKA-5440 On 5/16/17 8:46 AM, Eno Thereska wrote: > Hi Andreas, > > Thanks for reporting. This sounds like a bug, but could also be a semantic > thing. Couple of questions: > > - which version of Kafka are you using? > - what is the nature

Re: What could cause unavailable partitions issue and how to fix it?

Folks any input on this? Would help us a lot in case we get into this issue in future as right now only option is to delete the changelog table. On Sat, Jun 10, 2017 at 12:19 AM, Sachin Mittal wrote: > In course of our streaming application we discovered that it went into

Re: kafka in unrecoverable state

I managed to solve it by: - stopping and deleting all data on kafka & zookeeper - stopping all consumers and producers - starting kafka & zookeeper, waiting till they are up - start all consumers & producers, Is there a better way to do this, without data loss and halting everything? On Tue, Jun

Re: kafka in unrecoverable state

A few lines of the logs: [2017-06-13 15:25:37,343] INFO [GroupCoordinator 0]: Stabilized group summarizer generation 701 (kafka.coordinator.GroupCoordinator) [2017-06-13 15:25:37,345] INFO [GroupCoordinator 0]: Assignment received from leader for group summarizer for generation 701

kafka in unrecoverable state

Hi, I'm trying to find out how to at least get my kafka working again. Something went wrong and kafka has halted to a throughput of 0 messages. It keeps looping on stablizing consumer groups, and erroring on an append operation to the offset paritions, plus Not enough replicas. The weird things

ZookeeperLeaderElector broker shutdown rebalance time

On our production system I have observed problems with sending messages to kafka when one of the brokers gets taken down (controlled roll of an open-shift pod containing the kafka broker), the messages time out after 5 seconds (we have retries set up, but before the retries are exhausted, the wait

Re: Slow Consumer Group Startup

The topics already exist prior to starting any of the consumers On Mon, Jun 12, 2017 at 9:35 PM J Pai wrote: > When are the topics on which these consumer groups consume, created? > > -Jaikiran > On 13-Jun-2017, at 3:18 AM, Bryan Baugher wrote: > >

Single Key Aggregation

Hi, I witnessed a strange behaviour in KafkaStreams, need help in understanding the same. I created an application for aggregating clicks per user, I want to process it only for 1 user( i was writing only a single key). When I ran application on one machine, it was running fine.Now, to

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

Hi Vahid, +1 wrt OffsetFetch. The "Additional Food for Thought" mentions Heartbeat as a non-mutating action. I don't think that's true as the GroupCoordinator updates the latestHeartbeat field for the member and adds a new object to the heartbeatPurgatory, see