I let the tool running for an entire weekend on the test cluster and on Monday it was still saying "failed"...
I have 500 Go per Kafka node and it is a 8 nodes cluster. I am also wondering if I am using the tool correctly. Currently I am running the tool to rebalance everything across the entire cluster. As I have 3 replicas the tool requires at least 3 brokers. Should I add 3 new Kafka nodes and rebalance some topics to these new nodes only? I am afraid to unbalance the cluster with this option. Any suggestions? Thanks for your help. On Mon, Jul 7, 2014 at 9:29 PM, Jun Rao <jun...@gmail.com> wrote: > The failure could mean that the reassignment is still in progress. If you > have lots of data, it may take some time to move the data to new brokers. > You could observe the max lag in each broker to see how far behind new > replicas are (see http://kafka.apache.org/documentation.html#monitoring). > > Thanks, > > Jun > > > On Mon, Jul 7, 2014 at 4:42 PM, Florian Dambrine <flor...@gumgum.com> > wrote: > > > When I run the tool with the --verify option it says failed for the some > > partitions. > > > > The problem is I do not know if it is a zookeeper issue or if the tool > > really failed. > > > > I faced one time the zookeeper issue ( > > https://issues.apache.org/jira/browse/KAFKA-1382) and by killing the > > responsible Kafka the partition switched from failed to completed > > successfully. > > > > What should I do when the Kafka tool says that it failed to move the > > partition? > > > > > > > > > > On Mon, Jul 7, 2014 at 4:33 PM, Clark Haskins > > <chask...@linkedin.com.invalid > > > wrote: > > > > > How does it get stuck? > > > > > > -Clark > > > > > > Clark Elliott Haskins III > > > LinkedIn DDS Site Reliability Engineer > > > Kafka, Zookeeper, Samza SRE > > > Mobile: 505.385.1484 > > > BlueJeans: https://www.bluejeans.com/chaskins > > > > > > > > > chask...@linkedin.com > > > https://www.linkedin.com/in/clarkhaskins > > > There is no place like 127.0.0.1 > > > > > > > > > > > > > > > On 7/7/14, 3:49 PM, "Florian Dambrine" <flor...@gumgum.com> wrote: > > > > > > >Hi, > > > > > > > >I am trying to add new brokers to an existing 8 nodes Kafka cluster. > We > > > >have around 10 topics and the number of partition is set to 50. In > order > > > >to > > > >test the reassgin-partitions scripts, I tried on a sandbox cluster the > > > >following steps. > > > > > > > >I developed a script which is able to parse the reassignment partition > > > >plan > > > >given by the Kafka tool in smaller pieces (reassigning maximum 10 > > > >partitions at a time). > > > > > > > >Unfortunately I faced some issues with the tool that sometimes get > stuck > > > >on > > > >one partition. In this case I have to kill and restart the three > Kafkas > > on > > > >which the partition has been relocated to unlock the process (One > kafka > > at > > > >a time). > > > > > > > >Moreover, I have also faced these two issues that are already on Jira: > > > > > > > >https://issues.apache.org/jira/browse/KAFKA-1382 > > > >https://issues.apache.org/jira/browse/KAFKA-1479 > > > > > > > >We really need to add new nodes to our Kafka cluster, does anybody > have > > > >already rebalance a Kafka 0.8.1.1? What could you advise me? > > > > > > > >Thanks, and feel free to ask me if you need more details. > > > > > > > > > > > > > > > >-- > > > >*Florian Dambrine* | Intern, Big Data > > > >*GumGum* <http://www.gumgum.com/> | *Ads that stick* > > > >209-797-3994 | flor...@gumgum.com > > > > > > > > > > > > -- > > *Florian Dambrine* | Intern, Big Data > > *GumGum* <http://www.gumgum.com/> | *Ads that stick* > > 209-797-3994 | flor...@gumgum.com > > > -- *Florian Dambrine* | Intern, Big Data *GumGum* <http://www.gumgum.com/> | *Ads that stick* 209-797-3994 | flor...@gumgum.com
