Cluster expansion kafka 0.8.2.1
Hi, We are planning to expand cluster from 2 node to 8 node. The partition reassignment tool has the option to move topic or partition. Irrespective of number of node additions. If I give all the topics in the topic-to-move.json and all the brokers in the below command then it will give equal distribution of partition among nodes correct ? bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file topics-to-move.json --broker-list "2,3,4,5,6,7" --generate After this I am planning to apply the json --execute --reassignment-json-file option to rebalance cluster. Will this cause any problem ? This step seems to be more general but why it is not documented this way? Thanks Vimal
Re: Cluster expansion and upgrade
Partition reassignment wasn't fully working in 0.8-beta. So you probably will have to upgrade existing brokers to 0.8.1 before running partition reassignment. Also, 0.8.1.1 will be out soon. Thanks, Jun On Thu, Apr 24, 2014 at 9:49 AM, vimpy batra wrote: > Hello, > > We are currently running a kafka 0.8-beta cluster. We are planning to > expand the existing cluster and use 0.8.1 version on the new nodes. Before > upgrading the older ones we want the new ones to participate in the > cluster. We plan to use "reassign-partitions" tool in 0.8.1 to reassign > partitions to the newly added brokers. We will also add additional > partitions to the existing topics. Is there a best practice to go about > expanding clusters? Is this the recommended way to go or should we upgrade > our existing cluster to 0.8.1 first? > I also noticed that there are some fixes on top of 0.8.1. Are they > available to use? > > Thanks,
Cluster expansion and upgrade
Hello, We are currently running a kafka 0.8-beta cluster. We are planning to expand the existing cluster and use 0.8.1 version on the new nodes. Before upgrading the older ones we want the new ones to participate in the cluster. We plan to use "reassign-partitions" tool in 0.8.1 to reassign partitions to the newly added brokers. We will also add additional partitions to the existing topics. Is there a best practice to go about expanding clusters? Is this the recommended way to go or should we upgrade our existing cluster to 0.8.1 first? I also noticed that there are some fixes on top of 0.8.1. Are they available to use? Thanks,
Re: cluster expansion
>> They will occupy 4, 5, 6? Yes, some of the partitions can occupy 4,5,6. >> Can I keep the zk cluster running and shut down kafka processes before I run the partition addition tool? No. The tool won't run properly if you shut down kafka brokers. I think you meant shut down kafka consumer processes. On Mon, Dec 16, 2013 at 11:13 AM, Yu, Libo wrote: > For the 1st question, I don't quite get the answer. > Say I have broker 1,2,3,4,5,6. And the topic A has three partitions on > broker 1,2,3. > If I add another 3 partitions for A. They will occupy 4, 5, 6? > > One more question. Does the partition addition tool work > only with zookeeper? Can I keep the zk cluster running and shut down kafka > processes > before I run the partition addition tool? I am asking this question > because we try to avoid > any message loss when adding a partition in a production environment. > Thanks. > > Regards, > > Libo > > > -Original Message- > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > Sent: Monday, December 16, 2013 12:13 PM > To: users@kafka.apache.org > Subject: Re: cluster expansion > > If we have six brokers, and a topic has three partitions on broker 1, 2, 3. > Now if I add another three partitions. Will they be evenly distributed to > 1,2,3 or 4,5,6? > > Evenly distributed on all six brokers. > > If I use reassignment tool in 0.81 with 0.8 broker, will that work and get > around the bugs? > > Your broker also needs to be on 0.8.1 for it to work correctly. > > > > On Mon, Dec 16, 2013 at 9:06 AM, Yu, Libo wrote: > > > If we have six brokers, and a topic has three partitions on broker 1, 2, > 3. > > Now if I add another three partitions. Will they be evenly distributed > > to > > 1,2,3 or 4,5,6? > > > > One more question. > > If I use reassignment tool in 0.81 with 0.8 broker, will that work and > > get around the bugs? > > > > Regards, > > > > Libo > > > > > > -Original Message- > > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > > Sent: Monday, December 16, 2013 10:26 AM > > To: users@kafka.apache.org > > Subject: RE: cluster expansion > > > > They will be evenly distributed across the nodes in the cluster. > > > > Thanks, > > Neha > > On Dec 16, 2013 6:42 AM, "Yu, Libo" wrote: > > > > > Assume we have three brokers and a topic already has three partitions. > > > Now if I add another three partitions to this topic. Where will the > > > three partitions be located after running the script? > > > > > > Regards, > > > > > > Libo > > > > > > > > > -Original Message- > > > From: Jun Rao [mailto:jun...@gmail.com] > > > Sent: Sunday, December 15, 2013 11:43 PM > > > To: users@kafka.apache.org > > > Subject: Re: cluster expansion > > > > > > Yes, kafka-add-partitons.sh should be stable in 0.8.0. > > > > > > Thanks, > > > > > > Jun > > > > > > > > > On Sat, Dec 14, 2013 at 1:22 AM, Robert Turner > > wrote: > > > > > > > Is the kafka-add-partitons.sh tool stable in 0.8.0? > > > > > > > > > > > > On 13 December 2013 19:21, Neha Narkhede > > > wrote: > > > > > > > > > Partition movement is not an automatic operation in Kafka yet. > > > > > You need > > > > to > > > > > use the partition reassignment tool - > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Replication+tool > > > > s# Re plicationtools-6.ReassignPartitionsTool > > > > > . > > > > > > > > > > > > > > > Also, that feature is stable in 0.8.1. > > > > > > > > > > Thanks, > > > > > Neha > > > > > > > > > > > > > > > On Fri, Dec 13, 2013 at 6:48 AM, Robert Turner > > > > > > > > > wrote: > > > > > > > > > > > No the 6 partitions for each topic will remain on the original > > > brokers. > > > > > You > > > > > > could either reassign some partitions from all topics to the > > > > > > new > > > > brokers > > > > > or > > > > > > you could add partitions to the new brokers for each topic. In > > > > > > 0.8.0 > > > > > there > > > > > > is now an add-partitions tool. > > > > > > > > > > > > Cheers > > > > > >Rob Turner. > > > > > > > > > > > > > > > > > > On 13 December 2013 14:42, Yu, Libo wrote: > > > > > > > > > > > > > Hi folks, > > > > > > > > > > > > > > There are three brokers running 0.8-beta1 in our cluster > > currently. > > > > > > Assume > > > > > > > all the topics have six partitions. > > > > > > > I am going to add another three brokers to the cluster and > > > > > > > upgrade > > > > all > > > > > of > > > > > > > them to 0.8. My question is after the cluster is up, will > > > > > > > the partition be evenly distributed to all brokers? Thanks. > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > > > Libo > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Cheers > > > > > >Rob. > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Cheers > > > >Rob. > > > > > > > > > >
RE: cluster expansion
For the 1st question, I don't quite get the answer. Say I have broker 1,2,3,4,5,6. And the topic A has three partitions on broker 1,2,3. If I add another 3 partitions for A. They will occupy 4, 5, 6? One more question. Does the partition addition tool work only with zookeeper? Can I keep the zk cluster running and shut down kafka processes before I run the partition addition tool? I am asking this question because we try to avoid any message loss when adding a partition in a production environment. Thanks. Regards, Libo -Original Message- From: Neha Narkhede [mailto:neha.narkh...@gmail.com] Sent: Monday, December 16, 2013 12:13 PM To: users@kafka.apache.org Subject: Re: cluster expansion If we have six brokers, and a topic has three partitions on broker 1, 2, 3. Now if I add another three partitions. Will they be evenly distributed to 1,2,3 or 4,5,6? Evenly distributed on all six brokers. If I use reassignment tool in 0.81 with 0.8 broker, will that work and get around the bugs? Your broker also needs to be on 0.8.1 for it to work correctly. On Mon, Dec 16, 2013 at 9:06 AM, Yu, Libo wrote: > If we have six brokers, and a topic has three partitions on broker 1, 2, 3. > Now if I add another three partitions. Will they be evenly distributed > to > 1,2,3 or 4,5,6? > > One more question. > If I use reassignment tool in 0.81 with 0.8 broker, will that work and > get around the bugs? > > Regards, > > Libo > > > -Original Message- > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > Sent: Monday, December 16, 2013 10:26 AM > To: users@kafka.apache.org > Subject: RE: cluster expansion > > They will be evenly distributed across the nodes in the cluster. > > Thanks, > Neha > On Dec 16, 2013 6:42 AM, "Yu, Libo" wrote: > > > Assume we have three brokers and a topic already has three partitions. > > Now if I add another three partitions to this topic. Where will the > > three partitions be located after running the script? > > > > Regards, > > > > Libo > > > > > > -Original Message- > > From: Jun Rao [mailto:jun...@gmail.com] > > Sent: Sunday, December 15, 2013 11:43 PM > > To: users@kafka.apache.org > > Subject: Re: cluster expansion > > > > Yes, kafka-add-partitons.sh should be stable in 0.8.0. > > > > Thanks, > > > > Jun > > > > > > On Sat, Dec 14, 2013 at 1:22 AM, Robert Turner > wrote: > > > > > Is the kafka-add-partitons.sh tool stable in 0.8.0? > > > > > > > > > On 13 December 2013 19:21, Neha Narkhede > > wrote: > > > > > > > Partition movement is not an automatic operation in Kafka yet. > > > > You need > > > to > > > > use the partition reassignment tool - > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Replication+tool > > > s# Re plicationtools-6.ReassignPartitionsTool > > > > . > > > > > > > > > > > > Also, that feature is stable in 0.8.1. > > > > > > > > Thanks, > > > > Neha > > > > > > > > > > > > On Fri, Dec 13, 2013 at 6:48 AM, Robert Turner > > > > > > > wrote: > > > > > > > > > No the 6 partitions for each topic will remain on the original > > brokers. > > > > You > > > > > could either reassign some partitions from all topics to the > > > > > new > > > brokers > > > > or > > > > > you could add partitions to the new brokers for each topic. In > > > > > 0.8.0 > > > > there > > > > > is now an add-partitions tool. > > > > > > > > > > Cheers > > > > >Rob Turner. > > > > > > > > > > > > > > > On 13 December 2013 14:42, Yu, Libo wrote: > > > > > > > > > > > Hi folks, > > > > > > > > > > > > There are three brokers running 0.8-beta1 in our cluster > currently. > > > > > Assume > > > > > > all the topics have six partitions. > > > > > > I am going to add another three brokers to the cluster and > > > > > > upgrade > > > all > > > > of > > > > > > them to 0.8. My question is after the cluster is up, will > > > > > > the partition be evenly distributed to all brokers? Thanks. > > > > > > > > > > > > Regards, > > > > > > > > > > > > Libo > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Cheers > > > > >Rob. > > > > > > > > > > > > > > > > > > > > > -- > > > Cheers > > >Rob. > > > > > >
Re: cluster expansion
If we have six brokers, and a topic has three partitions on broker 1, 2, 3. Now if I add another three partitions. Will they be evenly distributed to 1,2,3 or 4,5,6? Evenly distributed on all six brokers. If I use reassignment tool in 0.81 with 0.8 broker, will that work and get around the bugs? Your broker also needs to be on 0.8.1 for it to work correctly. On Mon, Dec 16, 2013 at 9:06 AM, Yu, Libo wrote: > If we have six brokers, and a topic has three partitions on broker 1, 2, 3. > Now if I add another three partitions. Will they be evenly distributed to > 1,2,3 or 4,5,6? > > One more question. > If I use reassignment tool in 0.81 with 0.8 broker, will that work and get > around the bugs? > > Regards, > > Libo > > > -Original Message- > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > Sent: Monday, December 16, 2013 10:26 AM > To: users@kafka.apache.org > Subject: RE: cluster expansion > > They will be evenly distributed across the nodes in the cluster. > > Thanks, > Neha > On Dec 16, 2013 6:42 AM, "Yu, Libo" wrote: > > > Assume we have three brokers and a topic already has three partitions. > > Now if I add another three partitions to this topic. Where will the > > three partitions be located after running the script? > > > > Regards, > > > > Libo > > > > > > -Original Message- > > From: Jun Rao [mailto:jun...@gmail.com] > > Sent: Sunday, December 15, 2013 11:43 PM > > To: users@kafka.apache.org > > Subject: Re: cluster expansion > > > > Yes, kafka-add-partitons.sh should be stable in 0.8.0. > > > > Thanks, > > > > Jun > > > > > > On Sat, Dec 14, 2013 at 1:22 AM, Robert Turner > wrote: > > > > > Is the kafka-add-partitons.sh tool stable in 0.8.0? > > > > > > > > > On 13 December 2013 19:21, Neha Narkhede > > wrote: > > > > > > > Partition movement is not an automatic operation in Kafka yet. You > > > > need > > > to > > > > use the partition reassignment tool - > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools# > > > Re plicationtools-6.ReassignPartitionsTool > > > > . > > > > > > > > > > > > Also, that feature is stable in 0.8.1. > > > > > > > > Thanks, > > > > Neha > > > > > > > > > > > > On Fri, Dec 13, 2013 at 6:48 AM, Robert Turner > > > > > > > wrote: > > > > > > > > > No the 6 partitions for each topic will remain on the original > > brokers. > > > > You > > > > > could either reassign some partitions from all topics to the new > > > brokers > > > > or > > > > > you could add partitions to the new brokers for each topic. In > > > > > 0.8.0 > > > > there > > > > > is now an add-partitions tool. > > > > > > > > > > Cheers > > > > >Rob Turner. > > > > > > > > > > > > > > > On 13 December 2013 14:42, Yu, Libo wrote: > > > > > > > > > > > Hi folks, > > > > > > > > > > > > There are three brokers running 0.8-beta1 in our cluster > currently. > > > > > Assume > > > > > > all the topics have six partitions. > > > > > > I am going to add another three brokers to the cluster and > > > > > > upgrade > > > all > > > > of > > > > > > them to 0.8. My question is after the cluster is up, will the > > > > > > partition be evenly distributed to all brokers? Thanks. > > > > > > > > > > > > Regards, > > > > > > > > > > > > Libo > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Cheers > > > > >Rob. > > > > > > > > > > > > > > > > > > > > > -- > > > Cheers > > >Rob. > > > > > >
RE: cluster expansion
If we have six brokers, and a topic has three partitions on broker 1, 2, 3. Now if I add another three partitions. Will they be evenly distributed to 1,2,3 or 4,5,6? One more question. If I use reassignment tool in 0.81 with 0.8 broker, will that work and get around the bugs? Regards, Libo -Original Message- From: Neha Narkhede [mailto:neha.narkh...@gmail.com] Sent: Monday, December 16, 2013 10:26 AM To: users@kafka.apache.org Subject: RE: cluster expansion They will be evenly distributed across the nodes in the cluster. Thanks, Neha On Dec 16, 2013 6:42 AM, "Yu, Libo" wrote: > Assume we have three brokers and a topic already has three partitions. > Now if I add another three partitions to this topic. Where will the > three partitions be located after running the script? > > Regards, > > Libo > > > -Original Message- > From: Jun Rao [mailto:jun...@gmail.com] > Sent: Sunday, December 15, 2013 11:43 PM > To: users@kafka.apache.org > Subject: Re: cluster expansion > > Yes, kafka-add-partitons.sh should be stable in 0.8.0. > > Thanks, > > Jun > > > On Sat, Dec 14, 2013 at 1:22 AM, Robert Turner wrote: > > > Is the kafka-add-partitons.sh tool stable in 0.8.0? > > > > > > On 13 December 2013 19:21, Neha Narkhede > wrote: > > > > > Partition movement is not an automatic operation in Kafka yet. You > > > need > > to > > > use the partition reassignment tool - > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools# > > Re plicationtools-6.ReassignPartitionsTool > > > . > > > > > > > > > Also, that feature is stable in 0.8.1. > > > > > > Thanks, > > > Neha > > > > > > > > > On Fri, Dec 13, 2013 at 6:48 AM, Robert Turner > > > > > wrote: > > > > > > > No the 6 partitions for each topic will remain on the original > brokers. > > > You > > > > could either reassign some partitions from all topics to the new > > brokers > > > or > > > > you could add partitions to the new brokers for each topic. In > > > > 0.8.0 > > > there > > > > is now an add-partitions tool. > > > > > > > > Cheers > > > >Rob Turner. > > > > > > > > > > > > On 13 December 2013 14:42, Yu, Libo wrote: > > > > > > > > > Hi folks, > > > > > > > > > > There are three brokers running 0.8-beta1 in our cluster currently. > > > > Assume > > > > > all the topics have six partitions. > > > > > I am going to add another three brokers to the cluster and > > > > > upgrade > > all > > > of > > > > > them to 0.8. My question is after the cluster is up, will the > > > > > partition be evenly distributed to all brokers? Thanks. > > > > > > > > > > Regards, > > > > > > > > > > Libo > > > > > > > > > > > > > > > > > > > > > > -- > > > > Cheers > > > >Rob. > > > > > > > > > > > > > > > -- > > Cheers > >Rob. > > >
RE: cluster expansion
They will be evenly distributed across the nodes in the cluster. Thanks, Neha On Dec 16, 2013 6:42 AM, "Yu, Libo" wrote: > Assume we have three brokers and a topic already has three partitions. > Now if I add another three partitions to this topic. Where will the three > partitions be located after running the script? > > Regards, > > Libo > > > -Original Message- > From: Jun Rao [mailto:jun...@gmail.com] > Sent: Sunday, December 15, 2013 11:43 PM > To: users@kafka.apache.org > Subject: Re: cluster expansion > > Yes, kafka-add-partitons.sh should be stable in 0.8.0. > > Thanks, > > Jun > > > On Sat, Dec 14, 2013 at 1:22 AM, Robert Turner wrote: > > > Is the kafka-add-partitons.sh tool stable in 0.8.0? > > > > > > On 13 December 2013 19:21, Neha Narkhede > wrote: > > > > > Partition movement is not an automatic operation in Kafka yet. You > > > need > > to > > > use the partition reassignment tool - > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Re > > plicationtools-6.ReassignPartitionsTool > > > . > > > > > > > > > Also, that feature is stable in 0.8.1. > > > > > > Thanks, > > > Neha > > > > > > > > > On Fri, Dec 13, 2013 at 6:48 AM, Robert Turner > > wrote: > > > > > > > No the 6 partitions for each topic will remain on the original > brokers. > > > You > > > > could either reassign some partitions from all topics to the new > > brokers > > > or > > > > you could add partitions to the new brokers for each topic. In > > > > 0.8.0 > > > there > > > > is now an add-partitions tool. > > > > > > > > Cheers > > > >Rob Turner. > > > > > > > > > > > > On 13 December 2013 14:42, Yu, Libo wrote: > > > > > > > > > Hi folks, > > > > > > > > > > There are three brokers running 0.8-beta1 in our cluster currently. > > > > Assume > > > > > all the topics have six partitions. > > > > > I am going to add another three brokers to the cluster and > > > > > upgrade > > all > > > of > > > > > them to 0.8. My question is after the cluster is up, will the > > > > > partition be evenly distributed to all brokers? Thanks. > > > > > > > > > > Regards, > > > > > > > > > > Libo > > > > > > > > > > > > > > > > > > > > > > -- > > > > Cheers > > > >Rob. > > > > > > > > > > > > > > > -- > > Cheers > >Rob. > > >
RE: cluster expansion
Assume we have three brokers and a topic already has three partitions. Now if I add another three partitions to this topic. Where will the three partitions be located after running the script? Regards, Libo -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Sunday, December 15, 2013 11:43 PM To: users@kafka.apache.org Subject: Re: cluster expansion Yes, kafka-add-partitons.sh should be stable in 0.8.0. Thanks, Jun On Sat, Dec 14, 2013 at 1:22 AM, Robert Turner wrote: > Is the kafka-add-partitons.sh tool stable in 0.8.0? > > > On 13 December 2013 19:21, Neha Narkhede wrote: > > > Partition movement is not an automatic operation in Kafka yet. You > > need > to > > use the partition reassignment tool - > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Re > plicationtools-6.ReassignPartitionsTool > > . > > > > > > Also, that feature is stable in 0.8.1. > > > > Thanks, > > Neha > > > > > > On Fri, Dec 13, 2013 at 6:48 AM, Robert Turner > wrote: > > > > > No the 6 partitions for each topic will remain on the original brokers. > > You > > > could either reassign some partitions from all topics to the new > brokers > > or > > > you could add partitions to the new brokers for each topic. In > > > 0.8.0 > > there > > > is now an add-partitions tool. > > > > > > Cheers > > >Rob Turner. > > > > > > > > > On 13 December 2013 14:42, Yu, Libo wrote: > > > > > > > Hi folks, > > > > > > > > There are three brokers running 0.8-beta1 in our cluster currently. > > > Assume > > > > all the topics have six partitions. > > > > I am going to add another three brokers to the cluster and > > > > upgrade > all > > of > > > > them to 0.8. My question is after the cluster is up, will the > > > > partition be evenly distributed to all brokers? Thanks. > > > > > > > > Regards, > > > > > > > > Libo > > > > > > > > > > > > > > > > > -- > > > Cheers > > >Rob. > > > > > > > > > -- > Cheers >Rob. >
Re: cluster expansion
Yes, kafka-add-partitons.sh should be stable in 0.8.0. Thanks, Jun On Sat, Dec 14, 2013 at 1:22 AM, Robert Turner wrote: > Is the kafka-add-partitons.sh tool stable in 0.8.0? > > > On 13 December 2013 19:21, Neha Narkhede wrote: > > > Partition movement is not an automatic operation in Kafka yet. You need > to > > use the partition reassignment tool - > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool > > . > > > > > > Also, that feature is stable in 0.8.1. > > > > Thanks, > > Neha > > > > > > On Fri, Dec 13, 2013 at 6:48 AM, Robert Turner > wrote: > > > > > No the 6 partitions for each topic will remain on the original brokers. > > You > > > could either reassign some partitions from all topics to the new > brokers > > or > > > you could add partitions to the new brokers for each topic. In 0.8.0 > > there > > > is now an add-partitions tool. > > > > > > Cheers > > >Rob Turner. > > > > > > > > > On 13 December 2013 14:42, Yu, Libo wrote: > > > > > > > Hi folks, > > > > > > > > There are three brokers running 0.8-beta1 in our cluster currently. > > > Assume > > > > all the topics have six partitions. > > > > I am going to add another three brokers to the cluster and upgrade > all > > of > > > > them to 0.8. My question is after > > > > the cluster is up, will the partition be evenly distributed to all > > > > brokers? Thanks. > > > > > > > > Regards, > > > > > > > > Libo > > > > > > > > > > > > > > > > > -- > > > Cheers > > >Rob. > > > > > > > > > -- > Cheers >Rob. >
Re: cluster expansion
Is the kafka-add-partitons.sh tool stable in 0.8.0? On 13 December 2013 19:21, Neha Narkhede wrote: > Partition movement is not an automatic operation in Kafka yet. You need to > use the partition reassignment tool - > > https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool > . > > > Also, that feature is stable in 0.8.1. > > Thanks, > Neha > > > On Fri, Dec 13, 2013 at 6:48 AM, Robert Turner wrote: > > > No the 6 partitions for each topic will remain on the original brokers. > You > > could either reassign some partitions from all topics to the new brokers > or > > you could add partitions to the new brokers for each topic. In 0.8.0 > there > > is now an add-partitions tool. > > > > Cheers > >Rob Turner. > > > > > > On 13 December 2013 14:42, Yu, Libo wrote: > > > > > Hi folks, > > > > > > There are three brokers running 0.8-beta1 in our cluster currently. > > Assume > > > all the topics have six partitions. > > > I am going to add another three brokers to the cluster and upgrade all > of > > > them to 0.8. My question is after > > > the cluster is up, will the partition be evenly distributed to all > > > brokers? Thanks. > > > > > > Regards, > > > > > > Libo > > > > > > > > > > > > -- > > Cheers > >Rob. > > > -- Cheers Rob.
Re: cluster expansion
I will have to give that a try as well. I have been having a real tough time with the tool in 0.8.0. It fails frequently and I have to roll restarts on my brokers to get partial changes to stick. It should come with a warning! =) On Fri, Dec 13, 2013 at 2:03 PM, Neha Narkhede wrote: > It is HEAD from trunk. So far, it seems stable at LinkedIn. > > > On Fri, Dec 13, 2013 at 12:07 PM, David Birdsong > wrote: > > > On Fri, Dec 13, 2013 at 11:21 AM, Neha Narkhede > >wrote: > > > > > Partition movement is not an automatic operation in Kafka yet. You need > > to > > > use the partition reassignment tool - > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool > > > . > > > > > > > > > Also, that feature is stable in 0.8.1. > > > > > > > > What's the best way to try out 0.8.1? > > > > I have a git clone from git://git.apache.org/kafka.git but none of the > > branches or tags imply 0.8.1 > > > > Is it just HEAD from trunk? > > > > > > > Thanks, > > > Neha > > > > > > > > > On Fri, Dec 13, 2013 at 6:48 AM, Robert Turner > > wrote: > > > > > > > No the 6 partitions for each topic will remain on the original > brokers. > > > You > > > > could either reassign some partitions from all topics to the new > > brokers > > > or > > > > you could add partitions to the new brokers for each topic. In 0.8.0 > > > there > > > > is now an add-partitions tool. > > > > > > > > Cheers > > > >Rob Turner. > > > > > > > > > > > > On 13 December 2013 14:42, Yu, Libo wrote: > > > > > > > > > Hi folks, > > > > > > > > > > There are three brokers running 0.8-beta1 in our cluster currently. > > > > Assume > > > > > all the topics have six partitions. > > > > > I am going to add another three brokers to the cluster and upgrade > > all > > > of > > > > > them to 0.8. My question is after > > > > > the cluster is up, will the partition be evenly distributed to all > > > > > brokers? Thanks. > > > > > > > > > > Regards, > > > > > > > > > > Libo > > > > > > > > > > > > > > > > > > > > > > -- > > > > Cheers > > > >Rob. > > > > > > > > > >
Re: cluster expansion
It is HEAD from trunk. So far, it seems stable at LinkedIn. On Fri, Dec 13, 2013 at 12:07 PM, David Birdsong wrote: > On Fri, Dec 13, 2013 at 11:21 AM, Neha Narkhede >wrote: > > > Partition movement is not an automatic operation in Kafka yet. You need > to > > use the partition reassignment tool - > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool > > . > > > > > > Also, that feature is stable in 0.8.1. > > > > > What's the best way to try out 0.8.1? > > I have a git clone from git://git.apache.org/kafka.git but none of the > branches or tags imply 0.8.1 > > Is it just HEAD from trunk? > > > > Thanks, > > Neha > > > > > > On Fri, Dec 13, 2013 at 6:48 AM, Robert Turner > wrote: > > > > > No the 6 partitions for each topic will remain on the original brokers. > > You > > > could either reassign some partitions from all topics to the new > brokers > > or > > > you could add partitions to the new brokers for each topic. In 0.8.0 > > there > > > is now an add-partitions tool. > > > > > > Cheers > > >Rob Turner. > > > > > > > > > On 13 December 2013 14:42, Yu, Libo wrote: > > > > > > > Hi folks, > > > > > > > > There are three brokers running 0.8-beta1 in our cluster currently. > > > Assume > > > > all the topics have six partitions. > > > > I am going to add another three brokers to the cluster and upgrade > all > > of > > > > them to 0.8. My question is after > > > > the cluster is up, will the partition be evenly distributed to all > > > > brokers? Thanks. > > > > > > > > Regards, > > > > > > > > Libo > > > > > > > > > > > > > > > > > -- > > > Cheers > > >Rob. > > > > > >
Re: cluster expansion
On Fri, Dec 13, 2013 at 11:21 AM, Neha Narkhede wrote: > Partition movement is not an automatic operation in Kafka yet. You need to > use the partition reassignment tool - > > https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool > . > > > Also, that feature is stable in 0.8.1. > > What's the best way to try out 0.8.1? I have a git clone from git://git.apache.org/kafka.git but none of the branches or tags imply 0.8.1 Is it just HEAD from trunk? > Thanks, > Neha > > > On Fri, Dec 13, 2013 at 6:48 AM, Robert Turner wrote: > > > No the 6 partitions for each topic will remain on the original brokers. > You > > could either reassign some partitions from all topics to the new brokers > or > > you could add partitions to the new brokers for each topic. In 0.8.0 > there > > is now an add-partitions tool. > > > > Cheers > >Rob Turner. > > > > > > On 13 December 2013 14:42, Yu, Libo wrote: > > > > > Hi folks, > > > > > > There are three brokers running 0.8-beta1 in our cluster currently. > > Assume > > > all the topics have six partitions. > > > I am going to add another three brokers to the cluster and upgrade all > of > > > them to 0.8. My question is after > > > the cluster is up, will the partition be evenly distributed to all > > > brokers? Thanks. > > > > > > Regards, > > > > > > Libo > > > > > > > > > > > > -- > > Cheers > >Rob. > > >
Re: cluster expansion
Partition movement is not an automatic operation in Kafka yet. You need to use the partition reassignment tool - https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool. Also, that feature is stable in 0.8.1. Thanks, Neha On Fri, Dec 13, 2013 at 6:48 AM, Robert Turner wrote: > No the 6 partitions for each topic will remain on the original brokers. You > could either reassign some partitions from all topics to the new brokers or > you could add partitions to the new brokers for each topic. In 0.8.0 there > is now an add-partitions tool. > > Cheers >Rob Turner. > > > On 13 December 2013 14:42, Yu, Libo wrote: > > > Hi folks, > > > > There are three brokers running 0.8-beta1 in our cluster currently. > Assume > > all the topics have six partitions. > > I am going to add another three brokers to the cluster and upgrade all of > > them to 0.8. My question is after > > the cluster is up, will the partition be evenly distributed to all > > brokers? Thanks. > > > > Regards, > > > > Libo > > > > > > > -- > Cheers >Rob. >
Re: cluster expansion
No the 6 partitions for each topic will remain on the original brokers. You could either reassign some partitions from all topics to the new brokers or you could add partitions to the new brokers for each topic. In 0.8.0 there is now an add-partitions tool. Cheers Rob Turner. On 13 December 2013 14:42, Yu, Libo wrote: > Hi folks, > > There are three brokers running 0.8-beta1 in our cluster currently. Assume > all the topics have six partitions. > I am going to add another three brokers to the cluster and upgrade all of > them to 0.8. My question is after > the cluster is up, will the partition be evenly distributed to all > brokers? Thanks. > > Regards, > > Libo > > -- Cheers Rob.
cluster expansion
Hi folks, There are three brokers running 0.8-beta1 in our cluster currently. Assume all the topics have six partitions. I am going to add another three brokers to the cluster and upgrade all of them to 0.8. My question is after the cluster is up, will the partition be evenly distributed to all brokers? Thanks. Regards, Libo
