What you are doing should work. Although you might find better results by creating the partition assignment JSON file on your own rather than using the move topics to brokers JSON file to generate it. Doing this yourself will allow you to optimize the number of partition moves you are making and how exactly the data becomes balanced(data size, # of partitions etc).
The failure response you report could be caused by an already in-progress reassignment. I would ensure there are no active reassignments in progress and try again. -Clark Clark Elliott Haskins III LinkedIn DDS Site Reliability Engineer Kafka, Zookeeper, Samza SRE Mobile: 505.385.1484 BlueJeans: https://www.bluejeans.com/chaskins chask...@linkedin.com https://www.linkedin.com/in/clarkhaskins There is no place like 127.0.0.1 On 7/10/14, 10:28 AM, "Florian Dambrine" <flor...@gumgum.com> wrote: >Yes exactly, I am just trying to rebalance partitions to the new node. > >I just want to balance it evently across the all cluster. > >Am I doing wrong? > > >On Thu, Jul 10, 2014 at 9:58 AM, Clark Haskins < >chask...@linkedin.com.invalid> wrote: > >> I am confused as to exactly what you are trying to accomplish. Are you >> trying to evenly distribute partitions across your newly added nodes? >> >> -Clark >> >> Clark Elliott Haskins III >> LinkedIn DDS Site Reliability Engineer >> Kafka, Zookeeper, Samza SRE >> Mobile: 505.385.1484 >> BlueJeans: https://www.bluejeans.com/chaskins >> >> >> chask...@linkedin.com >> https://www.linkedin.com/in/clarkhaskins >> There is no place like 127.0.0.1 >> >> >> >> >> On 7/8/14, 11:11 AM, "Florian Dambrine" <flor...@gumgum.com> wrote: >> >> >I realized that I did not respond to you Clark, >> > >> >Here is the entire Json that I sliced in multiple pieces: >> > >> >>>{"version":1,"partitions":[{"topic":"SLOTS","partition":20,"replicas":[1 >>>01 >> >>>421743,105114483,101461702]},{"topic":"RTB","partition":12,"replicas":[1 >>>01 >> >>>671664,101812541,101862816]},{"topic":"B_IMPRESSION","partition":25,"rep >>>li >> >>>cas":[101461782,101812541,101862816]},{"topic":"RTB","partition":19,"rep >>>li >> >>>cas":[101461702,101811991,101812541]},{"topic":"B_IMPRESSION","partition >>>": >> >>>18,"replicas":[101811991,102311671,105114483]},{"topic":"AD_EVENTS","par >>>ti >> >>>tion":16,"replicas":[102311671,101862816,105114483]},{"topic":"ASSETS"," >>>pa >> >>>rtition":28,"replicas":[101671664,101862816,102311671]},{"topic":"AD_EVE >>>NT >> >>>S","partition":34,"replicas":[102311671,101421743,101461702]},{"topic":" >>>AD >> >>>_EVENTS","partition":43,"replicas":[102311671,101461702,101461782]},{"to >>>pi >> >>>c":"D_STATISTICS","partition":4,"replicas":[101811991,101461782,10167166 >>>4] >> >>>},{"topic":"ASSETS","partition":41,"replicas":[102311671,101461782,10167 >>>16 >> >>>64]},{"topic":"SLOTS","partition":16,"replicas":[101812541,101671664,101 >>>81 >> >>>1991]},{"topic":"ASSETS","partition":44,"replicas":[101461702,101812541, >>>10 >> >>>1862816]},{"topic":"ASSETS","partition":6,"replicas":[105114483,10231167 >>>1, >> >>>101421743]},{"topic":"RTB","partition":31,"replicas":[101811991,10511448 >>>3, >> >>>101421743]},{"topic":"B_IMPRESSION","partition":29,"replicas":[101862816 >>>,1 >> >>>01461702,101461782]},{"topic":"SLOTS","partition":21,"replicas":[1014617 >>>02 >> >>>,101421743,101461782]},{"topic":"ASSETS","partition":39,"replicas":[1018 >>>12 >> >>>541,101421743,101461702]},{"topic":"ASSETS","partition":49,"replicas":[1 >>>01 >> >>>862816,101461782,101671664]},{"topic":"AD_EVENTS","partition":32,"replic >>>as >> >>>":[101812541,102311671,105114483]},{"topic":"SLOTS","partition":31,"repl >>>ic >> >>>as":[101461782,101671664,101811991]},{"topic":"RTB","partition":23,"repl >>>ic >> >>>as":[101812541,105114483,101421743]},{"topic":"ASSETS","partition":23,"r >>>ep >> >>>licas":[102311671,101421743,101461702]},{"topic":"B_IMPRESSION","partiti >>>on >> >>>":26,"replicas":[101671664,101862816,102311671]},{"topic":"RTB","partiti >>>on >> >>>":47,"replicas":[101461782,105114483,101421743]},{"topic":"AD_EVENTS","p >>>ar >> >>>tition":26,"replicas":[105114483,101421743,101461702]},{"topic":"B_IMPRE >>>SS >> >>>ION","partition":40,"replicas":[105114483,101811991,101812541]},{"topic" >>>:" >> >>>D_STATISTICS","partition":36,"replicas":[101421743,101671664,101811991]} >>>,{ >> >>>"topic":"SLOTS","partition":28,"replicas":[105114483,101421743,101461702 >>>]} >> >>>,{"topic":"AD_EVENTS","partition":36,"replicas":[101421743,101671664,101 >>>81 >> >>>1991]},{"topic":"D_STATISTICS","partition":49,"replicas":[101811991,1051 >>>14 >> >>>483,101421743]},{"topic":"ASSETS","partition":11,"replicas":[101811991,1 >>>01 >> >>>812541,101862816]},{"topic":"AD_EVENTS","partition":24,"replicas":[10186 >>>28 >> >>>16,102311671,105114483]},{"topic":"RTB","partition":3,"replicas":[101671 >>>66 >> >>>4,101811991,101812541]},{"topic":"RTB","partition":41,"replicas":[101812 >>>54 >> >>>1,101461702,101461782]},{"topic":"ASSETS","partition":14,"replicas":[102 >>>31 >> >>>1671,105114483,101421743]},{"topic":"RTB","partition":30,"replicas":[101 >>>67 >> >>>1664,102311671,105114483]},{"topic":"SLOTS","partition":26,"replicas":[1 >>>01 >> >>>862816,101812541,102311671]},{"topic":"D_STATISTICS","partition":0,"repl >>>ic >> >>>as":[101421743,102311671,105114483]},{"topic":"D_STATISTICS","partition" >>>:8 >> >>>,"replicas":[105114483,101862816,102311671]},{"topic":"B_IMPRESSION","pa >>>rt >> >>>ition":36,"replicas":[101811991,101421743,101461702]},{"topic":"SLOTS"," >>>pa >> >>>rtition":49,"replicas":[101461782,101812541,101862816]},{"topic":"D_STAT >>>IS >> >>>TICS","partition":22,"replicas":[101811991,101812541,101862816]},{"topic >>>": >> >>>"B_IMPRESSION","partition":38,"replicas":[101862816,101461782,101671664] >>>}, >> >>>{"topic":"AD_EVENTS","partition":11,"replicas":[101461782,101461702,1016 >>>71 >> >>>664]},{"topic":"AD_EVENTS","partition":39,"replicas":[101671664,10186281 >>>6, >> >>>102311671]},{"topic":"SLOTS","partition":25,"replicas":[101812541,101811 >>>99 >> >>>1,101862816]},{"topic":"D_STATISTICS","partition":44,"replicas":[1051144 >>>83 >> >>>,101461782,101671664]},{"topic":"ASSETS","partition":47,"replicas":[1018 >>>11 >> >>>991,101421743,101461702]},{"topic":"AD_EVENTS","partition":12,"replicas" >>>:[ >> >>>101671664,101461782,101811991]},{"topic":"SLOTS","partition":4,"replicas >>>": >> >>>[101461782,105114483,101421743]},{"topic":"SLOTS","partition":47,"replic >>>as >> >>>":[101421743,101671664,101811991]},{"topic":"ASSETS","partition":29,"rep >>>li >> >>>cas":[101811991,102311671,105114483]},{"topic":"B_IMPRESSION","partition >>>": >> >>>22,"replicas":[105114483,101461782,101671664]},{"topic":"RTB","partition >>>": >> >>>14,"replicas":[101812541,102311671,105114483]},{"topic":"D_STATISTICS"," >>>pa >> >>>rtition":39,"replicas":[101671664,101862816,102311671]},{"topic":"ASSETS >>>", >> >>>"partition":12,"replicas":[101812541,101862816,102311671]},{"topic":"D_S >>>TA >> >>>TISTICS","partition":31,"replicas":[101811991,101862816,102311671]},{"to >>>pi >> >>>c":"SLOTS","partition":0,"replicas":[102311671,101811991,101812541]},{"t >>>op >> >>>ic":"RTB","partition":22,"replicas":[101811991,102311671,105114483]},{"t >>>op >> >>>ic":"D_STATISTICS","partition":2,"replicas":[101461782,101421743,1014617 >>>02 >> >>>]},{"topic":"SLOTS","partition":43,"replicas":[101812541,102311671,10511 >>>44 >> >>>83]},{"topic":"ASSETS","partition":36,"replicas":[101461782,101862816,10 >>>23 >> >>>11671]},{"topic":"B_IMPRESSION","partition":1,"replicas":[101812541,1018 >>>62 >> >>>816,102311671]},{"topic":"ASSETS","partition":34,"replicas":[101421743,1 >>>01 >> >>>671664,101811991]},{"topic":"AD_EVENTS","partition":40,"replicas":[10181 >>>19 >> >>>91,102311671,105114483]},{"topic":"AD_EVENTS","partition":5,"replicas":[ >>>10 >> >>>1812541,101671664,101811991]},{"topic":"ASSETS","partition":25,"replicas >>>": >> >>>[101421743,101461782,101671664]},{"topic":"RTB","partition":8,"replicas" >>>:[ >> >>>105114483,101421743,101461702]},{"topic":"B_IMPRESSION","partition":48," >>>re >> >>>plicas":[102311671,101811991,101812541]},{"topic":"RTB","partition":2,"r >>>ep >> >>>licas":[101461782,101671664,101811991]},{"topic":"AD_EVENTS","partition" >>>:1 >> >>>5,"replicas":[101862816,101812541,102311671]},{"topic":"RTB","partition" >>>:4 >> >>>,"replicas":[101811991,101812541,101862816]},{"topic":"SLOTS","partition >>>": >> >>>22,"replicas":[101461782,101461702,101671664]},{"topic":"SLOTS","partiti >>>on >> >>>":19,"replicas":[105114483,102311671,101421743]},{"topic":"AD_EVENTS","p >>>ar >> >>>tition":21,"replicas":[101671664,101811991,101812541]},{"topic":"D_STATI >>>ST >> >>>ICS","partition":26,"replicas":[105114483,101421743,101461702]},{"topic" >>>:" >> >>>ASSETS","partition":32,"replicas":[102311671,101461702,101461782]},{"top >>>ic >> >>>":"RTB","partition":38,"replicas":[101461782,102311671,105114483]},{"top >>>ic >> >>>":"D_STATISTICS","partition":6,"replicas":[101862816,101811991,101812541 >>>]} >> >>>,{"topic":"B_IMPRESSION","partition":28,"replicas":[101812541,101421743, >>>10 >> >>>1461702]},{"topic":"RTB","partition":36,"replicas":[101421743,101812541, >>>10 >> >>>1862816]},{"topic":"AD_EVENTS","partition":49,"replicas":[101811991,1051 >>>14 >> >>>483,101421743]},{"topic":"SLOTS","partition":17,"replicas":[101862816,10 >>>18 >> >>>11991,101812541]},{"topic":"D_STATISTICS","partition":24,"replicas":[101 >>>86 >> >>>2816,102311671,105114483]},{"topic":"D_STATISTICS","partition":37,"repli >>>ca >> >>>s":[101461702,101811991,101812541]},{"topic":"AD_EVENTS","partition":46, >>>"r >> >>>eplicas":[101461702,101812541,101862816]},{"topic":"B_IMPRESSION","parti >>>ti >> >>>on":6,"replicas":[101461702,101461782,101671664]},{"topic":"ASSETS","par >>>ti >> >>>tion":4,"replicas":[101862816,101812541,102311671]},{"topic":"ASSETS","p >>>ar >> >>>tition":0,"replicas":[101461782,101461702,101671664]},{"topic":"ASSETS", >>>"p >> >>>artition":30,"replicas":[101812541,105114483,101421743]},{"topic":"RTB", >>>"p >> >>>artition":18,"replicas":[101421743,101671664,101811991]},{"topic":"D_STA >>>TI >> >>>STICS","partition":29,"replicas":[101461782,101811991,101812541]},{"topi >>>c" >> >>>:"RTB","partition":9,"replicas":[101421743,101461782,101671664]},{"topic >>>": >> >>>"AD_EVENTS","partition":48,"replicas":[101671664,102311671,105114483]},{ >>>"t >> >>>opic":"D_STATISTICS","partition":47,"replicas":[101461782,101862816,1023 >>>11 >> >>>671]},{"topic":"SLOTS","partition":46,"replicas":[105114483,101461782,10 >>>16 >> >>>71664]},{"topic":"RTB","partition":48,"replicas":[101671664,101421743,10 >>>14 >> >>>61702]},{"topic":"ASSETS","partition":35,"replicas":[101461702,101811991 >>>,1 >> >>>01812541]},{"topic":"B_IMPRESSION","partition":34,"replicas":[101461782, >>>10 >> >>>1862816,102311671]},{"topic":"SLOTS","partition":1,"replicas":[105114483 >>>,1 >> >>>01812541,101862816]},{"topic":"AD_EVENTS","partition":18,"replicas":[101 >>>42 >> >>>1743,101461702,101461782]},{"topic":"D_STATISTICS","partition":38,"repli >>>ca >> >>>s":[101461782,101812541,101862816]},{"topic":"ASSETS","partition":8,"rep >>>li >> >>>cas":[101461702,101421743,101461782]},{"topic":"ASSETS","partition":26," >>>re >> >>>plicas":[101461702,101671664,101811991]},{"topic":"D_STATISTICS","partit >>>io >> >>>n":21,"replicas":[101671664,101811991,101812541]},{"topic":"B_IMPRESSION >>>", >> >>>"partition":10,"replicas":[101812541,102311671,105114483]},{"topic":"AD_ >>>EV >> >>>ENTS","partition":2,"replicas":[101461782,101421743,101461702]},{"topic" >>>:" >> >>>ASSETS","partition":19,"replicas":[101671664,101812541,101862816]},{"top >>>ic >> >>>":"D_STATISTICS","partition":48,"replicas":[101671664,102311671,10511448 >>>3] >> >>>},{"topic":"RTB","partition":32,"replicas":[101812541,101421743,10146170 >>>2] >> >>>},{"topic":"D_STATISTICS","partition":28,"replicas":[101461702,101671664 >>>,1 >> >>>01811991]},{"topic":"B_IMPRESSION","partition":35,"replicas":[101671664, >>>10 >> >>>2311671,105114483]},{"topic":"SLOTS","partition":3,"replicas":[101461702 >>>,1 >> >>>02311671,105114483]},{"topic":"D_STATISTICS","partition":41,"replicas":[ >>>10 >> >>>1812541,105114483,101421743]},{"topic":"B_IMPRESSION","partition":17,"re >>>pl >> >>>icas":[101671664,101812541,101862816]},{"topic":"B_IMPRESSION","partitio >>>n" >> >>>:4,"replicas":[105114483,101421743,101461702]},{"topic":"RTB","partition >>>": >> >>>11,"replicas":[101461782,101811991,101812541]},{"topic":"SLOTS","partiti >>>on >> >>>":41,"replicas":[101671664,101812541,101862816]},{"topic":"B_IMPRESSION" >>>," >> >>>partition":27,"replicas":[101811991,105114483,101421743]},{"topic":"RTB" >>>," >> >>>partition":17,"replicas":[105114483,101461702,101461782]},{"topic":"D_ST >>>AT >> >>>ISTICS","partition":15,"replicas":[101862816,101812541,102311671]},{"top >>>ic >> >>>":"RTB","partition":33,"replicas":[101862816,101461702,101461782]},{"top >>>ic >> >>>":"B_IMPRESSION","partition":0,"replicas":[101811991,101812541,101862816 >>>]} >> >>>,{"topic":"ASSETS","partition":2,"replicas":[101811991,101671664,1018125 >>>41 >> >>>]},{"topic":"AD_EVENTS","partition":28,"replicas":[101461702,101671664,1 >>>01 >> >>>811991]},{"topic":"D_STATISTICS","partition":3,"replicas":[101671664,101 >>>46 >> >>>1702,101461782]},{"topic":"RTB","partition":13,"replicas":[101811991,101 >>>86 >> >>>2816,102311671]},{"topic":"D_STATISTICS","partition":46,"replicas":[1014 >>>61 >> >>>702,101812541,101862816]},{"topic":"AD_EVENTS","partition":38,"replicas" >>>:[ >> >>>101461782,101812541,101862816]},{"topic":"RTB","partition":39,"replicas" >>>:[ >> >>>101671664,105114483,101421743]},{"topic":"RTB","partition":16,"replicas" >>>:[ >> >>>102311671,101421743,101461702]},{"topic":"AD_EVENTS","partition":33,"rep >>>li >> >>>cas":[101862816,105114483,101421743]},{"topic":"B_IMPRESSION","partition >>>": >> >>>20,"replicas":[101862816,101421743,101461702]},{"topic":"AD_EVENTS","par >>>ti >> >>>tion":30,"replicas":[101671664,101812541,101862816]},{"topic":"B_IMPRESS >>>IO >> >>>N","partition":9,"replicas":[101811991,101862816,102311671]},{"topic":"R >>>TB >> >>>","partition":5,"replicas":[101812541,101862816,102311671]},{"topic":"AD >>>_E >> >>>VENTS","partition":25,"replicas":[102311671,105114483,101421743]},{"topi >>>c" >> >>>:"AD_EVENTS","partition":4,"replicas":[101811991,101461782,101671664]},{ >>>"t >> >>>opic":"D_STATISTICS","partition":42,"replicas":[101862816,101421743,1014 >>>61 >> >>>702]},{"topic":"SLOTS","partition":32,"replicas":[101671664,101811991,10 >>>18 >> >>>12541]},{"topic":"SLOTS","partition":48,"replicas":[101461702,101811991, >>>10 >> >>>1812541]},{"topic":"D_STATISTICS","partition":27,"replicas":[101421743,1 >>>01 >> >>>461782,101671664]},{"topic":"ASSETS","partition":46,"replicas":[10167166 >>>4, >> >>>105114483,101421743]},{"topic":"RTB","partition":42,"replicas":[10186281 >>>6, >> >>>101461782,101671664]},{"topic":"D_STATISTICS","partition":5,"replicas":[ >>>10 >> >>>1812541,101671664,101811991]},{"topic":"RTB","partition":15,"replicas":[ >>>10 >> >>>1862816,105114483,101421743]},{"topic":"AD_EVENTS","partition":13,"repli >>>ca >> >>>s":[101811991,101671664,101812541]},{"topic":"SLOTS","partition":12,"rep >>>li >> >>>cas":[101461702,105114483,101421743]},{"topic":"ASSETS","partition":7,"r >>>ep >> >>>licas":[101421743,105114483,101461702]},{"topic":"ASSETS","partition":48 >>>," >> >>>replicas":[101812541,101461702,101461782]},{"topic":"B_IMPRESSION","part >>>it >> >>>ion":2,"replicas":[101862816,102311671,105114483]},{"topic":"D_STATISTIC >>>S" >> >>>,"partition":23,"replicas":[101812541,101862816,102311671]},{"topic":"AS >>>SE >> >>>TS","partition":3,"replicas":[101812541,101811991,101862816]},{"topic":" >>>SL >> >>>OTS","partition":33,"replicas":[101811991,101812541,101862816]},{"topic" >>>:" >> >>>ASSETS","partition":5,"replicas":[102311671,101862816,105114483]},{"topi >>>c" >> >>>:"SLOTS","partition":11,"replicas":[101421743,102311671,105114483]},{"to >>>pi >> >>>c":"SLOTS","partition":35,"replicas":[101862816,102311671,105114483]},{" >>>to >> >>>pic":"ASSETS","partition":22,"replicas":[101862816,105114483,101421743]} >>>,{ >> >>>"topic":"B_IMPRESSION","partition":32,"replicas":[101421743,101811991,10 >>>18 >> >>>12541]},{"topic":"AD_EVENTS","partition":1,"replicas":[101461702,1051144 >>>83 >> >>>,101421743]},{"topic":"ASSETS","partition":33,"replicas":[105114483,1014 >>>61 >> >>>782,101671664]},{"topic":"AD_EVENTS","partition":6,"replicas":[101862816 >>>,1 >> >>>01811991,101812541]},{"topic":"ASSETS","partition":17,"replicas":[101461 >>>70 >> >>>2,101461782,101671664]},{"topic":"B_IMPRESSION","partition":49,"replicas >>>": >> >>>[105114483,101812541,101862816]},{"topic":"RTB","partition":6,"replicas" >>>:[ >> >>>101862816,102311671,105114483]},{"topic":"RTB","partition":35,"replicas" >>>:[ >> >>>105114483,101671664,101811991]},{"topic":"SLOTS","partition":27,"replica >>>s" >> >>>:[102311671,105114483,101421743]},{"topic":"SLOTS","partition":34,"repli >>>ca >> >>>s":[101812541,101862816,102311671]},{"topic":"SLOTS","partition":30,"rep >>>li >> >>>cas":[101461702,101461782,101671664]},{"topic":"D_STATISTICS","partition >>>": >> >>>10,"replicas":[101461702,101421743,101461782]},{"topic":"AD_EVENTS","par >>>ti >> >>>tion":45,"replicas":[101421743,101811991,101812541]},{"topic":"SLOTS","p >>>ar >> >>>tition":8,"replicas":[101862816,101671664,101811991]},{"topic":"D_STATIS >>>TI >> >>>CS","partition":33,"replicas":[101862816,105114483,101421743]},{"topic": >>>"A >> >>>D_EVENTS","partition":14,"replicas":[101812541,101811991,101862816]},{"t >>>op >> >>>ic":"SLOTS","partition":13,"replicas":[101461782,101421743,101461702]},{ >>>"t >> >>>opic":"AD_EVENTS","partition":17,"replicas":[105114483,102311671,1014217 >>>43 >> >>>]},{"topic":"ASSETS","partition":24,"replicas":[105114483,101461702,1014 >>>61 >> >>>782]},{"topic":"ASSETS","partition":20,"replicas":[101811991,101862816,1 >>>02 >> >>>311671]},{"topic":"RTB","partition":43,"replicas":[102311671,101671664,1 >>>01 >> >>>811991]},{"topic":"RTB","partition":44,"replicas":[105114483,101811991,1 >>>01 >> >>>812541]},{"topic":"SLOTS","partition":15,"replicas":[101811991,101461782 >>>,1 >> >>>01671664]},{"topic":"ASSETS","partition":18,"replicas":[101461782,101811 >>>99 >> >>>1,101812541]},{"topic":"AD_EVENTS","partition":7,"replicas":[102311671,1 >>>01 >> >>>812541,101862816]},{"topic":"RTB","partition":27,"replicas":[101421743,1 >>>01 >> >>>811991,101812541]},{"topic":"RTB","partition":21,"replicas":[101671664,1 >>>01 >> >>>862816,102311671]},{"topic":"B_IMPRESSION","partition":33,"replicas":[10 >>>14 >> >>>61702,101812541,101862816]},{"topic":"RTB","partition":46,"replicas":[10 >>>14 >> >>>61702,102311671,105114483]},{"topic":"RTB","partition":10,"replicas":[10 >>>14 >> >>>61702,101671664,101811991]},{"topic":"AD_EVENTS","partition":41,"replica >>>s" >> >>>:[101812541,105114483,101421743]},{"topic":"RTB","partition":49,"replica >>>s" >> >>>:[101811991,101461702,101461782]},{"topic":"D_STATISTICS","partition":19 >>>," >> >>>replicas":[101461702,101461782,101671664]},{"topic":"AD_EVENTS","partiti >>>on >> >>>":10,"replicas":[101461702,101421743,101461782]},{"topic":"SLOTS","parti >>>ti >> >>>on":10,"replicas":[105114483,101862816,102311671]},{"topic":"ASSETS","pa >>>rt >> >>>ition":42,"replicas":[105114483,101671664,101811991]},{"topic":"B_IMPRES >>>SI >> >>>ON","partition":14,"replicas":[101421743,101461782,101671664]},{"topic": >>>"R >> >>>TB","partition":7,"replicas":[102311671,105114483,101421743]},{"topic":" >>>SL >> >>>OTS","partition":42,"replicas":[101811991,101862816,102311671]},{"topic" >>>:" >> >>>B_IMPRESSION","partition":8,"replicas":[101671664,101811991,101812541]}, >>>{" >> >>>topic":"SLOTS","partition":18,"replicas":[102311671,101862816,105114483] >>>}, >> >>>{"topic":"B_IMPRESSION","partition":23,"replicas":[101421743,101671664,1 >>>01 >> >>>811991]},{"topic":"B_IMPRESSION","partition":31,"replicas":[105114483,10 >>>16 >> >>>71664,101811991]},{"topic":"D_STATISTICS","partition":1,"replicas":[1014 >>>61 >> >>>702,105114483,101421743]},{"topic":"B_IMPRESSION","partition":44,"replic >>>as >> >>>":[101671664,105114483,101421743]},{"topic":"RTB","partition":28,"replic >>>as >> >>>":[101461702,101812541,101862816]},{"topic":"ASSETS","partition":31,"rep >>>li >> >>>cas":[101862816,101421743,101461702]},{"topic":"RTB","partition":25,"rep >>>li >> >>>cas":[102311671,101461702,101461782]},{"topic":"B_IMPRESSION","partition >>>": >> >>>37,"replicas":[101812541,101461702,101461782]},{"topic":"B_IMPRESSION"," >>>pa >> >>>rtition":13,"replicas":[105114483,101461702,101461782]},{"topic":"B_IMPR >>>ES >> >>>SION","partition":24,"replicas":[101461702,101811991,101812541]},{"topic >>>": >> >>>"AD_EVENTS","partition":8,"replicas":[105114483,101862816,102311671]},{" >>>to >> >>>pic":"ASSETS","partition":9,"replicas":[101461782,101671664,101811991]}, >>>{" >> >>>topic":"ASSETS","partition":43,"replicas":[101421743,101811991,101812541 >>>]} >> >>>,{"topic":"SLOTS","partition":37,"replicas":[105114483,101461702,1014617 >>>82 >> >>>]},{"topic":"B_IMPRESSION","partition":19,"replicas":[101812541,10511448 >>>3, >> >>>101421743]},{"topic":"AD_EVENTS","partition":3,"replicas":[101671664,101 >>>46 >> >>>1702,101461782]},{"topic":"SLOTS","partition":2,"replicas":[101421743,10 >>>18 >> >>>62816,102311671]},{"topic":"B_IMPRESSION","partition":12,"replicas":[102 >>>31 >> >>>1671,101421743,101461702]},{"topic":"SLOTS","partition":29,"replicas":[1 >>>01 >> >>>421743,101461702,101461782]},{"topic":"ASSETS","partition":38,"replicas" >>>:[ >> >>>101811991,105114483,101421743]},{"topic":"ASSETS","partition":45,"replic >>>as >> >>>":[101461782,102311671,105114483]},{"topic":"B_IMPRESSION","partition":4 >>>7, >> >>>"replicas":[101862816,101671664,101811991]},{"topic":"B_IMPRESSION","par >>>ti >> >>>tion":46,"replicas":[101812541,101461782,101671664]},{"topic":"RTB","par >>>ti >> >>>tion":29,"replicas":[101461782,101862816,102311671]},{"topic":"RTB","par >>>ti >> >>>tion":26,"replicas":[105114483,101461782,101671664]},{"topic":"B_IMPRESS >>>IO >> >>>N","partition":21,"replicas":[102311671,101461702,101461782]},{"topic":" >>>B_ >> >>>IMPRESSION","partition":45,"replicas":[101811991,101461702,101461782]},{ >>>"t >> >>>opic":"B_IMPRESSION","partition":5,"replicas":[101421743,101461702,10146 >>>17 >> >>>82]},{"topic":"RTB","partition":45,"replicas":[101421743,101862816,10231 >>>16 >> >>>71]},{"topic":"B_IMPRESSION","partition":15,"replicas":[101461702,101671 >>>66 >> >>>4,101811991]},{"topic":"D_STATISTICS","partition":7,"replicas":[10231167 >>>1, >> >>>101812541,101862816]},{"topic":"D_STATISTICS","partition":17,"replicas": >>>[1 >> >>>05114483,102311671,101421743]},{"topic":"SLOTS","partition":6,"replicas" >>>:[ >> >>>101811991,101461702,101461782]},{"topic":"SLOTS","partition":24,"replica >>>s" >> >>>:[101811991,101671664,101812541]},{"topic":"SLOTS","partition":9,"replic >>>as >> >>>":[102311671,101812541,101862816]},{"topic":"D_STATISTICS","partition":2 >>>5, >> >>>"replicas":[102311671,105114483,101421743]},{"topic":"AD_EVENTS","partit >>>io >> >>>n":44,"replicas":[105114483,101461782,101671664]},{"topic":"AD_EVENTS"," >>>pa >> >>>rtition":37,"replicas":[101461702,101811991,101812541]},{"topic":"ASSETS >>>", >> >>>"partition":16,"replicas":[101421743,101461702,101461782]},{"topic":"SLO >>>TS >> >>>","partition":44,"replicas":[101862816,105114483,101421743]},{"topic":"B >>>_I >> >>>MPRESSION","partition":16,"replicas":[101461782,101811991,101812541]},{" >>>to >> >>>pic":"ASSETS","partition":10,"replicas":[101671664,101811991,101812541]} >>>,{ >> >>>"topic":"B_IMPRESSION","partition":11,"replicas":[101862816,105114483,10 >>>14 >> >>>21743]},{"topic":"AD_EVENTS","partition":20,"replicas":[101461782,101671 >>>66 >> >>>4,101811991]},{"topic":"ASSETS","partition":1,"replicas":[101671664,1014 >>>61 >> >>>782,101811991]},{"topic":"SLOTS","partition":45,"replicas":[102311671,10 >>>14 >> >>>61702,101461782]},{"topic":"SLOTS","partition":38,"replicas":[101421743, >>>10 >> >>>1461782,101671664]},{"topic":"AD_EVENTS","partition":29,"replicas":[1014 >>>61 >> >>>782,101811991,101812541]},{"topic":"RTB","partition":24,"replicas":[1018 >>>62 >> >>>816,101421743,101461702]},{"topic":"RTB","partition":40,"replicas":[1018 >>>11 >> >>>991,101421743,101461702]},{"topic":"B_IMPRESSION","partition":30,"replic >>>as >> >>>":[102311671,101461782,101671664]},{"topic":"D_STATISTICS","partition":1 >>>8, >> >>>"replicas":[101421743,101461702,101461782]},{"topic":"D_STATISTICS","par >>>ti >> >>>tion":20,"replicas":[101461782,101671664,101811991]},{"topic":"ASSETS"," >>>pa >> >>>rtition":13,"replicas":[101862816,102311671,105114483]},{"topic":"B_IMPR >>>ES >> >>>SION","partition":42,"replicas":[101461702,101862816,102311671]},{"topic >>>": >> >>>"AD_EVENTS","partition":23,"replicas":[101812541,101862816,102311671]},{ >>>"t >> >>>opic":"RTB","partition":34,"replicas":[102311671,101461782,101671664]},{ >>>"t >> >>>opic":"D_STATISTICS","partition":16,"replicas":[102311671,101862816,1051 >>>14 >> >>>483]},{"topic":"SLOTS","partition":23,"replicas":[101671664,101461782,10 >>>18 >> >>>11991]},{"topic":"RTB","partition":0,"replicas":[101421743,101461702,101 >>>46 >> >>>1782]},{"topic":"AD_EVENTS","partition":19,"replicas":[101461702,1014617 >>>82 >> >>>,101671664]},{"topic":"ASSETS","partition":37,"replicas":[101671664,1023 >>>11 >> >>>671,105114483]},{"topic":"SLOTS","partition":39,"replicas":[101461702,10 >>>16 >> >>>71664,101811991]},{"topic":"AD_EVENTS","partition":9,"replicas":[1014217 >>>43 >> >>>,105114483,101461702]},{"topic":"D_STATISTICS","partition":43,"replicas" >>>:[ >> >>>102311671,101461702,101461782]},{"topic":"RTB","partition":37,"replicas" >>>:[ >> >>>101461702,101862816,102311671]},{"topic":"ASSETS","partition":15,"replic >>>as >> >>>":[105114483,101421743,101461702]},{"topic":"B_IMPRESSION","partition":7 >>>," >> >>>replicas":[101461782,101671664,101811991]},{"topic":"ASSETS","partition" >>>:2 >> >>>7,"replicas":[101461782,101812541,101862816]},{"topic":"SLOTS","partitio >>>n" >> >>>:40,"replicas":[101461782,101811991,101812541]},{"topic":"AD_EVENTS","pa >>>rt >> >>>ition":0,"replicas":[101421743,102311671,105114483]},{"topic":"D_STATIST >>>IC >> >>>S","partition":12,"replicas":[101671664,101461782,101811991]},{"topic":" >>>D_ >> >>>STATISTICS","partition":45,"replicas":[101421743,101811991,101812541]},{ >>>"t >> >>>opic":"B_IMPRESSION","partition":41,"replicas":[101421743,101812541,1018 >>>62 >> >>>816]},{"topic":"SLOTS","partition":36,"replicas":[102311671,101421743,10 >>>14 >> >>>61702]},{"topic":"AD_EVENTS","partition":22,"replicas":[101811991,101812 >>>54 >> >>>1,101862816]},{"topic":"RTB","partition":1,"replicas":[101461702,1014617 >>>82 >> >>>,101671664]},{"topic":"B_IMPRESSION","partition":43,"replicas":[10146178 >>>2, >> >>>102311671,105114483]},{"topic":"D_STATISTICS","partition":9,"replicas":[ >>>10 >> >>>1421743,105114483,101461702]},{"topic":"D_STATISTICS","partition":11,"re >>>pl >> >>>icas":[101461782,101461702,101671664]},{"topic":"D_STATISTICS","partitio >>>n" >> >>>:32,"replicas":[101812541,102311671,105114483]},{"topic":"D_STATISTICS", >>>"p >> >>>artition":34,"replicas":[102311671,101421743,101461702]},{"topic":"RTB", >>>"p >> >>>artition":20,"replicas":[101461782,101812541,101862816]},{"topic":"SLOTS >>>", >> >>>"partition":7,"replicas":[101812541,101461782,101671664]},{"topic":"AD_E >>>VE >> >>>NTS","partition":47,"replicas":[101461782,101862816,102311671]},{"topic" >>>:" >> >>>D_STATISTICS","partition":14,"replicas":[101812541,101811991,101862816]} >>>,{ >> >>>"topic":"D_STATISTICS","partition":35,"replicas":[105114483,101461702,10 >>>14 >> >>>61782]},{"topic":"AD_EVENTS","partition":35,"replicas":[105114483,101461 >>>70 >> >>>2,101461782]},{"topic":"ASSETS","partition":21,"replicas":[101812541,102 >>>31 >> >>>1671,105114483]},{"topic":"D_STATISTICS","partition":30,"replicas":[1016 >>>71 >> >>>664,101812541,101862816]},{"topic":"AD_EVENTS","partition":31,"replicas" >>>:[ >> >>>101811991,101862816,102311671]},{"topic":"B_IMPRESSION","partition":39," >>>re >> >>>plicas":[102311671,101671664,101811991]},{"topic":"D_STATISTICS","partit >>>io >> >>>n":13,"replicas":[101811991,101671664,101812541]},{"topic":"D_STATISTICS >>>", >> >>>"partition":40,"replicas":[101811991,102311671,105114483]},{"topic":"SLO >>>TS >> >>>","partition":14,"replicas":[101671664,101461702,101461782]},{"topic":"A >>>SS >> >>>ETS","partition":40,"replicas":[101862816,101461702,101461782]},{"topic" >>>:" >> >>>SLOTS","partition":5,"replicas":[101671664,101421743,101461702]},{"topic >>>": >> >>>"AD_EVENTS","partition":27,"replicas":[101421743,101461782,101671664]},{ >>>"t >> >>>opic":"B_IMPRESSION","partition":3,"replicas":[102311671,105114483,10142 >>>17 >> >>>43]},{"topic":"AD_EVENTS","partition":42,"replicas":[101862816,101421743 >>>,1 >> >01461702]}]} >> > >> > >> > >> >On Tue, Jul 8, 2014 at 10:59 AM, Florian Dambrine <flor...@gumgum.com> >> >wrote: >> > >> >> Here is the entire logic to rebalance the cluster which is done by >>this >> >> groovy script ( >> >> >> >> >> >>https://github.com/Lowess/Kafka/blob/master/KafkaPartitionRebalancer.groo >> >>vy >> >> ) >> >> >> >> #1: Request the zookeeper and get the broker id list >> >> #2: Request zookeeper and get the list of topic >> >> #3: Generate the topic-to-move.json which looks like: >> >> >> >> { >> >> "version": 1, >> >> "topics": [ >> >> { >> >> "topic": "SLOTS" >> >> }, >> >> { >> >> "topic": "ASSETS" >> >> }, >> >> { >> >> "topic": "AD_EVENTS" >> >> }, >> >> { >> >> "topic": "B_IMPRESSION" >> >> }, >> >> { >> >> "topic": "B_STATISTICS" >> >> }, >> >> { >> >> "topic": "PAGES" >> >> }, >> >> { >> >> "topic": "RTB" >> >> }, >> >> { >> >> "topic": "D_STATISTICS" >> >> }, >> >> { >> >> "topic": "D_REPORTING" >> >> } >> >> ] >> >> } >> >> >> >> #4: Upload this file on the kafka node (/tmp/topics-to-move.json) and >> >>run the following command: >> >> bin/kafka-reassign-partitions.sh --zookeeper ZK_IP:2181 >> >>--topics-to-move-json-file /tmp/topics-to-move.json --generate >> >>--broker-list "ALL_BROKERS_THAT_ARE_RETURNED_BY_ZOOKEEPER_ON_STEP_#1" >> >> >> >> #5: Parse the json returned by the previous step and slice it into >> >>smaller json (the number of partitions contained in a Json is limited >>by >> >>the groovy script (10 partitions in this example))that look like: >> >> >> >> { >> >> "version": 1, >> >> "partitions": [ >> >> { >> >> "topic": "D_REPORTING", >> >> "partition": 2, >> >> "replicas": [ >> >> 102311671, >> >> 10517222, >> >> 102311679 >> >> ] >> >> }, >> >> { >> >> "topic": "AD_EVENTS", >> >> "partition": 48, >> >> "replicas": [ >> >> 102311671, >> >> 109715277, >> >> 101531906 >> >> ] >> >> }, >> >> { >> >> "topic": "D_STATISTICS", >> >> "partition": 47, >> >> "replicas": [ >> >> 109715277, >> >> 10517222, >> >> 102311679 >> >> ] >> >> }, >> >> { >> >> "topic": "SLOTS", >> >> "partition": 46, >> >> "replicas": [ >> >> 101131445, >> >> 102336284, >> >> 10517222 >> >> ] >> >> }, >> >> { >> >> "topic": "RTB", >> >> "partition": 48, >> >> "replicas": [ >> >> 101021441, >> >> 102311671, >> >> 102336284 >> >> ] >> >> }, >> >> { >> >> "topic": "PAGES", >> >> "partition": 14, >> >> "replicas": [ >> >> 102311679, >> >> 102311671, >> >> 102336284 >> >> ] >> >> }, >> >> { >> >> "topic": "ASSETS", >> >> "partition": 35, >> >> "replicas": [ >> >> 10517222, >> >> 101131445, >> >> 102311679 >> >> ] >> >> }, >> >> { >> >> "topic": "B_IMPRESSION", >> >> "partition": 34, >> >> "replicas": [ >> >> 101131445, >> >> 102311672, >> >> 102311671 >> >> ] >> >> }, >> >> { >> >> "topic": "B_STATISTICS", >> >> "partition": 19, >> >> "replicas": [ >> >> 109715277, >> >> 101531906, >> >> 102311672 >> >> ] >> >> }, >> >> { >> >> "topic": "AD_EVENTS", >> >> "partition": 18, >> >> "replicas": [ >> >> 109715277, >> >> 102311671, >> >> 102336284 >> >> ] >> >> } >> >> ] >> >> } >> >> >> >> #6: Upload the previous Json on the kafka node >> >> (/tmp/expand-cluster-reassignment.json) and run the following >>command: >> >> bin/kafka-reassign-partitions.sh --zookeeper ZK_IP:2181 >> >>--reassignment-json-file >> >> /tmp/expand-cluster-reassignment.json --execute --broker-list >> >> "ALL_BROKERS_THAT_ARE_RETURNED_BY_ZOOKEEPER_ON_STEP_#1" >> >> >> >> #7: Loop on the verification step while the json returned by the >> >>following >> >> command contains failed partitions: >> >> bin/kafka-reassign-partitions.sh --zookeeper ZK_IP:2181 >> >>--reassignment-json-file >> >> /tmp/expand-cluster-reassignment.json --verify --broker-list >> >> "ALL_BROKERS_THAT_ARE_RETURNED_BY_ZOOKEEPER_ON_STEP_#1" >> >> >> >> #8 Execute a new json part file similar as step #5 util all of them >>have >> >> ran. >> >> >> >> >> >> Hope that will help you guys. >> >> >> >> >> >> On Tue, Jul 8, 2014 at 10:31 AM, Clark Haskins < >> >> chask...@linkedin.com.invalid> wrote: >> >> >> >>> Can you copy/paste the json you are passing to the reassignment >>tool? >> >>>Plus >> >>> the commands. Also do a describe on your topics. >> >>> >> >>> -Clark >> >>> >> >>> Clark Elliott Haskins III >> >>> LinkedIn DDS Site Reliability Engineer >> >>> Kafka, Zookeeper, Samza SRE >> >>> Mobile: 505.385.1484 >> >>> BlueJeans: https://www.bluejeans.com/chaskins >> >>> >> >>> >> >>> chask...@linkedin.com >> >>> https://www.linkedin.com/in/clarkhaskins >> >>> There is no place like 127.0.0.1 >> >>> >> >>> >> >>> >> >>> >> >>> On 7/8/14, 10:26 AM, "Florian Dambrine" <flor...@gumgum.com> wrote: >> >>> >> >>> >I let the tool running for an entire weekend on the test cluster >>and >> >>>on >> >>> >Monday it was still saying "failed"... >> >>> > >> >>> >I have 500 Go per Kafka node and it is a 8 nodes cluster. >> >>> > >> >>> >I am also wondering if I am using the tool correctly. Currently I >>am >> >>> >running the tool to rebalance everything across the entire cluster. >> >>>As I >> >>> >have 3 replicas the tool requires at least 3 brokers. >> >>> > >> >>> >Should I add 3 new Kafka nodes and rebalance some topics to these >>new >> >>> >nodes >> >>> >only? I am afraid to unbalance the cluster with this option. >> >>> > >> >>> >Any suggestions? >> >>> > >> >>> >Thanks for your help. >> >>> > >> >>> > >> >>> >On Mon, Jul 7, 2014 at 9:29 PM, Jun Rao <jun...@gmail.com> wrote: >> >>> > >> >>> >> The failure could mean that the reassignment is still in >>progress. >> >>>If >> >>> >>you >> >>> >> have lots of data, it may take some time to move the data to new >> >>> >>brokers. >> >>> >> You could observe the max lag in each broker to see how far >>behind >> >>>new >> >>> >> replicas are (see >> >>> >>http://kafka.apache.org/documentation.html#monitoring). >> >>> >> >> >>> >> Thanks, >> >>> >> >> >>> >> Jun >> >>> >> >> >>> >> >> >>> >> On Mon, Jul 7, 2014 at 4:42 PM, Florian Dambrine >> >>><flor...@gumgum.com> >> >>> >> wrote: >> >>> >> >> >>> >> > When I run the tool with the --verify option it says failed for >> >>>the >> >>> >>some >> >>> >> > partitions. >> >>> >> > >> >>> >> > The problem is I do not know if it is a zookeeper issue or if >>the >> >>> tool >> >>> >> > really failed. >> >>> >> > >> >>> >> > I faced one time the zookeeper issue ( >> >>> >> > https://issues.apache.org/jira/browse/KAFKA-1382) and by >>killing >> >>>the >> >>> >> > responsible Kafka the partition switched from failed to >>completed >> >>> >> > successfully. >> >>> >> > >> >>> >> > What should I do when the Kafka tool says that it failed to >>move >> >>>the >> >>> >> > partition? >> >>> >> > >> >>> >> > >> >>> >> > >> >>> >> > >> >>> >> > On Mon, Jul 7, 2014 at 4:33 PM, Clark Haskins >> >>> >> > <chask...@linkedin.com.invalid >> >>> >> > > wrote: >> >>> >> > >> >>> >> > > How does it get stuck? >> >>> >> > > >> >>> >> > > -Clark >> >>> >> > > >> >>> >> > > Clark Elliott Haskins III >> >>> >> > > LinkedIn DDS Site Reliability Engineer >> >>> >> > > Kafka, Zookeeper, Samza SRE >> >>> >> > > Mobile: 505.385.1484 >> >>> >> > > BlueJeans: https://www.bluejeans.com/chaskins >> >>> >> > > >> >>> >> > > >> >>> >> > > chask...@linkedin.com >> >>> >> > > https://www.linkedin.com/in/clarkhaskins >> >>> >> > > There is no place like 127.0.0.1 >> >>> >> > > >> >>> >> > > >> >>> >> > > >> >>> >> > > >> >>> >> > > On 7/7/14, 3:49 PM, "Florian Dambrine" <flor...@gumgum.com> >> >>>wrote: >> >>> >> > > >> >>> >> > > >Hi, >> >>> >> > > > >> >>> >> > > >I am trying to add new brokers to an existing 8 nodes Kafka >> >>> >>cluster. >> >>> >> We >> >>> >> > > >have around 10 topics and the number of partition is set to >> >>>50. In >> >>> >> order >> >>> >> > > >to >> >>> >> > > >test the reassgin-partitions scripts, I tried on a sandbox >> >>>cluster >> >>> >>the >> >>> >> > > >following steps. >> >>> >> > > > >> >>> >> > > >I developed a script which is able to parse the reassignment >> >>> >>partition >> >>> >> > > >plan >> >>> >> > > >given by the Kafka tool in smaller pieces (reassigning >>maximum >> >>>10 >> >>> >> > > >partitions at a time). >> >>> >> > > > >> >>> >> > > >Unfortunately I faced some issues with the tool that >>sometimes >> >>>get >> >>> >> stuck >> >>> >> > > >on >> >>> >> > > >one partition. In this case I have to kill and restart the >> >>>three >> >>> >> Kafkas >> >>> >> > on >> >>> >> > > >which the partition has been relocated to unlock the process >> >>>(One >> >>> >> kafka >> >>> >> > at >> >>> >> > > >a time). >> >>> >> > > > >> >>> >> > > >Moreover, I have also faced these two issues that are >>already >> >>>on >> >>> >>Jira: >> >>> >> > > > >> >>> >> > > >https://issues.apache.org/jira/browse/KAFKA-1382 >> >>> >> > > >https://issues.apache.org/jira/browse/KAFKA-1479 >> >>> >> > > > >> >>> >> > > >We really need to add new nodes to our Kafka cluster, does >> >>>anybody >> >>> >> have >> >>> >> > > >already rebalance a Kafka 0.8.1.1? What could you advise me? >> >>> >> > > > >> >>> >> > > >Thanks, and feel free to ask me if you need more details. >> >>> >> > > > >> >>> >> > > > >> >>> >> > > > >> >>> >> > > >-- >> >>> >> > > >*Florian Dambrine* | Intern, Big Data >> >>> >> > > >*GumGum* <http://www.gumgum.com/> | *Ads that stick* >> >>> >> > > >209-797-3994 | flor...@gumgum.com >> >>> >> > > >> >>> >> > > >> >>> >> > >> >>> >> > >> >>> >> > -- >> >>> >> > *Florian Dambrine* | Intern, Big Data >> >>> >> > *GumGum* <http://www.gumgum.com/> | *Ads that stick* >> >>> >> > 209-797-3994 | flor...@gumgum.com >> >>> >> > >> >>> >> >> >>> > >> >>> > >> >>> > >> >>> >-- >> >>> >*Florian Dambrine* | Intern, Big Data >> >>> >*GumGum* <http://www.gumgum.com/> | *Ads that stick* >> >>> >209-797-3994 | flor...@gumgum.com >> >>> >> >>> >> >> >> >> >> >> -- >> >> *Florian Dambrine* | Intern, Big Data >> >> *GumGum* <http://www.gumgum.com/> | *Ads that stick* >> >> 209-797-3994 | flor...@gumgum.com >> >> >> > >> > >> > >> >-- >> >*Florian Dambrine* | Intern, Big Data >> >*GumGum* <http://www.gumgum.com/> | *Ads that stick* >> >209-797-3994 | flor...@gumgum.com >> >> > > >-- >*Florian Dambrine* | Intern, Big Data >*GumGum* <http://www.gumgum.com/> | *Ads that stick* >209-797-3994 | flor...@gumgum.com
