Perfect. Thank you very much! Chris
On Sun, May 17, 2015 at 10:41 PM, Clark Haskins <cl...@kafka.guru> wrote: > Yeah you can start using the reassignment tool again. > > It's likely some issue with the controller that put you in this state. > Without gathering the logs it will be hard to tell. > > -Clark > > Sent from my iPhone > > > On May 17, 2015, at 3:42 PM, Chris Neal <cwn...@gmail.com> wrote: > > > > Done, and things look clean in Zookeeper now. I'm guessing that it is > safe > > for me to try to increase the replication count on this topic again? :) > > > > Any idea what might have happened to put me in this state? > > I had successfully increased probably 10 or so topics, but then I started > > getting this message and nothing worked after that. > > > > Again, thank you so much for your time and knowledge. Very much > > appreciated. > > Chris > > > >> On Sun, May 17, 2015 at 2:20 PM, Clark Haskins <cl...@kafka.guru> > wrote: > >> > >> No problem. > >> > >> Delete the reassign_partitions znode. Then do the controller again. > >> > >> The controller znode gets instantly re-created but with a different > broker > >> as the controller. > >> > >> If you do a get /controller you can see the values change > >> > >> -Clark > >> > >> Sent from my iPhone > >> > >>> On May 17, 2015, at 11:48 AM, Chris Neal <cwn...@gmail.com> wrote: > >>> > >>> Gah. Sorry for the spamming replies. > >>> > >>> The reassign_partitions znode is also still there, and still > empty....For > >>> what that's worth. :) > >>> Back to thinking that deleting the reassign_partitions znode is the > right > >>> way to go. > >>> > >>> Chris > >>> > >>>> On Sun, May 17, 2015 at 1:46 PM, Chris Neal <cwn...@gmail.com> wrote: > >>>> > >>>> Ok, I might have lied. Timezone got me. I do see this in the Kafka > >> logs, > >>>> so maybe it did work? > >>>> > >>>> [2015-05-17 14:36:57,924] INFO 0 successfully elected as leader > >>>> (kafka.server.ZookeeperLeaderElector) > >>>> [2015-05-17 14:36:57,924] INFO 0 successfully elected as leader > >>>> (kafka.server.ZookeeperLeaderElector) > >>>> [2015-05-17 14:36:58,975] INFO New leader is 0 > >>>> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) > >>>> [2015-05-17 14:36:58,975] INFO New leader is 0 > >>>> (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) > >>>> > >>>> Chris > >>>> > >>>>> On Sun, May 17, 2015 at 1:43 PM, Chris Neal <cwn...@gmail.com> > wrote: > >>>>> > >>>>> Thank you again. :) > >>>>> > >>>>> I ran the delete, but I'm not sure if it did anything: > >>>>> > >>>>> root> > >> > ./opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/zookeeper/bin/zkCli.sh > >>>>> -server myhost.mydomain.com:2181 delete /controller > >>>>> Connecting to myhost.mydomain.com:2181 > >>>>> > >>>>> WATCHER:: > >>>>> > >>>>> WatchedEvent state:SyncConnected type:None path:null > >>>>> root> > >>>>> > >>>>> I followed that up immediately by an "ls /", and the controller znode > >> was > >>>>> there. Either still there, or had already been re-created. There > was > >>>>> nothing in the Kafka logs indicating that anything had happened > >> either, so > >>>>> I'm thinking maybe I need to try and delete the znode partition > >> manually? > >>>>> > >>>>> Or, should I have seen the controller znode disappear for a time? > >>>>> > >>>>> Very much appreciate your help! > >>>>> Chris > >>>>> > >>>>>> On Sun, May 17, 2015 at 1:33 PM, Clark Haskins <cl...@kafka.guru> > >> wrote: > >>>>>> > >>>>>> Yeah it's empty. So I would try and force a new controller election > to > >>>>>> see if that cleans up the znode. > >>>>>> > >>>>>> Delete /controller > >>>>>> > >>>>>> If that doesn't work you will have to delete the reassign_partitions > >>>>>> znode manually > >>>>>> > >>>>>> -Clark > >>>>>> > >>>>>> Sent from my iPhone > >>>>>> > >>>>>>> On May 17, 2015, at 11:16 AM, Chris Neal <cwn...@gmail.com> wrote: > >>>>>>> > >>>>>>> Sorry :) > >>>>>>> Here you go: > >>>>>>> > >>>>>>> root> > >> > ./opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/zookeeper/bin/zkCli.sh > >>>>>>> -server myhost.mydomain.com <http://myhost.mydomain.com:2181/ > >:2181 > >>>>>> get > >>>>>>> /admin/reassign_partitions > >>>>>>> Connecting to myhost.mydomain.com < > http://myhost.mydomain.com:2181/ > >>>>>>> :2181 > >>>>>>> <snip> > >>>>>>> WATCHER:: > >>>>>>> > >>>>>>> WatchedEvent state:SyncConnected type:None path:null > >>>>>>> {"version":1,"partitions":[]} > >>>>>>> cZxid = 0x6001afde9 > >>>>>>> ctime = Wed Jan 21 18:37:40 UTC 2015 > >>>>>>> mZxid = 0x6001afde9 > >>>>>>> mtime = Wed Jan 21 18:37:40 UTC 2015 > >>>>>>> pZxid = 0x6001afde9 > >>>>>>> cversion = 0 > >>>>>>> dataVersion = 0 > >>>>>>> aclVersion = 0 > >>>>>>> ephemeralOwner = 0x0 > >>>>>>> dataLength = 29 > >>>>>>> numChildren = 0 > >>>>>>> > >>>>>>> Thank you! > >>>>>>> Chris > >>>>>>> > >>>>>>>> On Sun, May 17, 2015 at 12:58 PM, Clark Haskins <cl...@kafka.guru > > > >>>>>> wrote: > >>>>>>>> > >>>>>>>> Do a get /admin/reassign_partitions > >>>>>>>> > >>>>>>>> -Clark > >>>>>>>> > >>>>>>>> Sent from my iPhone > >>>>>>>> > >>>>>>>>> On May 17, 2015, at 10:20 AM, Chris Neal <cwn...@gmail.com> > wrote: > >>>>>>>>> > >>>>>>>>> Sure thing :) > >>>>>>>>> Hopefully I did this right. Somewhat of a Zookeeper noob. > >>>>>>>>> > >>>>>>>>> root> > >> > ./opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/zookeeper/bin/zkCli.sh > >>>>>>>>> -server myhost.mydomain.com <http://myhost.mydomain.com:2181/ > >>> :2181 > >>>>>> ls > >>>>>>>>> /admin/reassign_partitions > >>>>>>>>> Connecting to myhost.mydomain.com < > >> http://myhost.mydomain.com:2181/ > >>>>>>>>> :2181 > >>>>>>>>> <snip> > >>>>>>>>> WATCHER:: > >>>>>>>>> > >>>>>>>>> WatchedEvent state:SyncConnected type:None path:null > >>>>>>>>> [] > >>>>>>>>> > >>>>>>>>> Hope that is helpful :) > >>>>>>>>> If this is not what you were asking for, please just let me know. > >>>>>>>>> Thank you! > >>>>>>>>> Chris > >>>>>>>>> > >>>>>>>>>> On Sun, May 17, 2015 at 12:17 PM, Clark Haskins > <cl...@kafka.guru > >>> > >>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> The reassign_partitions znode is the important one. Please paste > >> the > >>>>>>>>>> contents of it. That node should only exist while there is a > >>>>>>>> reassignment > >>>>>>>>>> in progress. > >>>>>>>>>> > >>>>>>>>>> You can probably fix this up by forcing a new controller to come > >>>>>> online > >>>>>>>> by > >>>>>>>>>> deleting /controller > >>>>>>>>>> > >>>>>>>>>> -Clark > >>>>>>>>>> > >>>>>>>>>> Sent from my iPhone > >>>>>>>>>> > >>>>>>>>>>> On May 17, 2015, at 10:14 AM, Chris Neal <cwn...@gmail.com> > >> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Hi Clark, > >>>>>>>>>>> > >>>>>>>>>>> Thank you for your reply! I do see that znode under /admin: > >>>>>>>>>>> > >>>>>>>>>>> root> > >> > ./opt/cloudera/parcels/CDH-5.1.3-1.cdh5.1.3.p0.12/lib/zookeeper/bin/zkCli.sh > >>>>>>>>>>> -server myhost.mydomain.com:2181 ls /admin > >>>>>>>>>>> Connecting to myhost.mydomain.com:2181 > >>>>>>>>>>> <snip> > >>>>>>>>>>> WATCHER:: > >>>>>>>>>>> > >>>>>>>>>>> WatchedEvent state:SyncConnected type:None path:null > >>>>>>>>>>> [reassign_partitions, delete_topics] > >>>>>>>>>>> > >>>>>>>>>>> I'm not sure what this tells me though :) > >>>>>>>>>>> Again, thanks for your time. > >>>>>>>>>>> Chris > >>>>>>>>>>> > >>>>>>>>>>>> On Sun, May 17, 2015 at 12:20 AM, Clark Haskins > >> <cl...@kafka.guru > >>>>>>> > >>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> Does the partition reassignment znode exist under /admin in > >>>>>> zookeeper? > >>>>>>>>>>>> > >>>>>>>>>>>> -Clark > >>>>>>>>>>>> > >>>>>>>>>>>> Sent from my iPhone > >>>>>>>>>>>> > >>>>>>>>>>>>> On May 16, 2015, at 7:16 PM, Chris Neal <cwn...@gmail.com> > >>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Sorry for bumping my own thread. :S Just wanted to get it in > >>>>>> front > >>>>>>>> of > >>>>>>>>>>>> some > >>>>>>>>>>>>> eyes again! > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks for your time and help. > >>>>>>>>>>>>> Chris > >>>>>>>>>>>>> > >>>>>>>>>>>>>> On Thu, May 14, 2015 at 10:24 AM, Chris Neal < > >> cwn...@gmail.com> > >>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi All, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I am running: kafka_2.10-0.8.1.1, and when I run the > >>>>>>>>>>>>>> reassign-partitions.sh script, I get this: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Partitions reassignment failed due to Partition reassignment > >>>>>>>> currently > >>>>>>>>>>>> in > >>>>>>>>>>>>>> progress for Map(). Aborting operation > >>>>>>>>>>>>>> kafka.common.AdminCommandFailedException: Partition > >> reassignment > >>>>>>>>>>>> currently > >>>>>>>>>>>>>> in progress for Map(). Aborting operation > >>>>>>>>>>>>>> at > >> > kafka.admin.ReassignPartitionsCommand.reassignPartitions(ReassignPartitionsCommand.scala:204) > >>>>>>>>>>>>>> at > >> > kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:124) > >>>>>>>>>>>>>> at > >> > kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:49) > >>>>>>>>>>>>>> at > >> > kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I'm running it as: > >>>>>>>>>>>>>> /opt/kafka/kafka/bin/kafka-reassign-partitions.sh > --zookeeper > >>>>>>>>>> $ZOOKEEPER > >>>>>>>>>>>>>> --reassignment-json-file ./$1.json --execute > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> and my json file looks like: > >>>>>>>>>>>>>> {"version":1, > >>>>>>>>>>>>>> "partitions":[ > >>>>>>>>>>>>>> {"topic":"perf_mytopic_raw", > "partition":0,"replicas":[0,1]}, > >>>>>>>>>>>>>> > {"topic":"perf_mytopic_raw","partition":1,"replicas":[0,1]}]} > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The empty Map() looks suspicious, and I found another post > >> from > >>>>>>>> about > >>>>>>>>>> a > >>>>>>>>>>>>>> year ago where it was thought that this might be a bug? I > >>>>>> looked > >>>>>>>>>>>> through > >>>>>>>>>>>>>> JIRA and didn't see anything that looked like this issue was > >>>>>> created > >>>>>>>>>> or > >>>>>>>>>>>>>> fixed, so I wanted to ask again about it. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Any ideas why this might be failing? There is definitely > not > >>>>>>>> another > >>>>>>>>>>>>>> reassignment in progress. :) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks very much for your time. > >>>>>>>>>>>>>> Chris > >> >
