Same status after an hour. Should I delete them and re-add them or ??
On 5/29/13 7:30 AM, "Old, Curtis" <[email protected]> wrote: > > >Well one lab-cloud-9 is in an "Alert" State the other, lab-cloud-10 is >"Disconnected", and after a refresh on the UI lab-cloud-9 is back to >"Connecting" > > >Suggestions? > > > >On 5/29/13 7:17 AM, "Old, Curtis" <[email protected]> wrote: > >>Interesting the UUID's in the database where different then the Xenserver >>UUID's (there where 2 Xenservers hosts that had inode problems) But then >>again the other servers all had different UUID's as well <shrug>. >> >>Mysql DB: >> >> id: 21 >> name: lab-cloud-10.neustarlab.biz >> uuid: 5c65a56a-5a71-44e8-b4ae-89bf81ef4d24 >> >>And on the Host: >> >> >>uuid=dde1a6d7-8708-451f-a4a8-a9c5be79f57e >> >>It has been 20 to 25 min and one lab-cloud-9 its state is "Connecting" >>lab-cloud-10 is in "Disconnected" and the other 3 hosts (lab-cloud-6, 7, >>8) are "Up" >> >>Wait a bit longer? >> >> >> >>On 5/28/13 6:37 PM, "Kirk Kosinski" <[email protected]> wrote: >> >>>Before hacking the db, I suggest trying one more thing. >>> >>>1. In the CS UI, unmanage the cluster with the affected host. >>>2. From the CLI of a host in the XS pool, clear the host tags of the >>>affected host. >>>xe host-param-clear param-name=tags uuid=<UUID of affected host> >>>3. In the CS UI, manage the cluster with the affected host. >>> >>>This usually solves this type of problem (it might take a couple >>>minutes). Clearing host tags won't affect running VMs (if any). >>> >>>Best regards, >>>Kirk >>> >>>On 05/28/2013 03:03 PM, Ahmad Emneina wrote: >>>> first, I'd query to make sure its the right host, judging from the >>>>logs >>>>it >>>> has an id of 20... execute the following >>>> backup command from the cli that can access the db server: >>>> >>>> mysqldump <credentials if necessary> --databases cloud > cloud.sql >>>> >>>> Then start operating... >>>> >>>> select * from host where id=20\G (hit the enter key) >>>> >>>> if this is infact the misbehaving host, execute >>>> >>>> update host set status='Up' where id=20; (again hit enter) >>>> >>>> then try to launch a vm on that host (in the deployVirtualMachine api >>>> command you can/should be able to specify the exact host) >>>> >>>> >>>> On Tue, May 28, 2013 at 2:46 PM, Old, Curtis <[email protected]> >>>>wrote: >>>> >>>>> Oh god where do I find that sucker in the DB? Sorry I'M an admin >>>>>type >>>>> that isn't a mysql guru or Ji >>>>> >>>>> Curtis Old >>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer >>>>> 46000 Center Oak Plaza Sterling, VA 20166 >>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / >>>>>[email protected] >>>>> / www.neustar.biz <http://www.neustar.biz/> >>>>> ________________________________________ >>>>> >>>>> >>>>> The information contained in this e-mail message is intended only for >>>>>the >>>>> use of the recipient(s) named above and may contain confidential >>>>>and/or >>>>> privileged information. If you are not the intended recipient you >>>>>have >>>>> received this e-mail message in error and any review, dissemination, >>>>> distribution, or copying of this message is strictly prohibited. If >>>>>you >>>>> have received this communication in error, please notify us >>>>>immediately >>>>> and delete the original message. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 5/28/13 5:43 PM, "Ahmad Emneina" <[email protected]> wrote: >>>>> >>>>>> oh this looks like a bug. A potential workaround for it is marking >>>>>>the >>>>>> host >>>>>> as Up in the cloudstack db. I'd make a backup first then modify the >>>>>>db >>>>>> entry. >>>>>> >>>>>> >>>>>> On Tue, May 28, 2013 at 2:36 PM, Old, Curtis >>>>>><[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Tailed last 3000 lines right after trying a forced reconnect: >>>>>>> http://paste.cloudstack.org/hTOm/ >>>>>>> >>>>>>> Listed as Management log N*, >>>>>>> >>>>>>> Author Curtis Old >>>>>>> >>>>>>> >>>>>>> Can we see the latest logs? paste.cloudstack.org >>>>>>> >>>>>>> >>>>>>> On Tue, May 28, 2013 at 2:15 PM, Old, Curtis >>>>>>> <[email protected]>wrote: >>>>>>> >>>>>>>> Tried that already (like 2 times) >>>>>>>> >>>>>>>> Curtis Old >>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer >>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166 >>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / >>>>>>>> [email protected] >>>>>>>> / www.neustar.biz <http://www.neustar.biz/> >>>>>>>> ________________________________________ >>>>>>>> >>>>>>>> >>>>>>>> The information contained in this e-mail message is intended only >>>>>>>>for >>>>>>>> the >>>>>>>> use of the recipient(s) named above and may contain confidential >>>>>>>>and/or >>>>>>>> privileged information. If you are not the intended recipient you >>>>>>>>have >>>>>>>> received this e-mail message in error and any review, >>>>>>>>dissemination, >>>>>>>> distribution, or copying of this message is strictly prohibited. >>>>>>>>If >>>>>>>>you >>>>>>>> have received this communication in error, please notify us >>>>>>>>immediately >>>>>>>> and delete the original message. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 5/28/13 5:12 PM, "Ahmad Emneina" <[email protected]> wrote: >>>>>>>> >>>>>>>>> how about a reboot of the host, then force reconnect. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, May 28, 2013 at 2:05 PM, Old, Curtis >>>>>>>>><[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Bummer no goodŠ... >>>>>>>>>> >>>>>>>>>> Curtis Old >>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer >>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166 >>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / >>>>>>>> [email protected] >>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/> >>>>>>>>>> ________________________________________ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The information contained in this e-mail message is intended >>>>>>>>>>only >>>>>>>> for >>>>>>>>>> the >>>>>>>>>> use of the recipient(s) named above and may contain confidential >>>>>>>> and/or >>>>>>>>>> privileged information. If you are not the intended recipient >>>>>>>>>>you >>>>>>>> have >>>>>>>>>> received this e-mail message in error and any review, >>>>>>>>>>dissemination, >>>>>>>>>> distribution, or copying of this message is strictly prohibited. >>>>>>>>>>If >>>>>>>> you >>>>>>>>>> have received this communication in error, please notify us >>>>>>>> immediately >>>>>>>>>> and delete the original message. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 5/28/13 5:01 PM, "Ahmad Emneina" <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> dont do that :D do a toolstack restart on the xenserver host, >>>>>>>>>>>and a >>>>>>>>>> force >>>>>>>>>>> reconnect from cloudstack. see if that makes them play nice. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, May 28, 2013 at 1:58 PM, Old, Curtis >>>>>>>> <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Well the hypervisior looks fine from a Xenserver prospective >>>>>>>> but >>>>>>>>>>>> CloudStack won't "reconnect" it. I thought about deleting it >>>>>>>>>>>>in >>>>>>>> CS >>>>>>>>>> and >>>>>>>>>>>> re-adding it, but that bugs me lol >>>>>>>>>>>> >>>>>>>>>>>> *Curtis Old** >>>>>>>>>>>> **Neustar, Inc. / Neustarlabs / **Senior Research Engineer* >>>>>>>>>>>> >>>>>>>>>>>> *46000 Center Oak Plaza Sterling, VA 20166** >>>>>>>>>>>> **Office:** **+1.571.434.5384** **Mobile: **+1.540.532.2230** >>>>>>>> **/** >>>>>>>>>> ** >>>>>>>>>>>> [email protected]** / **www.neustar.biz** * >>>>>>>>>>>> * >>>>>>>>>>>> ------------------------------ >>>>>>>>>>>> * >>>>>>>>>>>> >>>>>>>>>>>> *The information contained in this e-mail message is intended >>>>>>>> only >>>>>>>>>> for >>>>>>>>>>>> the use of the recipient(s) named above and may contain >>>>>>>> confidential >>>>>>>>>>>> and/or >>>>>>>>>>>> privileged information. If you are not the intended recipient >>>>>>>>>>>>you >>>>>>>>>> have >>>>>>>>>>>> received this e-mail message in error and any review, >>>>>>>> dissemination, >>>>>>>>>>>> distribution, or copying of this message is strictly >>>>>>>>>>>>prohibited. >>>>>>>> If >>>>>>>>>> you >>>>>>>>>>>> have received this communication in error, please notify us >>>>>>>>>> immediately >>>>>>>>>>>> and >>>>>>>>>>>> delete the original message.* >>>>>>>>>>>> >>>>>>>>>>>> From: Ahmad Emneina <[email protected]> >>>>>>>>>>>> Reply-To: "[email protected]" <[email protected]> >>>>>>>>>>>> Date: Tuesday, May 28, 2013 4:56 PM >>>>>>>>>>>> To: "Old, Curtis" <[email protected]> >>>>>>>>>>>> Subject: Re: Xenserver hypervisior will not reconnect after >>>>>>>> reboot >>>>>>>>>>>> >>>>>>>>>>>> are you good to go again, after cleaning up some space? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, May 28, 2013 at 1:24 PM, Old, Curtis >>>>>>>>>>>> <[email protected]>wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Sorry the log entries are from the xenserver log >>>>>>>>>>>>> >>>>>>>>>>>>> Curtis Old >>>>>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer >>>>>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166 >>>>>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/> >>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> The information contained in this e-mail message is intended >>>>>>>> only >>>>>>>>>> for >>>>>>>>>>>>> the >>>>>>>>>>>>> use of the recipient(s) named above and may contain >>>>>>>>>>>>>confidential >>>>>>>>>> and/or >>>>>>>>>>>>> privileged information. If you are not the intended recipient >>>>>>>> you >>>>>>>>>> have >>>>>>>>>>>>> received this e-mail message in error and any review, >>>>>>>> dissemination, >>>>>>>>>>>>> distribution, or copying of this message is strictly >>>>>>>> prohibited. If >>>>>>>>>> you >>>>>>>>>>>>> have received this communication in error, please notify us >>>>>>>>>> immediately >>>>>>>>>>>>> and delete the original message. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 5/28/13 4:20 PM, "Old, Curtis" <[email protected]> >>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> The in innodes did fill, so after I cleaned them up I >>>>>>>>>>>>>>rebooted >>>>>>>> and >>>>>>>>>>>>> then >>>>>>>>>>>>>> the host was singing and dancing again in XenCenter, and >>>>>>>> primary >>>>>>>>>>>>> storage >>>>>>>>>>>>>> was mounted. What from the management log, if we can narrow >>>>>>>>>>>>>>it >>>>>>>>>> down? >>>>>>>>>>>>>> When I try the forced reconnect only this shows up: >>>>>>>>>>>>>> >>>>>>>>>>>>>> [20130528T20:17:39.118Z|debug|lab-cloud-9|8622 >>>>>>>>>>>>> inet_rpc||http_critical] >>>>>>>>>>>>>> Connection terminated >>>>>>>>>>>>>> [20130528T20:17:51.019Z| info|lab-cloud-9|28 >>>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23045 (cached = false) >>>>>>>> connected >>>>>>>>>> to >>>>>>>>>>>>>> 10.31.105.158:443 >>>>>>>>>>>>>> [20130528T20:17:51.019Z| info|lab-cloud-9|28 >>>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None >>>>>>>>>>>>> s_pid=23063 >>>>>>>>>>>>>> [20130528T20:18:21.089Z| info|lab-cloud-9|28 >>>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23063 (cached = false) >>>>>>>> connected >>>>>>>>>> to >>>>>>>>>>>>>> 10.31.105.158:443 >>>>>>>>>>>>>> [20130528T20:18:21.089Z| info|lab-cloud-9|28 >>>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None >>>>>>>>>>>>> s_pid=23063 >>>>>>>>>>>>>> [20130528T20:18:39.124Z|debug|lab-cloud-9|8623 >>>>>>>>>>>>> inet_rpc||http_critical] >>>>>>>>>>>>>> Connection terminated >>>>>>>>>>>>>> [20130528T20:18:51.159Z| info|lab-cloud-9|28 >>>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23068 (cached = false) >>>>>>>> connected >>>>>>>>>> to >>>>>>>>>>>>>> 10.31.105.158:443 >>>>>>>>>>>>>> [20130528T20:18:51.159Z| info|lab-cloud-9|28 >>>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None >>>>>>>>>>>>> s_pid=23068 >>>>>>>>>>>>>> [20130528T20:19:21.229Z| info|lab-cloud-9|28 >>>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>>> D:4de2ede3bcba|xapi] stunnel pid: 23072 (cached = false) >>>>>>>> connected >>>>>>>>>> to >>>>>>>>>>>>>> 10.31.105.158:443 >>>>>>>>>>>>>> [20130528T20:19:21.229Z| info|lab-cloud-9|28 >>>>>>>> heartbeat|Heartbeat >>>>>>>>>>>>>> D:4de2ede3bcba|xapi] with_recorded_stunnelpid task_opt=None >>>>>>>>>>>>> s_pid=23072 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Curtis Old >>>>>>>>>>>>>> Neustar, Inc. / Neustarlabs / Senior Research Engineer >>>>>>>>>>>>>> 46000 Center Oak Plaza Sterling, VA 20166 >>>>>>>>>>>>>> Office: +1.571.434.5384 Mobile: +1.540.532.2230 / >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> / www.neustar.biz <http://www.neustar.biz/> >>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> The information contained in this e-mail message is intended >>>>>>>> only >>>>>>>>>> for >>>>>>>>>>>>> the >>>>>>>>>>>>>> use of the recipient(s) named above and may contain >>>>>>>> confidential >>>>>>>>>>>>> and/or >>>>>>>>>>>>>> privileged information. If you are not the intended >>>>>>>>>>>>>>recipient >>>>>>>> you >>>>>>>>>> have >>>>>>>>>>>>>> received this e-mail message in error and any review, >>>>>>>>>> dissemination, >>>>>>>>>>>>>> distribution, or copying of this message is strictly >>>>>>>> prohibited. >>>>>>>> If >>>>>>>>>>>>> you >>>>>>>>>>>>>> have received this communication in error, please notify us >>>>>>>>>>>>> immediately >>>>>>>>>>>>>> and delete the original message. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 5/28/13 4:08 PM, "Ahmad Emneina" <[email protected]> >>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> pretty odd... how does the disk space look on the xenserver >>>>>>>> host? >>>>>>>>>>>>> we'd >>>>>>>>>>>>>>> need >>>>>>>>>>>>>>> more from the management log, as well as the xensource.log >>>>>>>>>>>>>>>off >>>>>>>> the >>>>>>>>>>>>>>> hypervisor. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, May 28, 2013 at 9:21 AM, Old, Curtis >>>>>>>>>> <[email protected] >>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Tried the force reconnect from CS 4.0.0 UI and I get >>>>>>>> "Command >>>>>>>>>>>>> Failed >>>>>>>>>>>>>>>> due >>>>>>>>>>>>>>>> to internal server Error" any ideas? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> management-server.log >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2013-05-24 10:33:00,974 DEBUG >>>>>>>> [cloud.async.AsyncJobManagerImpl] >>>>>>>>>>>>>>>> (catalina-exec-7:null) submit async job-230, details: >>>>>>>> AsyncJobVO >>>>>>>>>>>>>>>> {id:230, >>>>>>>>>>>>>>>> userId: 2, accountId: 2, sessionKey: null, instanceType: >>>>>>>> Host, >>>>>>>>>>>>>>>> instanceId: >>>>>>>>>>>>>>>> 20, cmd: com.cloud.api.commands.ReconnectHostCmd, >>>>>>>> cmdOriginator: >>>>>>>>>>>>> null, >>>>>>>>>>>>>>>> cmdInfo: >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>{"response":"json","id":"346bcc09-e56b-4b27-b269-29252bc2a6 >>>>>>>>>>>>>>>>5 >>>>>>>>>>>>>>>>3 >>>>>>>>>>>>>>>>"," >>>>>>>>>>>>>>>> ses >>>>>>>>>>>>>>>> si >>>>>>>>>>>>>>>> onk >>>>>>>>>>>>>>>> e >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>y":"k/+GoOS0g1IOaSr5bKA2BWg1yD4\u003d","ctxUserId":"2","_": >>>>>>>>>>>>>>>>" >>>>>>>>>>>>>>>>1 >>>>>>>>>>>>>>>>369 >>>>>>>>>>>>>>>> 405 >>>>>>>>>>>>>>>> 98 >>>>>>>>>>>>>>>> 077 >>>>>>>>>>>>>>>> 8 >>>>>>>>>>>>>>>> ","ctxAccountId":"2","ctxStartEventId":"1514"}, >>>>>>>>>>>>>>>> cmdVersion: 0, callbackType: 0, callbackAddress: null, >>>>>>>> status: >>>>>>>>>> 0, >>>>>>>>>>>>>>>> processStatus: 0, resultCode: 0, result: null, initMsid: >>>>>>>>>>>>> 112939138816, >>>>>>>>>>>>>>>> completeMsid: null, lastUpdated: null, lastPolled: null, >>>>>>>>>> created: >>>>>>>>>>>>> null} >>>>>>>>>>>>>>>> 2013-05-24 10:33:00,978 DEBUG >>>>>>>> [cloud.async.AsyncJobManagerImpl] >>>>>>>>>>>>>>>> (Job-Executor-16:job-230) Executing >>>>>>>>>>>>>>>> com.cloud.api.commands.ReconnectHostCmd >>>>>>>>>>>>>>>> for job-230 >>>>>>>>>>>>>>>> 2013-05-24 10:33:00,985 INFO >>>>>>>> [agent.manager.AgentManagerImpl] >>>>>>>>>>>>>>>> (Job-Executor-16:job-230) Unable to disconnect host >>>>>>>>>>>>>>>>because >>>>>>>> it >>>>>>>>>> is >>>>>>>>>>>>> not >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>> the correct state: host=20; Status=Disconnected >>>>>>>>>>>>>>>> 2013-05-24 10:33:00,986 WARN >>>>>>>> [api.commands.ReconnectHostCmd] >>>>>>>>>>>>>>>> (Job-Executor-16:job-230) Exception: >>>>>>>>>>>>>>>> com.cloud.api.ServerApiException >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHo >>>>>>>>>>>>>>>>s >>>>>>>>>>>>>>>>t >>>>>>>>>>>>>>>>Cmd >>>>>>>>>>>>>>>> .ja >>>>>>>>>>>>>>>> va >>>>>>>>>>>>>>>> :10 >>>>>>>>>>>>>>>> 8 >>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>> at >>>>>>>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerIm >>>>>>>>>>>>>>>>p >>>>>>>>>>>>>>>>l >>>>>>>>>>>>>>>>.ja >>>>>>>>>>>>>>>> va: >>>>>>>>>>>>>>>> 43 >>>>>>>>>>>>>>>> 2) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>java.util.concurrent.Executors$RunnableAdapter.call(Executo >>>>>>>>>>>>>>>>r >>>>>>>>>>>>>>>>s >>>>>>>>>>>>>>>>.ja >>>>>>>>>>>>>>>> va: >>>>>>>>>>>>>>>> 47 >>>>>>>>>>>>>>>> 1) >>>>>>>>>>>>>>>> at >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:33 >>>>>>>>>>>4 >>>>>>>>>>>) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoo >>>>>>>>>>>>>>>>l >>>>>>>>>>>>>>>>E >>>>>>>>>>>>>>>>xec >>>>>>>>>>>>>>>> uto >>>>>>>>>>>>>>>> r. >>>>>>>>>>>>>>>> jav >>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>> :1110) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPo >>>>>>>>>>>>>>>>o >>>>>>>>>>>>>>>>l >>>>>>>>>>>>>>>>Exe >>>>>>>>>>>>>>>> cut >>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>> .ja >>>>>>>>>>>>>>>> v >>>>>>>>>>>>>>>> a:603) >>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:679) >>>>>>>>>>>>>>>> 2013-05-24 10:33:00,986 WARN [cloud.api.ApiDispatcher] >>>>>>>>>>>>>>>> (Job-Executor-16:job-230) class >>>>>>>>>> com.cloud.api.ServerApiException : >>>>>>>>>>>>> null >>>>>>>>>>>>>>>> 2013-05-24 10:33:00,986 DEBUG >>>>>>>> [cloud.async.AsyncJobManagerImpl] >>>>>>>>>>>>>>>> (Job-Executor-16:job-230) Complete async job-230, >>>>>>>> jobStatus: 2, >>>>>>>>>>>>>>>> resultCode: >>>>>>>>>>>>>>>> 530, result: Error Code: 534 Error text: null >>>>>>>>>>>>>>>> 2013-05-24 10:33:05,998 DEBUG >>>>>>>> [cloud.async.AsyncJobManagerImpl] >>>>>>>>>>>>>>>> (catalina-exec-4:null) Async job-230 completed >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> catalina.out >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> INFO [agent.manager.AgentManagerImpl] >>>>>>>> (Job-Executor-16:job-230) >>>>>>>>>>>>> Unable >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> disconnect host because it is not in the correct state: >>>>>>>> host=20; >>>>>>>>>>>>>>>> Status=Disconnected >>>>>>>>>>>>>>>> WARN [api.commands.ReconnectHostCmd] >>>>>>>> (Job-Executor-16:job-230) >>>>>>>>>>>>>>>> Exception: >>>>>>>>>>>>>>>> com.cloud.api.ServerApiException >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>com.cloud.api.commands.ReconnectHostCmd.execute(ReconnectHo >>>>>>>>>>>>>>>>s >>>>>>>>>>>>>>>>t >>>>>>>>>>>>>>>>Cmd >>>>>>>>>>>>>>>> .ja >>>>>>>>>>>>>>>> va >>>>>>>>>>>>>>>> :10 >>>>>>>>>>>>>>>> 8 >>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>>>>com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.jav >>>>>>>>>>a >>>>>>>>>>: >>>>>>>>>>432 >>>>>>>>>> ) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>>>>>> >>>>>>>>>>java.util.concurrent.Executors$RunnableAdapter.call(Executors.jav >>>>>>>>>>a >>>>>>>>>>: >>>>>>>>>>471 >>>>>>>>>> ) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334 >>>>>>>>>>) >>>>>>>>>>>>>>>> at >>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:166) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoo >>>>>>>>>>>>>>>>l >>>>>>>>>>>>>>>>E >>>>>>>>>>>>>>>>xec >>>>>>>>>>>>>>>> uto >>>>>>>>>>>>>>>> r. >>>>>>>>>>>>>>>> jav >>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>> :1110) >>>>>>>>>>>>>>>> at >>>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPo >>>>>>>>>>>>>>>>o >>>>>>>>>>>>>>>>l >>>>>>>>>>>>>>>>Exe >>>>>>>>>>>>>>>> cut >>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>> .ja >>>>>>>>>>>>>>>> v >>>>>>>>>>>>> >>>a:603) >>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:679) >>>>>>>>>>>>>>>> WARN [cloud.api.ApiDispatcher] (Job-Executor-16:job-230) >>>>>>>> class >>>>>>>>>>>>>>>> com.cloud.api.ServerApiException : null >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>>> >>>> >>> >> >
