Nic, looks like when storage pool was removed, cloudStack kept references host-> removed storage pool in the DB - its a bug.
To workaround on your current setup: * save the db dump of cloud db just in case * delete all the stale storageHost refs to remove: "delete from storage_pool_host_ref where pool_id not in (select id from storage_pool)" Now listHosts should execute w/o any problems. Let me know if it works. -Alena. On 10/15/12 9:45 AM, "Nik Martin" <nik.mar...@nfinausa.com> wrote: >On 10/15/2012 11:40 AM, Alena Prokharchyk wrote: >> Nik, the error you see, doesn't state that the VM migration failed. NPE >>happens on listHosts command, when we are building the response and >>trying to figure out if local storage is enabled on the host. >> >> What version of cloudStack you are running on? Did you remove any of >>the primary storages in your setup? >> >> -Alena. >> > >Alena, > >This is a 3.02 CS with Xenserver hosts. I have removed and re-added >primary storage, to work around bug 251, which takes an entire cloud >offline if any primary storage is put into maintenance mode and >shutdown. I guess the fallout from that is more serious than earlier >expected. I'll need further direction on how to recover from bug 251, >which knocks me entirely offline. > >Nik > > > >> From: Nik Martin >><nik.mar...@nfinausa.com<mailto:nik.mar...@nfinausa.com>> >> Reply-To: "'cloudstack-users@incubator. org'" >><cloudstack-users@incubator.apache.org<mailto:cloudstack-users@incubator. >>apache.org>> >> To: "'cloudstack-users@incubator. org'" >><cloudstack-users@incubator.apache.org<mailto:cloudstack-users@incubator. >>apache.org>> >> Subject: Error Creating VMs and viewing hosts >> >> I migrated some VMs around to different storage tags (I have two primary >> SANs, HD and SSD), and at some point, a migration failed, with this >>error: >> >> >> 2012-10-15 09:34:32,399 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> (StatsCollector-1:null) Cleanup succeeded. Details null >> 2012-10-15 09:34:32,408 DEBUG [agent.manager.DirectAgentAttache] >> (DirectAgent-188:null) Seq 4-857023789: Executing request >> 2012-10-15 09:34:32,626 ERROR [cloud.api.ApiDispatcher] >> (catalina-exec-2:null) Exception while executing ListHostsCmd: >> java.lang.NullPointerException >> at >> >>com.cloud.storage.StorageManagerImpl.isLocalStorageActiveOnHost(StorageMa >>nagerImpl.java:409) >> at >>com.cloud.api.ApiDBUtils.isLocalStorageActiveOnHost(ApiDBUtils.java:340) >> at >> >>com.cloud.api.ApiResponseHelper.createHostResponse(ApiResponseHelper.java >>:640) >> at com.cloud.api.commands.ListHostsCmd.execute(ListHostsCmd.java:167) >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:132) >> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:509) >> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:416) >> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:300) >> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:59) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) >> at >> >>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicat >>ionFilterChain.java:290) >> at >> >>org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilte >>rChain.java:206) >> at >> >>org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve >>.java:233) >> at >> >>org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve >>.java:191) >> at >> >>org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java: >>127) >> at >> >>org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java: >>102) >> at >> >>org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) >> at >> >>org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.j >>ava:109) >> at >> >>org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:29 >>8) >> at >> >>org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.ja >>va:889) >> at >> >>org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.proces >>s(Http11NioProtocol.java:721) >> at >> >>org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.ja >>va:2268) >> at >> >>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java >>:1110) >> at >> >>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav >>a:603) >> at java.lang.Thread.run(Thread.java:679) >> 2012-10-15 09:34:33,597 DEBUG [agent.manager.AgentManagerImpl] >> (AgentManager-Handler-14:null) Ping from 14 >> 2012-10-15 09:34:33,658 DEBUG [agent.manager.DirectAgentAttache] >> (DirectAgent-188:null) Seq 4-857023789: Response Received: >> 2012-10-15 09:34:33,658 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> (DirectAgent-188:null) Cleanup succeeded. Details null >> 2012-10-15 09:34:33,658 DEBUG [agent.transport.Request] >> (StatsCollector-1:null) Seq 4-857023789: Received: { Ans: , MgmtId: >> 130577622632, via: 4, Ver: v1, Flags: 10, { GetStorageStatsAnswer } } >> 2012-10-15 09:34:33,658 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> (StatsCollector-1:null) Cleanup succeeded. Details null >> 2012-10-15 09:34:33,665 DEBUG [agent.manager.DirectAgentAttache] >> (DirectAgent-70:null) Seq 4-857023790: Executing request >> 2012-10-15 09:34:34,455 DEBUG [agent.manager.DirectAgentAttache] >> (DirectAgent-70:null) Seq 4-857023790: Response Received: >> 2012-10-15 09:34:34,456 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> (DirectAgent-70:null) Cleanup succeeded. Details null >> 2012-10-15 09:34:34,456 DEBUG [agent.transport.Request] >> (StatsCollector-1:null) Seq 4-857023790: Received: { Ans: , MgmtId: >> 130577622632, via: 4, Ver: v1, Flags: 10, { GetStorageStatsAnswer } } >> 2012-10-15 09:34:34,456 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> (StatsCollector-1:null) Cleanup succeeded. Details null >> 2012-10-15 09:34:34,461 DEBUG [agent.manager.DirectAgentAttache] >> (DirectAgent-459:null) Seq 4-857023791: Executing request >> 2012-10-15 09:34:35,387 DEBUG [agent.manager.DirectAgentAttache] >> (DirectAgent-459:null) Seq 4-857023791: Response Received: >> 2012-10-15 09:34:35,387 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> (DirectAgent-459:null) Cleanup succeeded. Details null >> 2012-10-15 09:34:35,388 DEBUG [agent.transport.Request] >> (StatsCollector-1:null) Seq 4-857023791: Received: { Ans: , MgmtId: >> 130577622632, via: 4, Ver: v1, Flags: 10, { GetStorageStatsAnswer } } >> 2012-10-15 09:34:35,388 DEBUG [cloud.vm.VirtualMachineManagerImpl] >> (StatsCollector-1:null) Cleanup succeeded. Details null >> 2012-10-15 09:34:35,547 DEBUG [cloud.server.StatsCollector] >> (StatsCollector-1:null) HostStatsCollector is running... >> 2012-10-15 09:34:35,556 DEBUG [agent.manager.DirectAgentAttache] >> (DirectAgent-137:null) Seq 4-857023792: Executing request >> 2 >> >> Now, I can't view my hosts under Infrastructure, and can't create new >> VMs. I don't have any local storage in my cloud, so I don't understand >> the reference to local storage in the log. When I go to view hosts I get >> an empty popup box. All my storage and hosts are online, so my guess >> this is a database error, but I don;t know where to look. >> >> -- >> Regards, >> >> Nik >> >> Nik Martin >> VP Business Development >> Nfina Technologies, Inc. >> +1.251.243.0043 x1003 >> Relentless Reliability >> >> > > >-- >Regards, > >Nik > >Nik Martin >VP Business Development >Nfina Technologies, Inc. >+1.251.243.0043 x1003 >Relentless Reliability >
