Hi, Do you try to ssh from the host running KVM and hosting the SystemVM ?
hug -----Message initial----- De:Sugandh S <s.suga...@rocketmail.com> Envoyé:mer. 19-03-2014 13:00 Sujet:Re: vm stuck in starting state, unable to delete it À:Rajesh Battala <rajesh.batt...@citrix.com>; users@cloudstack.apache.org; Sailaja Mada <sailaja.m...@citrix.com>; Suresh Sadhu <suresh.sa...@citrix.com>; Well, the router state seems to be "running" but I am not able to ping any of the system vms via their link local ip address or their public or private ips. When I try to ssh into console proxy vm, I get this: root@server2:~# ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.2.158 ssh: connect to host 169.254.2.158 port 3922: No route to host Thanks, Sugandh On Wednesday, 19 March 2014 5:31 PM, Rajesh Battala <rajesh.batt...@citrix.com> wrote: From the log, issue is while applying the dhcp entry in the VR hence deployment fails. Can you check whether the VR is up and network is in implemented state.? Thanks Rajesh Battala From:Sugandh S [mailto:s.suga...@rocketmail.com] Sent: Wednesday, March 19, 2014 5:07 PM To: Rajesh Battala; users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu Subject: Re: vm stuck in starting state, unable to delete it Hi, It took a couple of reboots to get system vms and router working again but now I have got another problem, whenever I create an instance I get "Unable to create a deployment for VM[User|<vmname>]" error. I have pasted the log here: http://tny.cz/1ee21d5e On Wednesday, 19 March 2014 4:39 PM, Rajesh Battala <rajesh.batt...@citrix.com> wrote: Can you just capture the log from when you started the action till you see the error. Thanks Rajesh Battala -----Original Message----- From: Sugandh S [mailto:s.suga...@rocketmail.com] Sent: Wednesday, March 19, 2014 4:23 PM To: Sailaja Mada; users@cloudstack.apache.org; Suresh Sadhu Subject: Re: vm stuck in starting state, unable to delete it Hi, The log file is around 10gb, I used pastebinit to upload it and it gave me "memory error". Is there any other way to provide the log file? Thanks, Sugandh On Wednesday, 19 March 2014 3:56 PM, Sailaja Mada <sailaja.m...@citrix.com> wrote: Hi, Can you please send the complete log using PasteBin. Thanks, Sailaja.M From:Sugandh S [mailto:s.suga...@rocketmail.com] Sent: 19 March 2014 15:30 To: users@cloudstack.apache.org; Sailaja Mada; Suresh Sadhu; Sugandh S Subject: Re: vm stuck in starting state, unable to delete it Hi, I just noticed that my domain router is also stuck in starting state and one of the vms I created is now showing error state. On Wednesday, 19 March 2014 3:25 PM, Sugandh S <s.suga...@rocketmail.com> wrote: Hi, > I have noticed VM in starting state when Template is getting Copied from > Secondary Storage to > Primary Storage . It's been over 150 minutes and I don't think it should take this long to copy the template. > size of the template and also value of global config parameter "wait" size of the iso is 700.29 MB and "wait" value is default "1800". Sugandh On Wednesday, 19 March 2014 3:16 PM, Sailaja Mada <sailaja.m...@citrix.com> wrote: Hi, I have noticed VM in starting state when Template is getting Copied from Secondary Storage to Primary Storage . VM gets deployed and will move to running state after this copy is completed. Can you please share the size of the template and also value of global config parameter "wait" One reason could be Storage Server is slow and Copy operation is taking longer time. It would help not to time out if you increase the "wait" value . But you may have to wait for the copy operation to complete to get the VM into running state. Thanks, Sailaja.M -----Original Message----- From: Sugandh S [mailto:s.suga...@rocketmail.com] Sent: 19 March 2014 15:01 To: Suresh Sadhu; users@cloudstack.apache.org Subject: Re: vm stuck in starting state, unable to delete it Here is another part of log e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600 2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi medoutException: Commands 1710489660 to Host 3 timed out after 3600 2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744901: Timed out on null 2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698 744901 to Host 4 timed out after 3600 2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati stics. 2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host : 4 2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running 2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null 2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600 2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi medoutException: Commands 1710489661 to Host 3 timed out after 3600 2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null 2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698 744902 to Host 4 timed out after 3600 2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati stics. 2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host : 4 2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null 2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600 2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta ts com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600 2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660 2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql] com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM] 2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null 2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after 3600 2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics. 2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4 2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720 2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner 2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested 2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4 2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720 2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM] 2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1, pod: pod1 2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null 2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600 2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600 2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null 2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698 744905 to Host 4 timed out after 3600 2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics. 2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4 2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null 2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600 2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600 2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:04:00,500 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null 2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600 2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics. 2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4 2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null 2014-03-19 14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600 2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600 2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null 2014-03-19 14:05:01,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600 2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics. 2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host : 4 2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null 2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600 2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600 2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null 2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed out after 3600 2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics. 2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4 2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null 2014-03-19 14:06:23,004 DEB On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <suresh.sa...@citrix.com> wrote: Can you please provide the logs and also did you notice any exception in the management log. For deleting vm : You can update the vm state in db as Stopped and try to delete them from CS. Regards Sadhu -----Original Message----- From: Sugandh S [mailto:s.suga...@rocketmail.com] Sent: 19 March 2014 14:30 To: users@cloudstack.apache.org Subject: vm stuck in starting state, unable to delete it Hello all, I am using CS 4.2 and my setup is as follows: One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary storage, it is /export/secondary. Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent. Now, when I create vms they are stuck in starting state and I am unable to delete them. Any and all help would be greatly appreciated. Thanks ahead, Sugandh
