Your agent snippet just looks like the system trying to stop the vm.
If a vm fails to start, it will also run through the stop routine to
clean up all of the prework, so the 'failed to stop' debug is all
normal. You may need to go above and look at why it failed to start.
On Fri, Apr 11, 2014 at 3:53 PM, Serg Senko <kernc...@gmail.com> wrote:
> Hi,
>
> It's can be some know bug?
> Possible it's already solved in new releases of CS but i need the
> work-around or fix before upgrade or reference to bug id.
>
> Environment:
> CS 4.1.1
> libvirt-1.0.1
> qemu-kvm-1.2
> NFS Storage ( as primary for VR's )
> Advanced VLAN isolation
>
> After hypervisor host crashing, one of VR's has failed to start in failover
> case,
> I have stopped it through UI with force, then was removed the VR for
> re-create it again by start/create VM API call.
>
>
> Try to start the Instance associated with this network, but failed because
> the VR can't be started when newly created.
>
> cloudstack-agent:
>
> 2014-04-11 07:05:34,546 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> 2014-04-11 07:05:34,547 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> 2014-04-11 07:05:34,548 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> 2014-04-11 07:05:34,548 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Executing:
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> destroy_network_rules_for_vm --vmname r-377-VM
>
> 2014-04-11 07:05:34,663 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Execution is successful.
>
> 2014-04-11 07:05:34,664 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Try to stop the vm at first
>
> 2014-04-11 07:05:34,665 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to stop VM :r-377-VM :
>
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> at org.libvirt.ErrorHandler.processError(Unknown Source)
>
> at org.libvirt.Connect.processError(Unknown Source)
>
> at org.libvirt.Connect.domainLookupByUUIDString(Unknown Source)
>
> at org.libvirt.Connect.domainLookupByUUID(Unknown Source)
>
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:4021)
>
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.stopVM(LibvirtComputingResource.java:3970)
>
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2894)
>
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1032)
>
> at com.cloud.agent.Agent.processRequest(Agent.java:525)
>
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
>
> at com.cloud.utils.nio.Task.run(Task.java:83)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>
> at java.lang.Thread.run(Thread.java:679)
>
> 2014-04-11 07:05:34,666 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
> domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> 2014-04-11 07:05:34,667 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
> domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
> 2014-04-11 07:05:34,668 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-2:null) Failed to get vm status:Domain not found: no
> domain with matching uuid '373ab4a9-cb8c-3275-a455-b9b4b963a983'
>
>
>
>
> Management CS:
>
> 2014-04-11 07:05:40,503 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-114:job-3001) Found 5 ip(s) to apply as a part of domR
> VM[DomainRouter|r-377-VM] start.
>
> 2014-04-11 07:05:40,528 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-114:job-3001) Resending ipAssoc, port forwarding, load
> balancing rules as a part of Virtual router start
>
> 2014-04-11 07:05:40,542 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-114:job-3001) Found 1 firewall Egress rule(s) to apply as a
> part of domR VM[DomainRouter|r-377-VM] start.
>
> 2014-04-11 07:05:40,581 ERROR [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-114:job-3001) Failed to start instance
> VM[DomainRouter|r-377-VM]
>
> java.lang.NullPointerException
>
> at
> com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346)
>
> at
> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241)
>
> at
> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924)
>
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902)
>
> at
> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175)
>
> at
> com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518)
>
> at
> com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1434)
>
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>
> at
> com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:1596)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:746)
>
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
>
> at
> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:212)
>
> at
> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
>
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3871)
>
> at
> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2579)
>
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>
> at
> org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:120)
>
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
>
> at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
>
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>
> at java.lang.Thread.run(Thread.java:679)
>
> 2014-04-11 07:05:40,584 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-114:job-3001) Cleaning up resources for the vm
> VM[DomainRouter|r-377-VM] in Starting state
>
> 2014-04-11 07:05:40,585 DEBUG [agent.transport.Request]
> (Job-Executor-114:job-3001) Seq 5-187434248: Sending { Cmd , MgmtId:
> 66290989385104, via: 5, Ver: v1, Flags: 100111,
> [{"StopCommand":{"isProxy":false,"vmName":"r-377-VM","wait":0}}] }
>
> 2014-04-11 07:05:40,707 DEBUG [agent.transport.Request]
> (AgentManager-Handler-16:null) Seq 5-187434248: Processing: { Ans: ,
> MgmtId: 66290989385104, via: 5, Ver: v1, Flags: 110,
> [{"StopAnswer":{"vncPort":0,"result":true,"wait":0}}] }
>
> 2014-04-11 07:05:40,707 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-16:null) Seq 5-187434248: No more commands found
>
> 2014-04-11 07:05:40,707 DEBUG [agent.transport.Request]
> (Job-Executor-114:job-3001) Seq 5-187434248: Received: { Ans: , MgmtId:
> 66290989385104, via: 5, Ver: v1, Flags: 110, { StopAnswer } }
>
> 2014-04-11 07:05:40,710 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (Job-Executor-114:job-3001) Successfully updated user statistics as a part
> of domR VM[DomainRouter|r-377-VM] reboot/stop
>
> 2014-04-11 07:05:40,718 DEBUG [cloud.network.NetworkModelImpl]
> (Job-Executor-114:job-3001) Service SecurityGroup is not supported in the
> network id=204
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking NiciraNvp to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [network.element.NiciraNvpElement]
> (Job-Executor-114:job-3001) Checking if NiciraNvpElement can handle service
> Connectivity on network inewdate
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking JuniperSRX to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking Netscaler to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking F5BigIP to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking CiscoNexus1000vVSM to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking BigSwitchVnsElement to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [network.element.BigSwitchVnsElement]
> (Job-Executor-114:job-3001) Checking if BigSwitchVnsElement can handle
> service Connectivity on network inewdate
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking VirtualRouter to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking Ovs to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking SecurityGroupProvider to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,722 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking VpcVirtualRouter to release
> Nic[490-377-5e36703c-ec99-4288-9850-3c93e8c188f7-10.1.1.1]
>
> 2014-04-11 07:05:40,724 DEBUG [network.guru.ControlNetworkGuru]
> (Job-Executor-114:job-3001) Released nic: NicProfile[491-377-null-null-null
>
> 2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking NiciraNvp to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,725 DEBUG [network.element.NiciraNvpElement]
> (Job-Executor-114:job-3001) Checking if NiciraNvpElement can handle service
> Connectivity on network null
>
> 2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking JuniperSRX to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking Netscaler to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking F5BigIP to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,725 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking CiscoNexus1000vVSM to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking BigSwitchVnsElement to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,726 DEBUG [network.element.BigSwitchVnsElement]
> (Job-Executor-114:job-3001) Checking if BigSwitchVnsElement can handle
> service Connectivity on network null
>
> 2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking VirtualRouter to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking Ovs to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking SecurityGroupProvider to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,726 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-114:job-3001) Asking VpcVirtualRouter to release
> Nic[491-377-5e36703c-ec99-4288-9850-3c93e8c188f7-null]
>
> 2014-04-11 07:05:40,728 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-114:job-3001) Successfully released network resources for the
> vm VM[DomainRouter|r-377-VM]
>
> 2014-04-11 07:05:40,728 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-114:job-3001) Successfully cleanued up resources for the vm
> VM[DomainRouter|r-377-VM] in Starting state
>
> 2014-04-11 07:05:40,731 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-114:job-3001) VM state transitted from :Starting to Stopped
> with event: OperationFailedvm's original host id: null new host id: null
> host id before state transition: 5
>
>
> Someone know such problem? Or help me to debug it please.
>
>
>
> --
> ttyv0 "/usr/libexec/gmail Pc" webcons on secure
