[
https://issues.apache.org/jira/browse/CLOUDSTACK-7179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073766#comment-14073766
]
edison su commented on CLOUDSTACK-7179:
---------------------------------------
Is there a network connection issue for primary storage?
Executing:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
nfs1-ccp.citrix.com -p /home/common/automation/SC_QA_AUTO4/primary2 -m
/mnt/e9b0d73b-9890-3aae-8934-44bd194592e0 -h 10.223.50.66
2014-07-23 22:50:34,976 DEBUG [kvm.resource.KVMHAMonitor] (Thread-1037:null)
Exit value is 1
2014-07-23 22:50:34,976 DEBUG [kvm.resource.KVMHAMonitor] (Thread-1037:null)
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh: line
131: echo: write error: Input/output error
2014-07-23 22:50:34,976 WARN [kvm.resource.KVMHAMonitor] (Thread-1037:null)
write heartbeat failed:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh: line
131: echo: write error: Input/output error, retry: 0
Then agent restart automatically, mgt server shutdown some routers because of
vm sync, as agent reports some routers in running state, while the mgt server
still thinks this routers are in the starting state.
> [Automation] Router deployment failing while calling "patchviasocket.pl"
> during router configuration
> ----------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-7179
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7179
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Virtual Router
> Affects Versions: 4.5.0
> Environment: KVM RHEL 6.3
> Reporter: Rayees Namathponnan
> Assignee: edison su
> Priority: Blocker
> Fix For: 4.5.0
>
>
> This issue is observed in automation environment, there are many router
> deployment failure, this issue observed while programming "patchviasocket.pl"
> 2014-07-23 22:50:23,660 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-384-VM -p
> %template=domP%name=r-384-VM%eth2ip=10.223.122.84%eth2mask=255.255.255.192%gateway=10.223.122.65%eth0ip=10.1.1.1%eth0mask=255.255.255.0%domain=csfeauto.advanced%cidrsize=24%dhcprange=10.1.1.1%eth1ip=169.254.1.76%eth1mask=255.255.0.0%type=router%disable_rp_filter=true%dns1=8.8.8.8
> 2014-07-23 22:50:24,973 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-3:null) Trying to connect to 169.254.3.89
> 2014-07-23 22:50:27,974 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-3:null) Could not connect to 169.254.3.89
> 2014-07-23 22:50:28,661 WARN [kvm.resource.LibvirtComputingResource]
> (Script-7:null) Interrupting script.
> 2014-07-23 22:50:28,662 WARN [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Timed out:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n
> r-384-VM -p
> %template=domP%name=r-384-VM%eth2ip=10.223.122.84%eth2mask=255.255.255.192%gateway=10.223.122.65%eth0ip=10.1.1.1%eth0mask=255.255.255.0%domain=csfeauto.advanced%cidrsize=24%dhcprange=10.1.1.1%eth1ip=169.254.1.76%eth1mask=255.255.0.0%type=router%disable_rp_filter=true%dns1=8.8.8.8
> . Output is:
> 2014-07-23 22:50:28,662 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) passcmd failed:timeout
> 2014-07-23 22:52:56,323 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Request:Seq 1-5899996986832060423: { Cmd ,
> MgmtId: 29066118877352, via: 1, Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":true,"vmName":"r-384-VM","wait":0}}]
> }
> 2014-07-23 22:52:56,323 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Processing command:
> com.cloud.agent.api.StopCommand
> 2014-07-23 22:52:56,326 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: KVM, for vm: r-384-VM,
> continue
> 2014-07-23 22:52:56,328 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: LXC, for vm: r-384-VM,
> continue
> 2014-07-23 22:52:56,328 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find which hypervisor the vm used , then
> use the default hypervisor
> 2014-07-23 22:52:56,330 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get vm status in case of
> checkboforecleanup is true
> org.libvirt.LibvirtException: Domain not found: no domain with matching name
> 'r-384-VM'
> at org.libvirt.ErrorHandler.processError(Unknown Source)
> at org.libvirt.Connect.processError(Unknown Source)
> at org.libvirt.Connect.processError(Unknown Source)
> at org.libvirt.Connect.domainLookupByName(Unknown Source)
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3506)
> at
> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1276)
> at com.cloud.agent.Agent.processRequest(Agent.java:501)
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
> at com.cloud.utils.nio.Task.run(Task.java:84)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2014-07-23 22:52:56,332 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: KVM, for vm: r-384-VM,
> continue
> 2014-07-23 22:52:56,334 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: LXC, for vm: r-384-VM,
> continue
> 2014-07-23 22:52:56,334 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find which hypervisor the vm used , then
> use the default hypervisor
> 2014-07-23 22:52:56,336 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching name
> 'r-384-VM'
> 2014-07-23 22:52:56,337 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching name
> 'r-384-VM'
> 2014-07-23 22:52:56,338 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching name
> 'r-384-VM'
> 2014-07-23 22:52:56,338 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Executing:
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
> destroy_network_rules_for_vm --vmname r-384-VM
> 2014-07-23 22:52:56,535 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Execution is successful.
> 2014-07-23 22:52:56,536 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Try to stop the vm at first
> 2014-07-23 22:52:56,537 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) VM r-384-VM doesn't exist, no need to stop it
> 2014-07-23 22:52:56,538 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Seq 1-5899996986832060423: { Ans: , MgmtId:
> 29066118877352, via: 1, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> 2014-07-23 22:52:56,641 DEBUG [cloud.agent.Agent] (Agent-Handler-1:null)
> Received response: Seq 0-1: { Ans: , MgmtId: 29066118877352, via: 1, Ver:
> v1, Flags: 100010,
> [{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":0,"wait":0},"result":true,"wait":0}}]
> }
> 2014-07-23 22:52:56,644 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Request:Seq 1-5899996986832060424: { Cmd ,
> MgmtId: 29066118877352, via: 1, Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.CheckVirtualMachineCommand":{"vmName":"r-386-VM","wait":20}}]
> }
> 2014-07-23 22:52:56,644 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Processing command:
> com.cloud.agent.api.CheckVirtualMachineCommand
--
This message was sent by Atlassian JIRA
(v6.2#6252)
