Hi guys, we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right before that, from CentOS 6.5 to CentOS 6.6)
And now I can not start SYSTEM VR - that is used for SHARED GUEST network anymore. And some VMs are down - and cant be started because they depend on this VR... VPC VRs are created fine, so new VR for VPC are created fine, but this one fro Guest network fails to start: Here you can see, after agent copies template from secondary storage, to primary local storage, it created base image, and backing file - so storage setup seems completed. Than all out of sudden we have errors: can't find connection: KVM, for vm: r-2281-VM, continue 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) can't find connection: LXC, for vm: r-2281-VM, continue 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) can't find which hypervisor the vm used , then use the default hypervisor 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'r-2281-VM' - and BTW - absolutely NOTHING in the libvirt log ! Here is the full log, with all the steps, on the host that doesnt host any other VMs (except one - so log is easy to read) - you can notice after 1st fail, agent copies new image/backing file, and tries again with new image... And BTW, same template is of course used for both System VR and VPC VRs... 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd , MgmtId: 161344838950, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":" http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2 ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs:// 10.44.253.13/var/lib/libvirt/SECONDARY ","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":" http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2 ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem:// 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}] } 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) Processing command: org.apache.cloudstack.storage.command.CopyCommand 2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) createStoragePool didn't find existing running pool: org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid 'M2��N0�e���,�, need to create it 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) Didn't find an existing storage pool 274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools with duplicate paths 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) Checking path of existing pool 1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to create 2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) Checking path of existing pool 93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to create 2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) Checking path of existing pool 5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to create 2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) Attempting to create storage pool 274d3297-d74e-30ce-81d7-65a3ffe52cfc 2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) <pool type='netfs'> <name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name> <uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid> <source> <host name='10.44.253.13'/> <dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/> </source> <target> <path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path> </target> </pool> 2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) <volume> <name>66db63ef-7e8d-4d49-9b8a-365babd88732</name> <capacity>2621440000</capacity> <target> <format type='qcow2'/> <permissions><mode>0744</mode></permissions></target> </volume> 2015-03-15 16:52:30,510 DEBUG [utils.script.Script] (agentRequest-Handler-4:null) Executing: qemu-img info /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2 2015-03-15 16:52:30,520 DEBUG [utils.script.Script] (agentRequest-Handler-4:null) Execution is successful. 2015-03-15 16:52:30,522 DEBUG [utils.script.Script] (agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2 /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732 2015-03-15 16:52:38,339 DEBUG [utils.script.Script] (agentRequest-Handler-4:null) Execution is successful. 2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId: 161344838950, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm":false}},"result":true,"wait":0}}] } 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd , MgmtId: 161344838950, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":" http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2 ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem:// 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7 "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem:// 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}] } 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Processing command: org.apache.cloudstack.storage.command.CopyCommand 2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) <volume> <name>72ec282b-f53d-478f-955b-3dfa275da19b</name> <capacity>2621440000</capacity> <target> <format type='qcow2'/> <permissions><mode>0744</mode></permissions></target> </volume> 2015-03-15 16:52:45,362 DEBUG [utils.script.Script] (agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2 -b /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732 /var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b 2015-03-15 16:52:45,371 DEBUG [utils.script.Script] (agentRequest-Handler-2:null) Execution is successful. 2015-03-15 16:52:45,372 DEBUG [utils.script.Script] (agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305) Formatting '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b', fmt=qcow2 size=2621440000 backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732' encryption=off cluster_size=65536 2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId: 161344838950, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}] } 2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd , MgmtId: 161344838950, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}] } 2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Processing command: com.cloud.agent.api.StopCommand 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) can't find connection: KVM, for vm: r-2281-VM, continue 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) can't find connection: LXC, for vm: r-2281-VM, continue 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) can't find which hypervisor the vm used , then use the default hypervisor 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'r-2281-VM' 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'r-2281-VM' 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Try to stop the vm at first 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need to stop it 2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId: 161344838950, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] } 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd , MgmtId: 161344838950, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":" http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2 ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem:// 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7 "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem:// 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}] } 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) Processing command: org.apache.cloudstack.storage.command.CopyCommand 2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) <volume> <name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name> <capacity>2621440000</capacity> <target> <format type='qcow2'/> <permissions><mode>0744</mode></permissions></target> </volume> 2015-03-15 16:52:46,376 DEBUG [utils.script.Script] (agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2 -b /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732 /var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46 2015-03-15 16:52:46,385 DEBUG [utils.script.Script] (agentRequest-Handler-1:null) Execution is successful. 2015-03-15 16:52:46,386 DEBUG [utils.script.Script] (agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305) Formatting '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46', fmt=qcow2 size=2621440000 backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732' encryption=off cluster_size=65536 2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId: 161344838950, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}] } 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd , MgmtId: 161344838950, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}] } 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Processing command: com.cloud.agent.api.StopCommand 2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) can't find connection: KVM, for vm: r-2281-VM, continue 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) can't find connection: LXC, for vm: r-2281-VM, continue 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) can't find which hypervisor the vm used , then use the default hypervisor 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'r-2281-VM' 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) Failed to get dom xml: org.libvirt.LibvirtException: Domain not found: no domain with matching name 'r-2281-VM' 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) Try to stop the vm at first 2015-03-15 16:52:46,695 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need to stop it 2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId: 161344838950, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] } -- Andrija Panić