Hi,
Has the yum update touch any of your qemu/libvirt packages?
Lucian
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Andrija Panic" <andrija.pa...@gmail.com>
> To: users@cloudstack.apache.org, d...@cloudstack.apache.org
> Sent: Sunday, 15 March, 2015 16:19:07
> Subject: [URGENT - HELP NEEDED]
> Hi guys,
>
> we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right before
> that, from CentOS 6.5 to CentOS 6.6)
>
> And now I can not start SYSTEM VR - that is used for SHARED GUEST network
> anymore.
> And some VMs are down - and cant be started because they depend on this
> VR...
>
> VPC VRs are created fine, so new VR for VPC are created fine, but this one
> fro Guest network fails to start:
>
> Here you can see, after agent copies template from secondary storage, to
> primary local storage, it created base image, and backing file - so storage
> setup seems completed.
>
> Than all out of sudden we have errors:
>
> can't find connection: KVM, for vm: r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> then use the default hypervisor
> 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
>
> - and BTW - absolutely NOTHING in the libvirt log !
>
>
> Here is the full log, with all the steps, on the host that doesnt host any
> other VMs (except one - so log is easy to read) - you can notice after 1st
> fail, agent copies new image/backing file, and tries again with new image...
>
> And BTW, same template is of course used for both System VR and VPC VRs...
>
> 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
> 10.44.253.13/var/lib/libvirt/SECONDARY
> ","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}]
> }
> 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) createStoragePool didn't find existing
> running pool: org.libvirt.LibvirtException: Storage pool not found: no
> storage pool with matching uuid 'M2��N0�e���,�, need to create it
> 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Didn't find an existing storage pool
> 274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools with
> duplicate paths
> 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Checking path of existing pool
> 1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to create
> 2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Checking path of existing pool
> 93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to create
> 2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Checking path of existing pool
> 5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to create
> 2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Attempting to create storage pool
> 274d3297-d74e-30ce-81d7-65a3ffe52cfc
> 2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) <pool type='netfs'>
> <name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name>
> <uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid>
> <source>
> <host name='10.44.253.13'/>
> <dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/>
> </source>
> <target>
> <path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path>
> </target>
> </pool>
>
> 2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) <volume>
> <name>66db63ef-7e8d-4d49-9b8a-365babd88732</name>
> <capacity>2621440000</capacity>
> <target>
> <format type='qcow2'/>
> <permissions><mode>0744</mode></permissions></target>
> </volume>
>
> 2015-03-15 16:52:30,510 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Executing: qemu-img info
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> 2015-03-15 16:52:30,520 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Execution is successful.
> 2015-03-15 16:52:30,522 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> 2015-03-15 16:52:38,339 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Execution is successful.
> 2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm":false}},"result":true,"wait":0}}]
> }
> 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> }
> 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-2:null) <volume>
> <name>72ec282b-f53d-478f-955b-3dfa275da19b</name>
> <capacity>2621440000</capacity>
> <target>
> <format type='qcow2'/>
> <permissions><mode>0744</mode></permissions></target>
> </volume>
>
> 2015-03-15 16:52:45,362 DEBUG [utils.script.Script]
> (agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2 -b
> /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> /var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b
> 2015-03-15 16:52:45,371 DEBUG [utils.script.Script]
> (agentRequest-Handler-2:null) Execution is successful.
> 2015-03-15 16:52:45,372 DEBUG [utils.script.Script]
> (agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME
> environment variable not set. Disabling LTTng-UST per-user tracing. (in
> setup_local_apps() at lttng-ust-comm.c:305)
> Formatting '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b',
> fmt=qcow2 size=2621440000
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> encryption=off cluster_size=65536
>
> 2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> }
> 2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> }
> 2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Processing command:
> com.cloud.agent.api.StopCommand
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: KVM, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> then use the default hypervisor
> 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) Try to stop the vm at first
> 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need to stop it
> 2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> }
> 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-1:null) <volume>
> <name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name>
> <capacity>2621440000</capacity>
> <target>
> <format type='qcow2'/>
> <permissions><mode>0744</mode></permissions></target>
> </volume>
>
> 2015-03-15 16:52:46,376 DEBUG [utils.script.Script]
> (agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2 -b
> /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> /var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46
> 2015-03-15 16:52:46,385 DEBUG [utils.script.Script]
> (agentRequest-Handler-1:null) Execution is successful.
> 2015-03-15 16:52:46,386 DEBUG [utils.script.Script]
> (agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME
> environment variable not set. Disabling LTTng-UST per-user tracing. (in
> setup_local_apps() at lttng-ust-comm.c:305)
> Formatting '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46',
> fmt=qcow2 size=2621440000
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> encryption=off cluster_size=65536
>
> 2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> }
> 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> }
> 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Processing command:
> com.cloud.agent.api.StopCommand
> 2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-5:null) can't find connection: KVM, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-5:null) can't find connection: LXC, for vm:
> r-2281-VM, continue
> 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> (agentRequest-Handler-5:null) can't find which hypervisor the vm used ,
> then use the default hypervisor
> 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Failed to get dom xml:
> org.libvirt.LibvirtException: Domain not found: no domain with matching
> name 'r-2281-VM'
> 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Try to stop the vm at first
> 2015-03-15 16:52:46,695 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need to stop it
> 2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId:
> 161344838950, via: 1, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
>
>
>
>
>
> --
>
> Andrija Panić
