Hi guys,
we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right before
that, from CentOS 6.5 to CentOS 6.6)
And now I can not start SYSTEM VR - that is used for SHARED GUEST network
anymore.
And some VMs are down - and cant be started because they depend on this
VR...
VPC VRs are created fine, so new VR for VPC are created fine, but this one
fro Guest network fails to start:
Here you can see, after agent copies template from secondary storage, to
primary local storage, it created base image, and backing file - so storage
setup seems completed.
Than all out of sudden we have errors:
can't find connection: KVM, for vm: r-2281-VM, continue
2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) can't find connection: LXC, for vm:
r-2281-VM, continue
2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
then use the default hypervisor
2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-2281-VM'
- and BTW - absolutely NOTHING in the libvirt log !
Here is the full log, with all the steps, on the host that doesnt host any
other VMs (except one - so log is easy to read) - you can notice after 1st
fail, agent copies new image/backing file, and tries again with new image...
And BTW, same template is of course used for both System VR and VPC VRs...
2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":"
http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
10.44.253.13/var/lib/libvirt/SECONDARY
","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}]
}
2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) Processing command:
org.apache.cloudstack.storage.command.CopyCommand
2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) createStoragePool didn't find existing
running pool: org.libvirt.LibvirtException: Storage pool not found: no
storage pool with matching uuid 'M2��N0�e���,�, need to create it
2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Didn't find an existing storage pool
274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools with
duplicate paths
2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Checking path of existing pool
1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to create
2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Checking path of existing pool
93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to create
2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Checking path of existing pool
5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to create
2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Attempting to create storage pool
274d3297-d74e-30ce-81d7-65a3ffe52cfc
2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) <pool type='netfs'>
<name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name>
<uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid>
<source>
<host name='10.44.253.13'/>
<dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/>
</source>
<target>
<path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path>
</target>
</pool>
2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) <volume>
<name>66db63ef-7e8d-4d49-9b8a-365babd88732</name>
<capacity>2621440000</capacity>
<target>
<format type='qcow2'/>
<permissions><mode>0744</mode></permissions></target>
</volume>
2015-03-15 16:52:30,510 DEBUG [utils.script.Script]
(agentRequest-Handler-4:null) Executing: qemu-img info
/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
2015-03-15 16:52:30,520 DEBUG [utils.script.Script]
(agentRequest-Handler-4:null) Execution is successful.
2015-03-15 16:52:30,522 DEBUG [utils.script.Script]
(agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f
/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
2015-03-15 16:52:38,339 DEBUG [utils.script.Script]
(agentRequest-Handler-4:null) Execution is successful.
2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm":false}},"result":true,"wait":0}}]
}
2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
}
2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:null) Processing command:
org.apache.cloudstack.storage.command.CopyCommand
2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-2:null) <volume>
<name>72ec282b-f53d-478f-955b-3dfa275da19b</name>
<capacity>2621440000</capacity>
<target>
<format type='qcow2'/>
<permissions><mode>0744</mode></permissions></target>
</volume>
2015-03-15 16:52:45,362 DEBUG [utils.script.Script]
(agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2 -b
/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b
2015-03-15 16:52:45,371 DEBUG [utils.script.Script]
(agentRequest-Handler-2:null) Execution is successful.
2015-03-15 16:52:45,372 DEBUG [utils.script.Script]
(agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME
environment variable not set. Disabling LTTng-UST per-user tracing. (in
setup_local_apps() at lttng-ust-comm.c:305)
Formatting '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b',
fmt=qcow2 size=2621440000
backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
encryption=off cluster_size=65536
2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
}
2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
}
2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) Processing command:
com.cloud.agent.api.StopCommand
2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) can't find connection: KVM, for vm:
r-2281-VM, continue
2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) can't find connection: LXC, for vm:
r-2281-VM, continue
2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
then use the default hypervisor
2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-2281-VM'
2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-2281-VM'
2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) Try to stop the vm at first
2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need to stop it
2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7"}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
}
2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:null) Processing command:
org.apache.cloudstack.storage.command.CopyCommand
2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-1:null) <volume>
<name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name>
<capacity>2621440000</capacity>
<target>
<format type='qcow2'/>
<permissions><mode>0744</mode></permissions></target>
</volume>
2015-03-15 16:52:46,376 DEBUG [utils.script.Script]
(agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2 -b
/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46
2015-03-15 16:52:46,385 DEBUG [utils.script.Script]
(agentRequest-Handler-1:null) Execution is successful.
2015-03-15 16:52:46,386 DEBUG [utils.script.Script]
(agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME
environment variable not set. Disabling LTTng-UST per-user tracing. (in
setup_local_apps() at lttng-ust-comm.c:305)
Formatting '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46',
fmt=qcow2 size=2621440000
backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
encryption=off cluster_size=65536
2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
}
2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
}
2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) Processing command:
com.cloud.agent.api.StopCommand
2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) can't find connection: KVM, for vm:
r-2281-VM, continue
2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) can't find connection: LXC, for vm:
r-2281-VM, continue
2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-5:null) can't find which hypervisor the vm used ,
then use the default hypervisor
2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-2281-VM'
2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) Failed to get dom xml:
org.libvirt.LibvirtException: Domain not found: no domain with matching
name 'r-2281-VM'
2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) Try to stop the vm at first
2015-03-15 16:52:46,695 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need to stop it
2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId:
161344838950, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
--
Andrija Panić
