Hi guys,
we just experienced a very strange problem:
ACS 4.5.1 vanila
Ubuntu 14.04.2 - latest Qemu* binaries (2.0.0+dfsg-2ubuntu1.11)
CEPH 0.94.1
We are unable to create VM from template that is bigger than 20GB - on CEPH.
Works with 10,14,18GB, but not with templates 20GB or bigger...
Problem with 20GB or more: mgmt server / agent tries 2 times to move/create
CEPH/RAW volume from NFS/QCOW2, although the first execution is successfull.
(some timeout problem?)
So, first time it detects it needs to copy from Qcow2/NFS to RAW/CEPH, it
issues qemu-img convert and gets success on execution.
Than again it tries to do EXACT same thing, so of course CEPH gives error
because destinatin image already exists and VM provisioning fails.
We dont see this behaviour with 10GB,14GB,18GB, but only with 20GB and
bigger.
Please find logs from a host, I marked critical points with "!!!!!!!":
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/204/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2","origUrl":"
http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2
","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
10.23.2.1/data/tank/secondary
","_role":"Image"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2
","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8457c284-cf5d-3979-b82e-32ea5efeb97b","id":1,"poolType":"RBD","host":"mon.swiss2.local","path":"cold-storage","port":6789,"url":"RBD://mon.swiss2.local/cold-storage/?ROLE=Primary&STOREUUID=8457c284-cf5d-3979-b82e-32ea5efeb97b"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"executeInSequence":true,"options":{},"wait":21600}}]
2015-06-10 01:42:36,661 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-4:null) Processing command:
org.apache.cloudstack.storage.command.CopyCommand
2015-06-10 01:42:36,661 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Attempting to create storage pool
b6d6c679-8475-31f6-b28f-a2e56746c7b9 (NetworkFilesystem) in libvirt
2015-06-10 01:42:36,669 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) <pool type='netfs'>
<name>b6d6c679-8475-31f6-b28f-a2e56746c7b9</name>
<uuid>b6d6c679-8475-31f6-b28f-a2e56746c7b9</uuid>
<source>
<host name='10.23.2.1'/>
<dir path='/data/tank/secondary/template/tmpl/2/204'/>
</source>
<target>
<path>/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9</path>
</target>
</pool>
2015-06-10 01:42:37,764 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Trying to fetch storage pool
b6d6c679-8475-31f6-b28f-a2e56746c7b9 from libvirt
2015-06-10 01:42:37,775 DEBUG [kvm.storage.KVMStorageProcessor]
(agentRequest-Handler-4:null) Copying template to primary storage, template
format is qcow2
2015-06-10 01:42:37,775 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Trying to fetch storage pool
8457c284-cf5d-3979-b82e-32ea5efeb97b from libvirt
2015-06-10 01:42:37,781 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) copyPhysicalDisk: disk size:1203378688,
virtualsize:25769803776 format:qcow2
2015-06-10 01:42:37,781 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) The source image is not RBD, but the
destination is. We will convert into RBD format 2
2015-06-10 01:42:37,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Starting copy from source image
/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
to RBD image cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2
(!!!!!!!!!!)
2015-06-10 01:42:37,782 DEBUG [utils.script.Script]
(agentRequest-Handler-4:null) Executing:
qemu-img convert -O raw
/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
rbd:cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2:mon_host=mon.swiss2.local:auth_supported=cephx:id=admin:key=AQA9E2pVzqotGRAALYrGr1zI+eO0i5F5ghU5/g==:rbd_default_format=2:client_mount_timeout=30
2015-06-10 01:47:12,430 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Succesfully converted source image
/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
to RBD image cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2
2015-06-10 01:47:12,549 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Succesfully connected to Ceph cluster at
mon.swiss2.local:6789
2015-06-10 01:47:12,576 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-4:null) Attempting to remove storage pool
b6d6c679-8475-31f6-b28f-a2e56746c7b9 from libvirt
2015-06-10 01:47:16,683 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) Request:Seq 13-5696772053646966888: { Cmd ,
MgmtId: 90520740254323, via: 13, Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/204/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2","origUrl":"
http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2
","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
10.23.2.1/data/tank/secondary
","_role":"Image"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8457c284-cf5d-3979-b82e-32ea5efeb97b","id":1,"poolType":"RBD","host":"mon.swiss2.local","path":"cold-storage","port":6789,"url":"RBD://mon.swiss2.local/cold-storage/?ROLE=Primary&STOREUUID=8457c284-cf5d-3979-b82e-32ea5efeb97b"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"executeInSequence":true,"options":{},"wait":21600}}]
}
2015-06-10 01:47:16,686 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) createStoragePool didn't find existing
running pool: org.libvirt.LibvirtException: Storage pool not found: no
storage pool with matching uuid ���y�u1�����gFǹ, need to create it
2015-06-10 01:47:16,692 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) Attempting to create storage pool
b6d6c679-8475-31f6-b28f-a2e56746c7b9
2015-06-10 01:47:16,693 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) <pool type='netfs'>
<name>b6d6c679-8475-31f6-b28f-a2e56746c7b9</name>
<uuid>b6d6c679-8475-31f6-b28f-a2e56746c7b9</uuid>
<source>
<host name='10.23.2.1'/>
<dir path='/data/tank/secondary/template/tmpl/2/204'/>
</source>
<target>
<path>/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9</path>
</target>
</pool>
2015-06-10 01:47:16,775 DEBUG [kvm.storage.KVMStorageProcessor]
(agentRequest-Handler-5:null) Copying template to primary storage, template
format is qcow2
2015-06-10 01:47:16,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) copyPhysicalDisk: disk size:1203378688,
virtualsize:25769803776 format:qcow2
2015-06-10 01:47:16,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) The source image is not RBD, but the
destination is. We will convert into RBD format 2
2015-06-10 01:47:16,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-5:null) Starting copy from source image
/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
to RBD image cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2
(!!!!!!!!!!)
2015-06-10 01:47:16,782 DEBUG [utils.script.Script]
(agentRequest-Handler-5:null) Executing: qemu-img convert -O raw
/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
rbd:cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2:mon_host=mon.swiss2.local:auth_supported=cephx:id=admin:key=AQA9E2pVzqotGRAALYrGr1zI+eO0i5F5ghU5/g==:rbd_default_format=2:client_mount_timeout=30
2015-06-10 01:47:16,844 DEBUG [utils.script.Script]
(agentRequest-Handler-5:null) Exit value is 1
2015-06-10 01:47:16,845 DEBUG [utils.script.Script]
(agentRequest-Handler-5:null) libust[3503/3503]: Warning: HOME environment
variable not set. Disabling LTTng-UST per-user tracing. (in
setup_local_apps() at lttng-ust-comm.c:305)qemu-img:
rbd:cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2:mon_host=mon.swiss2.local:auth_supported=cephx:id=admin:key=AQA9E2pVzqotGRAALYrGr1zI+eO0i5F5ghU5/g==:rbd_default_format=2:client_mount_timeout=30:
error while converting raw: Could not create image: File exists
--
Andrija Panić
