Actually,
my collegue made a patch for
the
plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java,
where we first check if the image is already present on CEPH, and if not,
only then it tries to copy it to the CEPH.
But this is not permanent solution, since in the first place, ACS should
not try to copy image 2 times, but only 1...
Anybody possible to test creating VM from bigger template on CEPH ?
Thanks,
On 10 June 2015 at 10:55, Andrija Panic <andrija.pa...@gmail.com> wrote:
> Hi guys,
>
> we just experienced a very strange problem:
>
> ACS 4.5.1 vanila
> Ubuntu 14.04.2 - latest Qemu* binaries (2.0.0+dfsg-2ubuntu1.11)
> CEPH 0.94.1
>
> We are unable to create VM from template that is bigger than 20GB - on
> CEPH.
> Works with 10,14,18GB, but not with templates 20GB or bigger...
>
> Problem with 20GB or more: mgmt server / agent tries 2 times to
> move/create CEPH/RAW volume from NFS/QCOW2, although the first execution is
> successfull.
> (some timeout problem?)
>
> So, first time it detects it needs to copy from Qcow2/NFS to RAW/CEPH, it
> issues qemu-img convert and gets success on execution.
> Than again it tries to do EXACT same thing, so of course CEPH gives error
> because destinatin image already exists and VM provisioning fails.
>
> We dont see this behaviour with 10GB,14GB,18GB, but only with 20GB and
> bigger.
>
>
> Please find logs from a host, I marked critical points with "!!!!!!!":
>
>
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/204/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2","origUrl":"
> http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2
> ","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
> 10.23.2.1/data/tank/secondary
> ","_role":"Image"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2
> ","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8457c284-cf5d-3979-b82e-32ea5efeb97b","id":1,"poolType":"RBD","host":"mon.swiss2.local","path":"cold-storage","port":6789,"url":"RBD://mon.swiss2.local/cold-storage/?ROLE=Primary&STOREUUID=8457c284-cf5d-3979-b82e-32ea5efeb97b"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"executeInSequence":true,"options":{},"wait":21600}}]
> 2015-06-10 01:42:36,661 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-4:null) Processing command:
> org.apache.cloudstack.storage.command.CopyCommand
> 2015-06-10 01:42:36,661 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Attempting to create storage pool
> b6d6c679-8475-31f6-b28f-a2e56746c7b9 (NetworkFilesystem) in libvirt
> 2015-06-10 01:42:36,669 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) <pool type='netfs'>
> <name>b6d6c679-8475-31f6-b28f-a2e56746c7b9</name>
> <uuid>b6d6c679-8475-31f6-b28f-a2e56746c7b9</uuid>
> <source>
> <host name='10.23.2.1'/>
> <dir path='/data/tank/secondary/template/tmpl/2/204'/>
> </source>
> <target>
> <path>/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9</path>
> </target>
> </pool>
>
> 2015-06-10 01:42:37,764 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Trying to fetch storage pool
> b6d6c679-8475-31f6-b28f-a2e56746c7b9 from libvirt
> 2015-06-10 01:42:37,775 DEBUG [kvm.storage.KVMStorageProcessor]
> (agentRequest-Handler-4:null) Copying template to primary storage, template
> format is qcow2
> 2015-06-10 01:42:37,775 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Trying to fetch storage pool
> 8457c284-cf5d-3979-b82e-32ea5efeb97b from libvirt
> 2015-06-10 01:42:37,781 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) copyPhysicalDisk: disk size:1203378688,
> virtualsize:25769803776 format:qcow2
> 2015-06-10 01:42:37,781 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) The source image is not RBD, but the
> destination is. We will convert into RBD format 2
>
> 2015-06-10 01:42:37,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Starting copy from source image
> /mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
> to RBD image cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2
>
> (!!!!!!!!!!)
> 2015-06-10 01:42:37,782 DEBUG [utils.script.Script]
> (agentRequest-Handler-4:null) Executing:
> qemu-img convert -O raw
> /mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
> rbd:cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2:mon_host=mon.swiss2.local:auth_supported=cephx:id=admin:key=AQA9E2pVzqotGRAALYrGr1zI+eO0i5F5ghU5/g==:rbd_default_format=2:client_mount_timeout=30
>
> 2015-06-10 01:47:12,430 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Succesfully converted source image
> /mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
> to RBD image cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2
> 2015-06-10 01:47:12,549 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Succesfully connected to Ceph cluster at
> mon.swiss2.local:6789
> 2015-06-10 01:47:12,576 INFO [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-4:null) Attempting to remove storage pool
> b6d6c679-8475-31f6-b28f-a2e56746c7b9 from libvirt
>
> 2015-06-10 01:47:16,683 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) Request:Seq 13-5696772053646966888: { Cmd ,
> MgmtId: 90520740254323, via: 13, Ver: v1, Flags: 100111,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/204/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2","origUrl":"
> http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2
> ","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
> 10.23.2.1/data/tank/secondary
> ","_role":"Image"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> http://xxx.yyy.180.244/userdata/d3494552-eb13-478e-84d6-da16f06a01a6.qcow2","uuid":"9a61c577-e74f-4c22-b7f8-5e642ea69cc2","id":204,"format":"QCOW2","accountId":2,"checksum":"b62eb99b8dd4ecfe40b61c91c8c037c0","hvm":true,"displayText":"andrija-debian7","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8457c284-cf5d-3979-b82e-32ea5efeb97b","id":1,"poolType":"RBD","host":"mon.swiss2.local","path":"cold-storage","port":6789,"url":"RBD://mon.swiss2.local/cold-storage/?ROLE=Primary&STOREUUID=8457c284-cf5d-3979-b82e-32ea5efeb97b"}},"name":"204-2-f7d41a8b-de2e-3a71-a8cf-301bc76cb2ab","hypervisorType":"KVM"}},"executeInSequence":true,"options":{},"wait":21600}}]
> }
> 2015-06-10 01:47:16,686 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) createStoragePool didn't find existing
> running pool: org.libvirt.LibvirtException: Storage pool not found: no
> storage pool with matching uuid ���y�u1�����gFǹ, need to create it
> 2015-06-10 01:47:16,692 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) Attempting to create storage pool
> b6d6c679-8475-31f6-b28f-a2e56746c7b9
> 2015-06-10 01:47:16,693 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) <pool type='netfs'>
> <name>b6d6c679-8475-31f6-b28f-a2e56746c7b9</name>
> <uuid>b6d6c679-8475-31f6-b28f-a2e56746c7b9</uuid>
> <source>
> <host name='10.23.2.1'/>
> <dir path='/data/tank/secondary/template/tmpl/2/204'/>
> </source>
> <target>
> <path>/mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9</path>
> </target>
> </pool>
>
> 2015-06-10 01:47:16,775 DEBUG [kvm.storage.KVMStorageProcessor]
> (agentRequest-Handler-5:null) Copying template to primary storage, template
> format is qcow2
> 2015-06-10 01:47:16,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) copyPhysicalDisk: disk size:1203378688,
> virtualsize:25769803776 format:qcow2
> 2015-06-10 01:47:16,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) The source image is not RBD, but the
> destination is. We will convert into RBD format 2
> 2015-06-10 01:47:16,782 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> (agentRequest-Handler-5:null) Starting copy from source image
> /mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
> to RBD image cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2
>
> (!!!!!!!!!!)
> 2015-06-10 01:47:16,782 DEBUG [utils.script.Script]
> (agentRequest-Handler-5:null) Executing: qemu-img convert -O raw
> /mnt/b6d6c679-8475-31f6-b28f-a2e56746c7b9/475f79e4-82ef-36f8-b1f2-acfcf541ca95.qcow2
> rbd:cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2:mon_host=mon.swiss2.local:auth_supported=cephx:id=admin:key=AQA9E2pVzqotGRAALYrGr1zI+eO0i5F5ghU5/g==:rbd_default_format=2:client_mount_timeout=30
> 2015-06-10 01:47:16,844 DEBUG [utils.script.Script]
> (agentRequest-Handler-5:null) Exit value is 1
>
> 2015-06-10 01:47:16,845 DEBUG [utils.script.Script]
> (agentRequest-Handler-5:null) libust[3503/3503]: Warning: HOME environment
> variable not set. Disabling LTTng-UST per-user tracing. (in
> setup_local_apps() at lttng-ust-comm.c:305)qemu-img:
> rbd:cold-storage/9a61c577-e74f-4c22-b7f8-5e642ea69cc2:mon_host=mon.swiss2.local:auth_supported=cephx:id=admin:key=AQA9E2pVzqotGRAALYrGr1zI+eO0i5F5ghU5/g==:rbd_default_format=2:client_mount_timeout=30:
> error while converting raw: Could not create image: File exists
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
