Hello guys,
My colleague has done some testing of the latest snapshot based on nightly
4.2.1 from https://dist.apache.org/repos/dist/dev/cloudstack/4.2.1/ and
discovered several issues with using S3 as a secondary storage, which are very
serious in my opinion. Here is our setup:
- Ubuntu 12.04.3 with latest updates and KVM Hypervisor
- Libvirt 1.1.1 / QEMU 1.5.0 from ubuntu-cloud ppa (Havana)
- CEPH RBD / RADOSGW up to Emperor 0.72.1 (RBD as Primary and RadosGW S3 as
Secondary with NFS staging)
All issues are easily reproducible.
1) Any instance created from S3 Storage template or installed from ISO can not
Live Migrate. No matter if the state of original ISO is attached or detached,
when trying to Live Migrate, management server and the destination host agent
produce the following errors (By the way, the OS type set to Ubuntu 12 64bit,
but shows up as Apple Mac OS X):
2013-12-10 18:48:28,067 DEBUG [cloud.agent.Agent] (agentRequest-Handler-4:null)
Request:Seq 26-349831812: { Cmd , MgmtId: 90520737989049, via: 26, Ver: v1,
Flags: 100111,
[{"com.cloud.agent.api.PrepareForMigrationCommand":{"vm":{"id":56,"name":"i-2-56-VM","type":"User","cpus":2,"minSpeed":2800,"maxSpeed":2800,"minRam":2147483648,"maxRam":2147483648,"arch":"x86_64","os":"Apple
Mac OS X 10.6
(32-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"5c7980f779d3ffa3","params":{},"uuid":"bd033b3d-f86a-4d6f-bb8c-06e61b7e1d62","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6c9c3134-bfcf-4b8f-8508-db7d8fea5404","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a1a6908-7c45-3232-a250-550650793b1c","id":9,"poolType":"RBD","host":"ceph.admin.lv","path":"cloudstack","port":6789}},"name":"ROOT-56","size":21474836480,"path":"754a16ec-662c-4303-97f9-3168f1affbfb","volumeId":78,"vmName":"i-2-56-VM","accountId":2,"format":"RAW","id":78,"hypervisorType":"KVM"}},"diskSeq":0,"type":"ROOT"},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/212/212-2-e6277a31-7fb6-3ca1-9486-c383c9027cdb/ub.iso","origUrl":"http://www.emigrant.lv/ub.iso","uuid":"75badc3e-ca5e-490c-8450-5f4397c43789","id":212,"format":"ISO","accountId":2,"hvm":true,"displayText":"Ubuntu
Server 12.04.3
64bit","imageDataStore":{"com.cloud.agent.api.to.S3TO":{"id":11,"uuid":"ee84fa05-3ad5-4822-89fd-0e1817421b19","endPoint":"s3.admin.lv","bucketName":"cs-secondary","httpsFlag":false,"created":"Dec
10, 2013 3:40:55
PM","enableRRS":false}},"name":"212-2-e6277a31-7fb6-3ca1-9486-c383c9027cdb","hypervisorType":"None"}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"58903a2b-ef3c-40e5-8b83-99b343ee7474","ip":"10.50.1.249","netmask":"255.255.255.0","gateway":"10.50.1.1","mac":"02:00:30:44:00:01","dns1":"91.224.1.10","dns2":"212.70.182.77","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://578","isolationUri":"vlan://578","isSecurityGroupEnabled":false,"name":"cloudbr0"},{"deviceId":1,"networkRateMbps":200,"defaultNic":false,"uuid":"c0df2b28-e97b-4eda-91e0-71a171ec5509","ip":"10.50.1.27","netmask":"255.255.255.0","gateway":"10.50.1.1","mac":"02:00:6a:3e:00:10","dns1":"91.224.1.10","dns2":"212.70.182.
2013-12-10 18:48:28,124 WARN [cloud.agent.Agent] (agentRequest-Handler-4:null)
Caught:
java.lang.ClassCastException: com.cloud.agent.api.to.S3TO cannot be cast to
com.cloud.agent.api.to.NfsTO
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.getVolumePath(LibvirtComputingResource.java:3628)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2985)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1196)
at com.cloud.agent.Agent.processRequest(Agent.java:525)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
at com.cloud.utils.nio.Task.run(Task.java:83)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
2) Spanshoting seems to be totally broken:
- any ROOT volume for any instance is always marked as OVM Hypervisor (not KVM)
which brakes the normal behaviour of the action buttons filter in
scripts/storage.js in the GUI
- the "Volume details" Action Filter functions in scripts/storage.js lack some
conditions for hypervysor and volume type / state combination
- due to those bugs the GUI doesn't provide Take Snapshot / Recurring Snapshot
buttons for most of the volumes (had to create a DATA volume, attach it to VM
and then detach for Snapshot buttons to appear).
3) Regardless the value of the snapshot.backup.rightafter setting snapshots
created on RBD are always being copied to Secondary S3 / NFS storage.
Is it only me, but I think that the above issues, especially issue 1 and 2
should be considered as blockers as they make the S3 secondary storage unusable
for production purposes.
Many thanks
Andrei
