Sanjeev N created CLOUDSTACK-4211:
-------------------------------------
Summary: [Object_store_refactor] Failed to create snapshot with S3
on kvm
Key: CLOUDSTACK-4211
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4211
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: KVM, Snapshot, Storage Controller
Affects Versions: 4.2.0
Environment: Latest build from ACS 4.2 branch
Storage: S3 for secondary, NFS for secondary staging, and local storage for
primary
Cluster: KVM
Reporter: Sanjeev N
Priority: Critical
Fix For: 4.2.0
Failed to create snapshot with S3 on kvm
Steps to Reproduce:
=================
1. Bring up CS with KVM cluster using S3 for secondary, NFS for secondary
staging, and local storage for primary
2.Deploy guest vm using default cent os template with both root and data disk
3.Create snapshot on root/data disk
Result:
=====
Snapshot creation failed with CloudRuntimeException: unsupported protocol
Observations:
============
Snapshot is created on primary but failed to copy to secondary staging storage.
Following is the log snippet:
2013-08-09 13:28:25,757 DEBUG [cloud.api.ApiServlet] (catalina-exec-6:null)
===START=== 10.146.0.20 -- GET
command=createSnapshot&volumeid=2b3fdf2f-579b-4771-8ac8-3d7ef2988a79&response=json&sessionkey=zXXHX4Nf5yJtILkB4BA8E75f9HA%3D&_=1376049616400
2013-08-09 13:28:25,956 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-6:null) submit async job-17 = [
0c5003b9-e341-46c4-b58f-c9181a7ccba1 ], details: AsyncJobVO {id:17, userId: 3,
accountId: 3, sessionKey: null, instanceType: Snapshot, instanceId: 3, cmd:
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
cmdOriginator: null, cmdInfo:
{"id":"3","response":"json","sessionkey":"zXXHX4Nf5yJtILkB4BA8E75f9HA\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"3","httpmethod":"GET","_":"1376049616400","volumeid":"2b3fdf2f-579b-4771-8ac8-3d7ef2988a79","ctxAccountId":"3","ctxStartEventId":"79"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 7332683579487,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-08-09 13:28:25,959 DEBUG [cloud.api.ApiServlet] (catalina-exec-6:null)
===END=== 10.146.0.20 -- GET
command=createSnapshot&volumeid=2b3fdf2f-579b-4771-8ac8-3d7ef2988a79&response=json&sessionkey=zXXHX4Nf5yJtILkB4BA8E75f9HA%3D&_=1376049616400
2013-08-09 13:28:25,962 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-2:job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) Executing
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-17 =
[ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]
2013-08-09 13:28:26,039 INFO [user.snapshot.CreateSnapshotCmd]
(Job-Executor-2:job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) VOLSS:
createSnapshotCmd starts:1376069306039
2013-08-09 13:28:26,313 DEBUG [agent.transport.Request] (Job-Executor-2:job-17
= [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) Seq 1-1681129511: Sending { Cmd ,
MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"2b3fdf2f-579b-4771-8ac8-3d7ef2988a79","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c8c1d524-3f88-4faa-9092-9ee37ac9e25e","id":1,"poolType":"Filesystem","host":"10.147.40.8","path":"/var/lib/libvirt/images","port":0}},"name":"DATA-5","size":5368709120,"path":"98700ea8-2e2f-4cac-bb10-fb6c6bb11032","volumeId":7,"vmName":"i-3-5-VM","accountId":3,"format":"QCOW2","id":7,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c8c1d524-3f88-4faa-9092-9ee37ac9e25e","id":1,"poolType":"Filesystem","host":"10.147.40.8","path":"/var/lib/libvirt/images","port":0}},"vmName":"i-3-5-VM","name":"test-vm_DATA-5_20130809172825","hypervisorType":"KVM","id":3}},"wait":0}}]
}
2013-08-09 13:28:27,219 DEBUG [agent.transport.Request]
(AgentManager-Handler-11:null) Seq 1-1681129511: Processing: { Ans: , MgmtId:
7332683579487, via: 1, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/var/lib/libvirt/images/98700ea8-2e2f-4cac-bb10-fb6c6bb11032/46d77288-3034-48f3-9864-c14bb47bc7bf","id":0}},"result":true,"wait":0}}]
}
2013-08-09 13:28:27,219 DEBUG [agent.transport.Request] (Job-Executor-2:job-17
= [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) Seq 1-1681129511: Received: { Ans:
, MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10, { CreateObjectAnswer } }
2013-08-09 13:28:27,284 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-2:job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) copyAsync
inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT
2013-08-09 13:28:27,330 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-2:job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ])
needCacheStorage true, dest at snapshots/3/7 dest role
Image/var/lib/libvirt/images/98700ea8-2e2f-4cac-bb10-fb6c6bb11032/46d77288-3034-48f3-9864-c14bb47bc7bf
src role Primary
2013-08-09 13:28:27,433 DEBUG [agent.transport.Request] (Job-Executor-2:job-17
= [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) Seq 1-1681129512: Sending { Cmd ,
MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/var/lib/libvirt/images/98700ea8-2e2f-4cac-bb10-fb6c6bb11032/46d77288-3034-48f3-9864-c14bb47bc7bf","volume":{"uuid":"2b3fdf2f-579b-4771-8ac8-3d7ef2988a79","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c8c1d524-3f88-4faa-9092-9ee37ac9e25e","id":1,"poolType":"Filesystem","host":"10.147.40.8","path":"/var/lib/libvirt/images","port":0}},"name":"DATA-5","size":5368709120,"path":"98700ea8-2e2f-4cac-bb10-fb6c6bb11032","volumeId":7,"vmName":"i-3-5-VM","accountId":3,"format":"QCOW2","id":7,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c8c1d524-3f88-4faa-9092-9ee37ac9e25e","id":1,"poolType":"Filesystem","host":"10.147.40.8","path":"/var/lib/libvirt/images","port":0}},"vmName":"i-3-5-VM","name":"test-vm_DATA-5_20130809172825","hypervisorType":"KVM","id":3}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/3/7","volume":{"uuid":"2b3fdf2f-579b-4771-8ac8-3d7ef2988a79","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c8c1d524-3f88-4faa-9092-9ee37ac9e25e","id":1,"poolType":"Filesystem","host":"10.147.40.8","path":"/var/lib/libvirt/images","port":0}},"name":"DATA-5","size":5368709120,"path":"98700ea8-2e2f-4cac-bb10-fb6c6bb11032","volumeId":7,"vmName":"i-3-5-VM","accountId":3,"format":"QCOW2","id":7,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.S3TO":{"id":2,"uuid":"28c97b44-c81a-41db-9644-1f28c674d121","endPoint":"10.147.29.56:8080","bucketName":"imagestore119","httpsFlag":false,"created":"Aug
9, 2013 12:09:22
PM","enableRRS":false}},"vmName":"i-3-5-VM","name":"test-vm_DATA-5_20130809172825","hypervisorType":"KVM","id":3}},"cacheTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/3/7","volume":{"uuid":"2b3fdf2f-579b-4771-8ac8-3d7ef2988a79","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c8c1d524-3f88-4faa-9092-9ee37ac9e25e","id":1,"poolType":"Filesystem","host":"10.147.40.8","path":"/var/lib/libvirt/images","port":0}},"name":"DATA-5","size":5368709120,"path":"98700ea8-2e2f-4cac-bb10-fb6c6bb11032","volumeId":7,"vmName":"i-3-5-VM","accountId":3,"format":"QCOW2","id":7,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_kvm_os","_role":"ImageCache"}},"vmName":"i-3-5-VM","name":"test-vm_DATA-5_20130809172825","hypervisorType":"KVM","id":3}},"executeInSequence":false,"wait":21600}}]
}
2013-08-09 13:28:27,444 DEBUG [agent.transport.Request]
(AgentManager-Handler-13:null) Seq 1-1681129512: Processing: { Ans: , MgmtId:
7332683579487, via: 1, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"unsupported
protocol","wait":0}}] }
2013-08-09 13:28:27,444 DEBUG [agent.transport.Request] (Job-Executor-2:job-17
= [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) Seq 1-1681129512: Received: { Ans:
, MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10, { CopyCmdAnswer } }
2013-08-09 13:28:27,462 DEBUG [agent.transport.Request] (Job-Executor-2:job-17
= [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) Seq 3-967704583: Sending { Cmd ,
MgmtId: 7332683579487, via: 3, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/3/7","volume":{"uuid":"2b3fdf2f-579b-4771-8ac8-3d7ef2988a79","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c8c1d524-3f88-4faa-9092-9ee37ac9e25e","id":1,"poolType":"Filesystem","host":"10.147.40.8","path":"/var/lib/libvirt/images","port":0}},"name":"DATA-5","size":5368709120,"path":"98700ea8-2e2f-4cac-bb10-fb6c6bb11032","volumeId":7,"vmName":"i-3-5-VM","accountId":3,"format":"QCOW2","id":7,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_kvm_os","_role":"ImageCache"}},"vmName":"i-3-5-VM","name":"test-vm_DATA-5_20130809172825","hypervisorType":"KVM","id":3}},"wait":0}}]
}
2013-08-09 13:28:27,784 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-12:null) SeqA 2-354: Processing Seq 2-354: { Cmd ,
MgmtId: -1, via: 2, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n
\"connections\": [\n {\n \"id\": 2,\n \"clientInfo\": \"\",\n
\"host\": \"10.147.40.8\",\n \"port\": 5905,\n \"tag\":
\"5084fb7b-0113-4574-85e5-f85618a97a04\",\n \"createTime\":
1376049502150,\n \"lastUsedTime\": 1376049608916\n }\n
]\n}","wait":0}}] }
2013-08-09 13:28:27,798 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-12:null) SeqA 2-354: Sending Seq 2-354: { Ans: , MgmtId:
7332683579487, via: 2, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2013-08-09 13:28:27,905 DEBUG [agent.transport.Request]
(AgentManager-Handler-14:null) Seq 3-967704583: Processing: { Ans: , MgmtId:
7332683579487, via: 3, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"snapshot directory 3
doesn't exist","wait":0}}] }
2013-08-09 13:28:27,906 DEBUG [agent.transport.Request] (Job-Executor-2:job-17
= [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) Seq 3-967704583: Received: { Ans:
, MgmtId: 7332683579487, via: 3, Ver: v1, Flags: 10, { Answer } }
2013-08-09 13:28:27,940 DEBUG [storage.snapshot.SnapshotManagerImpl]
(Job-Executor-2:job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) Failed to
create snapshot
com.cloud.utils.exception.CloudRuntimeException: unsupported protocol
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:286)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:137)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:256)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1004)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1256)
at
com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2674)
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
2013-08-09 13:28:27,954 DEBUG [storage.volume.VolumeServiceImpl]
(Job-Executor-2:job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) Take
snapshot: 7 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to
create snapshot
2013-08-09 13:28:27,955 DEBUG [cloud.server.StatsCollector]
(StatsCollector-2:null) VmStatsCollector is running...
2013-08-09 13:28:27,984 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-2:job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]) Complete
async job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ], jobStatus: 2,
resultCode: 530, result: Error Code: 530 Error text: Failed to create snapshot
due to an internal error creating snapshot for volume 7
2013-08-09 13:28:28,221 DEBUG [cloud.server.StatsCollector]
(StatsCollector-1:null) HostStatsCollector is running...
2013-08-09 13:28:28,702 DEBUG [agent.transport.Request] (StatsCollector-2:null)
Seq 1-1681129513: Received: { Ans: , MgmtId: 7332683579487, via: 1, Ver: v1,
Flags: 10, { GetVmStatsAnswer } }
2013-08-09 13:28:28,775 DEBUG [agent.transport.Request] (StatsCollector-1:null)
Seq 1-1681129514: Received: { Ans: , MgmtId: 7332683579487, via: 1, Ver: v1,
Flags: 10, { GetHostStatsAnswer } }
2013-08-09 13:28:29,045 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null)
===START=== 10.146.0.20 -- GET
command=queryAsyncJobResult&jobId=0c5003b9-e341-46c4-b58f-c9181a7ccba1&response=json&sessionkey=zXXHX4Nf5yJtILkB4BA8E75f9HA%3D&_=1376049619735
2013-08-09 13:28:29,065 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-24:null) Async job-17 = [ 0c5003b9-e341-46c4-b58f-c9181a7ccba1 ]
completed
2013-08-09 13:28:29,075 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null)
===END=== 10.146.0.20 -- GET
command=queryAsyncJobResult&jobId=0c5003b9-e341-46c4-b58f-c9181a7ccba1&response=json&sessionkey=zXXHX4Nf5yJtILkB4BA8E75f9HA%3D&_=1376049619735
Attaching management server log file and cloud DB.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
