[
https://issues.apache.org/jira/browse/CLOUDSTACK-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799313#comment-13799313
]
Ivan Kozlov commented on CLOUDSTACK-4892:
-----------------------------------------
managesnapshot.sh checks snapshot with the following command:
if ! dmsetup info -c --noheadings -o name ${vg_dm}-${snapshotname} > /dev/null
2>&1; then
it is successful on one host (i think host that created snaphot on primary)
dmsetup info -c --noheadings -o name
vg_primary-7ce7c39f59b6391a287a19906241060d
vg_primary-7ce7c39f59b6391a287a19906241060d
and fails on others
dmsetup info -c --noheadings -o name vg_primary-7ce7c39f59b6391a287a19906241060d
Device does not exist.
Command failed
However on every host vg_primary-7ce7c39f59b6391a287a19906241060d--cow is
present
> KVM snapshots are failing on CLVM
> ---------------------------------
>
> Key: CLOUDSTACK-4892
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4892
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM, Snapshot
> Affects Versions: 4.2.0
> Environment: CentOS 6.4, KVM, CLVM
> Reporter: Ivan Kozlov
>
> Creating snaphot fails hanging with state CreatedOnPrimary. Sometimes
> creating snaphot is successful.
> Snapshot logical volume is created and not deleted.
> When running snaphot with only single host snapshot is created normaly. Guess
> snapshot backup is trying access snapshot LV from host on which snapshot LV
> is not opened.
> Here is management log:
> 2013-10-18 17:32:58,512 DEBUG [cloud.async.AsyncJobManagerImpl]
> (catalina-exec-10:null) submit async job-41 = [
> 88ec27d7-78af-4664-a01b-eeca4469e37c ], details: AsyncJobVO {id:41, userId:
> 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 10,
> cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
> cmdOriginator: null, cmdInfo:
> {"id":"10","response":"json","sessionkey":"HKb50xNHyZm2wJx/IHi5S7UWBGQ\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1382106777170","volumeid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","ctxAccountId":"2","ctxStartEventId":"126"},
> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
> processStatus: 0, resultCode: 0, result: null, initMsid: 161342718518,
> completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2013-10-18 17:32:58,514 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Executing
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-41
> = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]
> 2013-10-18 17:32:58,549 INFO [user.snapshot.CreateSnapshotCmd]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) VOLSS:
> createSnapshotCmd starts:1382106778549
> 2013-10-18 17:32:58,925 DEBUG [agent.transport.Request]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq
> 1-111542657: Sending { Cmd , MgmtId: 161342718518, via: 1, Ver: v1, Flags:
> 100011,
> [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"name":"ROOT-5","size":8589934592,"path":"4f3e8cfc-d3be-4e55-bc13-5c236a689c83","volumeId":5,"vmName":"i-2-5-VM","accountId":2,"format":"RAW","id":5,"hypervisorType":"KVM"},"parentSnapshotPath":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/7e85ab28-4ea5-4b5e-8ec1-1abadf2d571e","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"vmName":"i-2-5-VM","name":"test-100_ROOT-5_20131018143258","hypervisorType":"KVM","id":10}},"wait":0}}]
> }
> 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request]
> (AgentManager-Handler-9:null) Seq 1-111542657: Processing: { Ans: , MgmtId:
> 161342718518, via: 1, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/c6c900d1-1377-4347-ba69-9ba09f264f69","id":0}},"result":true,"wait":0}}]
> }
> 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq
> 1-111542657: Received: { Ans: , MgmtId: 161342718518, via: 1, Ver: v1,
> Flags: 10, { CreateObjectAnswer } }
> 2013-10-18 17:33:00,497 DEBUG [storage.motion.AncientDataMotionStrategy]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) copyAsync
> inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT
> 2013-10-18 17:33:00,547 DEBUG [agent.transport.Request]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq
> 4-1918238786: Sending { Cmd , MgmtId: 161342718518, via: 4, Ver: v1, Flags:
> 100111,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/c6c900d1-1377-4347-ba69-9ba09f264f69","volume":{"uuid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"name":"ROOT-5","size":8589934592,"path":"4f3e8cfc-d3be-4e55-bc13-5c236a689c83","volumeId":5,"vmName":"i-2-5-VM","accountId":2,"format":"RAW","id":5,"hypervisorType":"KVM"},"parentSnapshotPath":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/7e85ab28-4ea5-4b5e-8ec1-1abadf2d571e","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"vmName":"i-2-5-VM","name":"test-100_ROOT-5_20131018143258","hypervisorType":"KVM","id":10}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/5","volume":{"uuid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"name":"ROOT-5","size":8589934592,"path":"4f3e8cfc-d3be-4e55-bc13-5c236a689c83","volumeId":5,"vmName":"i-2-5-VM","accountId":2,"format":"RAW","id":5,"hypervisorType":"KVM"},"parentSnapshotPath":"snapshots/2/5/7e85ab28-4ea5-4b5e-8ec1-1abadf2d571e","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.10.31/export/secondary","_role":"Image"}},"vmName":"i-2-5-VM","name":"test-100_ROOT-5_20131018143258","hypervisorType":"KVM","id":10}},"executeInSequence":true,"wait":21600}}]
> }
> 2013-10-18 17:33:01,069 DEBUG [agent.transport.Request]
> (AgentManager-Handler-8:null) Seq 4-1918238786: Processing: { Ans: , MgmtId:
> 161342718518, via: 4, Ver: v1, Flags: 110,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Disk
> /dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83 has no snapshot called
> 5ece5182b7f6f10ef1b66b558f3d0bbb.","wait":0}}] }
> 2013-10-18 17:33:01,070 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-8:null) Seq 4-1918238786: No more commands found
> 2013-10-18 17:33:01,070 DEBUG [agent.transport.Request]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq
> 4-1918238786: Received: { Ans: , MgmtId: 161342718518, via: 4, Ver: v1,
> Flags: 110, { CopyCmdAnswer } }
> 2013-10-18 17:33:01,336 DEBUG [storage.snapshot.SnapshotManagerImpl]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Failed to
> create snapshot
> com.cloud.utils.exception.CloudRuntimeException: Disk
> /dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83 has no snapshot called
> 5ece5182b7f6f10ef1b66b558f3d0bbb.
> at
> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264)
> at
> com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1307)
> at
> com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2720)
> at
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> 2013-10-18 17:33:01,427 DEBUG [storage.volume.VolumeServiceImpl]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Take
> snapshot: 5 failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
> at
> com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1040)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1307)
> at
> com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2720)
> at
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Disk
> /dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83 has no snapshot called
> 5ece5182b7f6f10ef1b66b558f3d0bbb.
> at
> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264)
> at
> com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013)
> ... 16 more
> 2013-10-18 17:33:01,509 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Complete
> async job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ], jobStatus: 2,
> resultCode: 530, result: Error Code: 530 Error text: Failed to create
> snapshot due to an internal error creating snapshot for volume 5
--
This message was sent by Atlassian JIRA
(v6.1#6144)
