[jira] [Commented] (CLOUDSTACK-4892) KVM snapshots are failing on CLVM
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799313#comment-13799313 ] Ivan Kozlov commented on CLOUDSTACK-4892: - managesnapshot.sh checks snapshot with the following command: if ! dmsetup info -c --noheadings -o name ${vg_dm}-${snapshotname} > /dev/null 2>&1; then it is successful on one host (i think host that created snaphot on primary) dmsetup info -c --noheadings -o name vg_primary-7ce7c39f59b6391a287a19906241060d vg_primary-7ce7c39f59b6391a287a19906241060d and fails on others dmsetup info -c --noheadings -o name vg_primary-7ce7c39f59b6391a287a19906241060d Device does not exist. Command failed However on every host vg_primary-7ce7c39f59b6391a287a19906241060d--cow is present > KVM snapshots are failing on CLVM > - > > Key: CLOUDSTACK-4892 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4892 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Snapshot >Affects Versions: 4.2.0 > Environment: CentOS 6.4, KVM, CLVM >Reporter: Ivan Kozlov > > Creating snaphot fails hanging with state CreatedOnPrimary. Sometimes > creating snaphot is successful. > Snapshot logical volume is created and not deleted. > When running snaphot with only single host snapshot is created normaly. Guess > snapshot backup is trying access snapshot LV from host on which snapshot LV > is not opened. > Here is management log: > 2013-10-18 17:32:58,512 DEBUG [cloud.async.AsyncJobManagerImpl] > (catalina-exec-10:null) submit async job-41 = [ > 88ec27d7-78af-4664-a01b-eeca4469e37c ], details: AsyncJobVO {id:41, userId: > 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 10, > cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, > cmdOriginator: null, cmdInfo: > {"id":"10","response":"json","sessionkey":"HKb50xNHyZm2wJx/IHi5S7UWBGQ\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1382106777170","volumeid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","ctxAccountId":"2","ctxStartEventId":"126"}, > cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, > processStatus: 0, resultCode: 0, result: null, initMsid: 161342718518, > completeMsid: null, lastUpdated: null, lastPolled: null, created: null} > 2013-10-18 17:32:58,514 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Executing > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-41 > = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ] > 2013-10-18 17:32:58,549 INFO [user.snapshot.CreateSnapshotCmd] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) VOLSS: > createSnapshotCmd starts:1382106778549 > 2013-10-18 17:32:58,925 DEBUG [agent.transport.Request] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq > 1-111542657: Sending { Cmd , MgmtId: 161342718518, via: 1, Ver: v1, Flags: > 100011, > [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"name":"ROOT-5","size":8589934592,"path":"4f3e8cfc-d3be-4e55-bc13-5c236a689c83","volumeId":5,"vmName":"i-2-5-VM","accountId":2,"format":"RAW","id":5,"hypervisorType":"KVM"},"parentSnapshotPath":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/7e85ab28-4ea5-4b5e-8ec1-1abadf2d571e","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"vmName":"i-2-5-VM","name":"test-100_ROOT-5_20131018143258","hypervisorType":"KVM","id":10}},"wait":0}}] > } > 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request] > (AgentManager-Handler-9:null) Seq 1-111542657: Processing: { Ans: , MgmtId: > 161342718518, via: 1, Ver: v1, Flags: 10, > [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/c6c900d1-1377-4347-ba69-9ba09f264f69","id":0}},"result":true,"wait":0}}] > } > 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq > 1-111542657: Received: { Ans: , MgmtId: 161342718518, via: 1, Ver: v1, > Flags: 10, { CreateObjectAnswer } } > 2013-10-18 17:33:00,497 DEBUG [storage.motion.AncientDataMotionStrategy] > (Job-
[jira] [Commented] (CLOUDSTACK-4892) KVM snapshots are failing on CLVM
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13819136#comment-13819136 ] Ivan Kozlov commented on CLOUDSTACK-4892: - I think this commit should solve the problem: https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=746896b unfortunately I cannot check it now. > KVM snapshots are failing on CLVM > - > > Key: CLOUDSTACK-4892 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4892 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Snapshot >Affects Versions: 4.2.0 > Environment: CentOS 6.4, KVM, CLVM >Reporter: Ivan Kozlov >Priority: Critical > > Creating snaphot fails hanging with state CreatedOnPrimary. Sometimes > creating snaphot is successful. > Snapshot logical volume is created and not deleted. > When running snaphot with only single host snapshot is created normaly. Guess > snapshot backup is trying access snapshot LV from host on which snapshot LV > is not opened. > Here is management log: > 2013-10-18 17:32:58,512 DEBUG [cloud.async.AsyncJobManagerImpl] > (catalina-exec-10:null) submit async job-41 = [ > 88ec27d7-78af-4664-a01b-eeca4469e37c ], details: AsyncJobVO {id:41, userId: > 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 10, > cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, > cmdOriginator: null, cmdInfo: > {"id":"10","response":"json","sessionkey":"HKb50xNHyZm2wJx/IHi5S7UWBGQ\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1382106777170","volumeid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","ctxAccountId":"2","ctxStartEventId":"126"}, > cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, > processStatus: 0, resultCode: 0, result: null, initMsid: 161342718518, > completeMsid: null, lastUpdated: null, lastPolled: null, created: null} > 2013-10-18 17:32:58,514 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Executing > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-41 > = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ] > 2013-10-18 17:32:58,549 INFO [user.snapshot.CreateSnapshotCmd] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) VOLSS: > createSnapshotCmd starts:1382106778549 > 2013-10-18 17:32:58,925 DEBUG [agent.transport.Request] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq > 1-111542657: Sending { Cmd , MgmtId: 161342718518, via: 1, Ver: v1, Flags: > 100011, > [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"name":"ROOT-5","size":8589934592,"path":"4f3e8cfc-d3be-4e55-bc13-5c236a689c83","volumeId":5,"vmName":"i-2-5-VM","accountId":2,"format":"RAW","id":5,"hypervisorType":"KVM"},"parentSnapshotPath":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/7e85ab28-4ea5-4b5e-8ec1-1abadf2d571e","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"vmName":"i-2-5-VM","name":"test-100_ROOT-5_20131018143258","hypervisorType":"KVM","id":10}},"wait":0}}] > } > 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request] > (AgentManager-Handler-9:null) Seq 1-111542657: Processing: { Ans: , MgmtId: > 161342718518, via: 1, Ver: v1, Flags: 10, > [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/c6c900d1-1377-4347-ba69-9ba09f264f69","id":0}},"result":true,"wait":0}}] > } > 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq > 1-111542657: Received: { Ans: , MgmtId: 161342718518, via: 1, Ver: v1, > Flags: 10, { CreateObjectAnswer } } > 2013-10-18 17:33:00,497 DEBUG [storage.motion.AncientDataMotionStrategy] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) copyAsync > inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT > 2013-10-18 17:33:00,547 DEBUG [agent.transport.Request] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq > 4-1918238786: Sending { Cmd , MgmtId: 161342718518, via: 4, Ver: v1, Flags: > 100111, > [{"org.apache.cloudstack.s
[jira] [Commented] (CLOUDSTACK-4892) KVM snapshots are failing on CLVM
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13819197#comment-13819197 ] Milamber commented on CLOUDSTACK-4892: -- With the lastest 4.2 branch (4.2.1-SNASHOT - 2013-11-11), the issue stills. (sometimes works / sometimes not works) 2013-11-11 18:18:15,894 INFO [user.snapshot.CreateSnapshotCmd] (Job-Executor-8:job-123 = [ 058ec85e-4ce1-42a1-aa22-9fa936c01b03 ]) VOLSS: createSnapshotCmd starts:1384193895894 com.cloud.utils.exception.CloudRuntimeException: Disk /dev/vg_baie/8d4626a2-c282-4a70-9e57-db5ad54b819a has no snapshot called a221071f6f7a198975b0907d79fa054d. at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:139) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:277) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1315) at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2773) at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:701) com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1040) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1315) at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2773) at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:701) Caused by: com.cloud.utils.exception.CloudRuntimeException: Disk /dev/vg_baie/8d4626a2-c282-4a70-9e57-db5ad54b819a has no snapshot called a221071f6f7a198975b0907d79fa054d. at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:139) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:277) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013) ... 16 more > KVM snapshots are failing on CLVM > - > > Key: CLOUDSTACK-4892 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4892 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Snapshot >Affects Versions: 4.2.0 > Environment: CentOS 6.4, KVM, CLVM >Reporter: Ivan Kozlov >Priority: Critical > > Creating snaphot fails hanging with state CreatedOnPrimary. Sometimes > creating snaphot is successful. > Snapshot logical volume is created and not deleted. > When running snaphot with only single host snapshot is created normaly. Guess > snapshot backup is trying acc