[jira] [Commented] (CLOUDSTACK-4892) KVM snapshots are failing on CLVM
[
https://issues.apache.org/jira/browse/CLOUDSTACK-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13819136#comment-13819136
]
Ivan Kozlov commented on CLOUDSTACK-4892:
-
I think this commit should solve the problem:
https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=746896b
unfortunately I cannot check it now.
KVM snapshots are failing on CLVM
-
Key: CLOUDSTACK-4892
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4892
Project: CloudStack
Issue Type: Bug
Security Level: Public(Anyone can view this level - this is the
default.)
Components: KVM, Snapshot
Affects Versions: 4.2.0
Environment: CentOS 6.4, KVM, CLVM
Reporter: Ivan Kozlov
Priority: Critical
Creating snaphot fails hanging with state CreatedOnPrimary. Sometimes
creating snaphot is successful.
Snapshot logical volume is created and not deleted.
When running snaphot with only single host snapshot is created normaly. Guess
snapshot backup is trying access snapshot LV from host on which snapshot LV
is not opened.
Here is management log:
2013-10-18 17:32:58,512 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-10:null) submit async job-41 = [
88ec27d7-78af-4664-a01b-eeca4469e37c ], details: AsyncJobVO {id:41, userId:
2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 10,
cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
cmdOriginator: null, cmdInfo:
{id:10,response:json,sessionkey:HKb50xNHyZm2wJx/IHi5S7UWBGQ\u003d,cmdEventType:SNAPSHOT.CREATE,ctxUserId:2,httpmethod:GET,_:1382106777170,volumeid:560a9f6e-9864-43cc-8096-ed9cd6c97311,ctxAccountId:2,ctxStartEventId:126},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 161342718518,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-10-18 17:32:58,514 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Executing
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-41
= [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]
2013-10-18 17:32:58,549 INFO [user.snapshot.CreateSnapshotCmd]
(Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) VOLSS:
createSnapshotCmd starts:1382106778549
2013-10-18 17:32:58,925 DEBUG [agent.transport.Request]
(Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq
1-111542657: Sending { Cmd , MgmtId: 161342718518, via: 1, Ver: v1, Flags:
100011,
[{org.apache.cloudstack.storage.command.CreateObjectCommand:{data:{org.apache.cloudstack.storage.to.SnapshotObjectTO:{volume:{uuid:560a9f6e-9864-43cc-8096-ed9cd6c97311,volumeType:ROOT,dataStore:{org.apache.cloudstack.storage.to.PrimaryDataStoreTO:{uuid:4a975c8c-997a-4d1d-aa88-810fd281cb04,id:1,poolType:CLVM,host:localhost,path:/vg_primary,port:0}},name:ROOT-5,size:8589934592,path:4f3e8cfc-d3be-4e55-bc13-5c236a689c83,volumeId:5,vmName:i-2-5-VM,accountId:2,format:RAW,id:5,hypervisorType:KVM},parentSnapshotPath:/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/7e85ab28-4ea5-4b5e-8ec1-1abadf2d571e,dataStore:{org.apache.cloudstack.storage.to.PrimaryDataStoreTO:{uuid:4a975c8c-997a-4d1d-aa88-810fd281cb04,id:1,poolType:CLVM,host:localhost,path:/vg_primary,port:0}},vmName:i-2-5-VM,name:test-100_ROOT-5_20131018143258,hypervisorType:KVM,id:10}},wait:0}}]
}
2013-10-18 17:32:59,986 DEBUG [agent.transport.Request]
(AgentManager-Handler-9:null) Seq 1-111542657: Processing: { Ans: , MgmtId:
161342718518, via: 1, Ver: v1, Flags: 10,
[{org.apache.cloudstack.storage.command.CreateObjectAnswer:{data:{org.apache.cloudstack.storage.to.SnapshotObjectTO:{path:/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/c6c900d1-1377-4347-ba69-9ba09f264f69,id:0}},result:true,wait:0}}]
}
2013-10-18 17:32:59,986 DEBUG [agent.transport.Request]
(Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq
1-111542657: Received: { Ans: , MgmtId: 161342718518, via: 1, Ver: v1,
Flags: 10, { CreateObjectAnswer } }
2013-10-18 17:33:00,497 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) copyAsync
inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT
2013-10-18 17:33:00,547 DEBUG [agent.transport.Request]
(Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq
4-1918238786: Sending { Cmd , MgmtId: 161342718518, via: 4, Ver: v1, Flags:
100111,
[jira] [Commented] (CLOUDSTACK-4892) KVM snapshots are failing on CLVM
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13819197#comment-13819197 ] Milamber commented on CLOUDSTACK-4892: -- With the lastest 4.2 branch (4.2.1-SNASHOT - 2013-11-11), the issue stills. (sometimes works / sometimes not works) 2013-11-11 18:18:15,894 INFO [user.snapshot.CreateSnapshotCmd] (Job-Executor-8:job-123 = [ 058ec85e-4ce1-42a1-aa22-9fa936c01b03 ]) VOLSS: createSnapshotCmd starts:1384193895894 com.cloud.utils.exception.CloudRuntimeException: Disk /dev/vg_baie/8d4626a2-c282-4a70-9e57-db5ad54b819a has no snapshot called a221071f6f7a198975b0907d79fa054d. at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:139) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:277) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1315) at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2773) at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:701) com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1040) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1315) at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2773) at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:701) Caused by: com.cloud.utils.exception.CloudRuntimeException: Disk /dev/vg_baie/8d4626a2-c282-4a70-9e57-db5ad54b819a has no snapshot called a221071f6f7a198975b0907d79fa054d. at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:139) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:277) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013) ... 16 more KVM snapshots are failing on CLVM - Key: CLOUDSTACK-4892 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4892 Project: CloudStack Issue Type: Bug Security Level: Public(Anyone can view this level - this is the default.) Components: KVM, Snapshot Affects Versions: 4.2.0 Environment: CentOS 6.4, KVM, CLVM Reporter: Ivan Kozlov Priority: Critical Creating snaphot fails hanging with state CreatedOnPrimary. Sometimes creating snaphot is successful. Snapshot logical volume is created and not deleted. When running snaphot with only single host snapshot is created normaly. Guess snapshot backup is trying access snapshot LV from
[jira] [Commented] (CLOUDSTACK-4892) KVM snapshots are failing on CLVM
[
https://issues.apache.org/jira/browse/CLOUDSTACK-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13799313#comment-13799313
]
Ivan Kozlov commented on CLOUDSTACK-4892:
-
managesnapshot.sh checks snapshot with the following command:
if ! dmsetup info -c --noheadings -o name ${vg_dm}-${snapshotname} /dev/null
21; then
it is successful on one host (i think host that created snaphot on primary)
dmsetup info -c --noheadings -o name
vg_primary-7ce7c39f59b6391a287a19906241060d
vg_primary-7ce7c39f59b6391a287a19906241060d
and fails on others
dmsetup info -c --noheadings -o name vg_primary-7ce7c39f59b6391a287a19906241060d
Device does not exist.
Command failed
However on every host vg_primary-7ce7c39f59b6391a287a19906241060d--cow is
present
KVM snapshots are failing on CLVM
-
Key: CLOUDSTACK-4892
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4892
Project: CloudStack
Issue Type: Bug
Security Level: Public(Anyone can view this level - this is the
default.)
Components: KVM, Snapshot
Affects Versions: 4.2.0
Environment: CentOS 6.4, KVM, CLVM
Reporter: Ivan Kozlov
Creating snaphot fails hanging with state CreatedOnPrimary. Sometimes
creating snaphot is successful.
Snapshot logical volume is created and not deleted.
When running snaphot with only single host snapshot is created normaly. Guess
snapshot backup is trying access snapshot LV from host on which snapshot LV
is not opened.
Here is management log:
2013-10-18 17:32:58,512 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-10:null) submit async job-41 = [
88ec27d7-78af-4664-a01b-eeca4469e37c ], details: AsyncJobVO {id:41, userId:
2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 10,
cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
cmdOriginator: null, cmdInfo:
{id:10,response:json,sessionkey:HKb50xNHyZm2wJx/IHi5S7UWBGQ\u003d,cmdEventType:SNAPSHOT.CREATE,ctxUserId:2,httpmethod:GET,_:1382106777170,volumeid:560a9f6e-9864-43cc-8096-ed9cd6c97311,ctxAccountId:2,ctxStartEventId:126},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 161342718518,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-10-18 17:32:58,514 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Executing
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-41
= [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]
2013-10-18 17:32:58,549 INFO [user.snapshot.CreateSnapshotCmd]
(Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) VOLSS:
createSnapshotCmd starts:1382106778549
2013-10-18 17:32:58,925 DEBUG [agent.transport.Request]
(Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq
1-111542657: Sending { Cmd , MgmtId: 161342718518, via: 1, Ver: v1, Flags:
100011,
[{org.apache.cloudstack.storage.command.CreateObjectCommand:{data:{org.apache.cloudstack.storage.to.SnapshotObjectTO:{volume:{uuid:560a9f6e-9864-43cc-8096-ed9cd6c97311,volumeType:ROOT,dataStore:{org.apache.cloudstack.storage.to.PrimaryDataStoreTO:{uuid:4a975c8c-997a-4d1d-aa88-810fd281cb04,id:1,poolType:CLVM,host:localhost,path:/vg_primary,port:0}},name:ROOT-5,size:8589934592,path:4f3e8cfc-d3be-4e55-bc13-5c236a689c83,volumeId:5,vmName:i-2-5-VM,accountId:2,format:RAW,id:5,hypervisorType:KVM},parentSnapshotPath:/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/7e85ab28-4ea5-4b5e-8ec1-1abadf2d571e,dataStore:{org.apache.cloudstack.storage.to.PrimaryDataStoreTO:{uuid:4a975c8c-997a-4d1d-aa88-810fd281cb04,id:1,poolType:CLVM,host:localhost,path:/vg_primary,port:0}},vmName:i-2-5-VM,name:test-100_ROOT-5_20131018143258,hypervisorType:KVM,id:10}},wait:0}}]
}
2013-10-18 17:32:59,986 DEBUG [agent.transport.Request]
(AgentManager-Handler-9:null) Seq 1-111542657: Processing: { Ans: , MgmtId:
161342718518, via: 1, Ver: v1, Flags: 10,
[{org.apache.cloudstack.storage.command.CreateObjectAnswer:{data:{org.apache.cloudstack.storage.to.SnapshotObjectTO:{path:/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/c6c900d1-1377-4347-ba69-9ba09f264f69,id:0}},result:true,wait:0}}]
}
2013-10-18 17:32:59,986 DEBUG [agent.transport.Request]
(Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq
1-111542657: Received: { Ans: , MgmtId: 161342718518, via: 1, Ver: v1,
Flags: 10, { CreateObjectAnswer } }
2013-10-18 17:33:00,497 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) copyAsync
inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT
2013-10-18 17:33:00,547 DEBUG [agent.transport.Request]
(Job-Executor-22:job-41
