[jira] [Created] (CLOUDSTACK-4892) KVM snapshots are failing on CLVM
Ivan Kozlov created CLOUDSTACK-4892:
---
Summary: KVM snapshots are failing on CLVM
Key: CLOUDSTACK-4892
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4892
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: KVM, Snapshot
Affects Versions: 4.2.0
Environment: CentOS 6.4, KVM, CLVM
Reporter: Ivan Kozlov
Creating snaphot fails hanging with state CreatedOnPrimary. Sometimes creating
snaphot is successful.
Snapshot logical volume is created and not deleted.
When running snaphot with only single host snapshot is created normaly. Guess
snapshot backup is trying access snapshot LV from host on which snapshot LV is
not opened.
Here is management log:
2013-10-18 17:32:58,512 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-10:null) submit async job-41 = [
88ec27d7-78af-4664-a01b-eeca4469e37c ], details: AsyncJobVO {id:41, userId: 2,
accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 10, cmd:
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
cmdOriginator: null, cmdInfo:
{"id":"10","response":"json","sessionkey":"HKb50xNHyZm2wJx/IHi5S7UWBGQ\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1382106777170","volumeid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","ctxAccountId":"2","ctxStartEventId":"126"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 161342718518,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-10-18 17:32:58,514 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Executing
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-41 =
[ 88ec27d7-78af-4664-a01b-eeca4469e37c ]
2013-10-18 17:32:58,549 INFO [user.snapshot.CreateSnapshotCmd]
(Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) VOLSS:
createSnapshotCmd starts:1382106778549
2013-10-18 17:32:58,925 DEBUG [agent.transport.Request] (Job-Executor-22:job-41
= [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq 1-111542657: Sending { Cmd ,
MgmtId: 161342718518, via: 1, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"name":"ROOT-5","size":8589934592,"path":"4f3e8cfc-d3be-4e55-bc13-5c236a689c83","volumeId":5,"vmName":"i-2-5-VM","accountId":2,"format":"RAW","id":5,"hypervisorType":"KVM"},"parentSnapshotPath":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/7e85ab28-4ea5-4b5e-8ec1-1abadf2d571e","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"vmName":"i-2-5-VM","name":"test-100_ROOT-5_20131018143258","hypervisorType":"KVM","id":10}},"wait":0}}]
}
2013-10-18 17:32:59,986 DEBUG [agent.transport.Request]
(AgentManager-Handler-9:null) Seq 1-111542657: Processing: { Ans: , MgmtId:
161342718518, via: 1, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/c6c900d1-1377-4347-ba69-9ba09f264f69","id":0}},"result":true,"wait":0}}]
}
2013-10-18 17:32:59,986 DEBUG [agent.transport.Request] (Job-Executor-22:job-41
= [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq 1-111542657: Received: { Ans:
, MgmtId: 161342718518, via: 1, Ver: v1, Flags: 10, { CreateObjectAnswer } }
2013-10-18 17:33:00,497 DEBUG [storage.motion.AncientDataMotionStrategy]
(Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) copyAsync
inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT
2013-10-18 17:33:00,547 DEBUG [agent.transport.Request] (Job-Executor-22:job-41
= [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq 4-1918238786: Sending { Cmd ,
MgmtId: 161342718518, via: 4, Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/c6c900d1-1377-4347-ba69-9ba09f264f69","volume":{"uuid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"name":"ROOT-5","size":8589934592,"path":
[jira] [Created] (CLOUDSTACK-4893) Documentation about Security Groups and OVS
Ivan Kozlov created CLOUDSTACK-4893: --- Summary: Documentation about Security Groups and OVS Key: CLOUDSTACK-4893 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4893 Project: CloudStack Issue Type: Improvement Security Level: Public (Anyone can view this level - this is the default.) Components: Doc Environment: KVM, OVS Reporter: Ivan Kozlov Priority: Trivial In Installation Guide (8.1. KVM Hypervisor Host Installation - 8.1.8. Configure the network using OpenVswitch) nothing is said about compatability of Security Groups and OVS. I think it should be mentioned. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CLOUDSTACK-4892) KVM snapshots are failing on CLVM
[
https://issues.apache.org/jira/browse/CLOUDSTACK-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799313#comment-13799313
]
Ivan Kozlov commented on CLOUDSTACK-4892:
-
managesnapshot.sh checks snapshot with the following command:
if ! dmsetup info -c --noheadings -o name ${vg_dm}-${snapshotname} > /dev/null
2>&1; then
it is successful on one host (i think host that created snaphot on primary)
dmsetup info -c --noheadings -o name
vg_primary-7ce7c39f59b6391a287a19906241060d
vg_primary-7ce7c39f59b6391a287a19906241060d
and fails on others
dmsetup info -c --noheadings -o name vg_primary-7ce7c39f59b6391a287a19906241060d
Device does not exist.
Command failed
However on every host vg_primary-7ce7c39f59b6391a287a19906241060d--cow is
present
> KVM snapshots are failing on CLVM
> -
>
> Key: CLOUDSTACK-4892
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4892
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM, Snapshot
>Affects Versions: 4.2.0
> Environment: CentOS 6.4, KVM, CLVM
>Reporter: Ivan Kozlov
>
> Creating snaphot fails hanging with state CreatedOnPrimary. Sometimes
> creating snaphot is successful.
> Snapshot logical volume is created and not deleted.
> When running snaphot with only single host snapshot is created normaly. Guess
> snapshot backup is trying access snapshot LV from host on which snapshot LV
> is not opened.
> Here is management log:
> 2013-10-18 17:32:58,512 DEBUG [cloud.async.AsyncJobManagerImpl]
> (catalina-exec-10:null) submit async job-41 = [
> 88ec27d7-78af-4664-a01b-eeca4469e37c ], details: AsyncJobVO {id:41, userId:
> 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 10,
> cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
> cmdOriginator: null, cmdInfo:
> {"id":"10","response":"json","sessionkey":"HKb50xNHyZm2wJx/IHi5S7UWBGQ\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1382106777170","volumeid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","ctxAccountId":"2","ctxStartEventId":"126"},
> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
> processStatus: 0, resultCode: 0, result: null, initMsid: 161342718518,
> completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2013-10-18 17:32:58,514 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Executing
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-41
> = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]
> 2013-10-18 17:32:58,549 INFO [user.snapshot.CreateSnapshotCmd]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) VOLSS:
> createSnapshotCmd starts:1382106778549
> 2013-10-18 17:32:58,925 DEBUG [agent.transport.Request]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq
> 1-111542657: Sending { Cmd , MgmtId: 161342718518, via: 1, Ver: v1, Flags:
> 100011,
> [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"name":"ROOT-5","size":8589934592,"path":"4f3e8cfc-d3be-4e55-bc13-5c236a689c83","volumeId":5,"vmName":"i-2-5-VM","accountId":2,"format":"RAW","id":5,"hypervisorType":"KVM"},"parentSnapshotPath":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/7e85ab28-4ea5-4b5e-8ec1-1abadf2d571e","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"vmName":"i-2-5-VM","name":"test-100_ROOT-5_20131018143258","hypervisorType":"KVM","id":10}},"wait":0}}]
> }
> 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request]
> (AgentManager-Handler-9:null) Seq 1-111542657: Processing: { Ans: , MgmtId:
> 161342718518, via: 1, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/c6c900d1-1377-4347-ba69-9ba09f264f69","id":0}},"result":true,"wait":0}}]
> }
> 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq
> 1-111542657: Received: { Ans: , MgmtId: 161342718518, via: 1, Ver: v1,
> Flags: 10, { CreateObjectAnswer } }
> 2013-10-18 17:33:00,497 DEBUG [storage.motion.AncientDataMotionStrategy]
> (Job-
[jira] [Created] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
Ivan Kozlov created CLOUDSTACK-4939:
---
Summary: Failed to create snaphot (KVM, GFS2)
Key: CLOUDSTACK-4939
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: KVM, Snapshot
Affects Versions: 4.2.0
Environment: CentOS 6.4, KVM, Shared mount point primary storage,
GFS2, iSCSI
Reporter: Ivan Kozlov
Priority: Blocker
With one host snapshots are created ok. After adding second host some snapshots
fail (Failed to create snapshot due to an internal error creating snapshot for
volume 14) stucking with state "CreatedOnPrimary". Even when all VMs are
running on the same host.
debug libvirt log shows:
2013-10-23 17:31:21.634+: 20007: debug :
virStorageFileGetMetadataInternal:673 :
path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
2013-10-23 17:32:57.189+: 20015: debug : qemuSnapObjFromName:233 : Domain
snapshot not found: no domain snapshot with matching name
'909848a0-b3ec-4657-a53a-c449dc24365b'
2013-10-23 17:32:57.474+: 20009: debug :
virStorageFileGetMetadataInternal:673 :
path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
2013-10-23 17:34:28.264+: 20008: debug : qemuSnapObjFromName:233 : Domain
snapshot not found: no domain snapshot with matching name
'f4e51b11-ac79-4a6a-b887-8926ffbd5cca'
management server log:
2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd]
(Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS:
createSnapshotCmd starts:1382549390561
2013-10-23 20:29:52,053 DEBUG [agent.transport.Request] (Job-Executor-52:job-94
= [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq 6-1170407437: Waiting for Seq
1170407434 Scheduling: { Cmd , MgmtId: 161342718518, via: 6, Ver: v1, Flags:
100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/f4e51b11-ac79-4a6a-b887-8926ffbd5cca","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"snapshots/2/14/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.10.31/export/secondary","_role":"Image"}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"executeInSequence":true,"wait":21600}}]
}
2013-10-23 20:31:21,560 DEBUG [agent.transport.Request]
(AgentManager-Handler-8:null) Seq 6-1170407434: Processing: { Ans: , MgmtId:
161342718518, via: 6, Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"org.libvirt.LibvirtException:
Domain snapshot not found: no domain snapshot with matching name
'65113136-dfb5-4cea-8e65-1065462ca2fe'","wait":0}}] }
2013-10-23 20:31:21,832 DEBUG [storage.snapshot.SnapshotManagerImpl]
(Job-Executor-49:job-91 = [ e2bf2454-4273-4a89-bc38-35add8297eb1 ]) Failed to
create snapshot
com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException:
Domain snapshot not found: no domain snapshot with matching name
'65113136-dfb5-4cea-8e65-1065462ca2fe'
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
at
org.apache.cloudstack.storage.snapshot.X
[jira] [Updated] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
[
https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Kozlov updated CLOUDSTACK-4939:
Affects Version/s: 4.2.1
> Failed to create snaphot (KVM, GFS2)
>
>
> Key: CLOUDSTACK-4939
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM, Snapshot
>Affects Versions: 4.2.0, 4.2.1
> Environment: CentOS 6.4, KVM, Shared mount point primary storage,
> GFS2, iSCSI
>Reporter: Ivan Kozlov
>Priority: Blocker
> Labels: kvm, sharedstorage, snapshot
>
> With one host snapshots are created ok. After adding second host some
> snapshots fail (Failed to create snapshot due to an internal error creating
> snapshot for volume 14) stucking with state "CreatedOnPrimary". Even when all
> VMs are running on the same host.
> debug libvirt log shows:
> 2013-10-23 17:31:21.634+: 20007: debug :
> virStorageFileGetMetadataInternal:673 :
> path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
> 2013-10-23 17:32:57.189+: 20015: debug : qemuSnapObjFromName:233 : Domain
> snapshot not found: no domain snapshot with matching name
> '909848a0-b3ec-4657-a53a-c449dc24365b'
> 2013-10-23 17:32:57.474+: 20009: debug :
> virStorageFileGetMetadataInternal:673 :
> path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
> 2013-10-23 17:34:28.264+: 20008: debug : qemuSnapObjFromName:233 : Domain
> snapshot not found: no domain snapshot with matching name
> 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca'
> management server log:
> 2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd]
> (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS:
> createSnapshotCmd starts:1382549390561
> 2013-10-23 20:29:52,053 DEBUG [agent.transport.Request]
> (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq
> 6-1170407437: Waiting for Seq 1170407434 Scheduling: { Cmd , MgmtId:
> 161342718518, via: 6, Ver: v1, Flags: 100111,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/f4e51b11-ac79-4a6a-b887-8926ffbd5cca","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"snapshots/2/14/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.10.31/export/secondary","_role":"Image"}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"executeInSequence":true,"wait":21600}}]
> }
> 2013-10-23 20:31:21,560 DEBUG [agent.transport.Request]
> (AgentManager-Handler-8:null) Seq 6-1170407434: Processing: { Ans: , MgmtId:
> 161342718518, via: 6, Ver: v1, Flags: 110,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"org.libvirt.LibvirtException:
> Domain snapshot not found: no domain snapshot with matching name
> '65113136-dfb5-4cea-8e65-1065462ca2fe'","wait":0}}] }
> 2013-10-23 20:31:21,832 DEBUG [storage.snapshot.SnapshotManagerImpl]
> (Job-Executor-49:job-91 = [ e2bf2454-4273-4a89-bc38-35add8297eb1 ]) Failed to
> create snapshot
> com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot
> with matching name '651
[jira] [Commented] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
[
https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804407#comment-13804407
]
Ivan Kozlov commented on CLOUDSTACK-4939:
-
I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still
failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in
maintainance snapshots are working fine. It makes me think it's not KVM bug but
ACS bug.
To recreate issue:
1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over
iscsi)
2. deploy 4-6 VMs and start simultaneous snapshot.
setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue.
> Failed to create snaphot (KVM, GFS2)
>
>
> Key: CLOUDSTACK-4939
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM, Snapshot
>Affects Versions: 4.2.0, 4.2.1
> Environment: CentOS 6.4, KVM, Shared mount point primary storage,
> GFS2, iSCSI
>Reporter: Ivan Kozlov
>Priority: Blocker
> Labels: kvm, sharedstorage, snapshot
>
> With one host snapshots are created ok. After adding second host some
> snapshots fail (Failed to create snapshot due to an internal error creating
> snapshot for volume 14) stucking with state "CreatedOnPrimary". Even when all
> VMs are running on the same host.
> debug libvirt log shows:
> 2013-10-23 17:31:21.634+: 20007: debug :
> virStorageFileGetMetadataInternal:673 :
> path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
> 2013-10-23 17:32:57.189+: 20015: debug : qemuSnapObjFromName:233 : Domain
> snapshot not found: no domain snapshot with matching name
> '909848a0-b3ec-4657-a53a-c449dc24365b'
> 2013-10-23 17:32:57.474+: 20009: debug :
> virStorageFileGetMetadataInternal:673 :
> path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
> 2013-10-23 17:34:28.264+: 20008: debug : qemuSnapObjFromName:233 : Domain
> snapshot not found: no domain snapshot with matching name
> 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca'
> management server log:
> 2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd]
> (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS:
> createSnapshotCmd starts:1382549390561
> 2013-10-23 20:29:52,053 DEBUG [agent.transport.Request]
> (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq
> 6-1170407437: Waiting for Seq 1170407434 Scheduling: { Cmd , MgmtId:
> 161342718518, via: 6, Ver: v1, Flags: 100111,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/f4e51b11-ac79-4a6a-b887-8926ffbd5cca","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"snapshots/2/14/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.10.31/export/secondary","_role":"Image"}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"executeInSequence":true,"wait":21600}}]
> }
> 2013-10-23 20:31:21,560 DEBUG [agent.transport.Request]
> (AgentManager-Handler-8:null) Seq 6-1170407434: Processing: { Ans: , MgmtId:
> 161342718518, via: 6, Ver: v1, Flags: 110,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswe
[jira] [Updated] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
[
https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Kozlov updated CLOUDSTACK-4939:
Fix Version/s: 4.2.1
> Failed to create snaphot (KVM, GFS2)
>
>
> Key: CLOUDSTACK-4939
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM, Snapshot
>Affects Versions: 4.2.0, 4.2.1
> Environment: CentOS 6.4, KVM, Shared mount point primary storage,
> GFS2, iSCSI
>Reporter: Ivan Kozlov
>Priority: Blocker
> Labels: kvm, sharedstorage, snapshot
> Fix For: 4.2.1
>
>
> With one host snapshots are created ok. After adding second host some
> snapshots fail (Failed to create snapshot due to an internal error creating
> snapshot for volume 14) stucking with state "CreatedOnPrimary". Even when all
> VMs are running on the same host.
> debug libvirt log shows:
> 2013-10-23 17:31:21.634+: 20007: debug :
> virStorageFileGetMetadataInternal:673 :
> path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
> 2013-10-23 17:32:57.189+: 20015: debug : qemuSnapObjFromName:233 : Domain
> snapshot not found: no domain snapshot with matching name
> '909848a0-b3ec-4657-a53a-c449dc24365b'
> 2013-10-23 17:32:57.474+: 20009: debug :
> virStorageFileGetMetadataInternal:673 :
> path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
> 2013-10-23 17:34:28.264+: 20008: debug : qemuSnapObjFromName:233 : Domain
> snapshot not found: no domain snapshot with matching name
> 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca'
> management server log:
> 2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd]
> (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS:
> createSnapshotCmd starts:1382549390561
> 2013-10-23 20:29:52,053 DEBUG [agent.transport.Request]
> (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq
> 6-1170407437: Waiting for Seq 1170407434 Scheduling: { Cmd , MgmtId:
> 161342718518, via: 6, Ver: v1, Flags: 100111,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/f4e51b11-ac79-4a6a-b887-8926ffbd5cca","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"snapshots/2/14/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.10.31/export/secondary","_role":"Image"}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"executeInSequence":true,"wait":21600}}]
> }
> 2013-10-23 20:31:21,560 DEBUG [agent.transport.Request]
> (AgentManager-Handler-8:null) Seq 6-1170407434: Processing: { Ans: , MgmtId:
> 161342718518, via: 6, Ver: v1, Flags: 110,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"org.libvirt.LibvirtException:
> Domain snapshot not found: no domain snapshot with matching name
> '65113136-dfb5-4cea-8e65-1065462ca2fe'","wait":0}}] }
> 2013-10-23 20:31:21,832 DEBUG [storage.snapshot.SnapshotManagerImpl]
> (Job-Executor-49:job-91 = [ e2bf2454-4273-4a89-bc38-35add8297eb1 ]) Failed to
> create snapshot
> com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot
[jira] [Comment Edited] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
[
https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804407#comment-13804407
]
Ivan Kozlov edited comment on CLOUDSTACK-4939 at 10/24/13 7:15 PM:
---
I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still
failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in
maintainance snapshots are working fine. It makes me think it's not KVM bug but
ACS bug.
To recreate issue:
1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over
iscsi)
2. deploy 4-6 VMs and start simultaneous snapshot.
setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue.
snapshot files are actually created on secondary storage. there is a record in
mysql cloud.snapshots, but there is no record in cloud.snapshot_store_ref.
Manually updating snapshot status to "BackedUp" and adding record with path to
snapshot_store_ref makes snapshot available. I was able to create template from
this snapshot and deploy VM from the template.
was (Author: eldorado):
I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still
failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in
maintainance snapshots are working fine. It makes me think it's not KVM bug but
ACS bug.
To recreate issue:
1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over
iscsi)
2. deploy 4-6 VMs and start simultaneous snapshot.
setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue.
> Failed to create snaphot (KVM, GFS2)
>
>
> Key: CLOUDSTACK-4939
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM, Snapshot
>Affects Versions: 4.2.0, 4.2.1
> Environment: CentOS 6.4, KVM, Shared mount point primary storage,
> GFS2, iSCSI
>Reporter: Ivan Kozlov
>Priority: Blocker
> Labels: kvm, sharedstorage, snapshot
> Fix For: 4.2.1
>
>
> With one host snapshots are created ok. After adding second host some
> snapshots fail (Failed to create snapshot due to an internal error creating
> snapshot for volume 14) stucking with state "CreatedOnPrimary". Even when all
> VMs are running on the same host.
> debug libvirt log shows:
> 2013-10-23 17:31:21.634+: 20007: debug :
> virStorageFileGetMetadataInternal:673 :
> path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
> 2013-10-23 17:32:57.189+: 20015: debug : qemuSnapObjFromName:233 : Domain
> snapshot not found: no domain snapshot with matching name
> '909848a0-b3ec-4657-a53a-c449dc24365b'
> 2013-10-23 17:32:57.474+: 20009: debug :
> virStorageFileGetMetadataInternal:673 :
> path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
> 2013-10-23 17:34:28.264+: 20008: debug : qemuSnapObjFromName:233 : Domain
> snapshot not found: no domain snapshot with matching name
> 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca'
> management server log:
> 2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd]
> (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS:
> createSnapshotCmd starts:1382549390561
> 2013-10-23 20:29:52,053 DEBUG [agent.transport.Request]
> (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq
> 6-1170407437: Waiting for Seq 1170407434 Scheduling: { Cmd , MgmtId:
> 161342718518, via: 6, Ver: v1, Flags: 100111,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/f4e51b11-ac79-4a6a-b887-8926ffbd5cca","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","
[jira] [Comment Edited] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
[
https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804407#comment-13804407
]
Ivan Kozlov edited comment on CLOUDSTACK-4939 at 10/25/13 2:13 PM:
---
I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still
failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in
maintainance snapshots are working fine. It makes me think it's not KVM bug but
ACS bug.
To recreate issue:
1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over
iscsi)
2. deploy 4-6 VMs and start simultaneous snapshot.
setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue.
snapshot files are actually created on secondary storage. there is a record in
mysql cloud.snapshots, but there is no record in cloud.snapshot_store_ref.
Manually updating snapshot status to "BackedUp" and adding record with path to
snapshot_store_ref makes snapshot available. I was able to create template from
this snapshot and deploy VM from the template.
ADD:
Continue researching. Finally enabled logging on agents (by the way that's
another bug, need to move /etc/cloudstack/agent/log4j-cloud.xml to
/etc/cloudstack/agent/log4j.xml to make logging work).
I created 4 instances on 2 hosts. Performed twice backup of each VM. We are
supposed to get 8 create snapshot commands, 8 backup snapshot and 8 remove
snapshot. However
on first host:
managesnapshot.sh -c was run 2 times
managesnapshot.sh -b was run 5 times
managesnapshot.sh -d was run 3 times
on the second host
managesnapshot.sh -c was run 2 times
managesnapshot.sh -b was run 3 times
managesnapshot.sh -d was run 2 times
so in total we get 4 create snapshot commands, 8 backup snapshot and 5 delete
snapshot. it seems to be very strange.
was (Author: eldorado):
I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still
failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in
maintainance snapshots are working fine. It makes me think it's not KVM bug but
ACS bug.
To recreate issue:
1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over
iscsi)
2. deploy 4-6 VMs and start simultaneous snapshot.
setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue.
snapshot files are actually created on secondary storage. there is a record in
mysql cloud.snapshots, but there is no record in cloud.snapshot_store_ref.
Manually updating snapshot status to "BackedUp" and adding record with path to
snapshot_store_ref makes snapshot available. I was able to create template from
this snapshot and deploy VM from the template.
> Failed to create snaphot (KVM, GFS2)
>
>
> Key: CLOUDSTACK-4939
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM, Snapshot
>Affects Versions: 4.2.0, 4.2.1
> Environment: CentOS 6.4, KVM, Shared mount point primary storage,
> GFS2, iSCSI
>Reporter: Ivan Kozlov
>Priority: Blocker
> Labels: kvm, sharedstorage, snapshot
> Fix For: 4.2.1
>
>
> With one host snapshots are created ok. After adding second host some
> snapshots fail (Failed to create snapshot due to an internal error creating
> snapshot for volume 14) stucking with state "CreatedOnPrimary". Even when all
> VMs are running on the same host.
> debug libvirt log shows:
> 2013-10-23 17:31:21.634+: 20007: debug :
> virStorageFileGetMetadataInternal:673 :
> path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
> 2013-10-23 17:32:57.189+: 20015: debug : qemuSnapObjFromName:233 : Domain
> snapshot not found: no domain snapshot with matching name
> '909848a0-b3ec-4657-a53a-c449dc24365b'
> 2013-10-23 17:32:57.474+: 20009: debug :
> virStorageFileGetMetadataInternal:673 :
> path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
> 2013-10-23 17:34:28.264+: 20008: debug : qemuSnapObjFromName:233 : Domain
> snapshot not found: no domain snapshot with matching name
> 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca'
> management server log:
> 2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd]
> (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS:
> createSnapshotCmd starts:1382549390561
> 2013-10-23 20:29:52,053 DEBUG [agent.transport.Request]
> (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq
> 6-1170407437: Waiting for Seq 1170407434 Scheduling: { Cmd , MgmtId:
> 161342718518, via: 6, Ver: v1, Flags: 100111,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudst
[jira] [Commented] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
[
https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805701#comment-13805701
]
Ivan Kozlov commented on CLOUDSTACK-4939:
-
I think I know what's going on. Management server send commands to
create/backup/delete snapshots randomly to any host. Depending on VM is running
or not on the host, host uses libvirt or qemu-img.
For example host1 creates snapshot using qemu-img. Command snapshot backup is
sent to host2, where VM is running. host2 tries to backup snapshot using
libvirt but there is no snapshot visible for that domain (because it was
created with qemu-img). so backup fails. the more hosts are in cluster the more
possible is snapshot failure.
So we need to
1. Check if VM is running before snapshot.
2. If VM is running send all commands (create/backup/delete snapshot) only to
the host where it is running.
To my mind this should solve the issue. Maybe someone can do this?
> Failed to create snaphot (KVM, GFS2)
>
>
> Key: CLOUDSTACK-4939
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM, Snapshot
>Affects Versions: 4.2.0, 4.2.1
> Environment: CentOS 6.4, KVM, Shared mount point primary storage,
> GFS2, iSCSI
>Reporter: Ivan Kozlov
>Priority: Blocker
> Labels: kvm, sharedstorage, snapshot
> Fix For: 4.2.1
>
>
> With one host snapshots are created ok. After adding second host some
> snapshots fail (Failed to create snapshot due to an internal error creating
> snapshot for volume 14) stucking with state "CreatedOnPrimary". Even when all
> VMs are running on the same host.
> debug libvirt log shows:
> 2013-10-23 17:31:21.634+: 20007: debug :
> virStorageFileGetMetadataInternal:673 :
> path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
> 2013-10-23 17:32:57.189+: 20015: debug : qemuSnapObjFromName:233 : Domain
> snapshot not found: no domain snapshot with matching name
> '909848a0-b3ec-4657-a53a-c449dc24365b'
> 2013-10-23 17:32:57.474+: 20009: debug :
> virStorageFileGetMetadataInternal:673 :
> path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
> 2013-10-23 17:34:28.264+: 20008: debug : qemuSnapObjFromName:233 : Domain
> snapshot not found: no domain snapshot with matching name
> 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca'
> management server log:
> 2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd]
> (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS:
> createSnapshotCmd starts:1382549390561
> 2013-10-23 20:29:52,053 DEBUG [agent.transport.Request]
> (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq
> 6-1170407437: Waiting for Seq 1170407434 Scheduling: { Cmd , MgmtId:
> 161342718518, via: 6, Ver: v1, Flags: 100111,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/f4e51b11-ac79-4a6a-b887-8926ffbd5cca","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"snapshots/2/14/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.10.31/export/secondary","_role":"Image"}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950"
[jira] [Comment Edited] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804407#comment-13804407 ] Ivan Kozlov edited comment on CLOUDSTACK-4939 at 10/25/13 9:06 PM: --- I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in maintainance snapshots are working fine. It makes me think it's not KVM bug but ACS bug. To recreate issue: 1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over iscsi) 2. deploy 4-6 VMs and start simultaneous snapshot. setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue. snapshot files are actually created on secondary storage. there is a record in mysql cloud.snapshots, but there is no record in cloud.snapshot_store_ref. Manually updating snapshot status to "BackedUp" and adding record with path to snapshot_store_ref makes snapshot available. I was able to create template from this snapshot and deploy VM from the template. ADD: Continue researching. Finally enabled logging on agents (by the way that's another bug, need to move /etc/cloudstack/agent/log4j-cloud.xml to /etc/cloudstack/agent/log4j.xml to make logging work). I created 4 instances on 2 hosts. Performed twice backup of each VM. We are supposed to get 8 create snapshot commands, 8 backup snapshot and 8 remove snapshot. However on first host: managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 5 times managesnapshot.sh -d was run 3 times on the second host managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 3 times managesnapshot.sh -d was run 2 times so in total we get 4 create snapshot commands, 8 backup snapshot and 5 delete snapshot. it seems to be very strange. ADD2 I think I know what's going on. Management server send commands to create/backup/delete snapshots rendomly to any host. Depending on VM is running or not on the host, host uses libvirt or qemu-img. For example host1 creates snapshot using qemu-img. Command snapshot backup is sent to host2, where VM is running. host2 tries to backup snapshot using libvirt but there is no snapshot visible for that domain (because it was created with qemu-img). so backup fails. the more hosts are in cluster the more possible is snapshot failure. So we need to 1. Check if VM is running before snapshot. 2. If VM is running send all commands (create/backup/delete snapshot) only to the host where it is running. To my mind this should solve the issue. Maybe someone can do this? was (Author: eldorado): I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in maintainance snapshots are working fine. It makes me think it's not KVM bug but ACS bug. To recreate issue: 1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over iscsi) 2. deploy 4-6 VMs and start simultaneous snapshot. setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue. snapshot files are actually created on secondary storage. there is a record in mysql cloud.snapshots, but there is no record in cloud.snapshot_store_ref. Manually updating snapshot status to "BackedUp" and adding record with path to snapshot_store_ref makes snapshot available. I was able to create template from this snapshot and deploy VM from the template. ADD: Continue researching. Finally enabled logging on agents (by the way that's another bug, need to move /etc/cloudstack/agent/log4j-cloud.xml to /etc/cloudstack/agent/log4j.xml to make logging work). I created 4 instances on 2 hosts. Performed twice backup of each VM. We are supposed to get 8 create snapshot commands, 8 backup snapshot and 8 remove snapshot. However on first host: managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 5 times managesnapshot.sh -d was run 3 times on the second host managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 3 times managesnapshot.sh -d was run 2 times so in total we get 4 create snapshot commands, 8 backup snapshot and 5 delete snapshot. it seems to be very strange. > Failed to create snaphot (KVM, GFS2) > > > Key: CLOUDSTACK-4939 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Snapshot >Affects Versions: 4.2.0, 4.2.1 > Environment: CentOS 6.4, KVM, Shared mount point primary storage, > GFS2, iSCSI >Reporter: Ivan Kozlov >Priority: Blocker > Labels: kvm, sharedstorage, snapshot > Fix For: 4.2.1 > > > With one host
[jira] [Comment Edited] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804407#comment-13804407 ] Ivan Kozlov edited comment on CLOUDSTACK-4939 at 10/25/13 9:07 PM: --- I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in maintainance snapshots are working fine. It makes me think it's not KVM bug but ACS bug. To recreate issue: 1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over iscsi) 2. deploy 4-6 VMs and start simultaneous snapshot. setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue. snapshot files are actually created on secondary storage. there is a record in mysql cloud.snapshots, but there is no record in cloud.snapshot_store_ref. Manually updating snapshot status to "BackedUp" and adding record with path to snapshot_store_ref makes snapshot available. I was able to create template from this snapshot and deploy VM from the template. ADD: Continue researching. Finally enabled logging on agents (by the way that's another bug, need to move /etc/cloudstack/agent/log4j-cloud.xml to /etc/cloudstack/agent/log4j.xml to make logging work). I created 4 instances on 2 hosts. Performed twice backup of each VM. We are supposed to get 8 create snapshot commands, 8 backup snapshot and 8 remove snapshot. However on first host: managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 5 times managesnapshot.sh -d was run 3 times on the second host managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 3 times managesnapshot.sh -d was run 2 times so in total we get 4 create snapshot commands, 8 backup snapshot and 5 delete snapshot. it seems to be very strange. was (Author: eldorado): I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in maintainance snapshots are working fine. It makes me think it's not KVM bug but ACS bug. To recreate issue: 1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over iscsi) 2. deploy 4-6 VMs and start simultaneous snapshot. setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue. snapshot files are actually created on secondary storage. there is a record in mysql cloud.snapshots, but there is no record in cloud.snapshot_store_ref. Manually updating snapshot status to "BackedUp" and adding record with path to snapshot_store_ref makes snapshot available. I was able to create template from this snapshot and deploy VM from the template. ADD: Continue researching. Finally enabled logging on agents (by the way that's another bug, need to move /etc/cloudstack/agent/log4j-cloud.xml to /etc/cloudstack/agent/log4j.xml to make logging work). I created 4 instances on 2 hosts. Performed twice backup of each VM. We are supposed to get 8 create snapshot commands, 8 backup snapshot and 8 remove snapshot. However on first host: managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 5 times managesnapshot.sh -d was run 3 times on the second host managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 3 times managesnapshot.sh -d was run 2 times so in total we get 4 create snapshot commands, 8 backup snapshot and 5 delete snapshot. it seems to be very strange. ADD2 I think I know what's going on. Management server send commands to create/backup/delete snapshots rendomly to any host. Depending on VM is running or not on the host, host uses libvirt or qemu-img. For example host1 creates snapshot using qemu-img. Command snapshot backup is sent to host2, where VM is running. host2 tries to backup snapshot using libvirt but there is no snapshot visible for that domain (because it was created with qemu-img). so backup fails. the more hosts are in cluster the more possible is snapshot failure. So we need to 1. Check if VM is running before snapshot. 2. If VM is running send all commands (create/backup/delete snapshot) only to the host where it is running. To my mind this should solve the issue. Maybe someone can do this? > Failed to create snaphot (KVM, GFS2) > > > Key: CLOUDSTACK-4939 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Snapshot >Affects Versions: 4.2.0, 4.2.1 > Environment: CentOS 6.4, KVM, Shared mount point primary storage, > GFS2, iSCSI >Reporter: Ivan Kozlov >Priority: Blocker > Labels: kvm, sharedstorage, snapshot > Fix For: 4.2.1 > > > With one ho
[jira] [Commented] (CLOUDSTACK-4902) Fail to create snapshot with KVM when run multiple Hosts in Cluster
[
https://issues.apache.org/jira/browse/CLOUDSTACK-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13806686#comment-13806686
]
Ivan Kozlov commented on CLOUDSTACK-4902:
-
I think this bug may be related to
https://issues.apache.org/jira/browse/CLOUDSTACK-4939
My investigation shows that issue is caused by sending commands to random
hosts. so sometimes we send commands to host where vm is running and everything
is ok and sometimes command is sent not to host where vm is running so it fails.
Are you using sharedmountpoint or clvm as primary storage?
> Fail to create snapshot with KVM when run multiple Hosts in Cluster
> ---
>
> Key: CLOUDSTACK-4902
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4902
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM, Management Server
>Affects Versions: 4.2.0
> Environment: Managegemnet server: CentOS 6.3 64bit
> KVM Host: Ubuntu 12.04.1 64bit
>Reporter: le cong duan
>Priority: Critical
> Fix For: 4.2.0
>
>
> When only one Host in the Cluster, I always success to create snapshot for
> the volume by manual or automation.
> But when have multiple Host in Cluster. The creating is failed with status
> "CreatedOnPrimary", sometimes it is successfull.
> The following is error log on Management when occur error, there are three
> error situations.
> --> First Situation
> 2013-10-18 22:46:27,163 DEBUG [agent.transport.Request]
> (Job-Executor-122:job-120 = [ d07688b0-
> bd86-4259-bdc2-441c36c4727d ]) Seq 11-714015661: Received: { Ans: , MgmtId:
> 113353561884, via: 11,
> Ver: v1, Flags: 110, { CopyCmdAnswer } }
> 2013-10-18 22:46:27,175 DEBUG [storage.snapshot.SnapshotManagerImpl]
> (Job-Executor-122:job-120 =
> [ d07688b0-bd86-4259-bdc2-441c36c4727d ]) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException:
> org.libvirt.LibvirtException: Domain snapshot not
> found: no snapshot with matching name '8a1d6db7-9ffe-43b5-a7df-df627329a168'
> at
> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot
> (SnapshotServiceImpl.java:280)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot
> (XenserverSnapshotStrategy.java:138)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot
> (XenserverSnapshotStrategy.java:264)
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot
> (SnapshotManagerImpl.java:1013)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor
> $InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot
> (VolumeServiceImpl.java:1307)
> at
> com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719)
> at
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute
> (CreateSnapshotCmd.java:170)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-10-18 22:46:27,181 DEBUG [storage.volume.VolumeServiceImpl]
> (Job-Executor-122:job-120 = [
> d07688b0-bd86-4259-bdc2-441c36c4727d ]) Take snapshot: 12 failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
> at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot
> (SnapshotManagerImpl.java:1040)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor
> $InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot
> (VolumeServiceImpl.java:1307)
> at
> com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719)
> at
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute
> (CreateSnapshotCmd.java:170)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Exec
[jira] [Updated] (CLOUDSTACK-4939) Failed to create snapshot (KVM, Multiple hosts, Sharedstorage)
[
https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Kozlov updated CLOUDSTACK-4939:
Summary: Failed to create snapshot (KVM, Multiple hosts, Sharedstorage)
(was: Failed to create snaphot (KVM, GFS2))
> Failed to create snapshot (KVM, Multiple hosts, Sharedstorage)
> --
>
> Key: CLOUDSTACK-4939
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM, Snapshot
>Affects Versions: 4.2.0, 4.2.1
> Environment: CentOS 6.4, KVM, Shared mount point primary storage,
> GFS2, iSCSI
>Reporter: Ivan Kozlov
>Priority: Blocker
> Labels: kvm, sharedstorage, snapshot
> Fix For: 4.2.1
>
>
> With one host snapshots are created ok. After adding second host some
> snapshots fail (Failed to create snapshot due to an internal error creating
> snapshot for volume 14) stucking with state "CreatedOnPrimary". Even when all
> VMs are running on the same host.
> debug libvirt log shows:
> 2013-10-23 17:31:21.634+: 20007: debug :
> virStorageFileGetMetadataInternal:673 :
> path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
> 2013-10-23 17:32:57.189+: 20015: debug : qemuSnapObjFromName:233 : Domain
> snapshot not found: no domain snapshot with matching name
> '909848a0-b3ec-4657-a53a-c449dc24365b'
> 2013-10-23 17:32:57.474+: 20009: debug :
> virStorageFileGetMetadataInternal:673 :
> path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2
> 2013-10-23 17:34:28.264+: 20008: debug : qemuSnapObjFromName:233 : Domain
> snapshot not found: no domain snapshot with matching name
> 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca'
> management server log:
> 2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd]
> (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS:
> createSnapshotCmd starts:1382549390561
> 2013-10-23 20:29:52,053 DEBUG [agent.transport.Request]
> (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq
> 6-1170407437: Waiting for Seq 1170407434 Scheduling: { Cmd , MgmtId:
> 161342718518, via: 6, Ver: v1, Flags: 100111,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/f4e51b11-ac79-4a6a-b887-8926ffbd5cca","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"snapshots/2/14/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.10.31/export/secondary","_role":"Image"}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"executeInSequence":true,"wait":21600}}]
> }
> 2013-10-23 20:31:21,560 DEBUG [agent.transport.Request]
> (AgentManager-Handler-8:null) Seq 6-1170407434: Processing: { Ans: , MgmtId:
> 161342718518, via: 6, Ver: v1, Flags: 110,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"org.libvirt.LibvirtException:
> Domain snapshot not found: no domain snapshot with matching name
> '65113136-dfb5-4cea-8e65-1065462ca2fe'","wait":0}}] }
> 2013-10-23 20:31:21,832 DEBUG [storage.snapshot.SnapshotManagerImpl]
> (Job-Executor-49:job-91 = [ e2bf2454-4273-4a89-bc38-35add8297eb1 ]) Failed to
[jira] [Commented] (CLOUDSTACK-4902) Fail to create snapshot with KVM when run multiple Hosts in Cluster
[
https://issues.apache.org/jira/browse/CLOUDSTACK-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13808129#comment-13808129
]
Ivan Kozlov commented on CLOUDSTACK-4902:
-
I have written patch that works for me. I do not know when cloudstack
developers will get to this issue. So if you want to try it here are my diffs:
1.
diff --git
a/engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/AncientDataMotionStrategy.java
b/tmp/cloud-snapshot-patch/AncientDataMotionStrategy.java
index 4aa0147..57a17e0 100644
---
a/engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/AncientDataMotionStrategy.java
+++ b/tmp/cloud-snapshot-patch/AncientDataMotionStrategy.java
@@ -78,6 +78,12 @@ import com.cloud.utils.NumbersUtil;
import com.cloud.utils.db.DB;
import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.dao.VMInstanceDao;
+import com.cloud.vm.VMInstanceVO;
+import com.cloud.host.HostVO;
+import org.apache.cloudstack.storage.RemoteHostEndPoint;
+
@Component
public class
AncientDataMotionStrategy implements DataMotionStrategy {
@@ -120,6 +126,8 @@ public class
StorageCacheManager cacheMgr;
@Inject
ManagementService _mgmtServer;
+@Inject
+VMInstanceDao vmDao;
@Override
public boolean canHandle(DataObject srcData, DataObject destData) {
@@ -461,6 +469,24 @@ public class
} else {
CopyCommand cmd = new CopyCommand(srcData.getTO(),
destData.getTO(), _backupsnapshotwait, _mgmt
EndPoint ep = selector.select(srcData, destData);
+SnapshotInfo snapshot = null;
+try {
+ snapshot = (SnapshotInfo) srcData;
+ Long VolumeId = snapshot.getVolumeId();
+ VolumeVO vol = volDao.findById(VolumeId);
+ Long VMId = vol.getInstanceId();
+ VMInstanceVO vm = vmDao.findById(VMId);
+ if (vm.getHostId() != null) {
+ HostVO host=hostDao.findById(vm.getHostId());
+ ep =
RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(), host.getPrivateI
+ s_logger.debug("VM seems to be running on a
host. Endpoint changed to " + ep.toS
+ }
+
+} catch (Exception e) {
+ s_logger.debug("Failed to find host with VM for
snapshot: " + snapshot.getId(), e
+}
answer = ep.sendMessage(cmd);
}
@@ -489,3 +515,4 @@ public class
return null;
}
}
2.
diff --git
a/plugins/storage/volume/default/src/org/apache/cloudstack/storage/datastore/driver/CloudStackPrimaryDataStoreDriverImpl.java
b/tmp/cloud-snapshot-patch/CloudStackPrimaryDataStoreDriv
index 683239c..cc37ed2 100644
--- a/CloudStackPrimaryDataStoreDriverImpl.java
+++ b/tmp/cloud-snapshot-patch/CloudStackPrimaryDataStoreDriverImpl.java
@@ -56,6 +56,13 @@ import org.apache.cloudstack.storage.to.TemplateObjectTO;
import org.apache.cloudstack.storage.volume.VolumeObject;
import org.apache.log4j.Logger;
+import com.cloud.storage.VolumeVO;
+import com.cloud.vm.VMInstanceVO;
+import com.cloud.host.HostVO;
+import org.apache.cloudstack.storage.RemoteHostEndPoint;
+
import javax.inject.Inject;
import java.util.UUID;
@@ -233,6 +240,23 @@ public class CloudStackPrimaryDataStoreDriverImpl
implements PrimaryDataStoreDri
CreateObjectCommand cmd = new CreateObjectCommand(snapshotTO);
EndPoint ep = this.epSelector.select(snapshot);
+
+try {
+ Long VolumeId = snapshot.getVolumeId();
+ VolumeVO vol = volumeDao.findById(VolumeId);
+ Long VMId = vol.getInstanceId();
+ VMInstanceVO vm = vmDao.findById(VMId);
+ if (vm.getHostId() != null) {
+ HostVO host=hostDao.findById(vm.getHostId());
+ ep =
RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(),
host.getPrivateIpAddress
+ s_logger.debug("VM seems to be running on a host.
Endpoint changed to " + ep.toString())
+ }
+
+} catch (Exception e) {
+ s_logger.debug("Failed to find host with VM for snapshot: " +
snapshot.getId(), e);
+}
+
+
Answer answer = null;
if ( ep == null ){
String errMsg = "No remote endpoint to send DeleteCommand,
check if host or ssvm is down?";
> Fail to create snapshot with KVM when run multiple Hosts in Cluster
> ---
>
> Key: CLOUDSTACK-4902
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4902
> Project: CloudStack
> Issue Type: Bug
> Security Level:
[jira] [Commented] (CLOUDSTACK-4892) KVM snapshots are failing on CLVM
[
https://issues.apache.org/jira/browse/CLOUDSTACK-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13819136#comment-13819136
]
Ivan Kozlov commented on CLOUDSTACK-4892:
-
I think this commit should solve the problem:
https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=746896b
unfortunately I cannot check it now.
> KVM snapshots are failing on CLVM
> -
>
> Key: CLOUDSTACK-4892
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4892
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM, Snapshot
>Affects Versions: 4.2.0
> Environment: CentOS 6.4, KVM, CLVM
>Reporter: Ivan Kozlov
>Priority: Critical
>
> Creating snaphot fails hanging with state CreatedOnPrimary. Sometimes
> creating snaphot is successful.
> Snapshot logical volume is created and not deleted.
> When running snaphot with only single host snapshot is created normaly. Guess
> snapshot backup is trying access snapshot LV from host on which snapshot LV
> is not opened.
> Here is management log:
> 2013-10-18 17:32:58,512 DEBUG [cloud.async.AsyncJobManagerImpl]
> (catalina-exec-10:null) submit async job-41 = [
> 88ec27d7-78af-4664-a01b-eeca4469e37c ], details: AsyncJobVO {id:41, userId:
> 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 10,
> cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
> cmdOriginator: null, cmdInfo:
> {"id":"10","response":"json","sessionkey":"HKb50xNHyZm2wJx/IHi5S7UWBGQ\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1382106777170","volumeid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","ctxAccountId":"2","ctxStartEventId":"126"},
> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
> processStatus: 0, resultCode: 0, result: null, initMsid: 161342718518,
> completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2013-10-18 17:32:58,514 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Executing
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-41
> = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]
> 2013-10-18 17:32:58,549 INFO [user.snapshot.CreateSnapshotCmd]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) VOLSS:
> createSnapshotCmd starts:1382106778549
> 2013-10-18 17:32:58,925 DEBUG [agent.transport.Request]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq
> 1-111542657: Sending { Cmd , MgmtId: 161342718518, via: 1, Ver: v1, Flags:
> 100011,
> [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"name":"ROOT-5","size":8589934592,"path":"4f3e8cfc-d3be-4e55-bc13-5c236a689c83","volumeId":5,"vmName":"i-2-5-VM","accountId":2,"format":"RAW","id":5,"hypervisorType":"KVM"},"parentSnapshotPath":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/7e85ab28-4ea5-4b5e-8ec1-1abadf2d571e","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"vmName":"i-2-5-VM","name":"test-100_ROOT-5_20131018143258","hypervisorType":"KVM","id":10}},"wait":0}}]
> }
> 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request]
> (AgentManager-Handler-9:null) Seq 1-111542657: Processing: { Ans: , MgmtId:
> 161342718518, via: 1, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/c6c900d1-1377-4347-ba69-9ba09f264f69","id":0}},"result":true,"wait":0}}]
> }
> 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq
> 1-111542657: Received: { Ans: , MgmtId: 161342718518, via: 1, Ver: v1,
> Flags: 10, { CreateObjectAnswer } }
> 2013-10-18 17:33:00,497 DEBUG [storage.motion.AncientDataMotionStrategy]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) copyAsync
> inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT
> 2013-10-18 17:33:00,547 DEBUG [agent.transport.Request]
> (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq
> 4-1918238786: Sending { Cmd , MgmtId: 161342718518, via: 4, Ver: v1, Flags:
> 100111,
> [{"org.apache.cloudstack.s
