[jira] [Created] (CLOUDSTACK-4892) KVM snapshots are failing on CLVM
Ivan Kozlov created CLOUDSTACK-4892: --- Summary: KVM snapshots are failing on CLVM Key: CLOUDSTACK-4892 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4892 Project: CloudStack Issue Type: Bug Security Level: Public (Anyone can view this level - this is the default.) Components: KVM, Snapshot Affects Versions: 4.2.0 Environment: CentOS 6.4, KVM, CLVM Reporter: Ivan Kozlov Creating snaphot fails hanging with state CreatedOnPrimary. Sometimes creating snaphot is successful. Snapshot logical volume is created and not deleted. When running snaphot with only single host snapshot is created normaly. Guess snapshot backup is trying access snapshot LV from host on which snapshot LV is not opened. Here is management log: 2013-10-18 17:32:58,512 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-10:null) submit async job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ], details: AsyncJobVO {id:41, userId: 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 10, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdOriginator: null, cmdInfo: {"id":"10","response":"json","sessionkey":"HKb50xNHyZm2wJx/IHi5S7UWBGQ\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1382106777170","volumeid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","ctxAccountId":"2","ctxStartEventId":"126"}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsid: 161342718518, completeMsid: null, lastUpdated: null, lastPolled: null, created: null} 2013-10-18 17:32:58,514 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ] 2013-10-18 17:32:58,549 INFO [user.snapshot.CreateSnapshotCmd] (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) VOLSS: createSnapshotCmd starts:1382106778549 2013-10-18 17:32:58,925 DEBUG [agent.transport.Request] (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq 1-111542657: Sending { Cmd , MgmtId: 161342718518, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"name":"ROOT-5","size":8589934592,"path":"4f3e8cfc-d3be-4e55-bc13-5c236a689c83","volumeId":5,"vmName":"i-2-5-VM","accountId":2,"format":"RAW","id":5,"hypervisorType":"KVM"},"parentSnapshotPath":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/7e85ab28-4ea5-4b5e-8ec1-1abadf2d571e","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"vmName":"i-2-5-VM","name":"test-100_ROOT-5_20131018143258","hypervisorType":"KVM","id":10}},"wait":0}}] } 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request] (AgentManager-Handler-9:null) Seq 1-111542657: Processing: { Ans: , MgmtId: 161342718518, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/c6c900d1-1377-4347-ba69-9ba09f264f69","id":0}},"result":true,"wait":0}}] } 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request] (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq 1-111542657: Received: { Ans: , MgmtId: 161342718518, via: 1, Ver: v1, Flags: 10, { CreateObjectAnswer } } 2013-10-18 17:33:00,497 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) copyAsync inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT 2013-10-18 17:33:00,547 DEBUG [agent.transport.Request] (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq 4-1918238786: Sending { Cmd , MgmtId: 161342718518, via: 4, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/c6c900d1-1377-4347-ba69-9ba09f264f69","volume":{"uuid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"name":"ROOT-5","size":8589934592,"path":
[jira] [Created] (CLOUDSTACK-4893) Documentation about Security Groups and OVS
Ivan Kozlov created CLOUDSTACK-4893: --- Summary: Documentation about Security Groups and OVS Key: CLOUDSTACK-4893 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4893 Project: CloudStack Issue Type: Improvement Security Level: Public (Anyone can view this level - this is the default.) Components: Doc Environment: KVM, OVS Reporter: Ivan Kozlov Priority: Trivial In Installation Guide (8.1. KVM Hypervisor Host Installation - 8.1.8. Configure the network using OpenVswitch) nothing is said about compatability of Security Groups and OVS. I think it should be mentioned. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CLOUDSTACK-4892) KVM snapshots are failing on CLVM
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799313#comment-13799313 ] Ivan Kozlov commented on CLOUDSTACK-4892: - managesnapshot.sh checks snapshot with the following command: if ! dmsetup info -c --noheadings -o name ${vg_dm}-${snapshotname} > /dev/null 2>&1; then it is successful on one host (i think host that created snaphot on primary) dmsetup info -c --noheadings -o name vg_primary-7ce7c39f59b6391a287a19906241060d vg_primary-7ce7c39f59b6391a287a19906241060d and fails on others dmsetup info -c --noheadings -o name vg_primary-7ce7c39f59b6391a287a19906241060d Device does not exist. Command failed However on every host vg_primary-7ce7c39f59b6391a287a19906241060d--cow is present > KVM snapshots are failing on CLVM > - > > Key: CLOUDSTACK-4892 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4892 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Snapshot >Affects Versions: 4.2.0 > Environment: CentOS 6.4, KVM, CLVM >Reporter: Ivan Kozlov > > Creating snaphot fails hanging with state CreatedOnPrimary. Sometimes > creating snaphot is successful. > Snapshot logical volume is created and not deleted. > When running snaphot with only single host snapshot is created normaly. Guess > snapshot backup is trying access snapshot LV from host on which snapshot LV > is not opened. > Here is management log: > 2013-10-18 17:32:58,512 DEBUG [cloud.async.AsyncJobManagerImpl] > (catalina-exec-10:null) submit async job-41 = [ > 88ec27d7-78af-4664-a01b-eeca4469e37c ], details: AsyncJobVO {id:41, userId: > 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 10, > cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, > cmdOriginator: null, cmdInfo: > {"id":"10","response":"json","sessionkey":"HKb50xNHyZm2wJx/IHi5S7UWBGQ\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1382106777170","volumeid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","ctxAccountId":"2","ctxStartEventId":"126"}, > cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, > processStatus: 0, resultCode: 0, result: null, initMsid: 161342718518, > completeMsid: null, lastUpdated: null, lastPolled: null, created: null} > 2013-10-18 17:32:58,514 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Executing > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-41 > = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ] > 2013-10-18 17:32:58,549 INFO [user.snapshot.CreateSnapshotCmd] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) VOLSS: > createSnapshotCmd starts:1382106778549 > 2013-10-18 17:32:58,925 DEBUG [agent.transport.Request] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq > 1-111542657: Sending { Cmd , MgmtId: 161342718518, via: 1, Ver: v1, Flags: > 100011, > [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"name":"ROOT-5","size":8589934592,"path":"4f3e8cfc-d3be-4e55-bc13-5c236a689c83","volumeId":5,"vmName":"i-2-5-VM","accountId":2,"format":"RAW","id":5,"hypervisorType":"KVM"},"parentSnapshotPath":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/7e85ab28-4ea5-4b5e-8ec1-1abadf2d571e","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"vmName":"i-2-5-VM","name":"test-100_ROOT-5_20131018143258","hypervisorType":"KVM","id":10}},"wait":0}}] > } > 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request] > (AgentManager-Handler-9:null) Seq 1-111542657: Processing: { Ans: , MgmtId: > 161342718518, via: 1, Ver: v1, Flags: 10, > [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/c6c900d1-1377-4347-ba69-9ba09f264f69","id":0}},"result":true,"wait":0}}] > } > 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq > 1-111542657: Received: { Ans: , MgmtId: 161342718518, via: 1, Ver: v1, > Flags: 10, { CreateObjectAnswer } } > 2013-10-18 17:33:00,497 DEBUG [storage.motion.AncientDataMotionStrategy] > (Job-
[jira] [Created] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
Ivan Kozlov created CLOUDSTACK-4939: --- Summary: Failed to create snaphot (KVM, GFS2) Key: CLOUDSTACK-4939 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939 Project: CloudStack Issue Type: Bug Security Level: Public (Anyone can view this level - this is the default.) Components: KVM, Snapshot Affects Versions: 4.2.0 Environment: CentOS 6.4, KVM, Shared mount point primary storage, GFS2, iSCSI Reporter: Ivan Kozlov Priority: Blocker With one host snapshots are created ok. After adding second host some snapshots fail (Failed to create snapshot due to an internal error creating snapshot for volume 14) stucking with state "CreatedOnPrimary". Even when all VMs are running on the same host. debug libvirt log shows: 2013-10-23 17:31:21.634+: 20007: debug : virStorageFileGetMetadataInternal:673 : path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 2013-10-23 17:32:57.189+: 20015: debug : qemuSnapObjFromName:233 : Domain snapshot not found: no domain snapshot with matching name '909848a0-b3ec-4657-a53a-c449dc24365b' 2013-10-23 17:32:57.474+: 20009: debug : virStorageFileGetMetadataInternal:673 : path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 2013-10-23 17:34:28.264+: 20008: debug : qemuSnapObjFromName:233 : Domain snapshot not found: no domain snapshot with matching name 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca' management server log: 2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd] (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS: createSnapshotCmd starts:1382549390561 2013-10-23 20:29:52,053 DEBUG [agent.transport.Request] (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq 6-1170407437: Waiting for Seq 1170407434 Scheduling: { Cmd , MgmtId: 161342718518, via: 6, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/f4e51b11-ac79-4a6a-b887-8926ffbd5cca","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"snapshots/2/14/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.10.31/export/secondary","_role":"Image"}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"executeInSequence":true,"wait":21600}}] } 2013-10-23 20:31:21,560 DEBUG [agent.transport.Request] (AgentManager-Handler-8:null) Seq 6-1170407434: Processing: { Ans: , MgmtId: 161342718518, via: 6, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot with matching name '65113136-dfb5-4cea-8e65-1065462ca2fe'","wait":0}}] } 2013-10-23 20:31:21,832 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-49:job-91 = [ e2bf2454-4273-4a89-bc38-35add8297eb1 ]) Failed to create snapshot com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot with matching name '65113136-dfb5-4cea-8e65-1065462ca2fe' at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138) at org.apache.cloudstack.storage.snapshot.X
[jira] [Updated] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kozlov updated CLOUDSTACK-4939: Affects Version/s: 4.2.1 > Failed to create snaphot (KVM, GFS2) > > > Key: CLOUDSTACK-4939 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Snapshot >Affects Versions: 4.2.0, 4.2.1 > Environment: CentOS 6.4, KVM, Shared mount point primary storage, > GFS2, iSCSI >Reporter: Ivan Kozlov >Priority: Blocker > Labels: kvm, sharedstorage, snapshot > > With one host snapshots are created ok. After adding second host some > snapshots fail (Failed to create snapshot due to an internal error creating > snapshot for volume 14) stucking with state "CreatedOnPrimary". Even when all > VMs are running on the same host. > debug libvirt log shows: > 2013-10-23 17:31:21.634+: 20007: debug : > virStorageFileGetMetadataInternal:673 : > path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 > 2013-10-23 17:32:57.189+: 20015: debug : qemuSnapObjFromName:233 : Domain > snapshot not found: no domain snapshot with matching name > '909848a0-b3ec-4657-a53a-c449dc24365b' > 2013-10-23 17:32:57.474+: 20009: debug : > virStorageFileGetMetadataInternal:673 : > path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 > 2013-10-23 17:34:28.264+: 20008: debug : qemuSnapObjFromName:233 : Domain > snapshot not found: no domain snapshot with matching name > 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca' > management server log: > 2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS: > createSnapshotCmd starts:1382549390561 > 2013-10-23 20:29:52,053 DEBUG [agent.transport.Request] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq > 6-1170407437: Waiting for Seq 1170407434 Scheduling: { Cmd , MgmtId: > 161342718518, via: 6, Ver: v1, Flags: 100111, > [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/f4e51b11-ac79-4a6a-b887-8926ffbd5cca","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"snapshots/2/14/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.10.31/export/secondary","_role":"Image"}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"executeInSequence":true,"wait":21600}}] > } > 2013-10-23 20:31:21,560 DEBUG [agent.transport.Request] > (AgentManager-Handler-8:null) Seq 6-1170407434: Processing: { Ans: , MgmtId: > 161342718518, via: 6, Ver: v1, Flags: 110, > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"org.libvirt.LibvirtException: > Domain snapshot not found: no domain snapshot with matching name > '65113136-dfb5-4cea-8e65-1065462ca2fe'","wait":0}}] } > 2013-10-23 20:31:21,832 DEBUG [storage.snapshot.SnapshotManagerImpl] > (Job-Executor-49:job-91 = [ e2bf2454-4273-4a89-bc38-35add8297eb1 ]) Failed to > create snapshot > com.cloud.utils.exception.CloudRuntimeException: > org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot > with matching name '651
[jira] [Commented] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804407#comment-13804407 ] Ivan Kozlov commented on CLOUDSTACK-4939: - I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in maintainance snapshots are working fine. It makes me think it's not KVM bug but ACS bug. To recreate issue: 1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over iscsi) 2. deploy 4-6 VMs and start simultaneous snapshot. setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue. > Failed to create snaphot (KVM, GFS2) > > > Key: CLOUDSTACK-4939 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Snapshot >Affects Versions: 4.2.0, 4.2.1 > Environment: CentOS 6.4, KVM, Shared mount point primary storage, > GFS2, iSCSI >Reporter: Ivan Kozlov >Priority: Blocker > Labels: kvm, sharedstorage, snapshot > > With one host snapshots are created ok. After adding second host some > snapshots fail (Failed to create snapshot due to an internal error creating > snapshot for volume 14) stucking with state "CreatedOnPrimary". Even when all > VMs are running on the same host. > debug libvirt log shows: > 2013-10-23 17:31:21.634+: 20007: debug : > virStorageFileGetMetadataInternal:673 : > path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 > 2013-10-23 17:32:57.189+: 20015: debug : qemuSnapObjFromName:233 : Domain > snapshot not found: no domain snapshot with matching name > '909848a0-b3ec-4657-a53a-c449dc24365b' > 2013-10-23 17:32:57.474+: 20009: debug : > virStorageFileGetMetadataInternal:673 : > path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 > 2013-10-23 17:34:28.264+: 20008: debug : qemuSnapObjFromName:233 : Domain > snapshot not found: no domain snapshot with matching name > 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca' > management server log: > 2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS: > createSnapshotCmd starts:1382549390561 > 2013-10-23 20:29:52,053 DEBUG [agent.transport.Request] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq > 6-1170407437: Waiting for Seq 1170407434 Scheduling: { Cmd , MgmtId: > 161342718518, via: 6, Ver: v1, Flags: 100111, > [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/f4e51b11-ac79-4a6a-b887-8926ffbd5cca","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"snapshots/2/14/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.10.31/export/secondary","_role":"Image"}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"executeInSequence":true,"wait":21600}}] > } > 2013-10-23 20:31:21,560 DEBUG [agent.transport.Request] > (AgentManager-Handler-8:null) Seq 6-1170407434: Processing: { Ans: , MgmtId: > 161342718518, via: 6, Ver: v1, Flags: 110, > [{"org.apache.cloudstack.storage.command.CopyCmdAnswe
[jira] [Updated] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kozlov updated CLOUDSTACK-4939: Fix Version/s: 4.2.1 > Failed to create snaphot (KVM, GFS2) > > > Key: CLOUDSTACK-4939 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Snapshot >Affects Versions: 4.2.0, 4.2.1 > Environment: CentOS 6.4, KVM, Shared mount point primary storage, > GFS2, iSCSI >Reporter: Ivan Kozlov >Priority: Blocker > Labels: kvm, sharedstorage, snapshot > Fix For: 4.2.1 > > > With one host snapshots are created ok. After adding second host some > snapshots fail (Failed to create snapshot due to an internal error creating > snapshot for volume 14) stucking with state "CreatedOnPrimary". Even when all > VMs are running on the same host. > debug libvirt log shows: > 2013-10-23 17:31:21.634+: 20007: debug : > virStorageFileGetMetadataInternal:673 : > path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 > 2013-10-23 17:32:57.189+: 20015: debug : qemuSnapObjFromName:233 : Domain > snapshot not found: no domain snapshot with matching name > '909848a0-b3ec-4657-a53a-c449dc24365b' > 2013-10-23 17:32:57.474+: 20009: debug : > virStorageFileGetMetadataInternal:673 : > path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 > 2013-10-23 17:34:28.264+: 20008: debug : qemuSnapObjFromName:233 : Domain > snapshot not found: no domain snapshot with matching name > 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca' > management server log: > 2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS: > createSnapshotCmd starts:1382549390561 > 2013-10-23 20:29:52,053 DEBUG [agent.transport.Request] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq > 6-1170407437: Waiting for Seq 1170407434 Scheduling: { Cmd , MgmtId: > 161342718518, via: 6, Ver: v1, Flags: 100111, > [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/f4e51b11-ac79-4a6a-b887-8926ffbd5cca","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"snapshots/2/14/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.10.31/export/secondary","_role":"Image"}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"executeInSequence":true,"wait":21600}}] > } > 2013-10-23 20:31:21,560 DEBUG [agent.transport.Request] > (AgentManager-Handler-8:null) Seq 6-1170407434: Processing: { Ans: , MgmtId: > 161342718518, via: 6, Ver: v1, Flags: 110, > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"org.libvirt.LibvirtException: > Domain snapshot not found: no domain snapshot with matching name > '65113136-dfb5-4cea-8e65-1065462ca2fe'","wait":0}}] } > 2013-10-23 20:31:21,832 DEBUG [storage.snapshot.SnapshotManagerImpl] > (Job-Executor-49:job-91 = [ e2bf2454-4273-4a89-bc38-35add8297eb1 ]) Failed to > create snapshot > com.cloud.utils.exception.CloudRuntimeException: > org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot
[jira] [Comment Edited] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804407#comment-13804407 ] Ivan Kozlov edited comment on CLOUDSTACK-4939 at 10/24/13 7:15 PM: --- I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in maintainance snapshots are working fine. It makes me think it's not KVM bug but ACS bug. To recreate issue: 1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over iscsi) 2. deploy 4-6 VMs and start simultaneous snapshot. setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue. snapshot files are actually created on secondary storage. there is a record in mysql cloud.snapshots, but there is no record in cloud.snapshot_store_ref. Manually updating snapshot status to "BackedUp" and adding record with path to snapshot_store_ref makes snapshot available. I was able to create template from this snapshot and deploy VM from the template. was (Author: eldorado): I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in maintainance snapshots are working fine. It makes me think it's not KVM bug but ACS bug. To recreate issue: 1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over iscsi) 2. deploy 4-6 VMs and start simultaneous snapshot. setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue. > Failed to create snaphot (KVM, GFS2) > > > Key: CLOUDSTACK-4939 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Snapshot >Affects Versions: 4.2.0, 4.2.1 > Environment: CentOS 6.4, KVM, Shared mount point primary storage, > GFS2, iSCSI >Reporter: Ivan Kozlov >Priority: Blocker > Labels: kvm, sharedstorage, snapshot > Fix For: 4.2.1 > > > With one host snapshots are created ok. After adding second host some > snapshots fail (Failed to create snapshot due to an internal error creating > snapshot for volume 14) stucking with state "CreatedOnPrimary". Even when all > VMs are running on the same host. > debug libvirt log shows: > 2013-10-23 17:31:21.634+: 20007: debug : > virStorageFileGetMetadataInternal:673 : > path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 > 2013-10-23 17:32:57.189+: 20015: debug : qemuSnapObjFromName:233 : Domain > snapshot not found: no domain snapshot with matching name > '909848a0-b3ec-4657-a53a-c449dc24365b' > 2013-10-23 17:32:57.474+: 20009: debug : > virStorageFileGetMetadataInternal:673 : > path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 > 2013-10-23 17:34:28.264+: 20008: debug : qemuSnapObjFromName:233 : Domain > snapshot not found: no domain snapshot with matching name > 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca' > management server log: > 2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS: > createSnapshotCmd starts:1382549390561 > 2013-10-23 20:29:52,053 DEBUG [agent.transport.Request] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq > 6-1170407437: Waiting for Seq 1170407434 Scheduling: { Cmd , MgmtId: > 161342718518, via: 6, Ver: v1, Flags: 100111, > [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/f4e51b11-ac79-4a6a-b887-8926ffbd5cca","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","
[jira] [Comment Edited] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804407#comment-13804407 ] Ivan Kozlov edited comment on CLOUDSTACK-4939 at 10/25/13 2:13 PM: --- I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in maintainance snapshots are working fine. It makes me think it's not KVM bug but ACS bug. To recreate issue: 1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over iscsi) 2. deploy 4-6 VMs and start simultaneous snapshot. setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue. snapshot files are actually created on secondary storage. there is a record in mysql cloud.snapshots, but there is no record in cloud.snapshot_store_ref. Manually updating snapshot status to "BackedUp" and adding record with path to snapshot_store_ref makes snapshot available. I was able to create template from this snapshot and deploy VM from the template. ADD: Continue researching. Finally enabled logging on agents (by the way that's another bug, need to move /etc/cloudstack/agent/log4j-cloud.xml to /etc/cloudstack/agent/log4j.xml to make logging work). I created 4 instances on 2 hosts. Performed twice backup of each VM. We are supposed to get 8 create snapshot commands, 8 backup snapshot and 8 remove snapshot. However on first host: managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 5 times managesnapshot.sh -d was run 3 times on the second host managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 3 times managesnapshot.sh -d was run 2 times so in total we get 4 create snapshot commands, 8 backup snapshot and 5 delete snapshot. it seems to be very strange. was (Author: eldorado): I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in maintainance snapshots are working fine. It makes me think it's not KVM bug but ACS bug. To recreate issue: 1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over iscsi) 2. deploy 4-6 VMs and start simultaneous snapshot. setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue. snapshot files are actually created on secondary storage. there is a record in mysql cloud.snapshots, but there is no record in cloud.snapshot_store_ref. Manually updating snapshot status to "BackedUp" and adding record with path to snapshot_store_ref makes snapshot available. I was able to create template from this snapshot and deploy VM from the template. > Failed to create snaphot (KVM, GFS2) > > > Key: CLOUDSTACK-4939 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Snapshot >Affects Versions: 4.2.0, 4.2.1 > Environment: CentOS 6.4, KVM, Shared mount point primary storage, > GFS2, iSCSI >Reporter: Ivan Kozlov >Priority: Blocker > Labels: kvm, sharedstorage, snapshot > Fix For: 4.2.1 > > > With one host snapshots are created ok. After adding second host some > snapshots fail (Failed to create snapshot due to an internal error creating > snapshot for volume 14) stucking with state "CreatedOnPrimary". Even when all > VMs are running on the same host. > debug libvirt log shows: > 2013-10-23 17:31:21.634+: 20007: debug : > virStorageFileGetMetadataInternal:673 : > path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 > 2013-10-23 17:32:57.189+: 20015: debug : qemuSnapObjFromName:233 : Domain > snapshot not found: no domain snapshot with matching name > '909848a0-b3ec-4657-a53a-c449dc24365b' > 2013-10-23 17:32:57.474+: 20009: debug : > virStorageFileGetMetadataInternal:673 : > path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 > 2013-10-23 17:34:28.264+: 20008: debug : qemuSnapObjFromName:233 : Domain > snapshot not found: no domain snapshot with matching name > 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca' > management server log: > 2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS: > createSnapshotCmd starts:1382549390561 > 2013-10-23 20:29:52,053 DEBUG [agent.transport.Request] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq > 6-1170407437: Waiting for Seq 1170407434 Scheduling: { Cmd , MgmtId: > 161342718518, via: 6, Ver: v1, Flags: 100111, > [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudst
[jira] [Commented] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805701#comment-13805701 ] Ivan Kozlov commented on CLOUDSTACK-4939: - I think I know what's going on. Management server send commands to create/backup/delete snapshots randomly to any host. Depending on VM is running or not on the host, host uses libvirt or qemu-img. For example host1 creates snapshot using qemu-img. Command snapshot backup is sent to host2, where VM is running. host2 tries to backup snapshot using libvirt but there is no snapshot visible for that domain (because it was created with qemu-img). so backup fails. the more hosts are in cluster the more possible is snapshot failure. So we need to 1. Check if VM is running before snapshot. 2. If VM is running send all commands (create/backup/delete snapshot) only to the host where it is running. To my mind this should solve the issue. Maybe someone can do this? > Failed to create snaphot (KVM, GFS2) > > > Key: CLOUDSTACK-4939 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Snapshot >Affects Versions: 4.2.0, 4.2.1 > Environment: CentOS 6.4, KVM, Shared mount point primary storage, > GFS2, iSCSI >Reporter: Ivan Kozlov >Priority: Blocker > Labels: kvm, sharedstorage, snapshot > Fix For: 4.2.1 > > > With one host snapshots are created ok. After adding second host some > snapshots fail (Failed to create snapshot due to an internal error creating > snapshot for volume 14) stucking with state "CreatedOnPrimary". Even when all > VMs are running on the same host. > debug libvirt log shows: > 2013-10-23 17:31:21.634+: 20007: debug : > virStorageFileGetMetadataInternal:673 : > path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 > 2013-10-23 17:32:57.189+: 20015: debug : qemuSnapObjFromName:233 : Domain > snapshot not found: no domain snapshot with matching name > '909848a0-b3ec-4657-a53a-c449dc24365b' > 2013-10-23 17:32:57.474+: 20009: debug : > virStorageFileGetMetadataInternal:673 : > path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 > 2013-10-23 17:34:28.264+: 20008: debug : qemuSnapObjFromName:233 : Domain > snapshot not found: no domain snapshot with matching name > 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca' > management server log: > 2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS: > createSnapshotCmd starts:1382549390561 > 2013-10-23 20:29:52,053 DEBUG [agent.transport.Request] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq > 6-1170407437: Waiting for Seq 1170407434 Scheduling: { Cmd , MgmtId: > 161342718518, via: 6, Ver: v1, Flags: 100111, > [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/f4e51b11-ac79-4a6a-b887-8926ffbd5cca","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"snapshots/2/14/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.10.31/export/secondary","_role":"Image"}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950"
[jira] [Comment Edited] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804407#comment-13804407 ] Ivan Kozlov edited comment on CLOUDSTACK-4939 at 10/25/13 9:06 PM: --- I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in maintainance snapshots are working fine. It makes me think it's not KVM bug but ACS bug. To recreate issue: 1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over iscsi) 2. deploy 4-6 VMs and start simultaneous snapshot. setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue. snapshot files are actually created on secondary storage. there is a record in mysql cloud.snapshots, but there is no record in cloud.snapshot_store_ref. Manually updating snapshot status to "BackedUp" and adding record with path to snapshot_store_ref makes snapshot available. I was able to create template from this snapshot and deploy VM from the template. ADD: Continue researching. Finally enabled logging on agents (by the way that's another bug, need to move /etc/cloudstack/agent/log4j-cloud.xml to /etc/cloudstack/agent/log4j.xml to make logging work). I created 4 instances on 2 hosts. Performed twice backup of each VM. We are supposed to get 8 create snapshot commands, 8 backup snapshot and 8 remove snapshot. However on first host: managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 5 times managesnapshot.sh -d was run 3 times on the second host managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 3 times managesnapshot.sh -d was run 2 times so in total we get 4 create snapshot commands, 8 backup snapshot and 5 delete snapshot. it seems to be very strange. ADD2 I think I know what's going on. Management server send commands to create/backup/delete snapshots rendomly to any host. Depending on VM is running or not on the host, host uses libvirt or qemu-img. For example host1 creates snapshot using qemu-img. Command snapshot backup is sent to host2, where VM is running. host2 tries to backup snapshot using libvirt but there is no snapshot visible for that domain (because it was created with qemu-img). so backup fails. the more hosts are in cluster the more possible is snapshot failure. So we need to 1. Check if VM is running before snapshot. 2. If VM is running send all commands (create/backup/delete snapshot) only to the host where it is running. To my mind this should solve the issue. Maybe someone can do this? was (Author: eldorado): I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in maintainance snapshots are working fine. It makes me think it's not KVM bug but ACS bug. To recreate issue: 1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over iscsi) 2. deploy 4-6 VMs and start simultaneous snapshot. setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue. snapshot files are actually created on secondary storage. there is a record in mysql cloud.snapshots, but there is no record in cloud.snapshot_store_ref. Manually updating snapshot status to "BackedUp" and adding record with path to snapshot_store_ref makes snapshot available. I was able to create template from this snapshot and deploy VM from the template. ADD: Continue researching. Finally enabled logging on agents (by the way that's another bug, need to move /etc/cloudstack/agent/log4j-cloud.xml to /etc/cloudstack/agent/log4j.xml to make logging work). I created 4 instances on 2 hosts. Performed twice backup of each VM. We are supposed to get 8 create snapshot commands, 8 backup snapshot and 8 remove snapshot. However on first host: managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 5 times managesnapshot.sh -d was run 3 times on the second host managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 3 times managesnapshot.sh -d was run 2 times so in total we get 4 create snapshot commands, 8 backup snapshot and 5 delete snapshot. it seems to be very strange. > Failed to create snaphot (KVM, GFS2) > > > Key: CLOUDSTACK-4939 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Snapshot >Affects Versions: 4.2.0, 4.2.1 > Environment: CentOS 6.4, KVM, Shared mount point primary storage, > GFS2, iSCSI >Reporter: Ivan Kozlov >Priority: Blocker > Labels: kvm, sharedstorage, snapshot > Fix For: 4.2.1 > > > With one host
[jira] [Comment Edited] (CLOUDSTACK-4939) Failed to create snaphot (KVM, GFS2)
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804407#comment-13804407 ] Ivan Kozlov edited comment on CLOUDSTACK-4939 at 10/25/13 9:07 PM: --- I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in maintainance snapshots are working fine. It makes me think it's not KVM bug but ACS bug. To recreate issue: 1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over iscsi) 2. deploy 4-6 VMs and start simultaneous snapshot. setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue. snapshot files are actually created on secondary storage. there is a record in mysql cloud.snapshots, but there is no record in cloud.snapshot_store_ref. Manually updating snapshot status to "BackedUp" and adding record with path to snapshot_store_ref makes snapshot available. I was able to create template from this snapshot and deploy VM from the template. ADD: Continue researching. Finally enabled logging on agents (by the way that's another bug, need to move /etc/cloudstack/agent/log4j-cloud.xml to /etc/cloudstack/agent/log4j.xml to make logging work). I created 4 instances on 2 hosts. Performed twice backup of each VM. We are supposed to get 8 create snapshot commands, 8 backup snapshot and 8 remove snapshot. However on first host: managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 5 times managesnapshot.sh -d was run 3 times on the second host managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 3 times managesnapshot.sh -d was run 2 times so in total we get 4 create snapshot commands, 8 backup snapshot and 5 delete snapshot. it seems to be very strange. was (Author: eldorado): I have checked this with ocfs2 and with build from ACS 4.2.1. Snapshots still failing. 4.2.1 is affected. NFS seems to work fine. When one of two hosts is in maintainance snapshots are working fine. It makes me think it's not KVM bug but ACS bug. To recreate issue: 1. set up cloudstack with 2 or more hosts and sharedstorage (gfs2/ocfs2 over iscsi) 2. deploy 4-6 VMs and start simultaneous snapshot. setting concurrent.snapshots.threshold.perhost = 1 doesn't solves the issue. snapshot files are actually created on secondary storage. there is a record in mysql cloud.snapshots, but there is no record in cloud.snapshot_store_ref. Manually updating snapshot status to "BackedUp" and adding record with path to snapshot_store_ref makes snapshot available. I was able to create template from this snapshot and deploy VM from the template. ADD: Continue researching. Finally enabled logging on agents (by the way that's another bug, need to move /etc/cloudstack/agent/log4j-cloud.xml to /etc/cloudstack/agent/log4j.xml to make logging work). I created 4 instances on 2 hosts. Performed twice backup of each VM. We are supposed to get 8 create snapshot commands, 8 backup snapshot and 8 remove snapshot. However on first host: managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 5 times managesnapshot.sh -d was run 3 times on the second host managesnapshot.sh -c was run 2 times managesnapshot.sh -b was run 3 times managesnapshot.sh -d was run 2 times so in total we get 4 create snapshot commands, 8 backup snapshot and 5 delete snapshot. it seems to be very strange. ADD2 I think I know what's going on. Management server send commands to create/backup/delete snapshots rendomly to any host. Depending on VM is running or not on the host, host uses libvirt or qemu-img. For example host1 creates snapshot using qemu-img. Command snapshot backup is sent to host2, where VM is running. host2 tries to backup snapshot using libvirt but there is no snapshot visible for that domain (because it was created with qemu-img). so backup fails. the more hosts are in cluster the more possible is snapshot failure. So we need to 1. Check if VM is running before snapshot. 2. If VM is running send all commands (create/backup/delete snapshot) only to the host where it is running. To my mind this should solve the issue. Maybe someone can do this? > Failed to create snaphot (KVM, GFS2) > > > Key: CLOUDSTACK-4939 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Snapshot >Affects Versions: 4.2.0, 4.2.1 > Environment: CentOS 6.4, KVM, Shared mount point primary storage, > GFS2, iSCSI >Reporter: Ivan Kozlov >Priority: Blocker > Labels: kvm, sharedstorage, snapshot > Fix For: 4.2.1 > > > With one ho
[jira] [Commented] (CLOUDSTACK-4902) Fail to create snapshot with KVM when run multiple Hosts in Cluster
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13806686#comment-13806686 ] Ivan Kozlov commented on CLOUDSTACK-4902: - I think this bug may be related to https://issues.apache.org/jira/browse/CLOUDSTACK-4939 My investigation shows that issue is caused by sending commands to random hosts. so sometimes we send commands to host where vm is running and everything is ok and sometimes command is sent not to host where vm is running so it fails. Are you using sharedmountpoint or clvm as primary storage? > Fail to create snapshot with KVM when run multiple Hosts in Cluster > --- > > Key: CLOUDSTACK-4902 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4902 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Management Server >Affects Versions: 4.2.0 > Environment: Managegemnet server: CentOS 6.3 64bit > KVM Host: Ubuntu 12.04.1 64bit >Reporter: le cong duan >Priority: Critical > Fix For: 4.2.0 > > > When only one Host in the Cluster, I always success to create snapshot for > the volume by manual or automation. > But when have multiple Host in Cluster. The creating is failed with status > "CreatedOnPrimary", sometimes it is successfull. > The following is error log on Management when occur error, there are three > error situations. > --> First Situation > 2013-10-18 22:46:27,163 DEBUG [agent.transport.Request] > (Job-Executor-122:job-120 = [ d07688b0- > bd86-4259-bdc2-441c36c4727d ]) Seq 11-714015661: Received: { Ans: , MgmtId: > 113353561884, via: 11, > Ver: v1, Flags: 110, { CopyCmdAnswer } } > 2013-10-18 22:46:27,175 DEBUG [storage.snapshot.SnapshotManagerImpl] > (Job-Executor-122:job-120 = > [ d07688b0-bd86-4259-bdc2-441c36c4727d ]) Failed to create snapshot > com.cloud.utils.exception.CloudRuntimeException: > org.libvirt.LibvirtException: Domain snapshot not > found: no snapshot with matching name '8a1d6db7-9ffe-43b5-a7df-df627329a168' > at > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot > (SnapshotServiceImpl.java:280) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot > (XenserverSnapshotStrategy.java:138) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot > (XenserverSnapshotStrategy.java:264) > at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot > (SnapshotManagerImpl.java:1013) > at com.cloud.utils.component.ComponentInstantiationPostProcessor > $InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot > (VolumeServiceImpl.java:1307) > at > com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719) > at > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute > (CreateSnapshotCmd.java:170) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-10-18 22:46:27,181 DEBUG [storage.volume.VolumeServiceImpl] > (Job-Executor-122:job-120 = [ > d07688b0-bd86-4259-bdc2-441c36c4727d ]) Take snapshot: 12 failed > com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot > at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot > (SnapshotManagerImpl.java:1040) > at com.cloud.utils.component.ComponentInstantiationPostProcessor > $InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot > (VolumeServiceImpl.java:1307) > at > com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2719) > at > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute > (CreateSnapshotCmd.java:170) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) > at > java.util.concurrent.Executors$RunnableAdapter.call(Exec
[jira] [Updated] (CLOUDSTACK-4939) Failed to create snapshot (KVM, Multiple hosts, Sharedstorage)
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kozlov updated CLOUDSTACK-4939: Summary: Failed to create snapshot (KVM, Multiple hosts, Sharedstorage) (was: Failed to create snaphot (KVM, GFS2)) > Failed to create snapshot (KVM, Multiple hosts, Sharedstorage) > -- > > Key: CLOUDSTACK-4939 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Snapshot >Affects Versions: 4.2.0, 4.2.1 > Environment: CentOS 6.4, KVM, Shared mount point primary storage, > GFS2, iSCSI >Reporter: Ivan Kozlov >Priority: Blocker > Labels: kvm, sharedstorage, snapshot > Fix For: 4.2.1 > > > With one host snapshots are created ok. After adding second host some > snapshots fail (Failed to create snapshot due to an internal error creating > snapshot for volume 14) stucking with state "CreatedOnPrimary". Even when all > VMs are running on the same host. > debug libvirt log shows: > 2013-10-23 17:31:21.634+: 20007: debug : > virStorageFileGetMetadataInternal:673 : > path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 > 2013-10-23 17:32:57.189+: 20015: debug : qemuSnapObjFromName:233 : Domain > snapshot not found: no domain snapshot with matching name > '909848a0-b3ec-4657-a53a-c449dc24365b' > 2013-10-23 17:32:57.474+: 20009: debug : > virStorageFileGetMetadataInternal:673 : > path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 > 2013-10-23 17:34:28.264+: 20008: debug : qemuSnapObjFromName:233 : Domain > snapshot not found: no domain snapshot with matching name > 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca' > management server log: > 2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS: > createSnapshotCmd starts:1382549390561 > 2013-10-23 20:29:52,053 DEBUG [agent.transport.Request] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq > 6-1170407437: Waiting for Seq 1170407434 Scheduling: { Cmd , MgmtId: > 161342718518, via: 6, Ver: v1, Flags: 100111, > [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/f4e51b11-ac79-4a6a-b887-8926ffbd5cca","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"snapshots/2/14/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.10.31/export/secondary","_role":"Image"}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"executeInSequence":true,"wait":21600}}] > } > 2013-10-23 20:31:21,560 DEBUG [agent.transport.Request] > (AgentManager-Handler-8:null) Seq 6-1170407434: Processing: { Ans: , MgmtId: > 161342718518, via: 6, Ver: v1, Flags: 110, > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"org.libvirt.LibvirtException: > Domain snapshot not found: no domain snapshot with matching name > '65113136-dfb5-4cea-8e65-1065462ca2fe'","wait":0}}] } > 2013-10-23 20:31:21,832 DEBUG [storage.snapshot.SnapshotManagerImpl] > (Job-Executor-49:job-91 = [ e2bf2454-4273-4a89-bc38-35add8297eb1 ]) Failed to
[jira] [Commented] (CLOUDSTACK-4902) Fail to create snapshot with KVM when run multiple Hosts in Cluster
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13808129#comment-13808129 ] Ivan Kozlov commented on CLOUDSTACK-4902: - I have written patch that works for me. I do not know when cloudstack developers will get to this issue. So if you want to try it here are my diffs: 1. diff --git a/engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/AncientDataMotionStrategy.java b/tmp/cloud-snapshot-patch/AncientDataMotionStrategy.java index 4aa0147..57a17e0 100644 --- a/engine/storage/datamotion/src/org/apache/cloudstack/storage/motion/AncientDataMotionStrategy.java +++ b/tmp/cloud-snapshot-patch/AncientDataMotionStrategy.java @@ -78,6 +78,12 @@ import com.cloud.utils.NumbersUtil; import com.cloud.utils.db.DB; import com.cloud.utils.exception.CloudRuntimeException; +import com.cloud.vm.dao.VMInstanceDao; +import com.cloud.vm.VMInstanceVO; +import com.cloud.host.HostVO; +import org.apache.cloudstack.storage.RemoteHostEndPoint; + @Component public class AncientDataMotionStrategy implements DataMotionStrategy { @@ -120,6 +126,8 @@ public class StorageCacheManager cacheMgr; @Inject ManagementService _mgmtServer; +@Inject +VMInstanceDao vmDao; @Override public boolean canHandle(DataObject srcData, DataObject destData) { @@ -461,6 +469,24 @@ public class } else { CopyCommand cmd = new CopyCommand(srcData.getTO(), destData.getTO(), _backupsnapshotwait, _mgmt EndPoint ep = selector.select(srcData, destData); +SnapshotInfo snapshot = null; +try { + snapshot = (SnapshotInfo) srcData; + Long VolumeId = snapshot.getVolumeId(); + VolumeVO vol = volDao.findById(VolumeId); + Long VMId = vol.getInstanceId(); + VMInstanceVO vm = vmDao.findById(VMId); + if (vm.getHostId() != null) { + HostVO host=hostDao.findById(vm.getHostId()); + ep = RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(), host.getPrivateI + s_logger.debug("VM seems to be running on a host. Endpoint changed to " + ep.toS + } + +} catch (Exception e) { + s_logger.debug("Failed to find host with VM for snapshot: " + snapshot.getId(), e +} answer = ep.sendMessage(cmd); } @@ -489,3 +515,4 @@ public class return null; } } 2. diff --git a/plugins/storage/volume/default/src/org/apache/cloudstack/storage/datastore/driver/CloudStackPrimaryDataStoreDriverImpl.java b/tmp/cloud-snapshot-patch/CloudStackPrimaryDataStoreDriv index 683239c..cc37ed2 100644 --- a/CloudStackPrimaryDataStoreDriverImpl.java +++ b/tmp/cloud-snapshot-patch/CloudStackPrimaryDataStoreDriverImpl.java @@ -56,6 +56,13 @@ import org.apache.cloudstack.storage.to.TemplateObjectTO; import org.apache.cloudstack.storage.volume.VolumeObject; import org.apache.log4j.Logger; +import com.cloud.storage.VolumeVO; +import com.cloud.vm.VMInstanceVO; +import com.cloud.host.HostVO; +import org.apache.cloudstack.storage.RemoteHostEndPoint; + import javax.inject.Inject; import java.util.UUID; @@ -233,6 +240,23 @@ public class CloudStackPrimaryDataStoreDriverImpl implements PrimaryDataStoreDri CreateObjectCommand cmd = new CreateObjectCommand(snapshotTO); EndPoint ep = this.epSelector.select(snapshot); + +try { + Long VolumeId = snapshot.getVolumeId(); + VolumeVO vol = volumeDao.findById(VolumeId); + Long VMId = vol.getInstanceId(); + VMInstanceVO vm = vmDao.findById(VMId); + if (vm.getHostId() != null) { + HostVO host=hostDao.findById(vm.getHostId()); + ep = RemoteHostEndPoint.getHypervisorHostEndPoint(host.getId(), host.getPrivateIpAddress + s_logger.debug("VM seems to be running on a host. Endpoint changed to " + ep.toString()) + } + +} catch (Exception e) { + s_logger.debug("Failed to find host with VM for snapshot: " + snapshot.getId(), e); +} + + Answer answer = null; if ( ep == null ){ String errMsg = "No remote endpoint to send DeleteCommand, check if host or ssvm is down?"; > Fail to create snapshot with KVM when run multiple Hosts in Cluster > --- > > Key: CLOUDSTACK-4902 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4902 > Project: CloudStack > Issue Type: Bug > Security Level:
[jira] [Commented] (CLOUDSTACK-4892) KVM snapshots are failing on CLVM
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13819136#comment-13819136 ] Ivan Kozlov commented on CLOUDSTACK-4892: - I think this commit should solve the problem: https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=746896b unfortunately I cannot check it now. > KVM snapshots are failing on CLVM > - > > Key: CLOUDSTACK-4892 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4892 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Snapshot >Affects Versions: 4.2.0 > Environment: CentOS 6.4, KVM, CLVM >Reporter: Ivan Kozlov >Priority: Critical > > Creating snaphot fails hanging with state CreatedOnPrimary. Sometimes > creating snaphot is successful. > Snapshot logical volume is created and not deleted. > When running snaphot with only single host snapshot is created normaly. Guess > snapshot backup is trying access snapshot LV from host on which snapshot LV > is not opened. > Here is management log: > 2013-10-18 17:32:58,512 DEBUG [cloud.async.AsyncJobManagerImpl] > (catalina-exec-10:null) submit async job-41 = [ > 88ec27d7-78af-4664-a01b-eeca4469e37c ], details: AsyncJobVO {id:41, userId: > 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 10, > cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, > cmdOriginator: null, cmdInfo: > {"id":"10","response":"json","sessionkey":"HKb50xNHyZm2wJx/IHi5S7UWBGQ\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1382106777170","volumeid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","ctxAccountId":"2","ctxStartEventId":"126"}, > cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, > processStatus: 0, resultCode: 0, result: null, initMsid: 161342718518, > completeMsid: null, lastUpdated: null, lastPolled: null, created: null} > 2013-10-18 17:32:58,514 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Executing > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-41 > = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ] > 2013-10-18 17:32:58,549 INFO [user.snapshot.CreateSnapshotCmd] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) VOLSS: > createSnapshotCmd starts:1382106778549 > 2013-10-18 17:32:58,925 DEBUG [agent.transport.Request] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq > 1-111542657: Sending { Cmd , MgmtId: 161342718518, via: 1, Ver: v1, Flags: > 100011, > [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"560a9f6e-9864-43cc-8096-ed9cd6c97311","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"name":"ROOT-5","size":8589934592,"path":"4f3e8cfc-d3be-4e55-bc13-5c236a689c83","volumeId":5,"vmName":"i-2-5-VM","accountId":2,"format":"RAW","id":5,"hypervisorType":"KVM"},"parentSnapshotPath":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/7e85ab28-4ea5-4b5e-8ec1-1abadf2d571e","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4a975c8c-997a-4d1d-aa88-810fd281cb04","id":1,"poolType":"CLVM","host":"localhost","path":"/vg_primary","port":0}},"vmName":"i-2-5-VM","name":"test-100_ROOT-5_20131018143258","hypervisorType":"KVM","id":10}},"wait":0}}] > } > 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request] > (AgentManager-Handler-9:null) Seq 1-111542657: Processing: { Ans: , MgmtId: > 161342718518, via: 1, Ver: v1, Flags: 10, > [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/dev/vg_primary/4f3e8cfc-d3be-4e55-bc13-5c236a689c83/c6c900d1-1377-4347-ba69-9ba09f264f69","id":0}},"result":true,"wait":0}}] > } > 2013-10-18 17:32:59,986 DEBUG [agent.transport.Request] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq > 1-111542657: Received: { Ans: , MgmtId: 161342718518, via: 1, Ver: v1, > Flags: 10, { CreateObjectAnswer } } > 2013-10-18 17:33:00,497 DEBUG [storage.motion.AncientDataMotionStrategy] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) copyAsync > inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT > 2013-10-18 17:33:00,547 DEBUG [agent.transport.Request] > (Job-Executor-22:job-41 = [ 88ec27d7-78af-4664-a01b-eeca4469e37c ]) Seq > 4-1918238786: Sending { Cmd , MgmtId: 161342718518, via: 4, Ver: v1, Flags: > 100111, > [{"org.apache.cloudstack.s