[ https://issues.apache.org/jira/browse/CLOUDSTACK-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kozlov updated CLOUDSTACK-4939: ------------------------------------ Affects Version/s: 4.2.1 > Failed to create snaphot (KVM, GFS2) > ------------------------------------ > > Key: CLOUDSTACK-4939 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4939 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: KVM, Snapshot > Affects Versions: 4.2.0, 4.2.1 > Environment: CentOS 6.4, KVM, Shared mount point primary storage, > GFS2, iSCSI > Reporter: Ivan Kozlov > Priority: Blocker > Labels: kvm, sharedstorage, snapshot > > With one host snapshots are created ok. After adding second host some > snapshots fail (Failed to create snapshot due to an internal error creating > snapshot for volume 14) stucking with state "CreatedOnPrimary". Even when all > VMs are running on the same host. > debug libvirt log shows: > 2013-10-23 17:31:21.634+0000: 20007: debug : > virStorageFileGetMetadataInternal:673 : > path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 > 2013-10-23 17:32:57.189+0000: 20015: debug : qemuSnapObjFromName:233 : Domain > snapshot not found: no domain snapshot with matching name > '909848a0-b3ec-4657-a53a-c449dc24365b' > 2013-10-23 17:32:57.474+0000: 20009: debug : > virStorageFileGetMetadataInternal:673 : > path=/mnt/48a148f6-3373-3af2-8667-2f240988163d/snapshots, fd=31, format=2 > 2013-10-23 17:34:28.264+0000: 20008: debug : qemuSnapObjFromName:233 : Domain > snapshot not found: no domain snapshot with matching name > 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca' > management server log: > 2013-10-23 20:29:50,561 INFO [user.snapshot.CreateSnapshotCmd] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) VOLSS: > createSnapshotCmd starts:1382549390561 > 2013-10-23 20:29:52,053 DEBUG [agent.transport.Request] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Seq > 6-1170407437: Waiting for Seq 1170407434 Scheduling: { Cmd , MgmtId: > 161342718518, via: 6, Ver: v1, Flags: 100111, > [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/f4e51b11-ac79-4a6a-b887-8926ffbd5cca","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"/primary/d59c6574-8ff9-41e4-86e5-ce560f30d717/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/14","volume":{"uuid":"02c07659-59d3-42f2-8928-1d899cef94e7","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2c8e7b93-2d02-4c47-99ce-7bcd8670554a","id":2,"poolType":"SharedMountPoint","host":"localhost","path":"/primary","port":0}},"name":"ROOT-14","size":8589934592,"path":"d59c6574-8ff9-41e4-86e5-ce560f30d717","volumeId":14,"vmName":"i-2-14-VM","accountId":2,"format":"QCOW2","id":14,"hypervisorType":"KVM"},"parentSnapshotPath":"snapshots/2/14/ab317705-7368-4a40-9d1c-da2c8a7b1824","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.10.31/export/secondary","_role":"Image"}},"vmName":"i-2-14-VM","name":"t1_ROOT-14_20131023172950","hypervisorType":"KVM","id":33}},"executeInSequence":true,"wait":21600}}] > } > 2013-10-23 20:31:21,560 DEBUG [agent.transport.Request] > (AgentManager-Handler-8:null) Seq 6-1170407434: Processing: { Ans: , MgmtId: > 161342718518, via: 6, Ver: v1, Flags: 110, > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"org.libvirt.LibvirtException: > Domain snapshot not found: no domain snapshot with matching name > '65113136-dfb5-4cea-8e65-1065462ca2fe'","wait":0}}] } > 2013-10-23 20:31:21,832 DEBUG [storage.snapshot.SnapshotManagerImpl] > (Job-Executor-49:job-91 = [ e2bf2454-4273-4a89-bc38-35add8297eb1 ]) Failed to > create snapshot > com.cloud.utils.exception.CloudRuntimeException: > org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot > with matching name '65113136-dfb5-4cea-8e65-1065462ca2fe' > at > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264) > at > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013) > at > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170) > 2013-10-23 20:31:21,999 DEBUG [storage.volume.VolumeServiceImpl] > (Job-Executor-49:job-91 = [ e2bf2454-4273-4a89-bc38-35add8297eb1 ]) Take > snapshot: 18 failed > com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot > at > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1040) > at > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170) > Caused by: com.cloud.utils.exception.CloudRuntimeException: > org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot > with matching name '65113136-dfb5-4cea-8e65-1065462ca2fe' > at > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264) > at > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013) > 2013-10-23 20:31:22,167 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-49:job-91 = [ e2bf2454-4273-4a89-bc38-35add8297eb1 ]) Complete > async job-91 = [ e2bf2454-4273-4a89-bc38-35add8297eb1 ], jobStatus: 2, > resultCode: 530, result: Error Code: 530 Error text: Failed to create > snapshot due to an internal error creating snapshot for volume 18 > 2013-10-23 20:31:24,709 DEBUG [agent.transport.Request] > (AgentManager-Handler-13:null) Seq 9-1437990929: Processing: { Ans: , > MgmtId: 161342718518, via: 9, Ver: v1, Flags: 110, > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/16/157016cb-5e57-428f-b747-5d9b628d2864","id":0}},"result":true,"wait":0}}] > } > 2013-10-23 20:31:25,760 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-51:job-93 = [ 25e157c0-f966-401e-9263-c42dac56e0c1 ]) Done > executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd > for job-93 = [ 25e157c0-f966-401e-9263-c42dac56e0c1 ] > 2013-10-23 20:32:57,416 DEBUG [agent.transport.Request] > (AgentManager-Handler-8:null) Seq 6-1170407435: Processing: { Ans: , MgmtId: > 161342718518, via: 6, Ver: v1, Flags: 110, > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"org.libvirt.LibvirtException: > Domain snapshot not found: no domain snapshot with matching name > '909848a0-b3ec-4657-a53a-c449dc24365b'","wait":0}}] } > 2013-10-23 20:32:57,680 DEBUG [storage.snapshot.SnapshotManagerImpl] > (Job-Executor-50:job-92 = [ b8bbb5be-54ba-43df-b429-5b5fb61416ad ]) Failed to > create snapshot > com.cloud.utils.exception.CloudRuntimeException: > org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot > with matching name '909848a0-b3ec-4657-a53a-c449dc24365b' > at > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264) > at > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013) > at > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170) > 2013-10-23 20:32:57,763 DEBUG [storage.volume.VolumeServiceImpl] > (Job-Executor-50:job-92 = [ b8bbb5be-54ba-43df-b429-5b5fb61416ad ]) Take > snapshot: 17 failed > com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot > at > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1040) > at > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170) > Caused by: com.cloud.utils.exception.CloudRuntimeException: > org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot > with matching name '909848a0-b3ec-4657-a53a-c449dc24365b' > at > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264) > at > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013) > 2013-10-23 20:32:57,849 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-50:job-92 = [ b8bbb5be-54ba-43df-b429-5b5fb61416ad ]) Complete > async job-92 = [ b8bbb5be-54ba-43df-b429-5b5fb61416ad ], jobStatus: 2, > resultCode: 530, result: Error Code: 530 Error text: Failed to create > snapshot due to an internal error creating snapshot for volume 17 > 2013-10-23 20:33:50,627 DEBUG [storage.snapshot.SnapshotSchedulerImpl] > (SnapshotPollTask:null) Snapshot scheduler.poll is being called at 2013-10-23 > 17:33:50 GMT > 2013-10-23 20:33:50,627 DEBUG [storage.snapshot.SnapshotSchedulerImpl] > (SnapshotPollTask:null) Got 0 snapshots to be executed at 2013-10-23 17:33:50 > GMT > 2013-10-23 20:34:28,514 DEBUG [agent.transport.Request] > (AgentManager-Handler-3:null) Seq 6-1170407437: Processing: { Ans: , MgmtId: > 161342718518, via: 6, Ver: v1, Flags: 110, > [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"org.libvirt.LibvirtException: > Domain snapshot not found: no domain snapshot with matching name > 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca'","wait":0}}] } > 2013-10-23 20:34:28,779 DEBUG [storage.snapshot.SnapshotManagerImpl] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Failed to > create snapshot > com.cloud.utils.exception.CloudRuntimeException: > org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot > with matching name 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca' > at > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264) > at > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013) > at > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170) > 2013-10-23 20:34:28,870 DEBUG [storage.volume.VolumeServiceImpl] > (Job-Executor-52:job-94 = [ 42f8d6e0-762e-4f01-a7d5-daff2e31be13 ]) Take > snapshot: 14 failed > com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot > at > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1040) > at > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170) > Caused by: com.cloud.utils.exception.CloudRuntimeException: > org.libvirt.LibvirtException: Domain snapshot not found: no domain snapshot > with matching name 'f4e51b11-ac79-4a6a-b887-8926ffbd5cca' > at > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:264) > at > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013) -- This message was sent by Atlassian JIRA (v6.1#6144)