[ 
https://issues.apache.org/jira/browse/CLOUDSTACK-692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joris van Lieshout updated CLOUDSTACK-692:
------------------------------------------

    Summary: The CleanupSnapshotBackup process on SSVM deletes snapshots that 
are still in the process of being copied to secondary storage.  (was: The 
CleanupSnapshotBackup process on SSVM deletes snapshots that are still in the 
process of being created.)
    
> The CleanupSnapshotBackup process on SSVM deletes snapshots that are still in 
> the process of being copied to secondary storage.
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-692
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-692
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the 
> default.) 
>          Components: Snapshot
>            Reporter: Joris van Lieshout
>            Priority: Minor
>
> Hi there,
> I think we ran into a bug due to a concurrence of circumstances regarding 
> snapshotting and the cleanup of snapshots.
> The CleanupSnapshotBackup process on the SSVM deletes vhd files that are not 
> known in the database but when, especially long running snapshot, are being 
> copied to secondary storeage there is a gap between the start and finish of 
> the VDI-copy, where the uuid of the destination vhd is not registered in the 
> database. If the CleanupSnapshotBackup deletes the destinaion vhd during this 
> window it results in hanging sparse_dd process on the XenServer hypervisor 
> pointing to a tapdisk2 process with no file behind it.
> ===Secondary storage vm (2 hour time difference due to time zone). second to 
> last line you see the vhd being deleted.
> 2013-09-04 03:14:45,721 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) 
> Request:Seq 261-1870805144:  { Cmd , MgmtId: 345052370018, via: 261, Ver: v1, 
> Flags: 100011, 
> [{"CleanupSnapshotBackupCommand":{"secondaryStoragePoolURL":"nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1","dcId":1,"accountId":45,"volumeId":5863,"validBackupUUIDs":["1a56760b-d1c0-4620-8cf7-271951500d70","b6157bc9-085b-4ed6-95c2-4341f31c64bf","1ff967e3-3606-4112-9155-b1145b2ef576","12fbe4e3-1fdd-4c35-a961-0fce07cff584","278e9915-4f94-40c8-bef4-9c6bc82d4653","6fba1dd7-4736-47b3-9eed-148304c0e192","b9d8c9d8-6445-463b-b4e1-ab3b3f3a67a2","40ba5d72-c69a-46c2-973b-0570c1cabeac","774f2b0e-cdaf-4594-a9f9-4f872dcaad6e","8269f50b-6bec-427c-8186-540df6a75dbf","7b0c6e75-40cf-4dd7-826a-09b39f3da7b5","df7eac9c-137a-4655-9d21-d781916351f1","11ec2db1-a2fc-4221-ae1a-c1ab2bd59509","dfc348e1-af50-4d77-b4a0-6e86fc954e1c","98f64c0f-7498-4c70-8b70-beaefd723b45","c42f9dd5-079d-4b77-86dc-c19b7fbed817"],"wait":0}}]
>  }
> 2013-09-04 03:14:45,722 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) 
> Processing command: com.cloud.agent.api.CleanupSnapshotBackupCommand
> 2013-09-04 03:14:45,723 DEBUG [storage.resource.NfsSecondaryStorageResource] 
> (agentRequest-Handler-2:) Executing: mount 
> 2013-09-04 03:14:45,732 DEBUG [storage.resource.NfsSecondaryStorageResource] 
> (agentRequest-Handler-2:) Execution is successful.
> 2013-09-04 03:14:45,772 WARN  [storage.resource.NfsSecondaryStorageResource] 
> (agentRequest-Handler-2:) snapshot 8ca9fea4-8a98-4cc3-bba7-cc1dcf32bb24.vhd 
> is not recorded in DB, remove it
> 2013-09-04 03:14:45,772 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:) 
> Seq 261-1870805144:  { Ans: , MgmtId: 345052370018, via: 261, Ver: v1, Flags: 
> 10, [{"Answer":{"result":true,"wait":0}}] }
> ==== management-server.log. here you see the snapshot being created, the 
> copyToSecStorage process starting, eventually timing out due to the hanging 
> vdi-copy, failing on retrying because vdi in use (although not existing any 
> more the vdi is still know on xen), retrying some more on another HV and 
> eventuall giving up because it tries to create a duplicate SR.
> 2013-09-04 04:27:10,931 DEBUG [cloud.async.AsyncJobManagerImpl] 
> (Job-Executor-69:job-95137) Executing 
> com.cloud.api.commands.CreateSnapshotCmd for job-95137
> 2013-09-04 04:27:10,971 DEBUG [agent.transport.Request] 
> (Job-Executor-69:job-95137) Seq 91-780303147: Sending  { Cmd , MgmtId: 
> 345052370017, via: 91, Ver: v1, Flags: 100011, 
> [{"ManageSnapshotCommand":{"_commandSwitch":"-c","_volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","_pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"_snapshotName":"vlstws3_ROOT-2736_20130904022710","_snapshotId":71889,"_vmName":"i-45-2736-VM","wait":0}}]
>  }
> 2013-09-04 04:27:10,971 DEBUG [agent.transport.Request] 
> (Job-Executor-69:job-95137) Seq 91-780303147: Executing:  { Cmd , MgmtId: 
> 345052370017, via: 91, Ver: v1, Flags: 100011, 
> [{"ManageSnapshotCommand":{"_commandSwitch":"-c","_volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","_pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"_snapshotName":"vlstws3_ROOT-2736_20130904022710","_snapshotId":71889,"_vmName":"i-45-2736-VM","wait":0}}]
>  }
> 2013-09-04 04:27:12,949 DEBUG [agent.transport.Request] 
> (Job-Executor-69:job-95137) Seq 91-780303147: Received:  { Ans: , MgmtId: 
> 345052370017, via: 91, Ver: v1, Flags: 10, { ManageSnapshotAnswer } }
> 2013-09-04 04:27:12,991 DEBUG [agent.transport.Request] 
> (Job-Executor-69:job-95137) Seq 91-780303148: Sending  { Cmd , MgmtId: 
> 345052370017, via: 91, Ver: v1, Flags: 100011, 
> [{"BackupSnapshotCommand":{"isVolumeInactive":false,"vmName":"i-45-2736-VM","snapshotId":71889,"pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"primaryStoragePoolNameLabel":"b290385b-466d-3243-a939-3d242164e034","snapshotUuid":"6396ec81-4128-44ca-a121-d9f1bf789d6c","snapshotName":"vlstws3_ROOT-2736_20130904022710","secondaryStorageUrl":"nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1","dcId":1,"accountId":45,"volumeId":5863,"volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","wait":21600}}]
>  }
> 2013-09-04 04:27:12,991 DEBUG [agent.transport.Request] 
> (Job-Executor-69:job-95137) Seq 91-780303148: Executing:  { Cmd , MgmtId: 
> 345052370017, via: 91, Ver: v1, Flags: 100011, 
> [{"BackupSnapshotCommand":{"isVolumeInactive":false,"vmName":"i-45-2736-VM","snapshotId":71889,"pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"primaryStoragePoolNameLabel":"b290385b-466d-3243-a939-3d242164e034","snapshotUuid":"6396ec81-4128-44ca-a121-d9f1bf789d6c","snapshotName":"vlstws3_ROOT-2736_20130904022710","secondaryStorageUrl":"nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1","dcId":1,"accountId":45,"volumeId":5863,"volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","wait":21600}}]
>  }
> 2013-09-04 04:27:12,992 DEBUG [agent.manager.DirectAgentAttache] 
> (DirectAgent-441:null) Seq 91-780303148: Executing request
> 2013-09-04 04:27:13,561 DEBUG [xen.resource.CitrixResourceBase] 
> (DirectAgent-441:null) Creating a not shared SR for 
> nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1/snapshots/45/5863
> 2013-09-04 04:27:14,499 DEBUG [xen.resource.CitrixResourceBase] 
> (DirectAgent-441:null) Checking 00e04f60-4c80-37e4-9abe-855b16fbc9e1 or SR 
> a4ca3b8c-6e6d-5f07-cb71-f6c9b7f950cb on 
> XS[d868bf32-d747-4ec0-b1c2-fba3df1c85bd-192.168.52.15]
> 2013-09-04 04:27:14,512 DEBUG [xen.resource.CitrixResourceBase] 
> (DirectAgent-441:null) Host 192.168.52.15 
> OpaqueRef:5ee9850b-b15c-b39b-dd27-ff226a1b3126: Created a SR; UUID is 
> a4ca3b8c-6e6d-5f07-cb71-f6c9b7f950cb device config is 
> {serverpath=/volumes/pool0/MCCP-SHARED-1-1/snapshots/45/5863, 
> server=mccpnas7.storage.mccp.mcinfra.net}
> 2013-09-04 10:27:12,992 DEBUG [agent.manager.AgentAttache] 
> (Job-Executor-69:job-95137) Seq 91-780303148: Waiting some more time because 
> this is the current command
> 2013-09-04 10:27:34,262 WARN  [xen.resource.CitrixResourceBase] 
> (DirectAgent-441:null) Async 21600 seconds timeout for task 
> com.xensource.xenapi.Task@ec1aac0e
> 2013-09-04 10:27:34,585 DEBUG [xen.resource.CitrixResourceBase] 
> (DirectAgent-441:null) Host 192.168.52.15 
> OpaqueRef:5ee9850b-b15c-b39b-dd27-ff226a1b3126: Removing SR
> 2013-09-04 10:27:34,624 DEBUG [xen.resource.CitrixResourceBase] 
> (DirectAgent-441:null) Host 192.168.52.15 
> OpaqueRef:5ee9850b-b15c-b39b-dd27-ff226a1b3126: Catch XenAPIException: This 
> operation cannot be performed because this VDI is in use by some other 
> operation
> 2013-09-04 10:27:34,650 DEBUG [xen.resource.CitrixResourceBase] 
> (DirectAgent-441:null) Host 192.168.52.15 
> OpaqueRef:5ee9850b-b15c-b39b-dd27-ff226a1b3126: Catch XenAPIException: This 
> operation cannot be performed because this VDI is in use by some other 
> operation
> 2013-09-04 10:27:34,650 WARN  [xen.resource.CitrixResourceBase] 
> (DirectAgent-441:null) Host 192.168.52.15 
> OpaqueRef:5ee9850b-b15c-b39b-dd27-ff226a1b3126: Unable to remove SR
> 2013-09-04 10:27:34,650 WARN  [xen.resource.CitrixResourceBase] 
> (DirectAgent-441:null) BackupSnapshot Failed due to Async 21600 seconds 
> timeout for task com.xensource.xenapi.Task@ec1aac0e
> 2013-09-04 10:27:34,651 DEBUG [agent.manager.DirectAgentAttache] 
> (DirectAgent-441:null) Seq 91-780303148: Response Received: 
> 2013-09-04 10:27:34,651 DEBUG [agent.transport.Request] 
> (DirectAgent-441:null) Seq 91-780303148: Processing:  { Ans: , MgmtId: 
> 345052370017, via: 91, Ver: v1, Flags: 10, 
> [{"BackupSnapshotAnswer":{"full":true,"result":false,"details":"BackupSnapshot
>  Failed due to Async 21600 seconds timeout for task 
> com.xensource.xenapi.Task@ec1aac0e","wait":0}}] }
> 2013-09-04 10:27:34,651 DEBUG [agent.transport.Request] 
> (Job-Executor-69:job-95137) Seq 91-780303148: Received:  { Ans: , MgmtId: 
> 345052370017, via: 91, Ver: v1, Flags: 10, { BackupSnapshotAnswer } }
> 2013-09-04 10:27:34,651 DEBUG [storage.snapshot.SnapshotManagerImpl] 
> (Job-Executor-69:job-95137) The result for 
> com.cloud.agent.api.BackupSnapshotCommand is BackupSnapshot Failed due to 
> Async 21600 seconds timeout for task com.xensource.xenapi.Task@ec1aac0e 
> through 91
> 2013-09-04 10:29:34,652 DEBUG [storage.snapshot.SnapshotManagerImpl] 
> (Job-Executor-69:job-95137) Retrying...
> 2013-09-04 10:29:34,659 DEBUG [agent.manager.ClusteredAgentAttache] 
> (Job-Executor-69:job-95137) Seq 89-1251159458: Forwarding Seq 89-1251159458:  
> { Cmd , MgmtId: 345052370017, via: 89, Ver: v1, Flags: 100011, 
> [{"BackupSnapshotCommand":{"isVolumeInactive":false,"vmName":"i-45-2736-VM","snapshotId":71889,"pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"primaryStoragePoolNameLabel":"b290385b-466d-3243-a939-3d242164e034","snapshotUuid":"6396ec81-4128-44ca-a121-d9f1bf789d6c","snapshotName":"vlstws3_ROOT-2736_20130904022710","secondaryStorageUrl":"nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1","dcId":1,"accountId":45,"volumeId":5863,"volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","wait":21600}}]
>  } to 345052370018
> 2013-09-04 10:29:43,085 DEBUG [agent.transport.Request] 
> (Job-Executor-69:job-95137) Seq 89-1251159458: Received:  { Ans: , MgmtId: 
> 345052370017, via: 89, Ver: v1, Flags: 10, { BackupSnapshotAnswer } }
> 2013-09-04 10:29:43,086 DEBUG [storage.snapshot.SnapshotManagerImpl] 
> (Job-Executor-69:job-95137) The result for 
> com.cloud.agent.api.BackupSnapshotCommand is BackupSnapshot Failed due to Can 
> not create second storage SR mountpoint: 
> mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1/snapshots/45/5863
>  due to SR_BACKEND_FAILURE_40The SR scan failed  [opterr=['INTERNAL_ERROR', 
> 'Db_exn.Uniqueness_constraint_violation("VDI", "uuid", 
> "df7eac9c-137a-4655-9d21-d781916351f1")']] through 89
> 2013-09-04 10:30:17,489 DEBUG [agent.manager.DirectAgentAttache] 
> (DirectAgent-441:null) Ping from 109
> 2013-09-04 10:31:18,345 DEBUG [agent.manager.DirectAgentAttache] 
> (DirectAgent-441:null) Seq 95-1001324546: Executing request
> 2013-09-04 10:31:18,642 DEBUG [agent.manager.DirectAgentAttache] 
> (DirectAgent-441:null) Seq 95-1001324546: Response Received: 
> 2013-09-04 10:31:18,642 DEBUG [agent.transport.Request] 
> (DirectAgent-441:null) Seq 95-1001324546: Processing:  { Ans: , MgmtId: 
> 345052370017, via: 95, Ver: v1, Flags: 10, 
> [{"Answer":{"result":true,"wait":0}}] }
> 2013-09-04 10:31:19,423 DEBUG [agent.manager.DirectAgentAttache] 
> (DirectAgent-441:null) Seq 41-1598291970: Executing request
> 2013-09-04 10:31:19,663 DEBUG [agent.manager.DirectAgentAttache] 
> (DirectAgent-441:null) Seq 41-1598291970: Response Received: 
> 2013-09-04 10:31:19,663 DEBUG [agent.transport.Request] 
> (DirectAgent-441:null) Seq 41-1598291970: Processing:  { Ans: , MgmtId: 
> 345052370017, via: 41, Ver: v1, Flags: 10, 
> [{"Answer":{"result":true,"wait":0}}] }
> 2013-09-04 10:31:43,086 DEBUG [storage.snapshot.SnapshotManagerImpl] 
> (Job-Executor-69:job-95137) Retrying...
> 2013-09-04 10:31:43,091 DEBUG [agent.transport.Request] 
> (Job-Executor-69:job-95137) Seq 93-156960768: Sending  { Cmd , MgmtId: 
> 345052370017, via: 93, Ver: v1, Flags: 100011, 
> [{"BackupSnapshotCommand":{"isVolumeInactive":false,"vmName":"i-45-2736-VM","snapshotId":71889,"pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"primaryStoragePoolNameLabel":"b290385b-466d-3243-a939-3d242164e034","snapshotUuid":"6396ec81-4128-44ca-a121-d9f1bf789d6c","snapshotName":"vlstws3_ROOT-2736_20130904022710","secondaryStorageUrl":"nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1","dcId":1,"accountId":45,"volumeId":5863,"volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","wait":21600}}]
>  }
> 2013-09-04 10:31:43,091 DEBUG [agent.transport.Request] 
> (Job-Executor-69:job-95137) Seq 93-156960768: Executing:  { Cmd , MgmtId: 
> 345052370017, via: 93, Ver: v1, Flags: 100011, 
> [{"BackupSnapshotCommand":{"isVolumeInactive":false,"vmName":"i-45-2736-VM","snapshotId":71889,"pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"primaryStoragePoolNameLabel":"b290385b-466d-3243-a939-3d242164e034","snapshotUuid":"6396ec81-4128-44ca-a121-d9f1bf789d6c","snapshotName":"vlstws3_ROOT-2736_20130904022710","secondaryStorageUrl":"nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1","dcId":1,"accountId":45,"volumeId":5863,"volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","wait":21600}}]
>  }
> 2013-09-04 10:31:56,375 DEBUG [agent.transport.Request] 
> (Job-Executor-69:job-95137) Seq 93-156960768: Received:  { Ans: , MgmtId: 
> 345052370017, via: 93, Ver: v1, Flags: 10, { BackupSnapshotAnswer } }
> 2013-09-04 10:31:56,376 DEBUG [storage.snapshot.SnapshotManagerImpl] 
> (Job-Executor-69:job-95137) The result for 
> com.cloud.agent.api.BackupSnapshotCommand is BackupSnapshot Failed due to Can 
> not create second storage SR mountpoint: 
> mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1/snapshots/45/5863
>  due to SR_BACKEND_FAILURE_40The SR scan failed  [opterr=['INTERNAL_ERROR', 
> 'Db_exn.Uniqueness_constraint_violation("VDI", "uuid", 
> "df7eac9c-137a-4655-9d21-d781916351f1")']] through 93
> 2013-09-04 10:33:56,376 DEBUG [storage.snapshot.SnapshotManagerImpl] 
> (Job-Executor-69:job-95137) Retrying...
> 2013-09-04 10:33:56,381 DEBUG [agent.transport.Request] 
> (Job-Executor-69:job-95137) Seq 95-1001356346: Sending  { Cmd , MgmtId: 
> 345052370017, via: 95, Ver: v1, Flags: 100011, 
> [{"BackupSnapshotCommand":{"isVolumeInactive":false,"vmName":"i-45-2736-VM","snapshotId":71889,"pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"primaryStoragePoolNameLabel":"b290385b-466d-3243-a939-3d242164e034","snapshotUuid":"6396ec81-4128-44ca-a121-d9f1bf789d6c","snapshotName":"vlstws3_ROOT-2736_20130904022710","secondaryStorageUrl":"nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1","dcId":1,"accountId":45,"volumeId":5863,"volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","wait":21600}}]
>  }
> 2013-09-04 10:33:56,381 DEBUG [agent.transport.Request] 
> (Job-Executor-69:job-95137) Seq 95-1001356346: Executing:  { Cmd , MgmtId: 
> 345052370017, via: 95, Ver: v1, Flags: 100011, 
> [{"BackupSnapshotCommand":{"isVolumeInactive":false,"vmName":"i-45-2736-VM","snapshotId":71889,"pool":{"id":208,"uuid":"b290385b-466d-3243-a939-3d242164e034","host":"mccpnas3-4-vip1.mccp.mcinfra.net","path":"/volumes/pool0/MCCP-S-SBP1-1_MCCP-XEN-1","port":2049,"type":"NetworkFilesystem"},"primaryStoragePoolNameLabel":"b290385b-466d-3243-a939-3d242164e034","snapshotUuid":"6396ec81-4128-44ca-a121-d9f1bf789d6c","snapshotName":"vlstws3_ROOT-2736_20130904022710","secondaryStorageUrl":"nfs://mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1","dcId":1,"accountId":45,"volumeId":5863,"volumePath":"9cb7af90-ca88-4b34-aa6f-bc21c3d4a3aa","wait":21600}}]
>  }
> 2013-09-04 10:34:13,682 DEBUG [agent.transport.Request] 
> (Job-Executor-69:job-95137) Seq 95-1001356346: Received:  { Ans: , MgmtId: 
> 345052370017, via: 95, Ver: v1, Flags: 10, { BackupSnapshotAnswer } }
> 2013-09-04 10:34:13,683 DEBUG [storage.snapshot.SnapshotManagerImpl] 
> (Job-Executor-69:job-95137) The result for 
> com.cloud.agent.api.BackupSnapshotCommand is BackupSnapshot Failed due to Can 
> not create second storage SR mountpoint: 
> mccpnas7.storage.mccp.mcinfra.net/volumes/pool0/MCCP-SHARED-1-1/snapshots/45/5863
>  due to SR_BACKEND_FAILURE_40The SR scan failed  [opterr=['INTERNAL_ERROR', 
> 'Db_exn.Uniqueness_constraint_violation("VDI", "uuid", 
> "df7eac9c-137a-4655-9d21-d781916351f1")']] through 95
> 2013-09-04 10:36:13,683 DEBUG [storage.snapshot.SnapshotManagerImpl] 
> (Job-Executor-69:job-95137) Retrying...
> 2013-09-04 10:36:13,685 WARN  [storage.snapshot.SnapshotManagerImpl] 
> (Job-Executor-69:job-95137) Storage unavailable 
> 2013-09-04 10:36:13,685 WARN  [storage.snapshot.SnapshotManagerImpl] 
> (Job-Executor-69:job-95137) Failed to back up snapshot on secondary storage, 
> deleting the record from the DB
> 2013-09-04 10:36:13,702 ERROR [cloud.api.ApiDispatcher] 
> (Job-Executor-69:job-95137) Exception while executing CreateSnapshotCmd:
> 2013-09-04 10:36:13,702 DEBUG [cloud.async.AsyncJobManagerImpl] 
> (Job-Executor-69:job-95137) Complete async job-95137, jobStatus: 2, 
> resultCode: 530, result: Error Code: 530 Error text: Created snapshot: 
> com.cloud.storage.SnapshotVO$$EnhancerByCGLIB$$3173e384@40abfe02 on primary 
> but failed to backup on secondary

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Reply via email to