[DISCUSS] Scalable Backup and Recovery
I’d like to revisit this topic focusing on a UI-only solution. I believe a reasonable approach is to allow users to select multiple volumes or vms (with the new multi-select UI widget) to perform snapshot operations. With this, we would kick off a snapshot API request per volume/vm that was selected so the UI would display the status and result of each request individually rather than lumping all the requests together and only being able to display a success or failure for the entire selection. Consequently, we would only need to make UI changes. To minimize the effort, we could tie directly into the existing filtering features of the volumes vms pages. This would allow users to select volumes vms for backup by the following criteria: -Name -Zone -Domain (if admin) -Account (if admin) -Tag key -Tag value I understand one lingering concern some people had was the user’s belief that the snapshots would all be consistent amongst each other. I’d like to hear if this is still a concern with this approach and if so, if there are any ideas of how to express to the user how the operation would actually be performed. -Chris -- Chris Suich chris.su...@netapp.commailto:chris.su...@netapp.com NetApp Software Engineer Data Center Platforms – Cloud Solutions Citrix, Cisco Red Hat On Sep 30, 2013, at 8:37 AM, SuichII, Christopher chris.su...@netapp.commailto:chris.su...@netapp.com wrote: See responses inline. -- Chris Suich chris.su...@netapp.commailto:chris.su...@netapp.com NetApp Software Engineer Data Center Platforms – Cloud Solutions Citrix, Cisco Red Hat On Sep 29, 2013, at 10:22 AM, kel...@backbonetechnology.commailto:kel...@backbonetechnology.com wrote: Even as a simple start it would be nice to have the option to back up/snapshot a VM (which means all volumes as a collection) and not just per volume. If I have 200 VMs with 3 volumes each, that's 600 snapshots and 400 attach disk actions. Now scale that out to your average production environment running 1000s of VMs. This becomes a logistical recovery nightmare. We should have some kind of routine that allows snapping multiple volumes as a collection so those same volumes can be restored as a collection. This is a really cool use case as well, I can definitely get behind it as well. Just a thought. Thanks, -Kelcey Sent from my HTC - Reply message - From: Darren Shepherd darren.s.sheph...@gmail.commailto:darren.s.sheph...@gmail.com To: dev@cloudstack.apache.orgmailto:dev@cloudstack.apache.org dev@cloudstack.apache.orgmailto:dev@cloudstack.apache.org Subject: Scalable Backup and Recovery Date: Sat, Sep 28, 2013 9:22 PM Based on your use cases it sounds like what your asking for is the ability to create a selection criteria for scheduled snapshots. So as long as your VM/Volume matches that criteria it will be backed up at some given time. I think that would be useful because a user could say something like all volumes in network 'production' should be backed up every night. Selection criteria is an interesting option. It is not necessarily what I was proposing, but that would be a cool solution depending on the flexibility of the criteria. So if such a thing was implemented it seems like if the storage provider implemented some capability to handle multiple snapshots at once, then it could be passed all volumes at once and it would do something intelligent. This is along the lines of what I was talking about in my other email - sometimes it is just easer to do things together, not one by one. Now reading between the lines, it seems like you're looking for some use case to exploit some functionality in netapp. I'll tell you what I'd like to see implemented. Having ran netapp in the past with CS we ran into this conundrum. We would take snapshots on the filter, but in reality they were pretty useless. If you ever needed to rollback to the snapshot, your screwed. Things have changed since the snapshot, VMs were created, deleted, VM snapshots had occurred, etc. So if you rollback the metadata in CS is completely out of sync now. What I think would be a great feature is to be able to do StoragePool snapshots. Now doing the snapshot is simple, the really complex thing is how to implement the StoragePool revert to snapshot functionality. If somebody could do that, that would be awesome. Without getting into much detail, we're actually not trying to exploit anything too NetApp specific. In fact, we're not really leveraging NetApp volume snapshots for BR for the reason you listed. If you try to recover the whole volume, you're potentially blowing away data that was created between clicking the button and when the snapshot was created. Again, that seems like a vendor specific thing - something we should add to the storage subsystem interface and let storage providers implement it if they can. Darren On Sat, Sep 28, 2013 at 5:41 PM, SuichII, Christopher chris.su...@netapp.commailto:chris.su
Re: Scalable Backup and Recovery
See responses inline. -- Chris Suich chris.su...@netapp.com NetApp Software Engineer Data Center Platforms – Cloud Solutions Citrix, Cisco Red Hat On Sep 29, 2013, at 10:22 AM, kel...@backbonetechnology.com wrote: Even as a simple start it would be nice to have the option to back up/snapshot a VM (which means all volumes as a collection) and not just per volume. If I have 200 VMs with 3 volumes each, that's 600 snapshots and 400 attach disk actions. Now scale that out to your average production environment running 1000s of VMs. This becomes a logistical recovery nightmare. We should have some kind of routine that allows snapping multiple volumes as a collection so those same volumes can be restored as a collection. This is a really cool use case as well, I can definitely get behind it as well. Just a thought. Thanks, -Kelcey Sent from my HTC - Reply message - From: Darren Shepherd darren.s.sheph...@gmail.com To: dev@cloudstack.apache.org dev@cloudstack.apache.org Subject: Scalable Backup and Recovery Date: Sat, Sep 28, 2013 9:22 PM Based on your use cases it sounds like what your asking for is the ability to create a selection criteria for scheduled snapshots. So as long as your VM/Volume matches that criteria it will be backed up at some given time. I think that would be useful because a user could say something like all volumes in network 'production' should be backed up every night. Selection criteria is an interesting option. It is not necessarily what I was proposing, but that would be a cool solution depending on the flexibility of the criteria. So if such a thing was implemented it seems like if the storage provider implemented some capability to handle multiple snapshots at once, then it could be passed all volumes at once and it would do something intelligent. This is along the lines of what I was talking about in my other email - sometimes it is just easer to do things together, not one by one. Now reading between the lines, it seems like you're looking for some use case to exploit some functionality in netapp. I'll tell you what I'd like to see implemented. Having ran netapp in the past with CS we ran into this conundrum. We would take snapshots on the filter, but in reality they were pretty useless. If you ever needed to rollback to the snapshot, your screwed. Things have changed since the snapshot, VMs were created, deleted, VM snapshots had occurred, etc. So if you rollback the metadata in CS is completely out of sync now. What I think would be a great feature is to be able to do StoragePool snapshots. Now doing the snapshot is simple, the really complex thing is how to implement the StoragePool revert to snapshot functionality. If somebody could do that, that would be awesome. Without getting into much detail, we're actually not trying to exploit anything too NetApp specific. In fact, we're not really leveraging NetApp volume snapshots for BR for the reason you listed. If you try to recover the whole volume, you're potentially blowing away data that was created between clicking the button and when the snapshot was created. Again, that seems like a vendor specific thing - something we should add to the storage subsystem interface and let storage providers implement it if they can. Darren On Sat, Sep 28, 2013 at 5:41 PM, SuichII, Christopher chris.su...@netapp.com wrote: Well, yes, in part. By scalable I mean that if CloudStack is expected to be able to manage such a large number of vms, it should be able to backup and recover those vms with minimal effort. Doing things one at a time does not necessarily scale well when you're talking about a cloud infrastructure. Also certain hypervisors have various quirks which stand in the way of an efficient solution. I absolutely agree. This is where the storage subsystem API comes in. Creating backups for some storage providers can be much faster, easier and more efficient than hypervisor. As the storage subsystem API gains more traction and true backup and recovery becomes available, I think we'll begin to see people asking why things must be done one at a time. The use cases I listed below would help us get ahead of the curve and have these features I predict people will be asking for (and it sounds like Kelcey is asking for it now!). -- Chris Suich chris.su...@netapp.com NetApp Software Engineer Data Center Platforms – Cloud Solutions Citrix, Cisco Red Hat On Sep 27, 2013, at 6:38 PM, Chiradeep Vittal chiradeep.vit...@citrix.com wrote: Ah I see. You mean a scalable user experience. The actual scalability of the snapshot process itself is limited by available disk and network bandwidth. Also certain hypervisors have various quirks which stand in the way of an efficient solution. On 9/27/13 10:27 AM, SuichII, Christopher chris.su...@netapp.com wrote: I'd like to start a discussion around
Re: Scalable Backup and Recovery
Even as a simple start it would be nice to have the option to back up/snapshot a VM (which means all volumes as a collection) and not just per volume. If I have 200 VMs with 3 volumes each, that's 600 snapshots and 400 attach disk actions. Now scale that out to your average production environment running 1000s of VMs. This becomes a logistical recovery nightmare. We should have some kind of routine that allows snapping multiple volumes as a collection so those same volumes can be restored as a collection. Just a thought. Thanks, -Kelcey Sent from my HTC - Reply message - From: Darren Shepherd darren.s.sheph...@gmail.com To: dev@cloudstack.apache.org dev@cloudstack.apache.org Subject: Scalable Backup and Recovery Date: Sat, Sep 28, 2013 9:22 PM Based on your use cases it sounds like what your asking for is the ability to create a selection criteria for scheduled snapshots. So as long as your VM/Volume matches that criteria it will be backed up at some given time. I think that would be useful because a user could say something like all volumes in network 'production' should be backed up every night. So if such a thing was implemented it seems like if the storage provider implemented some capability to handle multiple snapshots at once, then it could be passed all volumes at once and it would do something intelligent. Now reading between the lines, it seems like you're looking for some use case to exploit some functionality in netapp. I'll tell you what I'd like to see implemented. Having ran netapp in the past with CS we ran into this conundrum. We would take snapshots on the filter, but in reality they were pretty useless. If you ever needed to rollback to the snapshot, your screwed. Things have changed since the snapshot, VMs were created, deleted, VM snapshots had occurred, etc. So if you rollback the metadata in CS is completely out of sync now. What I think would be a great feature is to be able to do StoragePool snapshots. Now doing the snapshot is simple, the really complex thing is how to implement the StoragePool revert to snapshot functionality. If somebody could do that, that would be awesome. Darren On Sat, Sep 28, 2013 at 5:41 PM, SuichII, Christopher chris.su...@netapp.com wrote: Well, yes, in part. By scalable I mean that if CloudStack is expected to be able to manage such a large number of vms, it should be able to backup and recover those vms with minimal effort. Doing things one at a time does not necessarily scale well when you're talking about a cloud infrastructure. Also certain hypervisors have various quirks which stand in the way of an efficient solution. I absolutely agree. This is where the storage subsystem API comes in. Creating backups for some storage providers can be much faster, easier and more efficient than hypervisor. As the storage subsystem API gains more traction and true backup and recovery becomes available, I think we'll begin to see people asking why things must be done one at a time. The use cases I listed below would help us get ahead of the curve and have these features I predict people will be asking for (and it sounds like Kelcey is asking for it now!). -- Chris Suich chris.su...@netapp.com NetApp Software Engineer Data Center Platforms – Cloud Solutions Citrix, Cisco Red Hat On Sep 27, 2013, at 6:38 PM, Chiradeep Vittal chiradeep.vit...@citrix.com wrote: Ah I see. You mean a scalable user experience. The actual scalability of the snapshot process itself is limited by available disk and network bandwidth. Also certain hypervisors have various quirks which stand in the way of an efficient solution. On 9/27/13 10:27 AM, SuichII, Christopher chris.su...@netapp.com wrote: I'd like to start a discussion around the direction of scalable backup and recovery in CloudStack. Currently, the only want to backup and recover vms is by setting up a schedule or manually snapshotting up individual vm disks or manually snapshotting vms. Unfortunately, I don't believe this is a very scalable solution. What if a user wants all of their vm disks to be backed up on the same schedule? What if a domain administrator wants all of the vms in their domain to be backed up on the same schedule or to manually backup every vm in their domain? Here are some use cases I see for helping to scale things up: -Scheduled and manual backup of 1 to all of a user's vms and vm disks -Scheduled and manual backup of 1 to all of a domain's vms and vm disks (by a domain admin) -Scheduled and manual backup of 1 to all vms and vm disks on primary storage (by a cloud admin) - this one is tougher to find a valid use case for -Backup schedules attached to service offerings I know I previously started a discussion about backing up multiple vm disks at once, but I think these use cases, broken down by user type (user, domain admin and admin), should help clear things up and show the utility of being able
Re: Scalable Backup and Recovery
Well, yes, in part. By scalable I mean that if CloudStack is expected to be able to manage such a large number of vms, it should be able to backup and recover those vms with minimal effort. Doing things one at a time does not necessarily scale well when you're talking about a cloud infrastructure. Also certain hypervisors have various quirks which stand in the way of an efficient solution. I absolutely agree. This is where the storage subsystem API comes in. Creating backups for some storage providers can be much faster, easier and more efficient than hypervisor. As the storage subsystem API gains more traction and true backup and recovery becomes available, I think we'll begin to see people asking why things must be done one at a time. The use cases I listed below would help us get ahead of the curve and have these features I predict people will be asking for (and it sounds like Kelcey is asking for it now!). -- Chris Suich chris.su...@netapp.com NetApp Software Engineer Data Center Platforms – Cloud Solutions Citrix, Cisco Red Hat On Sep 27, 2013, at 6:38 PM, Chiradeep Vittal chiradeep.vit...@citrix.com wrote: Ah I see. You mean a scalable user experience. The actual scalability of the snapshot process itself is limited by available disk and network bandwidth. Also certain hypervisors have various quirks which stand in the way of an efficient solution. On 9/27/13 10:27 AM, SuichII, Christopher chris.su...@netapp.com wrote: I'd like to start a discussion around the direction of scalable backup and recovery in CloudStack. Currently, the only want to backup and recover vms is by setting up a schedule or manually snapshotting up individual vm disks or manually snapshotting vms. Unfortunately, I don't believe this is a very scalable solution. What if a user wants all of their vm disks to be backed up on the same schedule? What if a domain administrator wants all of the vms in their domain to be backed up on the same schedule or to manually backup every vm in their domain? Here are some use cases I see for helping to scale things up: -Scheduled and manual backup of 1 to all of a user's vms and vm disks -Scheduled and manual backup of 1 to all of a domain's vms and vm disks (by a domain admin) -Scheduled and manual backup of 1 to all vms and vm disks on primary storage (by a cloud admin) - this one is tougher to find a valid use case for -Backup schedules attached to service offerings I know I previously started a discussion about backing up multiple vm disks at once, but I think these use cases, broken down by user type (user, domain admin and admin), should help clear things up and show the utility of being able to backup multiple objects at once. Thanks! Chris -- Chris Suich chris.su...@netapp.com NetApp Software Engineer Data Center Platforms Cloud Solutions Citrix, Cisco Red Hat
Re: Scalable Backup and Recovery
Based on your use cases it sounds like what your asking for is the ability to create a selection criteria for scheduled snapshots. So as long as your VM/Volume matches that criteria it will be backed up at some given time. I think that would be useful because a user could say something like all volumes in network 'production' should be backed up every night. So if such a thing was implemented it seems like if the storage provider implemented some capability to handle multiple snapshots at once, then it could be passed all volumes at once and it would do something intelligent. Now reading between the lines, it seems like you're looking for some use case to exploit some functionality in netapp. I'll tell you what I'd like to see implemented. Having ran netapp in the past with CS we ran into this conundrum. We would take snapshots on the filter, but in reality they were pretty useless. If you ever needed to rollback to the snapshot, your screwed. Things have changed since the snapshot, VMs were created, deleted, VM snapshots had occurred, etc. So if you rollback the metadata in CS is completely out of sync now. What I think would be a great feature is to be able to do StoragePool snapshots. Now doing the snapshot is simple, the really complex thing is how to implement the StoragePool revert to snapshot functionality. If somebody could do that, that would be awesome. Darren On Sat, Sep 28, 2013 at 5:41 PM, SuichII, Christopher chris.su...@netapp.com wrote: Well, yes, in part. By scalable I mean that if CloudStack is expected to be able to manage such a large number of vms, it should be able to backup and recover those vms with minimal effort. Doing things one at a time does not necessarily scale well when you're talking about a cloud infrastructure. Also certain hypervisors have various quirks which stand in the way of an efficient solution. I absolutely agree. This is where the storage subsystem API comes in. Creating backups for some storage providers can be much faster, easier and more efficient than hypervisor. As the storage subsystem API gains more traction and true backup and recovery becomes available, I think we'll begin to see people asking why things must be done one at a time. The use cases I listed below would help us get ahead of the curve and have these features I predict people will be asking for (and it sounds like Kelcey is asking for it now!). -- Chris Suich chris.su...@netapp.com NetApp Software Engineer Data Center Platforms – Cloud Solutions Citrix, Cisco Red Hat On Sep 27, 2013, at 6:38 PM, Chiradeep Vittal chiradeep.vit...@citrix.com wrote: Ah I see. You mean a scalable user experience. The actual scalability of the snapshot process itself is limited by available disk and network bandwidth. Also certain hypervisors have various quirks which stand in the way of an efficient solution. On 9/27/13 10:27 AM, SuichII, Christopher chris.su...@netapp.com wrote: I'd like to start a discussion around the direction of scalable backup and recovery in CloudStack. Currently, the only want to backup and recover vms is by setting up a schedule or manually snapshotting up individual vm disks or manually snapshotting vms. Unfortunately, I don't believe this is a very scalable solution. What if a user wants all of their vm disks to be backed up on the same schedule? What if a domain administrator wants all of the vms in their domain to be backed up on the same schedule or to manually backup every vm in their domain? Here are some use cases I see for helping to scale things up: -Scheduled and manual backup of 1 to all of a user's vms and vm disks -Scheduled and manual backup of 1 to all of a domain's vms and vm disks (by a domain admin) -Scheduled and manual backup of 1 to all vms and vm disks on primary storage (by a cloud admin) - this one is tougher to find a valid use case for -Backup schedules attached to service offerings I know I previously started a discussion about backing up multiple vm disks at once, but I think these use cases, broken down by user type (user, domain admin and admin), should help clear things up and show the utility of being able to backup multiple objects at once. Thanks! Chris -- Chris Suich chris.su...@netapp.com NetApp Software Engineer Data Center Platforms Cloud Solutions Citrix, Cisco Red Hat
Scalable Backup and Recovery
I'd like to start a discussion around the direction of scalable backup and recovery in CloudStack. Currently, the only want to backup and recover vms is by setting up a schedule or manually snapshotting up individual vm disks or manually snapshotting vms. Unfortunately, I don't believe this is a very scalable solution. What if a user wants all of their vm disks to be backed up on the same schedule? What if a domain administrator wants all of the vms in their domain to be backed up on the same schedule or to manually backup every vm in their domain? Here are some use cases I see for helping to scale things up: -Scheduled and manual backup of 1 to all of a user's vms and vm disks -Scheduled and manual backup of 1 to all of a domain's vms and vm disks (by a domain admin) -Scheduled and manual backup of 1 to all vms and vm disks on primary storage (by a cloud admin) - this one is tougher to find a valid use case for -Backup schedules attached to service offerings I know I previously started a discussion about backing up multiple vm disks at once, but I think these use cases, broken down by user type (user, domain admin and admin), should help clear things up and show the utility of being able to backup multiple objects at once. Thanks! Chris -- Chris Suich chris.su...@netapp.com NetApp Software Engineer Data Center Platforms – Cloud Solutions Citrix, Cisco Red Hat
Re: Scalable Backup and Recovery
I agree this needs to be approached. I find it quite frustrating to have a VM with 4-5 volumes and not be able to backup all 5 volumes as a set. Plus the restore operation gets messy when you have to restore 5 volumes and re-attach them to the VM. The user and domain sets you discuss below I can also clearly see the use for. Thanks for starting this discussion. -Kelcey - Original Message - From: Christopher SuichII chris.su...@netapp.com To: dev@cloudstack.apache.org dev@cloudstack.apache.org Sent: Friday, September 27, 2013 10:27:29 AM Subject: Scalable Backup and Recovery I'd like to start a discussion around the direction of scalable backup and recovery in CloudStack. Currently, the only want to backup and recover vms is by setting up a schedule or manually snapshotting up individual vm disks or manually snapshotting vms. Unfortunately, I don't believe this is a very scalable solution. What if a user wants all of their vm disks to be backed up on the same schedule? What if a domain administrator wants all of the vms in their domain to be backed up on the same schedule or to manually backup every vm in their domain? Here are some use cases I see for helping to scale things up: -Scheduled and manual backup of 1 to all of a user's vms and vm disks -Scheduled and manual backup of 1 to all of a domain's vms and vm disks (by a domain admin) -Scheduled and manual backup of 1 to all vms and vm disks on primary storage (by a cloud admin) - this one is tougher to find a valid use case for -Backup schedules attached to service offerings I know I previously started a discussion about backing up multiple vm disks at once, but I think these use cases, broken down by user type (user, domain admin and admin), should help clear things up and show the utility of being able to backup multiple objects at once. Thanks! Chris -- Chris Suich chris.su...@netapp.com NetApp Software Engineer Data Center Platforms – Cloud Solutions Citrix, Cisco Red Hat
Re: Scalable Backup and Recovery
Ah I see. You mean a scalable user experience. The actual scalability of the snapshot process itself is limited by available disk and network bandwidth. Also certain hypervisors have various quirks which stand in the way of an efficient solution. On 9/27/13 10:27 AM, SuichII, Christopher chris.su...@netapp.com wrote: I'd like to start a discussion around the direction of scalable backup and recovery in CloudStack. Currently, the only want to backup and recover vms is by setting up a schedule or manually snapshotting up individual vm disks or manually snapshotting vms. Unfortunately, I don't believe this is a very scalable solution. What if a user wants all of their vm disks to be backed up on the same schedule? What if a domain administrator wants all of the vms in their domain to be backed up on the same schedule or to manually backup every vm in their domain? Here are some use cases I see for helping to scale things up: -Scheduled and manual backup of 1 to all of a user's vms and vm disks -Scheduled and manual backup of 1 to all of a domain's vms and vm disks (by a domain admin) -Scheduled and manual backup of 1 to all vms and vm disks on primary storage (by a cloud admin) - this one is tougher to find a valid use case for -Backup schedules attached to service offerings I know I previously started a discussion about backing up multiple vm disks at once, but I think these use cases, broken down by user type (user, domain admin and admin), should help clear things up and show the utility of being able to backup multiple objects at once. Thanks! Chris -- Chris Suich chris.su...@netapp.com NetApp Software Engineer Data Center Platforms Cloud Solutions Citrix, Cisco Red Hat
