I would hope that default behaviour in CloudStack is the traditional volume
snapshot moved to secondary storage. A global setting to change that
behaviour is probably ok if it is not default, but the user would want to
in certain cases make copies of those snapshots to secondary storage in
addition to taking the snapshot on the primary storage.
On Monday, February 8, 2016, Will Stevens <wstev...@cloudops.com> wrote:
> I don't think a global setting is a good option because we need both
> functionality to be available at the same time and for different use cases
> to be able to pick which they choose.
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Mon, Feb 8, 2016 at 1:48 PM, Mike Tutkowski <
> mike.tutkow...@solidfire.com <javascript:;>
> > wrote:
>
> > Now that I re-read your e-mail, it dawned on me: The end-user doesn't
> care
> > where the snapshot is.
> >
> > If that's true, then we should perhaps control this via Global Settings
> or
> > something.
> >
> > On Mon, Feb 8, 2016 at 11:46 AM, Mike Tutkowski <
> > mike.tutkow...@solidfire.com> wrote:
> >
> > > It's not ideal - true, but it does allow us to be backward compatible.
> > >
> > > If you have other ideas, though, about how to maintain backward
> > > compatibility, I'm definitely open to hear them.
> > >
> > > Thanks!
> > >
> > > On Mon, Feb 8, 2016 at 11:42 AM, Syed Mushtaq <syed1.mush...@gmail.com
> >
> > > wrote:
> > >
> > >> Hi Mike,
> > >>
> > >> Adding a flag to createSnapshot was the first and the most obvious
> thing
> > >> that came to our minds. The problem that I had with this was that:
> > >>
> > >> 1) I feel it is exposing something to the end user that is internal to
> > the
> > >> cloud.
> > >>
> > >> 2) We have to follow two different ways of restore/deletion in the
> same
> > >> code path depending on where the Snapshot resides which I find kind
> of a
> > >> bad design.
> > >>
> > >> But if exposing a archive flag to the end user is acceptable then we
> can
> > >> definitely use this instead of adding the StorageSnapshot API
> > >>
> > >> Thanks,
> > >> -Syed
> > >>
> > >>
> > >> On Mon, Feb 8, 2016 at 1:26 PM, Mike Tutkowski <
> > >> mike.tutkow...@solidfire.com
> > >> > wrote:
> > >>
> > >> > Hi Pierre-Luc,
> > >> >
> > >> > My recommendation would be this:
> > >> >
> > >> > Add an "archive" flag to the current volume-snapshot API. Its
> default
> > >> would
> > >> > be "false" because that would be backward compatible with how 4.6
> has
> > >> > volume snapshots implemented (i.e. they stay on the SAN in 4.6, 4.7,
> > and
> > >> > 4.8).
> > >> >
> > >> > If you set archive=true, then we would perform a background
> migration
> > of
> > >> > the snapshot from the SAN to the secondary storage (then delete the
> > SAN
> > >> > snapshot).
> > >> >
> > >> > That archive parameter would only be valid for managed storage.
> > >> >
> > >> > Sound reasonable?
> > >> >
> > >> > Also, a VM snapshot that includes disks provided by managed storage
> > >> should
> > >> > work.
> > >> >
> > >> > Talk to you later,
> > >> > Mike
> > >> >
> > >> > On Mon, Feb 8, 2016 at 9:22 AM, Pierre-Luc Dion <pd...@cloudops.com
> >
> > >> > wrote:
> > >> >
> > >> > > Mike,
> > >> > >
> > >> > > In terms of API's, would you prefer introducing a parameter to the
> > >> > existing
> > >> > > VolumeSnapshot, example: extract={true|false} with a default
> > value
> > >> of
> > >> > > true which would extract snapshot into the secondary storage,
> which
> > >> is
> > >> > the
> > >> > > current default behavior. Then for SAN snapshot that remain on the
> > >> SAN we
> > >> > > would just set "extract=false" ? as oppose to create a new
> > >> > > StorageSnapshot API ?
> > >> > >
> > >> > >
> > >> > > Paul,
> > >> > >
> > >> > > From what I'm seeing so far, we can't do a VM-snapshot when using
> > >> managed
> > >> > > storage for VM having more than one Volume. For the reason that
> > >> snapshot
> > >> > > are performed outside of the hypervisor awareness and
> > asynchronously.
> > >> If
> > >> > > someone have a way to address that, it would make thinks much more
> > >> > > attractive.
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Mon, Feb 8, 2016 at 10:57 AM, Ian Rae <i...@cloudops.com>
> wrote:
> > >> > >
> > >> > > > I think a service provider backup scenario is more likely to
> take
> > >> > > advantage
> > >> > > > of SAN snapshot. There are a few reasons for this. Traditional
> > >> backups
> > >> > > > involve access to the file system, and there is an expectation
> > that
> > >> > this
> > >> > > > can be done with reasonably short time frames without negatively
> > >> > > impacting
> > >> > > > VM performance, and that the backup orchestrator can apply
> various
> > >> > logic
> > >> > > > and or transformations to the data (compress, encrypt, deltas
> > >> etc...).
> > >> > > > While it is true that one could apply a backup process to a
> cloud
> > >> > > snapshot,
> > >> > > > this would be slow and inefficient requiring the data to be
> moved
> > >> > several
> > >> > > > times and there are some major bottlenecks with cloud snapshots.
> > >> With a
> > >> > > > cloud snapshot - there seems to be no reasonable expectation of
> > >> being
> > >> > > able
> > >> > > > to do differential snapshots (I think this depends on the
> > >> hypervisor)
> > >> > and
> > >> > > > if you do differential snapshots this will make file backups
> from
> > >> those
> > >> > > > snapshots even more complicated to orchestrate.
> > >> > > >
> > >> > > > Suspect there needs to be a different thread on how to better
> > enable
> > >> > > > backups, as a service. As per Paul's suggestion, but it is a
> > related
> > >> > > > workflow so relevant to this discussion.
> > >> > > >
> > >> > > > Ian
> > >> > > >
> > >> > > > On Monday, February 8, 2016, Mike Tutkowski <
> > >> > > mike.tutkow...@solidfire.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > To me it sounds like number two and number three are different
> > >> uses
> > >> > for
> > >> > > > the
> > >> > > > > same "thing"(which is totally fine).
> > >> > > > >
> > >> > > > > As for taking a fast SAN snapshot and exporting it
> > >> asynchronously, do
> > >> > > we
> > >> > > > > see the SSVM as performing the export?
> > >> > > > >
> > >> > > > > To be backwards compatible with what we have in 4.6 and later
> > for
> > >> > > volume
> > >> > > > > snapshots for managed storage, I think it might be easier if
> we
> > >> pass
> > >> > > in a
> > >> > > > > flag that says whether or not to archive the SAN snapshot
> > (which,
> > >> I
> > >> > > > think,
> > >> > > > > is something that you suggested, as well, Pierre-Luc).
> > >> > > > >
> > >> > > > > On Monday, February 8, 2016, Pierre-Luc Dion <
> > pd...@cloudops.com
> > >> > > > > <javascript:;>> wrote:
> > >> > > > >
> > >> > > > > > Hi Mike,
> > >> > > > > >
> > >> > > > > > The reason behind the creation of a SAN snapshot which is
> > >> exported
> > >> > > into
> > >> > > > > > secondary storage, is because creating a copy of the .VHD
> > >> directly
> > >> > > > would
> > >> > > > > > impact uptime of the VM as creating that copy take lots of
> > time.
> > >> > Has
> > >> > > > > oppose
> > >> > > > > > to a SAN snapshot that is close to instantaneous which can
> > >> > afterward
> > >> > > be
> > >> > > > > > clone into Secondary Storage asynchronously.
> > >> > > > > >
> > >> > > > > > I would suspect an extracted VolumeSnapshot taken from a SAN
> > >> > snapshot
> > >> > > > > could
> > >> > > > > > have is SAN snapshot deleted to avoid duplica and space
> > >> consumption
> > >> > > on
> > >> > > > > the
> > >> > > > > > Primary Storage side.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > I see 3 definitions in our current discussion regarding the
> > term
> > >> > > > snapshot
> > >> > > > > > (these are not official terminology but by own
> interpretation
> > of
> > >> > > them):
> > >> > > > > >
> > >> > > > > > 1. *Snapshot* (AKA: Storage Snapshot / Mike's definition of
> a
> > >> > > > snapshot):
> > >> > > > > > it's a volume snapshot at the storage level, point in time
> of
> > >> your
> > >> > > > data.
> > >> > > > > it
> > >> > > > > > reside on the primary storage. Useful and efficient for
> > software
> > >> > side
> > >> > > > > > incident.
> > >> > > > > > 2. *Cloud Snapshot *( AKA: CloudStack VolumeSnapshot/ cloud
> > >> backup
> > >> > > > aws-S3
> > >> > > > > > style ): Point in time copy of the Virtual Disk that reside
> > on a
> > >> > > > > different
> > >> > > > > > storage array then the original Volume. Facilitate data
> > >> migration
> > >> > > > between
> > >> > > > > > clusters and, in case of primary storage incident, Volume
> > >> snapshots
> > >> > > are
> > >> > > > > not
> > >> > > > > > impacted and can be reuse.
> > >> > > > > > 3. *Backup*: Archival of your Virtual-machines data that
> also
> > >> > > validate
> > >> > > > > data
> > >> > > > > > integrity, provide a storage efficient archiving method and
> an
> > >> > > > > independent
> > >> > > > > > way to restore your data in case of an major infrastructure
> > >> > disaster.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Regards,
> > >> > > > > >
> > >> > > > > > PL
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Fri, Feb 5, 2016 at 1:34 PM, Mike Tutkowski <
> > >> > > > > > mike.tutkow...@solidfire.com <javascript:;> <javascript:;>
> > >> > > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > So, let's see if I currently follow the requirements:
> > >> > > > > > >
> > >> > > > > > > * Augment volume snapshots for managed storage to
> > >> conditionally
> > >> > > > export
> > >> > > > > > data
> > >> > > > > > > to NFS. The current process of taking a snapshot on the
> SAN
> > is
> > >> > > fine,
> > >> > > > > but
> > >> > > > > > > we'd like the option to export the data to NFS, as well.
> > >> > > > > > >
> > >> > > > > > > Questions:
> > >> > > > > > >
> > >> > > > > > > Once the data has been exported to NFS, do we keep the SAN
> > >> > snapshot
> > >> > > > or
> > >> > > > > > > delete it?
> > >> > > > > > >
> > >> > > > > > > If we are deleting the SAN snapshot, then why don't we
> just
> > >> copy
> > >> > > the
> > >> > > > > VHD
> > >> > > > > > > from primary to secondary the way we do today for
> > non-managed
> > >> > (i.e.
> > >> > > > > > > traditional) storage? Why create a SAN snapshot in that
> > >> scenario?
> > >> > > > > Perhaps
> > >> > > > > > > to have the SSVM mount and perform the VHD copy to
> secondary
> > >> > > storage
> > >> > > > > > > instead of a XenServer host?
> > >> > > > > > >
> > >> > > > > > > Thanks for the clarification.
> > >> > > > > > >
> > >> > > > > > > By the way, to me a backup is when you copy data from one
> > >> storage
> > >> > > > > system
> > >> > > > > > to
> > >> > > > > > > another (regardless of features, if any, to restore the
> data
> > >> in
> > >> > the
> > >> > > > > > > future). A snapshot is a point-in-time view of the data
> of a
> > >> > volume
> > >> > > > and
> > >> > > > > > > it's stored on the same storage system as the volume.
> > >> > > > > > >
> > >> > > > > > > On Fri, Feb 5, 2016 at 11:09 AM, Pierre-Luc Dion <
> > >> > > pd...@cloudops.com
> > >> > > > > <javascript:;>
> > >> > > > > > <javascript:;>>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > That's fun to see that discussion happening. I 100%
> agree
> > >> with
> > >> > > > Paul's
> > >> > > > > > > > points of view. VolumeSnapshot are not a backup, but I
> do
> > >> > > consider
> > >> > > > > them
> > >> > > > > > > as
> > >> > > > > > > > a safety vest against Primary Storage failure, because
> > >> failure
> > >> > > > append
> > >> > > > > > > :-( .
> > >> > > > > > > >
> > >> > > > > > > > The current proposal around snapshots that reside on the
> > >> > primary
> > >> > > > > > storage
> > >> > > > > > > or
> > >> > > > > > > > snapshots that end in the Secondary Storage is not to
> > >> address
> > >> > > any
> > >> > > > > kind
> > >> > > > > > > of
> > >> > > > > > > > backups requirement because a snapshot is not a backup,
> > >> event
> > >> > an
> > >> > > > > > > extracted
> > >> > > > > > > > VM snapshot.
> > >> > > > > > > >
> > >> > > > > > > > The main idea, and again this is for managed storage;
> > >> > > > > > > >
> > >> > > > > > > > 1. StorageSnapshotAPI: Provide storage side snapshot
> > >> capability
> > >> > > for
> > >> > > > > > fast
> > >> > > > > > > > response time that support rollback to previous
> timestamp,
> > >> > create
> > >> > > > new
> > >> > > > > > > > volume and maybe create template.
> > >> > > > > > > > not required to be a new API if the work is already
> > >> done, I
> > >> > > > think
> > >> > > > > > > this
> > >> > > > > > > > is a different behaviors than the user expectation of a
> > >> > > > > > volume-snapshot.
> > >> > > > > > > > 2. VolumeSnapshotAPI: Provide current cloudstack
> behavior
> > >> that
> > >> > > > create
> > >> > > > > > an
> > >> > > > > > > > extraction of a volume into SecondaryStorage which can
> be
> > >> reuse
> > >> > > to
> > >> > > > > > > create a
> > >> > > > > > > > new volume into another Primary Storage. This type of
> > >> snapshot
> > >> > > is a
> > >> > > > > > slow
> > >> > > > > > > > job since yes it would have to copy the full volume size
> > on
> > >> the
> > >> > > > > > Secondary
> > >> > > > > > > > storage.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > PL
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Fri, Feb 5, 2016 at 12:45 PM, Syed Mushtaq <
> > >> > > > > syed1.mush...@gmail.com <javascript:;>
> > >> > > > > > <javascript:;>>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > I think I share you view on the 'Ideal world'. Backup
> > (via
> > >> > > Volume
> > >> > > > > > > > > Snapshots) is a huge bottleneck in Cloudstack. This is
> > >> > > amplified
> > >> > > > > > > > especially
> > >> > > > > > > > > when you have a object storage as your secondary
> storage
> > >> > > because
> > >> > > > it
> > >> > > > > > > > > requires two copies (one to an NFS staging area and
> from
> > >> > there
> > >> > > to
> > >> > > > > > > object
> > >> > > > > > > > > storage). And not to mention that all these copies are
> > >> > > consuming
> > >> > > > > > > > hypervisor
> > >> > > > > > > > > resources. Xenserver's Dom0 is also a huge bottleneck
> as
> > >> all
> > >> > > the
> > >> > > > > > > Network
> > >> > > > > > > > > and I/O flow through it. So our intention of proposing
> > the
> > >> > > > "Storage
> > >> > > > > > > > > Snapshots" is to give a better way of achiving
> snapshots
> > >> > while
> > >> > > > > still
> > >> > > > > > > > > keeping the original definition of volume snpashots
> (ie
> > >> > upload
> > >> > > to
> > >> > > > > sec
> > >> > > > > > > > > storage).
> > >> > > > > > > > >
> > >> > > > > > > > > But as Erik pointed out volume snapshots are not
> > backups.
> > >> > They
> > >> > > > > don't
> > >> > > > > > > work
> > >> > > > > > > > > form multi-disk LVM volume groups and dynamic disks. I
> > am
> > >> all
> > >> > > in
> > >> > > > > for
> > >> > > > > > a
> > >> > > > > > > > > better backup solution which handles these use cases
> and
> > >> > > utilizes
> > >> > > > > the
> > >> > > > > > > > > storage's advanced features.
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Fri, Feb 5, 2016 at 12:29 PM, Paul Angus <
> > >> > > > > > paul.an...@shapeblue.com <javascript:;> <javascript:;>>
> > >> > > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > In the beginning... there were CloudStack snapshots
> > and
> > >> > they
> > >> > > > were
> > >> > > > > > > > > actually
> > >> > > > > > > > > > volume snapshots not hypervisor point-in-time
> > snapshots.
> > >> > > > > > > > > > Then VM snapshots were created (which are
> > point-in-time
> > >> > > > > hypervisor
> > >> > > > > > > > > > snapshots) and we started referring to the original
> > >> > snapshots
> > >> > > > as
> > >> > > > > > > volume
> > >> > > > > > > > > > snapshots.
> > >> > > > > > > > > >
> > >> > > > > > > > > > CloudStack does not offer 'backups', but many people
> > use
> > >> > > volume
> > >> > > > > > > > snapshots
> > >> > > > > > > > > > as backups. However you can't in-place restore
> volume
> > >> > > snapshots
> > >> > > > > and
> > >> > > > > > > if
> > >> > > > > > > > > you
> > >> > > > > > > > > > have a VM with multiple volumes, the volume
> snapshots
> > >> must
> > >> > be
> > >> > > > > done
> > >> > > > > > in
> > >> > > > > > > > > > series, meaning that the state across all of the
> > >> volumes is
> > >> > > > > > unlikely
> > >> > > > > > > to
> > >> > > > > > > > > be
> > >> > > > > > > > > > consistent.
> > >> > > > > > > > > >
> > >> > > > > > > > > > 'Actual Backups' would enable all of the restore
> > options
> > >> > > which
> > >> > > > > > users
> > >> > > > > > > > > might
> > >> > > > > > > > > > expect as well options as to where they might be
> > >> stored. In
> > >> > > my
> > >> > > > > > ideal
> > >> > > > > > > > > world
> > >> > > > > > > > > > they would also be able to leverage back-end
> hardware
> > >> (such
> > >> > > as
> > >> > > > > > > > Solidfire,
> > >> > > > > > > > > > NetApp etc :) ) and software such as Veeam,
> Commvault
> > >> etc
> > >> > to
> > >> > > > > > > accelerate
> > >> > > > > > > > > the
> > >> > > > > > > > > > process.
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> > >> > > > > > > > > > Paul Angus
> > >> > > > > > > > > > VP Technology , ShapeBlue
> > >> > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > >> > > > > > > > > >
> > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540> |
> > >> > > > > m:
> > >> > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > >> > > > > > > > > > e: *paul.an...@shapeblue.com <javascript:;>
> > >> > <javascript:;>
> > >> > > |
> > >> > > > > t: @cloudyangus*
> > >> > > > > > > > > > <paul.an...@shapeblue.com <javascript:;>
> > >> > > > > <javascript:;>%20%7C%20t:%20@cloudyangus>
> > >> > > > > > | w:
> > >> > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > >> > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS
> UK
> > >> > > > > > > > > > Shape Blue Ltd is a company incorporated in England
> &
> > >> > Wales.
> > >> > > > > > > ShapeBlue
> > >> > > > > > > > > > Services India LLP is a company incorporated in
> India
> > >> and
> > >> > is
> > >> > > > > > operated
> > >> > > > > > > > > under
> > >> > > > > > > > > > license from Shape Blue Ltd. Shape Blue Brasil
> > >> Consultoria
> > >> > > Ltda
> > >> > > > > is
> > >> > > > > > a
> > >> > > > > > > > > > company incorporated in Brasil and is operated under
> > >> > license
> > >> > > > from
> > >> > > > > > > Shape
> > >> > > > > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company
> registered
> > >> by
> > >> > The
> > >> > > > > > > Republic
> > >> > > > > > > > of
> > >> > > > > > > > > > South Africa and is traded under license from Shape
> > Blue
> > >> > Ltd.
> > >> > > > > > > ShapeBlue
> > >> > > > > > > > > is
> > >> > > > > > > > > > a registered trademark.
> > >> > > > > > > > > > This email and any attachments to it may be
> > confidential
> > >> > and
> > >> > > > are
> > >> > > > > > > > intended
> > >> > > > > > > > > > solely for the use of the individual to whom it is
> > >> > addressed.
> > >> > > > Any
> > >> > > > > > > views
> > >> > > > > > > > > or
> > >> > > > > > > > > > opinions expressed are solely those of the author
> and
> > do
> > >> > not
> > >> > > > > > > > necessarily
> > >> > > > > > > > > > represent those of Shape Blue Ltd or related
> > companies.
> > >> If
> > >> > > you
> > >> > > > > are
> > >> > > > > > > not
> > >> > > > > > > > > the
> > >> > > > > > > > > > intended recipient of this email, you must neither
> > take
> > >> any
> > >> > > > > action
> > >> > > > > > > > based
> > >> > > > > > > > > > upon its contents, nor copy or show it to anyone.
> > Please
> > >> > > > contact
> > >> > > > > > the
> > >> > > > > > > > > sender
> > >> > > > > > > > > > if you believe you have received this email in
> error.
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > -----Original Message-----
> > >> > > > > > > > > > From: Syed Mushtaq [mailto:syed1.mush...@gmail.com
> > >> > > > > <javascript:;> <javascript:;>]
> > >> > > > > > > > > > Sent: Friday, February 5, 2016 4:58 PM
> > >> > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > >> > <javascript:;>
> > >> > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> Snapshots
> > >> > > > > > > > > >
> > >> > > > > > > > > > Paul,
> > >> > > > > > > > > >
> > >> > > > > > > > > > When you say actual backups, how would it be
> different
> > >> from
> > >> > > the
> > >> > > > > > > Volume
> > >> > > > > > > > > > Snapshots that exist currently. My understanding is
> > that
> > >> > > > Backups
> > >> > > > > > end
> > >> > > > > > > up
> > >> > > > > > > > > in
> > >> > > > > > > > > > Sec Storage whereas Snapshots are just a
> point-in-time
> > >> > state
> > >> > > of
> > >> > > > > > your
> > >> > > > > > > > > volume
> > >> > > > > > > > > > which can be restored back correct?
> > >> > > > > > > > > >
> > >> > > > > > > > > > -Syed
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Fri, Feb 5, 2016 at 11:23 AM, Paul Angus <
> > >> > > > > > > paul.an...@shapeblue.com <javascript:;> <javascript:;>>
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Hi Syed,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > As I understand it, the SolidFire plugin will
> export
> > >> the
> > >> > > > > snapshot
> > >> > > > > > > to
> > >> > > > > > > > > > > secondary storage if the user requests a template
> > from
> > >> > the
> > >> > > > > > snapshot
> > >> > > > > > > > or
> > >> > > > > > > > > > > wants to download the snapshot from the cloud.
> This
> > >> is a
> > >> > > > good,
> > >> > > > > > > > > > > pragmatic approach and yes Mike the SolidFire
> > storage
> > >> is
> > >> > > > super
> > >> > > > > > > > > > > reliable and snapshots on SolidFire arrays take up
> > >> next
> > >> > to
> > >> > > no
> > >> > > > > > > space.
> > >> > > > > > > > > > > BUT I think that we are talking about a more
> general
> > >> > > purpose
> > >> > > > > API,
> > >> > > > > > > and
> > >> > > > > > > > > > > other storage systems may not be as awesome as
> > Mike's.
> > >> > > That's
> > >> > > > > my
> > >> > > > > > > > > > > concern. Also, the time to transfer for say 1TB to
> > >> move
> > >> > > from
> > >> > > > > > > primary
> > >> > > > > > > > > > > to sec storage and then create a VM template out
> of
> > it
> > >> > may
> > >> > > be
> > >> > > > > too
> > >> > > > > > > > long
> > >> > > > > > > > > > for users.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > @Mike I don’t think 'we' use the term volume
> > snapshot
> > >> for
> > >> > > > > backup,
> > >> > > > > > > > it's
> > >> > > > > > > > > > > just that users want to do backups and a volume
> > >> snapshot
> > >> > is
> > >> > > > the
> > >> > > > > > > only
> > >> > > > > > > > > > > type of snapshot that copies the disk elsewhere
> and
> > >> can
> > >> > be
> > >> > > > > > > scheduled.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I'm 'pondering' the implications of enabling
> actual
> > >> > backups
> > >> > > > > > > (through
> > >> > > > > > > > > > > recognised backup providers) and the user
> > requirements
> > >> > > around
> > >> > > > > > them
> > >> > > > > > > > > > > (particularly restoration use cases) as a separate
> > >> thread
> > >> > > of
> > >> > > > > > work.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com>
> Paul
> > >> Angus
> > >> > > VP
> > >> > > > > > > > Technology
> > >> > > > > > > > > > > , ShapeBlue
> > >> > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > >> > > > > > > > > > >
> > >> > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > >> > > > |
> > >> > > > > m:
> > >> > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > >> > > > > > > > > > > e: *paul.an...@shapeblue.com <javascript:;>
> > >> > > <javascript:;> |
> > >> > > > > t: @cloudyangus*
> > >> > > > > > > > > > > <paul.an...@shapeblue.com <javascript:;>
> > >> <javascript:;>
> > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > >> > > > > > > > > > > *www.shapeblue.com* <http://www.shapeblue.com>
> > >> > > > > > > > > > > a: 53 Chandos Place, Covent Garden London WC2N 4HS
> > UK
> > >> > Shape
> > >> > > > > Blue
> > >> > > > > > > Ltd
> > >> > > > > > > > > > > is a company incorporated in England & Wales.
> > >> ShapeBlue
> > >> > > > > Services
> > >> > > > > > > > India
> > >> > > > > > > > > > > LLP is a company incorporated in India and is
> > operated
> > >> > > under
> > >> > > > > > > license
> > >> > > > > > > > > > > from Shape Blue Ltd. Shape Blue Brasil Consultoria
> > >> Ltda
> > >> > is
> > >> > > a
> > >> > > > > > > company
> > >> > > > > > > > > > > incorporated in Brasil and is operated under
> license
> > >> from
> > >> > > > Shape
> > >> > > > > > > Blue
> > >> > > > > > > > > > > Ltd. ShapeBlue SA Pty Ltd is a company registered
> by
> > >> The
> > >> > > > > Republic
> > >> > > > > > > of
> > >> > > > > > > > > > > South Africa and is traded under license from
> Shape
> > >> Blue
> > >> > > Ltd.
> > >> > > > > > > > > > > ShapeBlue is a registered trademark.
> > >> > > > > > > > > > > This email and any attachments to it may be
> > >> confidential
> > >> > > and
> > >> > > > > are
> > >> > > > > > > > > > > intended solely for the use of the individual to
> > whom
> > >> it
> > >> > is
> > >> > > > > > > > addressed.
> > >> > > > > > > > > > > Any views or opinions expressed are solely those
> of
> > >> the
> > >> > > > author
> > >> > > > > > and
> > >> > > > > > > do
> > >> > > > > > > > > > > not necessarily represent those of Shape Blue Ltd
> or
> > >> > > related
> > >> > > > > > > > > > > companies. If you are not the intended recipient
> of
> > >> this
> > >> > > > email,
> > >> > > > > > you
> > >> > > > > > > > > > > must neither take any action based upon its
> > contents,
> > >> nor
> > >> > > > copy
> > >> > > > > or
> > >> > > > > > > > show
> > >> > > > > > > > > > > it to anyone. Please contact the sender if you
> > believe
> > >> > you
> > >> > > > have
> > >> > > > > > > > > received
> > >> > > > > > > > > > this email in error.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > -----Original Message-----
> > >> > > > > > > > > > > From: Syed Mushtaq [mailto:
> syed1.mush...@gmail.com
> > >> > > > > <javascript:;>
> > >> > > > > > <javascript:;>]
> > >> > > > > > > > > > > Sent: 05 February 2016 15:31
> > >> > > > > > > > > > > To: dev@cloudstack.apache.org <javascript:;>
> > >> > > <javascript:;>
> > >> > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > Snapshots
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > I think the terminology confusion comes from AWS
> > where
> > >> > they
> > >> > > > do
> > >> > > > > > EBS
> > >> > > > > > > > > > > snapshots backed up to S3 and CloudStack sort of
> > >> followed
> > >> > > > that.
> > >> > > > > > And
> > >> > > > > > > > as
> > >> > > > > > > > > > > an end user who is oblivious to the internals of
> my
> > >> > > provider,
> > >> > > > > my
> > >> > > > > > > > > > > expectation would be something similar to what AWS
> > as
> > >> > that
> > >> > > is
> > >> > > > > my
> > >> > > > > > > > > > > biggest reference point.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > To your point Mike, I agree that a Primary Storage
> > >> > failure
> > >> > > on
> > >> > > > > > > > > > > SolidFire is unlikely, there are other motivations
> > >> for us
> > >> > > to
> > >> > > > > push
> > >> > > > > > > > data
> > >> > > > > > > > > > > to secondary storage. Primary storage (atleast for
> > us)
> > >> > > costs
> > >> > > > > > > around 3
> > >> > > > > > > > > > > times as much as secondary storage and snapshots
> on
> > >> > primary
> > >> > > > > > storage
> > >> > > > > > > > > > > are rarely used (especially for some of our
> > customers
> > >> who
> > >> > > do
> > >> > > > > > daily
> > >> > > > > > > > > > backups).
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Fri, Feb 5, 2016 at 10:18 AM, Mike Tutkowski <
> > >> > > > > > > > > > > mike.tutkow...@solidfire.com <javascript:;>
> > >> > > <javascript:;>>
> > >> > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > Some of the weirdness is around terminology.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > For most systems I've worked on, a snapshot and
> a
> > >> > backup
> > >> > > > are
> > >> > > > > > two
> > >> > > > > > > > > > > > completely different things (but CloudStack has
> > >> > > > traditionally
> > >> > > > > > > used
> > >> > > > > > > > > > > > the term "volume snapshot" to mean backup).
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > I will put in a SolidFire "plug" here and say,
> > >> though,
> > >> > > that
> > >> > > > > if
> > >> > > > > > > your
> > >> > > > > > > > > > > > primary storage is running on SolidFire that it
> is
> > >> > > unlikely
> > >> > > > > > > you'll
> > >> > > > > > > > > > > > encounter an issue where your primary storage
> goes
> > >> > > offline
> > >> > > > > (and
> > >> > > > > > > > > > > > you'll even maintain your performance guarantees
> > >> during
> > >> > > > > failure
> > >> > > > > > > > > > > > scenarios and upgrades, as well). That being the
> > >> case,
> > >> > it
> > >> > > > is
> > >> > > > > > less
> > >> > > > > > > > > > > > useful to require a backup to Swift (but it's
> > >> perfectly
> > >> > > OK
> > >> > > > if
> > >> > > > > > > > that's
> > >> > > > > > > > > > > > what we want to do
> > >> > > > > > > > > > > here).
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > On Fri, Feb 5, 2016 at 8:07 AM, Syed Mushtaq
> > >> > > > > > > > > > > > <syed1.mush...@gmail.com <javascript:;>
> > >> > <javascript:;>>
> > >> > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > Paul,
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > I believe with the current implementation of
> > >> > Snapshots
> > >> > > on
> > >> > > > > > > managed
> > >> > > > > > > > > > > > > storage
> > >> > > > > > > > > > > > > (SolidFire) the snapshots are never exported
> to
> > >> the
> > >> > > > > secondary
> > >> > > > > > > > > > storage.
> > >> > > > > > > > > > > > > While this solves the problem of having
> > snapshots
> > >> > > taking
> > >> > > > > > > forever
> > >> > > > > > > > > > > > > to get to sec storage, this leaves us with a
> > >> > > > > > > > > > > > huge
> > >> > > > > > > > > > > > > liability if our primary storage goes down. We
> > see
> > >> > > > > snapshots
> > >> > > > > > as
> > >> > > > > > > > > > > > > our recovery path because we store them in
> Swift
> > >> > which
> > >> > > is
> > >> > > > > > > > reliable
> > >> > > > > > > > > > > > > and resilient to failures.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > With Storage snpashots our goal is to have
> > Volume
> > >> > > > snapshots
> > >> > > > > > > > always
> > >> > > > > > > > > > > > > backed up to secondary storage and Storage
> > >> Snapshots
> > >> > > stay
> > >> > > > > on
> > >> > > > > > > the
> > >> > > > > > > > > > > > > primary
> > >> > > > > > > > > > > > storage.
> > >> > > > > > > > > > > > > A provider could potentially mix both these
> and
> > >> solve
> > >> > > the
> > >> > > > > > > problem
> > >> > > > > > > > > > > > > that you mentioned where you want to meet
> user's
> > >> > > > > expectation
> > >> > > > > > > of a
> > >> > > > > > > > > > > > > snapshot (ie backup to sec storage) while
> having
> > >> an
> > >> > > > ability
> > >> > > > > > to
> > >> > > > > > > > > > > > > utilize faster sanpshots (i.e. on the device)
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Hope this clarifies things.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > > > -Syed
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > On Fri, Feb 5, 2016 at 9:04 AM, Paul Angus
> > >> > > > > > > > > > > > > <paul.an...@shapeblue.com <javascript:;>
> > >> > > <javascript:;>>
> > >> > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > HI guys,
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Could someone point me to the Jira bug of FS
> > for
> > >> > the
> > >> > > > > > > > > > > > > > SAN-snapshot
> > >> > > > > > > > > > > > feature
> > >> > > > > > > > > > > > > > in 4.6 which is mentioned.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > From my discussions with users and operators
> > >> around
> > >> > > > > > snapshots
> > >> > > > > > > > > > > > > > I'd make
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > following observations:
> > >> > > > > > > > > > > > > > a. 'users' use snapshots as backups (both
> > >> long-term
> > >> > > and
> > >> > > > > > short
> > >> > > > > > > > > > > > > > term)
> > >> > > > > > > > > > > > with
> > >> > > > > > > > > > > > > > the expectation that they can use them for
> > >> recovery
> > >> > > if
> > >> > > > > > > > required.
> > >> > > > > > > > > > > > > > b. operators fall back to snapshots if
> > something
> > >> > has
> > >> > > > gone
> > >> > > > > > > wrong
> > >> > > > > > > > > > > > > > with primary storage.
> > >> > > > > > > > > > > > > > c. users sometimes want to be able to export
> > >> > > snapshots
> > >> > > > as
> > >> > > > > > > well
> > >> > > > > > > > > > > > > > as
> > >> > > > > > > > > > > > create
> > >> > > > > > > > > > > > > > new VMs from their snapshots
> > >> > > > > > > > > > > > > > d. snapshots are a currently a massive pain
> > for
> > >> > > > > operators,
> > >> > > > > > I
> > >> > > > > > > > > > > > > > know at
> > >> > > > > > > > > > > > > least
> > >> > > > > > > > > > > > > > one public cloud who have snapshots which
> > take 2
> > >> > days
> > >> > > > to
> > >> > > > > > > > > complete.
> > >> > > > > > > > > > > > > > e. snapshots (as they are) can't be used for
> > >> > multiple
> > >> > > > LVM
> > >> > > > > > > > disks.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > I think the process Mike has used in the
> > >> SolidFire
> > >> > > > plugin
> > >> > > > > > > (only
> > >> > > > > > > > > > > > > > moving
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > disk image to secondary storage when you
> > >> absolutely
> > >> > > > have
> > >> > > > > > to)
> > >> > > > > > > is
> > >> > > > > > > > > > > > > > a very
> > >> > > > > > > > > > > > > good
> > >> > > > > > > > > > > > > > and pragmatic solution. I wonder what
> problems
> > >> an
> > >> > > > > operator
> > >> > > > > > > > might
> > >> > > > > > > > > > > > > experience
> > >> > > > > > > > > > > > > > if they have an issue with a given primary
> > >> storage
> > >> > > pool
> > >> > > > > in
> > >> > > > > > a
> > >> > > > > > > > > > cluster.
> > >> > > > > > > > > > > > (I
> > >> > > > > > > > > > > > > > know that that is REALLY unlikely in the
> > >> SolidFire
> > >> > > case
> > >> > > > > > Mike
> > >> > > > > > > :)
> > >> > > > > > > > > > > > > > ) And
> > >> > > > > > > > > > > > if
> > >> > > > > > > > > > > > > > the transfer from primary to secondary is
> > slow,
> > >> the
> > >> > > > time
> > >> > > > > to
> > >> > > > > > > > > > > > > > being able
> > >> > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > create a template or export the volume will
> be
> > >> > slow.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > So for me the issue is around making sure
> that
> > >> the
> > >> > > end
> > >> > > > > > users
> > >> > > > > > > > > > > > expectations
> > >> > > > > > > > > > > > > > are met (while improving the
> speed/efficiency
> > of
> > >> > the
> > >> > > > back
> > >> > > > > > > end)
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > [image: ShapeBlue] <
> http://www.shapeblue.com>
> > >> Paul
> > >> > > > Angus
> > >> > > > > > VP
> > >> > > > > > > > > > > > > > Technology , ShapeBlue
> > >> > > > > > > > > > > > > > d: *+44 203 617 0528 | s: +44 203 603 0540*
> > >> > > > > > > > > > > > > >
> > >> > > > > > <+44%20203%20617%200528%20%7C%20s:%20+44%20203%20603%200540>
> > >> > > > > > > |
> > >> > > > > > > > m:
> > >> > > > > > > > > > > > > > *+44 7711 418784* <+44%207711%20418784>
> > >> > > > > > > > > > > > > > e: *paul.an...@shapeblue.com <javascript:;>
> > >> > > > > <javascript:;> | t:
> > >> > > > > > @cloudyangus*
> > >> > > > > > > > > > > > > > <paul.an...@shapeblue.com <javascript:;>
> > >> > > > <javascript:;>
> > >> > > > > > %20%7C%20t:%20@cloudyangus> | w:
> > >> > > > > > > > > > > > > > *www.shapeblue.com* <
> http://www.shapeblue.com
> > >
> > >> > > > > > > > > > > > > > a: 53 Chandos Place, Covent Garden London
> WC2N
> > >> 4HS
> > >> > UK
> > >> > > > > Shape
> > >> > > > > > > > Blue
> > >> > > > > > > > > > > > > > Ltd is a company incorporated in England &
> > >> Wales.
> > >> > > > > ShapeBlue
> > >> > > > > > > > > > > > > > Services India LLP is a company incorporated
> > in
> > >> > India
> > >> > > > and
> > >> > > > > > is
> > >> > > > > > > > > > > > > > operated
> > >> > > > > > > > > > > > > under
> > >> > > > > > > > > > > > > > license from Shape Blue Ltd. Shape Blue
> Brasil
> > >> > > > > Consultoria
> > >> > > > > > > Ltda
> > >> > > > > > > > > > > > > > is a company incorporated in Brasil and is
> > >> operated
> > >> > > > under
> > >> > > > > > > > > > > > > > license from Shape Blue Ltd. ShapeBlue SA
> Pty
> > >> Ltd
> > >> > is
> > >> > > a
> > >> > > > > > > company
> > >> > > > > > > > > > > > > > registered by The Republic
> > >> > > > > > > > > > > > of
> > >> > > > > > > > > > > > > > South Africa and is traded under license
> from
> > >> Shape
> > >> > > > Blue
> > >> > > > > > Ltd.
> > >> > > > > > > > > > > > > > ShapeBlue
> > >> > > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > a registered trademark.
> > >> > > > > > > > > > > > > > This email and any attachments to it may be
> > >> > > > confidential
> > >> > > > > > and
> > >> > > > > > > > are
> > >> > > > > > > > > > > > intended
> > >> > > > > > > > > > > > > > solely for the use of the individual to whom
> > it
> > >> is
> > >> > > > > > addressed.
> > >> > > > > > > > > > > > > > Any views
> > >> > > > > > > > > > > > > or
> > >> > > > > > > > > > > > > > opinions expressed are solely those of the
> > >> author
> > >> > and
> > >> > > > do
> > >> > > > > > not
> > >> > > > > > > > > > > > necessarily
> > >> > > > > > > > > > > > > > represent those of Shape Blue Ltd or related
> > >> > > companies.
> > >> > > > > If
> > >> > > > > > > you
> > >> > > > > > > > > > > > > > are not
> > >> > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > intended recipient of this email, you must
> > >> neither
> > >> > > take
> > >> > > > > any
> > >> > > > > > > > > > > > > > action
> > >> > > > > > > > > > > > based
> > >> > > > > > > > > > > > > > upon its contents, nor copy or show it to
> > >> anyone.
> > >> > > > Please
> > >> > > > > > > > contact
> > >> > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > sender
> > >> > > > > > > > > > > > > > if you believe you have received this email
> in
> > >> > error.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > -----Original Message-----
> > >> > > > > > > > > > > > > > From: Pierre-Luc Dion [mailto:
> > >> pd...@cloudops.com
> > >> > > > > <javascript:;>
> > >> > > > > > <javascript:;>]
> > >> > > > > > > > > > > > > > Sent: Friday, February 5, 2016 12:56 PM
> > >> > > > > > > > > > > > > > To: dev@cloudstack.apache.org
> <javascript:;>
> > >> > > > > <javascript:;>
> > >> > > > > > > > > > > > > > Subject: Re: [Propose][New Feature] Storage
> > >> > Snapshots
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Hi Mike,
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > The idea of introducing a new API:
> > >> StorageSnapshot
> > >> > > for
> > >> > > > > > > managed
> > >> > > > > > > > > > > > > > storage
> > >> > > > > > > > > > > > is
> > >> > > > > > > > > > > > > > because the VolumeSnapshot default, or
> > expected,
> > >> > > > behavior
> > >> > > > > > is
> > >> > > > > > > to
> > >> > > > > > > > > > > > > > archive snapshots into the Secondary
> Storage.
> > >> So a
> > >> > > > > > > > > > > > > > StorageSnapshot API would be
> > >> > > > > > > > > > > > > for
> > >> > > > > > > > > > > > > > snapshot that remain on the managed storage
> > >> > > appliance.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Quickly looking at the API doc and I don't
> > see a
> > >> > > strong
> > >> > > > > > > > > > > > > > requirement for volume snapshots to be moved
> > >> into
> > >> > > > > secondary
> > >> > > > > > > > > > > > > > storage. So, maybe StorageSnapshot API is
> not
> > >> > useful,
> > >> > > > but
> > >> > > > > > > both
> > >> > > > > > > > > > > > > > use cases are required. A snapshot that
> remain
> > >> on
> > >> > the
> > >> > > > > > managed
> > >> > > > > > > > > > > > > > storage, and another type of
> > >> > > > > > > > > > > > snapshot
> > >> > > > > > > > > > > > > > that end up into the secondary storage.
> Since
> > >> > you've
> > >> > > > > done a
> > >> > > > > > > lot
> > >> > > > > > > > > > > > > > of
> > >> > > > > > > > > > > > work,
> > >> > > > > > > > > > > > > > might easier to just add a parameter to the
> > >> current
> > >> > > > > > snapshot
> > >> > > > > > > > API
> > >> > > > > > > > > > > > > > that
> > >> > > > > > > > > > > > > would
> > >> > > > > > > > > > > > > > trigger an extraction of the storage
> snapshot
> > >> into
> > >> > > the
> > >> > > > > > > > secondary
> > >> > > > > > > > > > > > storage?
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > PL
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 9:02 PM, Mike
> > Tutkowski <
> > >> > > > > > > > > > > > > > mike.tutkow...@solidfire.com <javascript:;>
> > >> > > > > <javascript:;>
> > >> > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > I think that all sounds reasonable then -
> > >> thanks!
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > On Thu, Feb 4, 2016 at 6:52 PM, Syed
> > Mushtaq <
> > >> > > > > > > > > > > > syed1.mush...@gmail.com <javascript:;>
> > >> <javascript:;>>
> > >> > > > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >> You are correct Mike in terms of the
> > >> > requirements.
> > >> > > > One
> > >> > > > > > of
> > >> > > > > > > > our
> > >> > > > > > > > > > > > earlier
> > >> > > > > > > > > > > > > > >> iterations on this was to have an
> argument
> > to
> > >> > the
> > >> > > > > create
> > >> > > > > > > > > > > > > > >> snapshot
> > >> > > > > > > > > > > > API
> > >> > > > > > > > > > > > > > >> which decides whether to backup the
> volume
> > to
> > >> > sec
> > >> > > > > > storage
> > >> > > > > > > > but
> > >> > > > > > > > > > > > > > >> we realized it would make management of
> > >> > snapshots
> > >> > > > > quite
> > >> > > > > > > > messy
> > >> > > > > > > > > > > > > > >> so we proposed a new api instead.
> > >> > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > >> On Thu, Feb 4, 2016, 8:29 PM Mike
> Tutkowski
> > >> > > > > > > > > > > > > > >> <mike.tutkow...@solidfire.com
> > <javascript:;>
> > >> > > > > <javascript:;>>
> > >> > > > > > > > > > > > > > >> wrote:
> > >> > > > > > > > > > > > > > >>
> > >> > > > > > > > > > > > > > >>> Hi,
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> Just to make sure I understand all the
> > >> > > requirements
> > >> > > > > > here:
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> 1) This relates only to managed storage
> > (1:1
> > >> > > > mapping
> > >> > > > > > > > between
> > >> > > > > > > > > > > > > > >>> a virtual disk and a backend SAN
> volume).
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> 2) We want to take the current
> (introduced
> > >> in
> > >> > > 4.6)
> > >> > > > > > > > > > > > > > >>> functionality, which creates a snapshot
> on
> > >> the
> > >> > > SAN,
> > >> > > > > and
> > >> > > > > > > > > > > > > > >>> extend it via a config option (or
> > >> > > > > > > > > > > > > > >>> something) to not only take the SAN
> > >> snapshot,
> > >> > but
> > >> > > > to
> > >> > > > > > copy
> > >> > > > > > > > > > > > > > >>> the underlying VHD (XenServer only) to
> > NFS.
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> 3) The SAN snapshot is always taken.
> It's
> > >> the
> > >> > > > backup
> > >> > > > > to
> > >> > > > > > > NFS
> > >> > > > > > > > > > > > > > >>> that is optional.
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> 4) Templates can be created from the
> > >> snapshot
> > >> > > > that's
> > >> > > > > on
> > >> > > > > > > the
> > >> > > > > > > > > > > > > > >>> SAN (already works).
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> 5) CloudStack volumes can be created
> from
> > >> the
> > >> > > > > snapshot
> > >> > > > > > > > > > > > > > >>> that's on
> > >> > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > >>> SAN (already works as long as the new
> > >> > CloudStack
> > >> > > > > volume
> > >> > > > > > > > ends
> > >> > > > > > > > > > > > > > >>> up on the same primary storage).
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> Would we have a need for a storage
> > snapshot
> > >> API
> > >> > > > then
> > >> > > > > or
> > >> > > > > > > > > > > > > > >>> would that just be the standard volume
> > >> snapshot
> > >> > > > > without
> > >> > > > > > > the
> > >> > > > > > > > > > > > > > >>> backup to
> > >> > > > > > > > > > > NFS?
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> Thanks!
> > >> > > > > > > > > > > > > > >>> Mike
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>> On Thu, Feb 4, 2016 at 5:24 PM, Syed
> > Mushtaq
> > >> > > > > > > > > > > > > > >>> <syed1.mush...@gmail.com <javascript:;>
> > >> > > > > <javascript:;>>
> > >> > > > > > > > > > > > > > >>> wrote:
> > >> > > > > > > > > > > > > > >>>
> > >> > > > > > > > > > > > > > >>>> Is it possible to have both
> > functionalities
> > >> > > > > (snapshot
> > >> > > > > > on
> > >> > > > > > > > > > > > > > >>>> SAN & Sec
> > >> > > > > > > > > > > > > > >>>> Storage) coexist? Because Ideally, we
> > would
> > >> > like
> > >> > > > to
> > >> > > > > > have
> > >> > > > > > > > > both.
> > >> > > > > > > > > > > > > > >>>> For example, some of our customers want
> > to
> > >> > > > implement
> > >> > > > > > > their
> > >> > > > > > > > > > > > > > >>>> own backup strategies and do encryption
> > to
> > >> > their
> > >> > > > > > backups
> > >> > > > > > > > > > > > > > >>>> which is a perfect use case for Storage
> > >> > Snapshot
> > >> > > > > while
> > >> > > > > > > our
> > >> > > > > > > > > > > > > > >>>> other customers will still keep using
> the
> > >> > > standard
> > >> > > > > > > volume
> > >> > > > > > > > > > > snapshot.
> > >> > > > > > > > > > > > > > >>>>
> > >> > > > > > > > > > > > > > >>>> To keep things backward compatible, we
> > can
> > >> > add a
> > >> > > > > > setting
> > >> > > > > > > > > > > > > > >>>> which
> > >> > > > > > > > > > > > says
> > >> > > > > > > > > > > > > > >>>> to not upload on secondary storage,
> > >> because,
> > >> > > after
> > >> > > > > > all,
> > >> > > > > > > > you
> > >> > > > > > > > > > > > > > >>>> would take a SAN snapshot first when
> > doing
> > >> a
> > >> > > > Volume
> > >> > > > > > > > > Snapshot.
> > >> > > > > > > > > > > > > > >>>> You could stop the process there and
> not
> > do
> > >> > the
> > >> > > > > > upload.
> > >> > > > > > > > > > > > > > >>>>
> > >> > > > > > > > > > > > > > >>>> What do you think about this approach?
> > >> > > > > > > > > > > > > > >>>>
> > >> > > > > > > > > > > > > > >>>> Thanks,
> > >> > > > > > > > > > > > > > >>>> -Syed
> > >> > > > > > > > > > > > > > >>>>
> > >> > > > > > > > > > > > > > >>>> On Thu, Feb 4, 2016 at 4:25 PM, Mike
> > >> > Tutkowski <
> > >> > > > > > > > > > > > > > >>>> mike.tutkow...@solidfire.com
> > >> <javascript:;>
> > >> > > > > <javascript:;>> wrote:
> > >> > > > > > > > > > > > > > >>>>
> > >> > > > > > > > > > > > > > >>>>> So, this is just me thinking out load
> > >> here,
> > >> > but
> > >> > > > if
> > >> > > > > a
> > >> > > > > > > > given
> > >> > > > > > > > > > > > > > >>>>> CloudStack cloud doesn't actually need
> > to
> > >> > > provide
> > >> > > > > > both
> > >> > > > > > > > the
> > >> > > > > > > > > > > > ability
> > >> > > > > > > > > > > > > > >>>>> to take a SAN snapshot and export it
> to
> > >> NFS
> > >> > (if
> > >> > > > > just
> > >> > > > > > > > > > > > > > >>>>> taking a SAN snapshot is OK), then we
> > >> might
> > >> > be
> > >> > > > able
> > >> > > > > > to
> > >> > > > > > > > get
> > >> > > > > > > > > > > > > > >>>>> away with no new
> > >> > > > > > > > > > > > API
> > >> > > > > > > > > > > > > > >>>>> calls and simply implement a new
> custom
> > >> > > snapshot
> > >> > > > > > > strategy
> > >> > > > > > > > > > > > > > >>>>> and
> > >> > > > > > > > > > > > data
> > >> > > > > > > > > > > > > > >>>>> motion strategy to handle the case
> where
> > >> the
> > >> > > > > > CloudStack
> > >> > > > > > > > > > > > > > >>>>> cloud
> > >> > > > > > > > > > > > does
> > >> > > > > > > > > > > > > > >>>>> want both a SAN snapshot and
> > >> exported-to-NFS
> > >> > > > > backup.
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>> In other words, the "default" behavior
> > >> would
> > >> > be
> > >> > > > to
> > >> > > > > > use
> > >> > > > > > > > the
> > >> > > > > > > > > > > > > > >>>>> snapshot strategy and data motion
> > strategy
> > >> > that
> > >> > > > we
> > >> > > > > > > > already
> > >> > > > > > > > > > > > > > >>>>> have (the one that only takes a SAN
> > >> > snapshot).
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>> If your CloudStack cloud, however,
> wants
> > >> to
> > >> > > take
> > >> > > > a
> > >> > > > > > SAN
> > >> > > > > > > > > > > > > > >>>>> snapshot and have the data exported to
> > >> NFS,
> > >> > > then
> > >> > > > we
> > >> > > > > > > could
> > >> > > > > > > > > > > > > > >>>>> have you manipulate a Swing config
> file
> > to
> > >> > make
> > >> > > > use
> > >> > > > > > of
> > >> > > > > > > a
> > >> > > > > > > > > > > > > > >>>>> new snapshot strategy and data motion
> > >> > strategy
> > >> > > > that
> > >> > > > > > > > > > > > > > >>>>> performs both of these
> > >> > > > > > > > > > > > > > activities.
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>> This way, the old behavior is still
> the
> > >> > default
> > >> > > > for
> > >> > > > > > > > users,
> > >> > > > > > > > > > > > > > >>>>> but CloudStack admins can change this
> > >> > behavior
> > >> > > > via
> > >> > > > > > > > > > > configuration.
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>> Thoughts?
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>> On Thu, Feb 4, 2016 at 11:55 AM, Mike
> > >> > > Tutkowski <
> > >> > > > > > > > > > > > > > >>>>> mike.tutkow...@solidfire.com
> > >> <javascript:;>
> > >> > > > > <javascript:;>> wrote:
> > >> > > > > > > > > > > > > > >>>>>
> > >> > > > > > > > > > > > > > >>>>>> Right...I think we will need to come
> up
> > >> > with a
> > >> > > > > > viable
> > >> > > > > > > > > > > > > > >>>>>> upgrade path or some reasonable way
> for
> > >> them
> > >> > > to
> > >> > > > > move
> > >> > > > > > > > from
> > >> > > > > > > > > > > > > > >>>>>> the old way to the new way (and some
> > >> obvious
> > >> > > way
> > >> > > > > > that
> > >> > > > > > > > > > > > > > >>>>>> they will know they need
> > >> > > > > > > > > > > > to
> > >> > > > > > > > > > > > > > do this).
> > >> > > > > > > > > > > > > > >>>>>>
> > >> > > > > > > > > > > > > > >>>>>> On Thu, Feb 4, 2016 at 11:45 AM, Syed
> > >> > Mushtaq
> > >> > > <
> > >> > > > > > > > > > > > > > >>>>>> syed1.mush...@gmail.com
> <javascript:;>
> > >> > > > > <javascript:;>> wrote:
> > >> > > > > > > > > > > > > > >>>>>>
> > >> > > > > > > > > > > > > > >>>>>>> I'm not really sure about the
> upgrade
> > >> path
> > >> > > > > however,
> > >> > > > > > > > > > > > > > >>>>>>> customers who are using 4.6 and are
> > on a
> > >> > > > managed
> > >> > > > > > > > storage
> > >> > > > > > > > > > > > > > >>>>>>> would no longer have the same
> > >> functionality
> > >> > > > with
> > >> > > > > > > Volume
> > >> > > > > > > > >
--
Ian Rae
CEO | PDG
c: 514.944.4008
CloudOps | Cloud Infrastructure and Networking Solutions
www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J 1S6
