Sounds like a good idea.
On Wednesday 18 February 2015 10:29 PM, Mike Tutkowski wrote:
So, I spoke with Edison a couple months ago about my desire to take
snapshots on SolidFire's SAN instead of a making a backup on secondary
storage.
As Edison explained, CloudStack is theoretically flexible enough to
accommodate this if the developer implements the SnapshotStrategy interface
and orchestrates what needs to be done.
May be we can refactor the core to not assume that snapshots be
available only on secondary storage and implement a SnapshotStrategy
class that user can choose via a global setting, the core reads which
strategy it is and based on that decides that to assume.
I went ahead and implemented this approach successfully in XenServer
(although, as I mentioned earlier, not in as ideal a fashion as I would
have liked due to issues in XenServer). However, along the way, I found
that there was code in CloudStack "core" that assumed snapshots would be
stored on secondary storage.
I went ahead and made this core logic more flexible in 4.6 and so 4.6 is
the first release that I will be able to offer the ability for users to
take CloudStack snapshots that are stored on my SAN instead of on secondary
storage.
I then implemented a custom DataMotionStrategy to use with my snapshots so
that I could create templates for general use in CloudStack and so I could
make CloudStack volumes from my snapshots.
I think the bigger idea I'm seeing here is to allow primary storage to
be used as a staging storage, where both (immutable, so once made no one
changes them) snapshots and even templates can be stored and migrated in
background to secondary storage. The implementation would be easiest for
NFS and local storage, not sure about others, and except for Xen (since
snapshots are differential/incremental) it should be easier for KVM,
VMWare etc using full (immutable) snapshots.
Again...I know this thread is more about RBD, but I thought this might be
of interest as an FYI.
Keeping snapshots on primary storage would be a huge performance bonus,
not just for RBD but other storage technologies as well.
I think it's a good idea, as I've seen snapshots taking too much time
because of network latency issues.
On Wed, Feb 18, 2015 at 6:09 AM, Punith S <punit...@cloudbyte.com> wrote:
+1 with this feature.
like Logan said the current snapshot is just a copy of VDI/vmdk/qcow2's to
the secondary storage
hence the current feature acts as a backup feature taking a long time.
also the current cloudstack storage framework is not allowing the third
party storage vendors like cloudbyte and others to leverage their backend
storage snapshot feature where file systems support Copy on Write.
for example : In cloudbyte elastistor which is based on zfs filesystem, it
allows to take multiple snapshots within seconds.
and a volume clones can be created from each snapshot.
hence if the VDI/vmdk/qcow2's are already residing in
the volume, the clones will just replicate the
existing virtual disks. hence there will be no overhead of copying a
snapshot to a new volume or primary storage over the
network.
hence there should be an option provided to leverage a snapshot and
creating a volume out of it to the corresponding backend storage provider
in use.
thanks
On Tue, Feb 17, 2015 at 3:11 AM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:
Whatever way you think makes the most sense.
Either way, I'm working on this for XenServer and ESXi (eventually on
KVM,
I expect) for managed storage (SolidFire is an example of managed
storage).
On Mon, Feb 16, 2015 at 2:38 PM, Andrei Mikhailovsky <and...@arhont.com>
wrote:
I am happy to see the discussion is taking its pace and a lot of people
tend to agree that we should address this area. I have done the ticket
for
that, but I am not sure if this should be dealt in a more general way
as
suggested. Or perhaps having individual tickets for each hypervisor
would
achieve a faster response from the community?
Andrei
----- Original Message -----
From: "Mike Tutkowski" <mike.tutkow...@solidfire.com>
To: dev@cloudstack.apache.org
Sent: Monday, 16 February, 2015 9:17:26 PM
Subject: Re: Your thoughts on using Primary Storage for keeping
snapshots
Well...count me in on the general-purpose part (I'm already working
on that
and have much of it working).
If someone is interested in implementing the RBD part, he/she can
sync with
me and see if there is any overlapping work that I've already
implementing
from a general-purpose standpoint.
On Mon, Feb 16, 2015 at 1:39 PM, Ian Rae <i...@cloudops.com> wrote:
Agree with Logan. As fans of Ceph as well as SolidFire, we are
interested
in seeing this particular use case (RBD/KVM) being well
implemented,
however the concept of volume snapshots residing only on primary
storage vs
being transferred to secondary storage is a more generally useful
one that
is worth solving with the same terminology and interfaces, even if
the
mechanisms may be specific to the storage type and hypervisor.
It its not practical then its not practical, but seems like it
would be
worth trying.
On Mon, Feb 16, 2015 at 1:02 PM, Logan Barfield
<lbarfi...@tqhosting.com>
wrote:
Hi Mike,
I agree it is a general CloudStack issue that can be addressed
across
multiple primary storage options. It's a two stage issue since
some
changes will need to be implemented to support these features
across
the board, and others will need to be made to each storage
option.
It would be nice to see a single issue opened to cover this
across all
available storage options. Maybe have a community vote on what
support they want to see, and not consider the feature complete
until
all of the desired options are implemented? That would slow down
development for sure, but it would ensure that it was supported
where
it needs to be.
Thank You,
Logan Barfield
Tranquil Hosting
On Mon, Feb 16, 2015 at 12:42 PM, Mike Tutkowski
<mike.tutkow...@solidfire.com> wrote:
For example, Punith from CloudByte sent out an e-mail yesterday
that
was
very similar to this thread, but he was wondering how to
implement
such a
concept on his company's SAN technology.
On Mon, Feb 16, 2015 at 10:40 AM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:
Yeah, I think it's a similar concept, though.
You would want to take snapshots on Ceph (or some other
backend system
that acts as primary storage) instead of copying data to
secondary
storage
and calling it a snapshot.
For Ceph or any other backend system like that, the idea is to
speed
up
snapshots by not requiring CPU cycles on the front end or
network
bandwidth
to transfer the data.
In that sense, this is a general-purpose CloudStack problem
and it
appears
you are intending on discussing only the Ceph implementation
here,
which is
fine.
On Mon, Feb 16, 2015 at 10:34 AM, Logan Barfield <
lbarfi...@tqhosting.com>
wrote:
Hi Mike,
I think the interest in this issue is primarily for Ceph RBD,
which
doesn't use iSCSI or SAN concepts in general. As well I
believe RBD
is only currently supported in KVM (and VMware?). QEMU has
native
RBD
support, so it attaches the devices directly to the VMs in
question.
It also natively supports snapshotting, which is what this
discussion
is about.
Thank You,
Logan Barfield
Tranquil Hosting
On Mon, Feb 16, 2015 at 11:46 AM, Mike Tutkowski
<mike.tutkow...@solidfire.com> wrote:
I should have also commented on KVM (since that was the
hypervisor
called
out in the initial e-mail).
In my situation, most of my customers use XenServer and/or
ESXi, so
KVM
has
received the fewest of my cycles with regards to those
three
hypervisors.
KVM, though, is actually the simplest hypervisor for which
to
implement
these changes (since I am using the iSCSI adapter of the
KVM agent
and
it
just essentially passes my LUN to the VM in question).
For KVM, there is no clustered file system applied to my
backend
LUN,
so I
don't have to "worry" about that layer.
I don't see any hurdles like *immutable* UUIDs of SRs and
VDIs
(such
is
the
case with XenServer) or having to re-signature anything
(such is
the
case
with ESXi).
On Mon, Feb 16, 2015 at 9:33 AM, Mike Tutkowski <
mike.tutkow...@solidfire.com> wrote:
I have been working on this on and off for a while now (as
time
permits).
Here is an e-mail I sent to a customer of ours that helps
describe
some of
the issues:
*** Beginning of e-mail ***
The main requests were around the following features:
* The ability to leverage SolidFire snapshots.
* The ability to create CloudStack templates from
SolidFire
snapshots.
I had these on my roadmap, but bumped the priority up and
began
work on
them for the CS 4.6 release.
During design, I realized there were issues with the way
XenServer
is
architected that prevented me from directly using
SolidFire
snapshots.
I could definitely take a SolidFire snapshot of a
SolidFire
volume,
but
this snapshot would not be usable from XenServer if the
original
volume was
still in use.
Here is the gist of the problem:
When XenServer leverages an iSCSI target such as a
SolidFire
volume, it
applies a clustered files system to it, which they call a
storage
repository (SR). An SR has an *immutable* UUID associated
with it.
The virtual volume (which a VM sees as a disk) is
represented by a
virtual
disk image (VDI) in the SR. A VDI also has an *immutable*
UUID
associated
with it.
If I take a snapshot (or a clone) of the SolidFire volume
and then
later
try to use that snapshot from XenServer, XenServer
complains that
the
SR on
the snapshot has a UUID that conflicts with an existing
UUID.
In other words, it is not possible to use the original SR
and the
snapshot
of this SR from XenServer at the same time, which is
critical in a
cloud
environment (to enable creating templates from snapshots).
The way I have proposed circumventing this issue is not
ideal, but
technically works (this code is checked into the CS 4.6
branch):
When the time comes to take a CloudStack snapshot of a
CloudStack
volume
that is backed by SolidFire storage via the storage
plug-in, the
plug-in
will create a new SolidFire volume with characteristics
(size and
IOPS)
equal to those of the original volume.
We then have XenServer attach to this new SolidFire
volume,
create a
*new*
SR on it, and then copy the VDI from the source SR to the
destination
SR
(the new SR).
This leads to us having a copy of the VDI (a "snapshot" of
sorts),
but
it
requires CPU cycles on the compute cluster as well as
network
bandwidth to
write to the SAN (thus it is slower and more resource
intensive
than a
SolidFire snapshot).
I spoke with Tim Mackey (who works on XenServer at Citrix)
concerning
this
issue before and during the CloudStack Collaboration
Conference in
Budapest
in November. He agreed that this is a legitimate issue
with the
way
XenServer is designed and could not think of a way (other
than
what
I
was
doing) to get around it in current versions of XenServer.
One thought is to have a feature added to XenServer that
enables
you to
change the UUID of an SR and of a VDI.
If I could do that, then I could take a SolidFire snapshot
of the
SolidFire volume and issue commands to XenServer to have
it change
the
UUIDs of the original SR and the original VDI. I could
then
recored
the
necessary UUID info in the CS DB.
*** End of e-mail ***
I have since investigated this on ESXi.
ESXi does have a way for us to "re-signature" a datastore,
so
backend
snapshots can be taken and effectively used on this
hypervisor.
On Mon, Feb 16, 2015 at 8:19 AM, Logan Barfield <
lbarfi...@tqhosting.com>
wrote:
I'm just going to stick with the qemu-img option change
for RBD
for
now (which should cut snapshot time down drastically),
and look
forward to this in the future. I'd be happy to help get
this
moving,
but I'm not enough of a developer to lead the charge.
As far as renaming goes, I agree that maybe backups isn't
the
right
word. That being said calling a full-sized copy of a
volume a
"snapshot" also isn't the right word. Maybe "image" would
be
better?
I've also got my reservations about "accounts" vs "users"
(I
think
"departments" and "accounts or users" respectively is
less
confusing),
but that's a different thread.
Thank You,
Logan Barfield
Tranquil Hosting
On Mon, Feb 16, 2015 at 10:04 AM, Wido den Hollander <
w...@widodh.nl>
wrote:
On 16-02-15 15:38, Logan Barfield wrote:
I like this idea a lot for Ceph RBD. I do think there
should
still be
support for copying snapshots to secondary storage as
needed
(for
transfers between zones, etc.). I really think that
this
could
be
part of a larger move to clarify the naming
conventions used
for
disk
operations. Currently "Volume Snapshots" should
probably
really be
called "Backups". So having "snapshot" functionality,
and a
"convert
snapshot to backup/template" would be a good move.
I fully agree that this would be a very great addition.
I won't be able to work on this any time soon though.
Wido
Thank You,
Logan Barfield
Tranquil Hosting
On Mon, Feb 16, 2015 at 9:16 AM, Andrija Panic <
andrija.pa...@gmail.com> wrote:
BIG +1
My team should submit some patch to ACS for better
KVM
snapshots,
including
whole VM snapshot etc...but it's too early to give
details...
best
On 16 February 2015 at 13:01, Andrei Mikhailovsky <
and...@arhont.com>
wrote:
Hello guys,
I was hoping to have some feedback from the
community on the
subject
of
having an ability to keep snapshots on the primary
storage
where
it
is
supported by the storage backend.
The idea behind this functionality is to improve how
snapshots
are
currently handled on KVM hypervisors with Ceph
primary
storage.
At
the
moment, the snapshots are taken on the primary
storage and
being
copied to
the secondary storage. This method is very slow and
inefficient
even
on
small infrastructure. Even on medium deployments
using
snapshots
in
KVM
becomes nearly impossible. If you have tens or
hundreds
concurrent
snapshots taking place you will have a bunch of
timeouts and
errors,
your
network becomes clogged, etc. In addition, using
these
snapshots
for
creating new volumes or reverting back vms also slow
and
inefficient. As
above, when you have tens or hundreds concurrent
operations
it
will
not
succeed and you will have a majority of tasks with
errors or
timeouts.
At the moment, taking a single snapshot of
relatively small
volumes
(200GB
or 500GB for instance) takes tens if not hundreds of
minutes.
Taking
a
snapshot of the same volume on ceph primary storage
takes a
few
seconds at
most! Similarly, converting a snapshot to a volume
takes
tens
if
not
hundreds of minutes when secondary storage is
involved;
compared
with
seconds if done directly on the primary storage.
I suggest that the CloudStack should have the
ability to
keep
volume
snapshots on the primary storage where this is
supported by
the
storage.
Perhaps having a per primary storage setting that
enables
this
functionality. This will be beneficial for Ceph
primary
storage
on
KVM
hypervisors and perhaps on XenServer when Ceph will
be
supported
in
a near
future.
This will greatly speed up the process of using
snapshots on
KVM
and
users
will actually start using snapshotting rather than
giving up
with
frustration.
I have opened the ticket CLOUDSTACK-8256, so please
cast
your
vote
if you
are in agreement.
Thanks for your input
Andrei
--
Andrija Panić
--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*
--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*
--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*
--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*
--
*Ian Rae*
PDG *| *CEO
t *514.944.4008*
*CloudOps* Votre partenaire infonuagique* | *Cloud Solutions
Experts
w cloudops.com <http://www.cloudops.com/> *|* 420 rue Guy *|*
Montreal *|*
Quebec *|* H3J 1S6
<https://www.cloud.ca/>
<
http://www.cloudops.com/2014/11/cloudops-tops-deloittes-technology-fast-50/
--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*
--
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkow...@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
<http://solidfire.com/solution/overview/?video=play>*™*
--
regards,
punith s
cloudbyte.com
--
Regards,
Rohit Yadav
Software Architect, ShapeBlue
M. +91 8826230892 | rohit.ya...@shapeblue.com
Blog: bhaisaab.org | Twitter: @_bhaisaab
PS. If you see any footer below, I did not add it :)
Find out more about ShapeBlue and our range of CloudStack related services
IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
CloudStack Software
Engineering<http://shapeblue.com/cloudstack-software-engineering/>
CloudStack Infrastructure
Support<http://shapeblue.com/cloudstack-infrastructure-support/>
CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely
for the use of the individual to whom it is addressed. Any views or opinions
expressed are solely those of the author and do not necessarily represent those of
Shape Blue Ltd or related companies. If you are not the intended recipient of this
email, you must neither take any action based upon its contents, nor copy or show
it to anyone. Please contact the sender if you believe you have received this email
in error. Shape Blue Ltd is a company incorporated in England & Wales.
ShapeBlue Services India LLP is a company incorporated in India and is operated
under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company
incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue
SA Pty Ltd is a company registered by The Republic of South Africa and is traded
under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
