Re: [ceph-users] OSD + FlashCache vs. Cache Pool for RBD...

2014-03-22 Thread Kyle Bader 
> One downside of the above arrangement: I read that support for mapping
> newer-format RBDs is only present in fairly recent kernels.  I'm running
> Ubuntu 12.04 on the cluster at present with its stock 3.2 kernel.  There
> is a PPA for the 3.11 kernel used in Ubuntu 13.10, but if you're looking
> at a new deployment it might be better to wait until 14.04: then you'll
> get kernel 3.13.
>
> Anyone else have any ideas on the above?

I don't think there are any hairy udev issues or similar that will
make using a newer kernel on precise problematic. The only thing I can
think of that is a caveat of this kind of setup if if you lose a
hypervisor the cache will go with it and you likely wont be able to
migrate the guest to another host. The alternative is to use
flashcache on top of the OSD partition but then you introduce network
hops and is closer to what the tiering feature will offer, except the
flashcache OSD method is more particular about disk:ssd ratio, whereas
in a tier the flash could be on s completely separate hosts (possibly
dedicated flash machines).

-- 

Kyle
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD + FlashCache vs. Cache Pool for RBD...

2014-03-20 Thread Stuart Longland 
On 10/03/14 23:18, Xavier Trilla wrote:
> -What do you think is a better approach to improve the
> performance of RBD for VMs: Caching OSDs with FlashCache or using SSD
> Cache Pools?

Well as has been mentioned, Cache Pools isn't available yet however I'm
starting to do some thinking about using FlashCache to give the RBDs a
much-needed kick-along.

We're using Ceph with an in-testing OpenNebula deployment.  At this time
there appears to be little support for this, both on physical hosts and
within VM provisioning systems like OpenStack/OpenNebula.

On my TODO list is to investigate writing such a datastore driver for
OpenNebula.

Sébastian Han did a bit of a guide which goes into setting it up by hand:
http://www.sebastien-han.fr/blog/2012/11/15/make-your-rbd-fly-with-flashcache/

This tells us the idea is at least doable.

I'm thinking something along the lines of using a LVM partition on SSD
as a write-through (or even write-back) cache.

OpenNebula just uses the default RBD format, which from what I
understand, doesn't support COW cloning and such.

I've thus observed with our 3-node cluster (with Intel Core i3 3570T
CPUs, 8GB RAM, 2×3TB HDDs each and 10GB of SSD-based journal per OSD)
making temporary copies of images really does give the cluster a fair
hiding.  CPU load averages reaching 6 or more -- which is getting a bit
much for dual-core CPUs.

The thought I had was along the lines of using the newer RBD format
which does support COW.  Then the logic for provisioning storage would
be thus:

- If the image is non-persistent, make a COW clone of the original,
otherwise use the original image as-is.
- Create a local volume on the SSD LVM partition (ideally equal to the
size of the original image)
- Map the remote RBD device
- Set up flashcache to use the RBD and LVM cache volume to produce a
composite cached device.
- Point the VM at the device produced by flashcache.

The above can probably be applied to OpenStack as well, maybe some hocus
pocus work with ephemeral volumes in nova-volume or some such?

One downside of the above arrangement: I read that support for mapping
newer-format RBDs is only present in fairly recent kernels.  I'm running
Ubuntu 12.04 on the cluster at present with its stock 3.2 kernel.  There
is a PPA for the 3.11 kernel used in Ubuntu 13.10, but if you're looking
at a new deployment it might be better to wait until 14.04: then you'll
get kernel 3.13.

Anyone else have any ideas on the above?

Regards,
-- 
Stuart Longland
Systems Engineer
 _ ___
\  /|_) |   T: +61 7 3535 9619
 \/ | \ | 38b Douglas StreetF: +61 7 3535 9699
   SYSTEMSMilton QLD 4064   http://www.vrt.com.au


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD + FlashCache vs. Cache Pool for RBD...

2014-03-10 Thread Mark Nelson 

On 03/10/2014 08:18 AM, Xavier Trilla wrote:

Hi!

I’ve been checking some of the available information about Cache pools
and I’ve come out with some questions:

-What do you think is a better approach to improve the performance of
RBD for VMs: Caching OSDs with FlashCache or using SSD Cache Pools?


As cache pools aren't even out yet, and few people have really dug into 
flashcache backed OSDs, it's kind of tough to say. :)  If you do any 
testing, we'd leave to hear your results!




-As I understand kernel driver will not be able to mount  RBD images
from cached pools until it gets upgraded, am I right? (Kernel 3.14
probably?)

-As QEMU / KVM does use librbd, if I upgrade Ceph client packages on
Hypervisor host will VMs be able to mount RBD images from cached pools?
(As I understand QEMU / KVM link dynamically to librbd libraries, right?)

Actually, I plan to perform some tests by myself soon, but if someone
could give me some insight before I get my hands over proper HW to run
some tests it would be really great.


I think the only insight I can give right now is that flashcache is 
going to cache things at the block layer while a cache pool will cache 
things at the object layer.  There are potentially some advantages and 
disadvantages in each case.  I haven't really thought through this well 
yet, but here are some initial guesses:


flashcache:
+ All caching is local to a node, much less overhead (including network!).
+ May do better in situations with many object accesses and lots of hot 
inodes/dentries?

- No ability to define different replication/EC policy for cache only.
- cache writes are non-atomic?

Ceph Cache Tier:
+ Potentially can do things like replication for cache and erasure 
coding for cold data.

+ Maybe safer?
- More network overhead, potentially more CPU overhead.
- Takes longer to get things into cache?



Thanks!

Saludos cordiales,

Xavier Trilla P.

Silicon Hosting 

¿Todavía no conoces Bare Metal Cloud?

¡La evolución de los Servidores VPS ya ha llegado!

más información en: siliconhosting.com/cloud




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] OSD + FlashCache vs. Cache Pool for RBD...

2014-03-10 Thread Xavier Trilla 
Hi!

I've been checking some of the available information about Cache pools and I've 
come out with some questions:


-What do you think is a better approach to improve the performance of 
RBD for VMs: Caching OSDs with FlashCache or using SSD Cache Pools?

-As I understand kernel driver will not be able to mount  RBD images 
from cached pools until it gets upgraded, am I right? (Kernel 3.14 probably?)

-As QEMU / KVM does use librbd, if I upgrade Ceph client packages on 
Hypervisor host will VMs be able to mount RBD images from cached pools? (As I 
understand QEMU / KVM link dynamically to librbd libraries, right?)


Actually, I plan to perform some tests by myself soon, but if someone could 
give me some insight before I get my hands over proper HW to run some tests it 
would be really great.

Thanks!

Saludos cordiales,
Xavier Trilla P.
Silicon Hosting

¿Todavía no conoces Bare Metal Cloud?
¡La evolución de los Servidores VPS ya ha llegado!

más información en: siliconhosting.com/cloud

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com