Hi,
On 12/30/2012 10:38 PM, Miles Fidelman wrote:
Hi Folks,
I'm wondering how ceph would work in a small cluster that supports a mix
of engineering and modest production (email, lists, web server for
several small communities).
Specifically, we have a rack with 4 medium-horsepower servers, each with
4 disk drives, running Xen (debian dom0 and domUs) - all linked together
w/ 4 gigE ethernets.
Currently, 2 of the servers are running a high-availability
configuration, using DRBD to mirror specific volumes, and pacemaker for
failover.
For a while, I've been looking for a way to replace DRBD with something
that would mirror across more than 2 servers - so that we could migrate
VMs arbitrarily - and that will work without splitting up compute vs.
storage nodes (for the short term, at least, we're stuck with rack space
and server limitations).
The thing that looks closest to filling the bill is Sheepdog (at least
architecturally) - but it only provides a KVM interface. GlusterFS,
xTreemFS, and Ceph keep coming up as possibles - with ceph's rbd
interface looking like the easiest to integrate.
Which leads me to two questions:
- On a theoretical level, does using ceph as a storage pool for this
kind of small cluster make any sense (notably, I'd see running an OSD, a
MDS, a MON, and client DomUs on each of the 4 nodes, using LVM to pool
all the storage and it seems like folks recommend XFS as a production
filesystem)
Yes, that could work. But you have to keep in mind that OSDs can spike
in both CPU and memory when they have to do recovery work for a failed
node/OSD.
Also, with RBD you don't need an MDS. As a last note, you should always
have an odd number of monitors. So run a monitor on 3 of the 4 machines.
The monitors work by a voting principle where they need a majority. An
odd number is best in that situation.
- On a practical level, has anybody tried building this kind of small
cluster, and if so, what kind of results have you had?
I build some small Ceph cluster with sometimes just 3 nodes. It works,
but you have to keep in mind that when one node in a 4 node cluster
fails you will loose 25% of the capacity.
This will lead to a heavy recovery within the Ceph cluster which will
but a lot of pressure on that Gbit links and the CPUs and memory of the
nodes.
With RBD you might want to consider adding an SSD for the journaling of
the OSDs, that will give you a pretty nice performance boost.
Wido
Comments and suggestions please!
Thank you very much,
Miles Fidelman
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html