Re: Ceph performance improvement

2012-08-22 Thread David McBride
On 22/08/12 09:54, Denis Fondras wrote: The only point that prevents my from using it at datacenter-scale is performance. Here are some figures : * Test with "dd" on the OSD server (on drive /dev/disk/by-id/scsi-SATA_WDC_WD30EZRX-00_WD-WMAWZ0152201) : # dd if=/dev/zero of=testdd bs=4k count=4

Re: Ceph performance improvement

2012-08-22 Thread Denis Fondras
Thank you for the answer David. That looks like you're writing to a filesystem on that disk, rather than the block device itself -- but lets say you've got 139MB/sec (1112Mbit/sec) of straight-line performance. Note: this is already faster than your network link can go -- you can, at best, onl

Re: Ceph performance improvement

2012-08-22 Thread Mark Nelson
On 08/22/2012 03:54 AM, Denis Fondras wrote: Hello all, Hello! David had some good comments in his reply, so I'll just add in a couple of extra thoughts... I'm currently testing Ceph. So far it seems that HA and recovering are very good. The only point that prevents my from using it at da

Re: Ceph performance improvement

2012-08-22 Thread Alexandre DERUMIER
" À: "Denis Fondras" Cc: ceph-devel@vger.kernel.org Envoyé: Mercredi 22 Août 2012 14:35:28 Objet: Re: Ceph performance improvement On 08/22/2012 03:54 AM, Denis Fondras wrote: > Hello all, Hello! David had some good comments in his reply, so I'll just add in a couple

Re: Ceph performance improvement

2012-08-22 Thread Tommi Virtanen
On Wed, Aug 22, 2012 at 1:54 AM, Denis Fondras wrote: > First of all, here is my setup : > for the system, 1x 64GB SSD (Crucial C300 - /dev/sda) for the journal and 4x > 3TB drive (Western Digital WD30EZRX). Everything but the boot partition is > BTRFS-formated and 4K-aligned. ... > [osd] >

Re: Ceph performance improvement

2012-08-22 Thread Denis Fondras
Are you sure your osd data and journal are on the disks you think? The /home paths look suspicious -- especially for journal, which often should be a block device. I am :) Can you share output of "mount" and "ls -ld /home/osd.*" Here are some details : root@ceph-osd-0:~# ls -al /dev/disk/

Re: Ceph performance improvement

2012-08-22 Thread Tommi Virtanen
On Wed, Aug 22, 2012 at 9:23 AM, Denis Fondras wrote: >> Are you sure your osd data and journal are on the disks you think? The >> /home paths look suspicious -- especially for journal, which often >> should be a block device. > I am :) ... > -rw-r--r-- 1 root root 1048576000 août 22 17:22 /home/

Re: Ceph performance improvement

2012-08-22 Thread Mark Kirkwood
On 22/08/12 22:24, David McBride wrote: On 22/08/12 09:54, Denis Fondras wrote: * Test with "dd" from the client using CephFS : # dd if=/dev/zero of=testdd bs=4k count=4M 17179869184 bytes (17 GB) written, 338,29 s, 50,8 MB/s Again, the synchronous nature of 'dd' is probably severely affectin

Re: Ceph performance improvement

2012-08-24 Thread Denis Fondras
Hello Mark, Not sure what version of glibc Wheezy has, but try to make sure you have one that supports syncfs (you'll also need a semi-new kernel, 3.0+ should be fine). Wheezy has a fairly recent kernel : # uname -a Linux ceph-osd-0 3.2.0-3-amd64 #1 SMP Mon Jul 23 02:45:17 UTC 2012 x86_64 G

Re: Ceph performance improvement

2012-08-24 Thread Wido den Hollander
On 08/24/2012 06:41 PM, Denis Fondras wrote: In my opinions, performances from RBD client are decent. Unfortunately I need concurrent access and CephFS is really appealing in that respect. Ouch, that's taking a while! In addition to the comments that David made, be aware that you are also

Re: Ceph performance improvement / journal on block-dev

2012-08-22 Thread Dieter Kasper (KD)
On Wed, Aug 22, 2012 at 06:29:12PM +0200, Tommi Virtanen wrote: (...) > > Your journal is a file on a btrfs partition. That is probably a bad > idea for performance. I'd recommend partitioning the drive and using > partitions as journals directly. Hi Tommi, can you please teach me how to use the

Re: Ceph performance improvement / journal on block-dev

2012-08-22 Thread Tommi Virtanen
On Wed, Aug 22, 2012 at 12:12 PM, Dieter Kasper (KD) wrote: >> Your journal is a file on a btrfs partition. That is probably a bad >> idea for performance. I'd recommend partitioning the drive and using >> partitions as journals directly. > can you please teach me how to use the right parameter(s)