ceph on non-btrfs file systems

2011-10-23 Thread Sage Weil
Although running on ext4, xfs, or whatever other non-btrfs you want mostly works, there are a few important remaining issues: 1- ext4 limits total xattrs for 4KB. This can cause problems in some cases, as Ceph uses xattrs extensively. Most of the time we don't hit this. We do hit the limit w

Re: ceph on non-btrfs file systems

2011-10-24 Thread Christian Brunner
Thanks for explaining this. I don't have any objections against btrfs as a osd filesystem. Even the fact that there is no btrfs-fsck doesn't scare me, since I can use the ceph replication to recover a lost btrfs-filesystem. The only problem I have is, that btrfs is not stable on our side and I wond

ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-24 Thread Sage Weil
[adding linux-btrfs to cc] Josef, Chris, any ideas on the below issues? On Mon, 24 Oct 2011, Christian Brunner wrote: > Thanks for explaining this. I don't have any objections against btrfs > as a osd filesystem. Even the fact that there is no btrfs-fsck doesn't > scare me, since I can use the ce

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-24 Thread Josef Bacik
On Mon, Oct 24, 2011 at 10:06:49AM -0700, Sage Weil wrote: > [adding linux-btrfs to cc] > > Josef, Chris, any ideas on the below issues? > > On Mon, 24 Oct 2011, Christian Brunner wrote: > > Thanks for explaining this. I don't have any objections against btrfs > > as a osd filesystem. Even the fa

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-24 Thread Chris Mason
On Mon, Oct 24, 2011 at 03:51:47PM -0400, Josef Bacik wrote: > On Mon, Oct 24, 2011 at 10:06:49AM -0700, Sage Weil wrote: > > [adding linux-btrfs to cc] > > > > Josef, Chris, any ideas on the below issues? > > > > On Mon, 24 Oct 2011, Christian Brunner wrote: > > > Thanks for explaining this. I d

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-24 Thread Christian Brunner
2011/10/24 Chris Mason : > On Mon, Oct 24, 2011 at 03:51:47PM -0400, Josef Bacik wrote: >> On Mon, Oct 24, 2011 at 10:06:49AM -0700, Sage Weil wrote: >> > [adding linux-btrfs to cc] >> > >> > Josef, Chris, any ideas on the below issues? >> > >> > On Mon, 24 Oct 2011, Christian Brunner wrote: >> > >

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-24 Thread Arne Jansen
On 24.10.2011 23:34, Christian Brunner wrote: 2011/10/24 Chris Mason: On Mon, Oct 24, 2011 at 03:51:47PM -0400, Josef Bacik wrote: On Mon, Oct 24, 2011 at 10:06:49AM -0700, Sage Weil wrote: [adding linux-btrfs to cc] Josef, Chris, any ideas on the below issues? On Mon, 24 Oct 2011, Christian

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-25 Thread Christoph Hellwig
On Mon, Oct 24, 2011 at 10:06:49AM -0700, Sage Weil wrote: > > - When I run ceph with btrfs snaps disabled, the situation is getting > > slightly better. I can run an OSD for about 3 days without problems, > > but then again the load increases. This time, I can see that the > > ceph-osd (blkdev_iss

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-25 Thread Josef Bacik
On Tue, Oct 25, 2011 at 01:56:48PM +0200, Christian Brunner wrote: > 2011/10/24 Josef Bacik : > > On Mon, Oct 24, 2011 at 10:06:49AM -0700, Sage Weil wrote: > >> [adding linux-btrfs to cc] > >> > >> Josef, Chris, any ideas on the below issues? > >> > >> On Mon, 24 Oct 2011, Christian Brunner wrote:

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-25 Thread Christian Brunner
2011/10/25 Josef Bacik : > On Tue, Oct 25, 2011 at 01:56:48PM +0200, Christian Brunner wrote: >> 2011/10/24 Josef Bacik : >> > On Mon, Oct 24, 2011 at 10:06:49AM -0700, Sage Weil wrote: >> >> [adding linux-btrfs to cc] >> >> >> >> Josef, Chris, any ideas on the below issues? >> >> >> >> On Mon, 24

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-25 Thread Josef Bacik
On Tue, Oct 25, 2011 at 04:25:02PM +0200, Christian Brunner wrote: > 2011/10/25 Josef Bacik : > > On Tue, Oct 25, 2011 at 01:56:48PM +0200, Christian Brunner wrote: > >> 2011/10/24 Josef Bacik : > >> > On Mon, Oct 24, 2011 at 10:06:49AM -0700, Sage Weil wrote: > >> >> [adding linux-btrfs to cc] > >

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-25 Thread Josef Bacik
On Tue, Oct 25, 2011 at 04:25:02PM +0200, Christian Brunner wrote: > 2011/10/25 Josef Bacik : > > On Tue, Oct 25, 2011 at 01:56:48PM +0200, Christian Brunner wrote: > >> 2011/10/24 Josef Bacik : > >> > On Mon, Oct 24, 2011 at 10:06:49AM -0700, Sage Weil wrote: > >> >> [adding linux-btrfs to cc] > >

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-25 Thread Christian Brunner
2011/10/25 Josef Bacik : > On Tue, Oct 25, 2011 at 04:25:02PM +0200, Christian Brunner wrote: >> 2011/10/25 Josef Bacik : >> > On Tue, Oct 25, 2011 at 01:56:48PM +0200, Christian Brunner wrote: [...] >> >> >> >> In our Ceph-OSD server we have 4 disks with 4 btrfs filesystems. Ceph >> >> tries to ba

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-25 Thread Sage Weil
On Tue, 25 Oct 2011, Christoph Hellwig wrote: > On Mon, Oct 24, 2011 at 10:06:49AM -0700, Sage Weil wrote: > > > - When I run ceph with btrfs snaps disabled, the situation is getting > > > slightly better. I can run an OSD for about 3 days without problems, > > > but then again the load increases.

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-25 Thread Sage Weil
On Tue, 25 Oct 2011, Josef Bacik wrote: > At this point it seems like the biggest problem with latency in ceph-osd > is not related to btrfs, the latency seems to all be from the fact that > ceph-osd is fsyncing a block dev for whatever reason. There is one place where we sync_file_range() on t

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-25 Thread Christian Brunner
2011/10/25 Sage Weil : > On Tue, 25 Oct 2011, Josef Bacik wrote: >> At this point it seems like the biggest problem with latency in ceph-osd >> is not related to btrfs, the latency seems to all be from the fact that >> ceph-osd is fsyncing a block dev for whatever reason. > > There is one place whe

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-25 Thread Chris Mason
On Tue, Oct 25, 2011 at 11:05:12AM -0400, Josef Bacik wrote: > On Tue, Oct 25, 2011 at 04:25:02PM +0200, Christian Brunner wrote: > > > > Attached is a perf-report. I have included the whole report, so that > > you can see the difference between the good and the bad > > btrfs-endio-wri. > > > > W

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-25 Thread Josef Bacik
On Tue, Oct 25, 2011 at 04:15:45PM -0400, Chris Mason wrote: > On Tue, Oct 25, 2011 at 11:05:12AM -0400, Josef Bacik wrote: > > On Tue, Oct 25, 2011 at 04:25:02PM +0200, Christian Brunner wrote: > > > > > > Attached is a perf-report. I have included the whole report, so that > > > you can see the

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-25 Thread Sage Weil
On Tue, 25 Oct 2011, Christian Brunner wrote: > 2011/10/25 Sage Weil : > > On Tue, 25 Oct 2011, Josef Bacik wrote: > >> At this point it seems like the biggest problem with latency in ceph-osd > >> is not related to btrfs, the latency seems to all be from the fact that > >> ceph-osd is fsyncing a b

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-25 Thread Christian Brunner
2011/10/25 Josef Bacik : > On Tue, Oct 25, 2011 at 04:15:45PM -0400, Chris Mason wrote: >> On Tue, Oct 25, 2011 at 11:05:12AM -0400, Josef Bacik wrote: >> > On Tue, Oct 25, 2011 at 04:25:02PM +0200, Christian Brunner wrote: >> > > >> > > Attached is a perf-report. I have included the whole report,

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-26 Thread Christian Brunner
2011/10/26 Sage Weil : > On Wed, 26 Oct 2011, Christian Brunner wrote: >> >> > Christian, have you tweaked those settings in your ceph.conf?  It would >> >> > be >> >> > something like 'journal dio = false'.  If not, can you verify that >> >> > directio shows true when the journal is initialized f

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-26 Thread Christian Brunner
2011/10/26 Christian Brunner : > 2011/10/25 Josef Bacik : >> On Tue, Oct 25, 2011 at 04:15:45PM -0400, Chris Mason wrote: >>> On Tue, Oct 25, 2011 at 11:05:12AM -0400, Josef Bacik wrote: >>> > On Tue, Oct 25, 2011 at 04:25:02PM +0200, Christian Brunner wrote: >>> > > >>> > > Attached is a perf-repo

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-26 Thread Chris Mason
On Tue, Oct 25, 2011 at 04:22:48PM -0400, Josef Bacik wrote: > On Tue, Oct 25, 2011 at 04:15:45PM -0400, Chris Mason wrote: > > On Tue, Oct 25, 2011 at 11:05:12AM -0400, Josef Bacik wrote: > > > On Tue, Oct 25, 2011 at 04:25:02PM +0200, Christian Brunner wrote: > > > > > > > > Attached is a perf-r

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-26 Thread Sage Weil
On Wed, 26 Oct 2011, Christian Brunner wrote: > 2011/10/26 Sage Weil : > > On Wed, 26 Oct 2011, Christian Brunner wrote: > >> >> > Christian, have you tweaked those settings in your ceph.conf?  It > >> >> > would be > >> >> > something like 'journal dio = false'.  If not, can you verify that > >>

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-27 Thread Martin Mailand
Hi resend without the perf attachment, which could be found here: http://tuxadero.com/multistorage/perf.report.txt.bz2 Best Regards, martin Original-Nachricht Betreff: Re: ceph on btrfs [was Re: ceph on non-btrfs file systems] Datum: Wed, 26 Oct 2011 22:38:47 +0200 Von

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-27 Thread Stefan Majer
ut the perf attachment, which could be found here: > http://tuxadero.com/multistorage/perf.report.txt.bz2 > > Best Regards, >  martin > > Original-Nachricht > Betreff: Re: ceph on btrfs [was Re: ceph on non-btrfs file systems] > Datum: Wed, 26 Oct 2011 22:38:

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-27 Thread Martin Mailand
Majer On Thu, Oct 27, 2011 at 12:53 PM, Martin Mailand wrote: Hi resend without the perf attachment, which could be found here: http://tuxadero.com/multistorage/perf.report.txt.bz2 Best Regards, martin Original-Nachricht Betreff: Re: ceph on btrfs [was Re: ceph on non-btrfs

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-27 Thread Josef Bacik
On Wed, Oct 26, 2011 at 09:23:54AM -0400, Chris Mason wrote: > On Tue, Oct 25, 2011 at 04:22:48PM -0400, Josef Bacik wrote: > > On Tue, Oct 25, 2011 at 04:15:45PM -0400, Chris Mason wrote: > > > On Tue, Oct 25, 2011 at 11:05:12AM -0400, Josef Bacik wrote: > > > > On Tue, Oct 25, 2011 at 04:25:02PM

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-27 Thread Josef Bacik
On Thu, Oct 27, 2011 at 11:07:38AM -0400, Josef Bacik wrote: > On Wed, Oct 26, 2011 at 09:23:54AM -0400, Chris Mason wrote: > > On Tue, Oct 25, 2011 at 04:22:48PM -0400, Josef Bacik wrote: > > > On Tue, Oct 25, 2011 at 04:15:45PM -0400, Chris Mason wrote: > > > > On Tue, Oct 25, 2011 at 11:05:12AM

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-27 Thread Josef Bacik
On Tue, Oct 25, 2011 at 01:56:48PM +0200, Christian Brunner wrote: > 2011/10/24 Josef Bacik : > > On Mon, Oct 24, 2011 at 10:06:49AM -0700, Sage Weil wrote: > >> [adding linux-btrfs to cc] > >> > >> Josef, Chris, any ideas on the below issues? > >> > >> On Mon, 24 Oct 2011, Christian Brunner wrote:

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011-10-27 Thread Christian Brunner
2011/10/27 Josef Bacik : > On Tue, Oct 25, 2011 at 01:56:48PM +0200, Christian Brunner wrote: >> 2011/10/24 Josef Bacik : >> > On Mon, Oct 24, 2011 at 10:06:49AM -0700, Sage Weil wrote: >> >> [adding linux-btrfs to cc] >> >> >> >> Josef, Chris, any ideas on the below issues? >> >> >> >> On Mon, 24