Re: Ceph on btrfs 3.4rc

Same thing here. I've tried really hard, but even after 12 hours I wasn't able to get a single warning from btrfs. I think you cracked it! Thanks, Christian 2012/5/24 Martin Mailand : > Hi, > the ceph cluster is running under heavy load for the last 13 hours without a > problem, dmesg is empty

Re: Ceph on btrfs 3.4rc

2012/5/22 Josef Bacik : >> > > Yeah you would also need to change orphan_meta_reserved.  I fixed this by just > taking the BTRFS_I(inode)->lock when messing with these since we don't want to > take up all that space in the inode just for a marker.  I ran this patch for 3 > hours with no issues, let

Re: Ceph on btrfs 3.4rc

2012/5/21 Miao Xie : > Hi Josef, > > On fri, 18 May 2012 15:01:05 -0400, Josef Bacik wrote: >> diff --git a/fs/btrfs/btrfs_inode.h b/fs/btrfs/btrfs_inode.h >> index 9b9b15f..492c74f 100644 >> --- a/fs/btrfs/btrfs_inode.h >> +++ b/fs/btrfs/btrfs_inode.h >> @@ -57,9 +57,6 @@ struct btrfs_inode { >>  

Re: Ceph on btrfs 3.4rc

2012/5/17 Josef Bacik : > On Thu, May 17, 2012 at 05:12:55PM +0200, Martin Mailand wrote: >> Hi Josef, >> no there was nothing above. Here the is another dmesg output. >> > > Hrm ok give this a try and hopefully this is it, still couldn't reproduce. > Thanks, > > Josef Well, I hate to say it, but

Re: Ceph on btrfs 3.4rc

2012/5/10 Josef Bacik : > On Fri, Apr 27, 2012 at 01:02:08PM +0200, Christian Brunner wrote: >> Am 24. April 2012 18:26 schrieb Sage Weil : >> > On Tue, 24 Apr 2012, Josef Bacik wrote: >> >> On Fri, Apr 20, 2012 at 05:09:34PM +0200, Christian Brunner wrote: >>

Re: Ceph on btrfs 3.4rc

2012/5/3 Josef Bacik : > On Thu, May 03, 2012 at 09:38:27AM -0700, Josh Durgin wrote: >> On Thu, 3 May 2012 11:20:53 -0400, Josef Bacik >> wrote: >> > On Thu, May 03, 2012 at 08:17:43AM -0700, Josh Durgin wrote: >> > >> > Yeah all that was in the right place, I rebooted and I magically >> > stoppe

Re: Ceph on btrfs 3.4rc

2012/4/29 tsuna : > On Fri, Apr 20, 2012 at 8:09 AM, Christian Brunner > wrote: >> After running ceph on XFS for some time, I decided to try btrfs again. >> Performance with the current "for-linux-min" branch and big metadata >> is much better. > > I'

Re: Ceph on btrfs 3.4rc

Am 24. April 2012 18:26 schrieb Sage Weil : > On Tue, 24 Apr 2012, Josef Bacik wrote: >> On Fri, Apr 20, 2012 at 05:09:34PM +0200, Christian Brunner wrote: >> > After running ceph on XFS for some time, I decided to try btrfs again. >> > Performance with the current &qu

Re: Ceph on btrfs 3.4rc

.00MB, used=0.00 A few more btrfs_orphan_commit_root WARNINGS are present too. If needed I could upload the messages file. Regards, Christian Am 20. April 2012 17:09 schrieb Christian Brunner : > After running ceph on XFS for some time, I decided to try btrfs again. > Performance with the c

Ceph on btrfs 3.4rc

After running ceph on XFS for some time, I decided to try btrfs again. Performance with the current "for-linux-min" branch and big metadata is much better. The only problem (?) I'm still seeing is a warning that seems to occur from time to time: [87703.784552] [ cut here ]

Re: Strange prformance degradation when COW writes happen at fixed offsets

2012/2/24 Nik Markovic : > To add... I also tried nodatasum (only) and nodatacow otions. I found > somewhere that nodatacow doesn't really mean tthat COW is disabled. > Test data is still the same - CPU spikes and times are the same. > > On Fri, Feb 24, 2012 at 2:38 PM, Nik Markovic > wrote: >> O

Re: Btrfs slowdown with ceph (how to reproduce)

2012/1/23 Chris Mason : > On Mon, Jan 23, 2012 at 01:19:29PM -0500, Josef Bacik wrote: >> On Fri, Jan 20, 2012 at 01:13:37PM +0100, Christian Brunner wrote: >> > As you might know, I have been seeing btrfs slowdowns in our ceph >> > cluster for quite some time. Even wit

Btrfs slowdown with ceph (how to reproduce)

As you might know, I have been seeing btrfs slowdowns in our ceph cluster for quite some time. Even with the latest btrfs code for 3.3 I'm still seeing these problems. To make things reproducible, I've now written a small test, that imitates ceph's behavior: On a freshly created btrfs filesystem (

Re: [3.2-rc7] slowdown, warning + oops creating lots of files

2012/1/7 Christian Brunner : > 2012/1/5 Chris Mason : >> On Fri, Jan 06, 2012 at 07:12:16AM +1100, Dave Chinner wrote: >>> On Thu, Jan 05, 2012 at 02:45:00PM -0500, Chris Mason wrote: >>> > On Thu, Jan 05, 2012 at 01:46:57PM -0500, Chris Mason wrote: >>> &g

Re: [3.2-rc7] slowdown, warning + oops creating lots of files

2012/1/5 Chris Mason : > On Fri, Jan 06, 2012 at 07:12:16AM +1100, Dave Chinner wrote: >> On Thu, Jan 05, 2012 at 02:45:00PM -0500, Chris Mason wrote: >> > On Thu, Jan 05, 2012 at 01:46:57PM -0500, Chris Mason wrote: >> > > On Thu, Jan 05, 2012 at 10:01:22AM +1100, Dave Chinner wrote: >> > > > On T

Re: WARNING: at fs/btrfs/extent-tree.c:5980

Sorry - I forgot to mention, that I'm still seeing this with: [PATCH] Btrfs: update global block_rsv when creating a new block group Christian 2011/12/13 Christian Brunner : > Hi, > > with the latest btrfs for-linus I'm seeing seeing occasional > btrfs_alloc_free_block war

WARNING: at fs/btrfs/extent-tree.c:5980

Hi, with the latest btrfs for-linus I'm seeing seeing occasional btrfs_alloc_free_block warnings on several nodes in our ceph cluster. Before the warning there is an additional block rsv -28 message, but there is plenty of free space on the disk. [201653.774412] btrfs: block rsv returned -28 [2

Re: avoid redundant block group free-space checks

2011/12/12 Alexandre Oliva : > It was pointed out to me that the test for enough free space in a block > group was wrong in that it would skip a block group that had most of its > free space reserved by a cluster. > > I offer two mutually exclusive, (so far) very lightly tested patches to > address

Re: [PATCH 02/20] Btrfs: initialize new bitmaps' list

2011/12/12 Alexandre Oliva : > On Dec  7, 2011, Christian Brunner wrote: > >> With this patch applied I get much higher write-io values than without >> it. Some of the other patches help to reduce the effect, but it's >> still significant. > >> io

Re: [PATCH 02/20] Btrfs: initialize new bitmaps' list

2011/12/7 Christian Brunner : > 2011/12/1 Christian Brunner : >> 2011/12/1 Alexandre Oliva : >>> On Nov 29, 2011, Christian Brunner wrote: >>> >>>> When I'm doing havy reading in our ceph cluster. The load and wait-io >>>> on

Re: [PATCH 02/20] Btrfs: initialize new bitmaps' list

2011/12/1 Christian Brunner : > 2011/12/1 Alexandre Oliva : >> On Nov 29, 2011, Christian Brunner wrote: >> >>> When I'm doing havy reading in our ceph cluster. The load and wait-io >>> on the patched servers is higher than on the unpatched ones. >> &g

Re: [PATCH] Btrfs: protect orphan block rsv with spin_lock

2011/12/2 Josef Bacik : > We've been seeing warnings coming out of the orphan commit stuff forever from > ceph.  Turns out it's because we're racing with checking if the orphan block > reserve is set, because we clear it outside of the spin_lock.  So leave the > normal fastpath checks where they ar

Re: [PATCH 02/20] Btrfs: initialize new bitmaps' list

2011/12/1 Alexandre Oliva : > On Nov 29, 2011, Christian Brunner wrote: > >> When I'm doing havy reading in our ceph cluster. The load and wait-io >> on the patched servers is higher than on the unpatched ones. > > That's unexpected. > >> This seems

Re: [PATCH 02/20] Btrfs: initialize new bitmaps' list

2011/11/28 Alexandre Oliva : > We're failing to create clusters with bitmaps because > setup_cluster_no_bitmap checks that the list is empty before inserting > the bitmap entry in the list for setup_cluster_bitmap, but the list > field is only initialized when it is restored from the on-disk free >

Re: WARNING: at fs/btrfs/inode.c:2198 btrfs_orphan_commit_root+0xa8/0xc0

2011/11/26 Stefan Kleijkers : > Hello Josef, > > I've new results, is this the trace you are looking for? > > Trace of OSD0: http://pastebin.com/gddLBXE4 > Dmesg of OSD0: http://pastebin.com/Uebzgkjv > > OSD1 crashed a while later with the same messages. > > Stefan Hi Josef, I ran your patch on o

Re: BUG at fs/btrfs/inode.c:1587

2011/11/16 Chris Mason : > On Tue, Nov 15, 2011 at 09:19:53AM +0100, Christian Brunner wrote: >> Hi, >> >> this time I've hit a new bug. This happened while ceph was rebuilding >> his filestore (heavy io). >> >> The btrfs version is from 3.2-rc1, appl

BUG at fs/btrfs/inode.c:1587

Hi, this time I've hit a new bug. This happened while ceph was rebuilding his filestore (heavy io). The btrfs version is from 3.2-rc1, applied to a 3.0 kernel. Regards, Christian [28981.550478] [ cut here ] [28981.555625] kernel BUG at fs/btrfs/inode.c:1587! [28981.56077

Re: WARNING: at fs/btrfs/inode.c:2198 btrfs_orphan_commit_root+0xa8/0xc0

2011/11/9 Stefan Kleijkers : > Hello, > > I'm seeing a lot of warnings in dmesg with a BTRFS filesystem. I'm using the > 3.1 kernel, I found a patch for these warnings ( > http://marc.info/?l=linux-btrfs&m=131547325515336&w=2) > , but that patc

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011/10/31 Christian Brunner : > 2011/10/31 Christian Brunner : >> 2011/10/31 Christian Brunner : >>> >>> The patch didn't hurt, but I've to tell you that I'm still seeing the >>> same old problems. Load is going up again: >>> >>>

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011/10/31 Christian Brunner : > 2011/10/31 Christian Brunner : >> >> The patch didn't hurt, but I've to tell you that I'm still seeing the >> same old problems. Load is going up again: >> >>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIM

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011/10/27 Josef Bacik : > On Tue, Oct 25, 2011 at 01:56:48PM +0200, Christian Brunner wrote: >> 2011/10/24 Josef Bacik : >> > On Mon, Oct 24, 2011 at 10:06:49AM -0700, Sage Weil wrote: >> >> [adding linux-btrfs to cc] >> >> >> >> Josef, Chris,

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011/10/26 Christian Brunner : > 2011/10/25 Josef Bacik : >> On Tue, Oct 25, 2011 at 04:15:45PM -0400, Chris Mason wrote: >>> On Tue, Oct 25, 2011 at 11:05:12AM -0400, Josef Bacik wrote: >>> > On Tue, Oct 25, 2011 at 04:25:02PM +0200, Christian Brunner wrote: >

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011/10/26 Sage Weil : > On Wed, 26 Oct 2011, Christian Brunner wrote: >> >> > Christian, have you tweaked those settings in your ceph.conf?  It would >> >> > be >> >> > something like 'journal dio = false'.  If not, can you verify that &

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011/10/25 Josef Bacik : > On Tue, Oct 25, 2011 at 04:15:45PM -0400, Chris Mason wrote: >> On Tue, Oct 25, 2011 at 11:05:12AM -0400, Josef Bacik wrote: >> > On Tue, Oct 25, 2011 at 04:25:02PM +0200, Christian Brunner wrote: >> > > >> > > Attached is a per

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011/10/25 Sage Weil : > On Tue, 25 Oct 2011, Josef Bacik wrote: >> At this point it seems like the biggest problem with latency in ceph-osd >> is not related to btrfs, the latency seems to all be from the fact that >> ceph-osd is fsyncing a block dev for whatever reason. > > There is one place whe

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011/10/25 Josef Bacik : > On Tue, Oct 25, 2011 at 04:25:02PM +0200, Christian Brunner wrote: >> 2011/10/25 Josef Bacik : >> > On Tue, Oct 25, 2011 at 01:56:48PM +0200, Christian Brunner wrote: [...] >> >> >> >> In our Ceph-OSD server we have 4 disks wit

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011/10/25 Josef Bacik : > On Tue, Oct 25, 2011 at 01:56:48PM +0200, Christian Brunner wrote: >> 2011/10/24 Josef Bacik : >> > On Mon, Oct 24, 2011 at 10:06:49AM -0700, Sage Weil wrote: >> >> [adding linux-btrfs to cc] >> >> >> >> Josef, Chris,

Re: ceph on btrfs [was Re: ceph on non-btrfs file systems]

2011/10/24 Chris Mason : > On Mon, Oct 24, 2011 at 03:51:47PM -0400, Josef Bacik wrote: >> On Mon, Oct 24, 2011 at 10:06:49AM -0700, Sage Weil wrote: >> > [adding linux-btrfs to cc] >> > >> > Josef, Chris, any ideas on the below issues? >> > >

Re: WARNING: at fs/btrfs/inode.c:2114

2011/10/20 Liu Bo : > On 10/17/2011 11:23 PM, Christian Brunner wrote: >> 2011/10/11 Christian Brunner : >> >> I have updated to a 3.0.6 kernel, with all the btrfs patches from >> josef's git repo this weekend. But I'm still seeing the following >> warn

Re: WARNING: at fs/btrfs/inode.c:2114

2011/10/11 Christian Brunner : > 2011/10/11 Liu Bo : >> On 10/10/2011 12:41 AM, Christian Brunner wrote: >>> I just realized that this is still the same warning I reported some month >>> ago. >>> >>> I thought that this had been fixed with >

Re: [PATCH] Btrfs: allow us to overcommit our enospc reservations TEST THIS PLEASE!!!

2011/10/13 Josef Bacik : [...] >> >> [  175.956273] kernel BUG at fs/btrfs/inode.c:2176! >> > >> > Ok I think I see what's happening, this patch replaces the previous one, >> > let me >> > know how it goes.  Thanks, >> > >> >> Getting a slightly different BUG this time: >> > > Ok looks like I've f

Re: OSD: no current directory

2011/10/11 Sage Weil : > On Tue, 11 Oct 2011, Christian Brunner wrote: >> 2011/10/11 Sage Weil : >> > On Tue, 11 Oct 2011, Christian Brunner wrote: >> >> Maybe this one is easier: >> >> >> >> One of our OSDs isn't starting, because ther

Re: OSD: no current directory

2011/10/11 Sage Weil : > On Tue, 11 Oct 2011, Christian Brunner wrote: >> Maybe this one is easier: >> >> One of our OSDs isn't starting, because ther is no "current" >> directory. What I have are three snap directories. >> >> total 0 >>

Re: WARNING: at fs/btrfs/inode.c:2114

2011/10/11 Liu Bo : > On 10/10/2011 12:41 AM, Christian Brunner wrote: >> I just realized that this is still the same warning I reported some month >> ago. >> >> I thought that this had been fixed with >> >> 25d37af374263243214be9d912cbb46a8e469bc7 >>

Re: Btrfs High IO-Wait

I think this is related to the sync issues. You could try the josef's git tree: git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-work.git Since yesterday I'm using it in our ceph cluster and it seems to do a better job. Regards, Christian 2011/10/9 Martin Mailand : > Hi, > I have high

Re: WARNING: at fs/btrfs/inode.c:2114

I just realized that this is still the same warning I reported some month ago. I thought that this had been fixed with 25d37af374263243214be9d912cbb46a8e469bc7 which is included in the kernel I'm using. So I think there must be another Problem. Regards, Christian 2011/10/9 Christian Br

WARNING: at fs/btrfs/inode.c:2114

I gave btrfs "for-chris" from josef's github repo a try in our ceph cluster. During the rebuild I git the following warning. Everything still seems to work... Should I be concerned? Thanks, Christian [12554.886362] [ cut here ] [12554.891693] WARNING: at fs/btrfs/inode.c:

Re: Btrfs slowdown

oment. Regards Christian 2011/8/8 Sage Weil : > Hi Christian, > > Are you still seeing this slowness? > > sage > > > On Wed, 27 Jul 2011, Christian Brunner wrote: >> 2011/7/25 Chris Mason : >> > Excerpts from Christian Brunner's message of 2011-07-25 03:54:47

Re: Btrfs slowdown

2011/7/28 Marcus Sorensen : > Christian, > > Have you checked up on the disks themselves and hardware? High > utilization can mean that the i/o load has increased, but it can also > mean that the i/o capacity has decreased.  Your traces seem to > indicate that a good portion of the time is being sp

Btrfs slowdown

Hi, we are running a ceph cluster with btrfs as it's base filesystem (kernel 3.0). At the beginning everything worked very well, but after a few days (2-3) things are getting very slow. When I look at the object store servers I see heavy disk-i/o on the btrfs filesystems (disk utilization is betw

WARNING: at fs/btrfs/inode.c:2204

I'm running a Ceph Object Store with 3.0-rc7 and patches from Josef. Occasionally I get the attached warning. Everything seems to be working after this warning, but I am concerned... Thanks, Christian [13319.808020] [ cut here ] [13319.813284] WARNING: at fs/btrfs/inode.c

Re: [PATCH] Btrfs: don't be as agressive with delalloc metadata reservations V2

2011/7/18 Josef Bacik : > On 07/18/2011 02:11 PM, Josef Bacik wrote: >> Currently we reserve enough space to COW an entirely full btree for every >> extent >> we have reserved for an inode.  This _sucks_, because you only need to COW >> once, >> and then everybody else is ok.  Unfortunately we do

Re: Delayed inode operations not doing the right thing with enospc

2011/7/13 Josef Bacik : > On 07/12/2011 11:20 AM, Christian Brunner wrote: >> 2011/6/7 Josef Bacik : >>> On 06/06/2011 09:39 PM, Miao Xie wrote: >>>> On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote: >>>>> I got a lot of these when running stress.s

Re: Delayed inode operations not doing the right thing with enospc

2011/6/7 Josef Bacik : > On 06/06/2011 09:39 PM, Miao Xie wrote: >> On fri, 03 Jun 2011 14:46:10 -0400, Josef Bacik wrote: >>> I got a lot of these when running stress.sh on my test box >>> >>> >>> >>> This is because use_block_rsv() is having to do a >>> reserve_metadata_bytes(), which shouldn't h

kernel BUG at fs/btrfs/extent-tree.c:5637!

Hi, we are running a ceph cluster with a btrfs store. Last night we ran across this btrfs BUG. Any hints on how to solve this are welcome. Regards Christian May 19 06:10:07 os00 kernel: [247212.342712] [ cut here ] May 19 06:10:07 os00 kernel: [247212.347953] kernel BUG

Re: [PATCH] Prevent oopsing in posix_acl_valid()

2011/5/3 Josef Bacik : > On 05/03/2011 12:44 PM, Daniel J Blueman wrote: >> >> If posix_acl_from_xattr() returns an error code, a negative address is >> dereferenced causing an oops; fix by checking for error code first. >> >> Signed-off-by: Daniel J Blueman >> --- >>  fs/btrfs/acl.c |    5 +++-- >

Re: btrfs BUG during Ceph cosd open() syscall

The btrfs_orphan_commit_root warning is also reproducable in our ceph environment. Regards Christian 2011/1/26 Matt Weil : > heavy writes as well > > Jan  5 16:56:46 linuscs101 kernel: [ 3666.496742] [ cut here > ] >> >>  Jan  5 16:56:46 linuscs101 kernel: [ 3666.496754] W