Re: Btrfs slowdown with ceph (how to reproduce)

2012-01-24 Thread Martin Mailand
Hi Chris, great to hear that, could you give me a ping if you fixed it, than I can retry it? -martin Am 24.01.2012 20:40, schrieb Chris Mason: On Tue, Jan 24, 2012 at 08:15:58PM +0100, Martin Mailand wrote: Hi I tried the branch on one of my ceph osd, and there is a big difference in the per

Re: Btrfs slowdown with ceph (how to reproduce)

2012-01-24 Thread Chris Mason
On Tue, Jan 24, 2012 at 08:15:58PM +0100, Martin Mailand wrote: > Hi > I tried the branch on one of my ceph osd, and there is a big > difference in the performance. > The average request size stayed high, but after around a hour the > kernel crashed. > > IOstat > http://pastebin.com/xjuriJ6J > >

Re: Btrfs slowdown with ceph (how to reproduce)

2012-01-24 Thread Martin Mailand
Hi I tried the branch on one of my ceph osd, and there is a big difference in the performance. The average request size stayed high, but after around a hour the kernel crashed. IOstat http://pastebin.com/xjuriJ6J Kernel trace http://pastebin.com/SYE95GgH -martin Am 23.01.2012 19:50, schrieb

Re: Btrfs slowdown with ceph (how to reproduce)

2012-01-23 Thread Christian Brunner
2012/1/23 Chris Mason : > On Mon, Jan 23, 2012 at 01:19:29PM -0500, Josef Bacik wrote: >> On Fri, Jan 20, 2012 at 01:13:37PM +0100, Christian Brunner wrote: >> > As you might know, I have been seeing btrfs slowdowns in our ceph >> > cluster for quite some time. Even with the latest btrfs code for 3

Re: Btrfs slowdown with ceph (how to reproduce)

2012-01-23 Thread Chris Mason
On Mon, Jan 23, 2012 at 01:19:29PM -0500, Josef Bacik wrote: > On Fri, Jan 20, 2012 at 01:13:37PM +0100, Christian Brunner wrote: > > As you might know, I have been seeing btrfs slowdowns in our ceph > > cluster for quite some time. Even with the latest btrfs code for 3.3 > > I'm still seeing these

Re: Btrfs slowdown with ceph (how to reproduce)

2012-01-23 Thread Josef Bacik
On Fri, Jan 20, 2012 at 01:13:37PM +0100, Christian Brunner wrote: > As you might know, I have been seeing btrfs slowdowns in our ceph > cluster for quite some time. Even with the latest btrfs code for 3.3 > I'm still seeing these problems. To make things reproducible, I've now > written a small te

Btrfs slowdown with ceph (how to reproduce)

2012-01-20 Thread Christian Brunner
As you might know, I have been seeing btrfs slowdowns in our ceph cluster for quite some time. Even with the latest btrfs code for 3.3 I'm still seeing these problems. To make things reproducible, I've now written a small test, that imitates ceph's behavior: On a freshly created btrfs filesystem (

Re: Btrfs slowdown

2011-08-09 Thread Christian Brunner
Hi Sage, I did some testing with btrfs-unstable yesterday. With the recent commit from Chris it looks quite good: "Btrfs: force unplugs when switching from high to regular priority bios" However I can't test it extensively, because our main environment is on ext4 at the moment. Regards Christi

Re: Btrfs slowdown

2011-08-08 Thread Sage Weil
Hi Christian, Are you still seeing this slowness? sage On Wed, 27 Jul 2011, Christian Brunner wrote: > 2011/7/25 Chris Mason : > > Excerpts from Christian Brunner's message of 2011-07-25 03:54:47 -0400: > >> Hi, > >> > >> we are running a ceph cluster with btrfs as it's base filesystem > >> (ke

Re: Btrfs Slowdown (due to Memory Handling?)

2011-08-04 Thread Chris Mason
Excerpts from Mitch Harder's message of 2011-08-04 14:40:20 -0400: > On Thu, Aug 4, 2011 at 10:05 AM, Chris Mason wrote: > >> > >> Ok, so I'm going to guess that your problem is really with either file > >> layout or just us using more metadata pages than the others.  The file > >> layout part is

Re: Btrfs Slowdown (due to Memory Handling?)

2011-08-04 Thread Mitch Harder
On Thu, Aug 4, 2011 at 10:05 AM, Chris Mason wrote: > Excerpts from Chris Mason's message of 2011-08-04 11:04:54 -0400: >> Excerpts from Mitch Harder's message of 2011-08-04 10:45:51 -0400: >> > On Thu, Aug 4, 2011 at 9:22 AM, Chris Mason wrote: >> > > Excerpts from Mitch Harder's message of 2011

Re: Btrfs Slowdown (due to Memory Handling?)

2011-08-04 Thread Chris Mason
Excerpts from Mitch Harder's message of 2011-08-02 10:35:54 -0400: > I'm running into a significant slowdown in Btrfs (> 10x slower than > normal) that appears to be due to some issue between how Btrfs is > allocating memory, and how the kernel is expecting Btrfs to allocate > memory. > > The prob

Re: Btrfs slowdown

2011-08-03 Thread mck
I can confirm this as well (64-bit, Core i7, single-disk). > The issue seems to be gone in 3.0.0. After a few hours working 3.0.0 slows down on me too. The performance becomes unusable and a reboot is a must. Certain applications (particularly evolution ad firefox) are next to permanently greyed

Btrfs Slowdown (due to Memory Handling?)

2011-08-02 Thread Mitch Harder
I'm running into a significant slowdown in Btrfs (> 10x slower than normal) that appears to be due to some issue between how Btrfs is allocating memory, and how the kernel is expecting Btrfs to allocate memory. The problem does seem to be somewhat hardware specific. I can reproduce on two of my c

Re: Btrfs slowdown

2011-07-28 Thread Sage Weil
On Thu, 28 Jul 2011, Christian Brunner wrote: > When I look at the latencytop results, there is a high latency when > calling "btrfs_commit_transaction_async". Isn't "async" supposed to > return immediately? It depends. That function has to block until the commit has started before returning in

Re: Btrfs slowdown

2011-07-28 Thread Christian Brunner
2011/7/28 Marcus Sorensen : > Christian, > > Have you checked up on the disks themselves and hardware? High > utilization can mean that the i/o load has increased, but it can also > mean that the i/o capacity has decreased.  Your traces seem to > indicate that a good portion of the time is being sp

Re: Btrfs slowdown

2011-07-27 Thread Marcus Sorensen
Christian, Have you checked up on the disks themselves and hardware? High utilization can mean that the i/o load has increased, but it can also mean that the i/o capacity has decreased. Your traces seem to indicate that a good portion of the time is being spent on commits, that could be waiting o

Re: Btrfs slowdown

2011-07-25 Thread Chris Mason
Excerpts from Christian Brunner's message of 2011-07-25 03:54:47 -0400: > Hi, > > we are running a ceph cluster with btrfs as it's base filesystem > (kernel 3.0). At the beginning everything worked very well, but after > a few days (2-3) things are getting very slow. > > When I look at the object

Re: Btrfs slowdown

2011-07-25 Thread Jeremy Sanders
Christian Brunner wrote: > we are running a ceph cluster with btrfs as it's base filesystem > (kernel 3.0). At the beginning everything worked very well, but after > a few days (2-3) things are getting very slow. We get quite a slowdown over time, doing rsyncs to different snapshots. Btrfs seems

Re: Btrfs slowdown

2011-07-25 Thread Andrej Podzimek
Just a quick note: The issue seems to be gone in 3.0.0. But that's just a wild guess based on 1/2 hour without thrashing. :-) Andrej Hello, I can see something similar on the machines I maintain, mostly single-disk setups with a 2.6.39 kernel: 1) Heavy and frequent disk thrashing, although

Re: Btrfs slowdown

2011-07-25 Thread Andrej Podzimek
Hello, I can see something similar on the machines I maintain, mostly single-disk setups with a 2.6.39 kernel: 1) Heavy and frequent disk thrashing, although less than 20% of RAM is used and no swap usage is reported. 2) During the disk thrashing, some processors (usually 2 or

Btrfs slowdown

2011-07-25 Thread Christian Brunner
Hi, we are running a ceph cluster with btrfs as it's base filesystem (kernel 3.0). At the beginning everything worked very well, but after a few days (2-3) things are getting very slow. When I look at the object store servers I see heavy disk-i/o on the btrfs filesystems (disk utilization is betw