Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-04-01 Thread Holger Hoffstätte
On 04/01/16 03:01, Dave Chinner wrote: > Can you go back to your original kernel, and lower nr_requests to 8? Sure, did that and as expected it didn't help much. Under prolonged stress it was actually even a bit worse than writeback throttling. IMHO that's not really surprising either, since

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-04-01 Thread Holger Hoffstätte
On 04/01/16 03:01, Dave Chinner wrote: > Can you go back to your original kernel, and lower nr_requests to 8? Sure, did that and as expected it didn't help much. Under prolonged stress it was actually even a bit worse than writeback throttling. IMHO that's not really surprising either, since

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-04-01 Thread Jens Axboe
On 04/01/2016 12:27 AM, Dave Chinner wrote: On Thu, Mar 31, 2016 at 09:25:33PM -0600, Jens Axboe wrote: On 03/31/2016 06:46 PM, Dave Chinner wrote: virtio in guest, XFS direct IO -> no-op -> scsi in host. That has write back caching enabled on the guest, correct? No. It uses

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-04-01 Thread Jens Axboe
On 04/01/2016 12:27 AM, Dave Chinner wrote: On Thu, Mar 31, 2016 at 09:25:33PM -0600, Jens Axboe wrote: On 03/31/2016 06:46 PM, Dave Chinner wrote: virtio in guest, XFS direct IO -> no-op -> scsi in host. That has write back caching enabled on the guest, correct? No. It uses

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-04-01 Thread Jens Axboe
On 04/01/2016 12:16 AM, Dave Chinner wrote: On Thu, Mar 31, 2016 at 09:39:25PM -0600, Jens Axboe wrote: On 03/31/2016 09:29 PM, Jens Axboe wrote: I can't seem to reproduce this at all. On an nvme device, I get a fairly steady 60K/sec file creation rate, and we're nowhere near being IO bound.

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-04-01 Thread Jens Axboe
On 04/01/2016 12:16 AM, Dave Chinner wrote: On Thu, Mar 31, 2016 at 09:39:25PM -0600, Jens Axboe wrote: On 03/31/2016 09:29 PM, Jens Axboe wrote: I can't seem to reproduce this at all. On an nvme device, I get a fairly steady 60K/sec file creation rate, and we're nowhere near being IO bound.

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-04-01 Thread Dave Chinner
On Thu, Mar 31, 2016 at 09:25:33PM -0600, Jens Axboe wrote: > On 03/31/2016 06:46 PM, Dave Chinner wrote: > >>>virtio in guest, XFS direct IO -> no-op -> scsi in host. > >> > >>That has write back caching enabled on the guest, correct? > > > >No. It uses virtio,cache=none (that's the "XFS Direct

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-04-01 Thread Dave Chinner
On Thu, Mar 31, 2016 at 09:25:33PM -0600, Jens Axboe wrote: > On 03/31/2016 06:46 PM, Dave Chinner wrote: > >>>virtio in guest, XFS direct IO -> no-op -> scsi in host. > >> > >>That has write back caching enabled on the guest, correct? > > > >No. It uses virtio,cache=none (that's the "XFS Direct

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-04-01 Thread Dave Chinner
On Thu, Mar 31, 2016 at 09:39:25PM -0600, Jens Axboe wrote: > On 03/31/2016 09:29 PM, Jens Axboe wrote: > >>>I can't seem to reproduce this at all. On an nvme device, I get a > >>>fairly steady 60K/sec file creation rate, and we're nowhere near > >>>being IO bound. So the throttling has no effect

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-04-01 Thread Dave Chinner
On Thu, Mar 31, 2016 at 09:39:25PM -0600, Jens Axboe wrote: > On 03/31/2016 09:29 PM, Jens Axboe wrote: > >>>I can't seem to reproduce this at all. On an nvme device, I get a > >>>fairly steady 60K/sec file creation rate, and we're nowhere near > >>>being IO bound. So the throttling has no effect

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Dave Chinner
On Thu, Mar 31, 2016 at 09:29:30PM -0600, Jens Axboe wrote: > On 03/31/2016 06:56 PM, Dave Chinner wrote: > >I'm not changing the host kernels - it's a production machine and so > >it runs long uptime testing of stable kernels. (e.g. catch slow > >memory leaks, etc). So if you've disabled

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Dave Chinner
On Thu, Mar 31, 2016 at 09:29:30PM -0600, Jens Axboe wrote: > On 03/31/2016 06:56 PM, Dave Chinner wrote: > >I'm not changing the host kernels - it's a production machine and so > >it runs long uptime testing of stable kernels. (e.g. catch slow > >memory leaks, etc). So if you've disabled

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 09:29 PM, Jens Axboe wrote: I can't seem to reproduce this at all. On an nvme device, I get a fairly steady 60K/sec file creation rate, and we're nowhere near being IO bound. So the throttling has no effect at all. That's too slow to show the stalls - your likely concurrency

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 09:29 PM, Jens Axboe wrote: I can't seem to reproduce this at all. On an nvme device, I get a fairly steady 60K/sec file creation rate, and we're nowhere near being IO bound. So the throttling has no effect at all. That's too slow to show the stalls - your likely concurrency

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 09:29 PM, Jens Axboe wrote: I'm not changing the host kernels - it's a production machine and so it runs long uptime testing of stable kernels. (e.g. catch slow memory leaks, etc). So if you've disabled throttling in the guest, I can't test the throttling changes. Right, that'd

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 09:29 PM, Jens Axboe wrote: I'm not changing the host kernels - it's a production machine and so it runs long uptime testing of stable kernels. (e.g. catch slow memory leaks, etc). So if you've disabled throttling in the guest, I can't test the throttling changes. Right, that'd

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 06:56 PM, Dave Chinner wrote: On Thu, Mar 31, 2016 at 10:21:04AM -0600, Jens Axboe wrote: On 03/31/2016 08:29 AM, Jens Axboe wrote: What I see in these performance dips is the XFS transaction subsystem stalling *completely* - instead of running at a steady state of around 350,000

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 06:56 PM, Dave Chinner wrote: On Thu, Mar 31, 2016 at 10:21:04AM -0600, Jens Axboe wrote: On 03/31/2016 08:29 AM, Jens Axboe wrote: What I see in these performance dips is the XFS transaction subsystem stalling *completely* - instead of running at a steady state of around 350,000

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 06:46 PM, Dave Chinner wrote: On Thu, Mar 31, 2016 at 08:29:35AM -0600, Jens Axboe wrote: On 03/31/2016 02:24 AM, Dave Chinner wrote: On Wed, Mar 30, 2016 at 09:07:48AM -0600, Jens Axboe wrote: Hi, This patchset isn't as much a final solution, as it's demonstration of what I

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 06:46 PM, Dave Chinner wrote: On Thu, Mar 31, 2016 at 08:29:35AM -0600, Jens Axboe wrote: On 03/31/2016 02:24 AM, Dave Chinner wrote: On Wed, Mar 30, 2016 at 09:07:48AM -0600, Jens Axboe wrote: Hi, This patchset isn't as much a final solution, as it's demonstration of what I

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Dave Chinner
On Thu, Mar 31, 2016 at 10:09:56PM +, Holger Hoffstätte wrote: > > Hi, > > Jens mentioned on Twitter I should post my experience here as well, > so here we go. > > I've backported this series (incl. updates) to stable-4.4.x - not too > difficult, minus the NVM part which I don't need anyway

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Dave Chinner
On Thu, Mar 31, 2016 at 10:09:56PM +, Holger Hoffstätte wrote: > > Hi, > > Jens mentioned on Twitter I should post my experience here as well, > so here we go. > > I've backported this series (incl. updates) to stable-4.4.x - not too > difficult, minus the NVM part which I don't need anyway

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Dave Chinner
On Thu, Mar 31, 2016 at 10:21:04AM -0600, Jens Axboe wrote: > On 03/31/2016 08:29 AM, Jens Axboe wrote: > >>What I see in these performance dips is the XFS transaction > >>subsystem stalling *completely* - instead of running at a steady > >>state of around 350,000 transactions/s, there are *zero*

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Dave Chinner
On Thu, Mar 31, 2016 at 10:21:04AM -0600, Jens Axboe wrote: > On 03/31/2016 08:29 AM, Jens Axboe wrote: > >>What I see in these performance dips is the XFS transaction > >>subsystem stalling *completely* - instead of running at a steady > >>state of around 350,000 transactions/s, there are *zero*

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Dave Chinner
On Thu, Mar 31, 2016 at 08:29:35AM -0600, Jens Axboe wrote: > On 03/31/2016 02:24 AM, Dave Chinner wrote: > >On Wed, Mar 30, 2016 at 09:07:48AM -0600, Jens Axboe wrote: > >>Hi, > >> > >>This patchset isn't as much a final solution, as it's demonstration > >>of what I believe is a huge issue. Since

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Dave Chinner
On Thu, Mar 31, 2016 at 08:29:35AM -0600, Jens Axboe wrote: > On 03/31/2016 02:24 AM, Dave Chinner wrote: > >On Wed, Mar 30, 2016 at 09:07:48AM -0600, Jens Axboe wrote: > >>Hi, > >> > >>This patchset isn't as much a final solution, as it's demonstration > >>of what I believe is a huge issue. Since

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Holger Hoffstätte
Hi, Jens mentioned on Twitter I should post my experience here as well, so here we go. I've backported this series (incl. updates) to stable-4.4.x - not too difficult, minus the NVM part which I don't need anyway - and have been running it for the past few days without any problem whatsoever,

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Holger Hoffstätte
Hi, Jens mentioned on Twitter I should post my experience here as well, so here we go. I've backported this series (incl. updates) to stable-4.4.x - not too difficult, minus the NVM part which I don't need anyway - and have been running it for the past few days without any problem whatsoever,

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 08:29 AM, Jens Axboe wrote: What I see in these performance dips is the XFS transaction subsystem stalling *completely* - instead of running at a steady state of around 350,000 transactions/s, there are *zero* transactions running for periods of up to ten seconds. This co-incides

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 08:29 AM, Jens Axboe wrote: What I see in these performance dips is the XFS transaction subsystem stalling *completely* - instead of running at a steady state of around 350,000 transactions/s, there are *zero* transactions running for periods of up to ten seconds. This co-incides

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 02:24 AM, Dave Chinner wrote: On Wed, Mar 30, 2016 at 09:07:48AM -0600, Jens Axboe wrote: Hi, This patchset isn't as much a final solution, as it's demonstration of what I believe is a huge issue. Since the dawn of time, our background buffered writeback has sucked. When we do

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Jens Axboe
On 03/31/2016 02:24 AM, Dave Chinner wrote: On Wed, Mar 30, 2016 at 09:07:48AM -0600, Jens Axboe wrote: Hi, This patchset isn't as much a final solution, as it's demonstration of what I believe is a huge issue. Since the dawn of time, our background buffered writeback has sucked. When we do

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Dave Chinner
On Wed, Mar 30, 2016 at 09:07:48AM -0600, Jens Axboe wrote: > Hi, > > This patchset isn't as much a final solution, as it's demonstration > of what I believe is a huge issue. Since the dawn of time, our > background buffered writeback has sucked. When we do background > buffered writeback, it

Re: [PATCHSET v3][RFC] Make background writeback not suck

2016-03-31 Thread Dave Chinner
On Wed, Mar 30, 2016 at 09:07:48AM -0600, Jens Axboe wrote: > Hi, > > This patchset isn't as much a final solution, as it's demonstration > of what I believe is a huge issue. Since the dawn of time, our > background buffered writeback has sucked. When we do background > buffered writeback, it

[PATCHSET v3][RFC] Make background writeback not suck

2016-03-30 Thread Jens Axboe
Hi, This patchset isn't as much a final solution, as it's demonstration of what I believe is a huge issue. Since the dawn of time, our background buffered writeback has sucked. When we do background buffered writeback, it should have little impact on foreground activity. That's the definition of

[PATCHSET v3][RFC] Make background writeback not suck

2016-03-30 Thread Jens Axboe
Hi, This patchset isn't as much a final solution, as it's demonstration of what I believe is a huge issue. Since the dawn of time, our background buffered writeback has sucked. When we do background buffered writeback, it should have little impact on foreground activity. That's the definition of