Re: Local DoS through write heavy I/O on CFQ & Deadline

2012-10-18 Thread Jan Kara
On Thu 18-10-12 16:13:58, Chris Friesen wrote: > On 10/18/2012 03:28 PM, Jan Kara wrote: > > > Yeah, ionice has its limitations. The problem is that all buffered > >writes happen just into memory (so completely independently of ionice > >settings). Subsequent writing of dirty memory to disk

Re: Local DoS through write heavy I/O on CFQ & Deadline

2012-10-18 Thread Chris Friesen
On 10/18/2012 03:28 PM, Jan Kara wrote: Yeah, ionice has its limitations. The problem is that all buffered writes happen just into memory (so completely independently of ionice settings). Subsequent writing of dirty memory to disk happens using flusher thread which is a kernel process and it

Re: Local DoS through write heavy I/O on CFQ & Deadline

2012-10-18 Thread Jan Kara
Hello, On Fri 12-10-12 17:29:50, Alex Bligh wrote: > --On 12 October 2012 16:58:39 +0200 Michal Hocko wrote: Let me explain a couple of things... > >Once dirty_ratio (resp. dirty_bytes) limit is hit then the process which > >writes gets throttled. If this is not the case then there is a bug

Re: Local DoS through write heavy I/O on CFQ Deadline

2012-10-18 Thread Jan Kara
Hello, On Fri 12-10-12 17:29:50, Alex Bligh wrote: --On 12 October 2012 16:58:39 +0200 Michal Hocko mho...@suse.cz wrote: Let me explain a couple of things... Once dirty_ratio (resp. dirty_bytes) limit is hit then the process which writes gets throttled. If this is not the case then there

Re: Local DoS through write heavy I/O on CFQ Deadline

2012-10-18 Thread Chris Friesen
On 10/18/2012 03:28 PM, Jan Kara wrote: Yeah, ionice has its limitations. The problem is that all buffered writes happen just into memory (so completely independently of ionice settings). Subsequent writing of dirty memory to disk happens using flusher thread which is a kernel process and it

Re: Local DoS through write heavy I/O on CFQ Deadline

2012-10-18 Thread Jan Kara
On Thu 18-10-12 16:13:58, Chris Friesen wrote: On 10/18/2012 03:28 PM, Jan Kara wrote: Yeah, ionice has its limitations. The problem is that all buffered writes happen just into memory (so completely independently of ionice settings). Subsequent writing of dirty memory to disk happens

Re: Local DoS through write heavy I/O on CFQ & Deadline

2012-10-15 Thread Michal Hocko
On Fri 12-10-12 17:29:50, Alex Bligh wrote: > Michael, > > --On 12 October 2012 16:58:39 +0200 Michal Hocko wrote: > > >Once dirty_ratio (resp. dirty_bytes) limit is hit then the process which > >writes gets throttled. If this is not the case then there is a bug in > >the throttling code. > >

Re: Local DoS through write heavy I/O on CFQ Deadline

2012-10-15 Thread Michal Hocko
On Fri 12-10-12 17:29:50, Alex Bligh wrote: Michael, --On 12 October 2012 16:58:39 +0200 Michal Hocko mho...@suse.cz wrote: Once dirty_ratio (resp. dirty_bytes) limit is hit then the process which writes gets throttled. If this is not the case then there is a bug in the throttling code.

Re: Local DoS through write heavy I/O on CFQ & Deadline

2012-10-14 Thread Dave Chinner
On Thu, Oct 11, 2012 at 01:23:32PM +0100, Alex Bligh wrote: > We have noticed significant I/O scheduling issues on both the CFQ and the > deadline scheduler where a non-root user can starve any other process of > any I/O for minutes at a time. The problem is more serious using CFQ but is > still

Re: Local DoS through write heavy I/O on CFQ Deadline

2012-10-14 Thread Dave Chinner
On Thu, Oct 11, 2012 at 01:23:32PM +0100, Alex Bligh wrote: We have noticed significant I/O scheduling issues on both the CFQ and the deadline scheduler where a non-root user can starve any other process of any I/O for minutes at a time. The problem is more serious using CFQ but is still an

Re: Local DoS through write heavy I/O on CFQ & Deadline

2012-10-13 Thread Hillf Danton
On Sun, Oct 14, 2012 at 3:33 AM, Alex Bligh wrote: > Or perhaps I have the wrong end of the stick. Never mind a friendly link:) -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at

Re: Local DoS through write heavy I/O on CFQ & Deadline

2012-10-13 Thread Alex Bligh
--On 13 October 2012 21:53:09 +0800 Hillf Danton wrote: Take a look at the "wait for writeback" problem please. Linux 3.0+ Disk performance problem - wrong pdflush behaviour https://lkml.org/lkml/2012/10/10/412 I'm guessing that's related but may not be the whole story. My

Re: Local DoS through write heavy I/O on CFQ & Deadline

2012-10-13 Thread Hillf Danton
Hi Alex, On Sat, Oct 13, 2012 at 12:29 AM, Alex Bligh wrote: > Michael, > > --On 12 October 2012 16:58:39 +0200 Michal Hocko wrote: > >> Once dirty_ratio (resp. dirty_bytes) limit is hit then the process which >> writes gets throttled. If this is not the case then there is a bug in >> the

Re: Local DoS through write heavy I/O on CFQ Deadline

2012-10-13 Thread Hillf Danton
Hi Alex, On Sat, Oct 13, 2012 at 12:29 AM, Alex Bligh a...@alex.org.uk wrote: Michael, --On 12 October 2012 16:58:39 +0200 Michal Hocko mho...@suse.cz wrote: Once dirty_ratio (resp. dirty_bytes) limit is hit then the process which writes gets throttled. If this is not the case then there is

Re: Local DoS through write heavy I/O on CFQ Deadline

2012-10-13 Thread Alex Bligh
--On 13 October 2012 21:53:09 +0800 Hillf Danton dhi...@gmail.com wrote: Take a look at the wait for writeback problem please. Linux 3.0+ Disk performance problem - wrong pdflush behaviour https://lkml.org/lkml/2012/10/10/412 I'm guessing that's related but may not be the

Re: Local DoS through write heavy I/O on CFQ Deadline

2012-10-13 Thread Hillf Danton
On Sun, Oct 14, 2012 at 3:33 AM, Alex Bligh a...@alex.org.uk wrote: Or perhaps I have the wrong end of the stick. Never mind a friendly link:) -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at

Re: Local DoS through write heavy I/O on CFQ & Deadline

2012-10-12 Thread Alex Bligh
Michael, --On 12 October 2012 16:58:39 +0200 Michal Hocko wrote: Once dirty_ratio (resp. dirty_bytes) limit is hit then the process which writes gets throttled. If this is not the case then there is a bug in the throttling code. I believe that is the problem. Isn't the only thing that is

Re: Local DoS through write heavy I/O on CFQ & Deadline

2012-10-12 Thread Michal Hocko
On Fri 12-10-12 15:48:34, Alex Bligh wrote: > > > --On 12 October 2012 15:30:45 +0200 Michal Hocko wrote: > > >>Full info, including logs and scripts can be found at: > >> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1064521 > > > >You seem to have 8G of RAM and dirty_ratio=20 resp. >

Re: Local DoS through write heavy I/O on CFQ & Deadline

2012-10-12 Thread Alex Bligh
--On 12 October 2012 15:30:45 +0200 Michal Hocko wrote: Full info, including logs and scripts can be found at: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1064521 You seem to have 8G of RAM and dirty_ratio=20 resp. dirty_background_ratio=10 which means that 1.5G worth of dirty

Re: Local DoS through write heavy I/O on CFQ & Deadline

2012-10-12 Thread Michal Hocko
On Thu 11-10-12 13:23:32, Alex Bligh wrote: > We have noticed significant I/O scheduling issues on both the CFQ and the > deadline scheduler where a non-root user can starve any other process of > any I/O for minutes at a time. The problem is more serious using CFQ but is > still an effective

Re: Local DoS through write heavy I/O on CFQ & Deadline

2012-10-12 Thread Alex Bligh
Alan, --On 11 October 2012 14:46:34 +0100 Alan Cox wrote: We have reproduced this on multiple hardware environments, using 3.2 (/proc/version_signature gives "Ubuntu 3.2.0-29.46-generic 3.2.24"). Anecdotally we believe the situation has worsened since 3.0. I've certainly seen this on 3.0

Re: Local DoS through write heavy I/O on CFQ Deadline

2012-10-12 Thread Alex Bligh
Alan, --On 11 October 2012 14:46:34 +0100 Alan Cox a...@lxorguk.ukuu.org.uk wrote: We have reproduced this on multiple hardware environments, using 3.2 (/proc/version_signature gives Ubuntu 3.2.0-29.46-generic 3.2.24). Anecdotally we believe the situation has worsened since 3.0. I've

Re: Local DoS through write heavy I/O on CFQ Deadline

2012-10-12 Thread Michal Hocko
On Thu 11-10-12 13:23:32, Alex Bligh wrote: We have noticed significant I/O scheduling issues on both the CFQ and the deadline scheduler where a non-root user can starve any other process of any I/O for minutes at a time. The problem is more serious using CFQ but is still an effective local

Re: Local DoS through write heavy I/O on CFQ Deadline

2012-10-12 Thread Alex Bligh
--On 12 October 2012 15:30:45 +0200 Michal Hocko mho...@suse.cz wrote: Full info, including logs and scripts can be found at: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1064521 You seem to have 8G of RAM and dirty_ratio=20 resp. dirty_background_ratio=10 which means that 1.5G

Re: Local DoS through write heavy I/O on CFQ Deadline

2012-10-12 Thread Michal Hocko
On Fri 12-10-12 15:48:34, Alex Bligh wrote: --On 12 October 2012 15:30:45 +0200 Michal Hocko mho...@suse.cz wrote: Full info, including logs and scripts can be found at: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1064521 You seem to have 8G of RAM and dirty_ratio=20 resp.

Re: Local DoS through write heavy I/O on CFQ Deadline

2012-10-12 Thread Alex Bligh
Michael, --On 12 October 2012 16:58:39 +0200 Michal Hocko mho...@suse.cz wrote: Once dirty_ratio (resp. dirty_bytes) limit is hit then the process which writes gets throttled. If this is not the case then there is a bug in the throttling code. I believe that is the problem. Isn't the only

Re: Local DoS through write heavy I/O on CFQ & Deadline

2012-10-11 Thread Alan Cox
> We have reproduced this on multiple hardware environments, using 3.2 > (/proc/version_signature gives "Ubuntu 3.2.0-29.46-generic 3.2.24"). > Anecdotally we believe the situation has worsened since 3.0. I've certainly seen this on 3.0 and 3.2, but do you still see it on 3.5/6 ? -- To

Local DoS through write heavy I/O on CFQ & Deadline

2012-10-11 Thread Alex Bligh
We have noticed significant I/O scheduling issues on both the CFQ and the deadline scheduler where a non-root user can starve any other process of any I/O for minutes at a time. The problem is more serious using CFQ but is still an effective local DoS vector using Deadline. A simple way to

Local DoS through write heavy I/O on CFQ Deadline

2012-10-11 Thread Alex Bligh
We have noticed significant I/O scheduling issues on both the CFQ and the deadline scheduler where a non-root user can starve any other process of any I/O for minutes at a time. The problem is more serious using CFQ but is still an effective local DoS vector using Deadline. A simple way to

Re: Local DoS through write heavy I/O on CFQ Deadline

2012-10-11 Thread Alan Cox
We have reproduced this on multiple hardware environments, using 3.2 (/proc/version_signature gives Ubuntu 3.2.0-29.46-generic 3.2.24). Anecdotally we believe the situation has worsened since 3.0. I've certainly seen this on 3.0 and 3.2, but do you still see it on 3.5/6 ? -- To unsubscribe