Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-03-07 Thread Leroy van Logchem
> > actually a global dirty_ratio causes interference between devices which > > should otherwise not block each other... > > > > if you set up a "dd if=/dev/zero of=/dev/sdb bs=1M" it shouldn't affect > > write performance on sda -- but it does... because the dd basically > > dirties all of

Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-03-07 Thread Leroy van Logchem
actually a global dirty_ratio causes interference between devices which should otherwise not block each other... if you set up a dd if=/dev/zero of=/dev/sdb bs=1M it shouldn't affect write performance on sda -- but it does... because the dd basically dirties all of the

Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-01-11 Thread Andrew Morton
On Thu, 11 Jan 2007 14:35:06 -0800 (PST) dean gaudet <[EMAIL PROTECTED]> wrote: > actually a global dirty_ratio causes interference between devices which > should otherwise not block each other... > > if you set up a "dd if=/dev/zero of=/dev/sdb bs=1M" it shouldn't affect > write performance

Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-01-11 Thread dean gaudet
On Thu, 11 Jan 2007, Andrew Morton wrote: > On Thu, 11 Jan 2007 03:04:00 -0800 (PST) > dean gaudet <[EMAIL PROTECTED]> wrote: > > > On Tue, 9 Jan 2007, Neil Brown wrote: > > > > > Imagine a machine with lots of memory - say 100Gig. > > > > i've had these problems on machines as "small" as

Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-01-11 Thread Andrew Morton
On Thu, 11 Jan 2007 03:04:00 -0800 (PST) dean gaudet <[EMAIL PROTECTED]> wrote: > On Tue, 9 Jan 2007, Neil Brown wrote: > > > Imagine a machine with lots of memory - say 100Gig. > > i've had these problems on machines as "small" as 8GiB. the real problem > is that the kernel will let millions

Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-01-11 Thread dean gaudet
On Tue, 9 Jan 2007, Neil Brown wrote: > Imagine a machine with lots of memory - say 100Gig. i've had these problems on machines as "small" as 8GiB. the real problem is that the kernel will let millions of potential (write) IO ops stack up for a device which can handle only mere 100s of IOs

Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-01-11 Thread dean gaudet
On Tue, 9 Jan 2007, Neil Brown wrote: Imagine a machine with lots of memory - say 100Gig. i've had these problems on machines as small as 8GiB. the real problem is that the kernel will let millions of potential (write) IO ops stack up for a device which can handle only mere 100s of IOs per

Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-01-11 Thread Andrew Morton
On Thu, 11 Jan 2007 03:04:00 -0800 (PST) dean gaudet [EMAIL PROTECTED] wrote: On Tue, 9 Jan 2007, Neil Brown wrote: Imagine a machine with lots of memory - say 100Gig. i've had these problems on machines as small as 8GiB. the real problem is that the kernel will let millions of

Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-01-11 Thread dean gaudet
On Thu, 11 Jan 2007, Andrew Morton wrote: On Thu, 11 Jan 2007 03:04:00 -0800 (PST) dean gaudet [EMAIL PROTECTED] wrote: On Tue, 9 Jan 2007, Neil Brown wrote: Imagine a machine with lots of memory - say 100Gig. i've had these problems on machines as small as 8GiB. the real

Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-01-11 Thread Andrew Morton
On Thu, 11 Jan 2007 14:35:06 -0800 (PST) dean gaudet [EMAIL PROTECTED] wrote: actually a global dirty_ratio causes interference between devices which should otherwise not block each other... if you set up a dd if=/dev/zero of=/dev/sdb bs=1M it shouldn't affect write performance on sda --

Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-01-09 Thread Andrew Morton
On Wed, 10 Jan 2007 14:29:35 +1100 Neil Brown <[EMAIL PROTECTED]> wrote: > > > > It would be better if we can avoid creating the second global variable. Is > > it not possible to remove dirty_ratio? Make everything work off > > vm_dirty_kb and do arithmetricks at the /proc/sys/vm/dirty_ratio

Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-01-09 Thread Neil Brown
On Tuesday January 9, [EMAIL PROTECTED] wrote: > > Actually, ext3 doesn't work that way. The atime update will go into the > "running transaction", which is an instance of journal_t which is separate > from the committing transaction. Hmm... fair enough. start_this_handle (which is called

Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-01-09 Thread Neil Brown
On Tuesday January 9, [EMAIL PROTECTED] wrote: > > Could be IO scheduler borkage, could be ext3 borkage. A well-timed sysrq-T > will tell us, and is worth doing (please). The problem has been reported against reiserfs and ext3, and against SLES9 and SLES10. The big machine I can test with is

Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-01-09 Thread Andrew Morton
On Tue, 9 Jan 2007 19:57:50 +1100 Neil Brown <[EMAIL PROTECTED]> wrote: > > Imagine a machine with lots of memory - say 100Gig. > > Suppose there is one (largish) filesystem that is ext3 (or maybe > reiser) with the default data=ordered. > > Suppose this filesystem is being written to steadily

Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-01-09 Thread Andrew Morton
On Tue, 9 Jan 2007 19:57:50 +1100 Neil Brown [EMAIL PROTECTED] wrote: Imagine a machine with lots of memory - say 100Gig. Suppose there is one (largish) filesystem that is ext3 (or maybe reiser) with the default data=ordered. Suppose this filesystem is being written to steadily so that

Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-01-09 Thread Neil Brown
On Tuesday January 9, [EMAIL PROTECTED] wrote: Could be IO scheduler borkage, could be ext3 borkage. A well-timed sysrq-T will tell us, and is worth doing (please). The problem has been reported against reiserfs and ext3, and against SLES9 and SLES10. The big machine I can test with is

Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-01-09 Thread Neil Brown
On Tuesday January 9, [EMAIL PROTECTED] wrote: Actually, ext3 doesn't work that way. The atime update will go into the running transaction, which is an instance of journal_t which is separate from the committing transaction. Hmm... fair enough. start_this_handle (which is called eventually

Re: [PATCH - RFC] allow setting vm_dirty below 1% for large memory machines

2007-01-09 Thread Andrew Morton
On Wed, 10 Jan 2007 14:29:35 +1100 Neil Brown [EMAIL PROTECTED] wrote: It would be better if we can avoid creating the second global variable. Is it not possible to remove dirty_ratio? Make everything work off vm_dirty_kb and do arithmetricks at the /proc/sys/vm/dirty_ratio interface?