Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Rik van Riel
On Thu, 31 Jan 2008 12:32:24 +0100 Andi Kleen <[EMAIL PROTECTED]> wrote: > On Thu, Jan 31, 2008 at 05:52:09AM -0500, Rik van Riel wrote: > > Don't malloc() and free() hopelessly fragment memory > > over time, ensuring that little related data can be > > found inside each 1MB chunk if the process

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Andi Kleen
On Thu, Jan 31, 2008 at 05:52:09AM -0500, Rik van Riel wrote: > On Thu, 31 Jan 2008 12:06:10 +0100 > Andi Kleen <[EMAIL PROTECTED]> wrote: > > > > Yeah, the 2.5 switch to physical scanning killed us there. > > > > > > I still don't know why my > > >

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Rik van Riel
On Thu, 31 Jan 2008 12:06:10 +0100 Andi Kleen <[EMAIL PROTECTED]> wrote: > > Yeah, the 2.5 switch to physical scanning killed us there. > > > > I still don't know why my > > allocate-swapspace-according-to-virtual-address change didn't > > help. Much. Marcelo played with that a bit too. > >

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Andi Kleen
> Yeah, the 2.5 switch to physical scanning killed us there. > > I still don't know why my allocate-swapspace-according-to-virtual-address > change didn't help. Much. Marcelo played with that a bit too. I've been thinking about just always doing swap on > page clusters. Any reason swapping

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Andrew Morton
On Thu, 31 Jan 2008 11:15:08 +0100 Andi Kleen <[EMAIL PROTECTED]> wrote: > Peter Zijlstra <[EMAIL PROTECTED]> writes: > > > > Ah, that is Lennarts Pulse Audio thing, he has samples in memory which > > might not have been used for a while, and he wants to be able to > > pre-fetch those when he

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Andrew Morton
On Thu, 31 Jan 2008 11:10:13 +0100 Peter Zijlstra <[EMAIL PROTECTED]> wrote: > > On Thu, 2008-01-31 at 02:05 -0800, Andrew Morton wrote: > > On Thu, 31 Jan 2008 10:53:26 +0100 Peter Zijlstra <[EMAIL PROTECTED]> wrote: > > > > > > > > On Thu, 2008-01-31 at 01:47 -0800, Andrew Morton wrote: > >

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Andi Kleen
Peter Zijlstra <[EMAIL PROTECTED]> writes: > > Ah, that is Lennarts Pulse Audio thing, he has samples in memory which > might not have been used for a while, and he wants to be able to > pre-fetch those when he suspects they might need to be played. So that > once the audio thread comes along and

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Peter Zijlstra
On Thu, 2008-01-31 at 02:05 -0800, Andrew Morton wrote: > On Thu, 31 Jan 2008 10:53:26 +0100 Peter Zijlstra <[EMAIL PROTECTED]> wrote: > > > > > On Thu, 2008-01-31 at 01:47 -0800, Andrew Morton wrote: > > > On Thu, 31 Jan 2008 10:35:18 +0100 Peter Zijlstra <[EMAIL PROTECTED]> > > > wrote: > >

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Andrew Morton
On Thu, 31 Jan 2008 10:53:26 +0100 Peter Zijlstra <[EMAIL PROTECTED]> wrote: > > On Thu, 2008-01-31 at 01:47 -0800, Andrew Morton wrote: > > On Thu, 31 Jan 2008 10:35:18 +0100 Peter Zijlstra <[EMAIL PROTECTED]> wrote: > > > > > > > > On Thu, 2008-01-31 at 01:12 -0800, Andrew Morton wrote: > >

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Peter Zijlstra
On Thu, 2008-01-31 at 01:47 -0800, Andrew Morton wrote: > On Thu, 31 Jan 2008 10:35:18 +0100 Peter Zijlstra <[EMAIL PROTECTED]> wrote: > > > > > On Thu, 2008-01-31 at 01:12 -0800, Andrew Morton wrote: > > > > > Implementation-wise: make_pages_present() _can_ be converted to do this. > > > But

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Andrew Morton
On Thu, 31 Jan 2008 10:35:18 +0100 Peter Zijlstra <[EMAIL PROTECTED]> wrote: > > On Thu, 2008-01-31 at 01:12 -0800, Andrew Morton wrote: > > > Implementation-wise: make_pages_present() _can_ be converted to do this. > > But it's a lot of patching, and the result will be a cleaner, faster and >

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Peter Zijlstra
On Thu, 2008-01-31 at 01:12 -0800, Andrew Morton wrote: > Implementation-wise: make_pages_present() _can_ be converted to do this. > But it's a lot of patching, and the result will be a cleaner, faster and > smaller core MM. Whereas your approach is easy, but adds more code and > leaves the

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Andrew Morton
On Thu, 31 Jan 2008 09:44:00 +0100 Peter Zijlstra <[EMAIL PROTECTED]> wrote: > > On Wed, 2008-01-30 at 14:40 -0800, Andrew Morton wrote: > > On Wed, 30 Jan 2008 18:28:59 +0100 > > Peter Zijlstra <[EMAIL PROTECTED]> wrote: > > > > > Implement MADV_WILLNEED for anonymous pages by walking the page

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Peter Zijlstra
On Wed, 2008-01-30 at 14:40 -0800, Andrew Morton wrote: > On Wed, 30 Jan 2008 18:28:59 +0100 > Peter Zijlstra <[EMAIL PROTECTED]> wrote: > > > Implement MADV_WILLNEED for anonymous pages by walking the page tables and > > starting asynchonous swap cache reads for all encountered swap pages. > >

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Peter Zijlstra
On Wed, 2008-01-30 at 14:40 -0800, Andrew Morton wrote: On Wed, 30 Jan 2008 18:28:59 +0100 Peter Zijlstra [EMAIL PROTECTED] wrote: Implement MADV_WILLNEED for anonymous pages by walking the page tables and starting asynchonous swap cache reads for all encountered swap pages. Why cannot

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Andrew Morton
On Thu, 31 Jan 2008 09:44:00 +0100 Peter Zijlstra [EMAIL PROTECTED] wrote: On Wed, 2008-01-30 at 14:40 -0800, Andrew Morton wrote: On Wed, 30 Jan 2008 18:28:59 +0100 Peter Zijlstra [EMAIL PROTECTED] wrote: Implement MADV_WILLNEED for anonymous pages by walking the page tables and

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Peter Zijlstra
On Thu, 2008-01-31 at 01:12 -0800, Andrew Morton wrote: Implementation-wise: make_pages_present() _can_ be converted to do this. But it's a lot of patching, and the result will be a cleaner, faster and smaller core MM. Whereas your approach is easy, but adds more code and leaves the old

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Andrew Morton
On Thu, 31 Jan 2008 10:35:18 +0100 Peter Zijlstra [EMAIL PROTECTED] wrote: On Thu, 2008-01-31 at 01:12 -0800, Andrew Morton wrote: Implementation-wise: make_pages_present() _can_ be converted to do this. But it's a lot of patching, and the result will be a cleaner, faster and smaller

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Peter Zijlstra
On Thu, 2008-01-31 at 01:47 -0800, Andrew Morton wrote: On Thu, 31 Jan 2008 10:35:18 +0100 Peter Zijlstra [EMAIL PROTECTED] wrote: On Thu, 2008-01-31 at 01:12 -0800, Andrew Morton wrote: Implementation-wise: make_pages_present() _can_ be converted to do this. But it's a lot of

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Andrew Morton
On Thu, 31 Jan 2008 10:53:26 +0100 Peter Zijlstra [EMAIL PROTECTED] wrote: On Thu, 2008-01-31 at 01:47 -0800, Andrew Morton wrote: On Thu, 31 Jan 2008 10:35:18 +0100 Peter Zijlstra [EMAIL PROTECTED] wrote: On Thu, 2008-01-31 at 01:12 -0800, Andrew Morton wrote:

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Peter Zijlstra
On Thu, 2008-01-31 at 02:05 -0800, Andrew Morton wrote: On Thu, 31 Jan 2008 10:53:26 +0100 Peter Zijlstra [EMAIL PROTECTED] wrote: On Thu, 2008-01-31 at 01:47 -0800, Andrew Morton wrote: On Thu, 31 Jan 2008 10:35:18 +0100 Peter Zijlstra [EMAIL PROTECTED] wrote: On Thu,

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Andi Kleen
Peter Zijlstra [EMAIL PROTECTED] writes: Ah, that is Lennarts Pulse Audio thing, he has samples in memory which might not have been used for a while, and he wants to be able to pre-fetch those when he suspects they might need to be played. So that once the audio thread comes along and stuffs

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Andrew Morton
On Thu, 31 Jan 2008 11:10:13 +0100 Peter Zijlstra [EMAIL PROTECTED] wrote: On Thu, 2008-01-31 at 02:05 -0800, Andrew Morton wrote: On Thu, 31 Jan 2008 10:53:26 +0100 Peter Zijlstra [EMAIL PROTECTED] wrote: On Thu, 2008-01-31 at 01:47 -0800, Andrew Morton wrote: On Thu, 31 Jan

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Andrew Morton
On Thu, 31 Jan 2008 11:15:08 +0100 Andi Kleen [EMAIL PROTECTED] wrote: Peter Zijlstra [EMAIL PROTECTED] writes: Ah, that is Lennarts Pulse Audio thing, he has samples in memory which might not have been used for a while, and he wants to be able to pre-fetch those when he suspects they

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Andi Kleen
Yeah, the 2.5 switch to physical scanning killed us there. I still don't know why my allocate-swapspace-according-to-virtual-address change didn't help. Much. Marcelo played with that a bit too. I've been thinking about just always doing swap on page clusters. Any reason swapping

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Rik van Riel
On Thu, 31 Jan 2008 12:06:10 +0100 Andi Kleen [EMAIL PROTECTED] wrote: Yeah, the 2.5 switch to physical scanning killed us there. I still don't know why my allocate-swapspace-according-to-virtual-address change didn't help. Much. Marcelo played with that a bit too. I've been

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Rik van Riel
On Thu, 31 Jan 2008 12:32:24 +0100 Andi Kleen [EMAIL PROTECTED] wrote: On Thu, Jan 31, 2008 at 05:52:09AM -0500, Rik van Riel wrote: Don't malloc() and free() hopelessly fragment memory over time, ensuring that little related data can be found inside each 1MB chunk if the process is large

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-31 Thread Andi Kleen
On Thu, Jan 31, 2008 at 05:52:09AM -0500, Rik van Riel wrote: On Thu, 31 Jan 2008 12:06:10 +0100 Andi Kleen [EMAIL PROTECTED] wrote: Yeah, the 2.5 switch to physical scanning killed us there. I still don't know why my allocate-swapspace-according-to-virtual-address change didn't

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-30 Thread Andrew Morton
On Wed, 30 Jan 2008 18:28:59 +0100 Peter Zijlstra <[EMAIL PROTECTED]> wrote: > Implement MADV_WILLNEED for anonymous pages by walking the page tables and > starting asynchonous swap cache reads for all encountered swap pages. Why cannot this use (a perhaps suitably-modified)

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-30 Thread Matt Mackall
On Wed, 2008-01-30 at 18:28 +0100, Peter Zijlstra wrote: > Subject: mm: MADV_WILLNEED implementation for anonymous memory > > Implement MADV_WILLNEED for anonymous pages by walking the page tables and > starting asynchonous swap cache reads for all encountered swap pages. > > Doing so required

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-30 Thread Matt Mackall
On Wed, 2008-01-30 at 18:28 +0100, Peter Zijlstra wrote: Subject: mm: MADV_WILLNEED implementation for anonymous memory Implement MADV_WILLNEED for anonymous pages by walking the page tables and starting asynchonous swap cache reads for all encountered swap pages. Doing so required a

Re: [PATCH] mm: MADV_WILLNEED implementation for anonymous memory

2008-01-30 Thread Andrew Morton
On Wed, 30 Jan 2008 18:28:59 +0100 Peter Zijlstra [EMAIL PROTECTED] wrote: Implement MADV_WILLNEED for anonymous pages by walking the page tables and starting asynchonous swap cache reads for all encountered swap pages. Why cannot this use (a perhaps suitably-modified) make_pages_present()? --