Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-09-05 Thread Minchan Kim
On Fri, Sep 03, 2010 at 07:29:43PM +0900, KAMEZAWA Hiroyuki wrote: On Thu, 2 Sep 2010 17:54:24 +0900 KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: Here is a rough code for this. here is a _tested_ one. If I tested correctly, I allocated 40MB of contigous pages by the new

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-09-05 Thread KAMEZAWA Hiroyuki
On Mon, 6 Sep 2010 00:57:53 +0900 Minchan Kim minchan@gmail.com wrote: Thanks, -Kame == From: KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com This patch as a memory allocator for contiguous memory larger than MAX_ORDER. alloc_contig_pages(hint, size, list);

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-09-03 Thread KAMEZAWA Hiroyuki
On Thu, 2 Sep 2010 17:54:24 +0900 KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: Here is a rough code for this. here is a _tested_ one. If I tested correctly, I allocated 40MB of contigous pages by the new funciton. I'm grad this can be some hints for people. Thanks, -Kame == From:

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-09-02 Thread KAMEZAWA Hiroyuki
On Fri, 27 Aug 2010 17:16:39 +0900 KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: On Thu, 26 Aug 2010 18:36:24 +0900 Minchan Kim minchan@gmail.com wrote: On Thu, Aug 26, 2010 at 1:30 PM, KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: On Thu, 26 Aug 2010 13:06:28

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-30 Thread Clemens Ladisch
Andrew Morton wrote: It would help (a lot) if we could get more attention and buyin and fedback from the potential clients of this code. rmk's feedback is valuable. Have we heard from the linux-media people? What other subsystems might use it? ieee1394 perhaps? All FireWire controllers

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-28 Thread Hans Verkuil
On Thursday, August 26, 2010 00:58:14 Andrew Morton wrote: On Fri, 20 Aug 2010 15:15:10 +0200 Peter Zijlstra pet...@infradead.org wrote: On Fri, 2010-08-20 at 11:50 +0200, Michal Nazarewicz wrote: Hello everyone, The following patchset implements a Contiguous Memory Allocator. For

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-28 Thread Peter Zijlstra
On Sat, 2010-08-28 at 15:08 +0200, Hans Verkuil wrote: That would be good. Although I expect that the allocation would need to be 100% rock-solid reliable, otherwise the end user has a non-functioning device. Yes, indeed. And you have to be careful as well how you move pages around.

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-28 Thread Hans Verkuil
On Saturday, August 28, 2010 15:34:46 Peter Zijlstra wrote: On Sat, 2010-08-28 at 15:08 +0200, Hans Verkuil wrote: That would be good. Although I expect that the allocation would need to be 100% rock-solid reliable, otherwise the end user has a non-functioning device. Yes,

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-28 Thread Peter Zijlstra
On Sat, 2010-08-28 at 15:58 +0200, Hans Verkuil wrote: Isn't the proposed CMA thing vulnerable to the exact same problem? If you allow sharing of regions and plug some allocator in there you get the same problem. If you can solve it there, you can solve it for any kind of reservation

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-27 Thread KAMEZAWA Hiroyuki
On Thu, 26 Aug 2010 18:36:24 +0900 Minchan Kim minchan@gmail.com wrote: On Thu, Aug 26, 2010 at 1:30 PM, KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: On Thu, 26 Aug 2010 13:06:28 +0900 Minchan Kim minchan@gmail.com wrote: On Thu, Aug 26, 2010 at 12:44 PM, KAMEZAWA

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-27 Thread Peter Zijlstra
On Fri, 2010-08-27 at 17:16 +0900, KAMEZAWA Hiroyuki wrote: How about changing following this? The thing is MAX_ORDER is static. But we want to avoid too big MAX_ORDER of whole zones to support devices which requires big allocation chunk. So let's add MAX_ORDER into each zone and then,

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-26 Thread Peter Zijlstra
On Thu, 2010-08-26 at 03:28 +0200, Michał Nazarewicz wrote: On Fri, 20 Aug 2010 15:15:10 +0200, Peter Zijlstra pet...@infradead.org wrote: So the idea is to grab a large chunk of memory at boot time and then later allow some device to use it? I'd much rather we'd improve the regular

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-26 Thread Peter Zijlstra
On Thu, 2010-08-26 at 04:40 +0200, Michał Nazarewicz wrote: I think that the biggest problem is fragmentation here. For instance, I think that a situation where there is enough free space but it's fragmented so no single contiguous chunk can be allocated is a serious problem. However, I

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-26 Thread Peter Zijlstra
On Thu, 2010-08-26 at 11:49 +0900, Minchan Kim wrote: But one of problems is anonymous page which can be has a role of pinned page in non-swapsystem. Well, compaction can move those around, but if you've got too many of them its a simple matter of over-commit and for that we've got the

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-26 Thread Minchan Kim
On Thu, Aug 26, 2010 at 5:20 PM, Peter Zijlstra pet...@infradead.org wrote: On Thu, 2010-08-26 at 11:49 +0900, Minchan Kim wrote: But one of problems is anonymous page which can be has a role of pinned page in non-swapsystem. Well, compaction can move those around, but if you've got too many

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-26 Thread Minchan Kim
On Thu, Aug 26, 2010 at 1:30 PM, KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: On Thu, 26 Aug 2010 13:06:28 +0900 Minchan Kim minchan@gmail.com wrote: On Thu, Aug 26, 2010 at 12:44 PM, KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: On Thu, 26 Aug 2010 11:50:17 +0900

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-26 Thread Peter Zijlstra
On Thu, 2010-08-26 at 18:29 +0900, Minchan Kim wrote: As I said following mail, I said about free space problem. Of course, compaction could move anon pages into somewhere. What's is somewhere? At last, it's same zone. It can prevent fragment problem but not size of free space. So I mean it

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-26 Thread Mel Gorman
On Fri, Aug 20, 2010 at 03:15:10PM +0200, Peter Zijlstra wrote: On Fri, 2010-08-20 at 11:50 +0200, Michal Nazarewicz wrote: Hello everyone, The following patchset implements a Contiguous Memory Allocator. For those who have not yet stumbled across CMA an excerpt from documentation:

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-26 Thread Mel Gorman
On Thu, Aug 26, 2010 at 04:40:46AM +0200, Micha?? Nazarewicz wrote: Hello Andrew, I think Pawel has replied to most of your comments, so I'll just add my own 0.02 KRW. ;) Peter Zijlstra pet...@infradead.org wrote: So the idea is to grab a large chunk of memory at boot time and then later

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-26 Thread Minchan Kim
On Thu, Aug 26, 2010 at 7:06 PM, Peter Zijlstra pet...@infradead.org wrote: On Thu, 2010-08-26 at 18:29 +0900, Minchan Kim wrote: As I said following mail, I said about free space problem. Of course, compaction could move anon pages into somewhere. What's is somewhere? At last, it's same zone.

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-26 Thread Peter Zijlstra
Even more offtopic ;-) On Thu, 2010-08-26 at 19:21 +0900, Minchan Kim wrote: I agree highmem isn't a gorgeous. But my desktop isn't real machine? Important thing is that we already have a highmem and many guys include you(kmap stacking patch :))try to improve highmem problems. :) I have

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-26 Thread Michał Nazarewicz
On Thu, 26 Aug 2010 10:17:07 +0200, Peter Zijlstra pet...@infradead.org wrote: So why not work on the page allocator to improve its contiguous allocation behaviour. If you look at the thing you'll find pageblocks and migration types. If you change it so that you pin the migration type of one or

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread Andrew Morton
On Fri, 20 Aug 2010 15:15:10 +0200 Peter Zijlstra pet...@infradead.org wrote: On Fri, 2010-08-20 at 11:50 +0200, Michal Nazarewicz wrote: Hello everyone, The following patchset implements a Contiguous Memory Allocator. For those who have not yet stumbled across CMA an excerpt from

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread Daniel Walker
On Wed, 2010-08-25 at 15:58 -0700, Andrew Morton wrote: That would be good. Although I expect that the allocation would need to be 100% rock-solid reliable, otherwise the end user has a non-functioning device. Could generic core VM provide the required level of service? Anyway, these

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread KAMEZAWA Hiroyuki
On Wed, 25 Aug 2010 15:58:14 -0700 Andrew Morton a...@linux-foundation.org wrote: On Fri, 20 Aug 2010 15:15:10 +0200 Peter Zijlstra pet...@infradead.org wrote: On Fri, 2010-08-20 at 11:50 +0200, Michal Nazarewicz wrote: Hello everyone, The following patchset implements a

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread Pawel Osciak
Hi Andrew, Thank you for your comments and interest in this! On 08/26/2010 07:58 AM, Andrew Morton wrote: On Fri, 20 Aug 2010 15:15:10 +0200 Peter Zijlstrapet...@infradead.org wrote: On Fri, 2010-08-20 at 11:50 +0200, Michal Nazarewicz wrote: Hello everyone, The following

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread Michał Nazarewicz
On Fri, 20 Aug 2010 15:15:10 +0200, Peter Zijlstra pet...@infradead.org wrote: So the idea is to grab a large chunk of memory at boot time and then later allow some device to use it? I'd much rather we'd improve the regular page allocator to be smarter about this. We recently added a lot of

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread Michał Nazarewicz
On Thu, 26 Aug 2010 01:26:34 +0200, Daniel Walker dwal...@codeaurora.org wrote: If Michal is active, and follows community comments (including Zach's, but I haven't seen any) then we can defer to that solution .. Comments are always welcome. :) -- Best regards,

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread Pawel Osciak
On 08/26/2010 08:31 AM, Jonathan Corbet wrote: On Wed, 25 Aug 2010 15:58:14 -0700 Andrew Mortona...@linux-foundation.org wrote: If you want guarantees you can free stuff, why not add constraints to the page allocation type and only allow MIGRATE_MOVABLE pages inside a certain region,

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread Michał Nazarewicz
On Thu, 26 Aug 2010 01:31:25 +0200, Jonathan Corbet cor...@lwn.net wrote: The original OLPC has a camera controller which requires three contiguous, image-sized buffers in memory. That system is a little memory constrained (OK, it's desperately short of memory), so, in the past, the chances of

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread Michał Nazarewicz
On Thu, 26 Aug 2010 02:58:57 +0200, KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: Hmm, you may not like this..but how about following kind of interface ? Now, memoyr hotplug supports following operation to free and _isolate_ memory region. # echo offline

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread Michał Nazarewicz
Hello Andrew, I think Pawel has replied to most of your comments, so I'll just add my own 0.02 KRW. ;) Peter Zijlstra pet...@infradead.org wrote: So the idea is to grab a large chunk of memory at boot time and then later allow some device to use it? I'd much rather we'd improve the regular

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread Minchan Kim
On Thu, Aug 26, 2010 at 8:31 AM, Jonathan Corbet cor...@lwn.net wrote: On Wed, 25 Aug 2010 15:58:14 -0700 Andrew Morton a...@linux-foundation.org wrote: If you want guarantees you can free stuff, why not add constraints to the page allocation type and only allow MIGRATE_MOVABLE pages inside

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread KAMEZAWA Hiroyuki
On Thu, 26 Aug 2010 04:12:10 +0200 Michał Nazarewicz m.nazarew...@samsung.com wrote: On Thu, 26 Aug 2010 02:58:57 +0200, KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: Hmm, you may not like this..but how about following kind of interface ? Now, memoyr hotplug supports following

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread Minchan Kim
On Thu, Aug 26, 2010 at 11:49 AM, Minchan Kim minchan@gmail.com wrote: On Thu, Aug 26, 2010 at 8:31 AM, Jonathan Corbet cor...@lwn.net wrote: On Wed, 25 Aug 2010 15:58:14 -0700 Andrew Morton a...@linux-foundation.org wrote: If you want guarantees you can free stuff, why not add

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread KAMEZAWA Hiroyuki
On Thu, 26 Aug 2010 11:50:17 +0900 KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: 128MB...too big ? But it's depend on config. IBM's ppc guys used 16MB section, and recently, a new interface to shrink the number of /sys files are added, maybe usable. Something good with this

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread Michał Nazarewicz
KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: 128MB...too big ? But it's depend on config. On embedded systems it may be like half of the RAM. Or a quarter. So bigger granularity could be desired on some platforms. IBM's ppc guys used 16MB section, and recently, a new interface

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread Minchan Kim
On Thu, Aug 26, 2010 at 12:44 PM, KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: On Thu, 26 Aug 2010 11:50:17 +0900 KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: 128MB...too big ? But it's depend on config. IBM's ppc guys used 16MB section, and recently, a new interface to

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread Minchan Kim
On Thu, Aug 26, 2010 at 1:06 PM, Minchan Kim minchan@gmail.com wrote: On Thu, Aug 26, 2010 at 12:44 PM, KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: On Thu, 26 Aug 2010 11:50:17 +0900 KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: 128MB...too big ? But it's depend on

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread KAMEZAWA Hiroyuki
On Thu, 26 Aug 2010 13:06:28 +0900 Minchan Kim minchan@gmail.com wrote: On Thu, Aug 26, 2010 at 12:44 PM, KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: On Thu, 26 Aug 2010 11:50:17 +0900 KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: 128MB...too big ? But it's

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread KAMEZAWA Hiroyuki
On Thu, 26 Aug 2010 06:01:56 +0200 Michał Nazarewicz m.nazarew...@samsung.com wrote: KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: 128MB...too big ? But it's depend on config. On embedded systems it may be like half of the RAM. Or a quarter. So bigger granularity could be

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-25 Thread Américo Wang
On Thu, Aug 26, 2010 at 06:01:56AM +0200, Michał Nazarewicz wrote: KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com wrote: 128MB...too big ? But it's depend on config. On embedded systems it may be like half of the RAM. Or a quarter. So bigger granularity could be desired on some platforms.

[PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-20 Thread Michal Nazarewicz
Hello everyone, The following patchset implements a Contiguous Memory Allocator. For those who have not yet stumbled across CMA an excerpt from documentation: The Contiguous Memory Allocator (CMA) is a framework, which allows setting up a machine-specific configuration for

Re: [PATCH/RFCv4 0/6] The Contiguous Memory Allocator framework

2010-08-20 Thread Peter Zijlstra
On Fri, 2010-08-20 at 11:50 +0200, Michal Nazarewicz wrote: Hello everyone, The following patchset implements a Contiguous Memory Allocator. For those who have not yet stumbled across CMA an excerpt from documentation: The Contiguous Memory Allocator (CMA) is a framework, which allows