Re: [rfc] balance-on-fork NUMA placement

2007-08-05 Thread Nick Piggin
On Fri, Aug 03, 2007 at 01:10:13PM -0700, Suresh B wrote: > On Fri, Aug 03, 2007 at 02:20:10AM +0200, Nick Piggin wrote: > > On Thu, Aug 02, 2007 at 11:33:39AM -0700, Martin Bligh wrote: > > > Nick Piggin wrote: > > > >On Wed, Aug 01, 2007 at 03:52:11PM -0700, Martin Bligh wrote: > > > >>>And so fo

Re: [rfc] balance-on-fork NUMA placement

2007-08-03 Thread Siddha, Suresh B
On Fri, Aug 03, 2007 at 02:20:10AM +0200, Nick Piggin wrote: > On Thu, Aug 02, 2007 at 11:33:39AM -0700, Martin Bligh wrote: > > Nick Piggin wrote: > > >On Wed, Aug 01, 2007 at 03:52:11PM -0700, Martin Bligh wrote: > > >>>And so forth. Initial forks will balance. If the children refuse to > > >>>

Re: [rfc] balance-on-fork NUMA placement

2007-08-02 Thread Christoph Lameter
On Fri, 3 Aug 2007, Nick Piggin wrote: > Well what's wrong with it? It seems to use memory policies for exactly > what they are intended (aside from it being kernel directed...). Sure I think you could do it with some effort. They were primarily designed for user space. Lots of little side issue

Re: [rfc] balance-on-fork NUMA placement

2007-08-02 Thread Nick Piggin
On Thu, Aug 02, 2007 at 06:34:04PM -0700, Christoph Lameter wrote: > On Fri, 3 Aug 2007, Nick Piggin wrote: > > > Yeah it only gets set if the parent is initially using a default policy > > at this stage (and then is restored afterwards of course). > > Uggh. Looks like more hackery ahead. I think

Re: [rfc] balance-on-fork NUMA placement

2007-08-02 Thread Christoph Lameter
On Fri, 3 Aug 2007, Nick Piggin wrote: > Yeah it only gets set if the parent is initially using a default policy > at this stage (and then is restored afterwards of course). Uggh. Looks like more hackery ahead. I think this cannot be done in the desired clean way until we have some revving of th

Re: [rfc] balance-on-fork NUMA placement

2007-08-02 Thread Nick Piggin
On Thu, Aug 02, 2007 at 06:02:56PM -0700, Christoph Lameter wrote: > On Fri, 3 Aug 2007, Nick Piggin wrote: > > > > Ok. So MPOL_BIND on a single node. We would have to save the current > > > memory policy on the stack and then restore it later. Then you would need > > > a special call anyways. >

Re: [rfc] balance-on-fork NUMA placement

2007-08-02 Thread Christoph Lameter
On Fri, 3 Aug 2007, Nick Piggin wrote: > > Ok. So MPOL_BIND on a single node. We would have to save the current > > memory policy on the stack and then restore it later. Then you would need > > a special call anyways. > > Well the memory policy will already be set to MPOL_BIND at this point. >

Re: [rfc] balance-on-fork NUMA placement

2007-08-02 Thread Nick Piggin
On Thu, Aug 02, 2007 at 05:52:28PM -0700, Christoph Lameter wrote: > On Fri, 3 Aug 2007, Nick Piggin wrote: > > > > Add a (slow) kmalloc_policy? Strict Object round robin for interleave > > > right? It probably needs its own RR counter otherwise it disturbs the per > > > task page RR. > > > > I

Re: [rfc] balance-on-fork NUMA placement

2007-08-02 Thread Christoph Lameter
On Fri, 3 Aug 2007, Nick Piggin wrote: > > Add a (slow) kmalloc_policy? Strict Object round robin for interleave > > right? It probably needs its own RR counter otherwise it disturbs the per > > task page RR. > > I guess interleave could be nice for other things, but for this, I > just want MPO

Re: [rfc] balance-on-fork NUMA placement

2007-08-02 Thread Nick Piggin
On Thu, Aug 02, 2007 at 12:58:13PM -0700, Christoph Lameter wrote: > On Thu, 2 Aug 2007, Nick Piggin wrote: > > > > It does in the sense that slabs are allocated following policies. If you > > > want to place individual objects then you need to use kmalloc_node(). > > > > Is there no way to plac

Re: [rfc] balance-on-fork NUMA placement

2007-08-02 Thread Nick Piggin
On Thu, Aug 02, 2007 at 11:33:39AM -0700, Martin Bligh wrote: > Nick Piggin wrote: > >On Wed, Aug 01, 2007 at 03:52:11PM -0700, Martin Bligh wrote: > >>>And so forth. Initial forks will balance. If the children refuse to > >>>die, forks will continue to balance. If the parent starts seeing short

Re: [rfc] balance-on-fork NUMA placement

2007-08-02 Thread Christoph Lameter
On Thu, 2 Aug 2007, Nick Piggin wrote: > > It does in the sense that slabs are allocated following policies. If you > > want to place individual objects then you need to use kmalloc_node(). > > Is there no way to place objects via policy? At least kernel stack and page > tables on x86-64 should

Re: [rfc] balance-on-fork NUMA placement

2007-08-02 Thread Martin Bligh
Nick Piggin wrote: On Wed, Aug 01, 2007 at 03:52:11PM -0700, Martin Bligh wrote: And so forth. Initial forks will balance. If the children refuse to die, forks will continue to balance. If the parent starts seeing short lived children, fork()s will eventually start to stay local. Fork with

Re: [rfc] balance-on-fork NUMA placement

2007-08-02 Thread Lee Schermerhorn
On Wed, 2007-08-01 at 15:52 -0700, Martin Bligh wrote: > >> This topic seems to come up periodically every since we first introduced > >> the NUMA scheduler, and every time we decide it's a bad idea. What's > >> changed? What workloads does this improve (aside from some artificial > >> benchmark li

Re: [rfc] balance-on-fork NUMA placement

2007-08-01 Thread Nick Piggin
On Tue, Jul 31, 2007 at 04:40:18PM -0700, Christoph Lameter wrote: > On Tue, 31 Jul 2007, Andi Kleen wrote: > > > On Tuesday 31 July 2007 07:41, Nick Piggin wrote: > > > > > I haven't given this idea testing yet, but I just wanted to get some > > > opinions on it first. NUMA placement still isn't

Re: [rfc] balance-on-fork NUMA placement

2007-08-01 Thread Nick Piggin
On Wed, Aug 01, 2007 at 03:52:11PM -0700, Martin Bligh wrote: > > >And so forth. Initial forks will balance. If the children refuse to > >die, forks will continue to balance. If the parent starts seeing short > >lived children, fork()s will eventually start to stay local. > > Fork without ex

Re: [rfc] balance-on-fork NUMA placement

2007-08-01 Thread Martin Bligh
This topic seems to come up periodically every since we first introduced the NUMA scheduler, and every time we decide it's a bad idea. What's changed? What workloads does this improve (aside from some artificial benchmark like stream)? To repeat the conclusions of last time ... the primary prob

Re: [rfc] balance-on-fork NUMA placement

2007-08-01 Thread Lee Schermerhorn
On Wed, 2007-08-01 at 10:53 -0700, Martin Bligh wrote: > Nick Piggin wrote: > > On Tue, Jul 31, 2007 at 11:14:08AM +0200, Andi Kleen wrote: > >> On Tuesday 31 July 2007 07:41, Nick Piggin wrote: > >> > >>> I haven't given this idea testing yet, but I just wanted to get some > >>> opinions on it fir

Re: [rfc] balance-on-fork NUMA placement

2007-08-01 Thread Martin Bligh
Nick Piggin wrote: On Tue, Jul 31, 2007 at 11:14:08AM +0200, Andi Kleen wrote: On Tuesday 31 July 2007 07:41, Nick Piggin wrote: I haven't given this idea testing yet, but I just wanted to get some opinions on it first. NUMA placement still isn't ideal (eg. tasks with a memory policy will not

Re: [rfc] balance-on-fork NUMA placement

2007-08-01 Thread Andi Kleen
On Wednesday 01 August 2007 01:40:18 Christoph Lameter wrote: > It does in the sense that slabs are allocated following policies. If you > want to place individual objects then you need to use kmalloc_node(). Nick wants to place individual objects here -Andi - To unsubscribe from this list:

Re: [rfc] balance-on-fork NUMA placement

2007-07-31 Thread Ingo Molnar
* Nick Piggin <[EMAIL PROTECTED]> wrote: > > _after_ the dup_task_struct(). Then change sched_fork() to return a > > CPU number - hence we dont have a separate sched_fork_suggest_cpu() > > initialization function, only one, obvious sched_fork() function. > > Agreed? > > That puts task struct,

Re: [rfc] balance-on-fork NUMA placement

2007-07-31 Thread Nick Piggin
On Tue, Jul 31, 2007 at 11:14:08AM +0200, Andi Kleen wrote: > On Tuesday 31 July 2007 07:41, Nick Piggin wrote: > > > I haven't given this idea testing yet, but I just wanted to get some > > opinions on it first. NUMA placement still isn't ideal (eg. tasks with > > a memory policy will not do any

Re: [rfc] balance-on-fork NUMA placement

2007-07-31 Thread Nick Piggin
On Tue, Jul 31, 2007 at 10:01:14AM +0200, Ingo Molnar wrote: > > * Nick Piggin <[EMAIL PROTECTED]> wrote: > > > This patch uses memory policies to attempt to improve this. It > > requires that we ask the scheduler to suggest the child's new CPU > > earlier in the fork, but that is not a fundame

Re: [rfc] balance-on-fork NUMA placement

2007-07-31 Thread Christoph Lameter
On Tue, 31 Jul 2007, Andi Kleen wrote: > On Tuesday 31 July 2007 07:41, Nick Piggin wrote: > > > I haven't given this idea testing yet, but I just wanted to get some > > opinions on it first. NUMA placement still isn't ideal (eg. tasks with > > a memory policy will not do any placement, and proce

Re: [rfc] balance-on-fork NUMA placement

2007-07-31 Thread Andi Kleen
On Tuesday 31 July 2007 07:41, Nick Piggin wrote: > I haven't given this idea testing yet, but I just wanted to get some > opinions on it first. NUMA placement still isn't ideal (eg. tasks with > a memory policy will not do any placement, and process migrations of > course will leave the memory be

Re: [rfc] balance-on-fork NUMA placement

2007-07-31 Thread Ingo Molnar
* Nick Piggin <[EMAIL PROTECTED]> wrote: > This patch uses memory policies to attempt to improve this. It > requires that we ask the scheduler to suggest the child's new CPU > earlier in the fork, but that is not a fundamental difference. no fundamental objections, but i think we could simply