Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets (v0.2)

2005-04-23 Thread Paul Jackson
> Is the above code equivalant to what the comment states: > > if (is_cpu_isolated(trial) <= is_cpu_exclusive(trial)) > return -EINVAL; I think I got that backwards. How about: /* An isolated cpuset has to be exclusive */ if (!(is_cpu_isolated(trial) <=

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets (v0.2)

2005-04-23 Thread Paul Jackson
Dinakar's patch contains: + /* Make the change */ + par->cpus_allowed = t.cpus_allowed; + par->isolated_map = t.isolated_map; Doesn't the above make changes to the parent cpus_allowed without calling validate_change()? Couldn't we do nasty things like empty that cpus_allowed,

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets (v0.2)

2005-04-23 Thread Paul Jackson
Dinakar's patch contains: + /* Make the change */ + par-cpus_allowed = t.cpus_allowed; + par-isolated_map = t.isolated_map; Doesn't the above make changes to the parent cpus_allowed without calling validate_change()? Couldn't we do nasty things like empty that cpus_allowed,

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets (v0.2)

2005-04-23 Thread Paul Jackson
Is the above code equivalant to what the comment states: if (is_cpu_isolated(trial) = is_cpu_exclusive(trial)) return -EINVAL; I think I got that backwards. How about: /* An isolated cpuset has to be exclusive */ if (!(is_cpu_isolated(trial) =

Re: [Lse-tech] Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-22 Thread Paul Jackson
Dinakar wrote: > Ok, Let me begin at the beginning and attempt to define what I am > doing here The statement of requirements and approach help. Thank-you. And the comments in the code patch are much easier for me to understand. Thanks. Let me step back and consider where we are here. I've

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets (v0.2)

2005-04-22 Thread Paul Jackson
A few code details (still working on more substantive reply): + /* An isolated cpuset has to be exclusive */ + if ((is_cpu_isolated(trial) && !is_cpu_exclusive(cur)) + || (!is_cpu_exclusive(trial) && is_cpu_isolated(cur))) + return -EINVAL; Is the above code

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets (v0.2)

2005-04-22 Thread Paul Jackson
A few code details (still working on more substantive reply): + /* An isolated cpuset has to be exclusive */ + if ((is_cpu_isolated(trial) !is_cpu_exclusive(cur)) + || (!is_cpu_exclusive(trial) is_cpu_isolated(cur))) + return -EINVAL; Is the above code

Re: [Lse-tech] Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-22 Thread Paul Jackson
Dinakar wrote: Ok, Let me begin at the beginning and attempt to define what I am doing here The statement of requirements and approach help. Thank-you. And the comments in the code patch are much easier for me to understand. Thanks. Let me step back and consider where we are here. I've

[RFC PATCH] Dynamic sched domains aka Isolated cpusets (v0.2)

2005-04-21 Thread Dinakar Guniguntala
Based on the Paul's feedback, I have simplified and cleaned up the code quite a bit. o I have taken care of most of the nits, except for the output format change for cpusets with isolated children. o Also most of my documentation has been part of my earlier mails and I have not yet

Re: [Lse-tech] Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-21 Thread Dinakar Guniguntala
On Wed, Apr 20, 2005 at 12:09:46PM -0700, Paul Jackson wrote: > Earlier, I wrote to Dinakar: > > What are your invariants, and how can you assure yourself and us > > that your code preserves these invariants? Ok, Let me begin at the beginning and attempt to define what I am doing here 1. I need

Re: [Lse-tech] Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-21 Thread Dinakar Guniguntala
On Wed, Apr 20, 2005 at 12:09:46PM -0700, Paul Jackson wrote: Earlier, I wrote to Dinakar: What are your invariants, and how can you assure yourself and us that your code preserves these invariants? Ok, Let me begin at the beginning and attempt to define what I am doing here 1. I need a

[RFC PATCH] Dynamic sched domains aka Isolated cpusets (v0.2)

2005-04-21 Thread Dinakar Guniguntala
Based on the Paul's feedback, I have simplified and cleaned up the code quite a bit. o I have taken care of most of the nits, except for the output format change for cpusets with isolated children. o Also most of my documentation has been part of my earlier mails and I have not yet

Re: [Lse-tech] Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-20 Thread Paul Jackson
Earlier, I wrote to Dinakar: > What are your invariants, and how can you assure yourself and us > that your code preserves these invariants? I repeat that question. === On my first reading of your example, I see the following. It is sinking into my dense skull more than it had before that your

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-20 Thread Dinakar Guniguntala
On Tue, Apr 19, 2005 at 08:26:39AM -0700, Paul Jackson wrote: > * Your understanding of "cpu_exclusive" is not the same as mine. Sorry for creating confusion by what I said earlier, I do understand exactly what cpu_exclusive means. Its just that when I started working on this (a long time ago) I

Re: [Lse-tech] Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-20 Thread Dinakar Guniguntala
On Tue, Apr 19, 2005 at 10:23:48AM -0700, Paul Jackson wrote: > > How does this play out in your interface? Are you convinced that > your invariants are preserved at all times, to all users? Can > you present a convincing argument to others that this is so? Let me give an example of how the

Re: [Lse-tech] Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-20 Thread Dinakar Guniguntala
On Tue, Apr 19, 2005 at 10:23:48AM -0700, Paul Jackson wrote: How does this play out in your interface? Are you convinced that your invariants are preserved at all times, to all users? Can you present a convincing argument to others that this is so? Let me give an example of how the

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-20 Thread Dinakar Guniguntala
On Tue, Apr 19, 2005 at 08:26:39AM -0700, Paul Jackson wrote: * Your understanding of cpu_exclusive is not the same as mine. Sorry for creating confusion by what I said earlier, I do understand exactly what cpu_exclusive means. Its just that when I started working on this (a long time ago) I

Re: [Lse-tech] Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-20 Thread Paul Jackson
Earlier, I wrote to Dinakar: What are your invariants, and how can you assure yourself and us that your code preserves these invariants? I repeat that question. === On my first reading of your example, I see the following. It is sinking into my dense skull more than it had before that your

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
Dinakar wrote: > Also I think we can add further restrictions in terms not being able > to change (add/remove) cpus within a isolated cpuset. Instead one would > have to tear down an existing cpuset and make a new one with the > required configuration. that would simplify things even further My

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
Nick wrote: > Well the scheduler simply can't handle it, so it is not so much a > matter of pushing - you simply can't use partitioned domains and > meaningfully have a cpuset above them. Translating that into cpuset-speak, I think what you mean is that I can't have partitioned sched domains and

Re: [Lse-tech] Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
Dinakar wrote: > I was hoping that by the time we are done with this, we would > be able to completely get rid of the isolcpus= option. I won't miss it. Though, since it's in the main line kernel, do you need to mark it deprecated for a while first? > For that > ofcourse we need to be able

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
Simon wrote: > I guess we hit a limit of the filesystem-interface approach here. > Are the possible failure reasons really that complex ? Given the amount of head scratching my proposal has provoked so far, they might be that complex, yes. Failure reasons include: * The cpuset Foo whose

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
Dinakar, replying to Nick: > > It doesn't work if you have *most* jobs bound to either > > {0, 1, 2, 3} or {4, 5, 6, 7} but one which should be allowed > > to use any CPU from 0-7. > > That is the current definition of cpu_exclusive on cpusets. > I initially thought of attaching exclusive cpusets

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Dinakar Guniguntala
On Tue, Apr 19, 2005 at 04:19:35PM +1000, Nick Piggin wrote: [...Snip...] > Though I imagine this becomes a complete superset of the > isolcpus= functionality, and it would actually be easier to > manage a single isolated CPU and its associated processes with > the cpusets interfaces after this.

Re: [Lse-tech] Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Dinakar Guniguntala
On Mon, Apr 18, 2005 at 10:54:27PM -0700, Paul Jackson wrote: > Hmmm ... interesting patch. My reaction to the changes in > kernel/cpuset.c are complicated: Thanks Paul for taking time off your vaction to reply to this. I was expecting to see one of your huge mails but this has exceeded all my

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Simon Derr
On Mon, 18 Apr 2005, Paul Jackson wrote: > Hmmm ... interesting patch. My reaction to the changes in > kernel/cpuset.c are complicated: > > * I'm supposed to be on vacation the rest of this month, >so trying (entirely unsuccessfully so far) not to think >about this. > * This is

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Nick Piggin
On Tue, 2005-04-19 at 00:19 -0700, Paul Jackson wrote: > Nick wrote: > > It doesn't work if you have *most* jobs bound to either > > {0, 1, 2, 3} or {4, 5, 6, 7} but one which should be allowed > > to use any CPU from 0-7. > > How bad does it not work? > > My understanding is that Dinakar's

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Dinakar Guniguntala
On Tue, Apr 19, 2005 at 09:44:06AM +1000, Nick Piggin wrote: > Very good, I was wondering when someone would try to implement this ;) Thank you for the feedback ! > >-static void __devinit arch_init_sched_domains(void) > >+static void attach_domains(cpumask_t cpu_map) > > { > > This shouldn't

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
Nick wrote: > That would make sense. I'm not familiar with the workings of cpusets, > but that would require every task to be assigned to one of these > sets (or a subset within it), yes? That's the rub, as I noted a couple of messages ago, while you were writing this message. It doesn't require

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
> > So you do _not_ want to consider nested sched domains, just disjoint > > ones. Good. > > > > You don't either? Good. :) >From the point of view of cpusets, I'd rather not think about nested sched domains, for now at least. But my scheduler savvy colleagues on the big SGI boxes may well

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
Nick wrote: > It doesn't work if you have *most* jobs bound to either > {0, 1, 2, 3} or {4, 5, 6, 7} but one which should be allowed > to use any CPU from 0-7. How bad does it not work? My understanding is that Dinakar's patch did _not_ drive tasks out of larger cpusets that included two or more

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Nick Piggin
On Mon, 2005-04-18 at 23:59 -0700, Paul Jackson wrote: > Nick wrote: > > Basically you just have to know that it has the > > capability to partition the system in an arbitrary disjoint set > > of sets of cpus. > > > > If you can make use of that, then we're in business ;) > > You read fast ;) >

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
Nick wrote: > Basically you just have to know that it has the > capability to partition the system in an arbitrary disjoint set > of sets of cpus. > > If you can make use of that, then we're in business ;) You read fast ;) So you do _not_ want to consider nested sched domains, just disjoint

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Nick Piggin
On Mon, 2005-04-18 at 22:54 -0700, Paul Jackson wrote: > Now, onto the real stuff. > > This same issue, in a strange way, comes up on the memory side, > as well as on the cpu side. > > First, let me verify one thing. I understand that the _key_ > purpose of your patch is not so much to isolate

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Nick Piggin
On Mon, 2005-04-18 at 22:54 -0700, Paul Jackson wrote: Now, onto the real stuff. This same issue, in a strange way, comes up on the memory side, as well as on the cpu side. First, let me verify one thing. I understand that the _key_ purpose of your patch is not so much to isolate cpus,

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
Nick wrote: Basically you just have to know that it has the capability to partition the system in an arbitrary disjoint set of sets of cpus. If you can make use of that, then we're in business ;) You read fast ;) So you do _not_ want to consider nested sched domains, just disjoint ones.

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Nick Piggin
On Mon, 2005-04-18 at 23:59 -0700, Paul Jackson wrote: Nick wrote: Basically you just have to know that it has the capability to partition the system in an arbitrary disjoint set of sets of cpus. If you can make use of that, then we're in business ;) You read fast ;) So you do

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
Nick wrote: It doesn't work if you have *most* jobs bound to either {0, 1, 2, 3} or {4, 5, 6, 7} but one which should be allowed to use any CPU from 0-7. How bad does it not work? My understanding is that Dinakar's patch did _not_ drive tasks out of larger cpusets that included two or more of

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
So you do _not_ want to consider nested sched domains, just disjoint ones. Good. You don't either? Good. :) From the point of view of cpusets, I'd rather not think about nested sched domains, for now at least. But my scheduler savvy colleagues on the big SGI boxes may well have

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
Nick wrote: That would make sense. I'm not familiar with the workings of cpusets, but that would require every task to be assigned to one of these sets (or a subset within it), yes? That's the rub, as I noted a couple of messages ago, while you were writing this message. It doesn't require

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Dinakar Guniguntala
On Tue, Apr 19, 2005 at 09:44:06AM +1000, Nick Piggin wrote: Very good, I was wondering when someone would try to implement this ;) Thank you for the feedback ! -static void __devinit arch_init_sched_domains(void) +static void attach_domains(cpumask_t cpu_map) { This shouldn't be needed.

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Nick Piggin
On Tue, 2005-04-19 at 00:19 -0700, Paul Jackson wrote: Nick wrote: It doesn't work if you have *most* jobs bound to either {0, 1, 2, 3} or {4, 5, 6, 7} but one which should be allowed to use any CPU from 0-7. How bad does it not work? My understanding is that Dinakar's patch did _not_

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Simon Derr
On Mon, 18 Apr 2005, Paul Jackson wrote: Hmmm ... interesting patch. My reaction to the changes in kernel/cpuset.c are complicated: * I'm supposed to be on vacation the rest of this month, so trying (entirely unsuccessfully so far) not to think about this. * This is perhaps the

Re: [Lse-tech] Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Dinakar Guniguntala
On Mon, Apr 18, 2005 at 10:54:27PM -0700, Paul Jackson wrote: Hmmm ... interesting patch. My reaction to the changes in kernel/cpuset.c are complicated: Thanks Paul for taking time off your vaction to reply to this. I was expecting to see one of your huge mails but this has exceeded all my

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Dinakar Guniguntala
On Tue, Apr 19, 2005 at 04:19:35PM +1000, Nick Piggin wrote: [...Snip...] Though I imagine this becomes a complete superset of the isolcpus= functionality, and it would actually be easier to manage a single isolated CPU and its associated processes with the cpusets interfaces after this.

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
Dinakar, replying to Nick: It doesn't work if you have *most* jobs bound to either {0, 1, 2, 3} or {4, 5, 6, 7} but one which should be allowed to use any CPU from 0-7. That is the current definition of cpu_exclusive on cpusets. I initially thought of attaching exclusive cpusets to sched

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
Simon wrote: I guess we hit a limit of the filesystem-interface approach here. Are the possible failure reasons really that complex ? Given the amount of head scratching my proposal has provoked so far, they might be that complex, yes. Failure reasons include: * The cpuset Foo whose

Re: [Lse-tech] Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
Dinakar wrote: I was hoping that by the time we are done with this, we would be able to completely get rid of the isolcpus= option. I won't miss it. Though, since it's in the main line kernel, do you need to mark it deprecated for a while first? For that ofcourse we need to be able build

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
Nick wrote: Well the scheduler simply can't handle it, so it is not so much a matter of pushing - you simply can't use partitioned domains and meaningfully have a cpuset above them. Translating that into cpuset-speak, I think what you mean is that I can't have partitioned sched domains and

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-19 Thread Paul Jackson
Dinakar wrote: Also I think we can add further restrictions in terms not being able to change (add/remove) cpus within a isolated cpuset. Instead one would have to tear down an existing cpuset and make a new one with the required configuration. that would simplify things even further My

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-18 Thread Paul Jackson
Hmmm ... interesting patch. My reaction to the changes in kernel/cpuset.c are complicated: * I'm supposed to be on vacation the rest of this month, so trying (entirely unsuccessfully so far) not to think about this. * This is perhaps the first non-trivial cpuset patch to come in the

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-18 Thread Nick Piggin
Dinakar Guniguntala wrote: Here's an attempt at dynamic sched domains aka isolated cpusets Very good, I was wondering when someone would try to implement this ;) It needs some work. A few initial comments on the kernel/sched.c change - sorry, don't have too much time right now... ---

[RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-18 Thread Dinakar Guniguntala
Here's an attempt at dynamic sched domains aka isolated cpusets o This functionality is on top of CPUSETs and provides a way to completely isolate any set of CPUs dynamically. o There is a new cpu_isolated flag that allows users to convert an exclusive cpuset to an isolated one o The

[RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-18 Thread Dinakar Guniguntala
Here's an attempt at dynamic sched domains aka isolated cpusets o This functionality is on top of CPUSETs and provides a way to completely isolate any set of CPUs dynamically. o There is a new cpu_isolated flag that allows users to convert an exclusive cpuset to an isolated one o The

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-18 Thread Nick Piggin
Dinakar Guniguntala wrote: Here's an attempt at dynamic sched domains aka isolated cpusets Very good, I was wondering when someone would try to implement this ;) It needs some work. A few initial comments on the kernel/sched.c change - sorry, don't have too much time right now... ---

Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

2005-04-18 Thread Paul Jackson
Hmmm ... interesting patch. My reaction to the changes in kernel/cpuset.c are complicated: * I'm supposed to be on vacation the rest of this month, so trying (entirely unsuccessfully so far) not to think about this. * This is perhaps the first non-trivial cpuset patch to come in the