Re: [ANNOUNCE] 3.7-nohz1
On Sat, Jan 05, 2013 at 12:42:53AM +0100, Frederic Weisbecker wrote: > 2012/12/30 Paul E. McKenney : > > On Mon, Dec 24, 2012 at 12:43:25AM +0100, Frederic Weisbecker wrote: > >> 2012/12/21 Steven Rostedt : > >> > On Thu, 2012-12-20 at 19:32 +0100, Frederic Weisbecker wrote: > >> >> Let's imagine you have 4 CPUs. We keep the CPU 0 to offline RCU > >> >> callbacks there and to > >> >> handle the timekeeping. We set the rest as full dynticks. So you need > >> >> the following kernel > >> >> parameters: > >> >> > >> >> rcu_nocbs=1-3 full_nohz=1-3 > >> >> > >> >> (Note rcu_nocbs value must always be the same as full_nohz). > >> > > >> > Why? You can't have: rcu_nocbs=1-4 full_nohz=1-3 > >> > >> That should be allowed. > >> > >> > or: rcu_nocbs=1-3 full_nohz=1-4 ? > >> > >> But that not. > >> > >> You need to have: rcu_nocbs & full_nohz == full_nohz. This is because > >> the tick is not there to maintain the local RCU callbacks anymore. So > >> this must be offloaded to the rcu_nocb threads. > >> > >> I just have a doubt with rcu_nocb. Do we still need the tick to > >> complete the grace period for local rcu callbacks? I need to discuss > >> that with Paul. > > > > The tick is only needed if rcu_needs_cpu() returns false. Of course, > > this means that if you don't invoke rcu_needs_cpu() before returning to > > adaptive-idle usermode execution, you are correct that a full_nohz CPU > > would also have to be a rcu_nocbs CPU. > > > > That said, I am getting close to having an rcu_needs_cpu() that only > > returns false if there are callbacks immediately ready to invoke, at > > least if RCU_FAST_NO_HZ=y. > > Ok. Also when a CPU enqueues a callback and starts a grace period, the > tick polls on the grace period completion. If RCU_FAST_NO_HZ=n, then yes, this is the case, but only for !rcu_nocbs CPUs. >How is it handled with > rcu_nocbs CPUs? Does rcu_needs_cpu() return false until the grace > period is completed? If so I still need to restart the local tick > whenever a new callback is enqueued. Each rcu_nocbs CPU has a kthread, and that kthread is responsible for making sure that any needed grace periods move forward. In mainline, this is done via CPU 0, which is required to be a !rcu_nocbs CPU. In -rcu, the no-CBs kthreads communicate with the grace-period kthread via the rcu_node tree, so that if all CPUs are rcu_nocbs CPUs, rcu_needs_cpu() will always return false, even if RCU_FAST_NO_HZ=n. Thanx, Paul -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [ANNOUNCE] 3.7-nohz1
2012/12/30 Paul E. McKenney : > On Mon, Dec 24, 2012 at 12:43:25AM +0100, Frederic Weisbecker wrote: >> 2012/12/21 Steven Rostedt : >> > On Thu, 2012-12-20 at 19:32 +0100, Frederic Weisbecker wrote: >> >> Let's imagine you have 4 CPUs. We keep the CPU 0 to offline RCU callbacks >> >> there and to >> >> handle the timekeeping. We set the rest as full dynticks. So you need the >> >> following kernel >> >> parameters: >> >> >> >> rcu_nocbs=1-3 full_nohz=1-3 >> >> >> >> (Note rcu_nocbs value must always be the same as full_nohz). >> > >> > Why? You can't have: rcu_nocbs=1-4 full_nohz=1-3 >> >> That should be allowed. >> >> > or: rcu_nocbs=1-3 full_nohz=1-4 ? >> >> But that not. >> >> You need to have: rcu_nocbs & full_nohz == full_nohz. This is because >> the tick is not there to maintain the local RCU callbacks anymore. So >> this must be offloaded to the rcu_nocb threads. >> >> I just have a doubt with rcu_nocb. Do we still need the tick to >> complete the grace period for local rcu callbacks? I need to discuss >> that with Paul. > > The tick is only needed if rcu_needs_cpu() returns false. Of course, > this means that if you don't invoke rcu_needs_cpu() before returning to > adaptive-idle usermode execution, you are correct that a full_nohz CPU > would also have to be a rcu_nocbs CPU. > > That said, I am getting close to having an rcu_needs_cpu() that only > returns false if there are callbacks immediately ready to invoke, at > least if RCU_FAST_NO_HZ=y. Ok. Also when a CPU enqueues a callback and starts a grace period, the tick polls on the grace period completion. How is it handled with rcu_nocbs CPUs? Does rcu_needs_cpu() return false until the grace period is completed? If so I still need to restart the local tick whenever a new callback is enqueued. Thanks. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [ANNOUNCE] 3.7-nohz1
On Mon, Dec 24, 2012 at 12:43:25AM +0100, Frederic Weisbecker wrote: > 2012/12/21 Steven Rostedt : > > On Thu, 2012-12-20 at 19:32 +0100, Frederic Weisbecker wrote: > >> Let's imagine you have 4 CPUs. We keep the CPU 0 to offline RCU callbacks > >> there and to > >> handle the timekeeping. We set the rest as full dynticks. So you need the > >> following kernel > >> parameters: > >> > >> rcu_nocbs=1-3 full_nohz=1-3 > >> > >> (Note rcu_nocbs value must always be the same as full_nohz). > > > > Why? You can't have: rcu_nocbs=1-4 full_nohz=1-3 > > That should be allowed. > > > or: rcu_nocbs=1-3 full_nohz=1-4 ? > > But that not. > > You need to have: rcu_nocbs & full_nohz == full_nohz. This is because > the tick is not there to maintain the local RCU callbacks anymore. So > this must be offloaded to the rcu_nocb threads. > > I just have a doubt with rcu_nocb. Do we still need the tick to > complete the grace period for local rcu callbacks? I need to discuss > that with Paul. The tick is only needed if rcu_needs_cpu() returns false. Of course, this means that if you don't invoke rcu_needs_cpu() before returning to adaptive-idle usermode execution, you are correct that a full_nohz CPU would also have to be a rcu_nocbs CPU. That said, I am getting close to having an rcu_needs_cpu() that only returns false if there are callbacks immediately ready to invoke, at least if RCU_FAST_NO_HZ=y. Thanx, Paul > > That needs to be fixed. Either with a warning, and/or to force the two > > to be the same. That is, if they specify: > > > > rcu_nocbs=1-3 full_nohz=1-4 > > > > Then set rcu_nocbs=1-4 with a warning about it. Or simply set > > full_nohz=1-3. > > Yep, will do. > > Thanks! > > > > > -- Steve > > > >> > >> Now if you want proper isolation you need to: > >> > >> * Migrate your processes adequately > >> * Migrate your irqs to CPU 0 > >> * Migrate the RCU nocb threads to CPU 0. Example with the above > >> configuration: > >> > >> for p in $(ps -o pid= -C rcuo1,rcuo2,rcuo3) > >> do > >> taskset -cp 0 $p > >> done > >> > >> Then run what you want on the full dynticks CPUs. For best results, run 1 > >> task > >> per CPU, mostly in userspace and mostly CPU bound (otherwise more IO = > >> more kernel > >> mode execution = more chances to get IPIs, tick restarted, workqueues, > >> kthreads, etc...) > >> > >> This page contains a good reminder for those interested in CPU isolation: > >> https://github.com/gby/linux/wiki > >> > >> But keep in mind that my tree is not yet ready for serious production. > >> > > > > > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [ANNOUNCE] 3.7-nohz1
2012/12/21 Steven Rostedt : > On Thu, 2012-12-20 at 19:32 +0100, Frederic Weisbecker wrote: >> Let's imagine you have 4 CPUs. We keep the CPU 0 to offline RCU callbacks >> there and to >> handle the timekeeping. We set the rest as full dynticks. So you need the >> following kernel >> parameters: >> >> rcu_nocbs=1-3 full_nohz=1-3 >> >> (Note rcu_nocbs value must always be the same as full_nohz). > > Why? You can't have: rcu_nocbs=1-4 full_nohz=1-3 That should be allowed. > or: rcu_nocbs=1-3 full_nohz=1-4 ? But that not. You need to have: rcu_nocbs & full_nohz == full_nohz. This is because the tick is not there to maintain the local RCU callbacks anymore. So this must be offloaded to the rcu_nocb threads. I just have a doubt with rcu_nocb. Do we still need the tick to complete the grace period for local rcu callbacks? I need to discuss that with Paul. > > That needs to be fixed. Either with a warning, and/or to force the two > to be the same. That is, if they specify: > > rcu_nocbs=1-3 full_nohz=1-4 > > Then set rcu_nocbs=1-4 with a warning about it. Or simply set > full_nohz=1-3. Yep, will do. Thanks! > > -- Steve > >> >> Now if you want proper isolation you need to: >> >> * Migrate your processes adequately >> * Migrate your irqs to CPU 0 >> * Migrate the RCU nocb threads to CPU 0. Example with the above >> configuration: >> >> for p in $(ps -o pid= -C rcuo1,rcuo2,rcuo3) >> do >> taskset -cp 0 $p >> done >> >> Then run what you want on the full dynticks CPUs. For best results, run 1 >> task >> per CPU, mostly in userspace and mostly CPU bound (otherwise more IO = more >> kernel >> mode execution = more chances to get IPIs, tick restarted, workqueues, >> kthreads, etc...) >> >> This page contains a good reminder for those interested in CPU isolation: >> https://github.com/gby/linux/wiki >> >> But keep in mind that my tree is not yet ready for serious production. >> > > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [ANNOUNCE] 3.7-nohz1
Hi, On Thu, Dec 20, 2012 at 11:32 AM, Frederic Weisbecker wrote: > Hi, > > So this is a new version of the nohz cpusets based on 3.7, except it's not > using > cpusets anymore and I actually based it on the middle of the 3.8 merge window > in order to get latest upstream full dynticks preparatory work: cputime > cleanups, > RCU user mode, context tracking subsystem, nohz code consolidation, ... > > So the big changes since the last nohz cpuset release are: > > * printk now uses irq work so it doesn't rely on the tick anymore (provided > your arch implements irq work with IPIs or alike). This chunk has been > proposed > for the 3.8 merge window: https://lkml.org/lkml/2012/12/17/177 > May be Linus will pull, may be not. We'll see. In any case I've included it > in this tree > but I'm not reposting this part of the patchset to avoid spamming you. > > * cputime doesn't rely on IPIs anymore. Now the reader does a special > computation to > remotely get the tickless cputime. > > * No more cpusets interface. Paul McKenney suggested me to start with a boot > time > kernel parameter to define the full dynticks cpumask. And he was totally > right, it > makes the code much more simple. That's a good way to start and to make the > mainlining > easier. We can still add a runtime configuration later if necessary. It would be nice to have the runtime configuration ability. A percpu control file such as /sys/devices/system/cpu/cpuX/isol could configure that cpu with different levels of isolation. Users could echo bitmasks where each bit is associated with a level of isolation. echo 0 disables all isolation. Bit 1 disables RCU callbacks on that CPU, bit 2 isolates the CPU from the general scheduler just like isolcpus boot argument does, bit 3 pushes all irqs away, bit 4 turns off the ticks etc. I always hoped that someone will make isolcpus a runtime option so I guess it is time to get my hands dirty. Any pointers for this? > > * Now there is always a CPU handling the timekeeping. This can be further > optimized > and more power-friendly, I really did something simple-stupid. I guess we'll > try to get > that into a better shape with Hakan. But at least the timekeeping now works. Will look into it. > > * It uses the new RCU callbacks offlining feature. This way a full dynticks > CPU doesn't > need to keep the tick to handle local callbacks. This is still very > experimental though. > > * No more specific IPI vector for full dynticks. We just use the scheduler > ipi. > > The branch is: > > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git > 3.7-nohz1 > > There is still quite some work to do. > > == How to use? == > > Select: > CONFIG_NO_HZ > CONFIG_RCU_USER_QS > CONFIG_VIRT_CPU_ACCOUNTING_GEN > CONFIG_RCU_NOCB_CPU > CONFIG_NO_HZ_FULL > > You always need at least one timekeeping CPU. > > Let's imagine you have 4 CPUs. We keep the CPU 0 to offline RCU callbacks > there and to > handle the timekeeping. We set the rest as full dynticks. So you need the > following kernel > parameters: > > rcu_nocbs=1-3 full_nohz=1-3 > > (Note rcu_nocbs value must always be the same as full_nohz). > > Now if you want proper isolation you need to: > > * Migrate your processes adequately > * Migrate your irqs to CPU 0 > * Migrate the RCU nocb threads to CPU 0. Example with the above configuration: > > for p in $(ps -o pid= -C rcuo1,rcuo2,rcuo3) > do > taskset -cp 0 $p > done > > Then run what you want on the full dynticks CPUs. For best results, run 1 task > per CPU, mostly in userspace and mostly CPU bound (otherwise more IO = more > kernel > mode execution = more chances to get IPIs, tick restarted, workqueues, > kthreads, etc...) > > This page contains a good reminder for those interested in CPU isolation: > https://github.com/gby/linux/wiki > > But keep in mind that my tree is not yet ready for serious production. > > Happy Christmas, new year or whatever end of the world. > --- > > Frederic Weisbecker (32): > irq_work: Fix racy IRQ_WORK_BUSY flag setting > irq_work: Fix racy check on work pending flag > irq_work: Remove CONFIG_HAVE_IRQ_WORK > nohz: Add API to check tick state > irq_work: Don't stop the tick with pending works > irq_work: Make self-IPIs optable > printk: Wake up klogd using irq_work > Merge branch 'nohz/printk-v8' into 3.7-nohz1-stage > context_tracking: Add comments on interface and internals > cputime: Generic on-demand virtual cputime accounting > cputime: Allow dynamic switch between tick/virtual based cputime > accounting > cputime: Use accessors to read task cputime stats > cputime: Safely read cputime of full dynticks CPUs > nohz: Basic full dynticks interface > nohz: Assign timekeeping duty to a non-full-nohz CPU > nohz: Trace timekeeping update > nohz: Wake up full dynticks
Re: [ANNOUNCE] 3.7-nohz1
On Thu, 2012-12-20 at 19:32 +0100, Frederic Weisbecker wrote: > Hi, > Nice work Frederic! > So this is a new version of the nohz cpusets based on 3.7, except it's not > using > cpusets anymore and I actually based it on the middle of the 3.8 merge window > in order to get latest upstream full dynticks preparatory work: cputime > cleanups, > RCU user mode, context tracking subsystem, nohz code consolidation, ... > > So the big changes since the last nohz cpuset release are: > > * printk now uses irq work so it doesn't rely on the tick anymore (provided > your arch implements irq work with IPIs or alike). This chunk has been > proposed > for the 3.8 merge window: https://lkml.org/lkml/2012/12/17/177 > May be Linus will pull, may be not. We'll see. In any case I've included it > in this tree > but I'm not reposting this part of the patchset to avoid spamming you. > > * cputime doesn't rely on IPIs anymore. Now the reader does a special > computation to > remotely get the tickless cputime. > > * No more cpusets interface. Paul McKenney suggested me to start with a boot > time > kernel parameter to define the full dynticks cpumask. And he was totally > right, it > makes the code much more simple. That's a good way to start and to make the > mainlining > easier. We can still add a runtime configuration later if necessary. > > * Now there is always a CPU handling the timekeeping. This can be further > optimized > and more power-friendly, I really did something simple-stupid. I guess we'll > try to get > that into a better shape with Hakan. But at least the timekeeping now works. > > * It uses the new RCU callbacks offlining feature. This way a full dynticks > CPU doesn't > need to keep the tick to handle local callbacks. This is still very > experimental though. > > * No more specific IPI vector for full dynticks. We just use the scheduler > ipi. > > The branch is: > > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git > 3.7-nohz1 > > There is still quite some work to do. > > == How to use? == > > Select: > CONFIG_NO_HZ > CONFIG_RCU_USER_QS > CONFIG_VIRT_CPU_ACCOUNTING_GEN > CONFIG_RCU_NOCB_CPU > CONFIG_NO_HZ_FULL > > You always need at least one timekeeping CPU. > > Let's imagine you have 4 CPUs. We keep the CPU 0 to offline RCU callbacks > there and to > handle the timekeeping. We set the rest as full dynticks. So you need the > following kernel > parameters: > > rcu_nocbs=1-3 full_nohz=1-3 > > (Note rcu_nocbs value must always be the same as full_nohz). Why? You can't have: rcu_nocbs=1-4 full_nohz=1-3 or: rcu_nocbs=1-3 full_nohz=1-4 ? That needs to be fixed. Either with a warning, and/or to force the two to be the same. That is, if they specify: rcu_nocbs=1-3 full_nohz=1-4 Then set rcu_nocbs=1-4 with a warning about it. Or simply set full_nohz=1-3. -- Steve > > Now if you want proper isolation you need to: > > * Migrate your processes adequately > * Migrate your irqs to CPU 0 > * Migrate the RCU nocb threads to CPU 0. Example with the above configuration: > > for p in $(ps -o pid= -C rcuo1,rcuo2,rcuo3) > do > taskset -cp 0 $p > done > > Then run what you want on the full dynticks CPUs. For best results, run 1 task > per CPU, mostly in userspace and mostly CPU bound (otherwise more IO = more > kernel > mode execution = more chances to get IPIs, tick restarted, workqueues, > kthreads, etc...) > > This page contains a good reminder for those interested in CPU isolation: > https://github.com/gby/linux/wiki > > But keep in mind that my tree is not yet ready for serious production. > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[ANNOUNCE] 3.7-nohz1
Hi, So this is a new version of the nohz cpusets based on 3.7, except it's not using cpusets anymore and I actually based it on the middle of the 3.8 merge window in order to get latest upstream full dynticks preparatory work: cputime cleanups, RCU user mode, context tracking subsystem, nohz code consolidation, ... So the big changes since the last nohz cpuset release are: * printk now uses irq work so it doesn't rely on the tick anymore (provided your arch implements irq work with IPIs or alike). This chunk has been proposed for the 3.8 merge window: https://lkml.org/lkml/2012/12/17/177 May be Linus will pull, may be not. We'll see. In any case I've included it in this tree but I'm not reposting this part of the patchset to avoid spamming you. * cputime doesn't rely on IPIs anymore. Now the reader does a special computation to remotely get the tickless cputime. * No more cpusets interface. Paul McKenney suggested me to start with a boot time kernel parameter to define the full dynticks cpumask. And he was totally right, it makes the code much more simple. That's a good way to start and to make the mainlining easier. We can still add a runtime configuration later if necessary. * Now there is always a CPU handling the timekeeping. This can be further optimized and more power-friendly, I really did something simple-stupid. I guess we'll try to get that into a better shape with Hakan. But at least the timekeeping now works. * It uses the new RCU callbacks offlining feature. This way a full dynticks CPU doesn't need to keep the tick to handle local callbacks. This is still very experimental though. * No more specific IPI vector for full dynticks. We just use the scheduler ipi. The branch is: git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git 3.7-nohz1 There is still quite some work to do. == How to use? == Select: CONFIG_NO_HZ CONFIG_RCU_USER_QS CONFIG_VIRT_CPU_ACCOUNTING_GEN CONFIG_RCU_NOCB_CPU CONFIG_NO_HZ_FULL You always need at least one timekeeping CPU. Let's imagine you have 4 CPUs. We keep the CPU 0 to offline RCU callbacks there and to handle the timekeeping. We set the rest as full dynticks. So you need the following kernel parameters: rcu_nocbs=1-3 full_nohz=1-3 (Note rcu_nocbs value must always be the same as full_nohz). Now if you want proper isolation you need to: * Migrate your processes adequately * Migrate your irqs to CPU 0 * Migrate the RCU nocb threads to CPU 0. Example with the above configuration: for p in $(ps -o pid= -C rcuo1,rcuo2,rcuo3) do taskset -cp 0 $p done Then run what you want on the full dynticks CPUs. For best results, run 1 task per CPU, mostly in userspace and mostly CPU bound (otherwise more IO = more kernel mode execution = more chances to get IPIs, tick restarted, workqueues, kthreads, etc...) This page contains a good reminder for those interested in CPU isolation: https://github.com/gby/linux/wiki But keep in mind that my tree is not yet ready for serious production. Happy Christmas, new year or whatever end of the world. --- Frederic Weisbecker (32): irq_work: Fix racy IRQ_WORK_BUSY flag setting irq_work: Fix racy check on work pending flag irq_work: Remove CONFIG_HAVE_IRQ_WORK nohz: Add API to check tick state irq_work: Don't stop the tick with pending works irq_work: Make self-IPIs optable printk: Wake up klogd using irq_work Merge branch 'nohz/printk-v8' into 3.7-nohz1-stage context_tracking: Add comments on interface and internals cputime: Generic on-demand virtual cputime accounting cputime: Allow dynamic switch between tick/virtual based cputime accounting cputime: Use accessors to read task cputime stats cputime: Safely read cputime of full dynticks CPUs nohz: Basic full dynticks interface nohz: Assign timekeeping duty to a non-full-nohz CPU nohz: Trace timekeeping update nohz: Wake up full dynticks CPUs when a timer gets enqueued rcu: Restart the tick on non-responding full dynticks CPUs sched: Comment on rq->clock correctness in ttwu_do_wakeup() in nohz sched: Update rq clock on nohz CPU before migrating tasks sched: Update rq clock on nohz CPU before setting fair group shares sched: Update rq clock on tickless CPUs before calling check_preempt_curr() sched: Update rq clock earlier in unthrottle_cfs_rq sched: Update clock of nohz busiest rq before balancing sched: Update rq clock before idle balancing sched: Update nohz rq clock before searching busiest group on load balancing nohz: Move nohz load balancer selection into idle logic nohz: Full dynticks mode nohz: Only stop the tick on RCU nocb CPUs nohz: Don't turn off the tick if rcu needs it nohz: Don't stop the tick if posix cpu timers are running nohz: Add som
