Hi guys, This is my first post, so I assume we are talking about Linux here :) I'll take kernel 3.10.105 as example
On Monday, May 22, 2017 at 3:03:53 PM UTC+2, Wojciech Kudla wrote: > > There's lots of work being scheduled on even isolated cpus. > Indeed, isolcpus option only changes the default scheduler mask : parsing at kernel/sched/core.c:5768 <http://elixir.free-electrons.com/linux/v3.10.105/source/kernel/sched/core.c#L5768> and usage at init scheduler kernel/sched/core.c:6713 <http://elixir.free-electrons.com/linux/v3.10.105/source/kernel/sched/core.c#L6713> Apps not using the default allocation (kernel thread) and remasked user tasks can still use it. > If you are not running a tickless kernel, you should see around 1000 local > timer interrupts per second (by default). > This is the HZ constant of the kernel. On most distribution it's indeed 1000HZ. You can check this with a grep on your /proc/config.gz or /boot/config-$(uname -r) : # zgrep ^CONFIG_HZ= /proc/config.gz Or you can also check the value of the "LOC" interrupt. The added value each second is the value of HZ (can be 100, 250, 300, 1000). # watch -d -n1 grep LOC: /proc/interrupt (if you are lazy like me, check my ratethis <https://github.com/Saruspete/ratethis> script) You will also see soft irqs (if you haven't affinitized them with some > housekeeping cpu), non maskable interrupts/machine check errors, work queue > tasks, etc. > Are you talking about bottom-halves (kernel irq-related threads) ? For NMI and MCE, you shouldn't be seeing them to anything else than 0. NMI can be generated by watchdogs, but it's better to disable them in sake of avoiding jitters. > As for rcu, even with offloading you will see the isolated cores > performing work required to schedule the callbacks on the offloaded cpus. > You can solve that by switching to rcu callback polling, but my point here > is, there's a number of different types of tasks that will run on isolated > cpus. > In Linux, you can't fully isolate CPU0, and you can't set it to nohzfull : it's used for timekeeping and other management stuff : At each tick, there's a lot of work done: scheduling, jiffies, loadavg, timekeeping, timers, RCU callbacks... Also, for each cpu you have a number of kernel threads that cannot be moved : - ksoftirqd : a generic handler for "softirq" tasks, that is execute the "action" function of registered softirq vectors (check __do_softirq <http://elixir.free-electrons.com/linux/v3.10.105/source/kernel/softirq.c#L211> in kernel/softirq.c). this is mostly used by network drivers and tasklets. - migration : the only task running as realtime 99, that is responsible to migrate tasks from a cpu to another (to balance the load). - watchdog : this detects soft and hard lockups (if CONFIG_LOCKUP_DETECTOR is set in kernel config). Also as you can't fully isolate CPU0, that means socket0 will have parts of its LLC trashed, thus impacting other threads on the same socket, inducing jitter. There's also some vendor technology to reduce this, but it's vendor and model dependant. Adrien. > On Mon, 22 May 2017, 11:41 Himanshu Sharma, <imhima...@gmail.com > <javascript:>> wrote: > >> Thanks Wojciech for the quick reply. I read about rcu offloading and in >> my testing, can confirm that kernel rcu threads are scheduled on core 0 >> even if core 0 is isolated. Another thing I observed was that there are >> some kworker threads which run on isolated cpus other than 0. Is this >> expected behavior, because I used to think that isolated cpus are not >> touched by the kernel. These kworker threads will definitely lead to >> context switches and hamper performance a little bit. And I am afraid we >> can do nothing to get rid of them. >> >> Himanshu Sharma >> >> >> On Mon, May 22, 2017 at 2:08 PM, Wojciech Kudla <wojciec...@gmail.com >> <javascript:>> wrote: >> >>> There's a number of kernel tasks that are implicitly bound to cpu0. For >>> an example of one have a look at rcu offloading and its restrictions. >>> >>> On Mon, 22 May 2017, 08:59 Himanshu Sharma, <imhima...@gmail.com >>> <javascript:>> wrote: >>> >>>> Hi Michael >>>> >>>> Did you find a satisfactory reason for not isolating cpu 0, maybe some >>>> low level OS code that is bound to run on core 0? I am also stuck at this >>>> question right now and am thinking you might have an answer. >>>> >>>> Thanks >>>> Himanshu >>>> >>>> On Wednesday, March 4, 2015 at 8:08:22 PM UTC+5:30, Michael Mattoss >>>> wrote: >>>>> >>>>> Hi guys, >>>>> >>>>> I'm in the process of setting up a new dual-socket server for a low >>>>> latency workload. >>>>> The application will run exclusively on one CPU and everything else >>>>> (i.e. OS, non-critical processes) will run on the other CPU to avoid >>>>> cache >>>>> pollution. >>>>> I was wondering if it makes any difference as to which one of the >>>>> 2 CPU's is chosen for the workload. >>>>> Theoretically, there should be no difference but I was wondering if >>>>> there is some low-level stuff (e.g. core OS code, system management >>>>> interrupts handlers) that is statically allocated to CPU-0 as every >>>>> system >>>>> has at least 1 CPU. >>>>> Of course, if that's the case then CPU-1 is the better choice. >>>>> >>>>> Any thoughts/suggestions? >>>>> >>>>> Thanks, >>>>> Michael >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "mechanical-sympathy" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to mechanical-sympathy+unsubscr...@googlegroups.com >>>> <javascript:>. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "mechanical-sympathy" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/mechanical-sympathy/SnJ6LTKCjEU/unsubscribe >>> . >>> To unsubscribe from this group and all its topics, send an email to >>> mechanical-sympathy+unsubscr...@googlegroups.com <javascript:>. >>> >> >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "mechanical-sympathy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to mechanical-sympathy+unsubscr...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
