Re: [yocto] Can I disable RT throttling?
On 13-03-11 2:13 PM, David Mulder wrote: Will that work on a core that's offline? Nope. Only with an online core controlled by the Linux scheduler. If you do end up trying to get AMP working, you need to plumbing to load the other OS/kernel in a reserved memory location, set the program counter and start the OS. But that secondary OS has to know how to behave in a system that it doesn't control, and you'll need ways to communicate with it from Linux. remoteproc/rpmsg can solve some of the issues that I mention, but it is far from out of the box. That's why there's more interest in running a single task with exclusive CPU in userspace. The work and scaffolding required to get an AMP system up and running is non trivial. I kinda thought so, but I was hopeful. After speaking with some co-workers, I have a new perspective on these delays: yes, we are trying to do a 10us control loop, but if we miss a step or two occasionally we can accept that. And looking online I see people indicating context switch times well below 10us (Core-i cpus), which is better than I had anticipated, and should be workable. So I'm going to approach this problem by just trying to squeeze the kernel as much as I can. Some things that I see to squeeze are /dev/cpu_dma_latency (should be 0) or max_cstate (should be as low as possible (0, maybe 1)), possibly idle=poll. Are there other kernel parameters that can minimize kernel interference/time? And perhaps hints about how to set them in Yocto or menuconfig? There's nothing "out of the box" that I can recommend, outside of saying "it depends on your platform". It's a matter of knowing your devices, their interrupts, and the configuration of your kernel. Using things like CONFIG_NOHZ will remove the timer tick, and hence ticks that you may not need, you want to move device interrupts off your core, except for the one that you want. Use cgroups/cpusets to control resources and the scheduler off your core with "other" tasks. pin/lock memory to avoid page faults, etc. If you check out the preempt-rt wiki page on kernel.org, a lot of the information there applies to making sure that your prioritized thread gets the most run time that it can. As we progress with the meta-realtime layer, scripts for the above, system configuration, services and will be part of the layer and might be of use. Cheers, Bruce Thanks! Dave ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Can I disable RT throttling?
> > Will that work on a core that's offline? > > Nope. Only with an online core controlled by the Linux scheduler. > If you do end up trying to get AMP working, you need to plumbing > to load the other OS/kernel in a reserved memory location, set the > program counter and start the OS. > > But that secondary OS has to know how to behave in a system that > it doesn't control, and you'll need ways to communicate with it > from Linux. > > remoteproc/rpmsg can solve some of the issues that I mention, but > it is far from out of the box. > > That's why there's more interest in running a single task with > exclusive CPU in userspace. The work and scaffolding required to > get an AMP system up and running is non trivial. I kinda thought so, but I was hopeful. After speaking with some co-workers, I have a new perspective on these delays: yes, we are trying to do a 10us control loop, but if we miss a step or two occasionally we can accept that. And looking online I see people indicating context switch times well below 10us (Core-i cpus), which is better than I had anticipated, and should be workable. So I'm going to approach this problem by just trying to squeeze the kernel as much as I can. Some things that I see to squeeze are /dev/cpu_dma_latency (should be 0) or max_cstate (should be as low as possible (0, maybe 1)), possibly idle=poll. Are there other kernel parameters that can minimize kernel interference/time? And perhaps hints about how to set them in Yocto or menuconfig? Thanks! Dave ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Can I disable RT throttling?
On 13-03-11 01:38 PM, David Mulder wrote: From: Trevor Woerner [mailto:twoer...@gmail.com] You can call sched_setaffinity() before fork()ing your task, or use taskset on the cmdline. Will that work on a core that's offline? Nope. Only with an online core controlled by the Linux scheduler. If you do end up trying to get AMP working, you need to plumbing to load the other OS/kernel in a reserved memory location, set the program counter and start the OS. But that secondary OS has to know how to behave in a system that it doesn't control, and you'll need ways to communicate with it from Linux. remoteproc/rpmsg can solve some of the issues that I mention, but it is far from out of the box. That's why there's more interest in running a single task with exclusive CPU in userspace. The work and scaffolding required to get an AMP system up and running is non trivial. Cheers, Bruce ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Can I disable RT throttling?
> From: Trevor Woerner [mailto:twoer...@gmail.com] > You can call sched_setaffinity() before fork()ing your task, or use > taskset on the cmdline. Will that work on a core that's offline? ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Can I disable RT throttling?
On Mon, Mar 11, 2013 at 12:15 PM, David Mulder wrote: > but how can I make my run my code on that core? You can call sched_setaffinity() before fork()ing your task, or use taskset on the cmdline. (sorry if this has already been answered but I'm not seeing any responses due to mailing list delays) ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Can I disable RT throttling?
Wow! Thanks, Bruce, Fredrik, and everyone, for all this useful info! > > I'm running a 10us control loop by (under vxWorks) setting one thread's > > priority to max and not yielding ever > > wrote: > You are looking for kernel functionality that doesn't exist yet. In > any SMP system there are sources of cross cpu interference that can't > be removed, if you do, the global state machines of the kernel will > break and the system will eventually come to a halt. > > That's the mainline/scalable point of view, there are plenty of "custom" > and one off solutions to the problem, such as hotplugging the cpu out > of the system and running in an AMP configuration where a bare metal, > or RTOS can monopolize a CPU since the global state machines don't > interfere. These have their advantages and disadvantages, but if you > are coming from an existing RTOS application, they might be the > closest options to get you the performance you are looking for. An AMP configuration does seem like the closest thing to what the RTOS gave me. Do you have more info about that? Specifically, I see that I can remove a core from the scheduler ($ echo 0 > /sys/devices/system/cpu//online), but how can I make my run my code on that core? Do I get the code running on that core first and then remove the core, or vice-versa, or some other concept? I do PCIe reads/writes - are those affected? I use atomic read-write-modify locks for critical sections - will those still function correctly (I can't imagine they would break, but I thought I'd ask)? > > So maybe disabling RT throttling is all I need to do, maybe some other > > approach is needed. Hopefully someone will know more than I do. > > > wrote: > The RT-throttling can be disabled with: > > echo -1 >/proc/sys/kernel/sched_rt_runtime_us > echo -1 >/proc/sys/kernel/sched_rt_period_us > > But as mentioned in the other answers things like the system tick, ipi:s > etc. will interfere. But significantly less then the rt-throttling :) Thanks, Fredrick! I will try this. Dave Mulder ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Can I disable RT throttling?
On Sun, Mar 10, 2013 at 7:42 AM, Fredrik Markström wrote: > The RT-throttling can be disabled with: > > echo -1 >/proc/sys/kernel/sched_rt_runtime_us > echo -1 >/proc/sys/kernel/sched_rt_period_us > > But as mentioned in the other answers things like the system tick, ipi:s > etc. will interfere. But significantly less then the rt-throttling :) Absolutely! And as long as you really know what you are doing, you won't kill the system .. turning off the throttling definitely gives a bit more rope to that end as well :P Cheers, Bruce > > /Fredrik > > ___ > yocto mailing list > yocto@yoctoproject.org > https://lists.yoctoproject.org/listinfo/yocto > -- "Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end" ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Can I disable RT throttling?
The RT-throttling can be disabled with: echo -1 >/proc/sys/kernel/sched_rt_runtime_us echo -1 >/proc/sys/kernel/sched_rt_period_us But as mentioned in the other answers things like the system tick, ipi:s etc. will interfere. But significantly less then the rt-throttling :) /Fredrik ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Can I disable RT throttling?
On Sat, Mar 9, 2013 at 12:12 PM, Vin Shelton wrote: > On Fri, Mar 8, 2013 at 10:28 PM, Bruce Ashfield > wrote: >> On 13-03-08 5:36 PM, David Mulder wrote: >>> I’m running a 10us control loop by (under vxWorks) setting one thread’s >>> priority to max and not yielding ever (letting other tasks run on other >>> cores), but Linux seems to thwart that capability: Ubuntu swaps out my >>> thread occasionally for hundreds of microseconds; Yocto prints a >>> “[sched_delayed] sched: RT throttling activated” message as soon as I >>> start my thread, so it seems likely that it will swap out my thread >>> periodically (I’m not quite able to confirm that yet), and even the >>> fastest thread swap that I’ve heard of is too slow. >> >> >> You are looking for kernel functionality that doesn't exist yet. In >> any SMP system there are sources of cross cpu interference that can't >> be removed, if you do, the global state machines of the kernel will >> break and the system will eventually come to a halt. >> >> The system is trying to save you from yourself, by throttling the >> RT task from taking the entire system down. >> >> There is work in the mainline kernel and -rt communities around cpu >> and cpu isolation (some of which we'll try and make available via >> the yocto meta-virtualization or meta-realtime layer, when they are >> ready), it goes by names such as task_nohz or adaptive_nohz. It's >> a complex problem to solve (interrupts, rcu, lapic), and there's not >> likely to be anything available in the short term. > > At ELC, Steve Rostedt said that this feature is currently targeted at > the June mainline kernel release, but there's a lot of work still to > be done, so that's not a firm date. Absolutely. There's no such thing as a firm date for anything, I (among others) are following the development closely, testing, and helping wherever possible. Cheers, Bruce > > Regards, > Vin Shelton > ___ > yocto mailing list > yocto@yoctoproject.org > https://lists.yoctoproject.org/listinfo/yocto -- "Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end" ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Can I disable RT throttling?
On Fri, Mar 8, 2013 at 10:28 PM, Bruce Ashfield wrote: > On 13-03-08 5:36 PM, David Mulder wrote: >> I’m running a 10us control loop by (under vxWorks) setting one thread’s >> priority to max and not yielding ever (letting other tasks run on other >> cores), but Linux seems to thwart that capability: Ubuntu swaps out my >> thread occasionally for hundreds of microseconds; Yocto prints a >> “[sched_delayed] sched: RT throttling activated” message as soon as I >> start my thread, so it seems likely that it will swap out my thread >> periodically (I’m not quite able to confirm that yet), and even the >> fastest thread swap that I’ve heard of is too slow. > > > You are looking for kernel functionality that doesn't exist yet. In > any SMP system there are sources of cross cpu interference that can't > be removed, if you do, the global state machines of the kernel will > break and the system will eventually come to a halt. > > The system is trying to save you from yourself, by throttling the > RT task from taking the entire system down. > > There is work in the mainline kernel and -rt communities around cpu > and cpu isolation (some of which we'll try and make available via > the yocto meta-virtualization or meta-realtime layer, when they are > ready), it goes by names such as task_nohz or adaptive_nohz. It's > a complex problem to solve (interrupts, rcu, lapic), and there's not > likely to be anything available in the short term. At ELC, Steve Rostedt said that this feature is currently targeted at the June mainline kernel release, but there's a lot of work still to be done, so that's not a firm date. Regards, Vin Shelton ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
Re: [yocto] Can I disable RT throttling?
On 13-03-08 5:36 PM, David Mulder wrote: Hi. I’m running a 10us control loop by (under vxWorks) setting one thread’s priority to max and not yielding ever (letting other tasks run on other cores), but Linux seems to thwart that capability: Ubuntu swaps out my thread occasionally for hundreds of microseconds; Yocto prints a “[sched_delayed] sched: RT throttling activated” message as soon as I start my thread, so it seems likely that it will swap out my thread periodically (I’m not quite able to confirm that yet), and even the fastest thread swap that I’ve heard of is too slow. You are looking for kernel functionality that doesn't exist yet. In any SMP system there are sources of cross cpu interference that can't be removed, if you do, the global state machines of the kernel will break and the system will eventually come to a halt. The system is trying to save you from yourself, by throttling the RT task from taking the entire system down. There is work in the mainline kernel and -rt communities around cpu and cpu isolation (some of which we'll try and make available via the yocto meta-virtualization or meta-realtime layer, when they are ready), it goes by names such as task_nohz or adaptive_nohz. It's a complex problem to solve (interrupts, rcu, lapic), and there's not likely to be anything available in the short term. That's the mainline/scalable point of view, there are plenty of "custom" and one off solutions to the problem, such as hotplugging the cpu out of the system and running in an AMP configuration where a bare metal, or RTOS can monopolize a CPU since the global state machines don't interfere. These have their advantages and disadvantages, but if you are coming from an existing RTOS application, they might be the closest options to get you the performance you are looking for. Alternatively, there's the preempt-rt kernel (that we make available in linux-yocto, if that suits) that if your control loop is interrupt driven, you'll be closer to your needs since the kernel is far more preemptible and deterministic and might be able to meet your deadlines. There are experimental patches like sched_deadline (EDF) that might also be applicable, depending on your application architecture. Searching for "making Linux hard realtime", you'll find plenty of talks and research over the years for yet more custom ways to get things done. So that's my summary, as you can see, it's a long standing, evolving and complex story .. and not something that likely has an 'out of the box' solution at the moment. Cheers, Bruce I 0tried changing the kernel’s Preemption Model to “No Forced Preemption (Server)”, but that didn’t remove the RT throttling message. I browsed the rest of menuconfig but nothing looked related. I read an article from 2008 that talked about the kernel reserving 5% of the CPU for non-SCHED_FIFO tasks, so that’s the direction and terminology I looked for in menuconfig. So maybe disabling RT throttling is all I need to do, maybe some other approach is needed. Hopefully someone will know more than I do. Thanks! Dave Mulder ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
[yocto] Can I disable RT throttling?
Hi. I'm running a 10us control loop by (under vxWorks) setting one thread's priority to max and not yielding ever (letting other tasks run on other cores), but Linux seems to thwart that capability: Ubuntu swaps out my thread occasionally for hundreds of microseconds; Yocto prints a "[sched_delayed] sched: RT throttling activated" message as soon as I start my thread, so it seems likely that it will swap out my thread periodically (I'm not quite able to confirm that yet), and even the fastest thread swap that I've heard of is too slow. I tried changing the kernel's Preemption Model to "No Forced Preemption (Server)", but that didn't remove the RT throttling message. I browsed the rest of menuconfig but nothing looked related. I read an article from 2008 that talked about the kernel reserving 5% of the CPU for non-SCHED_FIFO tasks, so that's the direction and terminology I looked for in menuconfig. So maybe disabling RT throttling is all I need to do, maybe some other approach is needed. Hopefully someone will know more than I do. Thanks! Dave Mulder ___ yocto mailing list yocto@yoctoproject.org https://lists.yoctoproject.org/listinfo/yocto
