Re: [PATCH v5 0/9] "Task_isolation" mode
On Sun, Dec 06, 2020 at 12:25:45AM +0100, Thomas Gleixner wrote: > Pavel, > > On Sat, Dec 05 2020 at 21:40, Pavel Machek wrote: > > So... what kind of guarantees does this aim to provide / what tasks it > > is useful for? > > > > For real time response, we have other approaches. > > Depends on your requirements. Some problems are actually better solved > with busy polling. See below. > > > If you want to guarantee performnace of the "isolated" task... I don't > > see how that works. Other tasks on the system still compete for DRAM > > bandwidth, caches, etc... > > Applications which want to run as undisturbed as possible. There is > quite a range of those: > > - Hardware in the loop simulation is today often done with that crude > approach of "offlining" a CPU and then instead of playing dead > jumping to a preloaded bare metal executable. That's a horrible hack > and impossible to debug, but gives them the results they need to > achieve. These applications are well optimized vs. cache and memory > foot print, so they don't worry about these things too much and they > surely don't run on SMI and BIOS value add inflicted machines. > > Don't even think about waiting for an interrupt to achieve what > these folks are doing. So no, there are problems which a general > purpose realtime OS cannot solve ever. > > - HPC computations on large data sets. While the memory foot print is > large the access patterns are cache optimized. > > The problem there is that any unnecessary IPI, tick interrupt or > whatever nuisance is disturbing the carefully optimized cache usage > and alone getting rid of the timer interrupt gained them measurable > performance. Even very low single digit percentage of runtime saving > is valuable for these folks because the compute time on such beasts > is expensive. > > - Realtime guests in KVM. With posted interrupts and a fully populated > host side page table there is no point in running host side > interrupts or IPIs for random accounting or whatever purposes as > they affect the latency in the guest. With all the side effects > mitigated and a properly set up guest and host it is possible to get > to a zero exit situation after the bootup phase which means pretty > much matching bare metal behaviour. > > Yes, you can do that with e.g. Jailhouse as well, but you lose lots > of the fancy things KVM provides. And people care about these not > just because they are fancy. They care because their application > scenario needs them. > > There are more reasons why people want to be able to get as much > isolation from the OS as possible but at the same time have a sane > execution environment, debugging, performance monitoring and the OS > provided protection mechanisms instead of horrible hacks. > > Isolation makes sense for a range of applications and there is no reason > why Linux should not support them. One good client for the task isolation is Open Data Plane. There are even some code stubs supposed to enable isolation where needed. > > If you want to guarantee performnace of the "isolated" task... I don't > > see how that works. Other tasks on the system still compete for DRAM > > bandwidth, caches, etc... My experiments say that typical delay caused by dry IPI or syscall is 2000-2 'ticks'. Typical delay caused by cache miss is 3-30 ticks. To guarantee cache / memory bandwidth, one can use resctrl. Linux has implementation of it for x86 only, but arm64 has support for for resctrl on CPU side. Thanks, Yury > Thanks, > > tglx
Re: [PATCH v5 0/9] "Task_isolation" mode
Pavel, On Sat, Dec 05 2020 at 21:40, Pavel Machek wrote: > So... what kind of guarantees does this aim to provide / what tasks it > is useful for? > > For real time response, we have other approaches. Depends on your requirements. Some problems are actually better solved with busy polling. See below. > If you want to guarantee performnace of the "isolated" task... I don't > see how that works. Other tasks on the system still compete for DRAM > bandwidth, caches, etc... Applications which want to run as undisturbed as possible. There is quite a range of those: - Hardware in the loop simulation is today often done with that crude approach of "offlining" a CPU and then instead of playing dead jumping to a preloaded bare metal executable. That's a horrible hack and impossible to debug, but gives them the results they need to achieve. These applications are well optimized vs. cache and memory foot print, so they don't worry about these things too much and they surely don't run on SMI and BIOS value add inflicted machines. Don't even think about waiting for an interrupt to achieve what these folks are doing. So no, there are problems which a general purpose realtime OS cannot solve ever. - HPC computations on large data sets. While the memory foot print is large the access patterns are cache optimized. The problem there is that any unnecessary IPI, tick interrupt or whatever nuisance is disturbing the carefully optimized cache usage and alone getting rid of the timer interrupt gained them measurable performance. Even very low single digit percentage of runtime saving is valuable for these folks because the compute time on such beasts is expensive. - Realtime guests in KVM. With posted interrupts and a fully populated host side page table there is no point in running host side interrupts or IPIs for random accounting or whatever purposes as they affect the latency in the guest. With all the side effects mitigated and a properly set up guest and host it is possible to get to a zero exit situation after the bootup phase which means pretty much matching bare metal behaviour. Yes, you can do that with e.g. Jailhouse as well, but you lose lots of the fancy things KVM provides. And people care about these not just because they are fancy. They care because their application scenario needs them. There are more reasons why people want to be able to get as much isolation from the OS as possible but at the same time have a sane execution environment, debugging, performance monitoring and the OS provided protection mechanisms instead of horrible hacks. Isolation makes sense for a range of applications and there is no reason why Linux should not support them. Thanks, tglx
Re: [PATCH v5 0/9] "Task_isolation" mode
Hi! > General description > > This is the result of development and maintenance of task isolation > functionality that originally started based on task isolation patch > v15 and was later updated to include v16. It provided predictable > environment for userspace tasks running on arm64 processors alongside > with full-featured Linux environment. It is intended to provide > reliable interruption-free environment from the point when a userspace > task enters isolation and until the moment it leaves isolation or > receives a signal intentionally sent to it, and was successfully used > for this purpose. While CPU isolation with nohz provides an > environment that is close to this requirement, the remaining IPIs and > other disturbances keep it from being usable for tasks that require > complete predictability of CPU timing. So... what kind of guarantees does this aim to provide / what tasks it is useful for? For real time response, we have other approaches. If you want to guarantee performnace of the "isolated" task... I don't see how that works. Other tasks on the system still compete for DRAM bandwidth, caches, etc... So... what is the usecase? Pavel -- http://www.livejournal.com/~pavelmachek signature.asc Description: Digital signature
Re: [EXT] Re: [PATCH v5 0/9] "Task_isolation" mode
On Wed, 2020-12-02 at 14:02 +, Mark Rutland wrote: > On Tue, Nov 24, 2020 at 05:40:49PM +, Alex Belits wrote: > > > > > I am having problems applying the patchset to today's linux-next. > > > > > > Which kernel should I be using ? > > > > The patches are against Linus' tree, in particular, commit > > a349e4c659609fd20e4beea89e5c4a4038e33a95 > > Is there any reason to base on that commit in particular? No specific reason for that particular commit. > Generally it's preferred that a series is based on a tag (so either a > release or an -rc kernel), and that the cover letter explains what > the > base is. If you can do that in future it'll make the series much > easier > to work with. Ok. -- Alex
Re: [EXT] Re: [PATCH v5 0/9] "Task_isolation" mode
On Tue, Nov 24, 2020 at 05:40:49PM +, Alex Belits wrote: > > On Tue, 2020-11-24 at 08:36 -0800, Tom Rix wrote: > > External Email > > > > --- > > --- > > > > On 11/23/20 9:42 AM, Alex Belits wrote: > > > This is an update of task isolation work that was originally done > > > by > > > Chris Metcalf and maintained by him until > > > November 2017. It is adapted to the current kernel and cleaned up > > > to > > > implement its functionality in a more complete and cleaner manner. > > > > I am having problems applying the patchset to today's linux-next. > > > > Which kernel should I be using ? > > The patches are against Linus' tree, in particular, commit > a349e4c659609fd20e4beea89e5c4a4038e33a95 Is there any reason to base on that commit in particular? Generally it's preferred that a series is based on a tag (so either a release or an -rc kernel), and that the cover letter explains what the base is. If you can do that in future it'll make the series much easier to work with. Thanks, Mark.
Re: [EXT] Re: [PATCH v5 0/9] "Task_isolation" mode
On Tue, 2020-11-24 at 08:36 -0800, Tom Rix wrote: > External Email > > --- > --- > > On 11/23/20 9:42 AM, Alex Belits wrote: > > This is an update of task isolation work that was originally done > > by > > Chris Metcalf and maintained by him until > > November 2017. It is adapted to the current kernel and cleaned up > > to > > implement its functionality in a more complete and cleaner manner. > > I am having problems applying the patchset to today's linux-next. > > Which kernel should I be using ? The patches are against Linus' tree, in particular, commit a349e4c659609fd20e4beea89e5c4a4038e33a95 -- Alex
Re: [PATCH v5 0/9] "Task_isolation" mode
On 11/23/20 9:42 AM, Alex Belits wrote: > This is an update of task isolation work that was originally done by > Chris Metcalf and maintained by him until > November 2017. It is adapted to the current kernel and cleaned up to > implement its functionality in a more complete and cleaner manner. I am having problems applying the patchset to today's linux-next. Which kernel should I be using ? Thanks, Tom
