Re: [PATCH v2 0/6] arm64: provide pseudo NMI with GICv3
On 01/05/18 21:51, Joel Fernandes wrote: > On Mon, Apr 30, 2018 at 2:46 AM Julien Thierry > wrote: > [...] >>> On Wed, Jan 17, 2018 at 3:54 AM, Julien Thierry > wrote: Hi, This series is a continuation of the work started by Daniel [1]. The > goal is to use GICv3 interrupt priorities to simulate an NMI. To achieve this, set two priorities, one for standard interrupts and another, higher priority, for NMIs. Whenever we want to disable > interrupts, we mask the standard priority instead so NMIs can still be raised. Some corner cases though still require to actually mask all interrupts effectively disabling the NMI. Of course, using priority masking instead of PSR.I comes at some cost. > On hackbench, the drop of performance seems to be >1% on average for this version. I can only attribute that to recent changes in the kernel as >>> >>> Do you have more specific performance data on the performance overhead >>> with this series? >>> > >> Not at the moment. I was planning on doing a v3 anyway considering this >> series is getting a bit old and the GICv3 driver has had some > modifications. > > Great! Looking forward to it, will try to find some time to review this set > as well. > >> Once I get to it I can try to have more detailed performance data on a >> recent kernel. I've really only measured the performance on hackbench >> and on kernel build from defconfig (and for the kernel build the >> performance difference was completely hidden by the noise). > hackbench seems slightly slower compared to my other benchmarks while > the runs with the use of GICv3 priorities have stayed in the same time > frames. KVM Guests do not seem to be affected preformance-wise by the host > using PMR to mask interrupts or not. Currently, only PPIs and SPIs can be set as NMIs. IPIs being currently hardcoded IRQ numbers, there isn't a generic interface to set SGIs as > NMI for now. I don't think there is any reason LPIs should be allowed to > be set as NMI as they do not have an active state. When an NMI is active on a CPU, no other NMI can be triggered on the > CPU. Requirements to use this: - Have GICv3 - SCR_EL3.FIQ is set to 1 when linux runs >>> >>> Ah I see it mentioned here. Again, can you clarify if this is >>> something that can be misconfigured? Is it something that the >>> bootloader sets? >>> > >> Yes, this is something that the bootloader sets and we have seen a few >> cases where it is set to 0, so it can be "misconfigured". > >> It is not impossible to handle this case, but this bit affects the view >> the GICv3 CPU interface has on interrupt priority values. However it >> requires to add some conditions in both the interrupt handling and >> masking/unmasking code, so ideally we would avoid adding things to this. > >> But the idea is that Linux only deals with group 1 interrupts, and group >> 1 interrupts are only signaled as FIQs when the execution state is >> secure or at EL3, which should never happen in Linux's case. So ideally >> we'd like firmwares to set up this bit properly rather than to have to >> deal with both cases when only one of them makes sense for Linux. > > From what I see, on all our platforms, FIQs are delivered to the secure > monitor only. Which is the reason for this patchset in the first place. I > can't imagine a usecase that is not designed like this (and have not come > across this), so its probably Ok to just assume SCR_EL3.FIQ is to 1. > > In the future, if SCR_EL3.FIQ is set 0, then the NMI should use the FIQ > mechanism delivered to the non-secure OS. > > Does what I say make sense or was I just shooting arrows in the dark? :-P It would mean teaching Group-0 interrupts to the arm64 kernel. Not an impossible task, but that'd be catering for a minority of broken systems. In my book, that's at the absolute bottom of the priority range (pun intended...). M. -- Jazz is not dead. It just smells funny...
Re: [PATCH v2 0/6] arm64: provide pseudo NMI with GICv3
On Tue, May 01, 2018 at 06:18:44PM +, Joel Fernandes wrote: > > > From my understanding, Linux's hardlockup detector already uses the ARM > PMU > > > interrupt to check whether some task is stuck. I haven't looked at the > > > details of the implementation yet, but in theory having the PMU > interrupt as > > > NMI should make the hard lockup detector use the NMI. > > > > > > When I do the v3, I'll have a look at this to check whether the > hardlockup > > > detector works fine when using NMI. > > > That's what I saw on arch/arm (with some of the much older FIQ work). > > > Once you have PMU and the appropriate config to *admit* to supporting > > hard lockup then it will "just work" and be setup automatically during > > kernel boot. > > > Actually the problem then becomes that if you want to use the PMU > > for anything else then you may end up having to disable the hard > > lockup detector. > > This problem is not anything pseudo-NMI specific though right? > Contention/constraints on PMU resources should be a problem even on > platforms with real NMI. Quite so. Nothing specific to pseudo-NMI; merely part of life's rich tapestry. Moreover it is a potential surprise for anyone coming from x86 since I think the performance monitors make it easier to run hard lockup alongside some simple perf activity. Either way, if you are impacted, its easy to disable the hard lockup using procfs. Daniel.
Re: [PATCH v2 0/6] arm64: provide pseudo NMI with GICv3
On Mon, Apr 30, 2018 at 2:46 AM Julien Thierry wrote: [...] > > On Wed, Jan 17, 2018 at 3:54 AM, Julien Thierry wrote: > >> Hi, > >> > >> This series is a continuation of the work started by Daniel [1]. The goal > >> is to use GICv3 interrupt priorities to simulate an NMI. > >> > >> To achieve this, set two priorities, one for standard interrupts and > >> another, higher priority, for NMIs. Whenever we want to disable interrupts, > >> we mask the standard priority instead so NMIs can still be raised. Some > >> corner cases though still require to actually mask all interrupts > >> effectively disabling the NMI. > >> > >> Of course, using priority masking instead of PSR.I comes at some cost. On > >> hackbench, the drop of performance seems to be >1% on average for this > >> version. I can only attribute that to recent changes in the kernel as > > > > Do you have more specific performance data on the performance overhead > > with this series? > > > Not at the moment. I was planning on doing a v3 anyway considering this > series is getting a bit old and the GICv3 driver has had some modifications. Great! Looking forward to it, will try to find some time to review this set as well. > Once I get to it I can try to have more detailed performance data on a > recent kernel. I've really only measured the performance on hackbench > and on kernel build from defconfig (and for the kernel build the > performance difference was completely hidden by the noise). > >> hackbench seems slightly slower compared to my other benchmarks while the > >> runs with the use of GICv3 priorities have stayed in the same time frames. > >> KVM Guests do not seem to be affected preformance-wise by the host using > >> PMR to mask interrupts or not. > >> > >> Currently, only PPIs and SPIs can be set as NMIs. IPIs being currently > >> hardcoded IRQ numbers, there isn't a generic interface to set SGIs as NMI > >> for now. I don't think there is any reason LPIs should be allowed to be set > >> as NMI as they do not have an active state. > >> When an NMI is active on a CPU, no other NMI can be triggered on the CPU. > >> > >> > >> Requirements to use this: > >> - Have GICv3 > >> - SCR_EL3.FIQ is set to 1 when linux runs > > > > Ah I see it mentioned here. Again, can you clarify if this is > > something that can be misconfigured? Is it something that the > > bootloader sets? > > > Yes, this is something that the bootloader sets and we have seen a few > cases where it is set to 0, so it can be "misconfigured". > It is not impossible to handle this case, but this bit affects the view > the GICv3 CPU interface has on interrupt priority values. However it > requires to add some conditions in both the interrupt handling and > masking/unmasking code, so ideally we would avoid adding things to this. > But the idea is that Linux only deals with group 1 interrupts, and group > 1 interrupts are only signaled as FIQs when the execution state is > secure or at EL3, which should never happen in Linux's case. So ideally > we'd like firmwares to set up this bit properly rather than to have to > deal with both cases when only one of them makes sense for Linux. From what I see, on all our platforms, FIQs are delivered to the secure monitor only. Which is the reason for this patchset in the first place. I can't imagine a usecase that is not designed like this (and have not come across this), so its probably Ok to just assume SCR_EL3.FIQ is to 1. In the future, if SCR_EL3.FIQ is set 0, then the NMI should use the FIQ mechanism delivered to the non-secure OS. Does what I say make sense or was I just shooting arrows in the dark? :-P thanks, - Joel
Re: [PATCH v2 0/6] arm64: provide pseudo NMI with GICv3
> > On 29/04/18 07:37, Joel Fernandes wrote: > > > On Wed, Jan 17, 2018 at 4:10 AM, Julien Thierry < julien.thie...@arm.com> wrote: > > > > Hi, > > > > > > > > On 17/01/18 11:54, Julien Thierry wrote: > > > > > > > > > > This series is a continuation of the work started by Daniel [1]. The goal > > > > > is to use GICv3 interrupt priorities to simulate an NMI. > > > > > > > > > > > > > > > > > I have submitted a separate series making use of this feature for the ARM > > > > PMUv3 interrupt [1]. > > > > > > I guess the hard lockup detector using NMI could be a nice next step > > > to see how well it works with lock up detection. That's the main > > > usecase for my interest. However, perf profiling is also a strong one. > > > > > > > From my understanding, Linux's hardlockup detector already uses the ARM PMU > > interrupt to check whether some task is stuck. I haven't looked at the > > details of the implementation yet, but in theory having the PMU interrupt as > > NMI should make the hard lockup detector use the NMI. > > > > When I do the v3, I'll have a look at this to check whether the hardlockup > > detector works fine when using NMI. > That's what I saw on arch/arm (with some of the much older FIQ work). > Once you have PMU and the appropriate config to *admit* to supporting > hard lockup then it will "just work" and be setup automatically during > kernel boot. > Actually the problem then becomes that if you want to use the PMU > for anything else then you may end up having to disable the hard > lockup detector. This problem is not anything pseudo-NMI specific though right? Contention/constraints on PMU resources should be a problem even on platforms with real NMI. thanks, - Joel
Re: [PATCH v2 0/6] arm64: provide pseudo NMI with GICv3
On Mon, Apr 30, 2018 at 10:53:17AM +0100, Julien Thierry wrote: > > > On 29/04/18 07:37, Joel Fernandes wrote: > > On Wed, Jan 17, 2018 at 4:10 AM, Julien Thierry > > wrote: > > > Hi, > > > > > > On 17/01/18 11:54, Julien Thierry wrote: > > > > > > > > This series is a continuation of the work started by Daniel [1]. The > > > > goal > > > > is to use GICv3 interrupt priorities to simulate an NMI. > > > > > > > > > > > > > I have submitted a separate series making use of this feature for the ARM > > > PMUv3 interrupt [1]. > > > > I guess the hard lockup detector using NMI could be a nice next step > > to see how well it works with lock up detection. That's the main > > usecase for my interest. However, perf profiling is also a strong one. > > > > From my understanding, Linux's hardlockup detector already uses the ARM PMU > interrupt to check whether some task is stuck. I haven't looked at the > details of the implementation yet, but in theory having the PMU interrupt as > NMI should make the hard lockup detector use the NMI. > > When I do the v3, I'll have a look at this to check whether the hardlockup > detector works fine when using NMI. That's what I saw on arch/arm (with some of the much older FIQ work). Once you have PMU and the appropriate config to *admit* to supporting hard lockup then it will "just work" and be setup automatically during kernel boot. Actually the problem then becomes that if you want to use the PMU for anything else then you may end up having to disable the hard lockup detector. Daniel.
Re: [PATCH v2 0/6] arm64: provide pseudo NMI with GICv3
On 29/04/18 07:37, Joel Fernandes wrote: On Wed, Jan 17, 2018 at 4:10 AM, Julien Thierry wrote: Hi, On 17/01/18 11:54, Julien Thierry wrote: This series is a continuation of the work started by Daniel [1]. The goal is to use GICv3 interrupt priorities to simulate an NMI. I have submitted a separate series making use of this feature for the ARM PMUv3 interrupt [1]. I guess the hard lockup detector using NMI could be a nice next step to see how well it works with lock up detection. That's the main usecase for my interest. However, perf profiling is also a strong one. From my understanding, Linux's hardlockup detector already uses the ARM PMU interrupt to check whether some task is stuck. I haven't looked at the details of the implementation yet, but in theory having the PMU interrupt as NMI should make the hard lockup detector use the NMI. When I do the v3, I'll have a look at this to check whether the hardlockup detector works fine when using NMI. Cheers, -- Julien Thierry
Re: [PATCH v2 0/6] arm64: provide pseudo NMI with GICv3
Hi Joel, Thanks for the interest. On 29/04/18 07:35, Joel Fernandes wrote: Hi Julien, I am interested in evaluating if using this is feasible for our Android devices. There is quite a usecase for lockup detection that it seems worthwhile if it works well. Atleast I feel this can be used a debug option considering the performance downgrade. Do you have more details of if any GICv3 based system will work, or is there a way an SoC can be misconfigured so that this series will not work? I think Marc told me that's possible, but I wasn't sure. I will be quite happy if it works on SoC as long as they have the requisite GIC version. Some more questions below: On Wed, Jan 17, 2018 at 3:54 AM, Julien Thierry wrote: Hi, This series is a continuation of the work started by Daniel [1]. The goal is to use GICv3 interrupt priorities to simulate an NMI. To achieve this, set two priorities, one for standard interrupts and another, higher priority, for NMIs. Whenever we want to disable interrupts, we mask the standard priority instead so NMIs can still be raised. Some corner cases though still require to actually mask all interrupts effectively disabling the NMI. Of course, using priority masking instead of PSR.I comes at some cost. On hackbench, the drop of performance seems to be >1% on average for this version. I can only attribute that to recent changes in the kernel as Do you have more specific performance data on the performance overhead with this series? Not at the moment. I was planning on doing a v3 anyway considering this series is getting a bit old and the GICv3 driver has had some modifications. Once I get to it I can try to have more detailed performance data on a recent kernel. I've really only measured the performance on hackbench and on kernel build from defconfig (and for the kernel build the performance difference was completely hidden by the noise). hackbench seems slightly slower compared to my other benchmarks while the runs with the use of GICv3 priorities have stayed in the same time frames. KVM Guests do not seem to be affected preformance-wise by the host using PMR to mask interrupts or not. Currently, only PPIs and SPIs can be set as NMIs. IPIs being currently hardcoded IRQ numbers, there isn't a generic interface to set SGIs as NMI for now. I don't think there is any reason LPIs should be allowed to be set as NMI as they do not have an active state. When an NMI is active on a CPU, no other NMI can be triggered on the CPU. Requirements to use this: - Have GICv3 - SCR_EL3.FIQ is set to 1 when linux runs Ah I see it mentioned here. Again, can you clarify if this is something that can be misconfigured? Is it something that the bootloader sets? Yes, this is something that the bootloader sets and we have seen a few cases where it is set to 0, so it can be "misconfigured". It is not impossible to handle this case, but this bit affects the view the GICv3 CPU interface has on interrupt priority values. However it requires to add some conditions in both the interrupt handling and masking/unmasking code, so ideally we would avoid adding things to this. But the idea is that Linux only deals with group 1 interrupts, and group 1 interrupts are only signaled as FIQs when the execution state is secure or at EL3, which should never happen in Linux's case. So ideally we'd like firmwares to set up this bit properly rather than to have to deal with both cases when only one of them makes sense for Linux. Sorry if these questions sound premature, I haven't yet taken a closer look at the series. Cheers, -- Julien Thierry
Re: [PATCH v2 0/6] arm64: provide pseudo NMI with GICv3
On Wed, Jan 17, 2018 at 4:10 AM, Julien Thierry wrote: > Hi, > > On 17/01/18 11:54, Julien Thierry wrote: >> >> This series is a continuation of the work started by Daniel [1]. The goal >> is to use GICv3 interrupt priorities to simulate an NMI. >> > > > I have submitted a separate series making use of this feature for the ARM > PMUv3 interrupt [1]. I guess the hard lockup detector using NMI could be a nice next step to see how well it works with lock up detection. That's the main usecase for my interest. However, perf profiling is also a strong one. thanks, - Joel
Re: [PATCH v2 0/6] arm64: provide pseudo NMI with GICv3
Hi Julien, I am interested in evaluating if using this is feasible for our Android devices. There is quite a usecase for lockup detection that it seems worthwhile if it works well. Atleast I feel this can be used a debug option considering the performance downgrade. Do you have more details of if any GICv3 based system will work, or is there a way an SoC can be misconfigured so that this series will not work? I think Marc told me that's possible, but I wasn't sure. I will be quite happy if it works on SoC as long as they have the requisite GIC version. Some more questions below: On Wed, Jan 17, 2018 at 3:54 AM, Julien Thierry wrote: > Hi, > > This series is a continuation of the work started by Daniel [1]. The goal > is to use GICv3 interrupt priorities to simulate an NMI. > > To achieve this, set two priorities, one for standard interrupts and > another, higher priority, for NMIs. Whenever we want to disable interrupts, > we mask the standard priority instead so NMIs can still be raised. Some > corner cases though still require to actually mask all interrupts > effectively disabling the NMI. > > Of course, using priority masking instead of PSR.I comes at some cost. On > hackbench, the drop of performance seems to be >1% on average for this > version. I can only attribute that to recent changes in the kernel as Do you have more specific performance data on the performance overhead with this series? > hackbench seems slightly slower compared to my other benchmarks while the > runs with the use of GICv3 priorities have stayed in the same time frames. > KVM Guests do not seem to be affected preformance-wise by the host using > PMR to mask interrupts or not. > > Currently, only PPIs and SPIs can be set as NMIs. IPIs being currently > hardcoded IRQ numbers, there isn't a generic interface to set SGIs as NMI > for now. I don't think there is any reason LPIs should be allowed to be set > as NMI as they do not have an active state. > When an NMI is active on a CPU, no other NMI can be triggered on the CPU. > > > Requirements to use this: > - Have GICv3 > - SCR_EL3.FIQ is set to 1 when linux runs Ah I see it mentioned here. Again, can you clarify if this is something that can be misconfigured? Is it something that the bootloader sets? Sorry if these questions sound premature, I haven't yet taken a closer look at the series. thanks, - Joel
Re: [PATCH v2 0/6] arm64: provide pseudo NMI with GICv3
Hi, On 17/01/18 11:54, Julien Thierry wrote: This series is a continuation of the work started by Daniel [1]. The goal is to use GICv3 interrupt priorities to simulate an NMI. I have submitted a separate series making use of this feature for the ARM PMUv3 interrupt [1]. [1] https://www.spinics.net/lists/arm-kernel/msg629402.html Cheers, -- Julien Thierry
