Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
On Mon, 7 Sep 2020 13:41:40 +0200 pet...@infradead.org wrote: > > I may have once known the rationale behind all this, but it's been a > > long time since I worked on the PI code, and it's out of my cache. Now I'm trying to cache all this in from a long PTO ;-) > > I suffer much the same problem. > > So cenceptually there's the problem that idle must always be runnable, > and the moment you boost it, it becomes subject to a different > scheduling class. > > Imagine for example what happens when we boost it to RT and then have it > be subject to throttling. What are we going to run when the idle task > is no longer elegible to run. > > (it might all work out by accident, but ISTR we had a whole bunch of fun > in the earlier days of RT due to things like that) I'm thinking if a mutex_trylock() happens in an interrupt context, which I believe is the only way idle could trigger it, it could not be throttled if boosted, because it's in interrupt context. And it would have to release the mutex before leaving interrupt context putting it back to the idle state before throttling could take effect. Thus, I'm still not convinced that mutex_trylock() is still bad in interrupt context yet. At least not from what I can tell. -- Steve
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
Adding the kexec list as well.
Joerg Vehlow writes:
> Hi Eric,
>> What is this patch supposed to be doing?
>>
>> What bug is it fixing?
> This information is part in the first message of this mail thread.
> The patch was intendedfor the active discussion in this thread,
> not for a broad review.
> A short summary: In the rt kernel, a panic in an interrupt context does
> not start the dump-capture kernel, because there is a mutex_trylock in
> __crash_kexe. If this is called in interrupt context, it always fails.
> In the non-rt kernel calling mutex_trylock is not allowed according to
> the comment of the function, but it still works.
Thanks. For whatever reason I did not see the rest of this thread
when I was replying to your patch.
I get the feeling the rt kernel is breaking this case deliberately.
I don't know of any reason why a trylock couldn't work.
That said I won't propose fixing up the locks that way.
>> A BUG_ON that triggers inside of BUG_ONs seems not just suspect but
>> outright impossible to make use of.
> I am not entirely sure what would happen here. But even if it gets in
> some kind ofendless loop, I guess this is ok, because it allows finding
> the problem. A piece of code in the function, that ensures the precondition
> is a lot better than relying on only a comment.
> If this was in mtex_trylock, the bug described above wouldn't have sneaked
> in 12 years ago...
BUG_ON's are more likely to hide a problem then to show it.
Sometimes they are appropriate but the should be avoided as much as
possible.
>> I get the feeling skimming this that it is time to sort out and simplify
>> the locking here, rather than make it more complex, and more likely to
>> fail.
> I would very much like that, but sadly it looks like it is not possible.
> Either it wouldrequire blocking locks, that may fail, or not locking at
> all, that may also fail.Using a different kind of lock (like spinlock)
> is also not possible, becausespinlock_trylock again uses mutex_trylock
> in the rt kernel.
I think it is possible but the locking needs to be relooked at.
>> I get the feeling that over the years somehow the assumption that the
>> rest of the kernel is broken and that we need to get out of the broken
>> kernel as fast and as simply as possible has been lost.
> Yes I also have the feeling, that the mutexes need fixing, but I wouldn't
> to post any patch for that. At the moment, given the interface of the mutex,
> this is clearly a bug in kexec, even if it works in the non-rt kernel.
Cleanups that break the code. Sigh.
The code was written correctly for this case and was fine until
8c5a1cf0ad3a ("kexec: use a mutex for locking rather than xchg()").
Mostly because I didn't trust locks given their comparatively high level
of abstraction and what do you know that turned out to be correct in
this case.
It definitely looks time to see how the locking can be improved on the
kexec on panic code path.
Eric
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
Hi Eric, What is this patch supposed to be doing? What bug is it fixing? This information is part in the first message of this mail thread. The patch was intendedfor the active discussion in this thread, not for a broad review. A short summary: In the rt kernel, a panic in an interrupt context does not start the dump-capture kernel, because there is a mutex_trylock in __crash_kexe. If this is called in interrupt context, it always fails. In the non-rt kernel calling mutex_trylock is not allowed according to the comment of the function, but it still works. A BUG_ON that triggers inside of BUG_ONs seems not just suspect but outright impossible to make use of. I am not entirely sure what would happen here. But even if it gets in some kind ofendless loop, I guess this is ok, because it allows finding the problem. A piece of code in the function, that ensures the precondition is a lot better than relying on only a comment. If this was in mtex_trylock, the bug described above wouldn't have sneaked in 12 years ago... I get the feeling skimming this that it is time to sort out and simplify the locking here, rather than make it more complex, and more likely to fail. I would very much like that, but sadly it looks like it is not possible. Either it wouldrequire blocking locks, that may fail, or not locking at all, that may also fail.Using a different kind of lock (like spinlock) is also not possible, becausespinlock_trylock again uses mutex_trylock in the rt kernel. I get the feeling that over the years somehow the assumption that the rest of the kernel is broken and that we need to get out of the broken kernel as fast and as simply as possible has been lost. Yes I also have the feeling, that the mutexes need fixing, but I wouldn't to post any patch for that. At the moment, given the interface of the mutex, this is clearly a bug in kexec, even if it works in the non-rt kernel. Jörg
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
Joerg Vehlow writes:
> Hi,
>
> here is the new version of the patch based on Peters suggestion
> It looks like it works fine. I added the BUG_ON to __crash_kexec, because it
> is
> a precondition, that panic_cpu is set correctly, otherwise the whole locking
> logic fails.
>
> The mutex_trylock can still be used, because it is only in syscall context and
> no interrupt context.
What is this patch supposed to be doing?
What bug is it fixing?
A BUG_ON that triggers inside of BUG_ONs seems not just suspect but
outright impossible to make use of.
I get the feeling skimming this that it is time to sort out and simplify
the locking here, rather than make it more complex, and more likely to
fail.
I get the feeling that over the years somehow the assumption that the
rest of the kernel is broken and that we need to get out of the broken
kernel as fast and as simply as possible has been lost.
Eric
> ---
> kernel/kexec.c | 8 ++--
> kernel/kexec_core.c | 86 +++--
> kernel/kexec_file.c | 4 +-
> kernel/kexec_internal.h | 6 ++-
> 4 files changed, 69 insertions(+), 35 deletions(-)
>
> diff --git a/kernel/kexec.c b/kernel/kexec.c
> index f977786fe498..118a012aeac2 100644
> --- a/kernel/kexec.c
> +++ b/kernel/kexec.c
> @@ -255,12 +255,12 @@ SYSCALL_DEFINE4(kexec_load, unsigned long, entry,
> unsigned
> long, nr_segments,
> *
> * KISS: always take the mutex.
> */
> - if (!mutex_trylock(_mutex))
> + if (!kexec_trylock())
> return -EBUSY;
>
> result = do_kexec_load(entry, nr_segments, segments, flags);
>
> - mutex_unlock(_mutex);
> + kexec_unlock();
>
> return result;
> }
> @@ -309,12 +309,12 @@ COMPAT_SYSCALL_DEFINE4(kexec_load, compat_ulong_t,
> entry,
> *
> * KISS: always take the mutex.
> */
> - if (!mutex_trylock(_mutex))
> + if (!kexec_trylock())
> return -EBUSY;
>
> result = do_kexec_load(entry, nr_segments, ksegments, flags);
>
> - mutex_unlock(_mutex);
> + kexec_unlock();
>
> return result;
> }
> diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c
> index c19c0dad1ebe..71682a33b1ba 100644
> --- a/kernel/kexec_core.c
> +++ b/kernel/kexec_core.c
> @@ -45,7 +45,7 @@
> #include
> #include "kexec_internal.h"
>
> -DEFINE_MUTEX(kexec_mutex);
> +static DEFINE_MUTEX(kexec_mutex);
>
> /* Per cpu memory for storing cpu states in case of system crash. */
> note_buf_t __percpu *crash_notes;
> @@ -70,6 +70,43 @@ struct resource crashk_low_res = {
> .desc = IORES_DESC_CRASH_KERNEL
> };
>
> +void kexec_lock(void)
> +{
> + /*
> + * LOCK kexec_mutex cmpxchg(_cpu, INVALID, cpu)
> + * MB MB
> + * panic_cpu == INVALID kexec_mutex == LOCKED
> + *
> + * Ensures either we observe the cmpxchg, or crash_kernel() observes
> + * our lock acquisition.
> + */
> + mutex_lock(_mutex);
> + smp_mb();
> + atomic_cond_read_acquire(_cpu, VAL == PANIC_CPU_INVALID);
> +}
> +
> +int kexec_trylock(void) {
> + if (!mutex_trylock(_mutex)) {
> + return 0;
> + }
> + smp_mb();
> + if (atomic_read(_cpu) != PANIC_CPU_INVALID) {
> + mutex_unlock(_mutex);
> + return 0;
> + }
> + return 1;
> +}
> +
> +void kexec_unlock(void)
> +{
> + mutex_unlock(_mutex);
> +}
> +
> +int kexec_is_locked(void)
> +{
> + return mutex_is_locked(_mutex);
> +}
> +
> int kexec_should_crash(struct task_struct *p)
> {
> /*
> @@ -943,24 +980,15 @@ int kexec_load_disabled;
> */
> void __noclone __crash_kexec(struct pt_regs *regs)
> {
> - /* Take the kexec_mutex here to prevent sys_kexec_load
> - * running on one cpu from replacing the crash kernel
> - * we are using after a panic on a different cpu.
> - *
> - * If the crash kernel was not located in a fixed area
> - * of memory the xchg(_crash_image) would be
> - * sufficient. But since I reuse the memory...
> - */
> - if (mutex_trylock(_mutex)) {
> - if (kexec_crash_image) {
> - struct pt_regs fixed_regs;
> -
> - crash_setup_regs(_regs, regs);
> - crash_save_vmcoreinfo();
> - machine_crash_shutdown(_regs);
> - machine_kexec(kexec_crash_image);
> - }
> - mutex_unlock(_mutex);
> + BUG_ON(atomic_read(_cpu) != raw_smp_processor_id());
> +
> + if (!kexec_is_locked() && kexec_crash_image) {
> + struct pt_regs fixed_regs;
> +
> + crash_setup_regs(_regs, regs);
> + crash_save_vmcoreinfo();
> + machine_crash_shutdown(_regs);
> + machine_kexec(kexec_crash_image);
> }
> }
> STACK_FRAME_NON_STANDARD(__crash_kexec);
> @@ -977,9 +1005,11 @@ void crash_kexec(struct pt_regs *regs)
> this_cpu = raw_smp_processor_id();
> old_cpu = atomic_cmpxchg(_cpu, PANIC_CPU_INVALID, this_cpu);
> if (old_cpu == PANIC_CPU_INVALID) {
> - /* This is the 1st CPU which comes here, so go
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
Hi,
here is the new version of the patch based on Peters suggestion
It looks like it works fine. I added the BUG_ON to __crash_kexec,
because it is a precondition, that panic_cpu is set correctly, otherwise
the whole locking logic fails.
The mutex_trylock can still be used, because it is only in syscall
context and no interrupt context.
Jörg
---
kernel/kexec.c | 8 ++--
kernel/kexec_core.c | 86 +++--
kernel/kexec_file.c | 4 +-
kernel/kexec_internal.h | 6 ++-
4 files changed, 69 insertions(+), 35 deletions(-)
diff --git a/kernel/kexec.c b/kernel/kexec.c
index f977786fe498..118a012aeac2 100644
--- a/kernel/kexec.c
+++ b/kernel/kexec.c
@@ -255,12 +255,12 @@ SYSCALL_DEFINE4(kexec_load, unsigned long, entry,
unsigned long, nr_segments,
*
* KISS: always take the mutex.
*/
- if (!mutex_trylock(_mutex))
+ if (!kexec_trylock())
return -EBUSY;
result = do_kexec_load(entry, nr_segments, segments, flags);
- mutex_unlock(_mutex);
+ kexec_unlock();
return result;
}
@@ -309,12 +309,12 @@ COMPAT_SYSCALL_DEFINE4(kexec_load, compat_ulong_t,
entry,
*
* KISS: always take the mutex.
*/
- if (!mutex_trylock(_mutex))
+ if (!kexec_trylock())
return -EBUSY;
result = do_kexec_load(entry, nr_segments, ksegments, flags);
- mutex_unlock(_mutex);
+ kexec_unlock();
return result;
}
diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c
index c19c0dad1ebe..71682a33b1ba 100644
--- a/kernel/kexec_core.c
+++ b/kernel/kexec_core.c
@@ -45,7 +45,7 @@
#include
#include "kexec_internal.h"
-DEFINE_MUTEX(kexec_mutex);
+static DEFINE_MUTEX(kexec_mutex);
/* Per cpu memory for storing cpu states in case of system crash. */
note_buf_t __percpu *crash_notes;
@@ -70,6 +70,43 @@ struct resource crashk_low_res = {
.desc = IORES_DESC_CRASH_KERNEL
};
+void kexec_lock(void)
+{
+ /*
+ * LOCK kexec_mutex cmpxchg(_cpu, INVALID, cpu)
+ * MB MB
+ * panic_cpu == INVALID kexec_mutex == LOCKED
+ *
+ * Ensures either we observe the cmpxchg, or crash_kernel() observes
+ * our lock acquisition.
+ */
+ mutex_lock(_mutex);
+ smp_mb();
+ atomic_cond_read_acquire(_cpu, VAL == PANIC_CPU_INVALID);
+}
+
+int kexec_trylock(void) {
+ if (!mutex_trylock(_mutex)) {
+ return 0;
+ }
+ smp_mb();
+ if (atomic_read(_cpu) != PANIC_CPU_INVALID) {
+ mutex_unlock(_mutex);
+ return 0;
+ }
+ return 1;
+}
+
+void kexec_unlock(void)
+{
+ mutex_unlock(_mutex);
+}
+
+int kexec_is_locked(void)
+{
+ return mutex_is_locked(_mutex);
+}
+
int kexec_should_crash(struct task_struct *p)
{
/*
@@ -943,24 +980,15 @@ int kexec_load_disabled;
*/
void __noclone __crash_kexec(struct pt_regs *regs)
{
- /* Take the kexec_mutex here to prevent sys_kexec_load
- * running on one cpu from replacing the crash kernel
- * we are using after a panic on a different cpu.
- *
- * If the crash kernel was not located in a fixed area
- * of memory the xchg(_crash_image) would be
- * sufficient. But since I reuse the memory...
- */
- if (mutex_trylock(_mutex)) {
- if (kexec_crash_image) {
- struct pt_regs fixed_regs;
-
- crash_setup_regs(_regs, regs);
- crash_save_vmcoreinfo();
- machine_crash_shutdown(_regs);
- machine_kexec(kexec_crash_image);
- }
- mutex_unlock(_mutex);
+ BUG_ON(atomic_read(_cpu) != raw_smp_processor_id());
+
+ if (!kexec_is_locked() && kexec_crash_image) {
+ struct pt_regs fixed_regs;
+
+ crash_setup_regs(_regs, regs);
+ crash_save_vmcoreinfo();
+ machine_crash_shutdown(_regs);
+ machine_kexec(kexec_crash_image);
}
}
STACK_FRAME_NON_STANDARD(__crash_kexec);
@@ -977,9 +1005,11 @@ void crash_kexec(struct pt_regs *regs)
this_cpu = raw_smp_processor_id();
old_cpu = atomic_cmpxchg(_cpu, PANIC_CPU_INVALID, this_cpu);
if (old_cpu == PANIC_CPU_INVALID) {
- /* This is the 1st CPU which comes here, so go ahead. */
- printk_safe_flush_on_panic();
- __crash_kexec(regs);
+ if (!kexec_is_locked()) {
+ /* This is the 1st CPU which comes here, so go ahead. */
+ printk_safe_flush_on_panic();
+ __crash_kexec(regs);
+ }
/*
* Reset panic_cpu to allow another panic()/crash_kexec()
@@ -993,10 +1023,10 @@ size_t crash_get_memory_size(void)
{
size_t size = 0;
- mutex_lock(_mutex);
+ kexec_lock();
if (crashk_res.end != crashk_res.start)
size = resource_size(_res);
- mutex_unlock(_mutex);
+ kexec_unlock();
return size;
}
@@ -1016,7 +1046,7 @@ int crash_shrink_memory(unsigned long new_size)
unsigned long old_size;
struct resource *ram_res;
- mutex_lock(_mutex);
+ kexec_lock();
if
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
Hi Peter
On 9/7/2020 6:23 PM, pet...@infradead.org wrote:
According to the original comment in __crash_kexec, the mutex was used to
prevent a sys_kexec_load, while crash_kexec is executed. Your proposed patch
does not lock the mutex in crash_kexec.
Sure, but any mutex taker will (spin) wait for panic_cpu==CPU_INVALID.
And if the mutex is already held, we'll not run __crash_kexec() just
like the trylock() would do today.
Yes you are right, it should work.
This does not cover the original use
case anymore. The only thing that is protected now are two panicing cores at
the same time.
I'm not following. AFAICT it does exactly what the old code did.
Although maybe I didn't replace all kexec_mutex users, I now see that
thing isn't static.
Same thing here.
Actually, this implementation feels even more hacky to me
It's more minimal ;-) It's simpler in that it only provides the required
semantics (as I understand them) and does not attempt to implement a
more general trylock() like primitive that isn't needed.
Here I cannot agree with you. There is a second trylock in kernel_kexec,
that cannot
be protected using the panic_cpu, but it actually could still use
mutex_trylock and check
the panic_cpu. This should work I guess:
int kexec_trylock(void) {
if (!mutex_trylock(_mutex)) {
return 0;
}
smp_mb();
if (panic_cpu != PANIC_CPU_INVALID) {
mutex_unlock(_mutex);
return 0;
}
return 1;
}
Or do I miss something now? All functions protected by mutex_lock cannot
be executed, after
kexec_trylock resturned 1. kexec_crash will execute up to
mutex_is_locked and then roll back.
The only thing that can go wrong now is: kexec_trylock executes up to
smb_mb. At the same time
kexec_crash executes mutex_is_locked, which returns false now and then
before panic_cpu is reset,
kexec_trylock executes the panic_cpu check, and returns. Now both
functions did not get the lock and
nothing is executed.
Does that sound right to you? If you have no further objections I will
post it here
Jörg
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
On 07/09/20 12:41, pet...@infradead.org wrote: > So cenceptually there's the problem that idle must always be runnable, > and the moment you boost it, it becomes subject to a different > scheduling class. > > Imagine for example what happens when we boost it to RT and then have it > be subject to throttling. What are we going to run when the idle task > is no longer elegible to run. > > (it might all work out by accident, but ISTR we had a whole bunch of fun > in the earlier days of RT due to things like that) Doesn't that become a non-problem (conceptually) with proxy exec? In that case the idle task remains forever runnable, it just runs with its own scheduling context during the throttling window. Not trying to sell PE as the Billy Mays-backed solution to all kernel issues, but I think it's another thing for me to ponder on & play with.
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
On Mon, Sep 07, 2020 at 02:03:09PM +0200, Joerg Vehlow wrote: > > > On 9/7/2020 1:46 PM, pet...@infradead.org wrote: > > I think it's too complicated for that is needed, did you see my > > suggestion from a year ago? Did i miss something obvious? > > > This one? > https://lore.kernel.org/linux-fsdevel/20191219090535.gv2...@hirez.programming.kicks-ass.net/ > > I think it may be a bit incorrect? > According to the original comment in __crash_kexec, the mutex was used to > prevent a sys_kexec_load, while crash_kexec is executed. Your proposed patch > does not lock the mutex in crash_kexec. Sure, but any mutex taker will (spin) wait for panic_cpu==CPU_INVALID. And if the mutex is already held, we'll not run __crash_kexec() just like the trylock() would do today. > This does not cover the original use > case anymore. The only thing that is protected now are two panicing cores at > the same time. I'm not following. AFAICT it does exactly what the old code did. Although maybe I didn't replace all kexec_mutex users, I now see that thing isn't static. > Actually, this implementation feels even more hacky to me It's more minimal ;-) It's simpler in that it only provides the required semantics (as I understand them) and does not attempt to implement a more general trylock() like primitive that isn't needed. Also, read the kexec_lock() implementation you posted and explain to me what happens when kexec_busy is elevated. Also note the lack of confusing loops in my code.
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
On 9/7/2020 1:46 PM, pet...@infradead.org wrote: I think it's too complicated for that is needed, did you see my suggestion from a year ago? Did i miss something obvious? This one? https://lore.kernel.org/linux-fsdevel/20191219090535.gv2...@hirez.programming.kicks-ass.net/ I think it may be a bit incorrect? According to the original comment in __crash_kexec, the mutex was used to prevent a sys_kexec_load, while crash_kexec is executed. Your proposed patch does not lock the mutex in crash_kexec. This does not cover the original use case anymore. The only thing that is protected now are two panicing cores at the same time. Actually, this implementation feels even more hacky to me Jörg
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
On Mon, Sep 07, 2020 at 12:51:37PM +0200, Joerg Vehlow wrote: > Hi, > > I guess there is currently no other way than to use something like Steven > proposed. I implemented and tested the attached patch with a module, > that triggers the soft lockup detection and it works as expected. > I did not use inline functions, but normal function implemented in > kexec_core, > because there is nothing time critical here. > I also added the mutex_lock to the trylock variant, because then the unlock > function can be the same for both lock functions. > > What do you think? I think it's too complicated for that is needed, did you see my suggestion from a year ago? Did i miss something obvious?
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
On Sat, Aug 22, 2020 at 07:49:28PM -0400, Steven Rostedt wrote:
> From this email:
>
> > The problem happens when that owner is the idle task, this can happen
> > when the irq/softirq hits the idle task, in that case the contending
> > mutex_lock() will try and PI boost the idle task, and that is a big
> > no-no.
>
> What's wrong with priority boosting the idle task? It's not obvious,
> and I can't find comments in the code saying it would be bad.
> The idle task is not mentioned at all in rtmutex.c and not mentioned in
> kernel/locking except for some comments about RCU in lockdep.
There used to be a giant BUG() and comment somewhere in the PI code I
think.. but that's vage memories.
> I see that in the idle code the prio_change method does a BUG(), but
> there's no comment to say why it does so.
>
> The commit that added that BUG, doesn't explain why it can't happen:
>
> a8941d7ec8167 ("sched: Simplify the idle scheduling class")
That's like almost a decade ago ...
> I may have once known the rationale behind all this, but it's been a
> long time since I worked on the PI code, and it's out of my cache.
I suffer much the same problem.
So cenceptually there's the problem that idle must always be runnable,
and the moment you boost it, it becomes subject to a different
scheduling class.
Imagine for example what happens when we boost it to RT and then have it
be subject to throttling. What are we going to run when the idle task
is no longer elegible to run.
(it might all work out by accident, but ISTR we had a whole bunch of fun
in the earlier days of RT due to things like that)
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
Hi,
I guess there is currently no other way than to use something like Steven
proposed. I implemented and tested the attached patch with a module,
that triggers the soft lockup detection and it works as expected.
I did not use inline functions, but normal function implemented in
kexec_core,
because there is nothing time critical here.
I also added the mutex_lock to the trylock variant, because then the unlock
function can be the same for both lock functions.
What do you think?
Jörg
---
kernel/kexec.c | 8 ++---
kernel/kexec_core.c | 55 --
kernel/kexec_file.c | 6 ++--
kernel/kexec_internal.h | 6 +++-
security/integrity/ima/ima_kexec.c | 2 +-
5 files changed, 58 insertions(+), 19 deletions(-)
diff --git a/kernel/kexec.c b/kernel/kexec.c
index f977786fe498..118a012aeac2 100644
--- a/kernel/kexec.c
+++ b/kernel/kexec.c
@@ -255,12 +255,12 @@ SYSCALL_DEFINE4(kexec_load, unsigned long, entry,
unsigned long, nr_segments,
*
* KISS: always take the mutex.
*/
- if (!mutex_trylock(_mutex))
+ if (!kexec_trylock())
return -EBUSY;
result = do_kexec_load(entry, nr_segments, segments, flags);
- mutex_unlock(_mutex);
+ kexec_unlock();
return result;
}
@@ -309,12 +309,12 @@ COMPAT_SYSCALL_DEFINE4(kexec_load, compat_ulong_t,
entry,
*
* KISS: always take the mutex.
*/
- if (!mutex_trylock(_mutex))
+ if (!kexec_trylock())
return -EBUSY;
result = do_kexec_load(entry, nr_segments, ksegments, flags);
- mutex_unlock(_mutex);
+ kexec_unlock();
return result;
}
diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c
index c19c0dad1ebe..a8337e6114fb 100644
--- a/kernel/kexec_core.c
+++ b/kernel/kexec_core.c
@@ -45,7 +45,8 @@
#include
#include "kexec_internal.h"
-DEFINE_MUTEX(kexec_mutex);
+static atomic_t kexec_busy = ATOMIC_INIT(0);
+static DEFINE_MUTEX(kexec_mutex);
/* Per cpu memory for storing cpu states in case of system crash. */
note_buf_t __percpu *crash_notes;
@@ -70,6 +71,40 @@ struct resource crashk_low_res = {
.desc = IORES_DESC_CRASH_KERNEL
};
+int kexec_trylock(void)
+{
+ int ret = atomic_inc_return(_busy);
+ if (ret > 1) {
+ atomic_dec(_busy);
+ return 0;
+ }
+ mutex_lock(_mutex);
+ return 1;
+}
+
+void kexec_lock(void)
+{
+ int ret;
+ /* Prevent spinning, if the lock is busy */
+ mutex_lock(_mutex);
+ while (1) {
+ ret = atomic_inc_return(_busy);
+ if (ret > 1) {
+ atomic_dec(_busy);
+ mutex_unlock(_mutex);
+ } else {
+ break;
+ }
+ }
+}
+
+void kexec_unlock(void)
+{
+ BUG_ON(atomic_read(_busy) <= 0);
+ atomic_dec(_busy);
+ mutex_unlock(_mutex);
+}
+
int kexec_should_crash(struct task_struct *p)
{
/*
@@ -943,7 +978,7 @@ int kexec_load_disabled;
*/
void __noclone __crash_kexec(struct pt_regs *regs)
{
- /* Take the kexec_mutex here to prevent sys_kexec_load
+ /* Take the kexec_lock here to prevent sys_kexec_load
* running on one cpu from replacing the crash kernel
* we are using after a panic on a different cpu.
*
@@ -951,7 +986,7 @@ void __noclone __crash_kexec(struct pt_regs *regs)
* of memory the xchg(_crash_image) would be
* sufficient. But since I reuse the memory...
*/
- if (mutex_trylock(_mutex)) {
+ if (kexec_trylock()) {
if (kexec_crash_image) {
struct pt_regs fixed_regs;
@@ -960,7 +995,7 @@ void __noclone __crash_kexec(struct pt_regs *regs)
machine_crash_shutdown(_regs);
machine_kexec(kexec_crash_image);
}
- mutex_unlock(_mutex);
+ kexec_unlock();
}
}
STACK_FRAME_NON_STANDARD(__crash_kexec);
@@ -993,10 +1028,10 @@ size_t crash_get_memory_size(void)
{
size_t size = 0;
- mutex_lock(_mutex);
+ kexec_lock();
if (crashk_res.end != crashk_res.start)
size = resource_size(_res);
- mutex_unlock(_mutex);
+ kexec_unlock();
return size;
}
@@ -1016,7 +1051,7 @@ int crash_shrink_memory(unsigned long new_size)
unsigned long old_size;
struct resource *ram_res;
- mutex_lock(_mutex);
+ kexec_lock();
if (kexec_crash_image) {
ret = -ENOENT;
@@ -1054,7 +1089,7 @@ int crash_shrink_memory(unsigned long new_size)
insert_resource(_resource, ram_res);
unlock:
- mutex_unlock(_mutex);
+ kexec_unlock();
return ret;
}
@@ -1126,7 +1161,7 @@ int kernel_kexec(void)
{
int error = 0;
- if (!mutex_trylock(_mutex))
+ if (!kexec_trylock())
return -EBUSY;
if (!kexec_image) {
error = -EINVAL;
@@ -1203,7 +1238,7 @@ int kernel_kexec(void)
#endif
Unlock:
- mutex_unlock(_mutex);
+ kexec_unlock();
return error;
}
diff --git a/kernel/kexec_file.c b/kernel/kexec_file.c
index faa74d5f6941..52ac7573626e 100644
--- a/kernel/kexec_file.c
+++
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
On Sat, 22 Aug 2020 14:32:52 +0200
pet...@infradead.org wrote:
> On Fri, Aug 21, 2020 at 05:03:34PM -0400, Steven Rostedt wrote:
>
> > > Sigh. Is it too hard to make mutex_trylock() usable from interrupt
> > > context?
> >
> >
> > That's a question for Thomas and Peter Z.
>
> You should really know that too, the TL;DR answer is it's fundamentally
> buggered, can't work.
I knew there was an issue but I couldn't remember the reasoning, and
figured you could easily answer it without having to look back at the
code.
>
> The problem is that RT relies on being able to PI boost the mutex owner.
>
> ISTR we had a thread about all this last year or so, let me see if I can
> find that.
>
> Here goes:
>
>
> https://lkml.kernel.org/r/20191218135047.gs2...@hirez.programming.kicks-ass.net
>From this email:
> The problem happens when that owner is the idle task, this can happen
> when the irq/softirq hits the idle task, in that case the contending
> mutex_lock() will try and PI boost the idle task, and that is a big
> no-no.
What's wrong with priority boosting the idle task? It's not obvious,
and I can't find comments in the code saying it would be bad.
I looked around the code to see if I could find "why this is bad" but
couldn't find it. There's lots of places that say "Do not use
mutex_trylock in interrupt context, the implementation is not safe to
do so" but I can't find where it says "why" it is not safe to do so.
The idle task is not mentioned at all in rtmutex.c and not mentioned in
kernel/locking except for some comments about RCU in lockdep.
I see that in the idle code the prio_change method does a BUG(), but
there's no comment to say why it does so.
The commit that added that BUG, doesn't explain why it can't happen:
a8941d7ec8167 ("sched: Simplify the idle scheduling class")
I may have once known the rationale behind all this, but it's been a
long time since I worked on the PI code, and it's out of my cache.
-- Steve
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
On Fri, Aug 21, 2020 at 05:03:34PM -0400, Steven Rostedt wrote: > > Sigh. Is it too hard to make mutex_trylock() usable from interrupt > > context? > > > That's a question for Thomas and Peter Z. You should really know that too, the TL;DR answer is it's fundamentally buggered, can't work. The problem is that RT relies on being able to PI boost the mutex owner. ISTR we had a thread about all this last year or so, let me see if I can find that. Here goes: https://lkml.kernel.org/r/20191218135047.gs2...@hirez.programming.kicks-ass.net Kexec even gets mentioned: https://lkml.kernel.org/r/20191219090535.gv2...@hirez.programming.kicks-ass.net
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
On Fri, 21 Aug 2020 13:47:53 -0700
Andrew Morton wrote:
> On Fri, 21 Aug 2020 11:08:48 -0400 Steven Rostedt wrote:
>
> > On Fri, 21 Aug 2020 12:25:33 +0200
> > Joerg Vehlow wrote:
> >
> > > Hi Andrew and Others (please read at least the part with @RT developers),
> > >
> > > > Yup, mutex_trylock() from interrupt is improper. Well dang, that's a
> > > > bit silly. Presumably the 2006 spin_lock_mutex() wasn't taken with
> > > > irqs-off.
> > > >
> > > > Ho hum, did you look at switching the kexec code back to the xchg
> > > > approach?
> > > >
> > > I looked into reverting to the xchg approach, but that seems to be
> > > not a good solution anymore, because the mutex is used in many places,
> > > a lot with waiting locks and I guess that would require spinning now,
> > > if we do this with bare xchg.
> > >
> > > Instead I thought about using a spinlock, because they are supposed
> > > to be used in interrupt context as well, if I understand the documentation
> > > correctly ([1]).
> > > @RT developers
> > > Unfortunately the rt patches seem to interpret it a bit different and
> > > spin_trylock uses __rt_mutex_trylock again, with the same consequences as
> > > with the current code.
> > >
> > > I tried raw_spinlocks, but it looks like they result in a deadlock at
> > > least in the rt kernel. Thiy may be because of memory allocations in the
> > > critical sections, that are not allowed if I understand it correctly.
> > >
> > > I have no clue how to fix it at this point.
> > >
> > > Jörg
> > >
> > > [1] https://kernel.readthedocs.io/en/sphinx-samples/kernel-locking.html
> >
> > There's only two places that wait on the mutex, and all other places
> > try to get it, and if it fails, it simply exits.
> >
> > What I would do is introduce a kexec_busy counter, and have something
> > like this:
> >
> > For the two locations that actually wait on the mutex:
> >
> > loop:
> > mutex_lock(_mutex);
> > ret = atomic_inc_return(_busy);
> > if (ret > 1) {
> > /* Atomic context is busy on this counter, spin */
> > atomic_dec(_busy);
> > mutex_unlock(_mutex);
> > goto loop;
> > }
> > [..]
> > atomic_dec(_busy);
> > mutex_unlock(_mutex);
> >
> > And then all the other places that do the trylock:
> >
> > cant_sleep();
> > ret = atomic_inc_return(_busy);
> > if (ret > 1) {
> > atomic_dec(_busy);
> > return;
> > }
> > [..]
> > atomic_dec(_busy);
>
> Aw gee. Hide all this in include/linux/rostedt_lock.h...
Heh, if this was the way to go, I would have definitely recommended
packaging that up in static inline functions in some local header. Not
necessarily rostedt_lock.h, but I'll use that if people let me :-)
>
> Sigh. Is it too hard to make mutex_trylock() usable from interrupt
> context?
That's a question for Thomas and Peter Z.
-- Steve
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
On Fri, 21 Aug 2020 11:08:48 -0400 Steven Rostedt wrote:
> On Fri, 21 Aug 2020 12:25:33 +0200
> Joerg Vehlow wrote:
>
> > Hi Andrew and Others (please read at least the part with @RT developers),
> >
> > > Yup, mutex_trylock() from interrupt is improper. Well dang, that's a
> > > bit silly. Presumably the 2006 spin_lock_mutex() wasn't taken with
> > > irqs-off.
> > >
> > > Ho hum, did you look at switching the kexec code back to the xchg
> > > approach?
> > >
> > I looked into reverting to the xchg approach, but that seems to be
> > not a good solution anymore, because the mutex is used in many places,
> > a lot with waiting locks and I guess that would require spinning now,
> > if we do this with bare xchg.
> >
> > Instead I thought about using a spinlock, because they are supposed
> > to be used in interrupt context as well, if I understand the documentation
> > correctly ([1]).
> > @RT developers
> > Unfortunately the rt patches seem to interpret it a bit different and
> > spin_trylock uses __rt_mutex_trylock again, with the same consequences as
> > with the current code.
> >
> > I tried raw_spinlocks, but it looks like they result in a deadlock at
> > least in the rt kernel. Thiy may be because of memory allocations in the
> > critical sections, that are not allowed if I understand it correctly.
> >
> > I have no clue how to fix it at this point.
> >
> > Jörg
> >
> > [1] https://kernel.readthedocs.io/en/sphinx-samples/kernel-locking.html
>
> There's only two places that wait on the mutex, and all other places
> try to get it, and if it fails, it simply exits.
>
> What I would do is introduce a kexec_busy counter, and have something
> like this:
>
> For the two locations that actually wait on the mutex:
>
> loop:
> mutex_lock(_mutex);
> ret = atomic_inc_return(_busy);
> if (ret > 1) {
> /* Atomic context is busy on this counter, spin */
> atomic_dec(_busy);
> mutex_unlock(_mutex);
> goto loop;
> }
> [..]
> atomic_dec(_busy);
> mutex_unlock(_mutex);
>
> And then all the other places that do the trylock:
>
> cant_sleep();
> ret = atomic_inc_return(_busy);
> if (ret > 1) {
> atomic_dec(_busy);
> return;
> }
> [..]
> atomic_dec(_busy);
Aw gee. Hide all this in include/linux/rostedt_lock.h...
Sigh. Is it too hard to make mutex_trylock() usable from interrupt
context?
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
On Fri, 21 Aug 2020 12:25:33 +0200
Joerg Vehlow wrote:
> Hi Andrew and Others (please read at least the part with @RT developers),
>
> > Yup, mutex_trylock() from interrupt is improper. Well dang, that's a
> > bit silly. Presumably the 2006 spin_lock_mutex() wasn't taken with
> > irqs-off.
> >
> > Ho hum, did you look at switching the kexec code back to the xchg
> > approach?
> >
> I looked into reverting to the xchg approach, but that seems to be
> not a good solution anymore, because the mutex is used in many places,
> a lot with waiting locks and I guess that would require spinning now,
> if we do this with bare xchg.
>
> Instead I thought about using a spinlock, because they are supposed
> to be used in interrupt context as well, if I understand the documentation
> correctly ([1]).
> @RT developers
> Unfortunately the rt patches seem to interpret it a bit different and
> spin_trylock uses __rt_mutex_trylock again, with the same consequences as
> with the current code.
>
> I tried raw_spinlocks, but it looks like they result in a deadlock at
> least in the rt kernel. Thiy may be because of memory allocations in the
> critical sections, that are not allowed if I understand it correctly.
>
> I have no clue how to fix it at this point.
>
> Jörg
>
> [1] https://kernel.readthedocs.io/en/sphinx-samples/kernel-locking.html
There's only two places that wait on the mutex, and all other places
try to get it, and if it fails, it simply exits.
What I would do is introduce a kexec_busy counter, and have something
like this:
For the two locations that actually wait on the mutex:
loop:
mutex_lock(_mutex);
ret = atomic_inc_return(_busy);
if (ret > 1) {
/* Atomic context is busy on this counter, spin */
atomic_dec(_busy);
mutex_unlock(_mutex);
goto loop;
}
[..]
atomic_dec(_busy);
mutex_unlock(_mutex);
And then all the other places that do the trylock:
cant_sleep();
ret = atomic_inc_return(_busy);
if (ret > 1) {
atomic_dec(_busy);
return;
}
[..]
atomic_dec(_busy);
-- Steve
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
Hi Andrew and Others (please read at least the part with @RT developers), Yup, mutex_trylock() from interrupt is improper. Well dang, that's a bit silly. Presumably the 2006 spin_lock_mutex() wasn't taken with irqs-off. Ho hum, did you look at switching the kexec code back to the xchg approach? I looked into reverting to the xchg approach, but that seems to be not a good solution anymore, because the mutex is used in many places, a lot with waiting locks and I guess that would require spinning now, if we do this with bare xchg. Instead I thought about using a spinlock, because they are supposed to be used in interrupt context as well, if I understand the documentation correctly ([1]). @RT developers Unfortunately the rt patches seem to interpret it a bit different and spin_trylock uses __rt_mutex_trylock again, with the same consequences as with the current code. I tried raw_spinlocks, but it looks like they result in a deadlock at least in the rt kernel. Thiy may be because of memory allocations in the critical sections, that are not allowed if I understand it correctly. I have no clue how to fix it at this point. Jörg [1] https://kernel.readthedocs.io/en/sphinx-samples/kernel-locking.html
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
On Wed, 22 Jul 2020 06:30:53 +0200 Joerg Vehlow wrote: > >> About 12 years ago this was not implemented using a mutex, but using xchg. > >> See: 8c5a1cf0ad3ac5fcdf51314a63b16a440870f6a2 > > Yes, that commit is wrong, because mutex_trylock() is not to be taken in > > interrupt context, where crash_kexec() looks like it can be called. Yup, mutex_trylock() from interrupt is improper. Well dang, that's a bit silly. Presumably the 2006 spin_lock_mutex() wasn't taken with irqs-off. Ho hum, did you look at switching the kexec code back to the xchg approach?
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
On Wed, 22 Jul 2020 06:30:53 +0200 Joerg Vehlow wrote: > Hi Andrew, > > it's been two month now and no reaction from you. Maybe you did not see > this mail from Steven. > Please look at this issue. > Perhaps you need to send the report again without the RT (just [BUG]) to get Andrew's attention ;-) -- Steve
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
Hi Andrew, it's been two month now and no reaction from you. Maybe you did not see this mail from Steven. Please look at this issue. Greets, Jörg On 5/28/2020 2:46 PM, Steven Rostedt wrote: Hi Joerg, This does look like Andrew's commit (from 2008) is buggy (and this is a mainline bug, not an RT one). (top posting this so Andrew knows to look further ;-) On Thu, 28 May 2020 13:41:08 +0200 Joerg Vehlow wrote: Hi, I think I found a bug in the kernel with rt patches (or maybe even without). This applies to all kernels propably starting at 2.6.27. When a kernel panic is triggered from an interrupt handler, the dump-capture kernel is not started, instead the system acts as if it was not installed. The reason for this is, that panic calls __crash_kexec, which is protected by a mutex. On an rt kernel this mutex is an rt mutex and when trylock is called on an rt mutex, the first check is whether the current kthread is in an nmi or irq handler. If it is, the function just returns 0 -> locking failed. According to rt_mutex_trylock documentation, it is not allowed to call this function from an irq handler, but panic can be called from everywhere and thus rt_mutex_trylock can be called from everywhere. Actually even mutex_trylock has the comment, that it is not supposed to be used from interrupt context, but it still locks the mutex. I guess this could also be a bug in the non-rt kernel. I found this problem using a test module, that triggers the softlock detection. It is a pretty simple module, that creates a kthread, that disables preemption, spins 60 seconds in an endless loop and then reenables preemption and terminates the thread. This reliably triggers the softlock detection and if kernel.softlockup_panic=0, the system resumes perfectly fine afterwards. If kernel.softlockup_panic=1 I would expect the dump-capture kernel to be executed, but it is not due to the bug (without rt patches it works), instead the panic function is executed until the end to the endless loop. A stacktrace captured at the trylock call inside kexec_code looks like this: #0 __rt_mutex_trylock (lock=0x81701aa0 ) at /usr/src/kernel/kernel/locking/rtmutex.c:2110 #1 0x8087601a in _mutex_trylock (lock=) at /usr/src/kernel/kernel/locking/mutex-rt.c:185 #2 0x803022a0 in __crash_kexec (regs=0x0 ) at /usr/src/kernel/kernel/kexec_core.c:941 #3 0x8027af59 in panic (fmt=0x80fa3d66 "softlockup: hung tasks") at /usr/src/kernel/kernel/panic.c:198 #4 0x80325b6d in watchdog_timer_fn (hrtimer=) at /usr/src/kernel/kernel/watchdog.c:464 #5 0x802e6b90 in __run_hrtimer (flags=, now=, timer=, base=, cpu_base=) at /usr/src/kernel/kernel/time/hrtimer.c:1417 #6 __hrtimer_run_queues (cpu_base=0x88807db1c000, now=, flags=, active_mask=) at /usr/src/kernel/kernel/time/hrtimer.c:1479 #7 0x802e7704 in hrtimer_interrupt (dev=) at /usr/src/kernel/kernel/time/hrtimer.c:1539 #8 0x80a020f2 in local_apic_timer_interrupt () at /usr/src/kernel/arch/x86/kernel/apic/apic.c:1067 #9 smp_apic_timer_interrupt (regs=) at /usr/src/kernel/arch/x86/kernel/apic/apic.c:1092 #10 0x80a015df in apic_timer_interrupt () at /usr/src/kernel/arch/x86/entry/entry_64.S:909 Obviously and as expected the panic was triggered in the context of the apic interrupt. So in_irq() is true and trylock fails. About 12 years ago this was not implemented using a mutex, but using xchg. See: 8c5a1cf0ad3ac5fcdf51314a63b16a440870f6a2 Yes, that commit is wrong, because mutex_trylock() is not to be taken in interrupt context, where crash_kexec() looks like it can be called. Unless back then crash_kexec() wasn't called in interrupt context, then the commit that calls it from that combined with this commit is the issue. -- Steve Since my knowledege about mutexes inside the kernel is very limited, I do not know how this can be fixed and whether it should be fixed in the rt patches or if this really is a bug in mainline kernel (because trylock is also not allowed to be used in interrupt handlers. Jörg
Re: [BUG RT] dump-capture kernel not executed for panic in interrupt context
Hi Joerg, This does look like Andrew's commit (from 2008) is buggy (and this is a mainline bug, not an RT one). (top posting this so Andrew knows to look further ;-) On Thu, 28 May 2020 13:41:08 +0200 Joerg Vehlow wrote: > Hi, > > I think I found a bug in the kernel with rt patches (or maybe even without). > This applies to all kernels propably starting at 2.6.27. > > When a kernel panic is triggered from an interrupt handler, the dump-capture > kernel is not started, instead the system acts as if it was not installed. > The reason for this is, that panic calls __crash_kexec, which is protected > by a mutex. On an rt kernel this mutex is an rt mutex and when trylock > is called > on an rt mutex, the first check is whether the current kthread is in an > nmi or > irq handler. If it is, the function just returns 0 -> locking failed. > > According to rt_mutex_trylock documentation, it is not allowed to call this > function from an irq handler, but panic can be called from everywhere > and thus > rt_mutex_trylock can be called from everywhere. Actually even > mutex_trylock has > the comment, that it is not supposed to be used from interrupt context, > but it > still locks the mutex. I guess this could also be a bug in the non-rt > kernel. > > I found this problem using a test module, that triggers the softlock > detection. > It is a pretty simple module, that creates a kthread, that disables > preemption, > spins 60 seconds in an endless loop and then reenables preemption and > terminates > the thread. This reliably triggers the softlock detection and if > kernel.softlockup_panic=0, the system resumes perfectly fine afterwards. If > kernel.softlockup_panic=1 I would expect the dump-capture kernel to be > executed, > but it is not due to the bug (without rt patches it works), instead the > panic > function is executed until the end to the endless loop. > > > A stacktrace captured at the trylock call inside kexec_code looks like this: > #0 __rt_mutex_trylock (lock=0x81701aa0 ) at > /usr/src/kernel/kernel/locking/rtmutex.c:2110 > #1 0x8087601a in _mutex_trylock (lock=) at > /usr/src/kernel/kernel/locking/mutex-rt.c:185 > #2 0x803022a0 in __crash_kexec (regs=0x0 ) at > /usr/src/kernel/kernel/kexec_core.c:941 > #3 0x8027af59 in panic (fmt=0x80fa3d66 "softlockup: > hung tasks") at /usr/src/kernel/kernel/panic.c:198 > #4 0x80325b6d in watchdog_timer_fn (hrtimer=) at > /usr/src/kernel/kernel/watchdog.c:464 > #5 0x802e6b90 in __run_hrtimer (flags=, > now=, timer=, base=, > cpu_base=) at /usr/src/kernel/kernel/time/hrtimer.c:1417 > #6 __hrtimer_run_queues (cpu_base=0x88807db1c000, now= out>, flags=, active_mask=) at > /usr/src/kernel/kernel/time/hrtimer.c:1479 > #7 0x802e7704 in hrtimer_interrupt (dev=) at > /usr/src/kernel/kernel/time/hrtimer.c:1539 > #8 0x80a020f2 in local_apic_timer_interrupt () at > /usr/src/kernel/arch/x86/kernel/apic/apic.c:1067 > #9 smp_apic_timer_interrupt (regs=) at > /usr/src/kernel/arch/x86/kernel/apic/apic.c:1092 > #10 0x80a015df in apic_timer_interrupt () at > /usr/src/kernel/arch/x86/entry/entry_64.S:909 > > > Obviously and as expected the panic was triggered in the context of the apic > interrupt. So in_irq() is true and trylock fails. > > > About 12 years ago this was not implemented using a mutex, but using xchg. > See: 8c5a1cf0ad3ac5fcdf51314a63b16a440870f6a2 Yes, that commit is wrong, because mutex_trylock() is not to be taken in interrupt context, where crash_kexec() looks like it can be called. Unless back then crash_kexec() wasn't called in interrupt context, then the commit that calls it from that combined with this commit is the issue. -- Steve > > > Since my knowledege about mutexes inside the kernel is very limited, I > do not > know how this can be fixed and whether it should be fixed in the rt > patches or > if this really is a bug in mainline kernel (because trylock is also not > allowed > to be used in interrupt handlers. > > > Jörg
