RE: Cobalt Preemption of kernel update_fast_timekeeper can cause deadlocks

[Lange Norbert via Xenomai]

> >>
> >> If you are calling into an "unknown" non-RT blob, dropping from RT
> >> may actually be required. We do not promote explicit mode switches
> >> because they are not needed if you control (wrap) all your code. This
> >> might be  an exception.
> >
> > The non-RT "blob" is the regular linux rootfs in my case, ie.
> > libstdc++ and I plan to use libnttg-ust and stuff like xml parsers.
>
> That's all fine - as long as you are not in RT context.

To make sure I am not in RT Context needs alot knowledge of callstacks.
Easy to do if you create the code from scratch, not so easy if you are porting.

>
> Actually, if you use a SCHED_WEAK thread for calling into both RT and non-
> RT, you will not have to do the explicit switching because those threads fall
> back to non-RT as soon as they have no RT business (lock ownership or
> blocking) anymore, and then you are safe.

Thanks for clearing up another misunderstanding.

> >>
>
> >>
> >> Irrespective of that, I would definitely be interested in a
> >> LD_PRELOAD-based checker that you can attach to an application
> >> easily, without the need to switch to link-time wrapping (which is not
> needed with non-posix skins).
> >
> > If you don’t know lttng-ust, you could spend a hour or two playing
> > with it, Eg you can interpose and trace any malloc/free by just preloading
> the wrapper:
> > LD_PRELOAD=liblttng-ust-libc-wrapper your_app
> >
> > This could help with non-posix skins mixing with dangerous other functions
> aswell.
>
> It takes more than that if you look at how we decide whether to raise an
> alarm or not (context detection, warning flag evaluation, signal raising). 
> lttng-
> ust can be a nice tracing tool, but for a runtime equivalent to --mode-check, 
> I
> would rather set up a tool that behaves like the link-time version.

It does not take a lot, I had a stab at writing a preload checker for 
clock_gettime that bugged me a long time (most time was spent figuring out I 
have to enable PTHREAD_WARNSW for anything to happen).
Most function could be done similarly. Malloc/free are tricky as the dl loading 
might call those functions,
Packages like lttng already solved those issues, which was the point I was 
trying to make.

With such checkers you can find issues related to external DSOs, something the 
linker tricks won't be able to.

Norbert


This message and any attachments are solely for the use of the intended 
recipients. They may contain privileged and/or confidential information or 
other information protected from disclosure. If you are not an intended 
recipient, you are hereby notified that you received this email in error and 
that any review, dissemination, distribution or copying of this email and any 
attachment is strictly prohibited. If you have received this email in error, 
please contact the sender and delete the message and any attachment from your 
system.

ANDRITZ HYDRO GmbH


Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation

Firmensitz/ Registered seat: Wien

Firmenbuchgericht/ Court of registry: Handelsgericht Wien

Firmenbuchnummer/ Company registration: FN 61833 g

DVR: 0605077

UID-Nr.: ATU14756806


Thank You

-- next part --
An embedded and charset-unspecified text was scrubbed...
Name: cobalt_preload_helper.c
URL: 

RE: Cobalt Preemption of kernel update_fast_timekeeper can cause deadlocks

[Lange Norbert via Xenomai]

> -Original Message-
> From: Jan Kiszka 
> Sent: Donnerstag, 20. Dezember 2018 14:33
> To: Lange Norbert ; Xenomai
> (xenomai@xenomai.org) 
> Subject: Re: Cobalt Preemption of kernel update_fast_timekeeper can cause
> deadlocks
>
> E-MAIL FROM A NON-ANDRITZ SOURCE: AS A SECURITY MEASURE, PLEASE
> EXERCISE CAUTION WITH E-MAIL CONTENT AND ANY LINKS OR
> ATTACHMENTS.
>
>
> On 20.12.18 13:29, Lange Norbert via Xenomai wrote:
> >> On 19.12.18 19:26, Auel, Kendall via Xenomai wrote:
> >>> Jan,
> >>>
> >>> I'm very much in favor of providing a way to prevent Xenomai modules
> >> from using features which can result in deadlock, if there is a clean
> >> way to detect such a situation.
> >>>
> >>> We used gettimeofday in one of our modules and it mostly worked
> great.
> >> But once in a great while the system would deadlock. Most calls to
> >> gettimeofday are benign and appear to work normally, which is why it
> >> is especially problematic. It would have saved some debug cycles if
> >> there was a kernel log message to warn us of our danger.
> >>>
> >>> Or perhaps we could collect a blacklist of references which will
> >>> produce
> >> warnings when linking a Xenomai module. All of these things are 'nice
> >> to have' but certainly not urgent matters.
> >>
> >> We do have the infrastructure and a small use case for such RT traps
> already:
> >> If you use --mode-check on xeno-config, any usage of malloc and free
> >> from RT contexts will be detected and reported. These calls are evil
> >> as well because they tend no not trigger a syscall in the fast path
> >> and only fail on contention or empty-pool situations of the userspace
> allocator.
> >
> > There is still the issue that the cobald kernel can interrupt the
> > linux kernel while holding a lock.
> > Consider the case that you have a 4 core CPU, several cobalt threads are
> bound to eg. Core 0 (legacy code assuming single core).
> >
> > 1) linux wants to update the timekeeper struct
> > 2) now cobalt preempts the linux kernel while holding the lock on Core
> > 0
> > 3) the cobalt threads run close to each other and thus Core 0 remains in
> cobalt domain for hundreds of ms.
> > 4) finally all cobalt threads (that are bound to core 0) idle and
> > linux can free the lock
> >
> > This means that all Linux threads on *any core* that try to call some
> *gettime functions (possible others) will busywait on the lock.
> >
>
> You do not need to look at the GTOD lock to construct such delays: every
> Linux spinlock taken on one core that is then interrupted by RT workload for
> a longer period can delay other cores doing Linux stuff that needs that lock.
> That is a generic property of the co-kernel architecture - and the reason you
> should allow Linux to run every few ms, on *every* core.

You are right, I did not realize that.
Userspace usually does not spinlock, so I consider those functions a lot more 
critical,
clock_gettime is also heavily used (especially for tracing).

Funny enough, the linux x86 vdso handles clock_gettime(CLOCK_MONOTONIC) but not 
clock_gettime(CLOCK_MONOTONIC_RAW).
Seems the common denominator would be to use rdtsc directly =/
(I know about the pitfalls, but our hardware should have a stable, invariant 
tsc)

>
> > That a rt thread (potentially just temporary promoted non-rt thread, or not
> lazily demoted yet) can additionally deadlock the system sits just on top of
> this issue.
> >
> > Regarding to what I am allowed to do:
> > AFAIK a thread started as cobalt thread can freely switch between
> domains, typically around syscalls and the switches are "lazy". What are the
> rules for a thread that needs to collect some data RT (potentially using some
> RT Mutexes with prio inheritance) calling into DSOs that aren’t compiled with
> the "cobalt wrappings" active (say a logging framework that uses libcs
> clock_gettime).
> > Do I manually have to demote the thread somehow before calling DSO
> functions, is it not allowed at all to use DSOs that were compiled with 
> "cobalt
> wrappings"?
>
> If you are calling into an "unknown" non-RT blob, dropping from RT may
> actually be required. We do not promote explicit mode switches because
> they are not needed if you control (wrap) all your code. This might be  an
> exception.

The non-RT "blob" is the regular linux rootfs in my case, ie. libstdc++ and I 
plan
to use libnttg-ust and stuff like xml parsers.
I understand this as motivation to actually *have* the POSIX

Re: Cobalt Preemption of kernel update_fast_timekeeper can cause deadlocks

[Jan Kiszka via Xenomai]

On 20.12.18 13:29, Lange Norbert via Xenomai wrote:

On 19.12.18 19:26, Auel, Kendall via Xenomai wrote:

Jan,

I'm very much in favor of providing a way to prevent Xenomai modules

from using features which can result in deadlock, if there is a clean way to
detect such a situation.


We used gettimeofday in one of our modules and it mostly worked great.

But once in a great while the system would deadlock. Most calls to
gettimeofday are benign and appear to work normally, which is why it is
especially problematic. It would have saved some debug cycles if there was a
kernel log message to warn us of our danger.


Or perhaps we could collect a blacklist of references which will produce

warnings when linking a Xenomai module. All of these things are 'nice to
have' but certainly not urgent matters.

We do have the infrastructure and a small use case for such RT traps already:
If you use --mode-check on xeno-config, any usage of malloc and free from
RT contexts will be detected and reported. These calls are evil as well
because they tend no not trigger a syscall in the fast path and only fail on
contention or empty-pool situations of the userspace allocator.


There is still the issue that the cobald kernel can interrupt the linux kernel
while holding a lock.
Consider the case that you have a 4 core CPU, several cobalt threads are bound 
to eg. Core 0 (legacy code assuming single core).

1) linux wants to update the timekeeper struct
2) now cobalt preempts the linux kernel while holding the lock on Core 0
3) the cobalt threads run close to each other and thus Core 0 remains in cobalt 
domain for hundreds of ms.
4) finally all cobalt threads (that are bound to core 0) idle and linux can 
free the lock

This means that all Linux threads on *any core* that try to call some *gettime 
functions (possible others) will busywait on the lock.



You do not need to look at the GTOD lock to construct such delays: every Linux 
spinlock taken on one core that is then interrupted by RT workload for a longer 
period can delay other cores doing Linux stuff that needs that lock. That is a 
generic property of the co-kernel architecture - and the reason you should allow 
Linux to run every few ms, on *every* core.



That a rt thread (potentially just temporary promoted non-rt thread, or not 
lazily demoted yet) can additionally deadlock the system sits just on top of 
this issue.

Regarding to what I am allowed to do:
AFAIK a thread started as cobalt thread can freely switch between domains, typically around 
syscalls and the switches are "lazy". What are the rules for a thread that needs to 
collect some data RT (potentially using some RT Mutexes with prio inheritance) calling into DSOs 
that aren’t compiled with the "cobalt wrappings" active (say a logging framework that 
uses libcs clock_gettime).
Do I manually have to demote the thread somehow before calling DSO functions, is it not 
allowed at all to use DSOs that were compiled with "cobalt wrappings"?


If you are calling into an "unknown" non-RT blob, dropping from RT may actually 
be required. We do not promote explicit mode switches because they are not 
needed if you control (wrap) all your code. This might be  an exception.





with posix, you are already
redirected to the RT-safe implementations of those functions.


In my case (posix skin, not "native" as I replied earlier), the call came from 
another DSO which is unaffected by the
link-time wrapping.
I would likely have to LD_PRELOAD a checker DSO, seems more sane to me,
as the calls could originate from implicitly linked DSO aswell (C++ runtime 
library)


Is the reason that the other DSOs are not caught at link-time generic or 
specific to your build? The former case should be documented if it exists.


Irrespective of that, I would definitely be interested in a LD_PRELOAD-based 
checker that you can attach to an application easily, without the need to switch 
to link-time wrapping (which is not needed with non-posix skins).


Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

RE: Cobalt Preemption of kernel update_fast_timekeeper can cause deadlocks

[Lange Norbert via Xenomai]

> On 19.12.18 19:26, Auel, Kendall via Xenomai wrote:
> > Jan,
> >
> > I'm very much in favor of providing a way to prevent Xenomai modules
> from using features which can result in deadlock, if there is a clean way to
> detect such a situation.
> >
> > We used gettimeofday in one of our modules and it mostly worked great.
> But once in a great while the system would deadlock. Most calls to
> gettimeofday are benign and appear to work normally, which is why it is
> especially problematic. It would have saved some debug cycles if there was a
> kernel log message to warn us of our danger.
> >
> > Or perhaps we could collect a blacklist of references which will produce
> warnings when linking a Xenomai module. All of these things are 'nice to
> have' but certainly not urgent matters.
>
> We do have the infrastructure and a small use case for such RT traps already:
> If you use --mode-check on xeno-config, any usage of malloc and free from
> RT contexts will be detected and reported. These calls are evil as well
> because they tend no not trigger a syscall in the fast path and only fail on
> contention or empty-pool situations of the userspace allocator.

There is still the issue that the cobald kernel can interrupt the linux kernel
while holding a lock.
Consider the case that you have a 4 core CPU, several cobalt threads are bound 
to eg. Core 0 (legacy code assuming single core).

1) linux wants to update the timekeeper struct
2) now cobalt preempts the linux kernel while holding the lock on Core 0
3) the cobalt threads run close to each other and thus Core 0 remains in cobalt 
domain for hundreds of ms.
4) finally all cobalt threads (that are bound to core 0) idle and linux can 
free the lock

This means that all Linux threads on *any core* that try to call some *gettime 
functions (possible others) will busywait on the lock.

That a rt thread (potentially just temporary promoted non-rt thread, or not 
lazily demoted yet) can additionally deadlock the system sits just on top of 
this issue.

Regarding to what I am allowed to do:
AFAIK a thread started as cobalt thread can freely switch between domains, 
typically around syscalls and the switches are "lazy". What are the rules for a 
thread that needs to collect some data RT (potentially using some RT Mutexes 
with prio inheritance) calling into DSOs that aren’t compiled with the "cobalt 
wrappings" active (say a logging framework that uses libcs clock_gettime).
Do I manually have to demote the thread somehow before calling DSO functions, 
is it not allowed at all to use DSOs that were compiled with "cobalt wrappings"?

> with posix, you are already
> redirected to the RT-safe implementations of those functions.

In my case (posix skin, not "native" as I replied earlier), the call came from 
another DSO which is unaffected by the
link-time wrapping.
I would likely have to LD_PRELOAD a checker DSO, seems more sane to me,
as the calls could originate from implicitly linked DSO aswell (C++ runtime 
library)

Norbert


This message and any attachments are solely for the use of the intended 
recipients. They may contain privileged and/or confidential information or 
other information protected from disclosure. If you are not an intended 
recipient, you are hereby notified that you received this email in error and 
that any review, dissemination, distribution or copying of this email and any 
attachment is strictly prohibited. If you have received this email in error, 
please contact the sender and delete the message and any attachment from your 
system.

ANDRITZ HYDRO GmbH


Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation

Firmensitz/ Registered seat: Wien

Firmenbuchgericht/ Court of registry: Handelsgericht Wien

Firmenbuchnummer/ Company registration: FN 61833 g

DVR: 0605077

UID-Nr.: ATU14756806


Thank You

Re: Cobalt Preemption of kernel update_fast_timekeeper can cause deadlocks

[Jan Kiszka via Xenomai]

On 19.12.18 19:26, Auel, Kendall via Xenomai wrote:

Jan,

I'm very much in favor of providing a way to prevent Xenomai modules from using 
features which can result in deadlock, if there is a clean way to detect such a 
situation.

We used gettimeofday in one of our modules and it mostly worked great. But once 
in a great while the system would deadlock. Most calls to gettimeofday are 
benign and appear to work normally, which is why it is especially problematic. 
It would have saved some debug cycles if there was a kernel log message to warn 
us of our danger.

Or perhaps we could collect a blacklist of references which will produce 
warnings when linking a Xenomai module. All of these things are 'nice to have' 
but certainly not urgent matters.


We do have the infrastructure and a small use case for such RT traps already: If 
you use --mode-check on xeno-config, any usage of malloc and free from RT 
contexts will be detected and reported. These calls are evil as well because 
they tend no not trigger a syscall in the fast path and only fail on contention 
or empty-pool situations of the userspace allocator.


We could extend that mechanism for gettimeofday & Co. checks, but we need to 
limit that to non-posix applications: with posix, you are already redirected to 
the RT-safe implementations of those functions.


Patches welcome.

Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

RE: Cobalt Preemption of kernel update_fast_timekeeper can cause deadlocks

[Auel, Kendall via Xenomai]

Jan,

I'm very much in favor of providing a way to prevent Xenomai modules from using 
features which can result in deadlock, if there is a clean way to detect such a 
situation.

We used gettimeofday in one of our modules and it mostly worked great. But once 
in a great while the system would deadlock. Most calls to gettimeofday are 
benign and appear to work normally, which is why it is especially problematic. 
It would have saved some debug cycles if there was a kernel log message to warn 
us of our danger.

Or perhaps we could collect a blacklist of references which will produce 
warnings when linking a Xenomai module. All of these things are 'nice to have' 
but certainly not urgent matters.

Regards,
Kendall

> -Original Message-
> From: Xenomai  On Behalf Of Jan Kiszka
> via Xenomai
> Sent: Wednesday, December 19, 2018 4:45 AM
> To: Philippe Gerum ; Lange Norbert
> ; Xenomai (xenomai@xenomai.org)
> 
> Subject: Re: Cobalt Preemption of kernel update_fast_timekeeper can cause
> deadlocks
> 
> On 19.12.18 13:09, Philippe Gerum via Xenomai wrote:
> > On 12/19/18 11:20 AM, Lange Norbert via Xenomai wrote:
> >> There is a deadlock issue that haunted me for several weeks, it is
> >> caused by the kernels update of the user-visible timekeeping
> >> structures used by the VDSO clock_gettime functions.
> >>
> >> The kernel regularly updates a Timestamp structure, which is
> >> accessible in user-mode, it does so by something akin to a rw-lock in
> update_fast_timekeeper.
> >>
> >> If cobalt preempts the core during holding the lock, any thread
> >> trying to read the time will continue to spin. (This alone is an issue 
> >> IMHO).
> >> If the cobalt thread itself now call the vDSO function as reader, it
> >> will spin on the lock and block the lock from getting released.
> >>
> >>
> >> Either the update_fast_timekeeper funtion should not be preemptible
> >> by cobalt, or the spin-lock on reading could fallback to the syscall after 
> >> a
> certain amount of retries.
> >>
> >> The later is probably easier to implement, but then could randomly
> demote cobalt threads.
> >> (on the other hand, this would be always a demotion on platforms
> >> without the vdso function)
> >>
> >
> > update_vsyscall() is locking the write-side. If the analysis is correct, 
> > this
> patch may help at the expense of a some cycles more spent uninterruptible:
> >
> > diff --git a/arch/x86/entry/vsyscall/vsyscall_gtod.c
> > b/arch/x86/entry/vsyscall/vsyscall_gtod.c
> > index 9fb89b6e88c3..e9baa57e8385 100644
> > --- a/arch/x86/entry/vsyscall/vsyscall_gtod.c
> > +++ b/arch/x86/entry/vsyscall/vsyscall_gtod.c
> > @@ -32,11 +32,14 @@ void update_vsyscall(struct timekeeper *tk)
> >   {
> > int vclock_mode = tk->tkr_mono.clock->archdata.vclock_mode;
> > struct vsyscall_gtod_data *vdata = _gtod_data;
> > +   unsigned long flags;
> >
> > /* Mark the new vclock used. */
> > BUILD_BUG_ON(VCLOCK_MAX >= 32);
> > WRITE_ONCE(vclocks_used, READ_ONCE(vclocks_used) | (1 <<
> > vclock_mode));
> >
> > +   flags = hard_cond_local_irq_save();
> > +
> > gtod_write_begin(vdata);
> >
> > /* copy vsyscall data */
> > @@ -77,6 +80,8 @@ void update_vsyscall(struct timekeeper *tk)
> >
> > gtod_write_end(vdata);
> >
> > +   hard_cond_local_irq_restore(flags);
> > +
> > if (tk->tkr_mono.clock == _tsc)
> > ipipe_update_hostrt(tk);
> >   }
> >
> 
> This should rather be an application bug: An RT (Xenomai) thread is
> apparently using Linux gettimeofday & Co. (glibc) from RT context. That was
> never supported, we rather have RT services for that
> (CLOCK_HOST_REALTIME).
> 
> We may think about detecting such cases better, though. Norbert, are you
> using native/alchemy APIs?
> 
> Jan
> 
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate
> Competence Center Embedded Linux

Re: Cobalt Preemption of kernel update_fast_timekeeper can cause deadlocks

[Philippe Gerum via Xenomai]

On 12/19/18 1:44 PM, Jan Kiszka wrote:
> On 19.12.18 13:09, Philippe Gerum via Xenomai wrote:
>> On 12/19/18 11:20 AM, Lange Norbert via Xenomai wrote:
>>> There is a deadlock issue that haunted me for several weeks,
>>> it is caused by the kernels update of the user-visible
>>> timekeeping structures used by the VDSO clock_gettime functions.
>>>
>>> The kernel regularly updates a Timestamp structure, which is
>>> accessible in user-mode,
>>> it does so by something akin to a rw-lock in update_fast_timekeeper.
>>>
>>> If cobalt preempts the core during holding the lock, any thread
>>> trying to read the time
>>> will continue to spin. (This alone is an issue IMHO).
>>> If the cobalt thread itself now call the vDSO function as reader,
>>> it will spin on the lock and block the lock from getting released.
>>>
>>>
>>> Either the update_fast_timekeeper funtion should not be preemptible
>>> by cobalt,
>>> or the spin-lock on reading could fallback to the syscall after a
>>> certain amount of retries.
>>>
>>> The later is probably easier to implement, but then could randomly
>>> demote cobalt threads.
>>> (on the other hand, this would be always a demotion on platforms
>>> without the vdso function)
>>>
>>
>> update_vsyscall() is locking the write-side. If the analysis is
>> correct, this patch may help at the expense of a some cycles more
>> spent uninterruptible:
>>
>> diff --git a/arch/x86/entry/vsyscall/vsyscall_gtod.c
>> b/arch/x86/entry/vsyscall/vsyscall_gtod.c
>> index 9fb89b6e88c3..e9baa57e8385 100644
>> --- a/arch/x86/entry/vsyscall/vsyscall_gtod.c
>> +++ b/arch/x86/entry/vsyscall/vsyscall_gtod.c
>> @@ -32,11 +32,14 @@ void update_vsyscall(struct timekeeper *tk)
>>   {
>>   int vclock_mode = tk->tkr_mono.clock->archdata.vclock_mode;
>>   struct vsyscall_gtod_data *vdata = _gtod_data;
>> +    unsigned long flags;
>>     /* Mark the new vclock used. */
>>   BUILD_BUG_ON(VCLOCK_MAX >= 32);
>>   WRITE_ONCE(vclocks_used, READ_ONCE(vclocks_used) | (1 <<
>> vclock_mode));
>>   +    flags = hard_cond_local_irq_save();
>> +
>>   gtod_write_begin(vdata);
>>     /* copy vsyscall data */
>> @@ -77,6 +80,8 @@ void update_vsyscall(struct timekeeper *tk)
>>     gtod_write_end(vdata);
>>   +    hard_cond_local_irq_restore(flags);
>> +
>>   if (tk->tkr_mono.clock == _tsc)
>>   ipipe_update_hostrt(tk);
>>   }
>>
> 
> This should rather be an application bug: An RT (Xenomai) thread is
> apparently using Linux gettimeofday & Co. (glibc) from RT context.

Yes, it definitely is. But the issue is with coping with legacy.
CLOCK_HOST_REALTIME is indeed the best way around this.

 That
> was never supported, we rather have RT services for that
> (CLOCK_HOST_REALTIME).
> 
> We may think about detecting such cases better, though. Norbert, are you
> using native/alchemy APIs?
> 


-- 
Philippe.

RE: Cobalt Preemption of kernel update_fast_timekeeper can cause deadlocks

[Lange Norbert via Xenomai]

> -Original Message-
> From: Jan Kiszka 
> Sent: Mittwoch, 19. Dezember 2018 13:45
> To: Philippe Gerum ; Lange Norbert
> ; Xenomai (xenomai@xenomai.org)
> 
> Subject: Re: Cobalt Preemption of kernel update_fast_timekeeper can cause
> deadlocks
>
> E-MAIL FROM A NON-ANDRITZ SOURCE: AS A SECURITY MEASURE, PLEASE
> EXERCISE CAUTION WITH E-MAIL CONTENT AND ANY LINKS OR
> ATTACHMENTS.
>
>
> On 19.12.18 13:09, Philippe Gerum via Xenomai wrote:
> > On 12/19/18 11:20 AM, Lange Norbert via Xenomai wrote:
> >> There is a deadlock issue that haunted me for several weeks, it is
> >> caused by the kernels update of the user-visible timekeeping
> >> structures used by the VDSO clock_gettime functions.
> >>
> >> The kernel regularly updates a Timestamp structure, which is
> >> accessible in user-mode, it does so by something akin to a rw-lock in
> update_fast_timekeeper.
> >>
> >> If cobalt preempts the core during holding the lock, any thread
> >> trying to read the time will continue to spin. (This alone is an issue 
> >> IMHO).
> >> If the cobalt thread itself now call the vDSO function as reader, it
> >> will spin on the lock and block the lock from getting released.
> >>
> >>
> >> Either the update_fast_timekeeper funtion should not be preemptible
> >> by cobalt, or the spin-lock on reading could fallback to the syscall after 
> >> a
> certain amount of retries.
> >>
> >> The later is probably easier to implement, but then could randomly
> demote cobalt threads.
> >> (on the other hand, this would be always a demotion on platforms
> >> without the vdso function)
> >>
> >
> > update_vsyscall() is locking the write-side. If the analysis is correct, 
> > this
> patch may help at the expense of a some cycles more spent uninterruptible:
> >
> > diff --git a/arch/x86/entry/vsyscall/vsyscall_gtod.c
> > b/arch/x86/entry/vsyscall/vsyscall_gtod.c
> > index 9fb89b6e88c3..e9baa57e8385 100644
> > --- a/arch/x86/entry/vsyscall/vsyscall_gtod.c
> > +++ b/arch/x86/entry/vsyscall/vsyscall_gtod.c
> > @@ -32,11 +32,14 @@ void update_vsyscall(struct timekeeper *tk)
> >   {
> >   int vclock_mode = tk->tkr_mono.clock->archdata.vclock_mode;
> >   struct vsyscall_gtod_data *vdata = _gtod_data;
> > + unsigned long flags;
> >
> >   /* Mark the new vclock used. */
> >   BUILD_BUG_ON(VCLOCK_MAX >= 32);
> >   WRITE_ONCE(vclocks_used, READ_ONCE(vclocks_used) | (1 <<
> > vclock_mode));
> >
> > + flags = hard_cond_local_irq_save();
> > +
> >   gtod_write_begin(vdata);
> >
> >   /* copy vsyscall data */
> > @@ -77,6 +80,8 @@ void update_vsyscall(struct timekeeper *tk)
> >
> >   gtod_write_end(vdata);
> >
> > + hard_cond_local_irq_restore(flags);
> > +
> >   if (tk->tkr_mono.clock == _tsc)
> >   ipipe_update_hostrt(tk);
> >   }
> >
>
> This should rather be an application bug: An RT (Xenomai) thread is
> apparently using Linux gettimeofday & Co. (glibc) from RT context. That was
> never supported, we rather have RT services for that
> (CLOCK_HOST_REALTIME).

Any RT thread can preempt the kernel holding the write-lock. A deadlock only 
occurs if
*that* thread then tries to read-lock - true.
Keeping any number of linux threads in the reader "spin-lock" for the duration 
of the cobald mode will always happen
(I assume update_vsyscall is called regularly)

>
> We may think about detecting such cases better, though. Norbert, are you
> using native/alchemy APIs?

Native.
Btw. I thought mixing APIs is explicitly supported and one of the main features 
of Xenomai (like reading state, synchronized with RT mutexes and logging it to 
the filesystem), the existence of prio-inheritance would strongly imply that 
aswell. Otherwise, it's rather hard to guess which thread runs in the cobalt 
context, particularly it if might be just a temporal priority boost.
Compiling your own code with the "cobalt wrappers" is no issue, the deadlock 
above was caused by
Indirectly by another "linux" DSO (libstdc++).

Norbert


This message and any attachments are solely for the use of the intended 
recipients. They may contain privileged and/or confidential information or 
other information protected from disclosure. If you are not an intended 
recipient, you are hereby notified that you received this email in error and 
that any review, dissemination, distribution or copying of this email and any 
attachment is strictly prohibited. If you have received this email in error, 
please contact the sender and delete the message and any attachment from your 
system.

ANDRITZ HYDRO GmbH


Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation

Firmensitz/ Registered seat: Wien

Firmenbuchgericht/ Court of registry: Handelsgericht Wien

Firmenbuchnummer/ Company registration: FN 61833 g

DVR: 0605077

UID-Nr.: ATU14756806


Thank You

Re: Cobalt Preemption of kernel update_fast_timekeeper can cause deadlocks

[Jan Kiszka via Xenomai]

On 19.12.18 13:09, Philippe Gerum via Xenomai wrote:

On 12/19/18 11:20 AM, Lange Norbert via Xenomai wrote:

There is a deadlock issue that haunted me for several weeks,
it is caused by the kernels update of the user-visible
timekeeping structures used by the VDSO clock_gettime functions.

The kernel regularly updates a Timestamp structure, which is accessible in 
user-mode,
it does so by something akin to a rw-lock in update_fast_timekeeper.

If cobalt preempts the core during holding the lock, any thread trying to read 
the time
will continue to spin. (This alone is an issue IMHO).
If the cobalt thread itself now call the vDSO function as reader,
it will spin on the lock and block the lock from getting released.


Either the update_fast_timekeeper funtion should not be preemptible by cobalt,
or the spin-lock on reading could fallback to the syscall after a certain 
amount of retries.

The later is probably easier to implement, but then could randomly demote 
cobalt threads.
(on the other hand, this would be always a demotion on platforms without the 
vdso function)



update_vsyscall() is locking the write-side. If the analysis is correct, this 
patch may help at the expense of a some cycles more spent uninterruptible:

diff --git a/arch/x86/entry/vsyscall/vsyscall_gtod.c 
b/arch/x86/entry/vsyscall/vsyscall_gtod.c
index 9fb89b6e88c3..e9baa57e8385 100644
--- a/arch/x86/entry/vsyscall/vsyscall_gtod.c
+++ b/arch/x86/entry/vsyscall/vsyscall_gtod.c
@@ -32,11 +32,14 @@ void update_vsyscall(struct timekeeper *tk)
  {
int vclock_mode = tk->tkr_mono.clock->archdata.vclock_mode;
struct vsyscall_gtod_data *vdata = _gtod_data;
+   unsigned long flags;
  
  	/* Mark the new vclock used. */

BUILD_BUG_ON(VCLOCK_MAX >= 32);
WRITE_ONCE(vclocks_used, READ_ONCE(vclocks_used) | (1 << vclock_mode));
  
+	flags = hard_cond_local_irq_save();

+
gtod_write_begin(vdata);
  
  	/* copy vsyscall data */

@@ -77,6 +80,8 @@ void update_vsyscall(struct timekeeper *tk)
  
  	gtod_write_end(vdata);
  
+	hard_cond_local_irq_restore(flags);

+
if (tk->tkr_mono.clock == _tsc)
ipipe_update_hostrt(tk);
  }



This should rather be an application bug: An RT (Xenomai) thread is apparently 
using Linux gettimeofday & Co. (glibc) from RT context. That was never 
supported, we rather have RT services for that (CLOCK_HOST_REALTIME).


We may think about detecting such cases better, though. Norbert, are you using 
native/alchemy APIs?


Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Cobalt Preemption of kernel update_fast_timekeeper can cause deadlocks

[Philippe Gerum via Xenomai]

On 12/19/18 11:20 AM, Lange Norbert via Xenomai wrote:
> There is a deadlock issue that haunted me for several weeks,
> it is caused by the kernels update of the user-visible
> timekeeping structures used by the VDSO clock_gettime functions.
> 
> The kernel regularly updates a Timestamp structure, which is accessible in 
> user-mode,
> it does so by something akin to a rw-lock in update_fast_timekeeper.
> 
> If cobalt preempts the core during holding the lock, any thread trying to 
> read the time
> will continue to spin. (This alone is an issue IMHO).
> If the cobalt thread itself now call the vDSO function as reader,
> it will spin on the lock and block the lock from getting released.
> 
> 
> Either the update_fast_timekeeper funtion should not be preemptible by cobalt,
> or the spin-lock on reading could fallback to the syscall after a certain 
> amount of retries.
> 
> The later is probably easier to implement, but then could randomly demote 
> cobalt threads.
> (on the other hand, this would be always a demotion on platforms without the 
> vdso function)
> 

update_vsyscall() is locking the write-side. If the analysis is correct, this 
patch may help at the expense of a some cycles more spent uninterruptible:

diff --git a/arch/x86/entry/vsyscall/vsyscall_gtod.c 
b/arch/x86/entry/vsyscall/vsyscall_gtod.c
index 9fb89b6e88c3..e9baa57e8385 100644
--- a/arch/x86/entry/vsyscall/vsyscall_gtod.c
+++ b/arch/x86/entry/vsyscall/vsyscall_gtod.c
@@ -32,11 +32,14 @@ void update_vsyscall(struct timekeeper *tk)
 {
int vclock_mode = tk->tkr_mono.clock->archdata.vclock_mode;
struct vsyscall_gtod_data *vdata = _gtod_data;
+   unsigned long flags;
 
/* Mark the new vclock used. */
BUILD_BUG_ON(VCLOCK_MAX >= 32);
WRITE_ONCE(vclocks_used, READ_ONCE(vclocks_used) | (1 << vclock_mode));
 
+   flags = hard_cond_local_irq_save();
+
gtod_write_begin(vdata);
 
/* copy vsyscall data */
@@ -77,6 +80,8 @@ void update_vsyscall(struct timekeeper *tk)
 
gtod_write_end(vdata);
 
+   hard_cond_local_irq_restore(flags);
+
if (tk->tkr_mono.clock == _tsc)
ipipe_update_hostrt(tk);
 }

-- 
Philippe.

Cobalt Preemption of kernel update_fast_timekeeper can cause deadlocks

[Lange Norbert via Xenomai]

There is a deadlock issue that haunted me for several weeks,
it is caused by the kernels update of the user-visible
timekeeping structures used by the VDSO clock_gettime functions.

The kernel regularly updates a Timestamp structure, which is accessible in 
user-mode,
it does so by something akin to a rw-lock in update_fast_timekeeper.

If cobalt preempts the core during holding the lock, any thread trying to read 
the time
will continue to spin. (This alone is an issue IMHO).
If the cobalt thread itself now call the vDSO function as reader,
it will spin on the lock and block the lock from getting released.


Either the update_fast_timekeeper funtion should not be preemptible by cobalt,
or the spin-lock on reading could fallback to the syscall after a certain 
amount of retries.

The later is probably easier to implement, but then could randomly demote 
cobalt threads.
(on the other hand, this would be always a demotion on platforms without the 
vdso function)

Mit besten Grüßen / Kind regards

NORBERT LANGE
AT-DES

ANDRITZ HYDRO GmbH
Eibesbrunnergasse 20
1120 Vienna / AUSTRIA
p: +43 50805 56684
norbert.la...@andritz.com
andritz.com


This message and any attachments are solely for the use of the intended 
recipients. They may contain privileged and/or confidential information or 
other information protected from disclosure. If you are not an intended 
recipient, you are hereby notified that you received this email in error and 
that any review, dissemination, distribution or copying of this email and any 
attachment is strictly prohibited. If you have received this email in error, 
please contact the sender and delete the message and any attachment from your 
system.

ANDRITZ HYDRO GmbH


Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation

Firmensitz/ Registered seat: Wien

Firmenbuchgericht/ Court of registry: Handelsgericht Wien

Firmenbuchnummer/ Company registration: FN 61833 g

DVR: 0605077

UID-Nr.: ATU14756806


Thank You