On Sat, Mar 2, 2013 at 12:13 PM, Ronny Meeus <ronny.me...@gmail.com>
wrote:
On Fri, Mar 1, 2013 at 9:41 AM, Philippe Gerum <r...@xenomai.org> wrote:
On 03/01/2013 09:30 AM, Gilles Chanteperdrix wrote:
On 03/01/2013 09:30 AM, Philippe Gerum wrote:
On 03/01/2013 09:26 AM, Gilles Chanteperdrix wrote:
On 03/01/2013 09:22 AM, Philippe Gerum wrote:
On 02/28/2013 09:22 PM, Thomas De Schampheleire wrote:
On Thu, Feb 28, 2013 at 9:10 PM, Gilles Chanteperdrix
<gilles.chanteperd...@xenomai.org> wrote:
On 02/28/2013 08:19 PM, Ronny Meeus wrote:
Hello
we are using the PSOS interface of Xenomai forge, running
completely
in user-space using the mercury code.
We deploy our application on different processors, one product is
running on PPC multicore (P4040, P4080, P4034) and another one on
Cavium (8 core device).
The Linux version we use is 2.6.32 but I would assume that this
is
not
so relevant.
Our Xenomai application is running on one of the cores (affinity
is
set), while the other cores are running other code.
On both architectures we recently start to see issues that one
thread
is consuming 100% of the core on which the application is pinned.
The thread that monopolizes the core is the thread internally
used
to
manage the timers, running at the highest priority.
The trigger for running into this behavior is currently unclear.
If we only start a part of the application (platform management
only),
the issue is not observed.
We see this on both an old version of Xenomai and a very recent
one
(pulled from the git repo yesterday).
I will continue to debug this issue in the coming days and try
isolate
the code that is triggering it, but I can use hints from the
community.
Debugging is complex since once the load starts, the debugger is
not
reacting anymore.
If I put breakpoints in the functions that are called when the
timer
expires (both oneshot and periodic), the process starts to clone
itself and I endup with tens of them.
Has anybody seen an issue like this before or does somebody has
some
hints on how to debug this problem?
First enable the watchdog. It will send a signal to the
application
when
detecting a problem, then you can use the watchdog to trigger an
I-pipe
tracer trace when the bug happens. You will probably have to
increase
the watchdog polling frequency, in order to have a meaningful
trace.
I don't think an I-pipe tracer will be possible when using the
Mercury
core, right (xenomai-forge) ?
Correct.
I do not think so. The way I see it, you can enable the I-pipe tracer
without CONFIG_XENOMAI.
Mercury has NO pipeline in the kernel.
You mean mercury can not run with an I-pipe kernel?
I mean it does not care about the pipeline, it does not need it. So if
this
is about observing kernel activity, then ftrace should be fine, or
possibly
perf to find out where userland spends time.
--
Philippe.
_______________________________________________
Xenomai mailing list
Xenomai@xenomai.org
http://www.xenomai.org/mailman/listinfo/xenomai
Hello
An update on the investigation:
I was able to make this issue disappear by changing the timeout value
of the smallest timers we use.
We use a couple of timers with a timeout of 25ms. By enlarging these
to 25sec and the problem is gone.
Yesterday I was also able to see (using the"strace" tool) the process
executing constantly "clone" system calls.
Note that the process we use is large (2Gb) and uses an mlockall call.
In
http://stackoverflow.com/questions/4263958/some-information-on-timer-helper-thread-of-librt-so-1/4935895#4935895
I see that a new thread is created when the timer_create is called for
the first time. This thread stays alive until the program exits and is
used to process the timer expiries.
I have the feeling that there is an issue during the creation of this
thread. For example what would happen if the clone operation takes
longer than the time needed to perform the clone operation?
In the past we already observed issues with the clone call that we
could not explain (creation of the clone simply failed on our
application while it was working fine on a smaller application).
Do you guys know whether there is an impact on the clone operation by
this mlockall call?
I will try to make a small test application on which the issue can be
reproduced.
---
Ronny
I'm able to reproduce the issue on a small test build:
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <psos.h>
#include <copperplate/init.h>
#include <stdlib.h>
#include <string.h>
static void foo (u_long a0, u_long a1, u_long a2, u_long a3)
{
u_long ret, ev = 0, tmid,tmid2;
ret = tm_evevery(1,1,&tmid);
ret = tm_evafter(30000,4,&tmid2);
while (1) {
ret = ev_receive(0xFF,EV_ANY|EV_WAIT,0,&ev);
if (ev & 4) {
printf("%lx Restarting one-shot timer.
ev=%lx\n",ret,ev);
tm_evafter(30000,4,&tmid2);
}
ev = 0;
}
tm_wkafter(100);
}
int main(int argc, char * const *argv)
{
u_long ret, tid = 0, args[4];
mlockall(MCL_CURRENT | MCL_FUTURE);
copperplate_init(&argc,&argv);
ret = t_create("TEST",97, 0, 0, 0, &tid);
printf("t_create(tid=%lu) = %lu\n", tid, ret);
args[0] = 1;
args[1] = 2;
args[2] = 3;
args[3] = 4;
ret = t_start(tid, 0, foo, args);
printf("t_start(tid=%lu) = %lu\n", tid, ret);
while (1)
tm_wkafter(100);
return 0;
}
The TEST task starts 2 timers: one periodic and one 1shot timer.
Each time the one-shot timer expires, a print is done and the timer is
restarted.
Observation is that once the one-shot timer expires, the application
starts to use 100% cpuload on one core and the application code is not
executed anymore. So it looks like there is constant processing in
either Xenomai or the library code to process the timer handling. If
periodic timers are used the issue is not observed.
