Re: AW: [Xenomai-core] Fundamental Questions

[Philippe Gerum]

Christopher Stone wrote:
In light of the desire for support below, would the Xenomai team 
consider listing companies capable of commercial support on their 
website, or create another mailing list for us to announce commercial 
offerings around Xenomai. I am prepared to contribute to Xenomai in 
order to receive this privilege.  
 
Sorry for discussing commercial issues on your mailing list. If it makes 
it any better, we are a very small company, just trying to make a 
living, not a big corporate conglomerate.
 


We are about to start such listing on xenomai.org. Requests for inclusion should
be sent to Bruno who manages the website. He will likely ask for the graphical
elements needed to add the link (i.e. a logo of the proper size), and a short 
description of the services being advertised.


As a final point, I believe Xenomai is very well positioned to become 
very popular and "future proof". I believe the next frontier for Linux 
is industrial grade Linux, or Linux on the factory floor and Xenomai 
will end up the technology of choice to make that happen. Contrary to 
many opinions I have heard, I beleve the rt-preempt work done by Ingo 
Molnar will enhance Xenomai and not replace it.


My comments about this issue will likely be much longer than needed, but it's an
opportunity I'm going to take in order to clarify a few things, as an attempt to
explain why Xeno is Xeno, and which evolution process has led to the current
implementation.

Xenomai attempts to address four basic use cases:

- situations where revalidating the entire Linux stack, with the proper level of
confidence with respect to reliability and worst-case latency, regardless of the 
set of drivers or sub-systems one might use in a setup during the project 
lifetime, is not an option. A Xenomai-based application relying on the primary 
mode for getting determinism won't be impacted by some hidden piece of Linux 
kernel code which happens not to conform with the new locking semantics preempt_rt 
enforces. This advantage will last at least until all vanilla kernel code, and all 
vendor-supplied drivers one might want to use in her design, have been made 
conformant to such requirements. Of course, one could argue that "rogue" drivers 
could also mask off interrupts at hw level and thus impact Xenomai too, but the 
main difference there is about how complex it is to locate and fix the culprit in 
a short time, without adverse side-effects on the rest of the system. Debugging 
and fixing a tiny RTOS core is easier than chasing priority inversions in a 
full-blown GPOS kernel trying to cope with real-time issues.


- situations where real-time application duties tend to become more complex as the 
hardware improves, i.e. when applications need to cope at the same time with a 
broad spectrum of real-time constraints and non real-time duties, ranging from 
fast data acquisition to deterministic networking, and/or FFT crunching, while 
still being able to get a decent throughput for moving large amount of data over 
various storage/network devices without adverse effect on latencies.


- situations where the hardware the application needs to run on is fairly
limited. In such a case, a safe approach is to handle the most time-critical 
tasks
over a context which does not depend on the reserve of CPU power Linux has at 
hand
to complete its current unpreemptable duties, before turning its head to the
critical stuff. On the other hand, dealing with real-time principles at the core
level of a GPOS for all kernel activities - including those who have no 
real-time
constraints - is expensive in terms of CPU horsepower, and engineering 
complexity.

- situations where legacy code exists, that used to run over a non-POSIX API,
and/or over a non-Linux real-time environment.

So far, I've seen the following approaches implemented for addressing some of 
those issues:


1) pretending that anything that doesn't fall into the low microsecond-range
real-time constraint has no real-time constraint, enforcing a complete isolation
between the real-time core and Linux, so one has the only option to ask Linux 
its
"best effort" for dealing with "trivial" (i.e. "non ultra-low latency 
real-time")
duties. This just does not scale with the increasing complexity of applications,
and has the additional downside to lower the incentive to keep the regular Linux
programming model available to the real-time developer (i.e. "who cares for
user-space, GDB and/or Valgrind since we are only interested in coding data
acquisition loops!". This one is also known as the REMBO syndrom, as in "REal 
Men
only Bring an Oscilloscope").

2) pretending that burying the real-time principles deep enough inside the 
vanilla
Linux core will solve all problems for all real-time configurations, with high 
performances and stability, for every use, on every platform. Frankly, I have no 
clue whether the preempt_rt effort will eventually succeed or fail, the way it is 
currently advertised, 

Re: AW: [Xenomai-core] Fundamental Questions

[Herman Bruyninckx]

On Wed, 8 Mar 2006, Jan Kiszka wrote:


Christopher Stone wrote:

I do not have specific plans.

I am working on something I am currently calling the Xen Loadable Module or
XLM. It is an Xenomai application that when loaded, turns the Linux kernel
into a Xen compatible hypervisor. For the rationale behind it see here:
http://www.openembedded.biz/content/view/36/27.

When it is ready, in 6 to 8 weeks, I am prepared to contribute it. It is a
significant work, but, is an Xenomai application, so I don't know if you
guys want it. It does fit with your goals with respect to industrial grade
Linux.

I think that XLM actually illustrates a key point that people forget when
they compare rt-preempt to Xenomai. I believe that in the industrial grade
Linux world, the ability to support multiple OS's is key, especially in
light of the emerging dual core CPU's. Due to ADEOS, Xenomai has the
infrastructure to support doing things like running eCos on one core and
Linux on the other, or eCos and Linux, side by side, on the same CPU. XLM is
designed to make this easy. Rt-preempt has no such capability.


Sounds interesting, especially when considering future systems with
hardware support for virtualisation, thus removing the need to patch the
guest OS (there are still people with the desire to run Windows aside
the RTOS core for visualisation etc.). And if your approach have real
advantages over Xen, specifically on embedded systems or in combination
with hard real-time, this could become really great.


I have the same positive reaction towards this XLM suggestion!

In some of our projects with machine tool builders, this kind of
virtualization is high on their wish list, because it's a perfect way
(hopefully) to combine lots of legacy GUI code with the advantages of a
realtime Linux-based controller.

Herman

--
  K.U.Leuven, Mechanical Eng.,  Mechatronics & Robotics Research Group
 Tel: +32 16 322480

Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm


___
Xenomai-core mailing list
Xenomai-core@gna.org
https://mail.gna.org/listinfo/xenomai-core

Re: AW: [Xenomai-core] Fundamental Questions

[Jan Kiszka]

Christopher Stone wrote:
> I do not have specific plans. 
> 
> I am working on something I am currently calling the Xen Loadable Module or
> XLM. It is an Xenomai application that when loaded, turns the Linux kernel
> into a Xen compatible hypervisor. For the rationale behind it see here:
> http://www.openembedded.biz/content/view/36/27.
> 
> When it is ready, in 6 to 8 weeks, I am prepared to contribute it. It is a
> significant work, but, is an Xenomai application, so I don't know if you
> guys want it. It does fit with your goals with respect to industrial grade
> Linux. 
> 
> I think that XLM actually illustrates a key point that people forget when
> they compare rt-preempt to Xenomai. I believe that in the industrial grade
> Linux world, the ability to support multiple OS's is key, especially in
> light of the emerging dual core CPU's. Due to ADEOS, Xenomai has the
> infrastructure to support doing things like running eCos on one core and
> Linux on the other, or eCos and Linux, side by side, on the same CPU. XLM is
> designed to make this easy. Rt-preempt has no such capability. 

Sounds interesting, especially when considering future systems with
hardware support for virtualisation, thus removing the need to patch the
guest OS (there are still people with the desire to run Windows aside
the RTOS core for visualisation etc.). And if your approach have real
advantages over Xen, specifically on embedded systems or in combination
with hard real-time, this could become really great.

> 
> I am not trying to discredit rt-preempt. It is a significant and useful
> piece of work and contains some pretty smart coding. However, in my view,
> rt-preempt is just part of the solution required for industrial grade linux.
> Things like rt-preempt, Xenomai, and hopefully XLM will all be pieces of a
> comprehensive industrial grade linux solution.
> 
> If XLM is not a suitable contribution to Xenomai, then, I can contribute

Let's wait for some code first so that things like intrusiveness etc.
can be analysed.

> other ways such as other feature development or bug fixing. I would need
> some direction from the leaders in order to contribute in that way.

It's often best to pick a domain you are interested in on your own. This
can drive the overall development application-oriented in a very
constructive way. When you are using Xenomai for some projects, you may
happen to stumble over rough edges or lacking features.

Jan



signature.asc
Description: OpenPGP digital signature
___
Xenomai-core mailing list
Xenomai-core@gna.org
https://mail.gna.org/listinfo/xenomai-core

RE: AW: [Xenomai-core] Fundamental Questions

[Christopher Stone]

I do not have specific plans. 

I am working on something I am currently calling the Xen Loadable Module or
XLM. It is an Xenomai application that when loaded, turns the Linux kernel
into a Xen compatible hypervisor. For the rationale behind it see here:
http://www.openembedded.biz/content/view/36/27.

When it is ready, in 6 to 8 weeks, I am prepared to contribute it. It is a
significant work, but, is an Xenomai application, so I don't know if you
guys want it. It does fit with your goals with respect to industrial grade
Linux. 

I think that XLM actually illustrates a key point that people forget when
they compare rt-preempt to Xenomai. I believe that in the industrial grade
Linux world, the ability to support multiple OS's is key, especially in
light of the emerging dual core CPU's. Due to ADEOS, Xenomai has the
infrastructure to support doing things like running eCos on one core and
Linux on the other, or eCos and Linux, side by side, on the same CPU. XLM is
designed to make this easy. Rt-preempt has no such capability. 

I am not trying to discredit rt-preempt. It is a significant and useful
piece of work and contains some pretty smart coding. However, in my view,
rt-preempt is just part of the solution required for industrial grade linux.
Things like rt-preempt, Xenomai, and hopefully XLM will all be pieces of a
comprehensive industrial grade linux solution.

If XLM is not a suitable contribution to Xenomai, then, I can contribute
other ways such as other feature development or bug fixing. I would need
some direction from the leaders in order to contribute in that way.

Cheers,
   Chris.


Christopher Stone
Principal
Sombrio Systems Inc.
www.openembedded.biz
613-831-1892

-Original Message-
From: Jan Kiszka [mailto:[EMAIL PROTECTED] 
Sent: March 8, 2006 1:18 PM
To: [EMAIL PROTECTED]
Cc: xenomai-core@gna.org
Subject: Re: AW: [Xenomai-core] Fundamental Questions

Christopher Stone wrote:
> In light of the desire for support below, would the Xenomai team
> consider listing companies capable of commercial support on their
> website, or create another mailing list for us to announce commercial
> offerings around Xenomai. I am prepared to contribute to Xenomai in
> order to receive this privilege.

We had such a page on the old RTAI site, though not really populated I
think to remember. I personally have no concerns creating such a forum
for Xenomai as well. I think we could start with a link sub-page on
xenomai.org.

> 
> Sorry for discussing commercial issues on your mailing list. If it
> makes it any better, we are a very small company, just trying to make
> a living, not a big corporate conglomerate.

I'm convinced that this is not a question of big or small. There should
just be a good balance between taking and giving.

> 
> As a final point, I believe Xenomai is very well positioned to become
> very popular and "future proof". I believe the next frontier for
> Linux is industrial grade Linux, or Linux on the factory floor and
> Xenomai will end up the technology of choice to make that happen.
> Contrary to many opinions I have heard, I beleve the rt-preempt work
> done by Ingo Molnar will enhance Xenomai and not replace it. I also

Yep, that is one important point why Xenomai is future-proof in my eyes.
The day may come when significant parts of the PREEMPT_RT effort are
merged into mainline. But then Xenomai will still be able to offer
either staged degrees of hard-RT side by side with that approach, or
become a wrapping layer on top of the new deterministic and fast
infrastructure (PREEMPT_RT + RT-extended glibc). No one should expect
that this will happen soon, PREEMPT_RT has just started its tricky way
into the kernel.

> think the break from RTAI was very smart as it has given you the
> flexibility to move Xenomai in the direction it needs to go. The
> recent releases have made Xenomai ready for the commercial world. So,
> kudos to the Xenomai team. You guys are proving to be great leaders
> with the right technology at the right time.
> 

This nice compliment clearly has to be passed to our smart maintainer!

Jan


PS: Do you already have specific plans for potential contributions? I'm
just curious.




___
Xenomai-core mailing list
Xenomai-core@gna.org
https://mail.gna.org/listinfo/xenomai-core

Re: AW: [Xenomai-core] Fundamental Questions

[Jan Kiszka]

Christopher Stone wrote:
> In light of the desire for support below, would the Xenomai team
> consider listing companies capable of commercial support on their
> website, or create another mailing list for us to announce commercial
> offerings around Xenomai. I am prepared to contribute to Xenomai in
> order to receive this privilege.

We had such a page on the old RTAI site, though not really populated I
think to remember. I personally have no concerns creating such a forum
for Xenomai as well. I think we could start with a link sub-page on
xenomai.org.

> 
> Sorry for discussing commercial issues on your mailing list. If it
> makes it any better, we are a very small company, just trying to make
> a living, not a big corporate conglomerate.

I'm convinced that this is not a question of big or small. There should
just be a good balance between taking and giving.

> 
> As a final point, I believe Xenomai is very well positioned to become
> very popular and "future proof". I believe the next frontier for
> Linux is industrial grade Linux, or Linux on the factory floor and
> Xenomai will end up the technology of choice to make that happen.
> Contrary to many opinions I have heard, I beleve the rt-preempt work
> done by Ingo Molnar will enhance Xenomai and not replace it. I also

Yep, that is one important point why Xenomai is future-proof in my eyes.
The day may come when significant parts of the PREEMPT_RT effort are
merged into mainline. But then Xenomai will still be able to offer
either staged degrees of hard-RT side by side with that approach, or
become a wrapping layer on top of the new deterministic and fast
infrastructure (PREEMPT_RT + RT-extended glibc). No one should expect
that this will happen soon, PREEMPT_RT has just started its tricky way
into the kernel.

> think the break from RTAI was very smart as it has given you the
> flexibility to move Xenomai in the direction it needs to go. The
> recent releases have made Xenomai ready for the commercial world. So,
> kudos to the Xenomai team. You guys are proving to be great leaders
> with the right technology at the right time.
> 

This nice compliment clearly has to be passed to our smart maintainer!

Jan


PS: Do you already have specific plans for potential contributions? I'm
just curious.



signature.asc
Description: OpenPGP digital signature
___
Xenomai-core mailing list
Xenomai-core@gna.org
https://mail.gna.org/listinfo/xenomai-core

Re: AW: [Xenomai-core] Fundamental Questions

[Christopher Stone]

In light of the desire for support below, would the Xenomai team consider listing companies capable of commercial support on their website, or create another mailing list for us to announce commercial offerings around Xenomai. I am prepared to contribute to Xenomai in order to receive this privilege.       Sorry for discussing commercial issues on your mailing list. If it makes it any better, we are a very small company, just trying to make a living, not a big corporate conglomerate.      As a final point, I believe Xenomai is very well positioned to become very popular and "future proof". I believe the next frontier for Linux is industrial grade Linux, or Linux on the factory floor and Xenomai will end up the technology of choice to make that happen. Contrary to many opinions I have heard, I beleve the rt-preempt work done by Ingo Molnar will enhance Xenomai and not replace it. I also think the
 break from RTAI was very smart as it has given you the flexibility to move Xenomai in the direction it needs to go. The recent releases have made Xenomai ready for the commercial world. So, kudos to the Xenomai team. You guys are proving to be great leaders with the right technology at the right time.     Cheers,      Chris Stone.[EMAIL PROTECTED] wrote:  Dear Jan,thank you for taking time to answer my questions and sorry for the delayed response, but I have been busy with some other work. Please find my follow-up questions inserted in the text.> > > > 1.)> > Essentially the question deals with the problem, how long a > Xenomai task in secondary mode can be delayed by normal Linux tasks. > > In detail : we plan to to have a
 lot of "near realtime" > ethernet communication from within Xenomai using the normal > Linux network stack (calling the normal socket API). The > question now is, how our network communication is influenced > by other Linux tasks performing also network communication, > let´s say an FTP transfer ?> > Depending on the "normal" networking load, you will suffer > from more or less frequent (indeterministic) packet delays. Do you have an idea about the dimension weare talking about :less than a millisecond, few milliseconds, seconds, or is thedelay complete indeterministic ?> Xenomai will not improve this in any way. If your task in > secondary mode tries to send some data and requires to take a > networking lock currently held by another Linux task, it can > take a real long time until this request is completed. But at least, after a (linux-)systemcall (from what task
 ever) finished, Xenomai gets controll back before any other linux task, isn´t it ?This means : between systemcalls a rescheduling back to Xenomai is performed or isn´t it ??Sorry for the next stupid question, but what is a network lock. With what kind of action a task can lock the complete stack ? And how long could it block the stack ?Could you give me an example for better understanding ?> This > gets better with PREEMPT_RT but still remains non-RT because > the Linux networking stack is not designed for hard real-time.> Next stupid question : what is PREEMPT_RT ? Is this kernel 2.6 or is it the Monta Vista approach for real time (making the kernel more preemtable) ?> > If you communication can be soft-RT, you could indeed avoid > the separation - but you will then have to live with the side > effects. All you can do then is try to lower the number of > deadline
 misses by keeping the standard network traffic low > and managing the bandwidth of the participants (the Linux > network stack has some support for QoS, at least> 2.6 I think).> > BTW, as long as your network is not separated or you have no > control over the traffic arriving at your system, picking the > Linux stack in favour of RTnet (which is compatible with > non-RT networks) is indeed generally recommended. This way > you keep indeterministic load away from the real-time subsystem.> Unfortunatelly we don´t want to limit non realtime traffic, we just want to make shure, that deterministic traffic has a higher priority than non RT traffic (like in other RTOS like vxWorks). Indeterministic traffic should get just the leftover bandwith.What do you mean with : "Rtnet is compatible with non-RT networks" ? I thought RTnet uses a time slice mechanism and therefore could not be mixed
 with systems transmitting when ever they want. Do you refer to VNICs ?> > > > I have created a scheduling scenario and I would ask you to > have a look on it and to tell me whether it is correct or > not. Thank you !> > An corresponding question about this scheduling is : are there > > differences between a 2.4 and 2.6 Linux kernel ? (for our PowerPC > > plattform we intend to use the 2.4 kernel for performance reasons)> > > > Scheduling scenario :> > (I hope formating is not destroyed by email transfer)> > > > Time moves downwards> > > > v-Xenomai > > v-Linux kernel> > v-Linux processes> > > > l1 Linux task1 running> > s1 < l1 Linux task1 makes systemcall> > s1 Linux task1 systemcall processed

Re: AW: [Xenomai-core] Fundamental Questions

[Jan Kiszka]

[EMAIL PROTECTED] wrote:
> Dear Jan,
> thank you for taking time to answer my questions and 
> sorry for the delayed response, but I have been busy 
> with some other work. 
> Please find my follow-up questions inserted in the text.
> 
>>> 1.)
>>> Essentially the question deals with the problem, how long a 
>> Xenomai task in secondary mode can be delayed by normal Linux tasks. 
>>> In detail : we plan to to have a lot of "near realtime" 
>> ethernet communication from within Xenomai using the normal 
>> Linux network stack (calling the normal socket API). The 
>> question now is, how our network communication is influenced 
>> by other Linux tasks performing also network communication, 
>> let´s say an FTP transfer ?
>>
>> Depending on the "normal" networking load, you will suffer 
>> from more or less frequent (indeterministic) packet delays. 
> 
> Do you have an idea about the dimension weare talking about :
> less than a millisecond, few milliseconds, seconds, or is the
> delay complete indeterministic ?

Normally you will only see millisecond delays or less. But in case of
overload (heavy data transfers, miss-configured or virus-contaminated
nodes etc.) you may face several hundred milliseconds or more - up to
packets drops. That depends on the infrastructure (network layout,
switches and their QoS features) and cannot be answered generally. Good
QoS-aware switches can help here.

> 
>> Xenomai will not improve this in any way. If your task in 
>> secondary mode tries to send some data and requires to take a 
>> networking lock currently held by another Linux task, it can 
>> take a real long time until this request is completed. 
> 
> 
> 
> But at least, after a (linux-)systemcall (from what task ever) finished, 
> Xenomai gets controll back before any other linux task, isn´t it ?
> This means : between systemcalls a rescheduling back to Xenomai is performed 
> or isn´t it ??

The Xenomai domain regains control as soon as a) the real-time task in
secondary mode can continue or b) some other real-time task becomes
runnable.

> Sorry for the next stupid question, but what is a network lock. With what 
> kind of 
> action a task can lock the complete stack ? And how long could it block the 
> stack ?
> Could you give me an example for better understanding ?

"Network lock" was an oversimplification. Actually, this is a complex
topic, and I had a wrong model in mind. To clarify:

The networking stack contains a lot of locks (NIC transmission access,
routing and ARP tables, input and output queues, ...), but they are
non-preemptible for the transmission and reception path on standard
Linux (as long as I do not oversee some corner case). So, contention is
now happening to the point where your RT-task issues a Linux syscall.
The kernel then has to preempt the interrupted Linux task (= interrupted
by Xenomai when the RT-task became runnable) to let your task run in
secondary mode. This normally happens within a few hundred microseconds
(CONFIG_PREEMPT enabled in the kernel), but there can be scenarios where
preemption is disabled for several hundred milliseconds or more - though
very rarely with CONFIG_PREEMPT, but possible. That can be ok for soft
RT if you are able to handle the exceptions.

Besides this, another critical issue are delays due to buffer
acquisitions. If there is no fitting buffer at hand on packet arrival or
transmission request, it can take a while to free the necessary
resources, specifically if your whole system just happen to run short on
memory (swapping is taking place, some application leaked memory, ...).

> 
> 
>> This 
>> gets better with PREEMPT_RT but still remains non-RT because 
>> the Linux networking stack is not designed for hard real-time.
>>
> 
> 
> Next stupid question : what is PREEMPT_RT ? Is this kernel 2.6 or is it the 
> Monta Vista approach for real time (making the kernel more preemtable) ?

Actually, neither of both. It is the ongoing community effort (which
includes MontaVista) to make the kernel natively preemptible, it is not
MontaVista's own project. Watch out for "-rt" or "real-time preempt" on
the Linux kernel mailing list, or look at
http://people.redhat.com/mingo/realtime-preempt (don't ask me for the
latest user-space pieces, i.e. glibc patches, they are either still
unpublished, well hidden, or non-existent yet).

> 
>> If you communication can be soft-RT, you could indeed avoid 
>> the separation - but you will then have to live with the side 
>> effects. All you can do then is try to lower the number of 
>> deadline misses by keeping the standard network traffic low 
>> and managing the bandwidth of the participants (the Linux 
>> network stack has some support for QoS, at least
>> 2.6 I think).
>>
>> BTW, as long as your network is not separated or you have no 
>> control over the traffic arriving at your system, picking the 
>> Linux stack in favour of RTnet (which is compatible with 
>> non-RT networks) is indeed generally recommended. This way 
>>