RE: IPIPE patch causes hang during initrd boot; Ubuntu 18 desktop system
> From: Jan Kiszka > Sent: Thursday, June 13, 2019 3:12 PM > To: Proctor, Frederick M. (Fed) ; > xenomai@xenomai.org > Subject: Re: IPIPE patch causes hang during initrd boot; Ubuntu 18 desktop > system > > On 13.06.19 18:36, Jan Kiszka via Xenomai wrote: > > On 12.06.19 15:05, Jan Kiszka via Xenomai wrote: > >> On 12.06.19 14:37, Proctor, Frederick M. (Fed) via Xenomai wrote: > On 07.06.19 15:02, Proctor, Frederick M. (Fed) via Xenomai wrote: > > I am installing Xenomai + Cobalt with the IPIPE patch on a Ubuntu > > 18-based x86/64 desktop system. I have these versions: > > > > Linux kernel 4.14.71 PREEMPT_RT patch 4.14.71-rt44 IPIPE 4.14.71 > > Latest stable I-pipe version is ipipe-core-4.14.111-x86-3. > > >>> > >>> I was using 4.14.71 since it has patches for both IPIPE and > >>> PREEMPT_RT. I had used 4.14.111 also, but had the same kernel hang > problem at boot time. > >>> > >> > >> OK. > >> > > > > I have compiled the kernel three different ways: the stock > > configuration, with the PREEMPT_RT patch, and with the IPIPE patch > > (no PREEMPT_RT). I have configured out the ACPI processor, > > CPU_FREQ, and CPU_IDLE options in all cases. The first two boot > > fine (stock kernel, PREEMPT_RT). The IPIPE-patched kernel hangs > > during the initrd loading phase. > > > > I've tailored the kernel configuration somewhat based on the > > hardware I have, but nothing has worked to get a booting IPIPE > kernel. > > > > Does anyone have guidance on setting up the IPIPE patch on an > > x86/64 kernel on a desktop Ubuntu 18 machine? I am planning on > > continuing to trim down the kernel config options, but before I do > > this, I want to ask this group if anyone has had a similar problem. > > > > Does yous system start to boot again when the I-pipe kernel is > built without CONFIG_IPIPE and CONFIG_XENOMAI? If no, compare > the > config to a working one, step-wise aligning to it. > > Jan > >>> > >>> Yes it does - with the kernel patched with IPIPE and Xenomai, but > >>> with IPIPE and Xenomai *not selected* in the kernel config, the > >>> kernel boots fine. With IPIPE and Xenomai *selected* in a subsequent > >>> kernel config, that kernel hangs. Here's the diff on the config files: > >>> > >> > >> Can you share the full config? Maybe something subtly incompatible > >> with I-pipe is enabled, maybe I can reproduce in KVM (ie. with a chance to > debug). > >> > > > > With the configuration you sent offlist, I can confirm now that the > > system (in my case a KVM target) does not come up, rather gets stuck > > during secondary CPU boot. I'm not yet seeing the key difference, but we > will find out. > > > > I'm attaching the config I'm currently using for 4.14, maybe we can > > search in parallel for the relevant delta. > > > > Found: CONFIG_MAXSMP. Needs to be off, NR_CPUS at 64 eg. > > I need to check where our exact limit is, but this should likely be caught at > kconfig level. > > At the same time, you will likely want to tune down your config for your > needs. > CONFIG_MIGRATION (with CMA, transparent hugh pages etc.) is one thing > that you do not want in an RT kernel, but there is likely more ballast. > > Jan Thanks - this worked, I disabled CONFIG_MAXSMP, rebuilt and it booted fine. I'll go in and tune down things once I run the Xenomai tests. --Fred
Re: cobalts printf thread affects signalfd operation
On 13.06.19 19:08, Lange Norbert via Xenomai wrote: Hello, I ran into a problem with the automatically spawned printf thread. Short summary: The kernel delivers signals by picking a thread that can accept a signal and does not mask the specific signal, only if none is available then the signal will be queued for asynchronous delivery. signalfd is asynchronous delivery, thus if one thread accepts signals then signalfd won't work as expected, enter cobals printf thread. Good point... Solutions would be: - Mask all signals for this thread (is there any downside?) I don't see any, we do not use signals for the operation of the printer. - Mask all signals on the main thread before calling xenomai_init. This is ugly, as that removes most methods of interacting with a hung startup. - Somehow mask the thread for signals later (only ugly solutions?). The first option would be a rather small patch: @@ -642,7 +642,11 @@ static void spawn_printer_thread(void) pthread_attr_t thattr; pthread_attr_init(&thattr); +sigset_t sset, oset; +sigfillset(&sset) +sigprocmask(SIG_BLOCK, &sset, &oset); pthread_create(&printer_thread, &thattr, printer_loop, NULL); +sigprocmask(SIG_SETMASK, &oset, NULL); } Looks reasonable, please send a patch (then it can still make into next stable). Jan -- -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux
Re: IPIPE patch causes hang during initrd boot; Ubuntu 18 desktop system
On 13.06.19 18:36, Jan Kiszka via Xenomai wrote: On 12.06.19 15:05, Jan Kiszka via Xenomai wrote: On 12.06.19 14:37, Proctor, Frederick M. (Fed) via Xenomai wrote: On 07.06.19 15:02, Proctor, Frederick M. (Fed) via Xenomai wrote: I am installing Xenomai + Cobalt with the IPIPE patch on a Ubuntu 18-based x86/64 desktop system. I have these versions: Linux kernel 4.14.71 PREEMPT_RT patch 4.14.71-rt44 IPIPE 4.14.71 Latest stable I-pipe version is ipipe-core-4.14.111-x86-3. I was using 4.14.71 since it has patches for both IPIPE and PREEMPT_RT. I had used 4.14.111 also, but had the same kernel hang problem at boot time. OK. I have compiled the kernel three different ways: the stock configuration, with the PREEMPT_RT patch, and with the IPIPE patch (no PREEMPT_RT). I have configured out the ACPI processor, CPU_FREQ, and CPU_IDLE options in all cases. The first two boot fine (stock kernel, PREEMPT_RT). The IPIPE-patched kernel hangs during the initrd loading phase. I've tailored the kernel configuration somewhat based on the hardware I have, but nothing has worked to get a booting IPIPE kernel. Does anyone have guidance on setting up the IPIPE patch on an x86/64 kernel on a desktop Ubuntu 18 machine? I am planning on continuing to trim down the kernel config options, but before I do this, I want to ask this group if anyone has had a similar problem. Does yous system start to boot again when the I-pipe kernel is built without CONFIG_IPIPE and CONFIG_XENOMAI? If no, compare the config to a working one, step-wise aligning to it. Jan Yes it does - with the kernel patched with IPIPE and Xenomai, but with IPIPE and Xenomai *not selected* in the kernel config, the kernel boots fine. With IPIPE and Xenomai *selected* in a subsequent kernel config, that kernel hangs. Here's the diff on the config files: Can you share the full config? Maybe something subtly incompatible with I-pipe is enabled, maybe I can reproduce in KVM (ie. with a chance to debug). With the configuration you sent offlist, I can confirm now that the system (in my case a KVM target) does not come up, rather gets stuck during secondary CPU boot. I'm not yet seeing the key difference, but we will find out. I'm attaching the config I'm currently using for 4.14, maybe we can search in parallel for the relevant delta. Found: CONFIG_MAXSMP. Needs to be off, NR_CPUS at 64 eg. I need to check where our exact limit is, but this should likely be caught at kconfig level. At the same time, you will likely want to tune down your config for your needs. CONFIG_MIGRATION (with CMA, transparent hugh pages etc.) is one thing that you do not want in an RT kernel, but there is likely more ballast. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux
cobalts printf thread affects signalfd operation
Hello, I ran into a problem with the automatically spawned printf thread. Short summary: The kernel delivers signals by picking a thread that can accept a signal and does not mask the specific signal, only if none is available then the signal will be queued for asynchronous delivery. signalfd is asynchronous delivery, thus if one thread accepts signals then signalfd won't work as expected, enter cobals printf thread. Solutions would be: - Mask all signals for this thread (is there any downside?) - Mask all signals on the main thread before calling xenomai_init. This is ugly, as that removes most methods of interacting with a hung startup. - Somehow mask the thread for signals later (only ugly solutions?). The first option would be a rather small patch: @@ -642,7 +642,11 @@ static void spawn_printer_thread(void) pthread_attr_t thattr; pthread_attr_init(&thattr); +sigset_t sset, oset; +sigfillset(&sset) +sigprocmask(SIG_BLOCK, &sset, &oset); pthread_create(&printer_thread, &thattr, printer_loop, NULL); +sigprocmask(SIG_SETMASK, &oset, NULL); } Mit besten Grüßen / Kind regards NORBERT LANGE AT-DES ANDRITZ HYDRO GmbH Eibesbrunnergasse 20 1120 Vienna / AUSTRIA p: +43 50805 56684 norbert.la...@andritz.com andritz.com This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. ANDRITZ HYDRO GmbH Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation Firmensitz/ Registered seat: Wien Firmenbuchgericht/ Court of registry: Handelsgericht Wien Firmenbuchnummer/ Company registration: FN 61833 g DVR: 0605077 UID-Nr.: ATU14756806 Thank You
Re: IPIPE patch causes hang during initrd boot; Ubuntu 18 desktop system
On 12.06.19 15:05, Jan Kiszka via Xenomai wrote: On 12.06.19 14:37, Proctor, Frederick M. (Fed) via Xenomai wrote: On 07.06.19 15:02, Proctor, Frederick M. (Fed) via Xenomai wrote: I am installing Xenomai + Cobalt with the IPIPE patch on a Ubuntu 18-based x86/64 desktop system. I have these versions: Linux kernel 4.14.71 PREEMPT_RT patch 4.14.71-rt44 IPIPE 4.14.71 Latest stable I-pipe version is ipipe-core-4.14.111-x86-3. I was using 4.14.71 since it has patches for both IPIPE and PREEMPT_RT. I had used 4.14.111 also, but had the same kernel hang problem at boot time. OK. I have compiled the kernel three different ways: the stock configuration, with the PREEMPT_RT patch, and with the IPIPE patch (no PREEMPT_RT). I have configured out the ACPI processor, CPU_FREQ, and CPU_IDLE options in all cases. The first two boot fine (stock kernel, PREEMPT_RT). The IPIPE-patched kernel hangs during the initrd loading phase. I've tailored the kernel configuration somewhat based on the hardware I have, but nothing has worked to get a booting IPIPE kernel. Does anyone have guidance on setting up the IPIPE patch on an x86/64 kernel on a desktop Ubuntu 18 machine? I am planning on continuing to trim down the kernel config options, but before I do this, I want to ask this group if anyone has had a similar problem. Does yous system start to boot again when the I-pipe kernel is built without CONFIG_IPIPE and CONFIG_XENOMAI? If no, compare the config to a working one, step-wise aligning to it. Jan Yes it does - with the kernel patched with IPIPE and Xenomai, but with IPIPE and Xenomai *not selected* in the kernel config, the kernel boots fine. With IPIPE and Xenomai *selected* in a subsequent kernel config, that kernel hangs. Here's the diff on the config files: Can you share the full config? Maybe something subtly incompatible with I-pipe is enabled, maybe I can reproduce in KVM (ie. with a chance to debug). With the configuration you sent offlist, I can confirm now that the system (in my case a KVM target) does not come up, rather gets stuck during secondary CPU boot. I'm not yet seeing the key difference, but we will find out. I'm attaching the config I'm currently using for 4.14, maybe we can search in parallel for the relevant delta. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux -- next part -- A non-text attachment was scrubbed... Name: config-amd64.xz Type: application/x-xz Size: 22960 bytes Desc: not available URL: <http://xenomai.org/pipermail/xenomai/attachments/20190613/277c1b9e/attachment.bin>
Re: linux not getting gpio interrupts on xenomai 3.0.8 on raspberrypi raspbian/stretch kernel 4.9
On Wed, Jun 12, 2019 at 5:40 PM Harco Kuppens via Xenomai wrote: > > > I put the raspberry pi image with xenomai2 I build for the raspberry pi > 2 and 3, also supporting the latest raspberry pi3b+ online at: > >http://www.cs.ru.nl/lab/xenomai/raspberrypi.html > > I'm still wondering what the reason is that I don't receive the gpio > interrupts in linux. Unfortunately nobody could answer my questions so > far. If somebody knows the problem, then I will fix the image. > > Except for this linux gpio interrupt problem the above image seems to > work of for realtime xenomai gpio interrupts. > > Best regards, > Harco Kuppens > > Op 07/06/2019 om 12:34 schreef Harco Kuppens via Xenomai: > > Hi, > > > > In juli 2017 I managed to port xenomai 3.0.5 to rasbpian jessie on the > > rpi3b. See https://xenomai.org/pipermail/xenomai/2017-July/037515.html > > > > But this image doesn't boot on the new raspberrypi 3b+ board. So I had > > to build a new image. I succeeded to build this image. The > > instructions are at: > > > > https://github.com/harcokuppens/xenomai3_rpi_gpio/blob/master/install/install_xenomai-3.0.8_on_rpi2_3_3B%2B.txt > > > > > > Xenomai and handling gpio interrupts in realtime work fine on this > > image, however somehow handling gpio interrupts in linux don't work > > anymore. I have several test scripts using the wiringpi library within > > linux. Writing and reading gpio pins works fine, but somehow we don't > > get any interrupts. > > > > Even if do not load the xeno_gpio_bcm2835 module, then xenomai doesn't > > use the gpio pins, but still then in linux with wiringpi examples we > > don't get any gpio interrupts. > > > > However if I boot the standard raspbian image with the standard > > raspbian kernel which does not support xenomai/cobalt then the > > wiringpi examples work fine, and we get the gpio interrupts. > > > > In my older image from juli 2017 both in a realtime xenomai program or > > in a wiringpi linux program I received interrupts. (when run > > separately at different times, so they cannot influence each other; so > > they are not sharing interrupts!) > > > > So I wonder why do the interrupts only work in xenomai realtime, and > > not in linux anymore? > > > > Or is there something maybe changed because we now use > > > > * newer kernel version 4.9 instead of 4.1 > > * xenomai 3.08 instead of xenomai 3.05 > > > > I would expect that it just should work, they are different points at > > the pipeline, and either of them is only watching for interrupt it > > should just get it. So instead maybe I did do something wrong when > > patching the rpi 4.9 kernel during the installation? > > > > Patching was not so easy because the raspbian os on the raspberry pi > > comes with a customized kernel specificly tuned for the raspberry pi > > hardware. I call this the rpi kernel. This kernel is little bit > > different then the standard kernel from kernel.org. I call this the > > kernel.org kernel. > > > > The ipipe patches for the kernel are made for the kernel.org kernels. > > However I managed pretty easily to patch the rpi 4.9 kernel with the > > the ipipe patch for the kernel.org 4.9 kernel. > > Except for the file pinctrl-bcm2835.c which is the driver for gpio > > interrupts. In my installation patching this file for the rpi 4.9 > > kernel was difficult, but I thought I finally succeeded. > > > > However because the problem with linux not getting any interrupts I > > wonder if something still went wrong there. > > > > So if I look at the ipipe-patched version of the pinctrl-bcm2835.c for > > my older image using the rpi 4.1 kernel and my ipiped-patch for this > > file for rpi-4.9 kernel I am building the image now I find some > > differences between the files. > > The files you can find at: > > > > * > > https://raw.githubusercontent.com/harcokuppens/xenomai3_rpi_gpio/master/install/how_pinctrl-bcm2835_patch_for_rpi-4.9_is_derived/rpi-4.1.y.ipipe-patched/pinctrl-bcm2835.c > > * > > https://raw.githubusercontent.com/harcokuppens/xenomai3_rpi_gpio/master/install/how_pinctrl-bcm2835_patch_for_rpi-4.9_is_derived/rpi-4.9.y.ipipe-patched/pinctrl-bcm2835.c > > > > I found the following changes from the 4.1 to the 4.9 version: > > a) spin_lock_init/spin_lock_irqsave/spin_unlock_irqrestore are > > replaced with > > raw_spin_lock_init/raw_spin_lock_irqsave/raw_spin_unlock_irqrestore calls > > b) in bcm2835_gpio_irq_enable function we have an extra call > > ipipe_unlock_irq(data->irq) before calling raw_spin_unlock_irqrestore > > c) in bcm2835_gpio_irq_disable we have an extra call ipipe_lock_irq > > before > > calling raw_spin_unlock_irqrestore > > d) in static struct irq_chip bcm2835_gpio_irq_chip we defined extra > >functions for hold and release > > e) in bcm2835_pinctrl_probe the call > > > > err = devm_request_irq(dev, > > pc->irq[i],bcm2835_gpio_irq_handler, IRQF_SHARED,name, &pc->irq_data[i]); > > > >is replaced with > > > > if (IS_ENABLED(CONFIG_IPIPE))
