Re: INTR-REMAP error with UDD driver
On Friday, March 8, 2019 9:00 AM, Jan Kiszka wrote: > On 08.03.19 15:07, Jeff Webb wrote: > > > On Friday, March 8, 2019 1:14 AM, Jan Kiszka jan.kis...@siemens.com wrote: > > > > > On 08.03.19 02:35, Jeff Webb via Xenomai wrote: > > > > > > > I have implemented a simple UDD mini-driver (interrupt handler) and an > > > > associated Xenomai userspace test program. This pair of programs works > > > > as intended on one machine. When I tried running the same programs on > > > > another machine, I got this error when the first interrupt occurred: > > > > kernel: [ 75.122305] DMAR: DRHD: handling fault status reg 2 > > > > kernel: [ 75.123175] DMAR: [INTR-REMAP] Request device [f0:1f.0] fault > > > > index ee00 [fault reason 37] Blocked a compatibility format interrupt > > > > request > > > > > > This means the device is under IOMMU interrupt remapping regime, but it > > > was > > > programmed to use a standard MSI/MSI-X request, rather than requesting > > > that > > > parameters (address/data tuple) from the Linux interrupt management layer > > > that > > > will assign a remapping slot and hand out the right values. Now you > > > interrupt is > > > stuck in the filter. > > > Before suggesting the workaround/hack to disable interrupt remapping: How > > > did > > > you program MSI/-X? Via proper Linux pci_* helpers? > > > > The PCI card I am working with does not implement MSI interrupts, but just > > uses the legacy interrupt pin A. It's confusing to me that this error is > > associated with MSI. Since this behavior is slot-dependent, I figured at > > first that the error is from another device on the same interrupt line, but > > I get the same error on several different slots. There is only one slot > > where things work as expected. > > Ok, then we may have an issue with how we register legacy INTx at the IOAPIC > when interrupt remapping is on. Basically, the IOAPIC sends MSI-like messages > as > well that are subject to remapping. So its registers need to be programmed > with > the right translatable values, just like the MSI registers of a device. Linux > does that, it's in charge of the IOMMU. I don't recall how we (Xenomai/I-pipe) > fill the IOAPIC if a line was not used before under Linux. Possibly, there is > the wrong Linux function triggered. But maybe it's also something different. > > I assume you tested that your card would work find if it used only regular > Linux > driver services, no RTDM, right? That would be good in order to exclude a > general platform issue, maybe a BIOS (ACPI) bug in describing which PCI slot > is > under which IOMMU regime. I just wrote a similar UIO driver and user-space program. The UIO version works when the card is in any slot. The UDD version only works when the card is in one slot (out of 5). The UDD version does not work in the "bad" slots even if the UIO version is run first. All of the UIO testing is with the same kernel as the UDD testing. Thanks, -Jeff
Re: smokey's fork tests hangs?
On 08.03.19 15:05, Lange Norbert wrote: -Original Message- From: Jan Kiszka Sent: Freitag, 8. März 2019 14:59 To: Lange Norbert ; Xenomai (xenomai@xenomai.org) Subject: Re: smokey's fork tests hangs? E-MAIL FROM A NON-ANDRITZ SOURCE: AS A SECURITY MEASURE, PLEASE EXERCISE CAUTION WITH E-MAIL CONTENT AND ANY LINKS OR ATTACHMENTS. On 08.03.19 14:47, Lange Norbert wrote: Not reproducible here with stable/3.0.x or next, and with ipipe-x86- 4.14.y. What are your parameters? Not entirely upstream, but based on ipipe-core-4.14.89-x86-2 and xenomai master, the difference is contained in the rt_igb driver, which is not even loaded. Defconfig is attached. I mostly suspect glibc as the relevant difference, I am using glibc-2.28-69-g1e5c5303a522764d7e9d2302a60e4a32cdb902f1. Looking at the strace the child process 1039 is stuck at FUTEX_WAIT_PRIVATE, Don’t really know how to tackle this. Is this a regression? Then try bisecting the causing commit. Not really, I haven't ran the smokey suite often (and only for specific tests). I use buildroot for my rootfs, so I have no easy way of swapping around glibc versions. I have not encountered other issues outside of those tests. Which version of glibc is running on your end? Mine is in fact old: 2.19. Colleagues are on Debian 9 with 2.24 where this used to work but was not tested recently against master. I haven't tried buildroot with Xenomai yet - is there a working 3.x recipe, and everything is out-of-the-box? Regarding how to possibly debug this: If one thread is stuck, you could check if there are other threads in that application that may hold the lock. Or if the lock content is actually invalid and therefore blocking the caller (memory corruption). You should be able to read out the thread ID of the owner from a valid lock structure. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux
Re: INTR-REMAP error with UDD driver
On 08.03.19 15:07, Jeff Webb wrote: On Friday, March 8, 2019 1:14 AM, Jan Kiszka wrote: On 08.03.19 02:35, Jeff Webb via Xenomai wrote: I have implemented a simple UDD mini-driver (interrupt handler) and an associated Xenomai userspace test program. This pair of programs works as intended on one machine. When I tried running the same programs on another machine, I got this error when the first interrupt occurred: kernel: [ 75.122305] DMAR: DRHD: handling fault status reg 2 kernel: [ 75.123175] DMAR: [INTR-REMAP] Request device [f0:1f.0] fault index ee00 [fault reason 37] Blocked a compatibility format interrupt request This means the device is under IOMMU interrupt remapping regime, but it was programmed to use a standard MSI/MSI-X request, rather than requesting that parameters (address/data tuple) from the Linux interrupt management layer that will assign a remapping slot and hand out the right values. Now you interrupt is stuck in the filter. Before suggesting the workaround/hack to disable interrupt remapping: How did you program MSI/-X? Via proper Linux pci_* helpers? The PCI card I am working with does not implement MSI interrupts, but just uses the legacy interrupt pin A. It's confusing to me that this error is associated with MSI. Since this behavior is slot-dependent, I figured at first that the error is from another device on the same interrupt line, but I get the same error on several different slots. There is only one slot where things work as expected. Ok, then we may have an issue with how we register legacy INTx at the IOAPIC when interrupt remapping is on. Basically, the IOAPIC sends MSI-like messages as well that are subject to remapping. So its registers need to be programmed with the right translatable values, just like the MSI registers of a device. Linux does that, it's in charge of the IOMMU. I don't recall how we (Xenomai/I-pipe) fill the IOAPIC if a line was not used before under Linux. Possibly, there is the wrong Linux function triggered. But maybe it's also something different. I assume you tested that your card would work find if it used only regular Linux driver services, no RTDM, right? That would be good in order to exclude a general platform issue, maybe a BIOS (ACPI) bug in describing which PCI slot is under which IOMMU regime. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux
Re: INTR-REMAP error with UDD driver
On Friday, March 8, 2019 1:14 AM, Jan Kiszka wrote: > On 08.03.19 02:35, Jeff Webb via Xenomai wrote: > > > I have implemented a simple UDD mini-driver (interrupt handler) and an > > associated Xenomai userspace test program. This pair of programs works as > > intended on one machine. When I tried running the same programs on another > > machine, I got this error when the first interrupt occurred: > > kernel: [ 75.122305] DMAR: DRHD: handling fault status reg 2 > > kernel: [ 75.123175] DMAR: [INTR-REMAP] Request device [f0:1f.0] fault > > index ee00 [fault reason 37] Blocked a compatibility format interrupt > > request > > This means the device is under IOMMU interrupt remapping regime, but it was > programmed to use a standard MSI/MSI-X request, rather than requesting that > parameters (address/data tuple) from the Linux interrupt management layer that > will assign a remapping slot and hand out the right values. Now you interrupt > is > stuck in the filter. > > Before suggesting the workaround/hack to disable interrupt remapping: How did > you program MSI/-X? Via proper Linux pci_* helpers? The PCI card I am working with does not implement MSI interrupts, but just uses the legacy interrupt pin A. It's confusing to me that this error is associated with MSI. Since this behavior is slot-dependent, I figured at first that the error is from another device on the same interrupt line, but I get the same error on several different slots. There is only one slot where things work as expected. Thanks for the response, -Jeff > > Jan > > > I would guess that the [f0:1f.0] in the message is a PCI bus address, but I > > don't see this via lspci. The closest match is: > > 00:1f.0 ISA bridge [0601]: Intel Corporation Q67 Express Chipset LPC > > Controller [8086:1c4e] (rev 05) > > After the first attempt, I see no more of these messages until I reboot, > > but the UDD driver never receives an interrupt in any case. > > I tried moving my PCI card to five different slots, and finally found one > > where everything work properly on the second machine. I can at continue to > > make progress using the working slot, but I would really like to resolve > > the issue with the other slots. Does anyone have any idea what to try next? > > I am using these sources: > > linux-4.14.89.tar > > ipipe-core-4.14.89-x86-2.patch > > xenomai-3.0.8.tar > > I have attached the boot log, config, and cpu information. > > Thanks, > > -Jeff Webb > > -- > > Siemens AG, Corporate Technology, CT RDA IOT SES-DE > Corporate Competence Center Embedded Linux
RE: smokey's fork tests hangs?
> -Original Message- > From: Jan Kiszka > Sent: Freitag, 8. März 2019 14:59 > To: Lange Norbert ; Xenomai > (xenomai@xenomai.org) > Subject: Re: smokey's fork tests hangs? > > E-MAIL FROM A NON-ANDRITZ SOURCE: AS A SECURITY MEASURE, PLEASE > EXERCISE CAUTION WITH E-MAIL CONTENT AND ANY LINKS OR > ATTACHMENTS. > > > On 08.03.19 14:47, Lange Norbert wrote: > > > >> > >> Not reproducible here with stable/3.0.x or next, and with ipipe-x86- > 4.14.y. > >> What are your parameters? > > > > Not entirely upstream, but based on ipipe-core-4.14.89-x86-2 and > > xenomai master, the difference is contained in the rt_igb driver, which is > not even loaded. > > Defconfig is attached. > > > > I mostly suspect glibc as the relevant difference, I am using > > glibc-2.28-69-g1e5c5303a522764d7e9d2302a60e4a32cdb902f1. > > > > Looking at the strace the child process 1039 is stuck at > > FUTEX_WAIT_PRIVATE, Don’t really know how to tackle this. > > Is this a regression? Then try bisecting the causing commit. Not really, I haven't ran the smokey suite often (and only for specific tests). I use buildroot for my rootfs, so I have no easy way of swapping around glibc versions. I have not encountered other issues outside of those tests. Which version of glibc is running on your end? This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. ANDRITZ HYDRO GmbH Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation Firmensitz/ Registered seat: Wien Firmenbuchgericht/ Court of registry: Handelsgericht Wien Firmenbuchnummer/ Company registration: FN 61833 g DVR: 0605077 UID-Nr.: ATU14756806 Thank You
Re: smokey's fork tests hangs?
On 08.03.19 14:47, Lange Norbert wrote: Not reproducible here with stable/3.0.x or next, and with ipipe-x86-4.14.y. What are your parameters? Not entirely upstream, but based on ipipe-core-4.14.89-x86-2 and xenomai master, the difference is contained in the rt_igb driver, which is not even loaded. Defconfig is attached. I mostly suspect glibc as the relevant difference, I am using glibc-2.28-69-g1e5c5303a522764d7e9d2302a60e4a32cdb902f1. Looking at the strace the child process 1039 is stuck at FUTEX_WAIT_PRIVATE, Don’t really know how to tackle this. Is this a regression? Then try bisecting the causing commit. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux
RE: smokey's fork tests hangs?
> > Not reproducible here with stable/3.0.x or next, and with ipipe-x86-4.14.y. > What are your parameters? Not entirely upstream, but based on ipipe-core-4.14.89-x86-2 and xenomai master, the difference is contained in the rt_igb driver, which is not even loaded. Defconfig is attached. I mostly suspect glibc as the relevant difference, I am using glibc-2.28-69-g1e5c5303a522764d7e9d2302a60e4a32cdb902f1. Looking at the strace the child process 1039 is stuck at FUTEX_WAIT_PRIVATE, Don’t really know how to tackle this. Norbert This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. ANDRITZ HYDRO GmbH Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation Firmensitz/ Registered seat: Wien Firmenbuchgericht/ Court of registry: Handelsgericht Wien Firmenbuchnummer/ Company registration: FN 61833 g DVR: 0605077 UID-Nr.: ATU14756806 Thank You -- next part -- A non-text attachment was scrubbed... Name: defconfig_rtnet_static Type: application/octet-stream Size: 13373 bytes Desc: defconfig_rtnet_static URL: <http://xenomai.org/pipermail/xenomai/attachments/20190308/5b62cac5/attachment.obj> -- next part -- A non-text attachment was scrubbed... Name: strace.tar.xz Type: application/octet-stream Size: 2740 bytes Desc: strace.tar.xz URL: <http://xenomai.org/pipermail/xenomai/attachments/20190308/5b62cac5/attachment-0001.obj>
Re: Question about real-time networking
On 08.03.19 10:05, 김병진 via Xenomai wrote: HelloI wanna use tcp or udp for real-time network.i found an example from /home/kraptor/xenomai-v3.0.8/demo/posix/cobalt/eth_p_allBut it seems like not using rtnet.https://gitlab.denx.de/Xenomai/xenomai/wikis/RTnetWiki says that i need to install Rtnet separately.Is it right? I am trying to build rtnet tcp example without install.Cause all function I need is in /xenomai/kernel/drivers/net/stack/But what should I include or link?(cflags or ldflags)Is there anyone who can help me?Thank you. Something is mangling your emails... RTnet is part of Xenomai, no need for a separate installation anymore. Where exactly is our wiki suggesting something different? For rttcp, you need to use the development branch (master). Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux
Re: smokey's fork tests hangs?
On 07.03.19 15:55, Lange Norbert via Xenomai wrote: Hello, I have problems with the fork tests, they both hang for me. # xeno smokey --run=11 --verbose=2 # xeno smokey --run=20 --verbose=2 no leak withthread no leak withmutex no leak withcond no leak withsem no leak withnamed sem no leak withtimer no leak withmq When started under gdb, the parent is running into waitpid, The child is stuck at an innocent locking snippet [1]. #0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:63 #1 0x77ed74de in __GI___register_atfork ( prepare=0x77f5edc0 , prepare@entry=0x0, parent=parent@entry=0x0, child=child@entry=0x77f9f4e0 , dso_handle=0x77de0e5b <__lll_lock_wait_private+27>) at register-atfork.c:40 #2 0x77faca9d in __pthread_atfork (prepare=prepare@entry=0x0, parent=parent@entry=0x0, child=child@entry=0x77f9f4e0 ) at pthread_atfork.c:51 #3 0x77f9f307 in __cobalt_init () at /home/lano/buildroot/build/xenomai/lib/cobalt/init.c:212 #4 cobalt_init () at /home/lano/buildroot/build/xenomai/lib/cobalt/init.c:252 #5 0x77f9f4fc in cobalt_fork_handler () at /home/lano/buildroot/build/xenomai/lib/cobalt/init.c:192 #6 0x77ed7898 in __run_fork_handlers (who=who@entry=atfork_run_child) at register-atfork.c:134 #7 0x77e9b04a in __libc_fork () at ../sysdeps/nptl/fork.c:137 #8 0x00408d0b in ?? () #9 0x004051d1 in main () [1] - https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/x86_64/lowlevellock.S;h=71dd740202b1d6501a2ffd929aa99cbd99a019ef;hb=1e5c5303a522764d7e9d2302a60e4a32cdb902f1 Not reproducible here with stable/3.0.x or next, and with ipipe-x86-4.14.y. What are your parameters? Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux
Re: net/rtdev: Commit e972e78e reintroduces memory corruption fixed by 74464ee3
On 08.03.19 13:47, Jouko Haapaluoma wrote: Hello I have been investigating kernel memory corruption issues in our system (Xenomai 2.6.4 and RTnet 0.9.13.2). I traced the root cause into the rtdev_map_rtskb() function where a linked list is always grown but not shrinked. You have recently created a patch that fixes this: 74464ee3. Additionally, there seemed to be a regression and that was fixed in e972e78e. However, this regression fix reintroduces the memory corruption in cases where the rtdev does not have the rtdev->map_rtskb function defined. Therefore we end up in the situation that was before the 74464ee3: The rtskb_mapwait_list will grow but not shrink. If sockets are destroyed and recreated, the rtskb_mapwait_list->next will point to a destroyed skb object which causes a use-after-free memory corruption when calling list_add(). Hmpf, right, too bad. I suppose we should be fine by dropping the rtskb from the waitlist in rtdev_unmap_rtskb if it had no dma address set. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux
net/rtdev: Commit e972e78e reintroduces memory corruption fixed by 74464ee3
Hello I have been investigating kernel memory corruption issues in our system (Xenomai 2.6.4 and RTnet 0.9.13.2). I traced the root cause into the rtdev_map_rtskb() function where a linked list is always grown but not shrinked. You have recently created a patch that fixes this: 74464ee3. Additionally, there seemed to be a regression and that was fixed in e972e78e. However, this regression fix reintroduces the memory corruption in cases where the rtdev does not have the rtdev->map_rtskb function defined. Therefore we end up in the situation that was before the 74464ee3: The rtskb_mapwait_list will grow but not shrink. If sockets are destroyed and recreated, the rtskb_mapwait_list->next will point to a destroyed skb object which causes a use-after-free memory corruption when calling list_add(). Best Regards, Jouko Haapaluoma Senior Software Designer Embedded Linux Technology Specialist Wapice Ltd, Finland
Re: x86_64 kernel does not start under qemu
On 08.03.19 12:19, Richard Weinberger wrote: Am Mittwoch, 6. März 2019, 17:39:38 CET schrieb Jan Kiszka: On 06.03.19 16:33, Richard Weinberger wrote: Am Mittwoch, 6. März 2019, 14:43:55 CET schrieb Jan Kiszka: On 06.03.19 14:10, Jan Kiszka wrote: On 06.03.19 12:28, Lange Norbert via Xenomai wrote: Hello, I have a similar issue on real hardware (cc Philippe). Can you try booting with notscdeadline? I would be surprised if it's that: QEMU does not emulate this APIC feature, only KVM does. I need to reproduce. Which QEMU version? Just booted current ipipe-x86-4.14.y (4.14.103) in QEMU 3.1+ (some development snapshot) with -smp 4 - works fine. But these are likely quite a few variations from your setup. I'm on qemu 2.11.2. That one might still be serializing SMP, thus only use one core on the host. Well, why is this a problem for Xenomai? Does it deadlock if you have more than one cpu and these are not truly parallel? This seems a little odd to me. I the past, this de-parallelization triggered race conditions more easily as it widened the race windows massively. Whatever came out of it remained a Xenomai or I-pipe bug that could theoretically occur on real hardware as well. Currently trying to emulate that as it changes timing. Just retried with latest qemu (v3.1.0-2421-g9b748c5e061b), same problem. Yeah, I suspect the kernel config. Didn't have time to try defconfig so far. Is this plain ipipe? Just figured that you need to prepare the kernel with Xenomai to trigger the problem. Xenomai is 3.0.8. I have Xenomai stable/v3.0.x running as well. What kernel config are you using and what is your qemu command line? Currently used config attached, command line: qemu-system-x86_64 -drive file=/path/disk.img,discard=unmap,if=none,id=disk -device ide-hd,drive=disk -snapshot -m 1G -serial mon:stdio -s -smp 4 -machine q35,pit=off -fsdev local,path=/path/shared,security_model=none,id=vfs -device virtio-9p-pci,addr=1f.7,mount_tag=host,fsdev=vfs Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux -- next part -- A non-text attachment was scrubbed... Name: .config.xz Type: application/x-xz Size: 22936 bytes Desc: not available URL: <http://xenomai.org/pipermail/xenomai/attachments/20190308/e293ce8f/attachment.bin>
Re: x86_64 kernel does not start under qemu
Am Mittwoch, 6. März 2019, 17:39:38 CET schrieb Jan Kiszka: > On 06.03.19 16:33, Richard Weinberger wrote: > > Am Mittwoch, 6. März 2019, 14:43:55 CET schrieb Jan Kiszka: > >> On 06.03.19 14:10, Jan Kiszka wrote: > >>> On 06.03.19 12:28, Lange Norbert via Xenomai wrote: > Hello, > > I have a similar issue on real hardware (cc Philippe). Can you try > booting > with notscdeadline? > > >>> > >>> I would be surprised if it's that: QEMU does not emulate this APIC > >>> feature, only > >>> KVM does. > >>> > >>> I need to reproduce. Which QEMU version? > >>> > >> > >> Just booted current ipipe-x86-4.14.y (4.14.103) in QEMU 3.1+ (some > >> development > >> snapshot) with -smp 4 - works fine. But these are likely quite a few > >> variations > >> from your setup. > > > > I'm on qemu 2.11.2. > > That one might still be serializing SMP, thus only use one core on the host. Well, why is this a problem for Xenomai? Does it deadlock if you have more than one cpu and these are not truly parallel? This seems a little odd to me. > Currently trying to emulate that as it changes timing. Just retried with latest qemu (v3.1.0-2421-g9b748c5e061b), same problem. > > Is this plain ipipe? Just figured that you need to prepare the kernel > > with Xenomai to trigger the problem. > > Xenomai is 3.0.8. > > I have Xenomai stable/v3.0.x running as well. What kernel config are you using and what is your qemu command line? Thanks, //richard
Question about real-time networking
HelloI wanna use tcp or udp for real-time network.i found an example from /home/kraptor/xenomai-v3.0.8/demo/posix/cobalt/eth_p_allBut it seems like not using rtnet.https://gitlab.denx.de/Xenomai/xenomai/wikis/RTnetWiki says that i need to install Rtnet separately.Is it right? I am trying to build rtnet tcp example without install.Cause all function I need is in /xenomai/kernel/drivers/net/stack/But what should I include or link?(cflags or ldflags)Is there anyone who can help me?Thank you.
