clocksource 'tsc' marked unstable, CLOCK_HOST_REALTIME went bananas
Hello list! I am running Xenomai 3.0.7. released version with Kernel 4.9.90 patchlevel 6. We have a new x86-64 hardware, and during a test-run the clocksource went unstable, see (1) below for dmesg. After this, calls to "clock_gettime(CLOCK_HOST_REALTIME,&timeStamp)" went bananas. Like if time stopped completely, (or possibly jumped back in time at least a few hours). Stuff running on timers (using CLOCK_MONOTONIC)kept running fine. I tried switching time to hpet using "clocksource=hpet" but this didn't work at all. Either time (CLOCK_HOST_REALTIME) jumped back and forth or it completely stopped. Now I'm using "tsc=reliable" and haven't seen any issues since. I think I know why tsc may be deemed unreliable, but why can't I run using hpet? What are the drawbacks for xenomai with using tsc=reliable ? - (1) - [19139.502750] clocksource: timekeeping watchdog on CPU3: Marking clocksource 'tsc' as unstable because the skew is too large: [19139.503499] clocksource: 'hpet' wd_now: f347622d wd_last: 35ab9afb mask: [19139.504245] clocksource: 'tsc' cs_now: 31567c07fb36 cs_last: 30ffd3d5ec2d mask: [19139.505045] clocksource: Switched to clocksource hpet - (1) - Best regards Per Öberg
RTnet packet loss issue
Preface: Xenomai 2.6 Vanilla kernel 3.8.13 i686 RTnet 0.9.13 I have two PCs (M and S) connected via Ethernet exclusively for RTnet use. M--[switch]---S M's NIC is driven by the rt_r8169 driver S's NIC by the rt_e1000 driver The problem is that the connection is unstable and about 80% of the packets get lost. root@machinectrl:~# rtping 192.168.0.1 Real-time PING 192.168.0.1 56(84) bytes of data. 64 bytes from 192.168.0.1: icmp_seq=1 time=111.2 us 64 bytes from 192.168.0.1: icmp_seq=7 time=99.2 us 64 bytes from 192.168.0.1: icmp_seq=13 time=99.1 us 64 bytes from 192.168.0.1: icmp_seq=23 time=100.5 us 64 bytes from 192.168.0.1: icmp_seq=24 time=97.5 us 64 bytes from 192.168.0.1: icmp_seq=30 time=104.8 us 64 bytes from 192.168.0.1: icmp_seq=32 time=101.8 us 64 bytes from 192.168.0.1: icmp_seq=34 time=102.1 us 64 bytes from 192.168.0.1: icmp_seq=35 time=101.4 us 64 bytes from 192.168.0.1: icmp_seq=37 time=101.5 us 64 bytes from 192.168.0.1: icmp_seq=38 time=99.7 us 64 bytes from 192.168.0.1: icmp_seq=43 time=99.2 us 64 bytes from 192.168.0.1: icmp_seq=54 time=101.3 us 64 bytes from 192.168.0.1: icmp_seq=60 time=102.2 us 64 bytes from 192.168.0.1: icmp_seq=63 time=97.3 us 64 bytes from 192.168.0.1: icmp_seq=69 time=100.3 us 64 bytes from 192.168.0.1: icmp_seq=80 time=101.2 us 64 bytes from 192.168.0.1: icmp_seq=84 time=100.7 us 64 bytes from 192.168.0.1: icmp_seq=86 time=103.4 us 64 bytes from 192.168.0.1: icmp_seq=88 time=105.1 us 64 bytes from 192.168.0.1: icmp_seq=89 time=101.2 us 64 bytes from 192.168.0.1: icmp_seq=96 time=100.7 us 64 bytes from 192.168.0.1: icmp_seq=98 time=99.2 us 64 bytes from 192.168.0.1: icmp_seq=99 time=103.2 us 64 bytes from 192.168.0.1: icmp_seq=113 time=98.8 us 64 bytes from 192.168.0.1: icmp_seq=116 time=105.7 us 64 bytes from 192.168.0.1: icmp_seq=119 time=111.2 us 64 bytes from 192.168.0.1: icmp_seq=126 time=101.7 us 64 bytes from 192.168.0.1: icmp_seq=130 time=102.6 us ^C --- 192.168.0.1 rtping statistics --- 135 packets transmitted, 29 received, 79% packet loss worst case rtt = 111.2 us This happens sometimes from the start after both machines are booted without prior communication. If it is not broken from the start the RTnet works as expected. But it also happens after the system is running for several hours without any noticeable user action that would cause it. The cables and connectors have all been double checked. Is this a know error or is there anything you can tell me about this pattern? I'm grateful for every piece of advice as always. Cheers, Johannes
Re: latency test
I've done some testing with the X-window suggestions at: https://gitlab.denx.de/Xenomai/xenomai/wikis/Troubleshooting#the_latency_test_shows_high_latencies My results on an Intel PC with Xenomai 3.0.7, Linux 4.9, Debian 9.7 with xfce4, worst-case latencies: - Using Intel driver: 39.8 us - Using fbdev, 34.8 us - Setting Option "NoAccel", both with Intel and fbdev driver: 22.6 us - No GUI: 6.3 us So setting "NoAccel" helps in my case, but I'm also looking forward to further improve it. Bart On Thu, 28 Mar 2019 at 07:29, Alec Ari via Xenomai wrote: > Firefox bumps latency up too, I wonder if the Xenomai pros have any advice > on this. Good question! It'd be nice to do video editing and 4K encoding > while doing real-time processes too. > > > Alec > >
Re: [Xenomai] ipipe-4.14: planned release date
On 28.03.19 01:32, Virendra Kate wrote: >>> Did you all get an update on this issue ? Was it resolved after all? I have >>> the exact same hardware with the exact same issue. The board keeps resetting >>> after "Loading initial ramdisk..." >>> >>> My Setup is as follows: >>> ipipe version: ipipe-core-4.14.89-x86-2 >>> Linux-kernel version: linux-4.14.89 >>> xenomai version: xenomai-3.0.8 >>> Hardware: AMD® Ryzen embedded v1756b >>> >>> The board works fine with Linux kernel 3.9.146 patched with the appropriate >>> xenomai-ipipe files. I would like to move to linux-kernel_4.14 as the GPU >>> driver is officially supported by AMD for linux-kernel_4.14. >>> >> >> No direct progress regarding Ryzen, but I will try to glue together a new 4.14 >> version if pending patches soon so that you can try if that happens to help. >> Will let you know. >> > >Just done, see https://gitlab.denx.de/Xenomai/ipipe-x86/ The same issue persists. 1. I downloaded the kernel-patched with ipipe from the above link. 2. Added the co-kernel using: ../xenomai-3.0.8/scripts/prepare-kernel.sh --arch=x86_64 3. Made the appropriate kernel config changes. I have attached my .config file. To exclude or narow-down config-related issues, you could try starting of with the x86_64_defconfig, only adding essential Xenomai and board-related switches. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux
Kernel module symbol conflicts
Hello, We use a setup with multiple network controllers, and so far only one of them is realtime capable. The issue is, that there are nonRt and Rt drivers active at the same time (igb statically builtin and rt_igb as module). And while it works, I got some irregular weird crashes. Is there an issue with the loaded rt_igb module having a lot symbols that are identically named with the igb driver? Does that mean the rt_igp module will use the exported functions/variables from the igb driver, as a normal dynamic linker would? Whats the policy of xenomai, should this even work (igp and rt_igp at the same time)? Example of conflicting symbols: # cat /proc/kallsyms | grep ' T igb_config' 94f3b2e0 T igb_configure_tx_ring 94f3b6f0 T igb_configure_rx_ring 94f49c30 T igb_config_collision_dist 94f49d50 T igb_config_fc_after_link_up # nm /lib/modules/4.14.94-xeno2-static/kernel/drivers/xenomai/net/drivers/igb/rt_igb.ko | grep ' T igb_config' 4db0 T igb_config_collision_dist 4eb0 T igb_config_fc_after_link_up dd60 T igb_configure_rx_ring d9f0 T igb_configure_tx_ring Mit besten Grüßen / Kind regards NORBERT LANGE AT-DES ANDRITZ HYDRO GmbH Eibesbrunnergasse 20 1120 Vienna / AUSTRIA p: +43 50805 56684 norbert.la...@andritz.com andritz.com This message and any attachments are solely for the use of the intended recipients. They may contain privileged and/or confidential information or other information protected from disclosure. If you are not an intended recipient, you are hereby notified that you received this email in error and that any review, dissemination, distribution or copying of this email and any attachment is strictly prohibited. If you have received this email in error, please contact the sender and delete the message and any attachment from your system. ANDRITZ HYDRO GmbH Rechtsform/ Legal form: Gesellschaft mit beschränkter Haftung / Corporation Firmensitz/ Registered seat: Wien Firmenbuchgericht/ Court of registry: Handelsgericht Wien Firmenbuchnummer/ Company registration: FN 61833 g DVR: 0605077 UID-Nr.: ATU14756806 Thank You
Re: [Xenomai] ipipe-4.14: planned release date
Am Thu, 28 Mar 2019 11:19:06 +0100 schrieb Jan Kiszka : > On 28.03.19 01:32, Virendra Kate wrote: > > > > >>> Did you all get an update on this issue ? Was it resolved > > >>> after all? I have the exact same hardware with the exact same > > >>> issue. The board keeps resetting after "Loading initial > > >>> ramdisk..." > > >>> > > >>> My Setup is as follows: > > >>> ipipe version: ipipe-core-4.14.89-x86-2 > > >>> Linux-kernel version: linux-4.14.89 > > >>> xenomai version: xenomai-3.0.8 > > >>> Hardware: AMD® Ryzen embedded v1756b > > >>> > > >>> The board works fine with Linux kernel 3.9.146 patched with > > >>> the appropriate xenomai-ipipe files. I would like to move to > > >>> linux-kernel_4.14 as the GPU driver is officially supported by > > >>> AMD for linux-kernel_4.14. > > >> > > >> No direct progress regarding Ryzen, but I will try to glue > > >> together a new 4.14 version if pending patches soon so that you > > >> can try if that happens to help. Will let you know. > > >> > > > > > >Just done, see https://gitlab.denx.de/Xenomai/ipipe-x86/ > > > > The same issue persists. > > > > 1. I downloaded the kernel-patched with ipipe from the above link. > > 2. Added the co-kernel > > using: ../xenomai-3.0.8/scripts/prepare-kernel.sh --arch=x86_64 > > 3. Made the appropriate kernel config changes. > > > > I have attached my .config file. > > To exclude or narow-down config-related issues, you could try > starting of with the x86_64_defconfig, only adding essential Xenomai > and board-related switches. And an unpatched mainline kernel of the ~ same version, with the ~same config would be interesting as well. Henning > Jan >
Issues With rt_task_inquire and Associates
Folks, I noted a couple of issues with rt_task_inquire and wanted to share them. I will put together a patch later , but wanted to let folks know sooner rather than later. The documentation suggests that the info structure can be NULL. This is simply not true, as the actual code dereferences the pointer without checking. Thus a null info pointer will cause an exception and the thread will die an untimely (though outwardly quiet) death. I also note that the rt_task_inquire routine has an exit path that keeps the thread object in task control block locked. This is also a problem. There should really be a put_alchemy_task before the "goto out;" after the threadobj_stat call. I noted that this "keeps the tcb->thobj locked" issue occurs in a few of the other rt_task routines as well. A careful review is in order and I will do it when I submit the patches for rt_task_inquire. Warmest Regards, Steve
Re: INTR-REMAP error with UDD driver
Hey guys, I was wondering if there is an update on the INTR-REMAP UDD driver issue. We have not had any troubleshooting breakthroughs on our end. Best Regards, Jim Elliott From: Xenomai on behalf of Jeff Webb via Xenomai Sent: Friday, March 8, 2019 8:07 AM To: Jan Kiszka Cc: xenomai@xenomai.org Subject: Re: INTR-REMAP error with UDD driver On Friday, March 8, 2019 1:14 AM, Jan Kiszka wrote: > On 08.03.19 02:35, Jeff Webb via Xenomai wrote: > > > I have implemented a simple UDD mini-driver (interrupt handler) and an > > associated Xenomai userspace test program. This pair of programs works as > > intended on one machine. When I tried running the same programs on another > > machine, I got this error when the first interrupt occurred: > > kernel: [ 75.122305] DMAR: DRHD: handling fault status reg 2 > > kernel: [ 75.123175] DMAR: [INTR-REMAP] Request device [f0:1f.0] fault > > index ee00 [fault reason 37] Blocked a compatibility format interrupt > > request > > This means the device is under IOMMU interrupt remapping regime, but it was > programmed to use a standard MSI/MSI-X request, rather than requesting that > parameters (address/data tuple) from the Linux interrupt management layer that > will assign a remapping slot and hand out the right values. Now you interrupt > is > stuck in the filter. > > Before suggesting the workaround/hack to disable interrupt remapping: How did > you program MSI/-X? Via proper Linux pci_* helpers? The PCI card I am working with does not implement MSI interrupts, but just uses the legacy interrupt pin A. It's confusing to me that this error is associated with MSI. Since this behavior is slot-dependent, I figured at first that the error is from another device on the same interrupt line, but I get the same error on several different slots. There is only one slot where things work as expected. Thanks for the response, -Jeff > > Jan > > > I would guess that the [f0:1f.0] in the message is a PCI bus address, but I > > don't see this via lspci. The closest match is: > > 00:1f.0 ISA bridge [0601]: Intel Corporation Q67 Express Chipset LPC > > Controller [8086:1c4e] (rev 05) > > After the first attempt, I see no more of these messages until I reboot, > > but the UDD driver never receives an interrupt in any case. > > I tried moving my PCI card to five different slots, and finally found one > > where everything work properly on the second machine. I can at continue to > > make progress using the working slot, but I would really like to resolve > > the issue with the other slots. Does anyone have any idea what to try next? > > I am using these sources: > > linux-4.14.89.tar > > ipipe-core-4.14.89-x86-2.patch > > xenomai-3.0.8.tar > > I have attached the boot log, config, and cpu information. > > Thanks, > > -Jeff Webb > > -- > > Siemens AG, Corporate Technology, CT RDA IOT SES-DE > Corporate Competence Center Embedded Linux CONFIDENTIALITY NOTICE: The contents of this e-mail message may be privileged, confidential, proprietary, Controlled Unclassified Information (CUI) or otherwise protected against disclosure or dissemination. This e-mail message and any files transmitted with it are intended solely for the use of the individual/individuals or entity/entities to whom they are addressed. If you are not the intended recipient of this message, any dissemination, distribution, printing or copying is strictly prohibited. If you have received this message in error, please e-mail or call the sender (jim.elli...@nta-inc.net, ) and destroy all copies.
Re: INTR-REMAP error with UDD driver
- Den 28 mar 2019, på kl 18:10, xenomai xenomai@xenomai.org skrev: > Hey guys, I was wondering if there is an update on the INTR-REMAP UDD driver > issue. We have not had any troubleshooting breakthroughs on our end. > Best Regards, > Jim Elliott > > From: Xenomai on behalf of Jeff Webb via Xenomai > > Sent: Friday, March 8, 2019 8:07 AM > To: Jan Kiszka > Cc: xenomai@xenomai.org > Subject: Re: INTR-REMAP error with UDD driver > On Friday, March 8, 2019 1:14 AM, Jan Kiszka wrote: > > On 08.03.19 02:35, Jeff Webb via Xenomai wrote: >> > I have implemented a simple UDD mini-driver (interrupt handler) and an >> > associated Xenomai userspace test program. This pair of programs works as >> > intended on one machine. When I tried running the same programs on another > > > machine, I got this error when the first interrupt occurred: > > > kernel: [ 75.122305] DMAR: DRHD: handling fault status reg 2 >> > kernel: [ 75.123175] DMAR: [INTR-REMAP] Request device [f0:1f.0] fault >> > index > > > ee00 [fault reason 37] Blocked a compatibility format interrupt request > > This means the device is under IOMMU interrupt remapping regime, but it was > > programmed to use a standard MSI/MSI-X request, rather than requesting that > > parameters (address/data tuple) from the Linux interrupt management layer > > that > > will assign a remapping slot and hand out the right values. Now you > > interrupt is > > stuck in the filter. > > Before suggesting the workaround/hack to disable interrupt remapping: How > > did > > you program MSI/-X? Via proper Linux pci_* helpers? > The PCI card I am working with does not implement MSI interrupts, but just > uses > the legacy interrupt pin A. It's confusing to me that this error is associated > with MSI. Since this behavior is slot-dependent, I figured at first that the > error is from another device on the same interrupt line, but I get the same > error on several different slots. There is only one slot where things work as > expected. When I read this I realize I have a similar or the same issue. When I moved from 4.9.38 to 4.9.90 the xenomai_peak_pci driver stopped getting interrupts and I got a note about io-remapping and a blocked compability interrupt. It worked on another motherboard. The kernel-configs were not equal, and I haven't had time to troubleshoot this. When I switched to the new Peak driver it worked so I focused on other things instead. I may be able to contribute in the future. Best regards Per Öberg > Thanks for the response, > -Jeff > > Jan >> > I would guess that the [f0:1f.0] in the message is a PCI bus address, but I > > > don't see this via lspci. The closest match is: >> > 00:1f.0 ISA bridge [0601]: Intel Corporation Q67 Express Chipset LPC >> > Controller > > > [8086:1c4e] (rev 05) >> > After the first attempt, I see no more of these messages until I reboot, >> > but the > > > UDD driver never receives an interrupt in any case. >> > I tried moving my PCI card to five different slots, and finally found one >> > where >> > everything work properly on the second machine. I can at continue to make >> > progress using the working slot, but I would really like to resolve the >> > issue > > > with the other slots. Does anyone have any idea what to try next? > > > I am using these sources: > > > linux-4.14.89.tar > > > ipipe-core-4.14.89-x86-2.patch > > > xenomai-3.0.8.tar > > > I have attached the boot log, config, and cpu information. > > > Thanks, > > > -Jeff Webb > > -- > > Siemens AG, Corporate Technology, CT RDA IOT SES-DE > > Corporate Competence Center Embedded Linux > > CONFIDENTIALITY NOTICE: The contents of this e-mail message may be privileged, > confidential, proprietary, Controlled Unclassified Information (CUI) or > otherwise protected against disclosure or dissemination. This e-mail message > and any files transmitted with it are intended solely for the use of the > individual/individuals or entity/entities to whom they are addressed. If you > are not the intended recipient of this message, any dissemination, > distribution, printing or copying is strictly prohibited. If you have received > this message in error, please e-mail or call the sender > (jim.elli...@nta-inc.net, ) and destroy all copies.
irqchip tps65217 is not pipeline-safe!
Hi there, booting a 4.14.94-ti kernel with Cobalt 3.0.8 from the "stable/v3.0.x" branch, on an am3358 (PocketBeagle) I get a warning during boot "irqchip tps65217 is not pipeline-safe!". The tps65217 is the power-management IC for the board. What does this message mean exactly? Is it something to be concerned about? The board has behaved fine when running Xenomai tasks, in spite of this issue. Thanks, Giulio Relevant log below, full `dmesg` can be found here https://github.com/RobertCNelson/ti-linux-kernel-dev/issues/31#issuecomment-470118685 [2.010215] [ cut here ] [2.010255] WARNING: CPU: 0 PID: 64 at kernel/irq/chip.c:57 irq_set_chip+0xcc/0xdc [2.010262] irqchip tps65217 is not pipeline-safe! [2.010270] Modules linked in: [2.010292] CPU: 0 PID: 64 Comm: kworker/0:2 Not tainted 4.14.94-ti-xenomai-r95 #1 [2.010298] Hardware name: Generic AM33XX (Flattened Device Tree) [2.010305] I-pipe domain: Linux [2.010332] Workqueue: events deferred_probe_work_func [2.010377] [] (unwind_backtrace) from [] (show_stack+0x20/0x24) [2.010395] [] (show_stack) from [] (dump_stack+0x24/0x28) [2.010424] [] (dump_stack) from [] (__warn+0xf4/0x10c) [2.010440] [] (__warn) from [] (warn_slowpath_fmt+0x58/0x74) [2.010456] [] (warn_slowpath_fmt) from [] (irq_set_chip+0xcc/0xdc) [2.010473] [] (irq_set_chip) from [] (irq_set_chip_and_handler_name+0x24/0x3c) [2.010497] [] (irq_set_chip_and_handler_name) from [] (tps65217_irq_map+0x40/0x78) [2.010515] [] (tps65217_irq_map) from [] (irq_domain_associate+0x84/0x1f8) [2.010529] [] (irq_domain_associate) from [] (irq_create_mapping+0x144/0x1fc) [2.010548] [] (irq_create_mapping) from [] (mfd_add_device+0x204/0x348) [2.010563] [] (mfd_add_device) from [] (mfd_add_devices+0xa0/0x104) [2.010577] [] (mfd_add_devices) from [] (devm_mfd_add_devices+0x78/0xb8) [2.010592] [] (devm_mfd_add_devices) from [] (tps65217_probe+0x110/0x328) [2.010608] [] (tps65217_probe) from [] (i2c_device_probe+0x290/0x2dc) [2.010625] [] (i2c_device_probe) from [] (driver_probe_device+0x298/0x478) [2.010668] [] (driver_probe_device) from [] (__device_attach_driver+0xac/0x14c) [2.010683] [] (__device_attach_driver) from [] (bus_for_each_drv+0x68/0xc8) [2.010698] [] (bus_for_each_drv) from [] (__device_attach+0xe0/0x174) [2.010713] [] (__device_attach) from [] (device_initial_probe+0x1c/0x20) [2.010727] [] (device_initial_probe) from [] (bus_probe_device+0x94/0x9c) [2.010742] [] (bus_probe_device) from [] (device_add+0x418/0x618) [2.010756] [] (device_add) from [] (device_register+0x24/0x28) [2.010770] [] (device_register) from [] (i2c_new_device+0x154/0x308) [2.010787] [] (i2c_new_device) from [] (of_i2c_register_device+0x148/0x1fc) [2.010800] [] (of_i2c_register_device) from [] (of_i2c_register_devices+0x98/0x118) [2.010814] [] (of_i2c_register_devices) from [] (i2c_register_adapter+0x184/0x404) [2.010828] [] (i2c_register_adapter) from [] (__i2c_add_numbered_adapter+0x88/0x138) [2.010840] [] (__i2c_add_numbered_adapter) from [] (i2c_add_adapter+0xe8/0x154) [2.010853] [] (i2c_add_adapter) from [] (i2c_add_numbered_adapter+0x2c/0x30) [2.010869] [] (i2c_add_numbered_adapter) from [] (omap_i2c_probe+0x524/0x718) [2.010885] [] (omap_i2c_probe) from [] (platform_drv_probe+0x60/0xbc) [2.010900] [] (platform_drv_probe) from [] (driver_probe_device+0x298/0x478) [2.010915] [] (driver_probe_device) from [] (__device_attach_driver+0xac/0x14c) [2.010928] [] (__device_attach_driver) from [] (bus_for_each_drv+0x68/0xc8) [2.010942] [] (bus_for_each_drv) from [] (__device_attach+0xe0/0x174) [2.010956] [] (__device_attach) from [] (device_initial_probe+0x1c/0x20) [2.010970] [] (device_initial_probe) from [] (bus_probe_device+0x94/0x9c) [2.010984] [] (bus_probe_device) from [] (deferred_probe_work_func+0x74/0x19c) [2.011008] [] (deferred_probe_work_func) from [] (process_one_work+0x178/0x4e4) [2.011025] [] (process_one_work) from [] (worker_thread+0x26c/0x5ec) [2.011042] [] (worker_thread) from [] (kthread+0x168/0x170) [2.011064] [] (kthread) from [] (ret_from_fork+0x18/0x24) [2.011073] ---[ end trace 7699ea00e776f627 ]--- [2.014786] input: tps65217_pwr_but as /devices/platform/ocp/44e0b000.i2c/i2c-0/0-0024/tps65217-pwrbutton/input/input0 [2.015419] tps65217 0-0024: TPS65217 ID 0xe version 1.2
