Re: 5.10-dovetail regression?
Philippe Gerum writes:
> Jan Kiszka writes:
>
>> On 07.04.22 17:24, Philippe Gerum wrote:
>>>
>>> Jan Kiszka writes:
>>>
Hi Philippe,
does this already ring some bell?
https://source.denx.de/Xenomai/xenomai-images/-/jobs/419210
Only triggers with qemu-amd64, not on real HW and not with 5.15.
>>>
>>> I could not reproduce locally, but visual inspection revealed something
>>> fishy in #8e2c09ee5323. Could you try this on the failing kernel? TIA,
>>>
>>> diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
>>> index 2651c6cfd034..da6735d45a8a 100644
>>> --- a/kernel/time/clockevents.c
>>> +++ b/kernel/time/clockevents.c
>>> @@ -644,8 +644,8 @@ void clockevents_exchange_device(struct
>>> clock_event_device *old,
>>> * to the release list, keep it around but mark it as
>>> * reserved.
>>> */
>>> + list_del(>list);
>>> if (tick_check_is_proxy(new)) {
>>> - list_del(>list);
>>> clockevents_switch_state(old, CLOCK_EVT_STATE_RESERVED);
>>> } else {
>>> clockevents_switch_state(old, CLOCK_EVT_STATE_DETACHED);
>>>
>>
>> Didn't reproduce locally for me as well, though using the same image.
>> But the patch helped on the CI system.
>>
>
> It does not seem to be enough though, that patch fixes a different bug
> actually. So there are two of them:
>
> 1. lockup when running "corectl --stop" on 5.10/kvm_x86 configurations,
> not reproducible here on any other setup
>
> 2. list poisoning which triggers an assertion at boot on "some" x86
> configurations
>
> The patch above definitely fixes #1, makes sense. I managed to reproduce
> #2 on real hw, with kernel 5.15 this time. Same gremlin:
>
> [2.052096] smpboot: Estimated ratio of average max frequency by base
> frequency (times 1024): 1152
> [2.052273] [ cut here ]
> [2.053250] list_del corruption, 8881001ce0b8->next is LIST_POISON1
> (dead0100)
> [2.053250] WARNING: CPU: 0 PID: 1 at lib/list_debug.c:45
> __list_del_entry_valid+0x81/0xe0
> [2.053250] Modules linked in:
> [2.053250] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.15.32+ #100
> [2.053250] Hardware name: TQ-Group TQMxE39M/Type2 - Board Product Name,
> BIOS 5.12.09.16.05 07/26/2017
> [2.053250] IRQ stage: Linux
> [2.053250] RIP: 0010:__list_del_entry_valid+0x81/0xe0
> [2.053250] Code: 85 c5 ff 49 8b 55 08 4c 39 e2 75 5b b8 01 00 00 00 5d 41
> 5c 41 5d c3 4c 89 ea 48 8d 75 00 48 c7 c7 80 99 80 ad e8 ea fb 83 00 <0f> 0b
> 5d 41 5c 31 c0 41 5d c3 49 8d 14 24 48 8d 75 00 48 c7 c7 e0
> [2.053250] RSP: :888100287dc0 EFLAGS: 00010246
> [2.053250] RAX: RBX: 8881001ce000 RCX:
>
> [2.053250] RDX: 0002 RSI: 0008 RDI:
> ed1020050fae
> [2.053250] RBP: 8881001ce0b8 R08: ac22b384 R09:
> ac279120
> [2.053250] R10: 888100287aaf R11: ed1020050f55 R12:
> dead0122
> [2.053250] R13: dead0100 R14: 0002 R15:
> adff62a0
> [2.053250] FS: () GS:88815c80()
> knlGS:
> [2.053250] CS: 0010 DS: ES: CR0: 80050033
> [2.053250] CR2: 888104e01000 CR3: 000103e1 CR4:
> 003506f0
> [2.053250] Call Trace:
> [2.053250]
> [2.053250] clockevents_exchange_device+0x16c/0x2a0
> [2.053250] tick_check_new_device+0x1c3/0x230
> [2.053250] clockevents_register_device+0xc3/0x170
> [2.053250] setup_boot_APIC_clock+0x526/0x553
> [2.053250] ? default_ioapic_phys_id_map+0x40/0x40
> [2.053250] native_smp_prepare_cpus+0x2cd/0x3ef
> [2.053250] kernel_init_freeable+0xc0/0x290
> [2.053250] ? rest_init+0xe0/0xe0
> [2.053250] kernel_init+0x19/0x130
> [2.053250] ret_from_fork+0x22/0x30
> [2.053250]
>
> I'm on it.
Ok, so the first patch is not a fix, it's plain nonsense and is
responsible for the second issue in my test case. Back to square
#1. Still on it.
--
Philippe.
Re: 5.10-dovetail regression?
Jan Kiszka writes:
> On 07.04.22 17:24, Philippe Gerum wrote:
>>
>> Jan Kiszka writes:
>>
>>> Hi Philippe,
>>>
>>> does this already ring some bell?
>>>
>>> https://source.denx.de/Xenomai/xenomai-images/-/jobs/419210
>>>
>>> Only triggers with qemu-amd64, not on real HW and not with 5.15.
>>>
>>
>> I could not reproduce locally, but visual inspection revealed something
>> fishy in #8e2c09ee5323. Could you try this on the failing kernel? TIA,
>>
>> diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
>> index 2651c6cfd034..da6735d45a8a 100644
>> --- a/kernel/time/clockevents.c
>> +++ b/kernel/time/clockevents.c
>> @@ -644,8 +644,8 @@ void clockevents_exchange_device(struct
>> clock_event_device *old,
>> * to the release list, keep it around but mark it as
>> * reserved.
>> */
>> +list_del(>list);
>> if (tick_check_is_proxy(new)) {
>> -list_del(>list);
>> clockevents_switch_state(old, CLOCK_EVT_STATE_RESERVED);
>> } else {
>> clockevents_switch_state(old, CLOCK_EVT_STATE_DETACHED);
>>
>
> Didn't reproduce locally for me as well, though using the same image.
> But the patch helped on the CI system.
>
It does not seem to be enough though, that patch fixes a different bug
actually. So there are two of them:
1. lockup when running "corectl --stop" on 5.10/kvm_x86 configurations,
not reproducible here on any other setup
2. list poisoning which triggers an assertion at boot on "some" x86
configurations
The patch above definitely fixes #1, makes sense. I managed to reproduce
#2 on real hw, with kernel 5.15 this time. Same gremlin:
[2.052096] smpboot: Estimated ratio of average max frequency by base
frequency (times 1024): 1152
[2.052273] [ cut here ]
[2.053250] list_del corruption, 8881001ce0b8->next is LIST_POISON1
(dead0100)
[2.053250] WARNING: CPU: 0 PID: 1 at lib/list_debug.c:45
__list_del_entry_valid+0x81/0xe0
[2.053250] Modules linked in:
[2.053250] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.15.32+ #100
[2.053250] Hardware name: TQ-Group TQMxE39M/Type2 - Board Product Name,
BIOS 5.12.09.16.05 07/26/2017
[2.053250] IRQ stage: Linux
[2.053250] RIP: 0010:__list_del_entry_valid+0x81/0xe0
[2.053250] Code: 85 c5 ff 49 8b 55 08 4c 39 e2 75 5b b8 01 00 00 00 5d 41
5c 41 5d c3 4c 89 ea 48 8d 75 00 48 c7 c7 80 99 80 ad e8 ea fb 83 00 <0f> 0b 5d
41 5c 31 c0 41 5d c3 49 8d 14 24 48 8d 75 00 48 c7 c7 e0
[2.053250] RSP: :888100287dc0 EFLAGS: 00010246
[2.053250] RAX: RBX: 8881001ce000 RCX:
[2.053250] RDX: 0002 RSI: 0008 RDI: ed1020050fae
[2.053250] RBP: 8881001ce0b8 R08: ac22b384 R09: ac279120
[2.053250] R10: 888100287aaf R11: ed1020050f55 R12: dead0122
[2.053250] R13: dead0100 R14: 0002 R15: adff62a0
[2.053250] FS: () GS:88815c80()
knlGS:
[2.053250] CS: 0010 DS: ES: CR0: 80050033
[2.053250] CR2: 888104e01000 CR3: 000103e1 CR4: 003506f0
[2.053250] Call Trace:
[2.053250]
[2.053250] clockevents_exchange_device+0x16c/0x2a0
[2.053250] tick_check_new_device+0x1c3/0x230
[2.053250] clockevents_register_device+0xc3/0x170
[2.053250] setup_boot_APIC_clock+0x526/0x553
[2.053250] ? default_ioapic_phys_id_map+0x40/0x40
[2.053250] native_smp_prepare_cpus+0x2cd/0x3ef
[2.053250] kernel_init_freeable+0xc0/0x290
[2.053250] ? rest_init+0xe0/0xe0
[2.053250] kernel_init+0x19/0x130
[2.053250] ret_from_fork+0x22/0x30
[2.053250]
I'm on it.
--
Philippe.
Re: 5.10-dovetail regression?
On 07.04.22 17:24, Philippe Gerum wrote:
>
> Jan Kiszka writes:
>
>> Hi Philippe,
>>
>> does this already ring some bell?
>>
>> https://source.denx.de/Xenomai/xenomai-images/-/jobs/419210
>>
>> Only triggers with qemu-amd64, not on real HW and not with 5.15.
>>
>
> I could not reproduce locally, but visual inspection revealed something
> fishy in #8e2c09ee5323. Could you try this on the failing kernel? TIA,
>
> diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
> index 2651c6cfd034..da6735d45a8a 100644
> --- a/kernel/time/clockevents.c
> +++ b/kernel/time/clockevents.c
> @@ -644,8 +644,8 @@ void clockevents_exchange_device(struct
> clock_event_device *old,
>* to the release list, keep it around but mark it as
>* reserved.
>*/
> + list_del(>list);
> if (tick_check_is_proxy(new)) {
> - list_del(>list);
> clockevents_switch_state(old, CLOCK_EVT_STATE_RESERVED);
> } else {
> clockevents_switch_state(old, CLOCK_EVT_STATE_DETACHED);
>
Didn't reproduce locally for me as well, though using the same image.
But the patch helped on the CI system.
Thanks,
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
Re: 5.10-dovetail regression?
Jan Kiszka writes:
> Hi Philippe,
>
> does this already ring some bell?
>
> https://source.denx.de/Xenomai/xenomai-images/-/jobs/419210
>
> Only triggers with qemu-amd64, not on real HW and not with 5.15.
>
I could not reproduce locally, but visual inspection revealed something
fishy in #8e2c09ee5323. Could you try this on the failing kernel? TIA,
diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
index 2651c6cfd034..da6735d45a8a 100644
--- a/kernel/time/clockevents.c
+++ b/kernel/time/clockevents.c
@@ -644,8 +644,8 @@ void clockevents_exchange_device(struct clock_event_device
*old,
* to the release list, keep it around but mark it as
* reserved.
*/
+ list_del(>list);
if (tick_check_is_proxy(new)) {
- list_del(>list);
clockevents_switch_state(old, CLOCK_EVT_STATE_RESERVED);
} else {
clockevents_switch_state(old, CLOCK_EVT_STATE_DETACHED);
--
Philippe.
Re: 5.10-dovetail regression?
Philippe Gerum writes: > a > Jan Kiszka writes: > >> Hi Philippe, >> >> does this already ring some bell? >> >> https://source.denx.de/Xenomai/xenomai-images/-/jobs/419210 >> >> Only triggers with qemu-amd64, not on real HW and not with 5.15. >> >> Jan > > 8e2c09ee5323 is most likely causing this. It's a backport of the fix > developed for 5.15. I have a kvm-aarch64 setup which I routinely use > too, I'll reproduce and fix this. Sorry, I mean x86_64, not aarch64. -- Philippe.
Re: 5.10-dovetail regression?
a Jan Kiszka writes: > Hi Philippe, > > does this already ring some bell? > > https://source.denx.de/Xenomai/xenomai-images/-/jobs/419210 > > Only triggers with qemu-amd64, not on real HW and not with 5.15. > > Jan 8e2c09ee5323 is most likely causing this. It's a backport of the fix developed for 5.15. I have a kvm-aarch64 setup which I routinely use too, I'll reproduce and fix this. -- Philippe.
5.10-dovetail regression?
Hi Philippe, does this already ring some bell? https://source.denx.de/Xenomai/xenomai-images/-/jobs/419210 Only triggers with qemu-amd64, not on real HW and not with 5.15. Jan -- Siemens AG, Technology Competence Center Embedded Linux
