Re: Enable/Disable of ftrace events crashes kernel

[Richard Weinberger via Xenomai]

On Thu, Jul 11, 2019 at 11:20 PM Jan Kiszka  wrote:
>
> On 11.07.19 22:48, Richard Weinberger wrote:
> > On Thu, Jul 11, 2019 at 8:30 PM Jan Kiszka  wrote:
> >>
> >> On 11.07.19 12:25, Richard Weinberger wrote:
> >>> On Thu, Jul 11, 2019 at 12:21 PM Jan Kiszka  
> >>> wrote:
>  Can't reproduce so far, even with a while-true loop. Can you share your 
>  .config?
> >>>
> >>> Sure, see attachment.
> >>>
> >>
> >> This seems to fix the issue here:
> >>
> >> diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
> >> index 119fd66d111e..8f647c208cf2 100644
> >> --- a/arch/x86/entry/entry_64.S
> >> +++ b/arch/x86/entry/entry_64.S
> >> @@ -997,8 +997,8 @@ apicinterrupt IRQ_WORK_VECTOR   
> >> irq_work_interrupt  smp_irq_work_interrupt
> >>  \skip_label:
> >> UNWIND_HINT_REGS
> >> DISABLE_INTERRUPTS(CLBR_ANY)
> >> -   testl   %ebx, %ebx  /* %ebx: return to kernel mode */
> >> -   jnz retint_kernel_early
> >> +   testb   $3, CS(%rsp)
> >> +   jz  retint_kernel_early
> >> jmp retint_user_early
> >> .endif
> >>  1001:
> >>
> >> Tests welcome!
> >
> > With that change I can no longer trigger the crash.
>
> Perfect.
>
> > Can you please give more context? I'd like to understand the problem.
> >
>
> We were basing the decision whether to switch GS on return or not on a stale
> register (ebx). That register used to contain the information, but that 
> changed
> with "x86/entry/64: Remove %ebx handling from error_entry/exit". This caused 
> CPU
> state corruptions under certain conditions, apparently only when dealing with
> #DB exceptions, not with the way more frequent #PF.

Ah! Upstream b3681dd548d0 ("x86/entry/64: Remove %ebx handling from
error_entry/exit")
changed ebx to CS. Now things make sense again. :-)

Thanks for the quick fix and the explanation!

-- 
Thanks,
//richard

Re: Enable/Disable of ftrace events crashes kernel

[Jan Kiszka via Xenomai]

On 11.07.19 22:48, Richard Weinberger wrote:
> On Thu, Jul 11, 2019 at 8:30 PM Jan Kiszka  wrote:
>>
>> On 11.07.19 12:25, Richard Weinberger wrote:
>>> On Thu, Jul 11, 2019 at 12:21 PM Jan Kiszka  wrote:
 Can't reproduce so far, even with a while-true loop. Can you share your 
 .config?
>>>
>>> Sure, see attachment.
>>>
>>
>> This seems to fix the issue here:
>>
>> diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
>> index 119fd66d111e..8f647c208cf2 100644
>> --- a/arch/x86/entry/entry_64.S
>> +++ b/arch/x86/entry/entry_64.S
>> @@ -997,8 +997,8 @@ apicinterrupt IRQ_WORK_VECTOR   
>> irq_work_interrupt  smp_irq_work_interrupt
>>  \skip_label:
>> UNWIND_HINT_REGS
>> DISABLE_INTERRUPTS(CLBR_ANY)
>> -   testl   %ebx, %ebx  /* %ebx: return to kernel mode */
>> -   jnz retint_kernel_early
>> +   testb   $3, CS(%rsp)
>> +   jz  retint_kernel_early
>> jmp retint_user_early
>> .endif
>>  1001:
>>
>> Tests welcome!
> 
> With that change I can no longer trigger the crash.

Perfect.

> Can you please give more context? I'd like to understand the problem.
> 

We were basing the decision whether to switch GS on return or not on a stale
register (ebx). That register used to contain the information, but that changed
with "x86/entry/64: Remove %ebx handling from error_entry/exit". This caused CPU
state corruptions under certain conditions, apparently only when dealing with
#DB exceptions, not with the way more frequent #PF.

The issue is also present in 4.14, but in 4.4 and the unmaintained 4.9 as I
first thought.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Enable/Disable of ftrace events crashes kernel

[Richard Weinberger via Xenomai]

On Thu, Jul 11, 2019 at 8:30 PM Jan Kiszka  wrote:
>
> On 11.07.19 12:25, Richard Weinberger wrote:
> > On Thu, Jul 11, 2019 at 12:21 PM Jan Kiszka  wrote:
> >> Can't reproduce so far, even with a while-true loop. Can you share your 
> >> .config?
> >
> > Sure, see attachment.
> >
>
> This seems to fix the issue here:
>
> diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
> index 119fd66d111e..8f647c208cf2 100644
> --- a/arch/x86/entry/entry_64.S
> +++ b/arch/x86/entry/entry_64.S
> @@ -997,8 +997,8 @@ apicinterrupt IRQ_WORK_VECTOR   
> irq_work_interrupt  smp_irq_work_interrupt
>  \skip_label:
> UNWIND_HINT_REGS
> DISABLE_INTERRUPTS(CLBR_ANY)
> -   testl   %ebx, %ebx  /* %ebx: return to kernel mode */
> -   jnz retint_kernel_early
> +   testb   $3, CS(%rsp)
> +   jz  retint_kernel_early
> jmp retint_user_early
> .endif
>  1001:
>
> Tests welcome!

With that change I can no longer trigger the crash.
Can you please give more context? I'd like to understand the problem.

-- 
Thanks,
//richard

Re: Enable/Disable of ftrace events crashes kernel

[Jan Kiszka via Xenomai]

On 11.07.19 20:30, Jan Kiszka wrote:
> On 11.07.19 12:25, Richard Weinberger wrote:
>> On Thu, Jul 11, 2019 at 12:21 PM Jan Kiszka  wrote:
>>> Can't reproduce so far, even with a while-true loop. Can you share your 
>>> .config?
>>
>> Sure, see attachment.
>>
> 
> This seems to fix the issue here:
> 
> diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
> index 119fd66d111e..8f647c208cf2 100644
> --- a/arch/x86/entry/entry_64.S
> +++ b/arch/x86/entry/entry_64.S
> @@ -997,8 +997,8 @@ apicinterrupt IRQ_WORK_VECTOR 
> irq_work_interrupt  smp_irq_work_interrupt
>  \skip_label:
>   UNWIND_HINT_REGS
>   DISABLE_INTERRUPTS(CLBR_ANY)
> - testl   %ebx, %ebx  /* %ebx: return to kernel mode */
> - jnz retint_kernel_early
> + testb   $3, CS(%rsp)
> + jz  retint_kernel_early
>   jmp retint_user_early
>   .endif
>  1001:
> 
> Tests welcome!
> 
> Interestingly, 4.14 should have the same problem, but I failed to
> reproduce there so far.

Uhh, it's a regression in all our x86 stable trees, due to a backport of an
upstream commit. The above is definitely correct and hopefully also the fix for
this issue.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Enable/Disable of ftrace events crashes kernel

[Jan Kiszka via Xenomai]

On 11.07.19 12:25, Richard Weinberger wrote:
> On Thu, Jul 11, 2019 at 12:21 PM Jan Kiszka  wrote:
>> Can't reproduce so far, even with a while-true loop. Can you share your 
>> .config?
> 
> Sure, see attachment.
> 

This seems to fix the issue here:

diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 119fd66d111e..8f647c208cf2 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -997,8 +997,8 @@ apicinterrupt IRQ_WORK_VECTOR   
irq_work_interrupt  smp_irq_work_interrupt
 \skip_label:
UNWIND_HINT_REGS
DISABLE_INTERRUPTS(CLBR_ANY)
-   testl   %ebx, %ebx  /* %ebx: return to kernel mode */
-   jnz retint_kernel_early
+   testb   $3, CS(%rsp)
+   jz  retint_kernel_early
jmp retint_user_early
.endif
 1001:

Tests welcome!

Interestingly, 4.14 should have the same problem, but I failed to
reproduce there so far.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: Enable/Disable of ftrace events crashes kernel

[Richard Weinberger via Xenomai]

On Thu, Jul 11, 2019 at 12:21 PM Jan Kiszka  wrote:
> Can't reproduce so far, even with a while-true loop. Can you share your 
> .config?

Sure, see attachment.

-- 
Thanks,
//richard
-- next part --
A non-text attachment was scrubbed...
Name: .config
Type: application/x-config
Size: 121667 bytes
Desc: not available
URL: 

Re: Enable/Disable of ftrace events crashes kernel

[Jan Kiszka via Xenomai]

On 11.07.19 00:29, Richard Weinberger via Xenomai wrote:
> Hi!
> 
> I can reliable kill Linux on qemu by writing a few times 1 and 0 to
> /sys/kernel/debug/tracing/events/cobalt_core/enable
> 
> Didn't test on real hardware so far.
> The following splat happened on ipipe-core-4.19.57-x86-3 plus
> xenomai-git as of today.
> 
> [   33.664656] Kernel panic - not syncing: Machine halted.
> [   33.665323] CPU: 2 PID: 2088 Comm: bash Not tainted 4.19.57 #1
> [   33.666142] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> BIOS rel-1.11.0-0-g63451fc-prebuilt.qemu-project.org 04/01/2014
> [   33.667524] I-pipe domain: Linux
> [   33.667895] Call Trace:
> [   33.668354]  <#DF>
> [   33.668695]  dump_stack+0x8e/0xb3
> [   33.669104]  panic+0xdd/0x238
> [   33.669456]  df_debug+0x24/0x30
> [   33.669834]  do_double_fault+0x95/0x120
> [   33.670323]  double_fault+0x3f/0x60
> [   33.670794] RIP: 0010:xnintr_core_clock_handler+0xad/0x370
> [   33.671426] Code: c0 48 09 c2 49 89 96 80 1a 00 00 49 8d ae 88 1a
> 00 00 48 8d 59 08 48 87 5d 00 48 c7 c0 d0 e3 02 00 48 83 01 01 cc 1f
> 44 00 00 <41> 8b 86 10 03 00 00 49 81 4e 08 00 40 00 00 83 c0 01 41 89
> 86 10
> [   33.673615] RSP: 0018:964ebbb03f58 EFLAGS: 00010002
> [   33.674235] RAX: 0002e3d0 RBX: 964ebbb315c0 RCX: 
> 964ebbb3bb00
> [   33.675079] RDX: 0013d41dbbce RSI: fc25fc34 RDI: 
> 964ebbb315c0
> [   33.675923] RBP: 964ebbb31748 R08: 964ebb000249 R09: 
> 0002e320
> [   33.676761] R10: 0040 R11:  R12: 
> 0002
> [   33.677600] R13: 0002fcc0 R14: 964ebbb2fcc0 R15: 
> 964ebbb2fcc0
> [   33.678444]  
> [   33.678704]  
> [   33.678955]  dispatch_irq_head+0x84/0x110
> [   33.679437]  __ipipe_handle_irq+0x7c/0x1d0
> [   33.679927]  apic_timer_interrupt+0x12/0x40
> [   33.680448]  
> [   33.680805] RIP: 0010:smp_call_function_many+0x1e0/0x250
> [   33.681505] Code: 5f 97 00 3b 05 d5 70 47 01 0f 83 99 fe ff ff 48
> 63 c8 48 8b 13 48 03 14 cd 00 b7 c9 ac 8b 4a 18 83 e1 01 74 0a f3 90
> 8b 4a 18 <83> e1 01 75 f6 eb c8 48 c7 c2 20 b9 f5 ac 48 89 ee 89 df e8
> b8 5f
> [   33.684312] RSP: 0018:a2478079bc00 EFLAGS: 0202 ORIG_RAX:
> ff13
> [   33.685347] RAX: 0001 RBX: 964ebbb35a00 RCX: 
> 0003
> [   33.686198] RDX: 964ebbab9c80 RSI:  RDI: 
> 964ebbb35a08
> [   33.687044] RBP: 964ebbb35a08 R08: 000b R09: 
> aba22300
> [   33.687883] R10: a2478079bc20 R11: f000 R12: 
> aba22200
> [   33.688725] R13:  R14: 0001 R15: 
> 0040
> [   33.689577]  ? optimize_nops+0xe0/0xe0
> [   33.690055]  ? alternatives_text_reserved+0x60/0x60
> [   33.690643]  ? optimize_nops+0xe0/0xe0
> [   33.691092]  ? xnintr_core_clock_handler+0xa9/0x370
> [   33.691657]  ? trace_event_raw_event_irq_event+0xa0/0xa0
> [   33.692489]  on_each_cpu+0x23/0x50
> [   33.692902]  ? xnintr_core_clock_handler+0xa8/0x370
> [   33.693464]  text_poke_bp+0x63/0xe0
> [   33.693875]  __jump_label_transform.isra.0+0x12f/0x140
> [   33.694466]  arch_jump_label_transform+0x26/0x40
> [   33.695093]  __jump_label_update+0x78/0xb0
> [   33.695567]  static_key_slow_inc_cpuslocked+0x83/0x90
> [   33.696147]  static_key_slow_inc+0x11/0x20
> [   33.696622]  tracepoint_probe_register_prio+0x214/0x290
> [   33.697241]  __ftrace_event_enable_disable+0x96/0x260
> [   33.697905]  __ftrace_set_clr_event_nolock+0xe8/0x130
> [   33.698488]  system_enable_write+0xb3/0xf0
> [   33.698537] BUG: Unhandled exception over domain Xenomai at
> 0xabb5413d - switching to ROOT
> [   33.699032]  __vfs_write+0x31/0x180
> [   33.700443]  ? selinux_file_permission+0x118/0x130
> [   33.700979]  ? security_file_permission+0x27/0xb0
> [   33.701491]  vfs_write+0xa8/0x190
> [   33.701856]  ksys_write+0x55/0xd0
> [   33.702220]  do_syscall_64+0x64/0x160
> [   33.702644]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [   33.703210] RIP: 0033:0x7fcc38f5bd04
> [   33.703603] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f
> 1f 80 00 00 00 00 8b 05 2a fb 2c 00 48 63 ff 85 c0 75 13 b8 01 00 00
> 00 0f 05 <48> 3d 00 f0 ff ff 77 54 f3 c3 66 90 55 53 48 89 d5 48 89 f3
> 48 83
> [   33.705712] RSP: 002b:7ffd5b051008 EFLAGS: 0246 ORIG_RAX:
> 0001
> [   33.706673] RAX: ffda RBX: 0002 RCX: 
> 7fcc38f5bd04
> [   33.707552] RDX: 0002 RSI: 564c21421700 RDI: 
> 0001
> [   33.708399] RBP: 564c21421700 R08: 000a R09: 
> 
> [   33.709264] R10: 000a R11: 0246 R12: 
> 0002
> [   33.710197] R13: 0001 R14: 7fcc39227720 R15: 
> 0002
> [   33.711080] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.19.57 #1
> [   33.711974] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> BIOS rel-1.11.0-0-g63451fc-prebuilt.qemu-project.org 04/01/2014
> [