Re: [PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions

[Linus Torvalds]

On Fri, Sep 2, 2016 at 7:46 AM, Steven Rostedt  wrote:
>
> Can someone please pull this in. And mark it for stable, it goes
> probably as far back as 2.6.32.

Applied.

 Linus
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions

[Steven Rostedt]

I just spent half a day bisecting function tracing because I tripped
over this again. I thought this was merged, but I guess it was missed
again.

Can someone please pull this in. And mark it for stable, it goes
probably as far back as 2.6.32.

Thanks!

-- Steve

On Wed, 25 May 2016 13:47:26 -0400
Steven Rostedt  wrote:

> Łukasz Daniluk reported that on a RHEL kernel that his machine would lock up
> after enabling function tracer. I asked him to bisect the functions within
> available_filter_functions, which he did and it came down to three:
> 
>  _paravirt_nop(), _paravirt_ident_32() and _paravirt_ident_64()
> 
> It was found that this is only an issue when noreplace-paravirt is added to
> the kernel command line.
> 
> This means that those functions are most likely called within critical
> sections of the funtion tracer, and must not be traced.
> 
> In newer kenels _paravirt_nop() is defined within gcc asm(), and is no
> longer an issue. But both _paravirt_ident_{32,64}() causes the following
> splat when they are traced:
> 
>  mm/pgtable-generic.c:33: bad pmd 8800d2435150(01d00054)
>  mm/pgtable-generic.c:33: bad pmd 8800d3624190(01d00070)
>  mm/pgtable-generic.c:33: bad pmd 8800d36a5110(01d00054)
>  mm/pgtable-generic.c:33: bad pmd 880118eb1450(01d00054)
>  NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [systemd-journal:469]
>  Modules linked in: e1000e
>  CPU: 2 PID: 469 Comm: systemd-journal Not tainted 4.6.0-rc4-test+ #513
>  Hardware name: Hewlett-Packard HP Compaq Pro 6300 SFF/339A, BIOS K01 v02.05 
> 05/07/2012
>  task: 880118f740c0 ti: 8800d4aec000 task.ti: 8800d4aec000
>  RIP: 0010:[]  [] 
> queued_spin_lock_slowpath+0x118/0x1a0
>  RSP: 0018:8800d4aefb90  EFLAGS: 0246
>  RAX:  RBX:  RCX: 88011eb16d40
>  RDX: 82485760 RSI: 1f288820 RDI: ea008030
>  RBP: 8800d4aefb90 R08: 000c R09: 
>  R10: 821c8e0e R11:  R12: 88200fb8
>  R13: 7f7a4e3f7000 R14: ea000303f600 R15: 8800d4b562e0
>  FS:  7f7a4e3d7840() GS:88011eb0() knlGS:
>  CS:  0010 DS:  ES:  CR0: 80050033
>  CR2: 7f7a4e3f7000 CR3: d3e71000 CR4: 001406e0
>  Stack:
>   8800d4aefba0 81cc5f47 8800d4aefc60 8122c15b
>   8800d4aefcb0 8800d4aefbd0 811bf4cb 0002
>   0015 8800d2276050 8000c0fd8867 ea008030
>  Call Trace:
>   [] _raw_spin_lock+0x27/0x30
>   [] handle_pte_fault+0x13db/0x16b0
>   [] ? function_trace_call+0x15b/0x180
>   [] ? handle_pte_fault+0x5/0x16b0
>   [] handle_mm_fault+0x312/0x670
>   [] ? find_vma+0x68/0x70
>   [] __do_page_fault+0x1b1/0x4e0
>   [] do_page_fault+0x22/0x30
>   [] page_fault+0x28/0x30
>   [] ? copy_user_enhanced_fast_string+0x5/0x10
>   [] ? seq_read+0x305/0x370
>   [] __vfs_read+0x28/0xe0
>   [] ? __vfs_read+0x5/0xe0
>   [] ? __vfs_read+0x5/0xe0
>   [] vfs_read+0x86/0x130
>   [] SyS_read+0x46/0xa0
>   [] entry_SYSCALL_64_fastpath+0x1e/0xa8
>  Code: 12 48 c1 ea 0c 83 e8 01 83 e2 30 48 98 48 81 c2 40 6d 01 00 48 03 14
>  c5 80 6a 5d 82 48 89 0a 8b 41 08 85 c0 75 09 f3 90 8b 41 08 <85> c0 74 f7
>  4c 8b 09 4d 85 c9 74 08 41 0f 18 09 eb 02 f3 90 8b
> 
> Reported-by: Łukasz Daniluk 
> Signed-off-by: Steven Rostedt 
> 
> diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c
> index f08ac28b8136..f975d226be6e 100644
> --- a/arch/x86/kernel/paravirt.c
> +++ b/arch/x86/kernel/paravirt.c
> @@ -55,12 +55,12 @@ asm (".pushsection .entry.text, \"ax\"\n"
>   ".popsection");
>  
>  /* identity function, which can be inlined */
> -u32 _paravirt_ident_32(u32 x)
> +u32 notrace _paravirt_ident_32(u32 x)
>  {
>   return x;
>  }
>  
> -u64 _paravirt_ident_64(u64 x)
> +u64 notrace _paravirt_ident_64(u64 x)
>  {
>   return x;
>  }

___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions

[Steven Rostedt]

Hmm, I'm guessing this patch got lost.

-- Steve


On Wed, 25 May 2016 13:47:26 -0400
Steven Rostedt  wrote:

> Łukasz Daniluk reported that on a RHEL kernel that his machine would lock up
> after enabling function tracer. I asked him to bisect the functions within
> available_filter_functions, which he did and it came down to three:
> 
>  _paravirt_nop(), _paravirt_ident_32() and _paravirt_ident_64()
> 
> It was found that this is only an issue when noreplace-paravirt is added to
> the kernel command line.
> 
> This means that those functions are most likely called within critical
> sections of the funtion tracer, and must not be traced.
> 
> In newer kenels _paravirt_nop() is defined within gcc asm(), and is no
> longer an issue. But both _paravirt_ident_{32,64}() causes the following
> splat when they are traced:
> 
>  mm/pgtable-generic.c:33: bad pmd 8800d2435150(01d00054)
>  mm/pgtable-generic.c:33: bad pmd 8800d3624190(01d00070)
>  mm/pgtable-generic.c:33: bad pmd 8800d36a5110(01d00054)
>  mm/pgtable-generic.c:33: bad pmd 880118eb1450(01d00054)
>  NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [systemd-journal:469]
>  Modules linked in: e1000e
>  CPU: 2 PID: 469 Comm: systemd-journal Not tainted 4.6.0-rc4-test+ #513
>  Hardware name: Hewlett-Packard HP Compaq Pro 6300 SFF/339A, BIOS K01 v02.05 
> 05/07/2012
>  task: 880118f740c0 ti: 8800d4aec000 task.ti: 8800d4aec000
>  RIP: 0010:[]  [] 
> queued_spin_lock_slowpath+0x118/0x1a0
>  RSP: 0018:8800d4aefb90  EFLAGS: 0246
>  RAX:  RBX:  RCX: 88011eb16d40
>  RDX: 82485760 RSI: 1f288820 RDI: ea008030
>  RBP: 8800d4aefb90 R08: 000c R09: 
>  R10: 821c8e0e R11:  R12: 88200fb8
>  R13: 7f7a4e3f7000 R14: ea000303f600 R15: 8800d4b562e0
>  FS:  7f7a4e3d7840() GS:88011eb0() knlGS:
>  CS:  0010 DS:  ES:  CR0: 80050033
>  CR2: 7f7a4e3f7000 CR3: d3e71000 CR4: 001406e0
>  Stack:
>   8800d4aefba0 81cc5f47 8800d4aefc60 8122c15b
>   8800d4aefcb0 8800d4aefbd0 811bf4cb 0002
>   0015 8800d2276050 8000c0fd8867 ea008030
>  Call Trace:
>   [] _raw_spin_lock+0x27/0x30
>   [] handle_pte_fault+0x13db/0x16b0
>   [] ? function_trace_call+0x15b/0x180
>   [] ? handle_pte_fault+0x5/0x16b0
>   [] handle_mm_fault+0x312/0x670
>   [] ? find_vma+0x68/0x70
>   [] __do_page_fault+0x1b1/0x4e0
>   [] do_page_fault+0x22/0x30
>   [] page_fault+0x28/0x30
>   [] ? copy_user_enhanced_fast_string+0x5/0x10
>   [] ? seq_read+0x305/0x370
>   [] __vfs_read+0x28/0xe0
>   [] ? __vfs_read+0x5/0xe0
>   [] ? __vfs_read+0x5/0xe0
>   [] vfs_read+0x86/0x130
>   [] SyS_read+0x46/0xa0
>   [] entry_SYSCALL_64_fastpath+0x1e/0xa8
>  Code: 12 48 c1 ea 0c 83 e8 01 83 e2 30 48 98 48 81 c2 40 6d 01 00 48 03 14
>  c5 80 6a 5d 82 48 89 0a 8b 41 08 85 c0 75 09 f3 90 8b 41 08 <85> c0 74 f7
>  4c 8b 09 4d 85 c9 74 08 41 0f 18 09 eb 02 f3 90 8b
> 
> Reported-by: Łukasz Daniluk 
> Signed-off-by: Steven Rostedt 
> 
> diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c
> index f08ac28b8136..f975d226be6e 100644
> --- a/arch/x86/kernel/paravirt.c
> +++ b/arch/x86/kernel/paravirt.c
> @@ -55,12 +55,12 @@ asm (".pushsection .entry.text, \"ax\"\n"
>   ".popsection");
>  
>  /* identity function, which can be inlined */
> -u32 _paravirt_ident_32(u32 x)
> +u32 notrace _paravirt_ident_32(u32 x)
>  {
>   return x;
>  }
>  
> -u64 _paravirt_ident_64(u64 x)
> +u64 notrace _paravirt_ident_64(u64 x)
>  {
>   return x;
>  }

___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH] x86/paravirt: Do not trace _paravirt_ident_*() functions

[Steven Rostedt]

Łukasz Daniluk reported that on a RHEL kernel that his machine would lock up
after enabling function tracer. I asked him to bisect the functions within
available_filter_functions, which he did and it came down to three:

 _paravirt_nop(), _paravirt_ident_32() and _paravirt_ident_64()

It was found that this is only an issue when noreplace-paravirt is added to
the kernel command line.

This means that those functions are most likely called within critical
sections of the funtion tracer, and must not be traced.

In newer kenels _paravirt_nop() is defined within gcc asm(), and is no
longer an issue. But both _paravirt_ident_{32,64}() causes the following
splat when they are traced:

 mm/pgtable-generic.c:33: bad pmd 8800d2435150(01d00054)
 mm/pgtable-generic.c:33: bad pmd 8800d3624190(01d00070)
 mm/pgtable-generic.c:33: bad pmd 8800d36a5110(01d00054)
 mm/pgtable-generic.c:33: bad pmd 880118eb1450(01d00054)
 NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [systemd-journal:469]
 Modules linked in: e1000e
 CPU: 2 PID: 469 Comm: systemd-journal Not tainted 4.6.0-rc4-test+ #513
 Hardware name: Hewlett-Packard HP Compaq Pro 6300 SFF/339A, BIOS K01 v02.05 
05/07/2012
 task: 880118f740c0 ti: 8800d4aec000 task.ti: 8800d4aec000
 RIP: 0010:[]  [] 
queued_spin_lock_slowpath+0x118/0x1a0
 RSP: 0018:8800d4aefb90  EFLAGS: 0246
 RAX:  RBX:  RCX: 88011eb16d40
 RDX: 82485760 RSI: 1f288820 RDI: ea008030
 RBP: 8800d4aefb90 R08: 000c R09: 
 R10: 821c8e0e R11:  R12: 88200fb8
 R13: 7f7a4e3f7000 R14: ea000303f600 R15: 8800d4b562e0
 FS:  7f7a4e3d7840() GS:88011eb0() knlGS:
 CS:  0010 DS:  ES:  CR0: 80050033
 CR2: 7f7a4e3f7000 CR3: d3e71000 CR4: 001406e0
 Stack:
  8800d4aefba0 81cc5f47 8800d4aefc60 8122c15b
  8800d4aefcb0 8800d4aefbd0 811bf4cb 0002
  0015 8800d2276050 8000c0fd8867 ea008030
 Call Trace:
  [] _raw_spin_lock+0x27/0x30
  [] handle_pte_fault+0x13db/0x16b0
  [] ? function_trace_call+0x15b/0x180
  [] ? handle_pte_fault+0x5/0x16b0
  [] handle_mm_fault+0x312/0x670
  [] ? find_vma+0x68/0x70
  [] __do_page_fault+0x1b1/0x4e0
  [] do_page_fault+0x22/0x30
  [] page_fault+0x28/0x30
  [] ? copy_user_enhanced_fast_string+0x5/0x10
  [] ? seq_read+0x305/0x370
  [] __vfs_read+0x28/0xe0
  [] ? __vfs_read+0x5/0xe0
  [] ? __vfs_read+0x5/0xe0
  [] vfs_read+0x86/0x130
  [] SyS_read+0x46/0xa0
  [] entry_SYSCALL_64_fastpath+0x1e/0xa8
 Code: 12 48 c1 ea 0c 83 e8 01 83 e2 30 48 98 48 81 c2 40 6d 01 00 48 03 14
 c5 80 6a 5d 82 48 89 0a 8b 41 08 85 c0 75 09 f3 90 8b 41 08 <85> c0 74 f7
 4c 8b 09 4d 85 c9 74 08 41 0f 18 09 eb 02 f3 90 8b

Reported-by: Łukasz Daniluk 
Signed-off-by: Steven Rostedt 

diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c
index f08ac28b8136..f975d226be6e 100644
--- a/arch/x86/kernel/paravirt.c
+++ b/arch/x86/kernel/paravirt.c
@@ -55,12 +55,12 @@ asm (".pushsection .entry.text, \"ax\"\n"
  ".popsection");
 
 /* identity function, which can be inlined */
-u32 _paravirt_ident_32(u32 x)
+u32 notrace _paravirt_ident_32(u32 x)
 {
return x;
 }
 
-u64 _paravirt_ident_64(u64 x)
+u64 notrace _paravirt_ident_64(u64 x)
 {
return x;
 }
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization