Re: bpf: kernel BUG in htab_elem_free

[Dmitry Vyukov]

On Thu, Nov 3, 2016 at 10:36 AM, Daniel Borkmann  wrote:
> On 11/03/2016 03:15 PM, Dmitry Vyukov wrote:
>>
>> On Wed, Nov 2, 2016 at 11:14 PM, Dmitry Vyukov  wrote:
>>>
>>> Here we go.
>>>
>>> The following program triggers kernel BUG in htab_elem_free.
>>> On commit 0c183d92b20b5c84ca655b45ef57b3318b83eb9e (Oct 31).
>>> Run as "while true; do ./a.out; done".
>
>
> This one fixes it for me. Could you check it from your side as well?
> I'll submit an official fix then.

I've seen you mailed the fix already.
If you were able to reproduce it and test the fix, then there is
nothing else I can do.

> Thanks a lot for the catch!
> Daniel
>
> diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
> index 570eeca..ad1bc67 100644
> --- a/kernel/bpf/hashtab.c
> +++ b/kernel/bpf/hashtab.c
> @@ -687,7 +687,8 @@ static void delete_all_elements(struct bpf_htab *htab)
>
> hlist_for_each_entry_safe(l, n, head, hash_node) {
> hlist_del_rcu(>hash_node);
> -   htab_elem_free(htab, l);
> +   if (l->state != HTAB_EXTRA_ELEM_USED)
> +   htab_elem_free(htab, l);
> }
> }
>  }

Re: bpf: kernel BUG in htab_elem_free

[Daniel Borkmann]

On 11/03/2016 03:15 PM, Dmitry Vyukov wrote:

On Wed, Nov 2, 2016 at 11:14 PM, Dmitry Vyukov  wrote:

Here we go.

The following program triggers kernel BUG in htab_elem_free.
On commit 0c183d92b20b5c84ca655b45ef57b3318b83eb9e (Oct 31).
Run as "while true; do ./a.out; done".


This one fixes it for me. Could you check it from your side as well?
I'll submit an official fix then.

Thanks a lot for the catch!
Daniel

diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
index 570eeca..ad1bc67 100644
--- a/kernel/bpf/hashtab.c
+++ b/kernel/bpf/hashtab.c
@@ -687,7 +687,8 @@ static void delete_all_elements(struct bpf_htab *htab)

hlist_for_each_entry_safe(l, n, head, hash_node) {
hlist_del_rcu(>hash_node);
-   htab_elem_free(htab, l);
+   if (l->state != HTAB_EXTRA_ELEM_USED)
+   htab_elem_free(htab, l);
}
}
 }

Re: bpf: kernel BUG in htab_elem_free

[Dmitry Vyukov]

On Wed, Nov 2, 2016 at 11:14 PM, Dmitry Vyukov  wrote:
> Here we go.
>
> The following program triggers kernel BUG in htab_elem_free.
> On commit 0c183d92b20b5c84ca655b45ef57b3318b83eb9e (Oct 31).
> Run as "while true; do ./a.out; done".
>
> [ cut here ]
> kernel BUG at mm/slub.c:3866!
> invalid opcode:  [#1] SMP KASAN
> Modules linked in:
> CPU: 1 PID: 1542 Comm: kworker/1:2 Not tainted 4.9.0-rc3+ #20
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Workqueue: events bpf_map_free_deferred
> task: 88003b9c0040 task.stack: 88003cb7
> RIP: 0010:[]  [] kfree+0x140/0x1a0
> RSP: 0018:88003cb77c50  EFLAGS: 00010246
> RAX: eafb0aa0 RBX: 88003ec2a1a8 RCX: 
> RDX:  RSI: 110007b50401 RDI: 88003ec2a1a8
> RBP: 88003cb77c70 R08: 00021800 R09: 
> R10:  R11:  R12: eafb0a80
> R13: 81392bcb R14:  R15: 88003ec2a1a8
> FS:  () GS:88003ed0() knlGS:
> CS:  0010 DS:  ES:  CR0: 80050033
> CR2: 205d7000 CR3: 37d29000 CR4: 06e0
> Stack:
>  dc00 88003da82008 88003b75bb88 
>  88003cb77ce0 81392bcb 81acf4f8 88003b75bc04
>  88003b75bbe0 ed00076eb772 88003b75bb90 3cb77ce0
> Call Trace:
>  [< inline >] htab_elem_free kernel/bpf/hashtab.c:388
>  [< inline >] delete_all_elements kernel/bpf/hashtab.c:690
>  [] htab_map_free+0x30b/0x470 kernel/bpf/hashtab.c:711
>  [] bpf_map_free_deferred+0xac/0xd0 kernel/bpf/syscall.c:97
>  [] process_one_work+0x8a7/0x1300 kernel/workqueue.c:2096
>  [] worker_thread+0xed/0x14e0 kernel/workqueue.c:2230
>  [] kthread+0x1ec/0x260 kernel/kthread.c:209
>  [] ret_from_fork+0x25/0x30 arch/x86/entry/entry_64.S:433
> Code: 83 c4 18 48 89 da 4c 89 ee ff d0 49 8b 04 24 48 85 c0 75 e6 e9
> e9 fe ff ff 49 8b 04 24 f6 c4 40 75 0b 49 8b 44 24 20 a8 01 75 02 <0f>
> 0b 48 89 df e8 56 35 00 00 49 8b 04 24 31 f6 f6 c4 40 74 05
> RIP  [< inline >] PageCompound ./include/linux/page-flags.h:157
> RIP  [] kfree+0x140/0x1a0 mm/slub.c:3866
>  RSP 
> ---[ end trace 1dc58d6aeb2596aa ]---
> ==
> BUG: KASAN: stack-out-of-bounds in complete+0x68/0x70 at addr 88003cb77ed8
> Read of size 4 by task kworker/1:2/1542
> page:eaf2ddc0 count:0 mapcount:0 mapping:  (null) index:0x0
> flags: 0x100()
> page dumped because: kasan: bad access detected
> CPU: 1 PID: 1542 Comm: kworker/1:2 Tainted: G  D 4.9.0-rc3+ #20
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>  88003cb77ce0 81acf609 ed000796efdb ed000796efdb
>  0004  88003cb77d60 814cdbfb
>  88003c8d97c8 dc00 811dd038 0097
> Call Trace:
>  [< inline >] __dump_stack lib/dump_stack.c:15
>  [] dump_stack+0x83/0xba lib/dump_stack.c:51
>  [< inline >] kasan_report_error mm/kasan/report.c:204
>  [] kasan_report+0x4cb/0x500 mm/kasan/report.c:303
>  [] __asan_report_load4_noabort+0x14/0x20
> mm/kasan/report.c:328
>  [] complete+0x68/0x70 kernel/sched/completion.c:34
>  [< inline >] complete_vfork_done kernel/fork.c:1030
>  [] mm_release+0x222/0x3f0 kernel/fork.c:1114
>  [< inline >] exit_mm kernel/exit.c:467
>  [] do_exit+0x3a1/0x2960 kernel/exit.c:815
>  [] rewind_stack_do_exit+0x17/0x20
> arch/x86/entry/entry_64.S:1526
> Memory state around the buggy address:
>  88003cb77d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>  88003cb77e00: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 00 f4 f4 f4
>>88003cb77e80: f2 f2 f2 f2 00 f4 f4 f4 f2 f2 f2 f2 00 00 f4 f4
> ^
>  88003cb77f00: f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00
>  88003cb77f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> ==
> BUG: unable to handle kernel
> paging request at ffd8
> IP: [] kthread_data+0x4d/0x70 kernel/kthread.c:137
> PGD 360d067 [   48.581115] PUD 360f067
> PMD 0 [   48.581840]
> Oops:  [#2] SMP KASAN
> Modules linked in:
> CPU: 1 PID: 1542 Comm: kworker/1:2 Tainted: GB D 4.9.0-rc3+ #20
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> task: 88003b9c0040 task.stack: 88003cb7
> RIP: 0010:[]  [] kthread_data+0x4d/0x70
> RSP: 0018:88003cb77c78  EFLAGS: 00010046
> RAX: dc00 RBX:  RCX: 
> RDX: 1ffb RSI: 88003b9c00c0 RDI: ffd8
> RBP: 88003cb77c80 R08: 88003ed20a48 R09: 88003ed20a40
> R10:  R11:  R12: 

bpf: kernel BUG in htab_elem_free

[Dmitry Vyukov]

Here we go.

The following program triggers kernel BUG in htab_elem_free.
On commit 0c183d92b20b5c84ca655b45ef57b3318b83eb9e (Oct 31).
Run as "while true; do ./a.out; done".

[ cut here ]
kernel BUG at mm/slub.c:3866!
invalid opcode:  [#1] SMP KASAN
Modules linked in:
CPU: 1 PID: 1542 Comm: kworker/1:2 Not tainted 4.9.0-rc3+ #20
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Workqueue: events bpf_map_free_deferred
task: 88003b9c0040 task.stack: 88003cb7
RIP: 0010:[]  [] kfree+0x140/0x1a0
RSP: 0018:88003cb77c50  EFLAGS: 00010246
RAX: eafb0aa0 RBX: 88003ec2a1a8 RCX: 
RDX:  RSI: 110007b50401 RDI: 88003ec2a1a8
RBP: 88003cb77c70 R08: 00021800 R09: 
R10:  R11:  R12: eafb0a80
R13: 81392bcb R14:  R15: 88003ec2a1a8
FS:  () GS:88003ed0() knlGS:
CS:  0010 DS:  ES:  CR0: 80050033
CR2: 205d7000 CR3: 37d29000 CR4: 06e0
Stack:
 dc00 88003da82008 88003b75bb88 
 88003cb77ce0 81392bcb 81acf4f8 88003b75bc04
 88003b75bbe0 ed00076eb772 88003b75bb90 3cb77ce0
Call Trace:
 [< inline >] htab_elem_free kernel/bpf/hashtab.c:388
 [< inline >] delete_all_elements kernel/bpf/hashtab.c:690
 [] htab_map_free+0x30b/0x470 kernel/bpf/hashtab.c:711
 [] bpf_map_free_deferred+0xac/0xd0 kernel/bpf/syscall.c:97
 [] process_one_work+0x8a7/0x1300 kernel/workqueue.c:2096
 [] worker_thread+0xed/0x14e0 kernel/workqueue.c:2230
 [] kthread+0x1ec/0x260 kernel/kthread.c:209
 [] ret_from_fork+0x25/0x30 arch/x86/entry/entry_64.S:433
Code: 83 c4 18 48 89 da 4c 89 ee ff d0 49 8b 04 24 48 85 c0 75 e6 e9
e9 fe ff ff 49 8b 04 24 f6 c4 40 75 0b 49 8b 44 24 20 a8 01 75 02 <0f>
0b 48 89 df e8 56 35 00 00 49 8b 04 24 31 f6 f6 c4 40 74 05
RIP  [< inline >] PageCompound ./include/linux/page-flags.h:157
RIP  [] kfree+0x140/0x1a0 mm/slub.c:3866
 RSP 
---[ end trace 1dc58d6aeb2596aa ]---
==
BUG: KASAN: stack-out-of-bounds in complete+0x68/0x70 at addr 88003cb77ed8
Read of size 4 by task kworker/1:2/1542
page:eaf2ddc0 count:0 mapcount:0 mapping:  (null) index:0x0
flags: 0x100()
page dumped because: kasan: bad access detected
CPU: 1 PID: 1542 Comm: kworker/1:2 Tainted: G  D 4.9.0-rc3+ #20
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
 88003cb77ce0 81acf609 ed000796efdb ed000796efdb
 0004  88003cb77d60 814cdbfb
 88003c8d97c8 dc00 811dd038 0097
Call Trace:
 [< inline >] __dump_stack lib/dump_stack.c:15
 [] dump_stack+0x83/0xba lib/dump_stack.c:51
 [< inline >] kasan_report_error mm/kasan/report.c:204
 [] kasan_report+0x4cb/0x500 mm/kasan/report.c:303
 [] __asan_report_load4_noabort+0x14/0x20
mm/kasan/report.c:328
 [] complete+0x68/0x70 kernel/sched/completion.c:34
 [< inline >] complete_vfork_done kernel/fork.c:1030
 [] mm_release+0x222/0x3f0 kernel/fork.c:1114
 [< inline >] exit_mm kernel/exit.c:467
 [] do_exit+0x3a1/0x2960 kernel/exit.c:815
 [] rewind_stack_do_exit+0x17/0x20
arch/x86/entry/entry_64.S:1526
Memory state around the buggy address:
 88003cb77d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 88003cb77e00: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 00 f4 f4 f4
>88003cb77e80: f2 f2 f2 f2 00 f4 f4 f4 f2 f2 f2 f2 00 00 f4 f4
^
 88003cb77f00: f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00
 88003cb77f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
==
BUG: unable to handle kernel
paging request at ffd8
IP: [] kthread_data+0x4d/0x70 kernel/kthread.c:137
PGD 360d067 [   48.581115] PUD 360f067
PMD 0 [   48.581840]
Oops:  [#2] SMP KASAN
Modules linked in:
CPU: 1 PID: 1542 Comm: kworker/1:2 Tainted: GB D 4.9.0-rc3+ #20
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
task: 88003b9c0040 task.stack: 88003cb7
RIP: 0010:[]  [] kthread_data+0x4d/0x70
RSP: 0018:88003cb77c78  EFLAGS: 00010046
RAX: dc00 RBX:  RCX: 
RDX: 1ffb RSI: 88003b9c00c0 RDI: ffd8
RBP: 88003cb77c80 R08: 88003ed20a48 R09: 88003ed20a40
R10:  R11:  R12: 88003ed20980
R13: 88003b9c0040 R14: 88003b9c0094 R15: 0040
FS:  () GS:88003ed0() knlGS:
CS:  0010 DS:  ES:  CR0: 80050033
CR2: 0028 CR3: 0360c000 CR4: