Re: WARNING in refcount_sub_and_test (2)
On Fri, Mar 30, 2018 at 12:01 AM, syzbotwrote: > Hello, > > syzbot hit the following crash on bpf-next commit > 22527437e0a0c96ee3153e9d0382942b0fd4f9dd (Thu Mar 29 02:36:15 2018 +) > Merge branch 'nfp-bpf-updates' > syzbot dashboard link: > https://syzkaller.appspot.com/bug?extid=c7b0dde061c523bc4b0f > > C reproducer: https://syzkaller.appspot.com/x/repro.c?id=5996614741131264 > syzkaller reproducer: > https://syzkaller.appspot.com/x/repro.syz?id=5947747274326016 > Raw console output: > https://syzkaller.appspot.com/x/log.txt?id=6215237837520896 > Kernel config: > https://syzkaller.appspot.com/x/.config?id=-1280663959502969741 > compiler: gcc (GCC) 7.1.1 20170620 > > IMPORTANT: if you fix the bug, please add the following tag to the commit: > Reported-by: syzbot+c7b0dde061c523bc4...@syzkaller.appspotmail.com > It will help syzbot understand when the bug is fixed. See footer for > details. > If you forward the report, please keep this part and the footer. Messed kernel output. #syz dup: WARNING: refcount bug in sk_alloc > R13: 0030656c69662f2e R14: 0005 R15: 2f30656c69662f2e > [ cut here ] > [ cut here ] > refcount_t: increment on 0; use-after-free. > refcount_t: underflow; use-after-free. > WARNING: CPU: 0 PID: 4450 at lib/refcount.c:187 > refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:187 > WARNING: CPU: 1 PID: 4460 at lib/refcount.c:153 refcount_inc+0x47/0x50 > lib/refcount.c:153 > Kernel panic - not syncing: panic_on_warn set ... > > Modules linked in: > CPU: 0 PID: 4450 Comm: syzkaller428798 Not tainted 4.16.0-rc6+ #40 > CPU: 1 PID: 4460 Comm: syzkaller428798 Not tainted 4.16.0-rc6+ #40 > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS > Google 01/01/2011 > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS > Google 01/01/2011 > Call Trace: > RIP: 0010:refcount_inc+0x47/0x50 lib/refcount.c:153 > __dump_stack lib/dump_stack.c:17 [inline] > dump_stack+0x194/0x24d lib/dump_stack.c:53 > RSP: 0018:8801b534f860 EFLAGS: 00010286 > RAX: dc08 RBX: 8801b1b8c184 RCX: 815ba4be > panic+0x1e4/0x41c kernel/panic.c:183 > RDX: RSI: 110036a69ebc RDI: 110036a69e91 > RBP: 8801b534f868 R08: R09: > R10: R11: R12: 8801b534faf8 > R13: 8801b04db513 R14: 8801b1b8c180 R15: 8801b04db501 > FS: 008e6880() GS:8801db30() knlGS: > CS: 0010 DS: ES: CR0: 80050033 > __warn+0x1dc/0x200 kernel/panic.c:547 > CR2: 006ea510 CR3: 0001b106f005 CR4: 001606e0 > DR0: DR1: DR2: > report_bug+0x1f4/0x2b0 lib/bug.c:186 > DR3: DR6: fffe0ff0 DR7: 0400 > fixup_bug.part.11+0x37/0x80 arch/x86/kernel/traps.c:178 > Call Trace: > fixup_bug arch/x86/kernel/traps.c:247 [inline] > do_error_trap+0x2d7/0x3e0 arch/x86/kernel/traps.c:296 > get_net include/net/net_namespace.h:204 [inline] > sk_alloc+0x3f9/0x1440 net/core/sock.c:1540 > do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:315 > invalid_op+0x1b/0x40 arch/x86/entry/entry_64.S:986 > RIP: 0010:refcount_sub_and_test+0x167/0x1b0 lib/refcount.c:187 > RSP: 0018:8801b0e87728 EFLAGS: 00010286 > RAX: dc08 RBX: RCX: 815ba4be > RDX: RSI: 1100361d0e95 RDI: 0293 > RBP: 8801b0e877b8 R08: R09: > R10: 8801b0e87850 R11: R12: 1100361d0ee6 > inet_create+0x47c/0xf50 net/ipv4/af_inet.c:320 > R13: R14: 0001 R15: 8801b0816204 > __sock_create+0x4d4/0x850 net/socket.c:1285 > sock_create net/socket.c:1325 [inline] > SYSC_socket net/socket.c:1355 [inline] > SyS_socket+0xeb/0x1d0 net/socket.c:1335 > refcount_dec_and_test+0x1a/0x20 lib/refcount.c:212 > put_net include/net/net_namespace.h:222 [inline] > __sk_destruct+0x560/0x920 net/core/sock.c:1592 > do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287 > sk_destruct+0x47/0x80 net/core/sock.c:1601 > entry_SYSCALL_64_after_hwframe+0x42/0xb7 > __sk_free+0xf1/0x2b0 net/core/sock.c:1612 > RIP: 0033:0x44ac67 > sk_free+0x2a/0x40 net/core/sock.c:1623 > RSP: 002b:7ffcd4f45588 EFLAGS: 0202 > sock_put include/net/sock.h:1660 [inline] > tcp_close+0x967/0x1190 net/ipv4/tcp.c:2321 > ORIG_RAX: 0029 > RAX: ffda RBX: RCX: 0044ac67 > RDX: 0006 RSI: 0001 RDI: 0002 > RBP: 7ffcd4f456b0 R08: R09: 0001 > R10: 0006 R11: 0202 R12: 0002 > inet_release+0xed/0x1c0 net/ipv4/af_inet.c:427 > R13: 0002 R14: b38f R15: 7ffcd4f456d8 > sock_release+0x8d/0x1e0
Re: WARNING in refcount_sub_and_test
On Fri, Oct 27, 2017 at 11:36 AM, Eric Dumazetwrote: > On Fri, 2017-10-27 at 08:09 +0200, Dmitry Vyukov wrote: > >> Yes, I've noticed this one. It seems to happen on a first incoming >> network connection (ssh/scp). I have not seen it before. > > https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?h=timers/core=52f737c2da40259ac9962170ce608b6fb1b55ee4 > > ( Google-Bug-Id: 68003409 ) Good! I've noticed that it does not happen on latest Linus tree, now I know why.
Re: WARNING in refcount_sub_and_test
On Fri, 2017-10-27 at 08:09 +0200, Dmitry Vyukov wrote: > Yes, I've noticed this one. It seems to happen on a first incoming > network connection (ssh/scp). I have not seen it before. https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?h=timers/core=52f737c2da40259ac9962170ce608b6fb1b55ee4 ( Google-Bug-Id: 68003409 )
Re: WARNING in refcount_sub_and_test
On Thu, Oct 26, 2017 at 6:56 PM, Xin Longwrote: > Hi all, > > I am failed to reproduce it on target kernel with the reproducer file > or replaying the target syzkaller description log file, do I made > something wrong or there exists more subjects then the line in > repro.txt: > > #{Threaded:true Collide:true Repeat:false Procs:1 Sandbox:namespace > Fault:false FaultCall:-1 FaultNth:0 EnableTun:false UseTmpDir:true > HandleSegv:false WaitRepeat:false Debug:false Repro:false} Hi ChunYu, I've just re-tested the C repro and was able to trigger the bug in a second. I've checked out 49ca1943a7adb429b11b8e05d81bc821694b76c7, copied the provided config, run make olddefconfig, built with gcc-7 (you can get the exact one here https://storage.googleapis.com/syzkaller/gcc-7.tar.gz). Then run in qemu (most of the flags are probably irrelevant): qemu-system-x86_64 -hda wheezy.img -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic -nographic -kernel arch/x86/boot/bzImage -append "kvm-intel.nested=1 kvm-intel.unrestricted_guest=1 kvm-intel.ept=1 kvm-intel.flexpriority=1 kvm-intel.vpid=1 kvm-intel.emulate_invalid_guest_state=1 kvm-intel.eptad=1 kvm-intel.enable_shadow_vmcs=1 kvm-intel.pml=1 kvm-intel.enable_apicv=1 console=ttyS0 root=/dev/sda earlyprintk=serial slub_debug=UZ vsyscall=native rodata=n oops=panic panic_on_warn=1 panic=86400" -enable-kvm -pidfile vm_pid -m 2G -smp 4 -cpu host -usb -usbdevice mouse -usbdevice tablet -soundhw all >>> Just wondering where we can get wheezy.img, if I can't download >>> somewhere, can you provide one if possible ? >>> >>> I made some imgs before, with kernel built with the .config mail-list >>> usually gave, the guest always failed to boot. >> >> Makes sense. Added image/key links here: >> https://github.com/google/syzkaller/blob/master/docs/syzbot.md#crash-does-not-reproduce >> >> Here are commands to start qemu, ssh into the VM. This just worked for >> me to reproduce the crash. >> >> qemu-system-x86_64 -hda wheezy.img -net >> user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic -nographic -kernel >> arch/x86/boot/bzImage -append "kvm-intel.nested=1 >> kvm-intel.unrestricted_guest=1 kvm-intel.ept=1 >> kvm-intel.flexpriority=1 kvm-intel.vpid=1 >> kvm-intel.emulate_invalid_guest_state=1 kvm-intel.eptad=1 >> kvm-intel.enable_shadow_vmcs=1 kvm-intel.pml=1 >> kvm-intel.enable_apicv=1 console=ttyS0 root=/dev/sda >> earlyprintk=serial vsyscall=native rodata=n oops=panic panic_on_warn=1 >> panic=86400" -enable-kvm -m 2G -smp 4 -cpu host -usb -usbdevice mouse >> -usbdevice tablet -soundhw all >> >> ssh -i wheezy.img.key -p 10022 -o UserKnownHostsFile=/dev/null -o >> StrictHostKeyChecking=no -o IdentitiesOnly=yes root@localhost > Works, and be able to reproduce the issue. Thanks Dmitry. Great! > Another thing is (you might also notice): > https://paste.fedoraproject.org/paste/N~htmOMPUSiIXGUeLH7yIw > This call trace always comes up after kernel has started. Yes, I've noticed this one. It seems to happen on a first incoming network connection (ssh/scp). I have not seen it before.
Re: WARNING in refcount_sub_and_test
On Fri, Oct 27, 2017 at 4:30 AM, ChunYu Wangwrote: > Maybe I have just made some mistakes on understanding the reproduction > methods, will try it again. This is reproducible with the C program. If bot posts it, it was able to reproduce the bug with the compiled C program. If it was not able to reproduce with a C program, then it will post just syzkaller program. To answer your question re running these programs. To reproduce one needs to save this to a file: #{Threaded:true Collide:true Repeat:false Procs:1 Sandbox:namespace Fault:false FaultCall:-1 FaultNth:0 EnableTun:false UseTmpDir:true HandleSegv:false WaitRepeat:false Debug:false Repro:false} mmap(&(0x7f00/0xb5)=nil, 0xb5, 0x3, 0x32, 0x, 0x0) r0 = socket$inet_sctp(0x2, 0x1, 0x84) listen(r0, 0x11c8) accept4(r0, &(0x7fb54000-0x10)=@ethernet={0x0, @local={[0x0, 0x0, 0x0, 0x0, 0x0], 0x0}, [0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0]}, &(0x7f138000-0x4)=0x10, 0x8) listen(r0, 0x0) sendto$inet(r0, &(0x7f002000-0x68)="3755cecb8ecfa33eced658b46a028cba4565dff33dff05002377", 0x1a, 0x4, &(0x7f944000)={0x2, 0x3, @loopback=0x7f01, [0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0]}, 0x10) and then run: ./syz-execprog -sandbox=namespace saved.prog.file If syz-executor is not in the current dir, then also add -executor /path/to/syz-executor. -threaded and -collide flags are true by default, so it's not necessary to add them in this case. If it does not reproduce, it may be useful to run: ./syz-execprog -sandbox=namespace -procs=8 -repeat=0 saved.prog.file i.e. repeat executing it in infinite loop with 8 parallel processes, as lots of bugs are caused by races. > On Thu, Oct 26, 2017 at 10:49 PM, Dmitry Vyukov wrote: >> On Thu, Oct 26, 2017 at 10:53 AM, ChunYu Wang wrote: >>> Hi all, >>> >>> I am failed to reproduce it on target kernel with the reproducer file >>> or replaying the target syzkaller description log file, do I made >>> something wrong or there exists more subjects then the line in >>> repro.txt: >>> >>> #{Threaded:true Collide:true Repeat:false Procs:1 Sandbox:namespace >>> Fault:false FaultCall:-1 FaultNth:0 EnableTun:false UseTmpDir:true >>> HandleSegv:false WaitRepeat:false Debug:false Repro:false} >> >> >> Hi ChunYu, >> >> I've just re-tested the C repro and was able to trigger the bug in a second. >> I've checked out 49ca1943a7adb429b11b8e05d81bc821694b76c7, copied the >> provided config, run make olddefconfig, built with gcc-7 (you can get >> the exact one here >> https://storage.googleapis.com/syzkaller/gcc-7.tar.gz). Then run in >> qemu (most of the flags are probably irrelevant): >> >> qemu-system-x86_64 -hda wheezy.img -net >> user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic -nographic -kernel >> arch/x86/boot/bzImage -append "kvm-intel.nested=1 >> kvm-intel.unrestricted_guest=1 kvm-intel.ept=1 >> kvm-intel.flexpriority=1 kvm-intel.vpid=1 >> kvm-intel.emulate_invalid_guest_state=1 kvm-intel.eptad=1 >> kvm-intel.enable_shadow_vmcs=1 kvm-intel.pml=1 >> kvm-intel.enable_apicv=1 console=ttyS0 root=/dev/sda >> earlyprintk=serial slub_debug=UZ vsyscall=native rodata=n oops=panic >> panic_on_warn=1 panic=86400" -enable-kvm -pidfile vm_pid -m 2G -smp 4 >> -cpu host -usb -usbdevice mouse -usbdevice tablet -soundhw all >> >> And running the provided C program instantly spewed the following. >> >> Is there anything you did differently? I would like to understand >> common reasons why syzbot reproducers don't work and outline them >> here: >> https://github.com/google/syzkaller/blob/master/docs/syzbot.md >> >> Thanks >> >> >> [ 588.444300] refcount_t: underflow; use-after-free. >> [ 588.445812] [ cut here ] >> [ 588.447026] WARNING: CPU: 1 PID: 3086 at lib/refcount.c:186 >> refcount_sub_and_test+0x167/0x1b0 >> [ 588.449082] Kernel panic - not syncing: panic_on_warn set ... >> [ 588.449082] >> [ 588.450737] CPU: 1 PID: 3086 Comm: a.out Not tainted 4.14.0-rc5+ #9 >> [ 588.452160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), >> BIOS Bochs 01/01/2011 >> [ 588.454059] Call Trace: >> [ 588.454658] dump_stack+0x194/0x257 >> [ 588.455538] ? arch_local_irq_restore+0x53/0x53 >> [ 588.456630] panic+0x1e4/0x417 >> [ 588.457367] ? __warn+0x1d9/0x1d9 >> [ 588.458171] ? show_regs_print_info+0x65/0x65 >> [ 588.459234] ? refcount_sub_and_test+0x167/0x1b0 >> [ 588.460262] __warn+0x1c4/0x1d9 >> [ 588.460958] ? refcount_sub_and_test+0x167/0x1b0 >> [ 588.461965] report_bug+0x211/0x2d0 >> [ 588.462756] fixup_bug+0x40/0x90 >> [ 588.463597] do_trap+0x260/0x390 >> [ 588.464304] do_error_trap+0x120/0x390 >> [ 588.465105] ? vprintk_emit+0x49b/0x590 >> [ 588.465929] ? do_trap+0x390/0x390 >> [ 588.41] ? refcount_sub_and_test+0x167/0x1b0 >> [ 588.467646] ? vprintk_emit+0x3ea/0x590 >> [ 588.468475] ? trace_hardirqs_off_thunk+0x1a/0x1c >> [ 588.469482]
Re: WARNING in refcount_sub_and_test
Maybe I have just made some mistakes on understanding the reproduction methods, will try it again. Thanks, - ChunYu On Thu, Oct 26, 2017 at 10:49 PM, Dmitry Vyukovwrote: > On Thu, Oct 26, 2017 at 10:53 AM, ChunYu Wang wrote: >> Hi all, >> >> I am failed to reproduce it on target kernel with the reproducer file >> or replaying the target syzkaller description log file, do I made >> something wrong or there exists more subjects then the line in >> repro.txt: >> >> #{Threaded:true Collide:true Repeat:false Procs:1 Sandbox:namespace >> Fault:false FaultCall:-1 FaultNth:0 EnableTun:false UseTmpDir:true >> HandleSegv:false WaitRepeat:false Debug:false Repro:false} > > > Hi ChunYu, > > I've just re-tested the C repro and was able to trigger the bug in a second. > I've checked out 49ca1943a7adb429b11b8e05d81bc821694b76c7, copied the > provided config, run make olddefconfig, built with gcc-7 (you can get > the exact one here > https://storage.googleapis.com/syzkaller/gcc-7.tar.gz). Then run in > qemu (most of the flags are probably irrelevant): > > qemu-system-x86_64 -hda wheezy.img -net > user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic -nographic -kernel > arch/x86/boot/bzImage -append "kvm-intel.nested=1 > kvm-intel.unrestricted_guest=1 kvm-intel.ept=1 > kvm-intel.flexpriority=1 kvm-intel.vpid=1 > kvm-intel.emulate_invalid_guest_state=1 kvm-intel.eptad=1 > kvm-intel.enable_shadow_vmcs=1 kvm-intel.pml=1 > kvm-intel.enable_apicv=1 console=ttyS0 root=/dev/sda > earlyprintk=serial slub_debug=UZ vsyscall=native rodata=n oops=panic > panic_on_warn=1 panic=86400" -enable-kvm -pidfile vm_pid -m 2G -smp 4 > -cpu host -usb -usbdevice mouse -usbdevice tablet -soundhw all > > And running the provided C program instantly spewed the following. > > Is there anything you did differently? I would like to understand > common reasons why syzbot reproducers don't work and outline them > here: > https://github.com/google/syzkaller/blob/master/docs/syzbot.md > > Thanks > > > [ 588.444300] refcount_t: underflow; use-after-free. > [ 588.445812] [ cut here ] > [ 588.447026] WARNING: CPU: 1 PID: 3086 at lib/refcount.c:186 > refcount_sub_and_test+0x167/0x1b0 > [ 588.449082] Kernel panic - not syncing: panic_on_warn set ... > [ 588.449082] > [ 588.450737] CPU: 1 PID: 3086 Comm: a.out Not tainted 4.14.0-rc5+ #9 > [ 588.452160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), > BIOS Bochs 01/01/2011 > [ 588.454059] Call Trace: > [ 588.454658] dump_stack+0x194/0x257 > [ 588.455538] ? arch_local_irq_restore+0x53/0x53 > [ 588.456630] panic+0x1e4/0x417 > [ 588.457367] ? __warn+0x1d9/0x1d9 > [ 588.458171] ? show_regs_print_info+0x65/0x65 > [ 588.459234] ? refcount_sub_and_test+0x167/0x1b0 > [ 588.460262] __warn+0x1c4/0x1d9 > [ 588.460958] ? refcount_sub_and_test+0x167/0x1b0 > [ 588.461965] report_bug+0x211/0x2d0 > [ 588.462756] fixup_bug+0x40/0x90 > [ 588.463597] do_trap+0x260/0x390 > [ 588.464304] do_error_trap+0x120/0x390 > [ 588.465105] ? vprintk_emit+0x49b/0x590 > [ 588.465929] ? do_trap+0x390/0x390 > [ 588.41] ? refcount_sub_and_test+0x167/0x1b0 > [ 588.467646] ? vprintk_emit+0x3ea/0x590 > [ 588.468475] ? trace_hardirqs_off_thunk+0x1a/0x1c > [ 588.469482] do_invalid_op+0x1b/0x20 > [ 588.470262] invalid_op+0x18/0x20 > [ 588.470988] RIP: 0010:refcount_sub_and_test+0x167/0x1b0 > [ 588.472080] RSP: 0018:88006550e9c8 EFLAGS: 00010282 > [ 588.473224] RAX: 0026 RBX: 0001 RCX: > > [ 588.474643] RDX: 0026 RSI: 11000caa1cf9 RDI: > ed000caa1d2d > [ 588.476091] RBP: 88006550ea58 R08: R09: > 11000caa1ccb > [ 588.477520] R10: 88006550e7f8 R11: 85b2cb78 R12: > 11000caa1d3a > [ 588.478967] R13: ff01 R14: 0100 R15: > 88006a7f4a7c > [ 588.480413] ? refcount_sub_and_test+0x167/0x1b0 > [ 588.481337] ? refcount_inc+0x50/0x50 > [ 588.482081] ? __sctp_outq_teardown+0xa5b/0x1230 > [ 588.483004] ? sctp_association_free+0x2d0/0x930 > [ 588.484291] ? sctp_do_sm+0x271b/0x6a30 > [ 588.485247] ? sctp_primitive_SHUTDOWN+0xa0/0xd0 > [ 588.486295] ? sctp_close+0x3c6/0x980 > [ 588.487058] ? inet_release+0xed/0x1c0 > [ 588.488370] ? sock_release+0x8d/0x1e0 > [ 588.489080] ? sock_close+0x16/0x20 > [ 588.489759] sctp_wfree+0x183/0x620 > [ 588.490430] ? entry_SYSCALL_64_fastpath+0xbc/0xbe > [ 588.491323] ? __sctp_write_space+0x910/0x910 > [ 588.492177] skb_release_head_state+0x124/0x200 > [ 588.493078] skb_release_all+0x15/0x60 > [ 588.493938] consume_skb+0x153/0x490 > [ 588.494605] ? sctp_chunk_put+0x99/0x420 > [ 588.495388] ? alloc_skb_with_frags+0x750/0x750 > [ 588.496119] ? sctp_chunk_hold+0x20/0x20 > [ 588.496757] ? sctp_sched_dequeue_common+0x2aa/0x5d0 > [ 588.497554] ? refcount_sub_and_test+0x115/0x1b0 > [ 588.498296] ? refcount_inc+0x50/0x50 > [
Re: WARNING in refcount_sub_and_test
On Fri, Oct 27, 2017 at 12:56 AM, Xin Longwrote: > On Fri, Oct 27, 2017 at 12:13 AM, Dmitry Vyukov wrote: >> On Thu, Oct 26, 2017 at 5:49 PM, Xin Long wrote: >>> On Thu, Oct 26, 2017 at 10:49 PM, Dmitry Vyukov wrote: On Thu, Oct 26, 2017 at 10:53 AM, ChunYu Wang wrote: > Hi all, > > I am failed to reproduce it on target kernel with the reproducer file > or replaying the target syzkaller description log file, do I made > something wrong or there exists more subjects then the line in > repro.txt: > > #{Threaded:true Collide:true Repeat:false Procs:1 Sandbox:namespace > Fault:false FaultCall:-1 FaultNth:0 EnableTun:false UseTmpDir:true > HandleSegv:false WaitRepeat:false Debug:false Repro:false} Hi ChunYu, I've just re-tested the C repro and was able to trigger the bug in a second. I've checked out 49ca1943a7adb429b11b8e05d81bc821694b76c7, copied the provided config, run make olddefconfig, built with gcc-7 (you can get the exact one here https://storage.googleapis.com/syzkaller/gcc-7.tar.gz). Then run in qemu (most of the flags are probably irrelevant): qemu-system-x86_64 -hda wheezy.img -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic -nographic -kernel arch/x86/boot/bzImage -append "kvm-intel.nested=1 kvm-intel.unrestricted_guest=1 kvm-intel.ept=1 kvm-intel.flexpriority=1 kvm-intel.vpid=1 kvm-intel.emulate_invalid_guest_state=1 kvm-intel.eptad=1 kvm-intel.enable_shadow_vmcs=1 kvm-intel.pml=1 kvm-intel.enable_apicv=1 console=ttyS0 root=/dev/sda earlyprintk=serial slub_debug=UZ vsyscall=native rodata=n oops=panic panic_on_warn=1 panic=86400" -enable-kvm -pidfile vm_pid -m 2G -smp 4 -cpu host -usb -usbdevice mouse -usbdevice tablet -soundhw all >>> Just wondering where we can get wheezy.img, if I can't download >>> somewhere, can you provide one if possible ? >>> >>> I made some imgs before, with kernel built with the .config mail-list >>> usually gave, the guest always failed to boot. >> >> Makes sense. Added image/key links here: >> https://github.com/google/syzkaller/blob/master/docs/syzbot.md#crash-does-not-reproduce >> >> Here are commands to start qemu, ssh into the VM. This just worked for >> me to reproduce the crash. >> >> qemu-system-x86_64 -hda wheezy.img -net >> user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic -nographic -kernel >> arch/x86/boot/bzImage -append "kvm-intel.nested=1 >> kvm-intel.unrestricted_guest=1 kvm-intel.ept=1 >> kvm-intel.flexpriority=1 kvm-intel.vpid=1 >> kvm-intel.emulate_invalid_guest_state=1 kvm-intel.eptad=1 >> kvm-intel.enable_shadow_vmcs=1 kvm-intel.pml=1 >> kvm-intel.enable_apicv=1 console=ttyS0 root=/dev/sda >> earlyprintk=serial vsyscall=native rodata=n oops=panic panic_on_warn=1 >> panic=86400" -enable-kvm -m 2G -smp 4 -cpu host -usb -usbdevice mouse >> -usbdevice tablet -soundhw all >> >> ssh -i wheezy.img.key -p 10022 -o UserKnownHostsFile=/dev/null -o >> StrictHostKeyChecking=no -o IdentitiesOnly=yes root@localhost > Works, and be able to reproduce the issue. Thanks Dmitry. Fix for this crash: @@ -8276,6 +8279,7 @@ static void sctp_sock_migrate(struct sock *oldsk, struct sock *newsk, struct sk_buff *skb, *tmp; struct sctp_ulpevent *event; struct sctp_bind_hashbucket *head; + struct sctp_chunk *chunk; /* Migrate socket buffer sizes and all the socket level options to the * new socket. @@ -8379,7 +8383,12 @@ static void sctp_sock_migrate(struct sock *oldsk, struct sock *newsk, * paths won't try to lock it and then oldsk. */ lock_sock_nested(newsk, SINGLE_DEPTH_NESTING); +list_for_each_entry(chunk, >outqueue.out_chunk_list, list) + skb_orphan(chunk->skb); + sctp_assoc_migrate(assoc, newsk); +list_for_each_entry(chunk, >outqueue.out_chunk_list, list) + sctp_set_owner_w(chunk); Other lists in assoc->outqueue probably need to do the similar, will check for sure later.
Re: WARNING in refcount_sub_and_test
On Fri, Oct 27, 2017 at 12:13 AM, Dmitry Vyukovwrote: > On Thu, Oct 26, 2017 at 5:49 PM, Xin Long wrote: >> On Thu, Oct 26, 2017 at 10:49 PM, Dmitry Vyukov wrote: >>> On Thu, Oct 26, 2017 at 10:53 AM, ChunYu Wang wrote: Hi all, I am failed to reproduce it on target kernel with the reproducer file or replaying the target syzkaller description log file, do I made something wrong or there exists more subjects then the line in repro.txt: #{Threaded:true Collide:true Repeat:false Procs:1 Sandbox:namespace Fault:false FaultCall:-1 FaultNth:0 EnableTun:false UseTmpDir:true HandleSegv:false WaitRepeat:false Debug:false Repro:false} >>> >>> >>> Hi ChunYu, >>> >>> I've just re-tested the C repro and was able to trigger the bug in a second. >>> I've checked out 49ca1943a7adb429b11b8e05d81bc821694b76c7, copied the >>> provided config, run make olddefconfig, built with gcc-7 (you can get >>> the exact one here >>> https://storage.googleapis.com/syzkaller/gcc-7.tar.gz). Then run in >>> qemu (most of the flags are probably irrelevant): >>> >>> qemu-system-x86_64 -hda wheezy.img -net >>> user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic -nographic -kernel >>> arch/x86/boot/bzImage -append "kvm-intel.nested=1 >>> kvm-intel.unrestricted_guest=1 kvm-intel.ept=1 >>> kvm-intel.flexpriority=1 kvm-intel.vpid=1 >>> kvm-intel.emulate_invalid_guest_state=1 kvm-intel.eptad=1 >>> kvm-intel.enable_shadow_vmcs=1 kvm-intel.pml=1 >>> kvm-intel.enable_apicv=1 console=ttyS0 root=/dev/sda >>> earlyprintk=serial slub_debug=UZ vsyscall=native rodata=n oops=panic >>> panic_on_warn=1 panic=86400" -enable-kvm -pidfile vm_pid -m 2G -smp 4 >>> -cpu host -usb -usbdevice mouse -usbdevice tablet -soundhw all >> Just wondering where we can get wheezy.img, if I can't download >> somewhere, can you provide one if possible ? >> >> I made some imgs before, with kernel built with the .config mail-list >> usually gave, the guest always failed to boot. > > Makes sense. Added image/key links here: > https://github.com/google/syzkaller/blob/master/docs/syzbot.md#crash-does-not-reproduce > > Here are commands to start qemu, ssh into the VM. This just worked for > me to reproduce the crash. > > qemu-system-x86_64 -hda wheezy.img -net > user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic -nographic -kernel > arch/x86/boot/bzImage -append "kvm-intel.nested=1 > kvm-intel.unrestricted_guest=1 kvm-intel.ept=1 > kvm-intel.flexpriority=1 kvm-intel.vpid=1 > kvm-intel.emulate_invalid_guest_state=1 kvm-intel.eptad=1 > kvm-intel.enable_shadow_vmcs=1 kvm-intel.pml=1 > kvm-intel.enable_apicv=1 console=ttyS0 root=/dev/sda > earlyprintk=serial vsyscall=native rodata=n oops=panic panic_on_warn=1 > panic=86400" -enable-kvm -m 2G -smp 4 -cpu host -usb -usbdevice mouse > -usbdevice tablet -soundhw all > > ssh -i wheezy.img.key -p 10022 -o UserKnownHostsFile=/dev/null -o > StrictHostKeyChecking=no -o IdentitiesOnly=yes root@localhost Works, and be able to reproduce the issue. Thanks Dmitry. Another thing is (you might also notice): https://paste.fedoraproject.org/paste/N~htmOMPUSiIXGUeLH7yIw This call trace always comes up after kernel has started.
Re: WARNING in refcount_sub_and_test
On Thu, Oct 26, 2017 at 5:49 PM, Xin Longwrote: > On Thu, Oct 26, 2017 at 10:49 PM, Dmitry Vyukov wrote: >> On Thu, Oct 26, 2017 at 10:53 AM, ChunYu Wang wrote: >>> Hi all, >>> >>> I am failed to reproduce it on target kernel with the reproducer file >>> or replaying the target syzkaller description log file, do I made >>> something wrong or there exists more subjects then the line in >>> repro.txt: >>> >>> #{Threaded:true Collide:true Repeat:false Procs:1 Sandbox:namespace >>> Fault:false FaultCall:-1 FaultNth:0 EnableTun:false UseTmpDir:true >>> HandleSegv:false WaitRepeat:false Debug:false Repro:false} >> >> >> Hi ChunYu, >> >> I've just re-tested the C repro and was able to trigger the bug in a second. >> I've checked out 49ca1943a7adb429b11b8e05d81bc821694b76c7, copied the >> provided config, run make olddefconfig, built with gcc-7 (you can get >> the exact one here >> https://storage.googleapis.com/syzkaller/gcc-7.tar.gz). Then run in >> qemu (most of the flags are probably irrelevant): >> >> qemu-system-x86_64 -hda wheezy.img -net >> user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic -nographic -kernel >> arch/x86/boot/bzImage -append "kvm-intel.nested=1 >> kvm-intel.unrestricted_guest=1 kvm-intel.ept=1 >> kvm-intel.flexpriority=1 kvm-intel.vpid=1 >> kvm-intel.emulate_invalid_guest_state=1 kvm-intel.eptad=1 >> kvm-intel.enable_shadow_vmcs=1 kvm-intel.pml=1 >> kvm-intel.enable_apicv=1 console=ttyS0 root=/dev/sda >> earlyprintk=serial slub_debug=UZ vsyscall=native rodata=n oops=panic >> panic_on_warn=1 panic=86400" -enable-kvm -pidfile vm_pid -m 2G -smp 4 >> -cpu host -usb -usbdevice mouse -usbdevice tablet -soundhw all > Just wondering where we can get wheezy.img, if I can't download > somewhere, can you provide one if possible ? > > I made some imgs before, with kernel built with the .config mail-list > usually gave, the guest always failed to boot. Makes sense. Added image/key links here: https://github.com/google/syzkaller/blob/master/docs/syzbot.md#crash-does-not-reproduce Here are commands to start qemu, ssh into the VM. This just worked for me to reproduce the crash. qemu-system-x86_64 -hda wheezy.img -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic -nographic -kernel arch/x86/boot/bzImage -append "kvm-intel.nested=1 kvm-intel.unrestricted_guest=1 kvm-intel.ept=1 kvm-intel.flexpriority=1 kvm-intel.vpid=1 kvm-intel.emulate_invalid_guest_state=1 kvm-intel.eptad=1 kvm-intel.enable_shadow_vmcs=1 kvm-intel.pml=1 kvm-intel.enable_apicv=1 console=ttyS0 root=/dev/sda earlyprintk=serial vsyscall=native rodata=n oops=panic panic_on_warn=1 panic=86400" -enable-kvm -m 2G -smp 4 -cpu host -usb -usbdevice mouse -usbdevice tablet -soundhw all ssh -i wheezy.img.key -p 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes root@localhost
Re: WARNING in refcount_sub_and_test
On Thu, Oct 26, 2017 at 10:49 PM, Dmitry Vyukovwrote: > On Thu, Oct 26, 2017 at 10:53 AM, ChunYu Wang wrote: >> Hi all, >> >> I am failed to reproduce it on target kernel with the reproducer file >> or replaying the target syzkaller description log file, do I made >> something wrong or there exists more subjects then the line in >> repro.txt: >> >> #{Threaded:true Collide:true Repeat:false Procs:1 Sandbox:namespace >> Fault:false FaultCall:-1 FaultNth:0 EnableTun:false UseTmpDir:true >> HandleSegv:false WaitRepeat:false Debug:false Repro:false} > > > Hi ChunYu, > > I've just re-tested the C repro and was able to trigger the bug in a second. > I've checked out 49ca1943a7adb429b11b8e05d81bc821694b76c7, copied the > provided config, run make olddefconfig, built with gcc-7 (you can get > the exact one here > https://storage.googleapis.com/syzkaller/gcc-7.tar.gz). Then run in > qemu (most of the flags are probably irrelevant): > > qemu-system-x86_64 -hda wheezy.img -net > user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic -nographic -kernel > arch/x86/boot/bzImage -append "kvm-intel.nested=1 > kvm-intel.unrestricted_guest=1 kvm-intel.ept=1 > kvm-intel.flexpriority=1 kvm-intel.vpid=1 > kvm-intel.emulate_invalid_guest_state=1 kvm-intel.eptad=1 > kvm-intel.enable_shadow_vmcs=1 kvm-intel.pml=1 > kvm-intel.enable_apicv=1 console=ttyS0 root=/dev/sda > earlyprintk=serial slub_debug=UZ vsyscall=native rodata=n oops=panic > panic_on_warn=1 panic=86400" -enable-kvm -pidfile vm_pid -m 2G -smp 4 > -cpu host -usb -usbdevice mouse -usbdevice tablet -soundhw all Just wondering where we can get wheezy.img, if I can't download somewhere, can you provide one if possible ? I made some imgs before, with kernel built with the .config mail-list usually gave, the guest always failed to boot. Thanks.
Re: WARNING in refcount_sub_and_test
On Thu, Oct 26, 2017 at 10:53 AM, ChunYu Wangwrote: > Hi all, > > I am failed to reproduce it on target kernel with the reproducer file > or replaying the target syzkaller description log file, do I made > something wrong or there exists more subjects then the line in > repro.txt: > > #{Threaded:true Collide:true Repeat:false Procs:1 Sandbox:namespace > Fault:false FaultCall:-1 FaultNth:0 EnableTun:false UseTmpDir:true > HandleSegv:false WaitRepeat:false Debug:false Repro:false} Hi ChunYu, I've just re-tested the C repro and was able to trigger the bug in a second. I've checked out 49ca1943a7adb429b11b8e05d81bc821694b76c7, copied the provided config, run make olddefconfig, built with gcc-7 (you can get the exact one here https://storage.googleapis.com/syzkaller/gcc-7.tar.gz). Then run in qemu (most of the flags are probably irrelevant): qemu-system-x86_64 -hda wheezy.img -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic -nographic -kernel arch/x86/boot/bzImage -append "kvm-intel.nested=1 kvm-intel.unrestricted_guest=1 kvm-intel.ept=1 kvm-intel.flexpriority=1 kvm-intel.vpid=1 kvm-intel.emulate_invalid_guest_state=1 kvm-intel.eptad=1 kvm-intel.enable_shadow_vmcs=1 kvm-intel.pml=1 kvm-intel.enable_apicv=1 console=ttyS0 root=/dev/sda earlyprintk=serial slub_debug=UZ vsyscall=native rodata=n oops=panic panic_on_warn=1 panic=86400" -enable-kvm -pidfile vm_pid -m 2G -smp 4 -cpu host -usb -usbdevice mouse -usbdevice tablet -soundhw all And running the provided C program instantly spewed the following. Is there anything you did differently? I would like to understand common reasons why syzbot reproducers don't work and outline them here: https://github.com/google/syzkaller/blob/master/docs/syzbot.md Thanks [ 588.444300] refcount_t: underflow; use-after-free. [ 588.445812] [ cut here ] [ 588.447026] WARNING: CPU: 1 PID: 3086 at lib/refcount.c:186 refcount_sub_and_test+0x167/0x1b0 [ 588.449082] Kernel panic - not syncing: panic_on_warn set ... [ 588.449082] [ 588.450737] CPU: 1 PID: 3086 Comm: a.out Not tainted 4.14.0-rc5+ #9 [ 588.452160] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 [ 588.454059] Call Trace: [ 588.454658] dump_stack+0x194/0x257 [ 588.455538] ? arch_local_irq_restore+0x53/0x53 [ 588.456630] panic+0x1e4/0x417 [ 588.457367] ? __warn+0x1d9/0x1d9 [ 588.458171] ? show_regs_print_info+0x65/0x65 [ 588.459234] ? refcount_sub_and_test+0x167/0x1b0 [ 588.460262] __warn+0x1c4/0x1d9 [ 588.460958] ? refcount_sub_and_test+0x167/0x1b0 [ 588.461965] report_bug+0x211/0x2d0 [ 588.462756] fixup_bug+0x40/0x90 [ 588.463597] do_trap+0x260/0x390 [ 588.464304] do_error_trap+0x120/0x390 [ 588.465105] ? vprintk_emit+0x49b/0x590 [ 588.465929] ? do_trap+0x390/0x390 [ 588.41] ? refcount_sub_and_test+0x167/0x1b0 [ 588.467646] ? vprintk_emit+0x3ea/0x590 [ 588.468475] ? trace_hardirqs_off_thunk+0x1a/0x1c [ 588.469482] do_invalid_op+0x1b/0x20 [ 588.470262] invalid_op+0x18/0x20 [ 588.470988] RIP: 0010:refcount_sub_and_test+0x167/0x1b0 [ 588.472080] RSP: 0018:88006550e9c8 EFLAGS: 00010282 [ 588.473224] RAX: 0026 RBX: 0001 RCX: [ 588.474643] RDX: 0026 RSI: 11000caa1cf9 RDI: ed000caa1d2d [ 588.476091] RBP: 88006550ea58 R08: R09: 11000caa1ccb [ 588.477520] R10: 88006550e7f8 R11: 85b2cb78 R12: 11000caa1d3a [ 588.478967] R13: ff01 R14: 0100 R15: 88006a7f4a7c [ 588.480413] ? refcount_sub_and_test+0x167/0x1b0 [ 588.481337] ? refcount_inc+0x50/0x50 [ 588.482081] ? __sctp_outq_teardown+0xa5b/0x1230 [ 588.483004] ? sctp_association_free+0x2d0/0x930 [ 588.484291] ? sctp_do_sm+0x271b/0x6a30 [ 588.485247] ? sctp_primitive_SHUTDOWN+0xa0/0xd0 [ 588.486295] ? sctp_close+0x3c6/0x980 [ 588.487058] ? inet_release+0xed/0x1c0 [ 588.488370] ? sock_release+0x8d/0x1e0 [ 588.489080] ? sock_close+0x16/0x20 [ 588.489759] sctp_wfree+0x183/0x620 [ 588.490430] ? entry_SYSCALL_64_fastpath+0xbc/0xbe [ 588.491323] ? __sctp_write_space+0x910/0x910 [ 588.492177] skb_release_head_state+0x124/0x200 [ 588.493078] skb_release_all+0x15/0x60 [ 588.493938] consume_skb+0x153/0x490 [ 588.494605] ? sctp_chunk_put+0x99/0x420 [ 588.495388] ? alloc_skb_with_frags+0x750/0x750 [ 588.496119] ? sctp_chunk_hold+0x20/0x20 [ 588.496757] ? sctp_sched_dequeue_common+0x2aa/0x5d0 [ 588.497554] ? refcount_sub_and_test+0x115/0x1b0 [ 588.498296] ? refcount_inc+0x50/0x50 [ 588.49] ? trace_hardirqs_off+0xd/0x10 [ 588.499567] ? quarantine_put+0xeb/0x190 [ 588.500215] sctp_chunk_put+0x29c/0x420 [ 588.500836] ? sctp_chunk_hold+0x20/0x20 [ 588.501491] ? sctp_transport_dst_confirm+0x50/0x50 [ 588.502266] ? sctp_sched_fcfs_dequeue+0x198/0x290 [ 588.503027] ? sctp_sched_dequeue_common+0x5d0/0x5d0 [ 588.504001] sctp_chunk_free+0x53/0x60 [ 588.504692]
Re: WARNING in refcount_sub_and_test
Hi all, I am failed to reproduce it on target kernel with the reproducer file or replaying the target syzkaller description log file, do I made something wrong or there exists more subjects then the line in repro.txt: #{Threaded:true Collide:true Repeat:false Procs:1 Sandbox:namespace Fault:false FaultCall:-1 FaultNth:0 EnableTun:false UseTmpDir:true HandleSegv:false WaitRepeat:false Debug:false Repro:false} Thanks - ChunYu --- 2017/10/26 04:49:15 reproducing crash 'hang': testing program (duration=10s, {Threaded:true Collide:true Repeat:true Procs:32 Sandbox:namespace Fault:false FaultCall:-1 FaultNth:0 EnableTun:true UseTmpDir:true HandleSegv:true WaitRepeat:true Debug:false Repro:true}): mmap-socket$inet_sctp-listen-accept4-listen-sendto$inet 2017/10/26 04:49:15 running command: ssh []string{"-p", "22", "-o", "ConnectionAttempts=10", "-o", "ConnectTimeout=10", "-o", "BatchMode=yes", "-o", "UserKnownHostsFile=/dev/null", "-o", "IdentitiesOnly=yes", "-o", "StrictHostKeyChecking=no", "-o", "LogLevel=error", "-i", "/home/chunwang/.ssh/id_rsa", "root@10.73.5.213", "cd /home/user/tmp/syz && exec /home/user/tmp/syz/syz-execprog -executor /home/user/tmp/syz/syz-executor -arch=amd64 -cover=0 -procs=32 -repeat=0 -sandbox namespace -threaded=true -collide=true /home/user/tmp/syz/syzkaller140727249"} 2017/10/26 04:49:25 reproducing crash 'hang': program did not crash 2017/10/26 04:49:25 reproducing crash 'hang': single: failed to extract reproducer 2017/10/26 04:49:25 reproducing crash 'hang': bisect: bisecting 1 programs with base timeout 10s 2017/10/26 04:49:25 reproducing crash 'hang': bisect: bisecting 1 programs 2017/10/26 04:49:25 reproducing crash 'hang': bisect: executing all 1 programs