I see the problem. You only hit the breakpoint in the first forked process? I was able to sort-of work around this, using detach-on-fork.
gdb -q --args ./qemu-fuzz-i386 (gef) set follow-fork-mode child (gef) set detach-on-fork off (gef) b generic_fuzz.c:667 (gef) r --fuzz-target=generic-fuzz-virtio-vga -runs=1000 ... Thread 3.1 "qemu-fuzz-i386" hit Breakpoint 1, generic_fuzz (gef) c Continuing. [Thread 0x7ffff32defc0 (LWP 806) exited] (gef) info inferiors Num Description Executable 1 process 31866 /home/alxndr/Downloads/qemu/build-fuzz/qemu-fuzz-i386 2 process 802 /home/alxndr/Downloads/qemu/build-fuzz/qemu-fuzz-i386 * 3 <null> /home/alxndr/Downloads/qemu/build-fuzz/qemu-fuzz-i386 # I make a guess that the parent is 802. (gef) inferior 2 [Switching to inferior 2 [process 802] (/home/alxndr/Downloads/qemu/build-fuzz/qemu-fuzz-i386)] (gef) c ... [New Thread 0x7fffca9fe700 (LWP 4538)] [Switching to Thread 0x7ffff32defc0 (LWP 4537)] Thread 3.1 "qemu-fuzz-i386" hit Breakpoint 1, generic_fuzz (gef) c Continuing. [Thread 0x7ffff32defc0 (LWP 4537) exited] (gef) inferior 2 [Switching to inferior 2 [process 802] (/home/alxndr/Downloads/qemu/build-fuzz/qemu-fuzz-i386)] (gef) c ... There's probably ways to automate most of this with gdb breakpoint commands. I usually don't need to debug more than one child, so this is a new problem for me :) . Are you simply trying to reproduce a bug/crash? My basic workflow for that is: QEMU_FUZZ_TIMEOUT=0 QTEST_LOG=1 FUZZ_SERIALIZE_QTEST=1 \ ./qemu-fuzz-target \ --fuzz-target=generic-fuzz-virtio-vga ./crash-... > /tmp/out ./scripts/oss-fuzz/reorder_fuzzer_qtest_trace.py /tmp/out > /tmp/repro # In /tmp/out find the line "Starting qemu with Arguments:" and copy the # args ( without -qtest /dev/null) less /tmp/out export QEMU_ARGS="-display none -machine accel=qtest, -m 512M -machine q35 -nodefaults -device virtio-vga" # Reproduce the crash on a non-fuzz binary ./qemu-system-i386 $QEMU_ARGS -qtest stdio < /tmp/repro For most cases, this should work. Let me know if there's any problem -Alex On 210308 1058, Yan Zhiqiang wrote: > It hasn't changed. > Can you show me the specific details of the entire debugging process? > This is my debugging process as follows.(It will not hit the breakpoint as > you can see) > > ☁ build_qemu printenv > > XDG_RUNTIME_DIR=/run/user/500 > > XDG_SESSION_ID=2 > > ZSH=/home/ubuntu/.oh-my-zsh > > _=/usr/bin/printenv > > *QEMU_FUZZ_TIMEOUT=0* > > > > ☁ build_qemu gdb -q --args ./qemu-fuzz-i386 > > --fuzz-target=generic-fuzz-virtio-vga fuzz-output/ > > pwndbg: loaded 189 commands. Type pwndbg [filter] for a list. > > pwndbg: created $rebase, $ida gdb functions (can be used with print/break) > > Reading symbols from ./qemu-fuzz-i386...done. > > warning: File "/home/ubuntu/qemu-lastest-fuzz-test/qemu/.gdbinit" > > auto-loading has been declined by your `auto-load safe-path' set to > > "$debugdir:$datadir/auto-load". > > To enable execution of this file add > > add-auto-load-safe-path > > /home/ubuntu/qemu-lastest-fuzz-test/qemu/.gdbinit > > line to your configuration file "/home/ubuntu/.gdbinit". > > To completely disable this security protection add > > set auto-load safe-path / > > line to your configuration file "/home/ubuntu/.gdbinit". > > For more information about this security protection see the > > "Auto-loading safe path" section in the GDB manual. E.g., run from the > > shell: > > info "(gdb)Auto-loading safe path" > > pwndbg> b generic_fuzz.c:655 > > Breakpoint 1 at 0x2ca5938: file ../qemu/tests/qtest/fuzz/generic_fuzz.c, > > line 655. > > pwndbg> set follow-fork-mode child > > pwndbg> r > > Starting program: > > /home/ubuntu/qemu-lastest-fuzz-test/build_qemu/qemu-fuzz-i386 > > --fuzz-target=generic-fuzz-virtio-vga fuzz-output/ > > [Thread debugging using libthread_db enabled] > > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > > [New Thread 0x7ffff0fff700 (LWP 36468)] > > ==36456==WARNING: ASan doesn't fully support makecontext/swapcontext > > functions and may produce false positives in some cases! > > [New Thread 0x7fffd05ff700 (LWP 36469)] > > INFO: libFuzzer ignores flags that start with '--' > > INFO: Seed: 1537774383 > > INFO: Loaded 1 modules (791403 inline 8-bit counters): 791403 > > [0x55555f076000, 0x55555f13736b), > > INFO: Loaded 1 PC tables (791403 PCs): 791403 > > [0x55555e461b00,0x55555f0751b0), > > [New Thread 0x7fffcd179700 (LWP 36471)] > > INFO: 7892 files found in fuzz-output/ > > INFO: -max_len is not provided; libFuzzer will not generate inputs larger > > than 4096 bytes > > Matching objects by name virtio* > > This process will try to fuzz the following MemoryRegions: > > * vga ioports remapped[0] (size 20) > > * qemu extended regs[0] (size 8) > > * virtio-pci-device[0] (size 1000) > > * vga[4] (size 1) > > * msix-pba[0] (size 8) > > * vga[2] (size 10) > > * bochs dispi interface[0] (size 16) > > * virtio-pci-notify[0] (size 1000) > > * vga[0] (size 2) > > * bus master[0] (size 0) > > * msix-table[0] (size 30) > > * vga-lowmem[0] (size 20000) > > * virtio-pci-common[0] (size 800) > > * virtio-pci-notify-pio[0] (size 4) > > * vbe[0] (size 4) > > * bus master container[0] (size 0) > > * virtio-vga-msix[0] (size 1000) > > * vga[3] (size 2) > > * virtio-pci-isr[0] (size 800) > > * virtio-pci[0] (size 4000) > > * vga[1] (size 1) > > [New process 36472] > > [Thread debugging using libthread_db enabled] > > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > > [New Thread 0x7fffcd179700 (LWP 36473)] > > [Switching to Thread 0x7ffff7fdfcc0 (LWP 36472)] > > ERROR: Could not find ELF base! > > Thread 2.1 "qemu-fuzz-i386" hit Breakpoint 1, generic_fuzz > > (s=0x6120000001c0, Data=0x60200008ba30 "\276", Size=0) at > > ../qemu/tests/qtest/fuzz/generic_fuzz.c:655 > > 655 if (timeout) { > > ERROR: Could not find ELF base! > > LEGEND: STACK | HEAP | CODE | DATA | RWX | RODATA > > ────────────────────────────────────────────────────────────────[ > > REGISTERS ]──────────────────────────────────────────────────────────────── > > RAX 0x0 > > RBX 0x7fffffffc680 —▸ 0x55555f076000 (__start___sancov_cntrs) ◂— 0x0 > > RCX 0x2aaaac0fcc93c0 > > RDX 0x2aaaac0fcc9380 > > RDI 0x0 > > RSI 0x1 > > R8 0x555560337400 (fuzzer::TPC) ◂— 0x0 > > R9 0x0 > > R10 0x1 > > R11 0x201 > > R12 0x60200008ba30 ◂— 0xbe > > R13 0x555560337400 (fuzzer::TPC) ◂— 0x0 > > R14 0x0 > > R15 0x7fffffffca67 ◂— 0x80000 > > RBP 0x7fffffffc8d0 —▸ 0x7fffffffc9d0 —▸ 0x7fffffffca10 —▸ 0x7fffffffca90 > > —▸ 0x7fffffffcad0 ◂— ... > > RSP 0x7fffffffc4c0 ◂— 0x41b58ab3 > > RIP 0x5555581f9938 (generic_fuzz+616) ◂— mov eax, dword ptr [rip + > > 0x5300b82] > > ─────────────────────────────────────────────────────────────────[ DISASM > > ]────────────────────────────────────────────────────────────────── > > ► 0x5555581f9938 <generic_fuzz+616> mov eax, dword ptr [rip + > > 0x5300b82] <0x55555d4fa4c0> > > 0x5555581f993e <generic_fuzz+622> mov esi, eax > > 0x5555581f9940 <generic_fuzz+624> mov dword ptr [rbx + 0x170], eax > > 0x5555581f9946 <generic_fuzz+630> call > > __sanitizer_cov_trace_const_cmp4 <__sanitizer_cov_trace_const_cmp4> > > 0x5555581f994b <generic_fuzz+635> mov eax, dword ptr [rbx + 0x170] > > 0x5555581f9951 <generic_fuzz+641> cmp eax, 0 > > 0x5555581f9954 <generic_fuzz+644> jne generic_fuzz+669 > > <generic_fuzz+669> > > 0x5555581f995a <generic_fuzz+650> mov al, byte ptr [rip + > > 0x6e7c911] <0x55555f076271> > > 0x5555581f9960 <generic_fuzz+656> add al, 1 > > 0x5555581f9962 <generic_fuzz+658> mov byte ptr [rip + 0x6e7c909], > > al <0x55555f076271> > > 0x5555581f9968 <generic_fuzz+664> jmp generic_fuzz+2134 > > <generic_fuzz+2134> > > ──────────────────────────────────────────────────────────────[ SOURCE > > (CODE) ]────────────────────────────────────────────────────────────── > > In file: > > /home/ubuntu/qemu-lastest-fuzz-test/qemu/tests/qtest/fuzz/generic_fuzz.c > > 650 * Sometimes the fuzzer will find inputs that take quite a > > long time to > > 651 * process. Often times, these inputs do not result in new > > coverage. > > 652 * Even if these inputs might be interesting, they can slow > > down the > > 653 * fuzzer, overall. Set a timeout to avoid hurting > > performance, too much > > 654 */ > > ► 655 if (timeout) { > > 656 struct sigaction sact; > > 657 struct itimerval timer; > > 658 > > 659 sigemptyset(&sact.sa_mask); > > 660 sact.sa_flags = SA_NODEFER; > > ──────────────────────────────────────────────────────────────────[ STACK > > ]────────────────────────────────────────────────────────────────── > > 00:0000│ rsp 0x7fffffffc4c0 ◂— 0x41b58ab3 > > 01:0008│ 0x7fffffffc4c8 —▸ 0x55555c611c33 ◂— xor esp, dword ptr > > [rax] /* '3 32 80 7 ops:631 144 152 8 sact:656 368 32 9 timer:657' */ > > 02:0010│ 0x7fffffffc4d0 —▸ 0x5555581f96d0 (generic_fuzz) ◂— push rbp > > 03:0018│ 0x7fffffffc4d8 —▸ 0x5555581fbeb6 (pci_enum+1654) ◂— mov > > rdi, qword ptr [rbx + 0xb8] > > 04:0020│ 0x7fffffffc4e0 —▸ 0x5555581fc510 (op_in) ◂— push rbp > > 05:0028│ 0x7fffffffc4e8 —▸ 0x5555581fce80 (op_out) ◂— push rbp > > 06:0030│ 0x7fffffffc4f0 —▸ 0x5555581fda30 (op_read) ◂— push rbp > > 07:0038│ 0x7fffffffc4f8 —▸ 0x5555581fe4d0 (op_write) ◂— push rbp > > ────────────────────────────────────────────────────────────────[ > > BACKTRACE ]──────────────────────────────────────────────────────────────── > > ► f 0 5555581f9938 generic_fuzz+616 > > f 1 55555820802f LLVMFuzzerTestOneInput+1247 > > f 2 5555580ecb5b > > f 3 5555580eecdb > > f 4 5555580ef3b6 > > f 5 5555580e1e71 > > f 6 555558108d93 main+35 > > f 7 7ffff5668bf7 __libc_start_main+231 > > > > ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── > > pwndbg> c > > Continuing. > > [Thread 0x7fffcd179700 (LWP 36473) exited] > > [Inferior 2 (process 36472) exited normally] > > INFO: seed corpus: files: 7892 min: 1b max: 4096b total: 3532937b rss: > > 225Mb > > pwndbg> #2 pulse ft: 216 lim: 4 exec/s: 1 rss: 226Mb > > #4 pulse cov: 218 ft: 220 corp: 2/2b lim: 4 exec/s: 2 rss: 226Mb > > #8 pulse cov: 225 ft: 1268 corp: 6/6b lim: 4 exec/s: 4 rss: 226Mb > > #16 pulse cov: 1571 ft: 1803 corp: 12/22b lim: 4 exec/s: 8 rss: 226Mb > > #32 pulse cov: 1759 ft: 2257 corp: 26/90b lim: 4 exec/s: 16 rss: 226Mb > > #64 pulse cov: 1798 ft: 3062 corp: 50/220b lim: 4 exec/s: 32 rss: > > 227Mb > > #128 pulse cov: 3678 ft: 8040 corp: 105/703b lim: 4 exec/s: 42 rss: > > 229Mb > > #256 pulse cov: 3921 ft: 9791 corp: 204/1716b lim: 4 exec/s: 64 rss: > > 232Mb > > i386: wrong value for queue_enable 1818 > > i386: wrong value for queue_enable ffa17a18 > > i386: wrong value for queue_enable 185a55be > > i386: wrong value for queue_enable 35b5a01 > > i386: wrong value for queue_enable 5a55be00 > > #512 pulse cov: 5089 ft: 13954 corp: 386/5363b lim: 4 exec/s: 73 rss: > > 241Mb > > i386: wrong value for queue_enable 67666369 > > i386: wrong value for queue_enable 20ffff13 > > #1024 pulse cov: 5236 ft: 18351 corp: 679/16Kb lim: 4 exec/s: 85 rss: > > 264Mb > > i386: wrong value for queue_enable 13400200 > > i386: wrong value for queue_enable 943b > > i386: wrong value for queue_enable 555a > > i386: wrong value for queue_enable 13400200 > > ......... > > > > Regards, > Zhiqiang Yan > > Alexander Bulekov <alx...@bu.edu> 于2021年3月6日周六 下午1:11写道: > > > > > > > On 210305 1049, Yan Zhiqiang wrote: > > > * Yes, I tried "set follow-fork-mode child" as well, but it's also the > > > same situation. It will never hit the breakpoint twice(it's strange). > > > > Strange - I just tried this and I could set and hit a breakpoint in > > virtio-gpu code. Maybe the problem is the timeout. Can you try setting > > the environment variable QEMU_FUZZ_TIMEOUT=0 ? > > > > -Alex > > > > > > > > * Thank you for the method to debug crash, I learned a lot from it. > > > > > > Regards, > > > Zhiqiang Yan > > > > > > Alexander Bulekov <alx...@bu.edu> 于2021年3月4日周四 下午11:24写道: > > > > > > > On 210304 1843, Yan Zhiqiang wrote: > > > > > Hello Alex, > > > > > I'm learning the fuzz in QEMU recently, I review the fuzz code under > > > > > /tests/qtest/fuzz which is written by you. > > > > > I learn a lot from it, but I stuck when I want to debug the fuzz > > code. > > > > > I use the gdb with command as follows: > > > > > > > > > > > gdb -q --args ./qemu-fuzz-i386 > > --fuzz-target=generic-fuzz-virtio-vga > > > > > > ./fuzz-output > > > > > > > > > > and set breakpoint at generic_fuzz.c:generic_fuzz. > > > > > It acctually stop when hit the breakpoint. But the function argument > > Size > > > > > is zero and then goto _Exit(0). (try many times but always the same) > > > > > > > > Hi Zhiqiang, > > > > Happy to have more people look at the fuzzing code. > > > > We run each input in a forked process. Maybe you need to run > > > > "set follow-fork-mode child" in gdb? > > > > > > > > > Then input `c` to continue. > > > > > However, it never hit the breakpoint after that. Just as the picture > > > > showed. > > > > > [image: 1614854239086.jpg] > > > > > I tried the qtest debug method, but failed. > > > > > I want to know the real process state to learn the QEMU fuzz and add > > a > > > > new > > > > > fuzzer for QEMU. > > > > > Could you tell me what's the right method to debug the fuzz code? > > Thank > > > > you! > > > > > > > > For debugging crash, I usually build QEMU with --enable-sanitizers > > > > (ASAN), and I convert the crash to a "QTest" reproducer, so it can be > > > > debugged in a normal build of qemu. There's an RFC that has > > instructions > > > > for how to do this: > > > > https://lists.gnu.org/archive/html/qemu-devel/2021-02/msg06035.html > > > > > > > > Let me know if I can provide any more info. > > > > -Alex > > > > > > > > > > > > > > Regards, > > > > > Zhiqiang Yan > > > > > > > > > > > > > >