As part of Google Summer of Code 2019, I'm working on integrating
fuzzing of virtual devices into QEMU [1]. This is a highly WIP patchset
adding this functionality.
Fuzzers provide random data to a program and monitor its execution for
errors. Coverage-guided fuzzers also observe the parts of the program
that are exercised by each input, and use this information to
mutate/guide the inputs to reach additional parts of the program. They
are quite effective for finding bugs in a wide range of software.
Summary:
- The virtual-device fuzzers use libfuzzer [2] for coverage-guided
in-process fuzzing.
- To fuzz a device, create a new fuzz "target" - i.e. a function that
exercises QEMU based on inputs provided by the fuzzer.
- Fuzz targets rely on qtest and libqos to turn inputs into actions.
- Since libfuzzer does in-process fuzzing, the QEMU state needs to be
reset after each fuzz run. These patches provide three methods for
resetting state.
- There are currently few targets, but they have already helped
discover bugs in the console, and virtio-net, and have reproduced
previously-reported vulnerabilities.
Here are some main implementation details:
- The fuzzing occurs within a single process. QTest and QOS are
modified so the QTest client and server coexist within the same
process. They communicate with each other through direct function
calls. Similar to qtest, the fuzzer uses a lightweight accelerator to
skip CPU emulation. The fuzzing target is responsible for manually
executing the main loop.
- Since the same process is reused for many fuzzing runs, QEMU state
needs to be reset at the end of each run. There are currently three
implemented options for resetting state:
1. Reboot the guest between runs.
Pros: Straightforward and fast for simple fuzz targets.
Cons: Depending on the device, does not reset all device state. If
the device requires some initialization prior to being ready for
fuzzing (common for QOS-based targets), this initialization needs
to be done after each reboot.
Example target: --virtio-net-ctrl-fuzz
2. vmsave the state to RAM, once, and restore it after each run.
Alternatively, only save the device state
(savevm.c:qemu_save_device_state)
Pros: Do not need to initialize devices prior to each run.
VMStateDescriptions often specify more state than the device
resetting functions called during reboots.
Cons: Restoring state is often slower than rebooting. There is
currently no way to save the QOS object state, so the objects
usually needs to be re-allocated, defeating the purpose of
one-time device initialization.
Example target: --qtest-fuzz
3. Run each test case in a separate forked process and copy the
coverage information back to the parent. This is fairly similar to
AFL's "deferred" fork-server mode [3]
Pros: Relatively fast. Devices only need to be initialized once.
No need to do slow reboots or vmloads.
Cons: Not officially supported by libfuzzer and the implementation
is very flimsy. Does not work well for devices that rely on
dedicated threads.
Example target: --qtest-fork-fuzz
- Fuzz targets are registered using QEMU's module system, similar to
QOS test cases. Base qtest targets are registed with fuzz_add_target
and QOS-based targets with fuzz_add_qos_target.
- There are two entry points for the fuzzer:
LLVMFuzzerInitialize: Run once, prior to fuzzing. Here, we set up
qtest/qos, register the fuzz targets and partially execute vl.c:main.
This is also where we would take a snapshot, if using the vmsave
approach to resetting.
LLVMFuzzerTestOneInput: Run for each fuzzing input. This function is
responsible for taking care of device initialization, calling the
actual fuzz target, and resetting state at the end of each run.
Both of these functions are defined in tests/fuzz/fuzz.c
- There are many libfuzzer flags which should be used to configure the
coverage metrics and storage of interesting fuzz inputs. [2] These
flags can also be helpful in evaluating fuzzing performance through
metrics such as inputs/seconds and line-coverage.
Here are some key issues with the current state of the code:
- The patches change vl.c, main-loop.c, qtest.c, tests/libqtest.c,
savevm.c, memory.c. I wrapped the changes with #ifdef CONFIG_FUZZ,
but many of these changes can and should be avoided.
- tests/fuzz/qos_helpers.c is largely a copy of tests/qos-test.c.
- The fuzzer is not properly integrated into the build system.
Currently I simply added all of the necessary objects to
target/i386/Makefile.objs, but there should be a simple way to build
for other arches. The binary needs to be linked against libqemuutil,
libqtest, qos and the qos objects, and the requirements for softmmu
targets.
- Some of the fuzz targets leak memory during state-resetting that need
to be tracked down and fixed.
- As mentioned already, running each test in a separate process does
not seem to be supported by libfuzzer, and the implementation
reflects this (tests/fuzz/fuzzer_hooks.c)
- The existing fuzz targets should be cleaned up as they have issues
with memory alignment and contain redundant checks. The should also
use qtest's clock_step. The fork fuzz targets are dependant on
a hard-coded section size.
Building and running:
Libfuzzer requires clang.
$ CC=clang-7 CXX=clang++-7 ./configure --enable-fuzzing
$ make i386-softmmu/all
$ i386-softmmu/qemu-system-i386 --qtest-dma-fuzz -detect_leaks=0
Here "qtest-dma-fuzz" is the fuzz target name. Running qemu-system-i386
without any arguments should print all of the available fuzz targets.
The -help=1 command prints out the available libfuzzer options.
There are more details, including instructions for adding new fuzz
targets in docs/devel/fuzzing.txt
In the coming weeks I would like to fix the issues listed above, more
fuzzing targets, and ideally work on getting QEMU into oss-fuzz[4],
where it can be fuzzed continuously.
I appreciate any feedback. Thanks
-Alex
[1] https://wiki.qemu.org/Internships/ProjectIdeas/QtestOssFuzz
[2] Trophy Case section: http://lcamtuf.coredump.cx/afl/
[3] https://llvm.org/docs/LibFuzzer.html
[4] https://github.com/mirrorer/afl/blob/master/llvm_mode/README.llvm#L82
[5] https://github.com/google/oss-fuzz
Alexander Oleinik (19):
fuzz: add configure option and linker objects
fuzz: add FUZZ_TARGET type to qemu module system
fuzz: add fuzz accelerator
fuzz: Add qos support to fuzz targets
fuzz: expose qemu_savevm_state & skip state header
fuzz: Add ramfile for fast vmstate/vmload
fuzz: Modify libqtest to directly invoke qtest.c
fuzz: add shims to intercept libfuzzer init
fuzz: use mtree_info to find mapped addresses
fuzz: expose real_main (aka regular vl.c:main)
fuzz: add direct send/receive in qtest client
fuzz: hard-code all of the needed files for build
fuzz: add ctrl vq support to virtio-net in libqos
fuzz: hard-code a main-loop timeout
fuzz: add fuzz accelerator type
fuzz: add general fuzzer entrypoints
fuzz: add general qtest fuzz target
fuzz: Add virtio-net tx and ctrl fuzz targets
fuzz: Add documentation about the fuzzer to docs/
accel/fuzz.c | 47 ++++++
configure | 11 ++
docs/devel/fuzzing.txt | 145 +++++++++++++++++
include/qemu/module.h | 7 +-
include/sysemu/fuzz.h | 15 ++
include/sysemu/qtest.h | 7 +-
include/sysemu/sysemu.h | 4 +
memory.c | 34 ++++
migration/savevm.c | 8 +-
migration/savevm.h | 3 +
qtest.c | 19 ++-
target/i386/Makefile.objs | 19 +++
tests/fuzz/fuzz.c | 262 +++++++++++++++++++++++++++++++
tests/fuzz/fuzz.h | 96 ++++++++++++
tests/fuzz/fuzzer_hooks.c | 106 +++++++++++++
tests/fuzz/fuzzer_hooks.h | 9 ++
tests/fuzz/qos_fuzz.c | 63 ++++++++
tests/fuzz/qos_fuzz.h | 29 ++++
tests/fuzz/qos_helpers.c | 295 +++++++++++++++++++++++++++++++++++
tests/fuzz/qos_helpers.h | 17 ++
tests/fuzz/qtest_fuzz.c | 261 +++++++++++++++++++++++++++++++
tests/fuzz/qtest_fuzz.h | 38 +++++
tests/fuzz/ramfile.c | 127 +++++++++++++++
tests/fuzz/ramfile.h | 20 +++
tests/fuzz/virtio-net-fuzz.c | 226 +++++++++++++++++++++++++++
tests/libqos/virtio-net.c | 2 +-
tests/libqtest.c | 53 ++++++-
tests/libqtest.h | 6 +
util/main-loop.c | 3 +
vl.c | 21 ++-
30 files changed, 1945 insertions(+), 8 deletions(-)
create mode 100644 accel/fuzz.c
create mode 100644 docs/devel/fuzzing.txt
create mode 100644 include/sysemu/fuzz.h
create mode 100644 tests/fuzz/fuzz.c
create mode 100644 tests/fuzz/fuzz.h
create mode 100644 tests/fuzz/fuzzer_hooks.c
create mode 100644 tests/fuzz/fuzzer_hooks.h
create mode 100644 tests/fuzz/qos_fuzz.c
create mode 100644 tests/fuzz/qos_fuzz.h
create mode 100644 tests/fuzz/qos_helpers.c
create mode 100644 tests/fuzz/qos_helpers.h
create mode 100644 tests/fuzz/qtest_fuzz.c
create mode 100644 tests/fuzz/qtest_fuzz.h
create mode 100644 tests/fuzz/ramfile.c
create mode 100644 tests/fuzz/ramfile.h
create mode 100644 tests/fuzz/virtio-net-fuzz.c
--
2.20.1
