By popular demand, here is a tutorial describing how to fuzz Vim with afl-fuzz (American Fuzzy Lop) on Linux.
For details about afl-fuzz, make sure you read: http://lcamtuf.coredump.cx/afl/ http://lcamtuf.coredump.cx/afl/README.txt I also recommend reading this fascinating blog entry: http://lcamtuf.blogspot.nl/2014/11/pulling-jpegs-out-of-thin-air.html afl-fuzz has successfully found several bugs in Vim: http://permalink.gmane.org/gmane.editors.vim.devel/54481 (Jan 10, 2016) http://permalink.gmane.org/gmane.editors.vim.devel/54447 (Jan 10, 2016) http://permalink.gmane.org/gmane.editors.vim.devel/54279 (Jan 5, 2016) http://permalink.gmane.org/gmane.editors.vim.devel/54261 (Jan 5, 2016) http://permalink.gmane.org/gmane.editors.vim.devel/54227 (Jan 4, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/52921 (Nov 7, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/52920 (Nov 6, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/52863 (Nov 4, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/52861 (Nov 4, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/52306 (Sep 27, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/51655 (Aug 13, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/51643 (Aug 12, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/51637 (Aug 12, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/51619 (Aug 11, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/51587 (Aug 9, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/51570 (Aug 8, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/51568 (Aug 7, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/51478 (Aug 1, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/51473 (Aug 1, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/51425 (Jul 28, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/51400 (Jul 26, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/51394 (Jul 26, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/50722 (May 23, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/50487 (Apr 28, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/50406 (Apr 22, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/50381 (Apr 20, 2015) http://permalink.gmane.org/gmane.editors.vim.devel/50120 (Mar 30, 2015) Steps to use afl-fuzz with Vim: Step 1) Download and install afl-fuzz $ wget http://lcamtuf.coredump.cx/afl/releases/afl-latest.tgz $ tar zxvf afl-latest.tgz $ cd afl-1.96b # the version might be different. $ make -j4 $ sudo make install Step 2) Build vim with afl-gcc to instrument vim $ cd vim $ make clean $ rm src/auto/config.cache $ CC=/usr/local/bin/afl-gcc ./configure \ --with-features=huge \ --enable-gui=none $ make -j4 Instrumentation with afl-gcc guides the fuzzer to find new code paths in Vim. Step 3) Setup input and output directories of afl-fuzz $ cd vim/src $ mkdir in/ $ mkdir out/ The out/ directory should be left empty, it is used by afl-fuzz to store its state, the discovered test cases that cause crashes, hangs and a corpus of test cases that exercise many paths in vim. You need to put at least one input file in the in/ directory. You also need to chose the options to start vim. afl-fuzz will first run vim with the provided input example(s) and then mutate the input to create other interesting test cases that trigger new code paths in vim. What follows are two examples: the first example takes an input regexp to fuzz vim regexp engine: Example #1: $ cat in/regexp a*b\+\|[0-9]\|\d{1,9} $ export AFL_USE_ASAN=1 $ afl-fuzz -M fuzzer01 -m 99999999 -t 4000 -i in/ -o out/ \ ./vim -u NONE -X -Z -e -s \ -c 'call search(getline("."))'' -c ':qa!' @@ The second example takes a VimL source file as input to stress the VimL parser. Example #2: Copy a viml file into in/ the directory and start afl-fuzz with: $ export AFL_USE_ASAN=1 $ afl-fuzz -M fuzzer01 -m 99999999 -t 4000 -i in/ -o out/ \ ./vim -u NONE -X -Z -e -s -S @@ -c ':qa!' The @@ part is automatically replaced by the test case file being tested and its mutations. After a few seconds of running afl-fuzz, you should expect to see the value 'path' which keeps incrementing. If it stays at 1, it means that something is wrong (afl-fuzz is not discovering paths). Make sure that you keep the input in the in/ directory not too large as large inputs will slow down fuzzing. In particular, do not store vim backup files there i.e. you should not put two almost identical files like in/ex.vim and in/ex.viml~ as they would contain mostly useless duplication. Remove comments from VimL input file, as fuzzing comments is most likely a waste of time. You then need to let afl-fuzz run for a long time (days, or weeks) but sometimes bugs can be found in a few seconds. Afl-fuzz displays many stats. The most important stat is the number of discovered crashes. For more information about afl-fuzz status screen, see: http://lcamtuf.coredump.cx/afl/status_screen.txt Test cases that cause a crash are stored in fuzzer01/crashes/. You can re-run them later for debugging the crash. When a crash is found, it's good to simplify it as much as possible while still reproducing the crash, either manually or by using the afl-tmin tool. Different inputs and different command line options will fuzz different parts of Vim and will thus find different bugs. This is where you should be creative by selecting the input and command line options to start vim. The above examples use options -u NONE and -X to speed up vim startup. The faster Vim starts up, the faster afl-fuzz will be able to find bugs. Options -e -s are used to make vim silent to avoid the 'MORE' prompt which could block vim. The option -Z disables external commands which makes fuzzing safer. WARNING: fuzzing could in theory come up with running shell commands that deletes your files, open and modify existing files, etc. To be safer, it is best to fuzz as a user created only for the purpose of fuzzing. Besides the fuzzer01/crashes/ directory, afl-fuzz will also create a directory fuzzer01/hangs/ for test cases that cause vim to hang. There can be false positives there, depending on which time-out value (-t option of afl-fuzz) was used. afl-fuzz also creates a corpus of test cases which exercise many code paths in Vim in out/fuzzer01/queue/. The corpus can be used later to test vim with slower checkers such as valgrind, ubsan, msan, etc. Running valgrind for example is useful to find use of uninitialized memory or memory leaks which are not detected when running afl-fuzz with asan. If you want to fuzz in parallel on several cores, make sure that you read: https://github.com/mirrorer/afl/blob/master/docs/parallel_fuzzing.txt For parallel fuzzing, I run the following command in a 2nd terminal for example: $ afl-fuzz -S fuzzer02 -m 99999999 -t 4000 -i in/ -o out/ \ ./vim -u NONE -Z -X -e -s -S @@ -c ':qa!' Then in a 3rd terminal: $ afl-fuzz -S fuzzer03 -m 99999999 -t 4000 -i in/ -o out/ \ ./vim -u NONE -Z -X -e -s -S @@ -c ':qa!' Etc. Make sure that the first fuzzer is started with the -M option (for master) and the subsequent parallel fuzzers are run with the -S option (for slave) with different IDs (fuzzer01, fuzzer02, fuzzer03, etc.) That's it. There is certainly more to it. In particular, I have not yet explored the dictionary feature of afl-fuzz, which could be a powerful feature to fuzz Vim. See: http://lcamtuf.blogspot.nl/2015/01/afl-fuzz-making-up-grammar-with.html Good lock finding and sharing interesting bugs in Vim found with afl-fuzz. Regards Dominique -- -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php --- You received this message because you are subscribed to the Google Groups "vim_dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
