Hi, We've got an unusual use case for our qemu virtual machines. We're using a qemu-based system to run a bunch of instrumented test cases. QEMU provides the instrumented execution environment. Before we start testing we setup the test environment, which includes booting an operating system as a guest. Once everything is ready, we fork the qemu process once for each test case, with the parent controlling the test operation while the child performs a test. (For more details, see our blog post: https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2016/june/project-triforce-run-afl-on-everything/, full code can be found at https://github.com/nccgroup/TriforceAFL)
Right now our fork process is very hacky and gross, because linux fork() intentionally does not handle forking a multithreaded program. We have a special cpu instruction which triggers the forking to go into test mode. It sets a flag, which causes the one and only cpu thread to exit. Right before it exits, it sends a signal to the iothread by writing over a pipe. The iothread gets woken up and starts controlling test cases by forking children. After each fork, it starts up a new cpu to replace the one that exited before the fork. This is less than ideal for several reasons. Most importantly for us, we would like to be able to communicate informatoin about new JITs that occur from the child back to the parent so that the parent can reproduce the JIT and save future children from having to keep JITting the same thing. This doesnt currently work (I think because the cpu has exited, and there is no cpu left in the parent process). We would also like to have more elegant forking code. And so my question here -- Is there a better solution that we could use to support forking a qemu VM? Are there any single-threaded variants of qemu that would work well here? Are there any interesting tricks to pause and resume a cpu across forks in a clean fashion? Has anyone else already worked on this problem? -- Tim Newsham | www.thenewsh.com/~newsham | @newshtwit | thenewsh.blogspot.com
