[Qemu-devel] forking a virtual machine
Hi, We've got an unusual use case for our qemu virtual machines. We're using a qemu-based system to run a bunch of instrumented test cases. QEMU provides the instrumented execution environment. Before we start testing we setup the test environment, which includes booting an operating system as a guest. Once everything is ready, we fork the qemu process once for each test case, with the parent controlling the test operation while the child performs a test. (For more details, see our blog post: https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2016/june/project-triforce-run-afl-on-everything/, full code can be found at https://github.com/nccgroup/TriforceAFL) Right now our fork process is very hacky and gross, because linux fork() intentionally does not handle forking a multithreaded program. We have a special cpu instruction which triggers the forking to go into test mode. It sets a flag, which causes the one and only cpu thread to exit. Right before it exits, it sends a signal to the iothread by writing over a pipe. The iothread gets woken up and starts controlling test cases by forking children. After each fork, it starts up a new cpu to replace the one that exited before the fork. This is less than ideal for several reasons. Most importantly for us, we would like to be able to communicate informatoin about new JITs that occur from the child back to the parent so that the parent can reproduce the JIT and save future children from having to keep JITting the same thing. This doesnt currently work (I think because the cpu has exited, and there is no cpu left in the parent process). We would also like to have more elegant forking code. And so my question here -- Is there a better solution that we could use to support forking a qemu VM? Are there any single-threaded variants of qemu that would work well here? Are there any interesting tricks to pause and resume a cpu across forks in a clean fashion? Has anyone else already worked on this problem? -- Tim Newsham | www.thenewsh.com/~newsham | @newshtwit | thenewsh.blogspot.com
[Qemu-devel] basic block tracing question
Hi, I would like to create an accurate trace of basic blocks that get executed. I'm interested in a trace of what a CPU would execute, and not for the purposes of studying qemu itself. I'm currently emitting trace data from cpu_tb_exec https://github.com/qemu/qemu/blob/master/cpu-exec.c#L136 by printing out the env->eip (x86_64 only). This seems to be roughly the right place -- there's already cpu tracing in this function. I do notice that some basic blocks get printed twice here though, and I tracked it down to basic blocks being rescheduled if execution returns with TB_EXIT_* flags set https://github.com/qemu/qemu/blob/master/cpu-exec.c#L163 So I capture the PC before execution and only emit them if this is not the case, after execution. This gets rid of the duplicate edges in the trace, but there is still one problem left that I don't understand! Sometimes, when running the same program twice in a situation that should give the exact same trace, I see differences: exec 8100450a exec 81091130 -exec 812f2930 + basic block ff812f2930 returned with flag 3, setting pc to 812f285d +exec 812f285d exec 812f293d exec 81091142 In this case the basic block wasn't merely restarted. The PC was updated to a different value after the next_tb had the TB_EXIT_REQUESTED flag set. The particular basic block in question at 812f2930 ends with a callq to 0x812f2850 and then falls through to 0x812f293d. So I would expect to see the "..2930" and "..293d" in the trace, but not the "..285d" in the trace, unless it was just continuing mid-basic block after the exit? What exactly is going on here. What is the purpose of the TB_EXIT_REQUESTED here? What is causing execution to resume at "...285d" afterwards? Is there a simple way to filter out any extraneous basic blocks that get started midpoint (if that is the case) and keep the true basic block starts? Thank you in advance for any insights. -- Tim Newsham | www.thenewsh.com/~newsham | @newshtwit | thenewsh.blogspot.com
Re: [Qemu-devel] basic block tracing question
On Wed, Mar 16, 2016 at 10:52 AM, Peter Maydell <peter.mayd...@linaro.org> wrote: > > If you only emit tracing information after the TB has executed and > returned then you will miss the case where we execute half a TB > and take an exception (eg load/store that page faulted, or system call), > because in that case we'll longjmp() out of the generated code. That's > one of the reasons why the tracing we have in upstream traces before > TB execution. > What happens when the basic block gets interrupted mid execution and restarted? Will execution jump to the middle of the translated basic block, or will a new translation be performed starting at the midpoint? Firstly, are you running with -d nochain to disable QEMU's chaining > of TBs? (If not, then when we chain TBs together you'll only get > exec tracing for the first one, which is a good way to get confused. > The default tracing will tell you when we chain TBs together so you > can sort of unconfuse yourself, but it's easier to just turn it off > if you care about the TB logging.) > I was not using "-d nochain". Thank you! > thanks > -- PMM > -- Tim Newsham | www.thenewsh.com/~newsham | @newshtwit | thenewsh.blogspot.com
