We would like to contribute this patch for bug https://www.virtualbox.org/ticket/12611 under the MIT license. It fixes a crash in the VNC Extension of VirtualBoxVM when a VNC client connects while VirtualBox is still restoring saved state. The fix is to allocate a frame buffer of arbitrary size within VRDEEnableConnections that the VNC thread will use until VRDEResize.
Background: This is the standard sequence of events at VM startup: 1. VirtualBox calls extension entry point VRDECreateServer. 2. VirtualBox calls extension entry point VRDEEnableConnections, which listens for connections and starts VNC server thread. 3. VirtualBox calls extension entry point VRDEResize. Extension calls the callback VRDECallbackFramebufferQuery to obtain the new size and screen buffer, and allocates a frame buffer according to this size. 4. VirtualBox calls extension entry point VRDEUpdate whenever the pixels in the screen buffer change, and extension will copy the pixels from screen buffer to frame buffer. There can be a significant time between steps 2 and 3 (e.g., when VirtualBox has to restore saved state). If a VNC client happens to connect during that gap, then the VNC server tries to copy pixel data from the frame buffer to the client. Since the frame buffer was NULL at this point, this resulted in SEGV in the VNC event loop thread. Sample stack trace: Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000 Thread 41 Crashed: 0 libsystem_c.dylib 0x00007fff8bc9c9e7 memmove$VARIANT$sse42 + 31 1 libvncserver.0.dylib 0x000000010d3b5c73 rfbTranslateNone + 59 2 libvncserver.0.dylib 0x000000010d3b497d rfbSendRectEncodingHextile + 4630 3 libvncserver.0.dylib 0x000000010d3ae07e rfbSendFramebufferUpdate + 3314 4 libvncserver.0.dylib 0x000000010d3a92fb clientOutput + 320 5 libsystem_c.dylib 0x00007fff8bcaf742 _pthread_start + 327 6 libsystem_c.dylib 0x00007fff8bc9c181 thread_start + 13 Yonathan Randolph Acuitus
0001-Fix-bug-NULL-dereference-if-a-VNC-client-connects-wh.patch
Description: Binary data
_______________________________________________ vbox-dev mailing list [email protected] https://www.virtualbox.org/mailman/listinfo/vbox-dev
