Re: [Qemu-devel] [RFC 0/9] QContext: QOM class to support multiple event loops
On Mon, May 06, 2013 at 01:17:53PM -0500, mdroth wrote: > On Mon, May 06, 2013 at 09:54:14AM +0200, Paolo Bonzini wrote: > > Il 03/05/2013 18:03, Michael Roth ha scritto: > > > These patches apply on top of qemu.git master, and can also be obtained > > > from: > > > git://github.com/mdroth/qemu.git qcontext > > > > > > OVERVIEW > > > > > > This series introduces a set of QOM classes/interfaces for event > > > registration/handling: QContext and QSource, which are based closely on > > > their GMainContext/GSource GLib counterparts. > > > > > > QContexts can be created via the command-line via -object, and can also be > > > intructed (via -object params/properties) to automatically start a > > > thread/event-loop to handle QSources we attach to them. > > > > This is an awesome idea. > > > > However, it seems a bit overengineered. Why do we need QSource at all? > > In my opinion, we should first change dataplane to use AioContext as a > > GSource, and benchmark it thoroughly. If it is fast enough, we can > > I think it would be great to just stick with GSources. I didn't want to > rely too heavily on GLib for the RFC since there seems to be some > reservations about relying too heavily on GLib for our > OneTrueEventLoop interface (mainly, lack of PI mutexes in the context of > real-time device threads, or other performance considerations that might > pop up and cause us to rethink our use of glib). > > However, knowing that we *could* do something like porting to QSources and > using a different QContext implementation if the need ever became > evident is enough for me, and I'm happy to drop QSources until we > actually need them. The GSource->QSource conversions would be mostly > mechanical. > > > GSource, and benchmark it thoroughly. If it is fast enough, we can > > "just" introduce a glib-based QContext and be done with it. Hopefully > > that is the case... > > Sounds good to me. I'll look into that more, and talk to some of our > performance folks who were involved with the virtio-blk dataplane > testing. Great. I see value in QOM, it allows event loop threads to be specified on the command-line and monitor. But it would be nice to drop QSource as well as the QContext inheritance hierarchy. BTW there should be a command analogous to query-cpus that lists the QContexts and their thread IDs. This way CPU affinity can be set similar to how we do it for vcpu threads today. Stefan
Re: [Qemu-devel] [RFC 0/9] QContext: QOM class to support multiple event loops
Il 06/05/2013 20:35, mdroth ha scritto: > In the case of the former, I think a wrapper around GLib that we can > instantiate from the command-line line and query properties like TIDs > from is necessary for robust control over event loops and CPU resources. > We get this essentially for free with QOM, so I think it makes sense to > use it. > > In the case of the latter I'm not too sure. Without the QSource > abstraction there isn't much reason not to use the native GLib > interfaces on the underlying GSources/GMainContexts directly. In which > case GlibQContext would only need to be a container of sorts with some > minor additions like spawning an event thread for itself. > > If we ever did need to switch it out in favor of a non-GLib > implementation, it should be a mostly mechanical conversion of > GSource->QSource and adding some wrappers around > g_main_context_prepare/check/etc. I'm not sure it is that easy, but I agree entirely with everything else. > Also along that line, if we're taking the approach of not adding > infrastructure/cruft until we actually have a plan to use it, it probably > makes sense to make QContext a concrete class implemented via GLib, and we > can move the GLib stuff to a sub-class later if we ever end up with another > QContext implementation. > > Does this seem reasonable? Yes, very much. Paolo
Re: [Qemu-devel] [RFC 0/9] QContext: QOM class to support multiple event loops
On Mon, May 06, 2013 at 11:26:06AM +0800, liu ping fan wrote: > On Sat, May 4, 2013 at 12:03 AM, Michael Roth > wrote: > > These patches apply on top of qemu.git master, and can also be obtained > > from: > > git://github.com/mdroth/qemu.git qcontext > > > > OVERVIEW > > > > This series introduces a set of QOM classes/interfaces for event > > registration/handling: QContext and QSource, which are based closely on > > their GMainContext/GSource GLib counterparts. > > > > QContexts can be created via the command-line via -object, and can also be > > intructed (via -object params/properties) to automatically start a > > thread/event-loop to handle QSources we attach to them. > > > > The reference implementation of QContext is GlibQContext, which uses > > GMainContext/GSource interfaces underneath the covers, thus we can > > also attach GSource (and as a result, AioContexts) to it. > > > > As part of this series we also port the QEMU main loop to using QContext > > and drop virtio-blk's dataplane thread in favor of a GlibQContext thread, > > which virtio-blk dataplane attaches to via it's AioContext's GSource. > > > > With these patches in place we can do virtio-blk dataplane assignment > > like so: > > > > qemu ... \ > > -object glib-qcontext,id=ctx0,threaded=yes > > -drive file=file.raw,id=drive0,aio=native,cache=none \ > > -device virtio-blk,drive=drive0,scsi=off,x-data-plane=on,context=ctx0 > > > > And also gain the ability to assign a virtio-blk dataplane to the default > > QContext driven by the QEMU main iothread: > > > > qemu ... \ > > -drive file=file.raw,id=drive0,aio=native,cache=none \ > > -device virtio-blk,drive=drive0,scsi=off,x-data-plane=on,context=main > > > > The latter likely isn't particularly desirable, and the series is in rough > > shape in general, but the goal of this RFC to demonstrate the approach and > > get some feedback on how we might handle thread assignments for things like > > virtio-blk/virtio-net dataplane, and multi-threaded device models general. > > > > Input on this would be greatly appreciated. > > > > BACKGROUND > > > > There has been an outgoing discussion on qemu-devel about what event loop > > interface to consolidate around for virtio-blk dataplane, threaded > > virtio-net, > > and offloading device workloads to seperate threads in general. > > > > Currently the main candidates seem to be GLib GSources and AioContext, with > > virtio-blk/virtio-net dataplane being the use-cases under consideration. > > > > virtio-blk: > > > > In the case of virtio-blk dataplane, we need to drive virtqueue and AIO > > events. > > Since AioContext is used extensively throughout the block layer to drive > > AIO, > > it makes sense to re-use it here as we look toward pushing dataplane > > functionality deeper into the block layer to benefit from things like image > > format support/snapshots/etc. > > > > virtio-net: > > > > In the case of Ping Fan's virtio-net dataplane patches > > (http://thread.gmane.org/gmane.comp.emulators.qemu/196111/focus=196111), we > > need to drive virtqueue and NetClient peer events (such as TAP devices, or > > possibly things like slirp in the future). Currently NetClient events rely > > on > > IOHandler interfaces such as qemu_set_fd_handler(). These interfaces are > > global > > ones that rely on a single IOHandler list serviced by QEMU's main loop. An > > effort is currently underway to port these to GSources so that can be more > > easilly attached to other event loops (as opposed to the hooks used for the > > virtio-net dataplane series). > > > > Theoretically, much of the latter (such as TAP devices) can also be done > > around > > AioContext with some minor changes, but other NetClient backends such as > > slirp > > lend themselves to more open-ended event loop interfaces like GSources. > > Other > > devices might also find themselves needing something more open-ended (a > > somewhat > > silly but present example being virtio-serial + GSource-driven chardev) > > > > QContext: > > > > Since the direction for the forseeable future will likely continue to be > > GSources for some things, AioContext for others, a way to reconcile these > > approaches would be the age-old approach of adding a layer of abstration on > > top of the 2 so that we can handle device/backend thread assignments using > > a general mechanism. Building around this abstration also leaves open the > > ability to deal with things like locking considerations for real-time > > support > > in the future. > > > > A reasonable start to modeling abstraction layer this would be the > > open-ended > > GMainContext/GSource approach that QEMU relies on already, which is what > > the QContext/QSource interfaces do (with some minor differences/additions > > such as QSources storing and opaque instead of the GSource-subclassing > > approach > > for GLib). > > > I think, custom-ed function for readable or not , ex, tap_can_send() > should be passed into QSourc
Re: [Qemu-devel] [RFC 0/9] QContext: QOM class to support multiple event loops
On Mon, May 06, 2013 at 07:25:24AM -0500, Anthony Liguori wrote: > Paolo Bonzini writes: > > > Il 03/05/2013 18:03, Michael Roth ha scritto: > >> These patches apply on top of qemu.git master, and can also be obtained > >> from: > >> git://github.com/mdroth/qemu.git qcontext > >> > >> OVERVIEW > >> > >> This series introduces a set of QOM classes/interfaces for event > >> registration/handling: QContext and QSource, which are based closely on > >> their GMainContext/GSource GLib counterparts. > >> > >> QContexts can be created via the command-line via -object, and can also be > >> intructed (via -object params/properties) to automatically start a > >> thread/event-loop to handle QSources we attach to them. > > > > This is an awesome idea. > > Ack. > > > However, it seems a bit overengineered. > > Ack. > > > Why do we need QSource at all? > > In my opinion, we should first change dataplane to use AioContext as a > > GSource, and benchmark it thoroughly. If it is fast enough, we can > > "just" introduce a glib-based QContext and be done with it. Hopefully > > that is the case... > > Why even bother with QContext then? The QContext/GlibQContext object in general, or the QContext base class? In the case of the former, I think a wrapper around GLib that we can instantiate from the command-line line and query properties like TIDs from is necessary for robust control over event loops and CPU resources. We get this essentially for free with QOM, so I think it makes sense to use it. In the case of the latter I'm not too sure. Without the QSource abstraction there isn't much reason not to use the native GLib interfaces on the underlying GSources/GMainContexts directly. In which case GlibQContext would only need to be a container of sorts with some minor additions like spawning an event thread for itself. If we ever did need to switch it out in favor of a non-GLib implementation, it should be a mostly mechanical conversion of GSource->QSource and adding some wrappers around g_main_context_prepare/check/etc. Also along that line, if we're taking the approach of not adding infrastructure/cruft until we actually have a plan to use it, it probably makes sense to make QContext a concrete class implemented via GLib, and we can move the GLib stuff to a sub-class later if we ever end up with another QContext implementation. Does this seem reasonable? > > Regards, > > Anthony Liguori > > > > > Paolo >
Re: [Qemu-devel] [RFC 0/9] QContext: QOM class to support multiple event loops
On Mon, May 06, 2013 at 09:54:14AM +0200, Paolo Bonzini wrote: > Il 03/05/2013 18:03, Michael Roth ha scritto: > > These patches apply on top of qemu.git master, and can also be obtained > > from: > > git://github.com/mdroth/qemu.git qcontext > > > > OVERVIEW > > > > This series introduces a set of QOM classes/interfaces for event > > registration/handling: QContext and QSource, which are based closely on > > their GMainContext/GSource GLib counterparts. > > > > QContexts can be created via the command-line via -object, and can also be > > intructed (via -object params/properties) to automatically start a > > thread/event-loop to handle QSources we attach to them. > > This is an awesome idea. > > However, it seems a bit overengineered. Why do we need QSource at all? > In my opinion, we should first change dataplane to use AioContext as a > GSource, and benchmark it thoroughly. If it is fast enough, we can I think it would be great to just stick with GSources. I didn't want to rely too heavily on GLib for the RFC since there seems to be some reservations about relying too heavily on GLib for our OneTrueEventLoop interface (mainly, lack of PI mutexes in the context of real-time device threads, or other performance considerations that might pop up and cause us to rethink our use of glib). However, knowing that we *could* do something like porting to QSources and using a different QContext implementation if the need ever became evident is enough for me, and I'm happy to drop QSources until we actually need them. The GSource->QSource conversions would be mostly mechanical. > GSource, and benchmark it thoroughly. If it is fast enough, we can > "just" introduce a glib-based QContext and be done with it. Hopefully > that is the case... Sounds good to me. I'll look into that more, and talk to some of our performance folks who were involved with the virtio-blk dataplane testing. > > Paolo >
Re: [Qemu-devel] [RFC 0/9] QContext: QOM class to support multiple event loops
Paolo Bonzini writes: > Il 03/05/2013 18:03, Michael Roth ha scritto: >> These patches apply on top of qemu.git master, and can also be obtained from: >> git://github.com/mdroth/qemu.git qcontext >> >> OVERVIEW >> >> This series introduces a set of QOM classes/interfaces for event >> registration/handling: QContext and QSource, which are based closely on >> their GMainContext/GSource GLib counterparts. >> >> QContexts can be created via the command-line via -object, and can also be >> intructed (via -object params/properties) to automatically start a >> thread/event-loop to handle QSources we attach to them. > > This is an awesome idea. Ack. > However, it seems a bit overengineered. Ack. > Why do we need QSource at all? > In my opinion, we should first change dataplane to use AioContext as a > GSource, and benchmark it thoroughly. If it is fast enough, we can > "just" introduce a glib-based QContext and be done with it. Hopefully > that is the case... Why even bother with QContext then? Regards, Anthony Liguori > > Paolo
Re: [Qemu-devel] [RFC 0/9] QContext: QOM class to support multiple event loops
Il 03/05/2013 18:03, Michael Roth ha scritto: > These patches apply on top of qemu.git master, and can also be obtained from: > git://github.com/mdroth/qemu.git qcontext > > OVERVIEW > > This series introduces a set of QOM classes/interfaces for event > registration/handling: QContext and QSource, which are based closely on > their GMainContext/GSource GLib counterparts. > > QContexts can be created via the command-line via -object, and can also be > intructed (via -object params/properties) to automatically start a > thread/event-loop to handle QSources we attach to them. This is an awesome idea. However, it seems a bit overengineered. Why do we need QSource at all? In my opinion, we should first change dataplane to use AioContext as a GSource, and benchmark it thoroughly. If it is fast enough, we can "just" introduce a glib-based QContext and be done with it. Hopefully that is the case... Paolo
Re: [Qemu-devel] [RFC 0/9] QContext: QOM class to support multiple event loops
On Sat, May 4, 2013 at 12:03 AM, Michael Roth wrote: > These patches apply on top of qemu.git master, and can also be obtained from: > git://github.com/mdroth/qemu.git qcontext > > OVERVIEW > > This series introduces a set of QOM classes/interfaces for event > registration/handling: QContext and QSource, which are based closely on > their GMainContext/GSource GLib counterparts. > > QContexts can be created via the command-line via -object, and can also be > intructed (via -object params/properties) to automatically start a > thread/event-loop to handle QSources we attach to them. > > The reference implementation of QContext is GlibQContext, which uses > GMainContext/GSource interfaces underneath the covers, thus we can > also attach GSource (and as a result, AioContexts) to it. > > As part of this series we also port the QEMU main loop to using QContext > and drop virtio-blk's dataplane thread in favor of a GlibQContext thread, > which virtio-blk dataplane attaches to via it's AioContext's GSource. > > With these patches in place we can do virtio-blk dataplane assignment > like so: > > qemu ... \ > -object glib-qcontext,id=ctx0,threaded=yes > -drive file=file.raw,id=drive0,aio=native,cache=none \ > -device virtio-blk,drive=drive0,scsi=off,x-data-plane=on,context=ctx0 > > And also gain the ability to assign a virtio-blk dataplane to the default > QContext driven by the QEMU main iothread: > > qemu ... \ > -drive file=file.raw,id=drive0,aio=native,cache=none \ > -device virtio-blk,drive=drive0,scsi=off,x-data-plane=on,context=main > > The latter likely isn't particularly desirable, and the series is in rough > shape in general, but the goal of this RFC to demonstrate the approach and > get some feedback on how we might handle thread assignments for things like > virtio-blk/virtio-net dataplane, and multi-threaded device models general. > > Input on this would be greatly appreciated. > > BACKGROUND > > There has been an outgoing discussion on qemu-devel about what event loop > interface to consolidate around for virtio-blk dataplane, threaded virtio-net, > and offloading device workloads to seperate threads in general. > > Currently the main candidates seem to be GLib GSources and AioContext, with > virtio-blk/virtio-net dataplane being the use-cases under consideration. > > virtio-blk: > > In the case of virtio-blk dataplane, we need to drive virtqueue and AIO > events. > Since AioContext is used extensively throughout the block layer to drive AIO, > it makes sense to re-use it here as we look toward pushing dataplane > functionality deeper into the block layer to benefit from things like image > format support/snapshots/etc. > > virtio-net: > > In the case of Ping Fan's virtio-net dataplane patches > (http://thread.gmane.org/gmane.comp.emulators.qemu/196111/focus=196111), we > need to drive virtqueue and NetClient peer events (such as TAP devices, or > possibly things like slirp in the future). Currently NetClient events rely on > IOHandler interfaces such as qemu_set_fd_handler(). These interfaces are > global > ones that rely on a single IOHandler list serviced by QEMU's main loop. An > effort is currently underway to port these to GSources so that can be more > easilly attached to other event loops (as opposed to the hooks used for the > virtio-net dataplane series). > > Theoretically, much of the latter (such as TAP devices) can also be done > around > AioContext with some minor changes, but other NetClient backends such as slirp > lend themselves to more open-ended event loop interfaces like GSources. Other > devices might also find themselves needing something more open-ended (a > somewhat > silly but present example being virtio-serial + GSource-driven chardev) > > QContext: > > Since the direction for the forseeable future will likely continue to be > GSources for some things, AioContext for others, a way to reconcile these > approaches would be the age-old approach of adding a layer of abstration on > top of the 2 so that we can handle device/backend thread assignments using > a general mechanism. Building around this abstration also leaves open the > ability to deal with things like locking considerations for real-time support > in the future. > > A reasonable start to modeling abstraction layer this would be the open-ended > GMainContext/GSource approach that QEMU relies on already, which is what > the QContext/QSource interfaces do (with some minor differences/additions > such as QSources storing and opaque instead of the GSource-subclassing > approach > for GLib). > I think, custom-ed function for readable or not , ex, tap_can_send() should be passed into QSource, but I failed to find such things in [PATCH 3/9] QSource: QEMU event source object. Do I miss it? Thanks and regards, Pingfan > TODO/KNOWN ISSUES > > - GTK UI causes a crash during certain window refresh events. works fine with >VNC though, and no problems with other GSources. > - slirp isn't worki
[Qemu-devel] [RFC 0/9] QContext: QOM class to support multiple event loops
These patches apply on top of qemu.git master, and can also be obtained from: git://github.com/mdroth/qemu.git qcontext OVERVIEW This series introduces a set of QOM classes/interfaces for event registration/handling: QContext and QSource, which are based closely on their GMainContext/GSource GLib counterparts. QContexts can be created via the command-line via -object, and can also be intructed (via -object params/properties) to automatically start a thread/event-loop to handle QSources we attach to them. The reference implementation of QContext is GlibQContext, which uses GMainContext/GSource interfaces underneath the covers, thus we can also attach GSource (and as a result, AioContexts) to it. As part of this series we also port the QEMU main loop to using QContext and drop virtio-blk's dataplane thread in favor of a GlibQContext thread, which virtio-blk dataplane attaches to via it's AioContext's GSource. With these patches in place we can do virtio-blk dataplane assignment like so: qemu ... \ -object glib-qcontext,id=ctx0,threaded=yes -drive file=file.raw,id=drive0,aio=native,cache=none \ -device virtio-blk,drive=drive0,scsi=off,x-data-plane=on,context=ctx0 And also gain the ability to assign a virtio-blk dataplane to the default QContext driven by the QEMU main iothread: qemu ... \ -drive file=file.raw,id=drive0,aio=native,cache=none \ -device virtio-blk,drive=drive0,scsi=off,x-data-plane=on,context=main The latter likely isn't particularly desirable, and the series is in rough shape in general, but the goal of this RFC to demonstrate the approach and get some feedback on how we might handle thread assignments for things like virtio-blk/virtio-net dataplane, and multi-threaded device models general. Input on this would be greatly appreciated. BACKGROUND There has been an outgoing discussion on qemu-devel about what event loop interface to consolidate around for virtio-blk dataplane, threaded virtio-net, and offloading device workloads to seperate threads in general. Currently the main candidates seem to be GLib GSources and AioContext, with virtio-blk/virtio-net dataplane being the use-cases under consideration. virtio-blk: In the case of virtio-blk dataplane, we need to drive virtqueue and AIO events. Since AioContext is used extensively throughout the block layer to drive AIO, it makes sense to re-use it here as we look toward pushing dataplane functionality deeper into the block layer to benefit from things like image format support/snapshots/etc. virtio-net: In the case of Ping Fan's virtio-net dataplane patches (http://thread.gmane.org/gmane.comp.emulators.qemu/196111/focus=196111), we need to drive virtqueue and NetClient peer events (such as TAP devices, or possibly things like slirp in the future). Currently NetClient events rely on IOHandler interfaces such as qemu_set_fd_handler(). These interfaces are global ones that rely on a single IOHandler list serviced by QEMU's main loop. An effort is currently underway to port these to GSources so that can be more easilly attached to other event loops (as opposed to the hooks used for the virtio-net dataplane series). Theoretically, much of the latter (such as TAP devices) can also be done around AioContext with some minor changes, but other NetClient backends such as slirp lend themselves to more open-ended event loop interfaces like GSources. Other devices might also find themselves needing something more open-ended (a somewhat silly but present example being virtio-serial + GSource-driven chardev) QContext: Since the direction for the forseeable future will likely continue to be GSources for some things, AioContext for others, a way to reconcile these approaches would be the age-old approach of adding a layer of abstration on top of the 2 so that we can handle device/backend thread assignments using a general mechanism. Building around this abstration also leaves open the ability to deal with things like locking considerations for real-time support in the future. A reasonable start to modeling abstraction layer this would be the open-ended GMainContext/GSource approach that QEMU relies on already, which is what the QContext/QSource interfaces do (with some minor differences/additions such as QSources storing and opaque instead of the GSource-subclassing approach for GLib). TODO/KNOWN ISSUES - GTK UI causes a crash during certain window refresh events. works fine with VNC though, and no problems with other GSources. - slirp isn't working (will work with Ping Fan's slirp->GSource conversion) - add proper locking - integrate timers into a QSource to make timer-event driveable in seperate QContext event loops - consider a command-line wrapper to -object, such as: -context ,[threaded=],[backend=] Makefile.objs|6 +- async.c | 16 +++ hw/block/dataplane/virtio-blk.c | 46 ++- include/block/aio.h |6 + include/hw/vi
