Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-06-17 Thread Michael S. Tsirkin
On Thu, Jun 06, 2019 at 08:34:17AM -0400, Michael S. Tsirkin wrote: > On Tue, Apr 30, 2019 at 06:41:14PM -0400, Michael S. Tsirkin wrote: > > On Mon, Feb 11, 2019 at 08:52:01AM +, David Riddoch wrote: > > > > > Presumably we would like this to be an optional feature, as > > > > > implementatio

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-06-06 Thread Michael S. Tsirkin
On Tue, Apr 30, 2019 at 06:41:14PM -0400, Michael S. Tsirkin wrote: > On Mon, Feb 11, 2019 at 08:52:01AM +, David Riddoch wrote: > > > > Presumably we would like this to be an optional feature, as > > > > implementations > > > > of packed mode already exist in the wild.  How about > > > > VIRT

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-05-03 Thread Michael S. Tsirkin
On Mon, Feb 11, 2019 at 08:52:01AM +, David Riddoch wrote: > > > Presumably we would like this to be an optional feature, as > > > implementations > > > of packed mode already exist in the wild.  How about > > > VIRTIO_F_RING_PACKED_AVAIL_IDX? > > > If I prepare a patch to the spec is there st

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-19 Thread Michael S. Tsirkin
On Tue, Feb 19, 2019 at 02:33:04PM +0800, Jason Wang wrote: > > On 2019/2/14 下午12:04, Michael S. Tsirkin wrote: > > On Thu, Feb 14, 2019 at 11:34:22AM +0800, Jason Wang wrote: > > > On 2019/2/14 上午1:30, Michael S. Tsirkin wrote: > > > > On Wed, Feb 13, 2019 at 06:33:50PM +0800, Jason Wang wrote: >

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-19 Thread Michael S. Tsirkin
On Tue, Feb 19, 2019 at 02:21:01PM +0800, Jason Wang wrote: > > On 2019/2/15 下午12:23, Michael S. Tsirkin wrote: > > On Fri, Feb 15, 2019 at 11:59:55AM +0800, Jason Wang wrote: > > > On 2019/2/14 上午11:41, Michael S. Tsirkin wrote: > > > > > I think it's as simple as increase the avail idx by X? Sin

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-18 Thread Jason Wang
On 2019/2/14 下午12:04, Michael S. Tsirkin wrote: On Thu, Feb 14, 2019 at 11:34:22AM +0800, Jason Wang wrote: On 2019/2/14 上午1:30, Michael S. Tsirkin wrote: On Wed, Feb 13, 2019 at 06:33:50PM +0800, Jason Wang wrote: On 2019/2/13 上午1:35, Michael S. Tsirkin wrote: On Tue, Feb 12, 2019 at 04:47

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-18 Thread Jason Wang
On 2019/2/15 下午12:23, Michael S. Tsirkin wrote: On Fri, Feb 15, 2019 at 11:59:55AM +0800, Jason Wang wrote: On 2019/2/14 上午11:41, Michael S. Tsirkin wrote: I think it's as simple as increase the avail idx by X? Since descriptor were used in order, device can just read the next X-1 descriptors

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-14 Thread Michael S. Tsirkin
On Fri, Feb 15, 2019 at 11:59:55AM +0800, Jason Wang wrote: > > On 2019/2/14 上午11:41, Michael S. Tsirkin wrote: > > > I think it's as simple as increase the avail idx by X? Since descriptor > > > were > > > used in order, device can just read the next X-1 descriptors in this case. > > > > > > Th

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-14 Thread Jason Wang
On 2019/2/14 上午11:41, Michael S. Tsirkin wrote: I think it's as simple as increase the avail idx by X? Since descriptor were used in order, device can just read the next X-1 descriptors in this case. Thanks Right so a spec change would be needed, it's not transparent to guest. With the cha

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-13 Thread Michael S. Tsirkin
On Thu, Feb 14, 2019 at 11:34:22AM +0800, Jason Wang wrote: > > On 2019/2/14 上午1:30, Michael S. Tsirkin wrote: > > On Wed, Feb 13, 2019 at 06:33:50PM +0800, Jason Wang wrote: > > > On 2019/2/13 上午1:35, Michael S. Tsirkin wrote: > > > > On Tue, Feb 12, 2019 at 04:47:08PM +, David Riddoch wrote:

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-13 Thread Michael S. Tsirkin
On Thu, Feb 14, 2019 at 11:21:54AM +0800, Jason Wang wrote: > > On 2019/2/13 下午11:20, Michael S. Tsirkin wrote: > > On Wed, Feb 13, 2019 at 06:00:41PM +0800, Jason Wang wrote: > > > On 2019/2/12 下午9:44, Michael S. Tsirkin wrote: > > > > On Tue, Feb 12, 2019 at 03:28:26PM +0800, Jason Wang wrote: >

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-13 Thread Jason Wang
On 2019/2/14 上午1:30, Michael S. Tsirkin wrote: On Wed, Feb 13, 2019 at 06:33:50PM +0800, Jason Wang wrote: On 2019/2/13 上午1:35, Michael S. Tsirkin wrote: On Tue, Feb 12, 2019 at 04:47:08PM +, David Riddoch wrote: I would prefer to have the write barrier before writing the idx. Well that

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-13 Thread Jason Wang
On 2019/2/13 下午11:20, Michael S. Tsirkin wrote: On Wed, Feb 13, 2019 at 06:00:41PM +0800, Jason Wang wrote: On 2019/2/12 下午9:44, Michael S. Tsirkin wrote: On Tue, Feb 12, 2019 at 03:28:26PM +0800, Jason Wang wrote: On 2019/2/12 下午1:08, Michael S. Tsirkin wrote: On Mon, Feb 11, 2019 at 02:58

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-13 Thread Michael S. Tsirkin
On Tue, Feb 12, 2019 at 03:03:06PM -0500, Rob Miller wrote: > > > On Tue, Feb 12, 2019 at 1:55 PM Michael S. Tsirkin wrote: > > On Fri, Feb 01, 2019 at 09:43:02AM -0800, Rob Miller wrote: > > Agreed that this is needed. > > > > I would also like to suggest splitting the F_IN_ORD

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-13 Thread Michael S. Tsirkin
On Wed, Feb 13, 2019 at 06:33:50PM +0800, Jason Wang wrote: > > On 2019/2/13 上午1:35, Michael S. Tsirkin wrote: > > On Tue, Feb 12, 2019 at 04:47:08PM +, David Riddoch wrote: > > > > > > > I would prefer to have the write barrier before writing the idx. > > > > > > Well that's driver overhead f

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-13 Thread Michael S. Tsirkin
On Wed, Feb 13, 2019 at 06:00:41PM +0800, Jason Wang wrote: > > On 2019/2/12 下午9:44, Michael S. Tsirkin wrote: > > On Tue, Feb 12, 2019 at 03:28:26PM +0800, Jason Wang wrote: > > > On 2019/2/12 下午1:08, Michael S. Tsirkin wrote: > > > > On Mon, Feb 11, 2019 at 02:58:30PM +, David Riddoch wrote:

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-13 Thread Jason Wang
On 2019/2/13 上午1:35, Michael S. Tsirkin wrote: On Tue, Feb 12, 2019 at 04:47:08PM +, David Riddoch wrote: I would prefer to have the write barrier before writing the idx. Well that's driver overhead for something device might never utilise in a given workload. If we are optimizing let's o

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-13 Thread Jason Wang
On 2019/2/12 下午9:44, Michael S. Tsirkin wrote: On Tue, Feb 12, 2019 at 03:28:26PM +0800, Jason Wang wrote: On 2019/2/12 下午1:08, Michael S. Tsirkin wrote: On Mon, Feb 11, 2019 at 02:58:30PM +, David Riddoch wrote: This can result in a very high rate of doorbells with some drivers, which c

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-13 Thread David Riddoch
Maybe we should use a flag in event suppression structure. This way device can switch between being notification driven and being driven by index reads. On first thought it seems hard to avoid races: On receiving a doorbell the device wishes to transition from doorbells to polling, so driver->

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-12 Thread Rob Miller
On Tue, Feb 12, 2019 at 1:55 PM Michael S. Tsirkin wrote: > On Fri, Feb 01, 2019 at 09:43:02AM -0800, Rob Miller wrote: > > Agreed that this is needed. > > > > I would also like to suggest splitting the F_IN_ORDER into > > F_RX_IN_ORDER and F_TX_IN_ORDER to support hw LRO implementations, > > whi

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-12 Thread Michael S. Tsirkin
On Mon, Feb 04, 2019 at 01:36:28PM +0800, Stefan Hajnoczi wrote: > On Fri, Feb 01, 2019 at 09:43:02AM -0800, Rob Miller wrote: > > Agreed that this is needed. > > > > I would also like to suggest splitting the F_IN_ORDER into > > F_RX_IN_ORDER and F_TX_IN_ORDER to support hw LRO implementations, >

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-12 Thread Michael S. Tsirkin
On Fri, Feb 01, 2019 at 09:43:02AM -0800, Rob Miller wrote: > Agreed that this is needed. > > I would also like to suggest splitting the F_IN_ORDER into > F_RX_IN_ORDER and F_TX_IN_ORDER to support hw LRO implementations, > which can be more of a scatter/gather than tx. This would allow > batchmod

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-12 Thread Michael S. Tsirkin
On Tue, Feb 12, 2019 at 04:47:08PM +, David Riddoch wrote: > > > > > > I would prefer to have the write barrier before writing the idx. > > > > Well that's driver overhead for something device might never utilise in > > > > a given workload. If we are optimizing let's optimize for speed. > > >

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-12 Thread David Riddoch
I would prefer to have the write barrier before writing the idx. Well that's driver overhead for something device might never utilise in a given workload. If we are optimizing let's optimize for speed. I think doing the barrier before writing idx is best for speed (see below). I don't see it

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-12 Thread Michael S. Tsirkin
On Tue, Feb 12, 2019 at 11:40:25AM +, David Riddoch wrote: > On 12/02/2019 05:08, Michael S. Tsirkin wrote: > > On Mon, Feb 11, 2019 at 02:58:30PM +, David Riddoch wrote: > > > > > > > This can result in a very high rate of doorbells with some > > > > > > > drivers, which can become a sever

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-12 Thread Michael S. Tsirkin
On Tue, Feb 12, 2019 at 03:28:26PM +0800, Jason Wang wrote: > > On 2019/2/12 下午1:08, Michael S. Tsirkin wrote: > > On Mon, Feb 11, 2019 at 02:58:30PM +, David Riddoch wrote: > > > > > > > This can result in a very high rate of doorbells with some > > > > > > > drivers, which can become a sever

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-12 Thread Michael S. Tsirkin
On Tue, Feb 12, 2019 at 11:40:25AM +, David Riddoch wrote: > On 12/02/2019 05:08, Michael S. Tsirkin wrote: > > On Mon, Feb 11, 2019 at 02:58:30PM +, David Riddoch wrote: > > > > > > > This can result in a very high rate of doorbells with some > > > > > > > drivers, which can become a sever

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-12 Thread David Riddoch
On 12/02/2019 05:08, Michael S. Tsirkin wrote: On Mon, Feb 11, 2019 at 02:58:30PM +, David Riddoch wrote: This can result in a very high rate of doorbells with some drivers, which can become a severe bottleneck (because x86 CPUs can't emit MMIOs at very high rates). Interesting. Is there an

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-11 Thread Jason Wang
On 2019/2/12 下午1:08, Michael S. Tsirkin wrote: On Mon, Feb 11, 2019 at 02:58:30PM +, David Riddoch wrote: This can result in a very high rate of doorbells with some drivers, which can become a severe bottleneck (because x86 CPUs can't emit MMIOs at very high rates). Interesting. Is there

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-11 Thread Michael S. Tsirkin
On Mon, Feb 11, 2019 at 02:58:30PM +, David Riddoch wrote: > > > > > This can result in a very high rate of doorbells with some > > > > > drivers, which can become a severe bottleneck (because x86 CPUs can't > > > > > emit > > > > > MMIOs at very high rates). > > > > Interesting. Is there any

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-11 Thread David Riddoch
This can result in a very high rate of doorbells with some drivers, which can become a severe bottleneck (because x86 CPUs can't emit MMIOs at very high rates). Interesting. Is there any data you could share to help guide the design? E.g. what's the highest rate of MMIO writes supported etc? On

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-11 Thread Michael S. Tsirkin
On Mon, Feb 11, 2019 at 08:52:01AM +, David Riddoch wrote: > On 11/02/2019 07:33, Michael S. Tsirkin wrote: > > On Fri, Feb 01, 2019 at 02:23:55PM +, David Riddoch wrote: > > > All, > > > > > > I'd like to propose a small extension to the packed virtqueue mode.  My > > > proposal is to add

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-11 Thread David Riddoch
On 11/02/2019 07:33, Michael S. Tsirkin wrote: On Fri, Feb 01, 2019 at 02:23:55PM +, David Riddoch wrote: All, I'd like to propose a small extension to the packed virtqueue mode.  My proposal is to add an offset/wrap field, written by the driver, indicating how many available descriptors ha

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-10 Thread Michael S. Tsirkin
On Fri, Feb 01, 2019 at 02:23:55PM +, David Riddoch wrote: > All, > > I'd like to propose a small extension to the packed virtqueue mode.  My > proposal is to add an offset/wrap field, written by the driver, indicating > how many available descriptors have been added to the ring. > > The reas

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-03 Thread Stefan Hajnoczi
On Fri, Feb 01, 2019 at 09:43:02AM -0800, Rob Miller wrote: > Agreed that this is needed. > > I would also like to suggest splitting the F_IN_ORDER into > F_RX_IN_ORDER and F_TX_IN_ORDER to support hw LRO implementations, > which can be more of a scatter/gather than tx. This would allow > batchmod

Re: [virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-01 Thread Rob Miller
Agreed that this is needed. I would also like to suggest splitting the F_IN_ORDER into F_RX_IN_ORDER and F_TX_IN_ORDER to support hw LRO implementations, which can be more of a scatter/gather than tx. This would allow batchmode for tx at least in packed rings. Finally, i would suggest a means to

[virtio-dev] RFC: Doorbell suppression, packed-ring mode and hardware offload

2019-02-01 Thread David Riddoch
All, I'd like to propose a small extension to the packed virtqueue mode.  My proposal is to add an offset/wrap field, written by the driver, indicating how many available descriptors have been added to the ring. The reason for wanting this is to improve performance of hardware devices.  Beca