[Qemu-devel] Re: [RFC] vhost-blk implementation
On Wed, Mar 24, 2010 at 10:58:50AM -0700, Badari Pulavarty wrote: > Michael S. Tsirkin wrote: >> On Tue, Mar 23, 2010 at 12:55:07PM -0700, Badari Pulavarty wrote: >> >>> Michael S. Tsirkin wrote: >>> On Tue, Mar 23, 2010 at 10:57:33AM -0700, Badari Pulavarty wrote: > Michael S. Tsirkin wrote: > >> On Mon, Mar 22, 2010 at 05:34:04PM -0700, Badari Pulavarty wrote: >> >>> Write Results: >>> == >>> >>> I see degraded IO performance when doing sequential IO write >>> tests with vhost-blk compared to virtio-blk. >>> >>> # time dd of=/dev/vda if=/dev/zero bs=2M oflag=direct >>> >>> I get ~110MB/sec with virtio-blk, but I get only ~60MB/sec with >>> vhost-blk. Wondering why ? >>> >> Try to look and number of interrupts and/or number of exits. >> > I checked interrupts and IO exits - there is no major noticeable > difference between > vhost-blk and virtio-blk scenerios. > >> It could also be that you are overrunning some queue. >> >> I don't see any exit mitigation strategy in your patch: >> when there are already lots of requests in a queue, it's usually >> a good idea to disable notifications and poll the >> queue as requests complete. That could help performance. >> > Do you mean poll eventfd for new requests instead of waiting for > new notifications ? > Where do you do that in vhost-net code ? > vhost_disable_notify does this. > Unlike network socket, since we are dealing with a file, there is > no ->poll support for it. > So I can't poll for the data. And also, Issue I am having is on > the write() side. > Not sure I understand. > I looked at it some more - I see 512K write requests on the > virtio-queue in both vhost-blk and virtio-blk cases. Both qemu or > vhost is doing synchronous writes to page cache (there is no write > batching in qemu that is affecting this case). I still puzzled on > why virtio-blk outperforms vhost-blk. > > Thanks, > Badari > If you say the number of requests is the same, we are left with: - requests are smaller for some reason? - something is causing retries? >>> No. IO requests sizes are exactly same (512K) in both cases. There >>> are no retries or >>> errors in both cases. One thing I am not clear is - for some reason >>> guest kernel >>> could push more data into virtio-ring in case of virtio-blk vs >>> vhost-blk. Is this possible ? >>> Does guest gets to run much sooner in virtio-blk case than vhost-blk >>> ? Sorry, if its dumb question - >>> I don't understand all the vhost details :( >>> >>> Thanks, >>> Badari >>> >> >> BTW, did you put the backend in non-blocking mode? >> As I said, vhost net passes non-blocking flag to >> socket backend, but vfs_write/read that you use does >> not have an option to do this. >> >> So we'll need to extend the backend to fix that, >> but just for demo purposes, you could set non-blocking >> mode on the file from userspace. >> >> > Michael, > > Atleast I understand why the performance difference now.. My debug > code is changed the behaviour of virtio-blk which confused me. > > 1) virtio-blk is able to batch up writes from various requests. > 2) virtio-blk offloads the writes to different thread > > Where as vhost-blk, I do each request syncrhonously. Hence > the difference. You are right - i have to make backend async. > I will working on handing off work to in-kernel threads. > I am not sure about IO completion handling and calling > vhost_add_used_and_signal() out of context. But let > me take a stab at it and ask your help if I run into > issues. > > Thanks, > Badari > The way I did it for vhost net, requests are synchronous but non-blocking. So if it can't be done directly, I delay it. -- MST
[Qemu-devel] Re: [RFC] vhost-blk implementation
Michael S. Tsirkin wrote: On Tue, Mar 23, 2010 at 12:55:07PM -0700, Badari Pulavarty wrote: Michael S. Tsirkin wrote: On Tue, Mar 23, 2010 at 10:57:33AM -0700, Badari Pulavarty wrote: Michael S. Tsirkin wrote: On Mon, Mar 22, 2010 at 05:34:04PM -0700, Badari Pulavarty wrote: Write Results: == I see degraded IO performance when doing sequential IO write tests with vhost-blk compared to virtio-blk. # time dd of=/dev/vda if=/dev/zero bs=2M oflag=direct I get ~110MB/sec with virtio-blk, but I get only ~60MB/sec with vhost-blk. Wondering why ? Try to look and number of interrupts and/or number of exits. I checked interrupts and IO exits - there is no major noticeable difference between vhost-blk and virtio-blk scenerios. It could also be that you are overrunning some queue. I don't see any exit mitigation strategy in your patch: when there are already lots of requests in a queue, it's usually a good idea to disable notifications and poll the queue as requests complete. That could help performance. Do you mean poll eventfd for new requests instead of waiting for new notifications ? Where do you do that in vhost-net code ? vhost_disable_notify does this. Unlike network socket, since we are dealing with a file, there is no ->poll support for it. So I can't poll for the data. And also, Issue I am having is on the write() side. Not sure I understand. I looked at it some more - I see 512K write requests on the virtio-queue in both vhost-blk and virtio-blk cases. Both qemu or vhost is doing synchronous writes to page cache (there is no write batching in qemu that is affecting this case). I still puzzled on why virtio-blk outperforms vhost-blk. Thanks, Badari If you say the number of requests is the same, we are left with: - requests are smaller for some reason? - something is causing retries? No. IO requests sizes are exactly same (512K) in both cases. There are no retries or errors in both cases. One thing I am not clear is - for some reason guest kernel could push more data into virtio-ring in case of virtio-blk vs vhost-blk. Is this possible ? Does guest gets to run much sooner in virtio-blk case than vhost-blk ? Sorry, if its dumb question - I don't understand all the vhost details :( Thanks, Badari BTW, did you put the backend in non-blocking mode? As I said, vhost net passes non-blocking flag to socket backend, but vfs_write/read that you use does not have an option to do this. So we'll need to extend the backend to fix that, but just for demo purposes, you could set non-blocking mode on the file from userspace. Michael, Atleast I understand why the performance difference now.. My debug code is changed the behaviour of virtio-blk which confused me. 1) virtio-blk is able to batch up writes from various requests. 2) virtio-blk offloads the writes to different thread Where as vhost-blk, I do each request syncrhonously. Hence the difference. You are right - i have to make backend async. I will working on handing off work to in-kernel threads. I am not sure about IO completion handling and calling vhost_add_used_and_signal() out of context. But let me take a stab at it and ask your help if I run into issues. Thanks, Badari
[Qemu-devel] Re: [RFC] vhost-blk implementation
On Tue, Mar 23, 2010 at 12:55:07PM -0700, Badari Pulavarty wrote: > Michael S. Tsirkin wrote: >> On Tue, Mar 23, 2010 at 10:57:33AM -0700, Badari Pulavarty wrote: >> >>> Michael S. Tsirkin wrote: >>> On Mon, Mar 22, 2010 at 05:34:04PM -0700, Badari Pulavarty wrote: > Write Results: > == > > I see degraded IO performance when doing sequential IO write > tests with vhost-blk compared to virtio-blk. > > # time dd of=/dev/vda if=/dev/zero bs=2M oflag=direct > > I get ~110MB/sec with virtio-blk, but I get only ~60MB/sec with > vhost-blk. Wondering why ? > Try to look and number of interrupts and/or number of exits. >>> I checked interrupts and IO exits - there is no major noticeable >>> difference between >>> vhost-blk and virtio-blk scenerios. >>> It could also be that you are overrunning some queue. I don't see any exit mitigation strategy in your patch: when there are already lots of requests in a queue, it's usually a good idea to disable notifications and poll the queue as requests complete. That could help performance. >>> Do you mean poll eventfd for new requests instead of waiting for new >>> notifications ? >>> Where do you do that in vhost-net code ? >>> >> >> vhost_disable_notify does this. >> >> >>> Unlike network socket, since we are dealing with a file, there is no >>> ->poll support for it. >>> So I can't poll for the data. And also, Issue I am having is on the >>> write() side. >>> >> >> Not sure I understand. >> >> >>> I looked at it some more - I see 512K write requests on the >>> virtio-queue in both vhost-blk and virtio-blk cases. Both qemu or >>> vhost is doing synchronous writes to page cache (there is no write >>> batching in qemu that is affecting this case). I still puzzled on >>> why virtio-blk outperforms vhost-blk. >>> >>> Thanks, >>> Badari >>> >> >> If you say the number of requests is the same, we are left with: >> - requests are smaller for some reason? >> - something is causing retries? >> > No. IO requests sizes are exactly same (512K) in both cases. There are > no retries or > errors in both cases. One thing I am not clear is - for some reason > guest kernel > could push more data into virtio-ring in case of virtio-blk vs > vhost-blk. Is this possible ? > Does guest gets to run much sooner in virtio-blk case than vhost-blk ? > Sorry, if its dumb question - > I don't understand all the vhost details :( > > Thanks, > Badari BTW, did you put the backend in non-blocking mode? As I said, vhost net passes non-blocking flag to socket backend, but vfs_write/read that you use does not have an option to do this. So we'll need to extend the backend to fix that, but just for demo purposes, you could set non-blocking mode on the file from userspace. -- MST
[Qemu-devel] Re: [RFC] vhost-blk implementation
On Tue, Mar 23, 2010 at 12:55:07PM -0700, Badari Pulavarty wrote: > Michael S. Tsirkin wrote: >> On Tue, Mar 23, 2010 at 10:57:33AM -0700, Badari Pulavarty wrote: >> >>> Michael S. Tsirkin wrote: >>> On Mon, Mar 22, 2010 at 05:34:04PM -0700, Badari Pulavarty wrote: > Write Results: > == > > I see degraded IO performance when doing sequential IO write > tests with vhost-blk compared to virtio-blk. > > # time dd of=/dev/vda if=/dev/zero bs=2M oflag=direct > > I get ~110MB/sec with virtio-blk, but I get only ~60MB/sec with > vhost-blk. Wondering why ? > Try to look and number of interrupts and/or number of exits. >>> I checked interrupts and IO exits - there is no major noticeable >>> difference between >>> vhost-blk and virtio-blk scenerios. >>> It could also be that you are overrunning some queue. I don't see any exit mitigation strategy in your patch: when there are already lots of requests in a queue, it's usually a good idea to disable notifications and poll the queue as requests complete. That could help performance. >>> Do you mean poll eventfd for new requests instead of waiting for new >>> notifications ? >>> Where do you do that in vhost-net code ? >>> >> >> vhost_disable_notify does this. >> >> >>> Unlike network socket, since we are dealing with a file, there is no >>> ->poll support for it. >>> So I can't poll for the data. And also, Issue I am having is on the >>> write() side. >>> >> >> Not sure I understand. >> >> >>> I looked at it some more - I see 512K write requests on the >>> virtio-queue in both vhost-blk and virtio-blk cases. Both qemu or >>> vhost is doing synchronous writes to page cache (there is no write >>> batching in qemu that is affecting this case). I still puzzled on >>> why virtio-blk outperforms vhost-blk. >>> >>> Thanks, >>> Badari >>> >> >> If you say the number of requests is the same, we are left with: >> - requests are smaller for some reason? >> - something is causing retries? >> > No. IO requests sizes are exactly same (512K) in both cases. There are > no retries or > errors in both cases. One thing I am not clear is - for some reason > guest kernel > could push more data into virtio-ring in case of virtio-blk vs > vhost-blk. Is this possible ? > Does guest gets to run much sooner in virtio-blk case than vhost-blk ? > Sorry, if its dumb question - > I don't understand all the vhost details :( > > Thanks, > Badari > You said you observed same number of requests in userspace versus kernel above. And request size is the same as well. But somehow more data is transferred? I'm confused. -- MST
[Qemu-devel] Re: [RFC] vhost-blk implementation
Michael S. Tsirkin wrote: On Tue, Mar 23, 2010 at 10:57:33AM -0700, Badari Pulavarty wrote: Michael S. Tsirkin wrote: On Mon, Mar 22, 2010 at 05:34:04PM -0700, Badari Pulavarty wrote: Write Results: == I see degraded IO performance when doing sequential IO write tests with vhost-blk compared to virtio-blk. # time dd of=/dev/vda if=/dev/zero bs=2M oflag=direct I get ~110MB/sec with virtio-blk, but I get only ~60MB/sec with vhost-blk. Wondering why ? Try to look and number of interrupts and/or number of exits. I checked interrupts and IO exits - there is no major noticeable difference between vhost-blk and virtio-blk scenerios. It could also be that you are overrunning some queue. I don't see any exit mitigation strategy in your patch: when there are already lots of requests in a queue, it's usually a good idea to disable notifications and poll the queue as requests complete. That could help performance. Do you mean poll eventfd for new requests instead of waiting for new notifications ? Where do you do that in vhost-net code ? vhost_disable_notify does this. Unlike network socket, since we are dealing with a file, there is no ->poll support for it. So I can't poll for the data. And also, Issue I am having is on the write() side. Not sure I understand. I looked at it some more - I see 512K write requests on the virtio-queue in both vhost-blk and virtio-blk cases. Both qemu or vhost is doing synchronous writes to page cache (there is no write batching in qemu that is affecting this case). I still puzzled on why virtio-blk outperforms vhost-blk. Thanks, Badari If you say the number of requests is the same, we are left with: - requests are smaller for some reason? - something is causing retries? No. IO requests sizes are exactly same (512K) in both cases. There are no retries or errors in both cases. One thing I am not clear is - for some reason guest kernel could push more data into virtio-ring in case of virtio-blk vs vhost-blk. Is this possible ? Does guest gets to run much sooner in virtio-blk case than vhost-blk ? Sorry, if its dumb question - I don't understand all the vhost details :( Thanks, Badari
[Qemu-devel] Re: [RFC] vhost-blk implementation
Michael S. Tsirkin wrote: On Mon, Mar 22, 2010 at 05:34:04PM -0700, Badari Pulavarty wrote: Write Results: == I see degraded IO performance when doing sequential IO write tests with vhost-blk compared to virtio-blk. # time dd of=/dev/vda if=/dev/zero bs=2M oflag=direct I get ~110MB/sec with virtio-blk, but I get only ~60MB/sec with vhost-blk. Wondering why ? Try to look and number of interrupts and/or number of exits. I checked interrupts and IO exits - there is no major noticeable difference between vhost-blk and virtio-blk scenerios. It could also be that you are overrunning some queue. I don't see any exit mitigation strategy in your patch: when there are already lots of requests in a queue, it's usually a good idea to disable notifications and poll the queue as requests complete. That could help performance. Do you mean poll eventfd for new requests instead of waiting for new notifications ? Where do you do that in vhost-net code ? Unlike network socket, since we are dealing with a file, there is no ->poll support for it. So I can't poll for the data. And also, Issue I am having is on the write() side. I looked at it some more - I see 512K write requests on the virtio-queue in both vhost-blk and virtio-blk cases. Both qemu or vhost is doing synchronous writes to page cache (there is no write batching in qemu that is affecting this case). I still puzzled on why virtio-blk outperforms vhost-blk. Thanks, Badari
[Qemu-devel] Re: [RFC] vhost-blk implementation
Michael S. Tsirkin wrote: On Mon, Mar 22, 2010 at 05:34:04PM -0700, Badari Pulavarty wrote: Write Results: == I see degraded IO performance when doing sequential IO write tests with vhost-blk compared to virtio-blk. # time dd of=/dev/vda if=/dev/zero bs=2M oflag=direct I get ~110MB/sec with virtio-blk, but I get only ~60MB/sec with vhost-blk. Wondering why ? Try to look and number of interrupts and/or number of exits. It could also be that you are overrunning some queue. Yeah. I don't see any exit mitigation strategy in your patch: when there are already lots of requests in a queue, it's usually a good idea to disable notifications and poll the queue as requests complete. That could help performance. Thanks for the suggestions. I will take a closer look. Thanks, Badari
[Qemu-devel] Re: [RFC] vhost-blk implementation
On Tue, Mar 23, 2010 at 10:57:33AM -0700, Badari Pulavarty wrote: > Michael S. Tsirkin wrote: >> On Mon, Mar 22, 2010 at 05:34:04PM -0700, Badari Pulavarty wrote: >> >>> Write Results: >>> == >>> >>> I see degraded IO performance when doing sequential IO write >>> tests with vhost-blk compared to virtio-blk. >>> >>> # time dd of=/dev/vda if=/dev/zero bs=2M oflag=direct >>> >>> I get ~110MB/sec with virtio-blk, but I get only ~60MB/sec with >>> vhost-blk. Wondering why ? >>> >> >> Try to look and number of interrupts and/or number of exits. >> > > I checked interrupts and IO exits - there is no major noticeable > difference between > vhost-blk and virtio-blk scenerios. >> It could also be that you are overrunning some queue. >> >> I don't see any exit mitigation strategy in your patch: >> when there are already lots of requests in a queue, it's usually >> a good idea to disable notifications and poll the >> queue as requests complete. That could help performance. >> > Do you mean poll eventfd for new requests instead of waiting for new > notifications ? > Where do you do that in vhost-net code ? vhost_disable_notify does this. > Unlike network socket, since we are dealing with a file, there is no > ->poll support for it. > So I can't poll for the data. And also, Issue I am having is on the > write() side. Not sure I understand. > I looked at it some more - I see 512K write requests on the > virtio-queue in both vhost-blk and virtio-blk cases. Both qemu or > vhost is doing synchronous writes to page cache (there is no write > batching in qemu that is affecting this case). I still puzzled on > why virtio-blk outperforms vhost-blk. > > Thanks, > Badari If you say the number of requests is the same, we are left with: - requests are smaller for some reason? - something is causing retries? -- MST
[Qemu-devel] Re: [RFC] vhost-blk implementation
On Mon, Mar 22, 2010 at 05:34:04PM -0700, Badari Pulavarty wrote: > Write Results: > == > > I see degraded IO performance when doing sequential IO write > tests with vhost-blk compared to virtio-blk. > > # time dd of=/dev/vda if=/dev/zero bs=2M oflag=direct > > I get ~110MB/sec with virtio-blk, but I get only ~60MB/sec with > vhost-blk. Wondering why ? Try to look and number of interrupts and/or number of exits. It could also be that you are overrunning some queue. I don't see any exit mitigation strategy in your patch: when there are already lots of requests in a queue, it's usually a good idea to disable notifications and poll the queue as requests complete. That could help performance. -- MST
