[Qemu-devel] [RFC] block io lost in the guest , possible related to qemu?
Hi Experts, We've seen guest block io lost in a VM.any response will be helpful environment is: guest os: Ubuntu 1304 running busy database workload with xfs on a disk export with virtio-blk the exported vdb has very high infight io over 300. Some times later a lot io process in D state, looks a lot requests is lost in below storage stack. We're use qemu-kvm 1.0, host kernel 3.4.51 In qemu log of virtio-blk.c I found below commit, I wonder is it possible the workload generate some unknown reqests to qemu that lost in virtio_blk_handle_read? I do some fio test myself, I cann't generate so call unknown request type. Any response will be helpful. Jack commit 9e72c45033770b81b536ac6091e91807247cc25a Author: Alexey Zaytsev Date: Thu Dec 13 09:03:43 2012 +0200 virtio-blk: Return UNSUPP for unknown request types Currently, all unknown requests are treated as VIRTIO_BLK_T_IN Signed-off-by: Alexey Zaytsev Signed-off-by: Stefan Hajnoczi diff --git a/hw/virtio-blk.c b/hw/virtio-blk.c index 92c745a..df57b35 100644 --- a/hw/virtio-blk.c +++ b/hw/virtio-blk.c @@ -398,10 +398,14 @@ static void virtio_blk_handle_request(VirtIOBlockReq *req, qemu_iovec_init_external(&req->qiov, &req->elem.out_sg[1], req->elem.out_num - 1); virtio_blk_handle_write(req, mrb); -} else { +} else if (type == VIRTIO_BLK_T_IN || type == VIRTIO_BLK_T_BARRIER) { +/* VIRTIO_BLK_T_IN is 0, so we can't just & it. */ qemu_iovec_init_external(&req->qiov, &req->elem.in_sg[0], req->elem.in_num - 1); virtio_blk_handle_read(req); +} else { +virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP); +g_free(req); } }
Re: [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu?
Hello Kevin & Stefan Any comments or wild guess about the bug? Regards, Jack On 10/25/2013 05:01 PM, Jack Wang wrote: > Hi Experts, > > We've seen guest block io lost in a VM.any response will be helpful > > environment is: > guest os: Ubuntu 1304 > running busy database workload with xfs on a disk export with virtio-blk > > the exported vdb has very high infight io over 300. Some times later a > lot io process in D state, looks a lot requests is lost in below storage > stack. > > We're use qemu-kvm 1.0, host kernel 3.4.51 > > In qemu log of virtio-blk.c > I found below commit, I wonder is it possible the workload generate some > unknown reqests to qemu that lost in virtio_blk_handle_read? > I do some fio test myself, I cann't generate so call unknown request type. > > Any response will be helpful. > > Jack > > > commit 9e72c45033770b81b536ac6091e91807247cc25a > Author: Alexey Zaytsev > Date: Thu Dec 13 09:03:43 2012 +0200 > > virtio-blk: Return UNSUPP for unknown request types > > Currently, all unknown requests are treated as VIRTIO_BLK_T_IN > > Signed-off-by: Alexey Zaytsev > Signed-off-by: Stefan Hajnoczi > > diff --git a/hw/virtio-blk.c b/hw/virtio-blk.c > index 92c745a..df57b35 100644 > --- a/hw/virtio-blk.c > +++ b/hw/virtio-blk.c > @@ -398,10 +398,14 @@ static void > virtio_blk_handle_request(VirtIOBlockReq *req, > qemu_iovec_init_external(&req->qiov, &req->elem.out_sg[1], > req->elem.out_num - 1); > virtio_blk_handle_write(req, mrb); > -} else { > +} else if (type == VIRTIO_BLK_T_IN || type == VIRTIO_BLK_T_BARRIER) { > +/* VIRTIO_BLK_T_IN is 0, so we can't just & it. */ > qemu_iovec_init_external(&req->qiov, &req->elem.in_sg[0], > req->elem.in_num - 1); > virtio_blk_handle_read(req); > +} else { > +virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP); > +g_free(req); > } > } >
Re: [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu?
On 10/28/2013 10:54 AM, Alexey Zaytsev wrote: > Hey. > > I very much doubt this commit could be causing the problem, as qemu > would never set wrong request type in the first place. You can easily > check by either reverting it, or adding a printk() before > virtio_blk_req_complete(VIRTIO_BLK_S_UNSUPP). Hi Alexey, Thanks for you input. According to my test results, yes, as you said, virtio-blk never generate wrong request type. So the commit is only a small cosmetic extra check:( As there's nothing abnormal in host server and storage, there must be some hidden bug somewhere, damn it. Regards, Jack > > On Mon, Oct 28, 2013 at 10:15 AM, Jack Wang wrote: >> Hello Kevin & Stefan >> >> Any comments or wild guess about the bug? >> >> Regards, >> Jack >> >> On 10/25/2013 05:01 PM, Jack Wang wrote: >>> Hi Experts, >>> >>> We've seen guest block io lost in a VM.any response will be helpful >>> >>> environment is: >>> guest os: Ubuntu 1304 >>> running busy database workload with xfs on a disk export with virtio-blk >>> >>> the exported vdb has very high infight io over 300. Some times later a >>> lot io process in D state, looks a lot requests is lost in below storage >>> stack. >>> >>> We're use qemu-kvm 1.0, host kernel 3.4.51 >>> >>> In qemu log of virtio-blk.c >>> I found below commit, I wonder is it possible the workload generate some >>> unknown reqests to qemu that lost in virtio_blk_handle_read? >>> I do some fio test myself, I cann't generate so call unknown request type. >>> >>> Any response will be helpful. >>> >>> Jack >>> >>> >>> commit 9e72c45033770b81b536ac6091e91807247cc25a >>> Author: Alexey Zaytsev >>> Date: Thu Dec 13 09:03:43 2012 +0200 >>> >>> virtio-blk: Return UNSUPP for unknown request types >>> >>> Currently, all unknown requests are treated as VIRTIO_BLK_T_IN >>> >>> Signed-off-by: Alexey Zaytsev >>> Signed-off-by: Stefan Hajnoczi >>> >>> diff --git a/hw/virtio-blk.c b/hw/virtio-blk.c >>> index 92c745a..df57b35 100644 >>> --- a/hw/virtio-blk.c >>> +++ b/hw/virtio-blk.c >>> @@ -398,10 +398,14 @@ static void >>> virtio_blk_handle_request(VirtIOBlockReq *req, >>> qemu_iovec_init_external(&req->qiov, &req->elem.out_sg[1], >>> req->elem.out_num - 1); >>> virtio_blk_handle_write(req, mrb); >>> -} else { >>> +} else if (type == VIRTIO_BLK_T_IN || type == VIRTIO_BLK_T_BARRIER) { >>> +/* VIRTIO_BLK_T_IN is 0, so we can't just & it. */ >>> qemu_iovec_init_external(&req->qiov, &req->elem.in_sg[0], >>> req->elem.in_num - 1); >>> virtio_blk_handle_read(req); >>> +} else { >>> +virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP); >>> +g_free(req); >>> } >>> } >>> >>
Re: [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu?
Hey. I very much doubt this commit could be causing the problem, as qemu would never set wrong request type in the first place. You can easily check by either reverting it, or adding a printk() before virtio_blk_req_complete(VIRTIO_BLK_S_UNSUPP). On Mon, Oct 28, 2013 at 10:15 AM, Jack Wang wrote: > Hello Kevin & Stefan > > Any comments or wild guess about the bug? > > Regards, > Jack > > On 10/25/2013 05:01 PM, Jack Wang wrote: >> Hi Experts, >> >> We've seen guest block io lost in a VM.any response will be helpful >> >> environment is: >> guest os: Ubuntu 1304 >> running busy database workload with xfs on a disk export with virtio-blk >> >> the exported vdb has very high infight io over 300. Some times later a >> lot io process in D state, looks a lot requests is lost in below storage >> stack. >> >> We're use qemu-kvm 1.0, host kernel 3.4.51 >> >> In qemu log of virtio-blk.c >> I found below commit, I wonder is it possible the workload generate some >> unknown reqests to qemu that lost in virtio_blk_handle_read? >> I do some fio test myself, I cann't generate so call unknown request type. >> >> Any response will be helpful. >> >> Jack >> >> >> commit 9e72c45033770b81b536ac6091e91807247cc25a >> Author: Alexey Zaytsev >> Date: Thu Dec 13 09:03:43 2012 +0200 >> >> virtio-blk: Return UNSUPP for unknown request types >> >> Currently, all unknown requests are treated as VIRTIO_BLK_T_IN >> >> Signed-off-by: Alexey Zaytsev >> Signed-off-by: Stefan Hajnoczi >> >> diff --git a/hw/virtio-blk.c b/hw/virtio-blk.c >> index 92c745a..df57b35 100644 >> --- a/hw/virtio-blk.c >> +++ b/hw/virtio-blk.c >> @@ -398,10 +398,14 @@ static void >> virtio_blk_handle_request(VirtIOBlockReq *req, >> qemu_iovec_init_external(&req->qiov, &req->elem.out_sg[1], >> req->elem.out_num - 1); >> virtio_blk_handle_write(req, mrb); >> -} else { >> +} else if (type == VIRTIO_BLK_T_IN || type == VIRTIO_BLK_T_BARRIER) { >> +/* VIRTIO_BLK_T_IN is 0, so we can't just & it. */ >> qemu_iovec_init_external(&req->qiov, &req->elem.in_sg[0], >> req->elem.in_num - 1); >> virtio_blk_handle_read(req); >> +} else { >> +virtio_blk_req_complete(req, VIRTIO_BLK_S_UNSUPP); >> +g_free(req); >> } >> } >> >
Re: [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu?
On Fri, Oct 25, 2013 at 05:01:54PM +0200, Jack Wang wrote: > We've seen guest block io lost in a VM.any response will be helpful > > environment is: > guest os: Ubuntu 1304 > running busy database workload with xfs on a disk export with virtio-blk > > the exported vdb has very high infight io over 300. Some times later a > lot io process in D state, looks a lot requests is lost in below storage > stack. Is the image file on a local file system or are you using a network storage system (e.g. NFS, Gluster, Ceph, Sheepdog)? If you run "vmstat 5" inside the guest, do you see "bi"/"bo" block I/O activity? If that number is very low or zero then there may be a starvation problem. If that number is reasonable then the workload is simply bottlenecked on disk I/O. virtio-blk only has 128 descriptors available so it's not possible to have 300 requests pending at the virtio-blk layer. If you suspect QEMU, try building qemu.git/master from source in case the bug has already been fixed. If you want to trace I/O requests, you might find this blog post on writing trace analysis scripts useful: http://blog.vmsplice.net/2011/03/how-to-write-trace-analysis-scripts-for.html Stefan
Re: [Qemu-devel] [RFC] block io lost in the guest , possible related to qemu?
On 10/30/2013 10:50 AM, Stefan Hajnoczi wrote: > On Fri, Oct 25, 2013 at 05:01:54PM +0200, Jack Wang wrote: >> We've seen guest block io lost in a VM.any response will be helpful >> >> environment is: >> guest os: Ubuntu 1304 >> running busy database workload with xfs on a disk export with virtio-blk >> >> the exported vdb has very high infight io over 300. Some times later a >> lot io process in D state, looks a lot requests is lost in below storage >> stack. > > Is the image file on a local file system or are you using a network > storage system (e.g. NFS, Gluster, Ceph, Sheepdog)? > > If you run "vmstat 5" inside the guest, do you see "bi"/"bo" block I/O > activity? If that number is very low or zero then there may be a > starvation problem. If that number is reasonable then the workload is > simply bottlenecked on disk I/O. > > virtio-blk only has 128 descriptors available so it's not possible to > have 300 requests pending at the virtio-blk layer. > > If you suspect QEMU, try building qemu.git/master from source in case > the bug has already been fixed. > > If you want to trace I/O requests, you might find this blog post on > writing trace analysis scripts useful: > http://blog.vmsplice.net/2011/03/how-to-write-trace-analysis-scripts-for.html > > Stefan > Thanks Stefan for your valuable input. The image is on device exported with InfiniBand srp/srpt. Will follow your suggestions to do further investigation. The 300 infight ios I memtioned is from the /proc/diskstats Field 9 -- # of I/Os currently in progress. Jack