Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload
On Tue, Feb 15, 2022 at 12:14:11PM +0100, Laszlo Ersek wrote: > On 02/15/22 11:43, Richard W.M. Jones wrote: > > On Mon, Feb 14, 2022 at 04:08:21PM +, Richard W.M. Jones wrote: > >> On Mon, Feb 14, 2022 at 04:52:17PM +0100, Laszlo Ersek wrote: > >>> On 02/14/22 14:01, Richard W.M. Jones wrote: > But nbdcopy needs to be reworked to make the input and output requests > separate, so that nbdcopy will coalesce and split blocks as it copies. > This is difficult. > > Another problem I'm finding (eg > https://bugzilla.redhat.com/show_bug.cgi?id=2039255#c9) is that > performance of new virt-v2v is extremely specific to input and output > mode, and hardware and network configurations. For reasons that I > don't fully understand. > >>> > >>> How are the nbdcopy source and destination coupled with each other? From > >>> work I'd done a decade ago, I remember that connecting two > >>> network-oriented (UDP) processes with a small-buffer pipe between them > >>> caused very bad effects. Whenever either process was blocked on the > >>> network (or on a timer, for example), the pipe went immediately full or > >>> empty (dependent on the particular blocked process), which in turn > >>> blocked the other process almost immediately. So the mitigation for that > >>> was to create a simple local app, to be inserted between the two > >>> network-oriented processes in the pipeline, just to de-couple them from > >>> each other, and make sure that a write to the pipe, or a read from it, > >>> would effectively never block. (The app-in-the-middle did have a maximum > >>> buffer size, but it was configurable, so not a practical limitation; it > >>> could be multiple tens of MB if needed.) > >>> > >>> If nbdcopy does some internal queueing (perhaps implicitly, i.e. by > >>> allowing multiple requests to be in flight at the same time), then > >>> seeing some stats on those "in real time" could be enlightening. > >> > >> So the way it works at the moment is it's event driven. Ignoring > >> extents to keep the description simple, we issue asynch read requests > >> (ie. nbd_aio_pread) and in the completion callbacks of those requests, > >> asynchronous write requests are started (ie. nbd_aio_pwrite). > >> > >> https://gitlab.com/nbdkit/libnbd/-/blob/6725fa0e129f9a60d7b89707ef8604e0aeeeaf43/copy/multi-thread-copying.c#L372 > >> > >> There is a limit on the number of parallel requests in flight > >> (nbdcopy --requests, default 64). This limits the implicit buffer to > >> max_requests * request_size. That's 16MB in the default > >> configuration. Quite small actually ... > >> > >> https://gitlab.com/nbdkit/libnbd/-/blob/6725fa0e129f9a60d7b89707ef8604e0aeeeaf43/copy/multi-thread-copying.c#L239 > > > > You might be on to something! > > > > I asked Ming Xie to run a special build of virt-v2v with all datapath > > debugging enabled and this allows me to calculate the size of the > > nbdcopy implicit buffer, ie. the value returned by the in_flight > > function (see second link above). > > > > The results (attached) show that the internal buffer is full (~ 64 > > requests) just about the whole time. (Note that because of request > > splitting, it's possible for the buffer to grow larger than 64 > > requests, which explains occasional bursts above this "limit".) > > > > Anyway I've done another build of virt-v2v which calls nbdcopy with > > --requests=1024, so we'll see if that improves performance. > > > > It may not do if the problem is really that one side is just slow. > > The above problem might combine with the small HTTP request size + > > synchronous request issue that Nir pointed out in his patch, if there > > are longer round trips on the QE machines than in my local testing. > > > > If --requests=1024 alone doesn't make any difference I'll try another > > test build that combines this with larger request size. > > It could be interesting to see src in_flight vs. dst in_flight. > > Also, even if one side is significantly slower than the other, that > slower speed may still not be reached end-to-end, if either end is > "bursty" (regardless of speed). A big buffer in the middle can smooth > over bursts, and so the experienced speed can approach the expected > average speed. The current log is not necessarily a sign of bad things > happening; after all we expect nbdcopy to be busy, so requests should be > in flight. I guess what I had in mind was this: one of the ends attempts > to produce or consume many requests in a burst, but there's no room (or > no data) for that burst in the buffer, so even though the wire speed > would allow for more, we're blocked elsewhere. Eventually the queue > should fill up, but then a bursty "get" should never block, > alternatively, eventually the queue should be almost always empty, but > then a bursty "put" should never block. With a small buffer however, > those "should not" cases can occur easily. > > Sorry that I can't express myself bette
Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload
On 02/15/22 11:43, Richard W.M. Jones wrote: > On Mon, Feb 14, 2022 at 04:08:21PM +, Richard W.M. Jones wrote: >> On Mon, Feb 14, 2022 at 04:52:17PM +0100, Laszlo Ersek wrote: >>> On 02/14/22 14:01, Richard W.M. Jones wrote: But nbdcopy needs to be reworked to make the input and output requests separate, so that nbdcopy will coalesce and split blocks as it copies. This is difficult. Another problem I'm finding (eg https://bugzilla.redhat.com/show_bug.cgi?id=2039255#c9) is that performance of new virt-v2v is extremely specific to input and output mode, and hardware and network configurations. For reasons that I don't fully understand. >>> >>> How are the nbdcopy source and destination coupled with each other? From >>> work I'd done a decade ago, I remember that connecting two >>> network-oriented (UDP) processes with a small-buffer pipe between them >>> caused very bad effects. Whenever either process was blocked on the >>> network (or on a timer, for example), the pipe went immediately full or >>> empty (dependent on the particular blocked process), which in turn >>> blocked the other process almost immediately. So the mitigation for that >>> was to create a simple local app, to be inserted between the two >>> network-oriented processes in the pipeline, just to de-couple them from >>> each other, and make sure that a write to the pipe, or a read from it, >>> would effectively never block. (The app-in-the-middle did have a maximum >>> buffer size, but it was configurable, so not a practical limitation; it >>> could be multiple tens of MB if needed.) >>> >>> If nbdcopy does some internal queueing (perhaps implicitly, i.e. by >>> allowing multiple requests to be in flight at the same time), then >>> seeing some stats on those "in real time" could be enlightening. >> >> So the way it works at the moment is it's event driven. Ignoring >> extents to keep the description simple, we issue asynch read requests >> (ie. nbd_aio_pread) and in the completion callbacks of those requests, >> asynchronous write requests are started (ie. nbd_aio_pwrite). >> >> https://gitlab.com/nbdkit/libnbd/-/blob/6725fa0e129f9a60d7b89707ef8604e0aeeeaf43/copy/multi-thread-copying.c#L372 >> >> There is a limit on the number of parallel requests in flight >> (nbdcopy --requests, default 64). This limits the implicit buffer to >> max_requests * request_size. That's 16MB in the default >> configuration. Quite small actually ... >> >> https://gitlab.com/nbdkit/libnbd/-/blob/6725fa0e129f9a60d7b89707ef8604e0aeeeaf43/copy/multi-thread-copying.c#L239 > > You might be on to something! > > I asked Ming Xie to run a special build of virt-v2v with all datapath > debugging enabled and this allows me to calculate the size of the > nbdcopy implicit buffer, ie. the value returned by the in_flight > function (see second link above). > > The results (attached) show that the internal buffer is full (~ 64 > requests) just about the whole time. (Note that because of request > splitting, it's possible for the buffer to grow larger than 64 > requests, which explains occasional bursts above this "limit".) > > Anyway I've done another build of virt-v2v which calls nbdcopy with > --requests=1024, so we'll see if that improves performance. > > It may not do if the problem is really that one side is just slow. > The above problem might combine with the small HTTP request size + > synchronous request issue that Nir pointed out in his patch, if there > are longer round trips on the QE machines than in my local testing. > > If --requests=1024 alone doesn't make any difference I'll try another > test build that combines this with larger request size. It could be interesting to see src in_flight vs. dst in_flight. Also, even if one side is significantly slower than the other, that slower speed may still not be reached end-to-end, if either end is "bursty" (regardless of speed). A big buffer in the middle can smooth over bursts, and so the experienced speed can approach the expected average speed. The current log is not necessarily a sign of bad things happening; after all we expect nbdcopy to be busy, so requests should be in flight. I guess what I had in mind was this: one of the ends attempts to produce or consume many requests in a burst, but there's no room (or no data) for that burst in the buffer, so even though the wire speed would allow for more, we're blocked elsewhere. Eventually the queue should fill up, but then a bursty "get" should never block, alternatively, eventually the queue should be almost always empty, but then a bursty "put" should never block. With a small buffer however, those "should not" cases can occur easily. Sorry that I can't express myself better... and this "burst" stuff may not apply to nbdcopy in the first place. Thanks Laszlo ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestf
Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload
On Mon, Feb 14, 2022 at 04:08:21PM +, Richard W.M. Jones wrote: > On Mon, Feb 14, 2022 at 04:52:17PM +0100, Laszlo Ersek wrote: > > On 02/14/22 14:01, Richard W.M. Jones wrote: > > > But nbdcopy needs to be reworked to make the input and output requests > > > separate, so that nbdcopy will coalesce and split blocks as it copies. > > > This is difficult. > > > > > > Another problem I'm finding (eg > > > https://bugzilla.redhat.com/show_bug.cgi?id=2039255#c9) is that > > > performance of new virt-v2v is extremely specific to input and output > > > mode, and hardware and network configurations. For reasons that I > > > don't fully understand. > > > > How are the nbdcopy source and destination coupled with each other? From > > work I'd done a decade ago, I remember that connecting two > > network-oriented (UDP) processes with a small-buffer pipe between them > > caused very bad effects. Whenever either process was blocked on the > > network (or on a timer, for example), the pipe went immediately full or > > empty (dependent on the particular blocked process), which in turn > > blocked the other process almost immediately. So the mitigation for that > > was to create a simple local app, to be inserted between the two > > network-oriented processes in the pipeline, just to de-couple them from > > each other, and make sure that a write to the pipe, or a read from it, > > would effectively never block. (The app-in-the-middle did have a maximum > > buffer size, but it was configurable, so not a practical limitation; it > > could be multiple tens of MB if needed.) > > > > If nbdcopy does some internal queueing (perhaps implicitly, i.e. by > > allowing multiple requests to be in flight at the same time), then > > seeing some stats on those "in real time" could be enlightening. > > So the way it works at the moment is it's event driven. Ignoring > extents to keep the description simple, we issue asynch read requests > (ie. nbd_aio_pread) and in the completion callbacks of those requests, > asynchronous write requests are started (ie. nbd_aio_pwrite). > > https://gitlab.com/nbdkit/libnbd/-/blob/6725fa0e129f9a60d7b89707ef8604e0aeeeaf43/copy/multi-thread-copying.c#L372 > > There is a limit on the number of parallel requests in flight > (nbdcopy --requests, default 64). This limits the implicit buffer to > max_requests * request_size. That's 16MB in the default > configuration. Quite small actually ... > > https://gitlab.com/nbdkit/libnbd/-/blob/6725fa0e129f9a60d7b89707ef8604e0aeeeaf43/copy/multi-thread-copying.c#L239 You might be on to something! I asked Ming Xie to run a special build of virt-v2v with all datapath debugging enabled and this allows me to calculate the size of the nbdcopy implicit buffer, ie. the value returned by the in_flight function (see second link above). The results (attached) show that the internal buffer is full (~ 64 requests) just about the whole time. (Note that because of request splitting, it's possible for the buffer to grow larger than 64 requests, which explains occasional bursts above this "limit".) Anyway I've done another build of virt-v2v which calls nbdcopy with --requests=1024, so we'll see if that improves performance. It may not do if the problem is really that one side is just slow. The above problem might combine with the small HTTP request size + synchronous request issue that Nir pointed out in his patch, if there are longer round trips on the QE machines than in my local testing. If --requests=1024 alone doesn't make any difference I'll try another test build that combines this with larger request size. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org buffer-size.txt.gz Description: application/gzip ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload
On Mon, Feb 14, 2022 at 04:08:21PM +, Richard W.M. Jones wrote: > There is a limit on the number of parallel requests in flight > (nbdcopy --requests, default 64). This limits the implicit buffer to > max_requests * request_size. That's 16MB in the default > configuration. Quite small actually ... > > https://gitlab.com/nbdkit/libnbd/-/blob/6725fa0e129f9a60d7b89707ef8604e0aeeeaf43/copy/multi-thread-copying.c#L239 > > I've managed to reproduce the problem locally now so I can try playing > with this limit. Or rather, I thought I'd reproduced it, but after repeating the test a few times it's just ordinary variability. Anyhow I tried increasing the nbdcopy --requests parameter to 64->256 but it didn't change the speed (faster or slower). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload
On Mon, Feb 14, 2022 at 04:52:17PM +0100, Laszlo Ersek wrote: > On 02/14/22 14:01, Richard W.M. Jones wrote: > > But nbdcopy needs to be reworked to make the input and output requests > > separate, so that nbdcopy will coalesce and split blocks as it copies. > > This is difficult. > > > > Another problem I'm finding (eg > > https://bugzilla.redhat.com/show_bug.cgi?id=2039255#c9) is that > > performance of new virt-v2v is extremely specific to input and output > > mode, and hardware and network configurations. For reasons that I > > don't fully understand. > > How are the nbdcopy source and destination coupled with each other? From > work I'd done a decade ago, I remember that connecting two > network-oriented (UDP) processes with a small-buffer pipe between them > caused very bad effects. Whenever either process was blocked on the > network (or on a timer, for example), the pipe went immediately full or > empty (dependent on the particular blocked process), which in turn > blocked the other process almost immediately. So the mitigation for that > was to create a simple local app, to be inserted between the two > network-oriented processes in the pipeline, just to de-couple them from > each other, and make sure that a write to the pipe, or a read from it, > would effectively never block. (The app-in-the-middle did have a maximum > buffer size, but it was configurable, so not a practical limitation; it > could be multiple tens of MB if needed.) > > If nbdcopy does some internal queueing (perhaps implicitly, i.e. by > allowing multiple requests to be in flight at the same time), then > seeing some stats on those "in real time" could be enlightening. So the way it works at the moment is it's event driven. Ignoring extents to keep the description simple, we issue asynch read requests (ie. nbd_aio_pread) and in the completion callbacks of those requests, asynchronous write requests are started (ie. nbd_aio_pwrite). https://gitlab.com/nbdkit/libnbd/-/blob/6725fa0e129f9a60d7b89707ef8604e0aeeeaf43/copy/multi-thread-copying.c#L372 There is a limit on the number of parallel requests in flight (nbdcopy --requests, default 64). This limits the implicit buffer to max_requests * request_size. That's 16MB in the default configuration. Quite small actually ... https://gitlab.com/nbdkit/libnbd/-/blob/6725fa0e129f9a60d7b89707ef8604e0aeeeaf43/copy/multi-thread-copying.c#L239 I've managed to reproduce the problem locally now so I can try playing with this limit. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload
On 02/14/22 14:01, Richard W.M. Jones wrote: > On Mon, Feb 14, 2022 at 12:53:01PM +0100, Laszlo Ersek wrote: >> On 02/14/22 10:56, Richard W.M. Jones wrote: >>> This change slowed things down (slightly) for me, although the change >>> is within the margin of error so it probably made no difference. >>> >>> Before: >>> >>> $ time ./run virt-v2v -i disk /var/tmp/fedora-35.qcow2 -o rhv-upload -oc >>> https://ovirt4410/ovirt-engine/api -op /tmp/ovirt-passwd -oo rhv-direct -os >>> ovirt-data -on test14 -of raw >>> [ 0.0] Setting up the source: -i disk /var/tmp/fedora-35.qcow2 >>> [ 1.0] Opening the source >>> [ 6.5] Inspecting the source >>> [ 10.5] Checking for sufficient free disk space in the guest >>> [ 10.5] Converting Fedora Linux 35 (Thirty Five) to run on KVM >>> virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown >>> device "vda". You may have to fix this entry manually after conversion. >>> virt-v2v: This guest has virtio drivers installed. >>> [ 57.0] Mapping filesystem data to avoid copying unused and blank areas >>> [ 59.0] Closing the overlay >>> [ 59.6] Assigning disks to buses >>> [ 59.6] Checking if the guest needs BIOS or UEFI to boot >>> [ 59.6] Setting up the destination: -o rhv-upload -oc >>> https://ovirt4410/ovirt-engine/api -os ovirt-data >>> [ 79.3] Copying disk 1/1 >>> █ 100% [] >>> [ 89.9] Creating output metadata >>> [ 94.0] Finishing off >>> >>> real 1m34.213s >>> user 0m6.585s >>> sys 0m11.880s >>> >>> >>> After: >>> >>> $ time ./run virt-v2v -i disk /var/tmp/fedora-35.qcow2 -o rhv-upload -oc >>> https://ovirt4410/ovirt-engine/api -op /tmp/ovirt-passwd -oo rhv-direct -os >>> ovirt-data -on test15 -of raw >>> [ 0.0] Setting up the source: -i disk /var/tmp/fedora-35.qcow2 >>> [ 1.0] Opening the source >>> [ 7.4] Inspecting the source >>> [ 11.7] Checking for sufficient free disk space in the guest >>> [ 11.7] Converting Fedora Linux 35 (Thirty Five) to run on KVM >>> virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown >>> device "vda". You may have to fix this entry manually after conversion. >>> virt-v2v: This guest has virtio drivers installed. >>> [ 59.6] Mapping filesystem data to avoid copying unused and blank areas >>> [ 61.5] Closing the overlay >>> [ 62.2] Assigning disks to buses >>> [ 62.2] Checking if the guest needs BIOS or UEFI to boot >>> [ 62.2] Setting up the destination: -o rhv-upload -oc >>> https://ovirt4410/ovirt-engine/api -os ovirt-data >>> [ 81.6] Copying disk 1/1 >>> █ 100% [] >>> [ 91.3] Creating output metadata >>> [ 96.0] Finishing off >>> >>> real 1m36.275s >>> user 0m4.700s >>> sys 0m14.070s >> >> My ACK on Nir's v2 patch basically means that I defer to you on its >> review -- I don't have anything against it, but I understand it's >> (perhaps a temporary) workaround until we find a more sustainable (and >> likely much more complex) solution. > > Sure, I don't mind taking this as a temporary solution. The code > itself is perfectly fine. The request size here is essentially an > optimization hint, it doesn't affect the architecture. > > An architectural problem that affects both nbdkit & nbdcopy is that > NBD commands drive the nbdkit backend and the nbdcopy loop. If we > make the nbdcopy --request-size larger, NBD commands ask for more > data, nbdkit-vddk-plugin makes larger VixDiskLib_ReadAsynch requests, > which at some point breaks the VMware server. (This is fairly easy to > solve in nbdkit-vddk-plugin or with a filter.) > > But nbdcopy needs to be reworked to make the input and output requests > separate, so that nbdcopy will coalesce and split blocks as it copies. > This is difficult. > > Another problem I'm finding (eg > https://bugzilla.redhat.com/show_bug.cgi?id=2039255#c9) is that > performance of new virt-v2v is extremely specific to input and output > mode, and hardware and network configurations. For reasons that I > don't fully understand. How are the nbdcopy source and destination coupled with each other? From work I'd done a decade ago, I remember that connecting two network-oriented (UDP) processes with a small-buffer pipe between them caused very bad effects. Whenever either process was blocked on the network (or on a timer, for example), the pipe went immediately full or empty (dependent on the particular blocked process), which in turn blocked the other process almost immediately. So the mitigation for that was to create a simple local app, to be inserted between the two network-oriented processes in the pipeline, just to de-couple them from each other, and make sure that a write to the pipe, or a read from it, would effectively never block. (The app-in-the-middle did have a maximum buffer size, but it was configurable, so not a practical limitation; it could be multiple tens of MB if needed.) If nbdcopy does some internal queueing (
Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload
On Mon, Feb 14, 2022 at 03:59:34PM +0200, Nir Soffer wrote: > On Mon, Feb 14, 2022 at 3:01 PM Richard W.M. Jones wrote: > > > > On Mon, Feb 14, 2022 at 12:53:01PM +0100, Laszlo Ersek wrote: > > > On 02/14/22 10:56, Richard W.M. Jones wrote: > > > > This change slowed things down (slightly) for me, although the change > > > > is within the margin of error so it probably made no difference. > > > > > > > > Before: > > > > > > > > $ time ./run virt-v2v -i disk /var/tmp/fedora-35.qcow2 -o rhv-upload > > > > -oc https://ovirt4410/ovirt-engine/api -op /tmp/ovirt-passwd -oo > > > > rhv-direct -os ovirt-data -on test14 -of raw > > > > [ 0.0] Setting up the source: -i disk /var/tmp/fedora-35.qcow2 > > > > [ 1.0] Opening the source > > > > [ 6.5] Inspecting the source > > > > [ 10.5] Checking for sufficient free disk space in the guest > > > > [ 10.5] Converting Fedora Linux 35 (Thirty Five) to run on KVM > > > > virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown > > > > device "vda". You may have to fix this entry manually after conversion. > > > > virt-v2v: This guest has virtio drivers installed. > > > > [ 57.0] Mapping filesystem data to avoid copying unused and blank areas > > > > [ 59.0] Closing the overlay > > > > [ 59.6] Assigning disks to buses > > > > [ 59.6] Checking if the guest needs BIOS or UEFI to boot > > > > [ 59.6] Setting up the destination: -o rhv-upload -oc > > > > https://ovirt4410/ovirt-engine/api -os ovirt-data > > > > [ 79.3] Copying disk 1/1 > > > > █ 100% [] > > > > [ 89.9] Creating output metadata > > > > [ 94.0] Finishing off > > > > > > > > real 1m34.213s > > > > user 0m6.585s > > > > sys 0m11.880s > > > > > > > > > > > > After: > > > > > > > > $ time ./run virt-v2v -i disk /var/tmp/fedora-35.qcow2 -o rhv-upload > > > > -oc https://ovirt4410/ovirt-engine/api -op /tmp/ovirt-passwd -oo > > > > rhv-direct -os ovirt-data -on test15 -of raw > > > > [ 0.0] Setting up the source: -i disk /var/tmp/fedora-35.qcow2 > > > > [ 1.0] Opening the source > > > > [ 7.4] Inspecting the source > > > > [ 11.7] Checking for sufficient free disk space in the guest > > > > [ 11.7] Converting Fedora Linux 35 (Thirty Five) to run on KVM > > > > virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown > > > > device "vda". You may have to fix this entry manually after conversion. > > > > virt-v2v: This guest has virtio drivers installed. > > > > [ 59.6] Mapping filesystem data to avoid copying unused and blank areas > > > > [ 61.5] Closing the overlay > > > > [ 62.2] Assigning disks to buses > > > > [ 62.2] Checking if the guest needs BIOS or UEFI to boot > > > > [ 62.2] Setting up the destination: -o rhv-upload -oc > > > > https://ovirt4410/ovirt-engine/api -os ovirt-data > > > > [ 81.6] Copying disk 1/1 > > > > █ 100% [] > > > > [ 91.3] Creating output metadata > > > > [ 96.0] Finishing off > > > > > > > > real 1m36.275s > > > > user 0m4.700s > > > > sys 0m14.070s > > > > > > My ACK on Nir's v2 patch basically means that I defer to you on its > > > review -- I don't have anything against it, but I understand it's > > > (perhaps a temporary) workaround until we find a more sustainable (and > > > likely much more complex) solution. > > > > Sure, I don't mind taking this as a temporary solution. The code > > itself is perfectly fine. The request size here is essentially an > > optimization hint, it doesn't affect the architecture. > > > > An architectural problem that affects both nbdkit & nbdcopy is that > > NBD commands drive the nbdkit backend and the nbdcopy loop. If we > > make the nbdcopy --request-size larger, NBD commands ask for more > > data, nbdkit-vddk-plugin makes larger VixDiskLib_ReadAsynch requests, > > which at some point breaks the VMware server. (This is fairly easy to > > solve in nbdkit-vddk-plugin or with a filter.) > > > > But nbdcopy needs to be reworked to make the input and output requests > > separate, so that nbdcopy will coalesce and split blocks as it copies. > > This is difficult. > > > > Another problem I'm finding (eg > > https://bugzilla.redhat.com/show_bug.cgi?id=2039255#c9) is that > > performance of new virt-v2v is extremely specific to input and output > > mode, and hardware and network configurations. For reasons that I > > don't fully understand. > > It would be interesting to test this patch in the same environment and see > how it affects the results. > > Do we see slow down only when using vddk? maybe it is related to the slow > extent calls? I was trying as far as possible to test the input and output sides separately, so I've only tested input from VDDK -> local disk, and local disk -> rhv-upload before. Now I am testing end to end because of Ming Xie's reports, but it's giving me very noisy results at the moment. Still trying ... Rich. -- Richard Jones, Virtualization G
Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload
On Mon, Feb 14, 2022 at 3:01 PM Richard W.M. Jones wrote: > > On Mon, Feb 14, 2022 at 12:53:01PM +0100, Laszlo Ersek wrote: > > On 02/14/22 10:56, Richard W.M. Jones wrote: > > > This change slowed things down (slightly) for me, although the change > > > is within the margin of error so it probably made no difference. > > > > > > Before: > > > > > > $ time ./run virt-v2v -i disk /var/tmp/fedora-35.qcow2 -o rhv-upload -oc > > > https://ovirt4410/ovirt-engine/api -op /tmp/ovirt-passwd -oo rhv-direct > > > -os ovirt-data -on test14 -of raw > > > [ 0.0] Setting up the source: -i disk /var/tmp/fedora-35.qcow2 > > > [ 1.0] Opening the source > > > [ 6.5] Inspecting the source > > > [ 10.5] Checking for sufficient free disk space in the guest > > > [ 10.5] Converting Fedora Linux 35 (Thirty Five) to run on KVM > > > virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown > > > device "vda". You may have to fix this entry manually after conversion. > > > virt-v2v: This guest has virtio drivers installed. > > > [ 57.0] Mapping filesystem data to avoid copying unused and blank areas > > > [ 59.0] Closing the overlay > > > [ 59.6] Assigning disks to buses > > > [ 59.6] Checking if the guest needs BIOS or UEFI to boot > > > [ 59.6] Setting up the destination: -o rhv-upload -oc > > > https://ovirt4410/ovirt-engine/api -os ovirt-data > > > [ 79.3] Copying disk 1/1 > > > █ 100% [] > > > [ 89.9] Creating output metadata > > > [ 94.0] Finishing off > > > > > > real 1m34.213s > > > user 0m6.585s > > > sys 0m11.880s > > > > > > > > > After: > > > > > > $ time ./run virt-v2v -i disk /var/tmp/fedora-35.qcow2 -o rhv-upload -oc > > > https://ovirt4410/ovirt-engine/api -op /tmp/ovirt-passwd -oo rhv-direct > > > -os ovirt-data -on test15 -of raw > > > [ 0.0] Setting up the source: -i disk /var/tmp/fedora-35.qcow2 > > > [ 1.0] Opening the source > > > [ 7.4] Inspecting the source > > > [ 11.7] Checking for sufficient free disk space in the guest > > > [ 11.7] Converting Fedora Linux 35 (Thirty Five) to run on KVM > > > virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown > > > device "vda". You may have to fix this entry manually after conversion. > > > virt-v2v: This guest has virtio drivers installed. > > > [ 59.6] Mapping filesystem data to avoid copying unused and blank areas > > > [ 61.5] Closing the overlay > > > [ 62.2] Assigning disks to buses > > > [ 62.2] Checking if the guest needs BIOS or UEFI to boot > > > [ 62.2] Setting up the destination: -o rhv-upload -oc > > > https://ovirt4410/ovirt-engine/api -os ovirt-data > > > [ 81.6] Copying disk 1/1 > > > █ 100% [] > > > [ 91.3] Creating output metadata > > > [ 96.0] Finishing off > > > > > > real 1m36.275s > > > user 0m4.700s > > > sys 0m14.070s > > > > My ACK on Nir's v2 patch basically means that I defer to you on its > > review -- I don't have anything against it, but I understand it's > > (perhaps a temporary) workaround until we find a more sustainable (and > > likely much more complex) solution. > > Sure, I don't mind taking this as a temporary solution. The code > itself is perfectly fine. The request size here is essentially an > optimization hint, it doesn't affect the architecture. > > An architectural problem that affects both nbdkit & nbdcopy is that > NBD commands drive the nbdkit backend and the nbdcopy loop. If we > make the nbdcopy --request-size larger, NBD commands ask for more > data, nbdkit-vddk-plugin makes larger VixDiskLib_ReadAsynch requests, > which at some point breaks the VMware server. (This is fairly easy to > solve in nbdkit-vddk-plugin or with a filter.) > > But nbdcopy needs to be reworked to make the input and output requests > separate, so that nbdcopy will coalesce and split blocks as it copies. > This is difficult. > > Another problem I'm finding (eg > https://bugzilla.redhat.com/show_bug.cgi?id=2039255#c9) is that > performance of new virt-v2v is extremely specific to input and output > mode, and hardware and network configurations. For reasons that I > don't fully understand. It would be interesting to test this patch in the same environment and see how it affects the results. Do we see slow down only when using vddk? maybe it is related to the slow extent calls? Nir ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload
On Mon, Feb 14, 2022 at 12:53:01PM +0100, Laszlo Ersek wrote: > On 02/14/22 10:56, Richard W.M. Jones wrote: > > This change slowed things down (slightly) for me, although the change > > is within the margin of error so it probably made no difference. > > > > Before: > > > > $ time ./run virt-v2v -i disk /var/tmp/fedora-35.qcow2 -o rhv-upload -oc > > https://ovirt4410/ovirt-engine/api -op /tmp/ovirt-passwd -oo rhv-direct -os > > ovirt-data -on test14 -of raw > > [ 0.0] Setting up the source: -i disk /var/tmp/fedora-35.qcow2 > > [ 1.0] Opening the source > > [ 6.5] Inspecting the source > > [ 10.5] Checking for sufficient free disk space in the guest > > [ 10.5] Converting Fedora Linux 35 (Thirty Five) to run on KVM > > virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown > > device "vda". You may have to fix this entry manually after conversion. > > virt-v2v: This guest has virtio drivers installed. > > [ 57.0] Mapping filesystem data to avoid copying unused and blank areas > > [ 59.0] Closing the overlay > > [ 59.6] Assigning disks to buses > > [ 59.6] Checking if the guest needs BIOS or UEFI to boot > > [ 59.6] Setting up the destination: -o rhv-upload -oc > > https://ovirt4410/ovirt-engine/api -os ovirt-data > > [ 79.3] Copying disk 1/1 > > █ 100% [] > > [ 89.9] Creating output metadata > > [ 94.0] Finishing off > > > > real 1m34.213s > > user 0m6.585s > > sys 0m11.880s > > > > > > After: > > > > $ time ./run virt-v2v -i disk /var/tmp/fedora-35.qcow2 -o rhv-upload -oc > > https://ovirt4410/ovirt-engine/api -op /tmp/ovirt-passwd -oo rhv-direct -os > > ovirt-data -on test15 -of raw > > [ 0.0] Setting up the source: -i disk /var/tmp/fedora-35.qcow2 > > [ 1.0] Opening the source > > [ 7.4] Inspecting the source > > [ 11.7] Checking for sufficient free disk space in the guest > > [ 11.7] Converting Fedora Linux 35 (Thirty Five) to run on KVM > > virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown > > device "vda". You may have to fix this entry manually after conversion. > > virt-v2v: This guest has virtio drivers installed. > > [ 59.6] Mapping filesystem data to avoid copying unused and blank areas > > [ 61.5] Closing the overlay > > [ 62.2] Assigning disks to buses > > [ 62.2] Checking if the guest needs BIOS or UEFI to boot > > [ 62.2] Setting up the destination: -o rhv-upload -oc > > https://ovirt4410/ovirt-engine/api -os ovirt-data > > [ 81.6] Copying disk 1/1 > > █ 100% [] > > [ 91.3] Creating output metadata > > [ 96.0] Finishing off > > > > real 1m36.275s > > user 0m4.700s > > sys 0m14.070s > > My ACK on Nir's v2 patch basically means that I defer to you on its > review -- I don't have anything against it, but I understand it's > (perhaps a temporary) workaround until we find a more sustainable (and > likely much more complex) solution. Sure, I don't mind taking this as a temporary solution. The code itself is perfectly fine. The request size here is essentially an optimization hint, it doesn't affect the architecture. An architectural problem that affects both nbdkit & nbdcopy is that NBD commands drive the nbdkit backend and the nbdcopy loop. If we make the nbdcopy --request-size larger, NBD commands ask for more data, nbdkit-vddk-plugin makes larger VixDiskLib_ReadAsynch requests, which at some point breaks the VMware server. (This is fairly easy to solve in nbdkit-vddk-plugin or with a filter.) But nbdcopy needs to be reworked to make the input and output requests separate, so that nbdcopy will coalesce and split blocks as it copies. This is difficult. Another problem I'm finding (eg https://bugzilla.redhat.com/show_bug.cgi?id=2039255#c9) is that performance of new virt-v2v is extremely specific to input and output mode, and hardware and network configurations. For reasons that I don't fully understand. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload
On Mon, Feb 14, 2022 at 01:11:52PM +0200, Nir Soffer wrote: > On Mon, Feb 14, 2022 at 11:56 AM Richard W.M. Jones wrote: > > > > This change slowed things down (slightly) for me, although the change > > is within the margin of error so it probably made no difference. > > > > Before: > ... > > [ 79.3] Copying disk 1/1 > > █ 100% [] > > [ 89.9] Creating output metadata > > 10.6 seconds... > > > After: > ... > > [ 81.6] Copying disk 1/1 > > █ 100% [] > > [ 91.3] Creating output metadata > > 9.7 seconds - 9% speedup. > > We cannot compare the total time since creating a disk can be take 4-16 > seconds. > > What kind of storage is this? Is this the local storage hack used as NFS? > You may get much better performance with local disk, hiding the delays in > writing to shared storage. But most oVirt users use NFS or GlsuterFS. Yes, it's the Posix local storage attached to a single oVirt node (the only node). > I tested on NFS, tuned to simulate a fast NFS server. > > Testing such changes should be done on a real server with real storage. > I'll try to get a server in our scale lab to do more real testing. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload
On 02/14/22 10:56, Richard W.M. Jones wrote: > This change slowed things down (slightly) for me, although the change > is within the margin of error so it probably made no difference. > > Before: > > $ time ./run virt-v2v -i disk /var/tmp/fedora-35.qcow2 -o rhv-upload -oc > https://ovirt4410/ovirt-engine/api -op /tmp/ovirt-passwd -oo rhv-direct -os > ovirt-data -on test14 -of raw > [ 0.0] Setting up the source: -i disk /var/tmp/fedora-35.qcow2 > [ 1.0] Opening the source > [ 6.5] Inspecting the source > [ 10.5] Checking for sufficient free disk space in the guest > [ 10.5] Converting Fedora Linux 35 (Thirty Five) to run on KVM > virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown > device "vda". You may have to fix this entry manually after conversion. > virt-v2v: This guest has virtio drivers installed. > [ 57.0] Mapping filesystem data to avoid copying unused and blank areas > [ 59.0] Closing the overlay > [ 59.6] Assigning disks to buses > [ 59.6] Checking if the guest needs BIOS or UEFI to boot > [ 59.6] Setting up the destination: -o rhv-upload -oc > https://ovirt4410/ovirt-engine/api -os ovirt-data > [ 79.3] Copying disk 1/1 > █ 100% [] > [ 89.9] Creating output metadata > [ 94.0] Finishing off > > real 1m34.213s > user 0m6.585s > sys0m11.880s > > > After: > > $ time ./run virt-v2v -i disk /var/tmp/fedora-35.qcow2 -o rhv-upload -oc > https://ovirt4410/ovirt-engine/api -op /tmp/ovirt-passwd -oo rhv-direct -os > ovirt-data -on test15 -of raw > [ 0.0] Setting up the source: -i disk /var/tmp/fedora-35.qcow2 > [ 1.0] Opening the source > [ 7.4] Inspecting the source > [ 11.7] Checking for sufficient free disk space in the guest > [ 11.7] Converting Fedora Linux 35 (Thirty Five) to run on KVM > virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown > device "vda". You may have to fix this entry manually after conversion. > virt-v2v: This guest has virtio drivers installed. > [ 59.6] Mapping filesystem data to avoid copying unused and blank areas > [ 61.5] Closing the overlay > [ 62.2] Assigning disks to buses > [ 62.2] Checking if the guest needs BIOS or UEFI to boot > [ 62.2] Setting up the destination: -o rhv-upload -oc > https://ovirt4410/ovirt-engine/api -os ovirt-data > [ 81.6] Copying disk 1/1 > █ 100% [] > [ 91.3] Creating output metadata > [ 96.0] Finishing off > > real 1m36.275s > user 0m4.700s > sys0m14.070s My ACK on Nir's v2 patch basically means that I defer to you on its review -- I don't have anything against it, but I understand it's (perhaps a temporary) workaround until we find a more sustainable (and likely much more complex) solution. Thanks Laszlo ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload
On 02/13/22 20:56, Nir Soffer wrote:
> Output modules can specify now request_size to override the default
> request size in nbdcopy.
>
> The rhv-upload plugin is translating every NBD command to HTTP request,
> translated back to NBD command on imageio server. The HTTP client and
> server, and the NBD client on the imageio server side are synchronous
> and implemented in python, so they have high overhead per request. To
> get good performance we need to use larger request size.
>
> Testing shows that request size of 4 MiB speeds up the copy disk phase
> from 14.7 seconds to 7.9 seconds (1.8x times faster). Request size of 8
> MiB is a little bit faster but is not compatible with VDDK input.
>
> Here are stats extracted from imageio log when importing Fedora 35 image
> with 3 GiB of random data. For each copy, we have 4 connection stats.
>
> Before:
>
> connection 1 ops, 14.767843 s
> dispatch 4023 ops, 11.427662 s
> zero 38 ops, 0.053840 s, 327.91 MiB, 5.95 GiB/s
> write 3981 ops, 8.975877 s, 988.61 MiB, 110.14 MiB/s
> flush 4 ops, 0.001023 s
>
> connection 1 ops, 14.770026 s
> dispatch 4006 ops, 11.408732 s
> zero 37 ops, 0.057205 s, 633.21 MiB, 10.81 GiB/s
> write 3965 ops, 8.907420 s, 986.65 MiB, 110.77 MiB/s
> flush 4 ops, 0.000280 s
>
> connection 1 ops, 14.768180 s
> dispatch 4057 ops, 11.430712 s
> zero 42 ops, 0.030011 s, 470.47 MiB, 15.31 GiB/s
> write 4011 ops, 9.002055 s, 996.98 MiB, 110.75 MiB/s
> flush 4 ops, 0.000261 s
>
> connection 1 ops, 14.770744 s
> dispatch 4037 ops, 11.462050 s
> zero 45 ops, 0.026668 s, 750.82 MiB, 27.49 GiB/s
> write 3988 ops, 9.002721 s, 989.36 MiB, 109.90 MiB/s
> flush 4 ops, 0.000282 s
>
> After:
>
> connection 1 ops, 7.940377 s
> dispatch 323 ops, 6.695582 s
> zero 37 ops, 0.079958 s, 641.12 MiB, 7.83 GiB/s
> write 282 ops, 6.300242 s, 1.01 GiB, 164.54 MiB/s
> flush 4 ops, 0.000537 s
>
> connection 1 ops, 7.908156 s
> dispatch 305 ops, 6.643475 s
> zero 36 ops, 0.144166 s, 509.43 MiB, 3.45 GiB/s
> write 265 ops, 6.179187 s, 941.23 MiB, 152.32 MiB/s
> flush 4 ops, 0.000324 s
>
> connection 1 ops, 7.942349 s
> dispatch 325 ops, 6.744800 s
> zero 45 ops, 0.185335 s, 622.19 MiB, 3.28 GiB/s
> write 276 ops, 6.236819 s, 995.45 MiB, 159.61 MiB/s
> flush 4 ops, 0.000369 s
>
> connection 1 ops, 7.955572 s
> dispatch 317 ops, 6.721212 s
> zero 43 ops, 0.135771 s, 409.68 MiB, 2.95 GiB/s
> write 270 ops, 6.326366 s, 988.26 MiB, 156.21 MiB/s
> flush 4 ops, 0.001439 s
> ---
>
> Changes since v1:
>
> - Decrease request size to 4 MiB for compatibility with VDDK input.
> (Richard)
> - Reimplement in a nicer way based on
>
> https://github.com/libguestfs/virt-v2v/commit/08e764959ec9dadd71a95d22d3d88d647a18d165
> (Richard)
>
> v1 was here:
> https://listman.redhat.com/archives/libguestfs/2022-February/msg00183.html
>
> output/output.ml| 1 +
> output/output.mli | 2 ++
> output/output_disk.ml | 2 ++
> output/output_glance.ml | 2 ++
> output/output_json.ml | 2 ++
> output/output_libvirt.ml| 2 ++
> output/output_null.ml | 2 ++
> output/output_openstack.ml | 2 +-
> output/output_qemu.ml | 2 ++
> output/output_rhv.ml| 2 ++
> output/output_rhv_upload.ml | 7 +++
> output/output_vdsm.ml | 2 ++
> v2v/v2v.ml | 11 ---
> 13 files changed, 35 insertions(+), 4 deletions(-)
>
> diff --git a/output/output.ml b/output/output.ml
> index 786ee5d5..7256b547 100644
> --- a/output/output.ml
> +++ b/output/output.ml
> @@ -39,20 +39,21 @@ type options = {
> module type OUTPUT = sig
>type poptions
>type t
>val to_string : options -> string
>val query_output_options : unit -> unit
>val parse_options : options -> Types.source -> poptions
>val setup : string -> poptions -> Types.source -> t
>val finalize : string -> poptions -> t ->
> Types.source -> Types.inspect -> Types.target_meta ->
> unit
> + val request_size : int option
> end
>
> let error_option_cannot_be_used_in_output_mode mode opt =
>error (f_"-o %s: %s option cannot be used in this output mode") mode opt
>
> let get_disks dir =
>let rec loop acc i =
> let socket = sprintf "%s/in%d" dir i in
> if Sys.file_exists socket then (
>let size = Utils.with_nbd_connect_unix ~socket NBD.get_size in
> diff --git a/output/output.mli b/output/output.mli
> index eed204ed..8e3efd8e 100644
> --- a/output/output.mli
> +++ b/output/output.mli
> @@ -52,20 +52,22 @@ module type OUTPUT = sig
>
>Set up the output mode. Sets up a disk pipeline
>[dir // "outX"] for each output disk. *)
>
>val finalize : string -> poptions -> t ->
> Types.source -> Types.inspect -> Types.target_meta ->
> unit
>(** [finalize dir poptions t inspect target_meta]
>
>Finalizes the conversion and writes metadata. *)
> +
> + val request_size : int option
> end
>
> (** Help
Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload
On Mon, Feb 14, 2022 at 11:56 AM Richard W.M. Jones wrote: > > This change slowed things down (slightly) for me, although the change > is within the margin of error so it probably made no difference. > > Before: ... > [ 79.3] Copying disk 1/1 > █ 100% [] > [ 89.9] Creating output metadata 10.6 seconds... > After: ... > [ 81.6] Copying disk 1/1 > █ 100% [] > [ 91.3] Creating output metadata 9.7 seconds - 9% speedup. We cannot compare the total time since creating a disk can be take 4-16 seconds. What kind of storage is this? Is this the local storage hack used as NFS? You may get much better performance with local disk, hiding the delays in writing to shared storage. But most oVirt users use NFS or GlsuterFS. I tested on NFS, tuned to simulate a fast NFS server. Testing such changes should be done on a real server with real storage. I'll try to get a server in our scale lab to do more real testing. Nir ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload
This change slowed things down (slightly) for me, although the change is within the margin of error so it probably made no difference. Before: $ time ./run virt-v2v -i disk /var/tmp/fedora-35.qcow2 -o rhv-upload -oc https://ovirt4410/ovirt-engine/api -op /tmp/ovirt-passwd -oo rhv-direct -os ovirt-data -on test14 -of raw [ 0.0] Setting up the source: -i disk /var/tmp/fedora-35.qcow2 [ 1.0] Opening the source [ 6.5] Inspecting the source [ 10.5] Checking for sufficient free disk space in the guest [ 10.5] Converting Fedora Linux 35 (Thirty Five) to run on KVM virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown device "vda". You may have to fix this entry manually after conversion. virt-v2v: This guest has virtio drivers installed. [ 57.0] Mapping filesystem data to avoid copying unused and blank areas [ 59.0] Closing the overlay [ 59.6] Assigning disks to buses [ 59.6] Checking if the guest needs BIOS or UEFI to boot [ 59.6] Setting up the destination: -o rhv-upload -oc https://ovirt4410/ovirt-engine/api -os ovirt-data [ 79.3] Copying disk 1/1 █ 100% [] [ 89.9] Creating output metadata [ 94.0] Finishing off real 1m34.213s user 0m6.585s sys 0m11.880s After: $ time ./run virt-v2v -i disk /var/tmp/fedora-35.qcow2 -o rhv-upload -oc https://ovirt4410/ovirt-engine/api -op /tmp/ovirt-passwd -oo rhv-direct -os ovirt-data -on test15 -of raw [ 0.0] Setting up the source: -i disk /var/tmp/fedora-35.qcow2 [ 1.0] Opening the source [ 7.4] Inspecting the source [ 11.7] Checking for sufficient free disk space in the guest [ 11.7] Converting Fedora Linux 35 (Thirty Five) to run on KVM virt-v2v: warning: /files/boot/grub2/device.map/hd0 references unknown device "vda". You may have to fix this entry manually after conversion. virt-v2v: This guest has virtio drivers installed. [ 59.6] Mapping filesystem data to avoid copying unused and blank areas [ 61.5] Closing the overlay [ 62.2] Assigning disks to buses [ 62.2] Checking if the guest needs BIOS or UEFI to boot [ 62.2] Setting up the destination: -o rhv-upload -oc https://ovirt4410/ovirt-engine/api -os ovirt-data [ 81.6] Copying disk 1/1 █ 100% [] [ 91.3] Creating output metadata [ 96.0] Finishing off real 1m36.275s user 0m4.700s sys 0m14.070s Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://people.redhat.com/~rjones/virt-df/ ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
[Libguestfs] [PATCH v2] v2v/v2v.ml: Use larger request size for -o rhv-upload
Output modules can specify now request_size to override the default
request size in nbdcopy.
The rhv-upload plugin is translating every NBD command to HTTP request,
translated back to NBD command on imageio server. The HTTP client and
server, and the NBD client on the imageio server side are synchronous
and implemented in python, so they have high overhead per request. To
get good performance we need to use larger request size.
Testing shows that request size of 4 MiB speeds up the copy disk phase
from 14.7 seconds to 7.9 seconds (1.8x times faster). Request size of 8
MiB is a little bit faster but is not compatible with VDDK input.
Here are stats extracted from imageio log when importing Fedora 35 image
with 3 GiB of random data. For each copy, we have 4 connection stats.
Before:
connection 1 ops, 14.767843 s
dispatch 4023 ops, 11.427662 s
zero 38 ops, 0.053840 s, 327.91 MiB, 5.95 GiB/s
write 3981 ops, 8.975877 s, 988.61 MiB, 110.14 MiB/s
flush 4 ops, 0.001023 s
connection 1 ops, 14.770026 s
dispatch 4006 ops, 11.408732 s
zero 37 ops, 0.057205 s, 633.21 MiB, 10.81 GiB/s
write 3965 ops, 8.907420 s, 986.65 MiB, 110.77 MiB/s
flush 4 ops, 0.000280 s
connection 1 ops, 14.768180 s
dispatch 4057 ops, 11.430712 s
zero 42 ops, 0.030011 s, 470.47 MiB, 15.31 GiB/s
write 4011 ops, 9.002055 s, 996.98 MiB, 110.75 MiB/s
flush 4 ops, 0.000261 s
connection 1 ops, 14.770744 s
dispatch 4037 ops, 11.462050 s
zero 45 ops, 0.026668 s, 750.82 MiB, 27.49 GiB/s
write 3988 ops, 9.002721 s, 989.36 MiB, 109.90 MiB/s
flush 4 ops, 0.000282 s
After:
connection 1 ops, 7.940377 s
dispatch 323 ops, 6.695582 s
zero 37 ops, 0.079958 s, 641.12 MiB, 7.83 GiB/s
write 282 ops, 6.300242 s, 1.01 GiB, 164.54 MiB/s
flush 4 ops, 0.000537 s
connection 1 ops, 7.908156 s
dispatch 305 ops, 6.643475 s
zero 36 ops, 0.144166 s, 509.43 MiB, 3.45 GiB/s
write 265 ops, 6.179187 s, 941.23 MiB, 152.32 MiB/s
flush 4 ops, 0.000324 s
connection 1 ops, 7.942349 s
dispatch 325 ops, 6.744800 s
zero 45 ops, 0.185335 s, 622.19 MiB, 3.28 GiB/s
write 276 ops, 6.236819 s, 995.45 MiB, 159.61 MiB/s
flush 4 ops, 0.000369 s
connection 1 ops, 7.955572 s
dispatch 317 ops, 6.721212 s
zero 43 ops, 0.135771 s, 409.68 MiB, 2.95 GiB/s
write 270 ops, 6.326366 s, 988.26 MiB, 156.21 MiB/s
flush 4 ops, 0.001439 s
---
Changes since v1:
- Decrease request size to 4 MiB for compatibility with VDDK input.
(Richard)
- Reimplement in a nicer way based on
https://github.com/libguestfs/virt-v2v/commit/08e764959ec9dadd71a95d22d3d88d647a18d165
(Richard)
v1 was here:
https://listman.redhat.com/archives/libguestfs/2022-February/msg00183.html
output/output.ml| 1 +
output/output.mli | 2 ++
output/output_disk.ml | 2 ++
output/output_glance.ml | 2 ++
output/output_json.ml | 2 ++
output/output_libvirt.ml| 2 ++
output/output_null.ml | 2 ++
output/output_openstack.ml | 2 +-
output/output_qemu.ml | 2 ++
output/output_rhv.ml| 2 ++
output/output_rhv_upload.ml | 7 +++
output/output_vdsm.ml | 2 ++
v2v/v2v.ml | 11 ---
13 files changed, 35 insertions(+), 4 deletions(-)
diff --git a/output/output.ml b/output/output.ml
index 786ee5d5..7256b547 100644
--- a/output/output.ml
+++ b/output/output.ml
@@ -39,20 +39,21 @@ type options = {
module type OUTPUT = sig
type poptions
type t
val to_string : options -> string
val query_output_options : unit -> unit
val parse_options : options -> Types.source -> poptions
val setup : string -> poptions -> Types.source -> t
val finalize : string -> poptions -> t ->
Types.source -> Types.inspect -> Types.target_meta ->
unit
+ val request_size : int option
end
let error_option_cannot_be_used_in_output_mode mode opt =
error (f_"-o %s: %s option cannot be used in this output mode") mode opt
let get_disks dir =
let rec loop acc i =
let socket = sprintf "%s/in%d" dir i in
if Sys.file_exists socket then (
let size = Utils.with_nbd_connect_unix ~socket NBD.get_size in
diff --git a/output/output.mli b/output/output.mli
index eed204ed..8e3efd8e 100644
--- a/output/output.mli
+++ b/output/output.mli
@@ -52,20 +52,22 @@ module type OUTPUT = sig
Set up the output mode. Sets up a disk pipeline
[dir // "outX"] for each output disk. *)
val finalize : string -> poptions -> t ->
Types.source -> Types.inspect -> Types.target_meta ->
unit
(** [finalize dir poptions t inspect target_meta]
Finalizes the conversion and writes metadata. *)
+
+ val request_size : int option
end
(** Helper functions for output modes. *)
val error_option_cannot_be_used_in_output_mode : string -> string -> unit
(** [error_option_cannot_be_used_in_output_mode mode option]
prints error message that option cannot be used in this output mode. *)
val get_disks : string -> (int * int64) list
(** Examines the v2v d
