Re: [Libguestfs] [nbdkit PATCH] RFC: noextents: Add extentmode=data config parameter

2022-02-08 Thread Laszlo Ersek
On 02/08/22 21:29, Eric Blake wrote: > On Tue, Feb 08, 2022 at 12:39:02PM +0100, Laszlo Ersek wrote: >> On 02/04/22 17:44, Eric Blake wrote: >>> While writing tests for an nbdcopy bug, I found myself wanting a way >>> to easily view an entire image as data, but without disabling extents >>> support

Re: [Libguestfs] [libnbd PATCH] copy: Fail nbdcopy if NBD read or write fails

2022-02-08 Thread Laszlo Ersek
On 02/08/22 14:43, Richard W.M. Jones wrote: > On Fri, Feb 04, 2022 at 09:18:07AM +0100, Laszlo Ersek wrote: >> (And there was no easy way to skip connect-tcp6.) > > Wwe use this in libnbd.spec: > > https://src.fedoraproject.org/rpms/libnbd/blob/rawhide/f/libnbd.spec#_214 Yes (I know what the li

Re: [Libguestfs] [nbdkit PATCH] RFC: noextents: Add extentmode=data config parameter

2022-02-08 Thread Laszlo Ersek
On 02/04/22 17:44, Eric Blake wrote: > While writing tests for an nbdcopy bug, I found myself wanting a way > to easily view an entire image as data, but without disabling extents > support altogether. The existing extentlist filter can do this, but > requires a secondary file. > > Still an RFC b

Re: [Libguestfs] [libnbd PATCH v2 1/5] python: tests: Fix error handling

2022-02-08 Thread Laszlo Ersek
On 02/07/22 02:03, Nir Soffer wrote: > On Fri, Feb 4, 2022 at 10:37 AM Laszlo Ersek wrote: >> >> On 02/03/22 23:49, Nir Soffer wrote: >>> On Thu, Feb 3, 2022 at 10:26 PM Eric Blake wrote: Like a lot of the C examples, the aio copy test ignores read and write errors in the completio

Re: [Libguestfs] [nbdkit PATCH] RFC: noextents: Add extentmode=data config parameter

2022-02-08 Thread Richard W.M. Jones
On Fri, Feb 04, 2022 at 10:44:38AM -0600, Eric Blake wrote: > While writing tests for an nbdcopy bug, I found myself wanting a way > to easily view an entire image as data, but without disabling extents > support altogether. The existing extentlist filter can do this, but > requires a secondary fi

Re: [Libguestfs] [PATCH libnbd] golang: make-dist.sh: Generate the list file

2022-02-08 Thread Richard W.M. Jones
On Mon, Feb 07, 2022 at 01:28:30AM +0200, Nir Soffer wrote: > On Sun, Feb 6, 2022 at 9:13 PM Richard W.M. Jones wrote: > > > > On Sun, Feb 06, 2022 at 07:21:02PM +0200, Nir Soffer wrote: > > > Generated the list file when creating the distribution. Since the Go > > > tool treat the list file on th

[Libguestfs] LIBNBD SECURITY: Silent image corruption with nbdcopy

2022-02-08 Thread Eric Blake
We have discovered a security flaw with potential moderate impact in libnbd. Lifecycle - Reported: 2022-01-24 Fixed: 2022-02-04 Published: 2022-01-27 This has been assigned CVE-2022-0485. Credit -- Reported by Nir Soffer and patched by Eric Blake Description --- libnb

Re: [Libguestfs] [PATCH libnbd] golang: make-dist.sh: Generate the list file

2022-02-08 Thread Eric Blake
On Sun, Feb 06, 2022 at 07:21:02PM +0200, Nir Soffer wrote: > Generated the list file when creating the distribution. Since the Go > tool treat the list file on the proxy server as the source of truth, we > do the same. The new list file is created by downloading the current > list file, sorting it

Re: [Libguestfs] [PATCH libnbd] golang: make-dist.sh: Generate the list file

2022-02-08 Thread Nir Soffer
On Mon, Feb 7, 2022 at 3:13 PM Eric Blake wrote: > > On Sun, Feb 06, 2022 at 07:21:02PM +0200, Nir Soffer wrote: > > Generated the list file when creating the distribution. Since the Go > > tool treat the list file on the proxy server as the source of truth, we > > do the same. The new list file i

Re: [Libguestfs] [PATCH libnbd 7/9] golang: aio_buffer.go: Speed up FromBytes()

2022-02-08 Thread Eric Blake
On Sun, Jan 30, 2022 at 01:33:35AM +0200, Nir Soffer wrote: > Using Slice() we can use builtin copy() instead of a manual loop, which > is 4.6 times faster with a typical 256k buffer: Again, the benefits of the underlying libraries being able to vectorize over larger chunks of memory ;) > > Befo

Re: [Libguestfs] [PATCH libnbd 6/9] golang: tests: Use AioBuffer.Slice()

2022-02-08 Thread Eric Blake
On Sun, Jan 30, 2022 at 01:33:34AM +0200, Nir Soffer wrote: > Slice() is easier to use and faster than Get() or Bytes(). Lets use the Let's > new way. > > Signed-off-by: Nir Soffer > --- > golang/libnbd_500_aio_pread_test.go | 2 +- > golang/libnbd_510_aio_pwrite_test.go | 8 +--- > golan

Re: [Libguestfs] [PATCH libnbd 4/9] golang: aio_buffer.go: Add MakeAioBufferZero()

2022-02-08 Thread Eric Blake
On Sun, Jan 30, 2022 at 01:33:32AM +0200, Nir Soffer wrote: > Make it easy to create a zeroed buffer via calloc(), preventing leaking > sensitive info from the heap. > > Benchmarking show that creating a zeroed buffer is much slower compared shows > with uninitialized buffer, but much faster com

Re: [Libguestfs] [PATCH libnbd 2/9] golang: aio_buffer.go: Make it safer to use

2022-02-08 Thread Eric Blake
On Sun, Jan 30, 2022 at 01:33:30AM +0200, Nir Soffer wrote: > If a Go program tries to use AioBuffer after calling AioBuffer.Free(), > the program may silently corrupt data, accessing memory that does not > belong to the buffer any more, or segfault if the address is not mapped. > In the worst case

Re: [Libguestfs] [nbdkit PATCH] RFC: noextents: Add extentmode=data config parameter

2022-02-08 Thread Eric Blake
On Tue, Feb 08, 2022 at 12:39:02PM +0100, Laszlo Ersek wrote: > On 02/04/22 17:44, Eric Blake wrote: > > While writing tests for an nbdcopy bug, I found myself wanting a way > > to easily view an entire image as data, but without disabling extents > > support altogether. The existing extentlist fi

Re: [Libguestfs] [PATCH libnbd 1/9] golang: tests: Add test for AioBuffer

2022-02-08 Thread Eric Blake
On Sun, Jan 30, 2022 at 01:33:29AM +0200, Nir Soffer wrote: > Add unit tests and benchmarks for AioBuffer. The tests are trivial but > they server as running documentation, and they point out important > details about the type. > > The benchmarks how efficient is allocation a new buffer, zeroing i

Re: [Libguestfs] [PATCH libnbd] generator/Go.ml: Simplify copy_uint32_array

2022-02-08 Thread Eric Blake
On Sun, Feb 06, 2022 at 07:45:20PM +0200, Nir Soffer wrote: > Create a slice backed up by the entries pointer, and copy the data with > builtin copy(). This can be 3x times faster but I did not measure it. > > Eric posted a similar patch[1] last year, but the patch seems to be > stuck with unrelat

Re: [Libguestfs] [PATCH 1/5] output/rhv-upload-plugin: Fix flush and close

2022-02-08 Thread Richard W.M. Jones
On Sat, Dec 18, 2021 at 10:36:29PM +0200, Nir Soffer wrote: > When locking the http pool, we wait until all connections are idle, and > take them from the pool. But since we used pool.qsize(), which is the > number of items currently in the queue, we did not wait for all > connections. > > This le

Re: [Libguestfs] [PATCH 5/5] output/rhv-upload-plugin: Keep connections alive

2022-02-08 Thread Richard W.M. Jones
On Sat, Dec 18, 2021 at 10:36:33PM +0200, Nir Soffer wrote: > When importing from vddk, nbdcopy may be blocked for few minutes(!) > trying to get extents. While nbdcopy is blocked, imageio server closes > the idle connections. When we finally get a request from nbdcopy, we > fail to detect that the

Re: [Libguestfs] [PATCH 3/5] output/rhv-upload-plugin: Extract send_flush() helper

2022-02-08 Thread Richard W.M. Jones
On Sat, Dec 18, 2021 at 10:36:31PM +0200, Nir Soffer wrote: > Extract a helper for sending flush request for single connection, and > inline the iter_http_pool() helper into flush(), its only user. > --- > output/rhv-upload-plugin.py | 54 - > 1 file changed, 23

Re: [Libguestfs] Performance regression in -o rhv-upload questions

2022-02-08 Thread Richard W.M. Jones
On Tue, Feb 08, 2022 at 04:44:33PM +0200, Nir Soffer wrote: > This happens because virt-v2v try to finalize the transfer *before* closing > the > connections to imageio server. > > Current imageio release mark the a ticket as canceled, but will not remove it > if the ticket has open connections f

[Libguestfs] [PATCH v2v] output: -o rhv-upload: Kill nbdkit instances before finalizing

2022-02-08 Thread Richard W.M. Jones
If nbdkit instance(s) are still running then they will hold open some http connections to imageio. In some versions of imageio, starting finalization in this state causes a 60 second timeout. See-also: https://listman.redhat.com/archives/libguestfs/2022-February/msg00111.html Thanks: Nir Soffer

Re: [Libguestfs] Performance regression in -o rhv-upload questions

2022-02-08 Thread Richard W.M. Jones
On Tue, Feb 08, 2022 at 04:44:33PM +0200, Nir Soffer wrote: > On Tue, Feb 8, 2022 at 3:27 PM Richard W.M. Jones wrote: > > virt-v2v 1.44.2: > > Complete log: http://oirase.annexia.org/tmp/virt-v2v-1.44.2-rhv-upload.log > > > > [ 63.1] Copying disk 1/1 > > ... > > transfer 7aecd359-3706-49f5

Re: [Libguestfs] Performance regression in -o rhv-upload questions

2022-02-08 Thread Nir Soffer
On Tue, Feb 8, 2022 at 3:27 PM Richard W.M. Jones wrote: > > Hi Nir, > > https://bugzilla.redhat.com/show_bug.cgi?id=2039255#c4 > > I'm looking for some advice/help with a performance regression in > virt-v2v between 1.44.2 and the latest version. It's very obvious and > reproducible when I do a

[Libguestfs] Performance regression in -o rhv-upload questions

2022-02-08 Thread Richard W.M. Jones
Hi Nir, https://bugzilla.redhat.com/show_bug.cgi?id=2039255#c4 I'm looking for some advice/help with a performance regression in virt-v2v between 1.44.2 and the latest version. It's very obvious and reproducible when I do a conversion from a local disk to local RHV using -o rhv-upload, specifica

Re: [Libguestfs] [libnbd PATCH] copy: Fail nbdcopy if NBD read or write fails

2022-02-08 Thread Richard W.M. Jones
On Fri, Feb 04, 2022 at 09:18:07AM +0100, Laszlo Ersek wrote: > (And there was no easy way to skip connect-tcp6.) Wwe use this in libnbd.spec: https://src.fedoraproject.org/rpms/libnbd/blob/rawhide/f/libnbd.spec#_214 In libguestfs we have a whole infrastructure for skipping tests by setting $SKI