Re: [Libguestfs] Checksums and other verification
On Tue, Feb 28, 2023 at 4:13 PM Laszlo Ersek wrote: > > On 2/28/23 12:39, Richard W.M. Jones wrote: > > On Tue, Feb 28, 2023 at 12:24:04PM +0100, Laszlo Ersek wrote: > >> On 2/27/23 17:44, Richard W.M. Jones wrote: > >>> On Mon, Feb 27, 2023 at 08:42:23AM -0600, Eric Blake wrote: > Or intentionally choose a hash that can be computed out-of-order, > such as a Merkle Tree. But we'd need a standard setup for all > parties to agree on how the hash is to be computed and checked, if > it is going to be anything more than just a linear hash of the > entire guest-visible contents. > >>> > >>> Unfortunately I suspect that by far the easiest way for people who > >>> host images to compute checksums is to run 'shaXXXsum' on them or > >>> sign them with a GPG signature, rather than engaging in a novel hash > >>> function. Indeed that's what is happening now: > >>> > >>> https://alt.fedoraproject.org/en/verify.html > >> > >> If the output is produced with unordered writes, but the complete > >> output needs to be verified with a hash *chain*, that still allows > >> for some level of asynchrony. The start of the hashing need not be > >> delayed until after the end of output, only after the start of > >> output. > >> > >> For example, nbdcopy could maintain the highest offset up to which > >> the output is contiguous, and on a separate thread, it could be > >> hashing the output up to that offset. > >> > >> Considering a gigantic output, as yet unassembled blocks could likely > >> not be buffered in memory (that's why the writes are unordered in the > >> first place!), so the hashing thread would have to re-read the output > >> via NBD. Whether that would cause performance to improve or to > >> deteriorate is undecided IMO. If the far end of the output network > >> block device can accommodate a reader that is independent of the > >> writers, then this level of overlap is beneficial. Otherwise, this > >> extra reader thread would just add more thrashing, and we'd be better > >> off with a separate read-through once writing is complete. > > > > In my mind I'm wondering if there's any mathematical result that lets > > you combine each hash(block_i) into the final hash(block[1..N]) > > without needing to compute the hash of each block in order. > > I've now checked: > > https://en.wikipedia.org/wiki/SHA-2 > https://en.wikipedia.org/wiki/Merkle%E2%80%93Damg%C3%A5rd_construction > https://en.wikipedia.org/wiki/One-way_compression_function#Construction_from_block_ciphers > https://en.wikipedia.org/wiki/One-way_compression_function#Davies%E2%80%93Meyer > > Consider the following order of steps: > > - precompute hash(block[n]), with some initial IV > - throw away block[n] > - wait until block[n-1] is processed, providing the actual IV for > hashing block[n] > - mix the new IV into hash(block[n]) without having access to block[n] > > If such a method existed, it would break the security (i.e., the > original design) of the hash, IMO, as it would separate the IV from > block[n]. In a way, it would make the "mix" and "concat" operators (of > the underlying block cipher's chaining method) distributive. I believe > then you could generate a bunch of *valid* hash(block[n]) values as a > mere function of the IV, without having access to block[n]. You could > perhaps use that for probing against other hash(block[m]) values, and > maybe determine repeating patterns in the plaintext. I'm not a > cryptographer so I can't exactly show what security property is broken > by separating the IV from block[n]. > > > (This is what blkhash solves, but unfortunately the output isn't > > compatible with standard hashes.) > > Assuming blkhash is a Merkle Tree implementation, blkhash solves a > different problem IMO. blkhash uses a flat Merkle tree, described here: https://www.researchgate.net/publication/323243320_Foundations_of_Applied_Cryptography_and_Cybersecurity seection 3.9.4 2lMT: the Flat Merkle Tree construction To support parallel hashing it uses more complex construction, but this can be simplified to a single flat Merkle tree. Nir ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [V2V PATCH 0/5] Bring support for virtio-scsi back to Windows
On Tue, Feb 28, 2023 at 03:24:46PM +0100, Laszlo Ersek wrote:
> On 2/28/23 14:01, Richard W.M. Jones wrote:
> > On Wed, Feb 22, 2023 at 08:20:43PM +0200, Andrey Drobyshev wrote:
> >> Since commits b28cd1dc ("Remove requested_guestcaps / rcaps"), f0afc439
> >> ("Remove guestcaps_block_type Virtio_SCSI") support for installing
> >> virtio-scsi driver is missing in virt-v2v. AFAIU plans and demands for
> >> bringing this feature back have been out there for a while. E.g. I've
> >> found a corresponding issue which is still open [1].
> >>
> >> The code in b28cd1dc, f0afc439 was removed due to removing the old in-place
> >> support. However, having the new in-place support present and bringing
> >> this same code (partially) back with several additions and improvements,
> >> I'm able to successfully convert and boot a Win guest with a virtio-scsi
> >> disk controller. So please consider the following implementation of
> >> this feature.
> >>
> >> [1] https://github.com/libguestfs/virt-v2v/issues/12
> >
> > So you're right that we ended up removing virtio-scsi support because
> > in-place conversion was temporarily removed, and it wasn't restored
> > when the new virt-v2v-in-place tool was added.
> >
> > I looked at the patches, and I don't have any objection to 1-4.
> >
> > Patch 5 is the tricky one because it changes the default, but we
> > (Red Hat) are likely to stick with virtio-blk for better or worse.
> >
> > Could we make this configurable? We've traditionally shied away from
> > adding virt-v2v command line options which are purely for fine-tuning
> > conversions. The reason for this is that when you're doing bulk
> > conversions of hundreds of VMs from VMware, you want virt-v2v to do
> > "the best it can", and it's not really feasible to hand configure
> > every conversion. (There are some historical exceptions to this, like
> > --root, but don't look at those!) I know the bulk conversion case
> > doesn't apply to Virtuozzo, but it does matter for us.
> >
> > It could be argued that this isn't the same, because if we're bulk
> > converting, you're not fine-tuning virtio-scsi vs virtio-blk per VM,
> > but per target hypervisor. So a command line option would be OK in
> > this case.
> >
> > BTW previously we "configured" this preference through the input
> > libvirt XML. That was a terrible idea, so I'd prefer to avoid it this
> > time around.
>
> Ah, that's new information to me. I thought the "rcaps" (requested caps,
> derived from the input XML) was exactly what was missing here. Not
> because of the particular reason(s) that may have motivated rcaps in the
> past, but because in the use case at hand, the tweaking of the domain
> XML precedes the conversion (more like, the injection of the desired
> virtio drivers).
>
> Why was it a terrible idea?
It was a hack. For copying (normal) conversions -i libvirtxml
describes the input metadata. For in-place conversions "input" is a
fuzzy concept, so it was overloaded to describe the conversion
preferences. We can do better with a command line parameter.
> > Anyway if you can think of a better version of patch 5 then I think we
> > can accept this.
>
> ITYM "if you *cannot* think of a better version"...
>
> But anyway, if a command line option is acceptable to you, I'm perfectly
> fine with it as well. I'd like to avoid changing the default. I quite
> foresee tens of QE test runs breaking, resulting in as many bug reports,
> and then us eventually reverting the change-of-the-default downstream.
> I'd rather favor an experimental command line switch.
Yes I'm sure we don't want to change the default; or at least we
(meaning Red Hat, downstream) want to choose whether and when to
change the default.
BTW/FYI we don't ship virt-v2v-in-place in RHEL (we do in Fedora).
Thanks,
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [V2V PATCH 0/5] Bring support for virtio-scsi back to Windows
On 2/28/23 14:01, Richard W.M. Jones wrote:
> On Wed, Feb 22, 2023 at 08:20:43PM +0200, Andrey Drobyshev wrote:
>> Since commits b28cd1dc ("Remove requested_guestcaps / rcaps"), f0afc439
>> ("Remove guestcaps_block_type Virtio_SCSI") support for installing
>> virtio-scsi driver is missing in virt-v2v. AFAIU plans and demands for
>> bringing this feature back have been out there for a while. E.g. I've
>> found a corresponding issue which is still open [1].
>>
>> The code in b28cd1dc, f0afc439 was removed due to removing the old in-place
>> support. However, having the new in-place support present and bringing
>> this same code (partially) back with several additions and improvements,
>> I'm able to successfully convert and boot a Win guest with a virtio-scsi
>> disk controller. So please consider the following implementation of
>> this feature.
>>
>> [1] https://github.com/libguestfs/virt-v2v/issues/12
>
> So you're right that we ended up removing virtio-scsi support because
> in-place conversion was temporarily removed, and it wasn't restored
> when the new virt-v2v-in-place tool was added.
>
> I looked at the patches, and I don't have any objection to 1-4.
>
> Patch 5 is the tricky one because it changes the default, but we
> (Red Hat) are likely to stick with virtio-blk for better or worse.
>
> Could we make this configurable? We've traditionally shied away from
> adding virt-v2v command line options which are purely for fine-tuning
> conversions. The reason for this is that when you're doing bulk
> conversions of hundreds of VMs from VMware, you want virt-v2v to do
> "the best it can", and it's not really feasible to hand configure
> every conversion. (There are some historical exceptions to this, like
> --root, but don't look at those!) I know the bulk conversion case
> doesn't apply to Virtuozzo, but it does matter for us.
>
> It could be argued that this isn't the same, because if we're bulk
> converting, you're not fine-tuning virtio-scsi vs virtio-blk per VM,
> but per target hypervisor. So a command line option would be OK in
> this case.
>
> BTW previously we "configured" this preference through the input
> libvirt XML. That was a terrible idea, so I'd prefer to avoid it this
> time around.
Ah, that's new information to me. I thought the "rcaps" (requested caps,
derived from the input XML) was exactly what was missing here. Not
because of the particular reason(s) that may have motivated rcaps in the
past, but because in the use case at hand, the tweaking of the domain
XML precedes the conversion (more like, the injection of the desired
virtio drivers).
Why was it a terrible idea?
> Anyway if you can think of a better version of patch 5 then I think we
> can accept this.
ITYM "if you *cannot* think of a better version"...
But anyway, if a command line option is acceptable to you, I'm perfectly
fine with it as well. I'd like to avoid changing the default. I quite
foresee tens of QE test runs breaking, resulting in as many bug reports,
and then us eventually reverting the change-of-the-default downstream.
I'd rather favor an experimental command line switch.
Laszlo
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] Checksums and other verification
On 2/28/23 12:39, Richard W.M. Jones wrote: > On Tue, Feb 28, 2023 at 12:24:04PM +0100, Laszlo Ersek wrote: >> On 2/27/23 17:44, Richard W.M. Jones wrote: >>> On Mon, Feb 27, 2023 at 08:42:23AM -0600, Eric Blake wrote: Or intentionally choose a hash that can be computed out-of-order, such as a Merkle Tree. But we'd need a standard setup for all parties to agree on how the hash is to be computed and checked, if it is going to be anything more than just a linear hash of the entire guest-visible contents. >>> >>> Unfortunately I suspect that by far the easiest way for people who >>> host images to compute checksums is to run 'shaXXXsum' on them or >>> sign them with a GPG signature, rather than engaging in a novel hash >>> function. Indeed that's what is happening now: >>> >>> https://alt.fedoraproject.org/en/verify.html >> >> If the output is produced with unordered writes, but the complete >> output needs to be verified with a hash *chain*, that still allows >> for some level of asynchrony. The start of the hashing need not be >> delayed until after the end of output, only after the start of >> output. >> >> For example, nbdcopy could maintain the highest offset up to which >> the output is contiguous, and on a separate thread, it could be >> hashing the output up to that offset. >> >> Considering a gigantic output, as yet unassembled blocks could likely >> not be buffered in memory (that's why the writes are unordered in the >> first place!), so the hashing thread would have to re-read the output >> via NBD. Whether that would cause performance to improve or to >> deteriorate is undecided IMO. If the far end of the output network >> block device can accommodate a reader that is independent of the >> writers, then this level of overlap is beneficial. Otherwise, this >> extra reader thread would just add more thrashing, and we'd be better >> off with a separate read-through once writing is complete. > > In my mind I'm wondering if there's any mathematical result that lets > you combine each hash(block_i) into the final hash(block[1..N]) > without needing to compute the hash of each block in order. I've now checked: https://en.wikipedia.org/wiki/SHA-2 https://en.wikipedia.org/wiki/Merkle%E2%80%93Damg%C3%A5rd_construction https://en.wikipedia.org/wiki/One-way_compression_function#Construction_from_block_ciphers https://en.wikipedia.org/wiki/One-way_compression_function#Davies%E2%80%93Meyer Consider the following order of steps: - precompute hash(block[n]), with some initial IV - throw away block[n] - wait until block[n-1] is processed, providing the actual IV for hashing block[n] - mix the new IV into hash(block[n]) without having access to block[n] If such a method existed, it would break the security (i.e., the original design) of the hash, IMO, as it would separate the IV from block[n]. In a way, it would make the "mix" and "concat" operators (of the underlying block cipher's chaining method) distributive. I believe then you could generate a bunch of *valid* hash(block[n]) values as a mere function of the IV, without having access to block[n]. You could perhaps use that for probing against other hash(block[m]) values, and maybe determine repeating patterns in the plaintext. I'm not a cryptographer so I can't exactly show what security property is broken by separating the IV from block[n]. > (This is what blkhash solves, but unfortunately the output isn't > compatible with standard hashes.) Assuming blkhash is a Merkle Tree implementation, blkhash solves a different problem IMO. In your above notation, hash(block[1..N]) is a hash of the concatenated plaintext blocks, and that's not what a Merkle Tree describes. The "mix" and "concat" operators remain non-distributive; it's the operator trees that differ. With a Merkle Tree, there are sub-trees that can be evaluated independently of each other. With SHA256, we have a fully imbalanced operator tree, one where the tree depth is maximal. Laszlo ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [V2V PATCH 0/5] Bring support for virtio-scsi back to Windows
On Wed, Feb 22, 2023 at 08:20:43PM +0200, Andrey Drobyshev wrote:
> Since commits b28cd1dc ("Remove requested_guestcaps / rcaps"), f0afc439
> ("Remove guestcaps_block_type Virtio_SCSI") support for installing
> virtio-scsi driver is missing in virt-v2v. AFAIU plans and demands for
> bringing this feature back have been out there for a while. E.g. I've
> found a corresponding issue which is still open [1].
>
> The code in b28cd1dc, f0afc439 was removed due to removing the old in-place
> support. However, having the new in-place support present and bringing
> this same code (partially) back with several additions and improvements,
> I'm able to successfully convert and boot a Win guest with a virtio-scsi
> disk controller. So please consider the following implementation of
> this feature.
>
> [1] https://github.com/libguestfs/virt-v2v/issues/12
So you're right that we ended up removing virtio-scsi support because
in-place conversion was temporarily removed, and it wasn't restored
when the new virt-v2v-in-place tool was added.
I looked at the patches, and I don't have any objection to 1-4.
Patch 5 is the tricky one because it changes the default, but we
(Red Hat) are likely to stick with virtio-blk for better or worse.
Could we make this configurable? We've traditionally shied away from
adding virt-v2v command line options which are purely for fine-tuning
conversions. The reason for this is that when you're doing bulk
conversions of hundreds of VMs from VMware, you want virt-v2v to do
"the best it can", and it's not really feasible to hand configure
every conversion. (There are some historical exceptions to this, like
--root, but don't look at those!) I know the bulk conversion case
doesn't apply to Virtuozzo, but it does matter for us.
It could be argued that this isn't the same, because if we're bulk
converting, you're not fine-tuning virtio-scsi vs virtio-blk per VM,
but per target hypervisor. So a command line option would be OK in
this case.
BTW previously we "configured" this preference through the input
libvirt XML. That was a terrible idea, so I'd prefer to avoid it this
time around.
Anyway if you can think of a better version of patch 5 then I think we
can accept this.
Thanks,
Rich.
>
> v2v:
>
> Andrey Drobyshev (2):
> Revert "Remove guestcaps_block_type Virtio_SCSI"
> convert_windows: add Inject_virtio_win.Virtio_SCSI as a possible block
> type
>
> convert/convert.ml | 2 +-
> convert/convert_linux.ml | 9 +++--
> convert/convert_windows.ml | 1 +
> convert/target_bus_assignment.ml | 1 +
> lib/create_ovf.ml| 1 +
> lib/types.ml | 3 ++-
> lib/types.mli| 2 +-
> output/openstack_image_properties.ml | 7 +++
> 9 files changed, 22 insertions(+), 6 deletions(-)
>
> common:
>
> Andrey Drobyshev (2):
> inject_virtio_win: add Virtio_SCSI to block_type
> inject_virtio_win: make virtio-scsi the default block driver
>
> Roman Kagan (1):
> inject_virtio_win: match only vendor/device
>
> mlcustomize/inject_virtio_win.ml | 25 -
> mlcustomize/inject_virtio_win.mli | 2 +-
> 2 files changed, 17 insertions(+), 10 deletions(-)
>
> --
> 2.31.1
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH v5 0/5] vector: introduce DEFINE_POINTER_VECTOR_TYPE()
Thanks for this series. I checked it upstream & it looks great. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] Checksums and other verification
On Tue, Feb 28, 2023 at 12:24:04PM +0100, Laszlo Ersek wrote: > On 2/27/23 17:44, Richard W.M. Jones wrote: > > On Mon, Feb 27, 2023 at 08:42:23AM -0600, Eric Blake wrote: > >> Or intentionally choose a hash that can be computed out-of-order, such > >> as a Merkle Tree. But we'd need a standard setup for all parties to > >> agree on how the hash is to be computed and checked, if it is going to > >> be anything more than just a linear hash of the entire guest-visible > >> contents. > > > > Unfortunately I suspect that by far the easiest way for people who > > host images to compute checksums is to run 'shaXXXsum' on them or sign > > them with a GPG signature, rather than engaging in a novel hash > > function. Indeed that's what is happening now: > > > > https://alt.fedoraproject.org/en/verify.html > > If the output is produced with unordered writes, but the complete output > needs to be verified with a hash *chain*, that still allows for some > level of asynchrony. The start of the hashing need not be delayed until > after the end of output, only after the start of output. > > For example, nbdcopy could maintain the highest offset up to which the > output is contiguous, and on a separate thread, it could be hashing the > output up to that offset. > > Considering a gigantic output, as yet unassembled blocks could likely > not be buffered in memory (that's why the writes are unordered in the > first place!), so the hashing thread would have to re-read the output > via NBD. Whether that would cause performance to improve or to > deteriorate is undecided IMO. If the far end of the output network block > device can accommodate a reader that is independent of the writers, then > this level of overlap is beneficial. Otherwise, this extra reader thread > would just add more thrashing, and we'd be better off with a separate > read-through once writing is complete. In my mind I'm wondering if there's any mathematical result that lets you combine each hash(block_i) into the final hash(block[1..N]) without needing to compute the hash of each block in order. (This is what blkhash solves, but unfortunately the output isn't compatible with standard hashes.) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH v3 05/29] vector: (mostly) factor out DEFINE_VECTOR_EMPTY
On 2/23/23 18:54, Laszlo Ersek wrote:
> On 2/22/23 18:01, Richard W.M. Jones wrote:
>> On Wed, Feb 22, 2023 at 05:22:57PM +0100, Laszlo Ersek wrote:
>>> On 2/22/23 10:40, Laszlo Ersek wrote:
On 2/22/23 09:17, Richard W.M. Jones wrote:
> On Tue, Feb 21, 2023 at 05:23:52PM +0100, Laszlo Ersek wrote:
>> This is doable, but I hope it's not expected that
>> DEFINE_POINTER_VECTOR_TYPE() *enforce* that the element type be a
>> pointer :)
>
>
> You might ignore this for a first draft, but it is apparently possible
> to statically detect this (at least, if using GCC/clang):
>
> https://stackoverflow.com/questions/19255148/check-if-a-macro-argument-is-a-pointer-or-not
Right, we already use at least __builtin_types_compatible_p in
TYPE_IS_ARRAY(); that's what I wouldn't want more of, at least via this
series.
>>>
>>> OK, summarizing the TODOs for this particular patch:
>>>
>>> 1. keep _iter
>>>
>>> 2. rename DEFINE_VECTOR_EMPTY to ADD_VECTOR_EMPTY_METHOD
>>>
>>> 3. introduce DEFINE_POINTER_VECTOR_TYPE that expands to DEFINE_VECTOR_TYPE
>>> + ADD_VECTOR_EMPTY_METHOD
>>>
>>> 4. DEFINE_POINTER_VECTOR_TYPE (and ADD_VECTOR_EMPTY_METHOD) should not take
>>> "free" as a parameter; ADD_VECTOR_EMPTY_METHOD should hard-code it
>>>
>>> 5. in "common/utils/string-vector.h", don't ADD_VECTOR_EMPTY_METHOD;
>>> instead, replace DEFINE_VECTOR_TYPE with DEFINE_POINTER_VECTOR_TYPE.
>>>
>>> 6. in ADD_VECTOR_EMPTY_METHOD, consider checking that the element type is a
>>> pointer type.
>>
>> Sounds like a plan.
>>
>>> --*--
>>>
>>> Re: 6, I find that the stackoverflow solution above is too complicated. I
>>> mentioned our existent TYPE_IS_ARRAY macro:
>>>
>>> #define TYPE_IS_ARRAY(a) \
>>> (!__builtin_types_compatible_p (typeof (a), typeof (&(a)[0])))
>>>
>>> This is perfectly usable for our purposes, as !TYPE_IS_ARRAY(). The reason
>>> is that when this macro is applied to anything that's *neither* a pointer
>>> *nor* an array, we get a build error at once:
>>>
>>> #define TYPE_IS_ARRAY(a) \
>>> (!__builtin_types_compatible_p (typeof (a), typeof (&(a)[0])))
>>>
>>> int main(void)
>>> {
>>> int x[5];
>>> int *y;
>>> int z;
>>>
>>> TYPE_IS_ARRAY (x);
>>> TYPE_IS_ARRAY (y);
>>> TYPE_IS_ARRAY (z);
>>> return 0;
>>> }
>>>
>>> --->
>>>
>>> isptr.c: In function ‘main’:
>>> isptr.c:2:59: error: subscripted value is neither array nor pointer nor
>>> vector
>>> 2 | (!__builtin_types_compatible_p (typeof (a), typeof (&(a)[0])))
>>> | ^
>>> isptr.c:12:3: note: in expansion of macro ‘TYPE_IS_ARRAY’
>>>12 | TYPE_IS_ARRAY (z);
>>> | ^
>>>
>>> (the "nor vector" part of the error message can most likely be ignored, I
>>> believe it refers to the C++ standard library vector class)
>>>
>>> Thus, with types that are neither pointers nor arrays nicely caught at
>>> compilation time, !TYPE_IS_ARRAY stands for "pointer".
>>
>> Nice .. but probably want to define:
>>
>> #define TYPE_IS_POINTER(p) (!TYPE_IS_ARRAY(p))
>>
>>> I'll experiment with this a bit, but if it becomes too complex, I'll likely
>>> drop step 6.
>
> Sigh, this patch has now turned into a series of 5 patches. I'll post it
> separately tomorrow. /smh
This patch has been rewritten and merged as commit range
254ac22dde42..c7ff70101e8c.
Laszlo
___
Libguestfs mailing list
Libguestfs@redhat.com
https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] [libnbd PATCH v5 0/5] vector: introduce DEFINE_POINTER_VECTOR_TYPE()
On 2/26/23 12:45, Laszlo Ersek wrote: > V4: > 20230224043937.392235-1-lersek@redhat.com">http://mid.mail-archive.com/20230224043937.392235-1-lersek@redhat.com > > Please see the Notes section in each patch for the updates. > > Laszlo Merged as commit range 254ac22dde42..c7ff70101e8c. Thanks Laszlo > > Laszlo Ersek (5): > common/include: add TYPE_IS_POINTER() macro > common/include: extract STATIC_ASSERT() macro > force semicolon after DEFINE_VECTOR_TYPE() macro invocations > vector: introduce DEFINE_POINTER_VECTOR_TYPE() > convert string_vector_(iter(free) + reset()) to string_vector_empty() > > common/include/Makefile.am | 1 + > common/include/checked-overflow.h| 8 ++-- > common/include/compiler-macros.h | 31 - > common/include/static-assert.h | 48 > common/include/test-array-size.c | 45 ++ > common/utils/string-vector.h | 2 +- > common/utils/test-vector.c | 3 +- > common/utils/vector.h| 42 - > copy/file-ops.c | 2 +- > copy/nbd-ops.c | 2 +- > dump/dump.c | 2 +- > fuse/nbdfuse.h | 2 +- > generator/states-connect-socket-activation.c | 9 ++-- > info/list.c | 2 +- > info/map.c | 2 +- > info/show.c | 3 +- > lib/handle.c | 12 ++--- > lib/uri.c| 2 +- > lib/utils.c | 6 +-- > ublk/nbdublk.h | 2 +- > ublk/tgt.c | 2 +- > 21 files changed, 157 insertions(+), 71 deletions(-) > create mode 100644 common/include/static-assert.h > > > base-commit: 594f7a738e0aba23eda37965c96b3df6b8f76960 > ___ > Libguestfs mailing list > Libguestfs@redhat.com > https://listman.redhat.com/mailman/listinfo/libguestfs > ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
Re: [Libguestfs] Checksums and other verification
On 2/27/23 17:44, Richard W.M. Jones wrote: > On Mon, Feb 27, 2023 at 08:42:23AM -0600, Eric Blake wrote: >> Or intentionally choose a hash that can be computed out-of-order, such >> as a Merkle Tree. But we'd need a standard setup for all parties to >> agree on how the hash is to be computed and checked, if it is going to >> be anything more than just a linear hash of the entire guest-visible >> contents. > > Unfortunately I suspect that by far the easiest way for people who > host images to compute checksums is to run 'shaXXXsum' on them or sign > them with a GPG signature, rather than engaging in a novel hash > function. Indeed that's what is happening now: > > https://alt.fedoraproject.org/en/verify.html If the output is produced with unordered writes, but the complete output needs to be verified with a hash *chain*, that still allows for some level of asynchrony. The start of the hashing need not be delayed until after the end of output, only after the start of output. For example, nbdcopy could maintain the highest offset up to which the output is contiguous, and on a separate thread, it could be hashing the output up to that offset. Considering a gigantic output, as yet unassembled blocks could likely not be buffered in memory (that's why the writes are unordered in the first place!), so the hashing thread would have to re-read the output via NBD. Whether that would cause performance to improve or to deteriorate is undecided IMO. If the far end of the output network block device can accommodate a reader that is independent of the writers, then this level of overlap is beneficial. Otherwise, this extra reader thread would just add more thrashing, and we'd be better off with a separate read-through once writing is complete. Laszlo ___ Libguestfs mailing list Libguestfs@redhat.com https://listman.redhat.com/mailman/listinfo/libguestfs
