Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
Am 29.06.2020 um 15:08 hat Nir Soffer geschrieben:
> On Mon, Jun 29, 2020 at 3:06 PM Kevin Wolf wrote:
> >
> > Am 26.06.2020 um 21:42 hat Nir Soffer geschrieben:
> > > On Tue, Jun 23, 2020 at 1:21 AM Nir Soffer wrote:
> > > >
> > > > I'm trying to export qcow2 images from ova format using qemu-nbd.
> > > >
> > > > I create 2 compressed qcow2 images, with different data:
> > > >
> > > > $ qemu-img info disk1.qcow2
> > > > image: disk1.qcow2
> > > > file format: qcow2
> > > > virtual size: 200 MiB (209715200 bytes)
> > > > disk size: 384 KiB
> > > > ...
> > > >
> > > > $ qemu-img info disk2.qcow2
> > > > image: disk2.qcow2
> > > > file format: qcow2
> > > > virtual size: 200 MiB (209715200 bytes)
> > > > disk size: 384 KiB
> > > > ...
> > > >
> > > > And packed them in a tar file. This is not a valid ova but good enough
> > > > for this test:
> > > >
> > > > $ tar tvf vm.ova
> > > > -rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk1.qcow2
> > > > -rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk2.qcow2
> > > >
> > > > To get info about the disks in ova file, we can use:
> > > >
> > > > $ python -c 'import tarfile; print(list({"name": m.name, "offset":
> > > > m.offset_data, "size": m.size} for m in tarfile.open("vm.ova")))'
> > > > [{'name': 'disk1.qcow2', 'offset': 512, 'size': 454144}, {'name':
> > > > 'disk2.qcow2', 'offset': 455168, 'size': 454144}]
> > > >
> > > > First I tried the obvious:
> > > >
> > > > $ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only --offset=512
> > > > vm.ova
> > > >
> > > > And it works, but it exposes the qcow2 data. I want to raw data so I
> > > > can upload the guest
> > > > data to ovirt, where is may be converted to qcow2 format.
> > > >
> > > > $ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
> > > > {
> > > > "virtual-size": 209715200,
> > > > "filename": "nbd+unix://?socket=/tmp/nbd.sock",
> > > > "format": "qcow2",
> > > > ...
> > > > }
> > > >
> > > > Looking in qemu manual and qapi/block-core.json, I could construct this
> > > > command:
> > > >
> > > > $ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only
> > > > 'json:{"driver": "qcow2", "file": {"driver": "raw", "offset": 512,
> > > > "size": 454144, "file": {"driver": "file", "filename": "vm.ova"}}}'
> > > >
> > > > And it works:
> > > >
> > > > $ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
> > > > {
> > > > "virtual-size": 209715200,
> > > > "filename": "nbd+unix://?socket=/tmp/nbd.sock",
> > > > "format": "raw"
> > > > }
> > > >
> > > > $ qemu-img map --output json "nbd+unix://?socket=/tmp/nbd.sock"
> > > > [{ "start": 0, "length": 104857600, "depth": 0, "zero": false, "data":
> > > > true, "offset": 0},
> > > > { "start": 104857600, "length": 104857600, "depth": 0, "zero": true,
> > > > "data": false, "offset": 104857600}]
> > > >
> > > > $ qemu-img map --output json disk1.qcow2
> > > > [{ "start": 0, "length": 104857600, "depth": 0, "zero": false, "data":
> > > > true},
> > > > { "start": 104857600, "length": 104857600, "depth": 0, "zero": true,
> > > > "data": false}]
> > > >
> > > > $ qemu-img convert -f raw -O raw nbd+unix://?socket=/tmp/nbd.sock
> > > > disk1.raw
> > > >
> > > > $ qemu-img info disk1.raw
> > > > image: disk1.raw
> > > > file format: raw
> > > > virtual size: 200 MiB (209715200 bytes)
> > > > disk size: 100 MiB
> > > >
> > > > $ qemu-img compare disk1.raw disk1.qcow2
> > > > Images are identical.
> > > >
> > > > I wonder if this is the best way to stack a qcow2 driver on top of a
> > > > raw driver exposing a range from a tar file.
> >
> > Yes, if you want to specify an offset and a size to access only part of
> > a file as the disk image, sticking a raw driver in the middle is the way
> > to go.
> >
> > > Other related challenges with this are:
> > >
> > > 1. probing image format
> > >
> > > With standalone images, we probe image format using:
> > >
> > > qemu-img info image
> > >
> > > I know probing is considered dangerous, but I think this ok when user
> > > run this code on his machine, on an image they want to upload to
> > > oVirt. On a hypervisor we use prlimit to limit the resources used by
> > > qemu-img, so we can use the same solution also when running by a user
> > > if needed.
> > >
> > > However not being able to probe image format is a usability issue. It
> > > does not make sense that qemu-img cannot probe image format safely, at
> > > least for qcow2 format.
> > >
> > > I can get image info using:
> > >
> > > $ qemu-img info 'json:{"driver": "qcow2", "file": {"driver": "raw",
> > > "offset": 1536, "file": {"driver": "file", "filename":
> > > "fedora-32.ova"}}}'
> > > image: json:{"driver": "qcow2", "file": {"offset": 1536, "driver":
> > > "raw", "file": {"driver": "file", "filename": "fedora-32.ova"}}}
> > > file format: qcow2
> > > virtual size: 6 GiB (6442450944 bytes)
> > > disk size: 645 MiB
> > > cluster_size: 65536
> > > Format specific information:
> > > compat: 1.1
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
On Mon, Jun 29, 2020 at 3:06 PM Kevin Wolf wrote:
>
> Am 26.06.2020 um 21:42 hat Nir Soffer geschrieben:
> > On Tue, Jun 23, 2020 at 1:21 AM Nir Soffer wrote:
> > >
> > > I'm trying to export qcow2 images from ova format using qemu-nbd.
> > >
> > > I create 2 compressed qcow2 images, with different data:
> > >
> > > $ qemu-img info disk1.qcow2
> > > image: disk1.qcow2
> > > file format: qcow2
> > > virtual size: 200 MiB (209715200 bytes)
> > > disk size: 384 KiB
> > > ...
> > >
> > > $ qemu-img info disk2.qcow2
> > > image: disk2.qcow2
> > > file format: qcow2
> > > virtual size: 200 MiB (209715200 bytes)
> > > disk size: 384 KiB
> > > ...
> > >
> > > And packed them in a tar file. This is not a valid ova but good enough
> > > for this test:
> > >
> > > $ tar tvf vm.ova
> > > -rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk1.qcow2
> > > -rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk2.qcow2
> > >
> > > To get info about the disks in ova file, we can use:
> > >
> > > $ python -c 'import tarfile; print(list({"name": m.name, "offset":
> > > m.offset_data, "size": m.size} for m in tarfile.open("vm.ova")))'
> > > [{'name': 'disk1.qcow2', 'offset': 512, 'size': 454144}, {'name':
> > > 'disk2.qcow2', 'offset': 455168, 'size': 454144}]
> > >
> > > First I tried the obvious:
> > >
> > > $ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only --offset=512
> > > vm.ova
> > >
> > > And it works, but it exposes the qcow2 data. I want to raw data so I
> > > can upload the guest
> > > data to ovirt, where is may be converted to qcow2 format.
> > >
> > > $ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
> > > {
> > > "virtual-size": 209715200,
> > > "filename": "nbd+unix://?socket=/tmp/nbd.sock",
> > > "format": "qcow2",
> > > ...
> > > }
> > >
> > > Looking in qemu manual and qapi/block-core.json, I could construct this
> > > command:
> > >
> > > $ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only
> > > 'json:{"driver": "qcow2", "file": {"driver": "raw", "offset": 512,
> > > "size": 454144, "file": {"driver": "file", "filename": "vm.ova"}}}'
> > >
> > > And it works:
> > >
> > > $ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
> > > {
> > > "virtual-size": 209715200,
> > > "filename": "nbd+unix://?socket=/tmp/nbd.sock",
> > > "format": "raw"
> > > }
> > >
> > > $ qemu-img map --output json "nbd+unix://?socket=/tmp/nbd.sock"
> > > [{ "start": 0, "length": 104857600, "depth": 0, "zero": false, "data":
> > > true, "offset": 0},
> > > { "start": 104857600, "length": 104857600, "depth": 0, "zero": true,
> > > "data": false, "offset": 104857600}]
> > >
> > > $ qemu-img map --output json disk1.qcow2
> > > [{ "start": 0, "length": 104857600, "depth": 0, "zero": false, "data":
> > > true},
> > > { "start": 104857600, "length": 104857600, "depth": 0, "zero": true,
> > > "data": false}]
> > >
> > > $ qemu-img convert -f raw -O raw nbd+unix://?socket=/tmp/nbd.sock
> > > disk1.raw
> > >
> > > $ qemu-img info disk1.raw
> > > image: disk1.raw
> > > file format: raw
> > > virtual size: 200 MiB (209715200 bytes)
> > > disk size: 100 MiB
> > >
> > > $ qemu-img compare disk1.raw disk1.qcow2
> > > Images are identical.
> > >
> > > I wonder if this is the best way to stack a qcow2 driver on top of a
> > > raw driver exposing a range from a tar file.
>
> Yes, if you want to specify an offset and a size to access only part of
> a file as the disk image, sticking a raw driver in the middle is the way
> to go.
>
> > Other related challenges with this are:
> >
> > 1. probing image format
> >
> > With standalone images, we probe image format using:
> >
> > qemu-img info image
> >
> > I know probing is considered dangerous, but I think this ok when user
> > run this code on his machine, on an image they want to upload to
> > oVirt. On a hypervisor we use prlimit to limit the resources used by
> > qemu-img, so we can use the same solution also when running by a user
> > if needed.
> >
> > However not being able to probe image format is a usability issue. It
> > does not make sense that qemu-img cannot probe image format safely, at
> > least for qcow2 format.
> >
> > I can get image info using:
> >
> > $ qemu-img info 'json:{"driver": "qcow2", "file": {"driver": "raw",
> > "offset": 1536, "file": {"driver": "file", "filename":
> > "fedora-32.ova"}}}'
> > image: json:{"driver": "qcow2", "file": {"offset": 1536, "driver":
> > "raw", "file": {"driver": "file", "filename": "fedora-32.ova"}}}
> > file format: qcow2
> > virtual size: 6 GiB (6442450944 bytes)
> > disk size: 645 MiB
> > cluster_size: 65536
> > Format specific information:
> > compat: 1.1
> > lazy refcounts: false
> > refcount bits: 16
> > corrupt: false
> >
> > But there is no way to probe the format, unless I try first with
> > qcow2, and consider the image as raw otherwise.
>
> Just leave out the top-level "driver" option. This isn't -blockdev
> (which does indeed require a
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
Am 26.06.2020 um 21:42 hat Nir Soffer geschrieben:
> On Tue, Jun 23, 2020 at 1:21 AM Nir Soffer wrote:
> >
> > I'm trying to export qcow2 images from ova format using qemu-nbd.
> >
> > I create 2 compressed qcow2 images, with different data:
> >
> > $ qemu-img info disk1.qcow2
> > image: disk1.qcow2
> > file format: qcow2
> > virtual size: 200 MiB (209715200 bytes)
> > disk size: 384 KiB
> > ...
> >
> > $ qemu-img info disk2.qcow2
> > image: disk2.qcow2
> > file format: qcow2
> > virtual size: 200 MiB (209715200 bytes)
> > disk size: 384 KiB
> > ...
> >
> > And packed them in a tar file. This is not a valid ova but good enough
> > for this test:
> >
> > $ tar tvf vm.ova
> > -rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk1.qcow2
> > -rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk2.qcow2
> >
> > To get info about the disks in ova file, we can use:
> >
> > $ python -c 'import tarfile; print(list({"name": m.name, "offset":
> > m.offset_data, "size": m.size} for m in tarfile.open("vm.ova")))'
> > [{'name': 'disk1.qcow2', 'offset': 512, 'size': 454144}, {'name':
> > 'disk2.qcow2', 'offset': 455168, 'size': 454144}]
> >
> > First I tried the obvious:
> >
> > $ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only --offset=512
> > vm.ova
> >
> > And it works, but it exposes the qcow2 data. I want to raw data so I
> > can upload the guest
> > data to ovirt, where is may be converted to qcow2 format.
> >
> > $ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
> > {
> > "virtual-size": 209715200,
> > "filename": "nbd+unix://?socket=/tmp/nbd.sock",
> > "format": "qcow2",
> > ...
> > }
> >
> > Looking in qemu manual and qapi/block-core.json, I could construct this
> > command:
> >
> > $ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only
> > 'json:{"driver": "qcow2", "file": {"driver": "raw", "offset": 512,
> > "size": 454144, "file": {"driver": "file", "filename": "vm.ova"}}}'
> >
> > And it works:
> >
> > $ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
> > {
> > "virtual-size": 209715200,
> > "filename": "nbd+unix://?socket=/tmp/nbd.sock",
> > "format": "raw"
> > }
> >
> > $ qemu-img map --output json "nbd+unix://?socket=/tmp/nbd.sock"
> > [{ "start": 0, "length": 104857600, "depth": 0, "zero": false, "data":
> > true, "offset": 0},
> > { "start": 104857600, "length": 104857600, "depth": 0, "zero": true,
> > "data": false, "offset": 104857600}]
> >
> > $ qemu-img map --output json disk1.qcow2
> > [{ "start": 0, "length": 104857600, "depth": 0, "zero": false, "data":
> > true},
> > { "start": 104857600, "length": 104857600, "depth": 0, "zero": true,
> > "data": false}]
> >
> > $ qemu-img convert -f raw -O raw nbd+unix://?socket=/tmp/nbd.sock disk1.raw
> >
> > $ qemu-img info disk1.raw
> > image: disk1.raw
> > file format: raw
> > virtual size: 200 MiB (209715200 bytes)
> > disk size: 100 MiB
> >
> > $ qemu-img compare disk1.raw disk1.qcow2
> > Images are identical.
> >
> > I wonder if this is the best way to stack a qcow2 driver on top of a
> > raw driver exposing a range from a tar file.
Yes, if you want to specify an offset and a size to access only part of
a file as the disk image, sticking a raw driver in the middle is the way
to go.
> Other related challenges with this are:
>
> 1. probing image format
>
> With standalone images, we probe image format using:
>
> qemu-img info image
>
> I know probing is considered dangerous, but I think this ok when user
> run this code on his machine, on an image they want to upload to
> oVirt. On a hypervisor we use prlimit to limit the resources used by
> qemu-img, so we can use the same solution also when running by a user
> if needed.
>
> However not being able to probe image format is a usability issue. It
> does not make sense that qemu-img cannot probe image format safely, at
> least for qcow2 format.
>
> I can get image info using:
>
> $ qemu-img info 'json:{"driver": "qcow2", "file": {"driver": "raw",
> "offset": 1536, "file": {"driver": "file", "filename":
> "fedora-32.ova"}}}'
> image: json:{"driver": "qcow2", "file": {"offset": 1536, "driver":
> "raw", "file": {"driver": "file", "filename": "fedora-32.ova"}}}
> file format: qcow2
> virtual size: 6 GiB (6442450944 bytes)
> disk size: 645 MiB
> cluster_size: 65536
> Format specific information:
> compat: 1.1
> lazy refcounts: false
> refcount bits: 16
> corrupt: false
>
> But there is no way to probe the format, unless I try first with
> qcow2, and consider the image as raw otherwise.
Just leave out the top-level "driver" option. This isn't -blockdev
(which does indeed require a "driver"), but uses the same logic as
-drive and therefore supports format probing:
$ ./qemu-img info
'json:{"file":{"driver":"raw","offset":512,"size":2424832,"file":{"filename":"/tmp/test.ova"}}}'
image: json:{"driver": "qcow2", "file": {"offset": 512, "driver": "raw",
"size": 2424832, "file": {"driver": "file", "filename":
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
On Fri, Jun 26, 2020 at 09:47:24PM +0300, Nir Soffer wrote: > Whats's missing in the current solution is supporting compressed raw > disk. I don't think oVirt creates such ova files, so this should not > be an issue. However if nbdkit tar plugin can support this (current > implementation does not), this will be a good reason to integrate > with it. As far as I know the OVA format requires that the tar file is uncompressed. Also the disk images inside the tar file are supposed to be uncompressed (except maybe for qcow2 compression, but that's different). > I wonder if it is possible to add gzip driver in qemu block layer? Kevin already answered this, and as he said no it's not possible. There's already a gzip plugin in nbdkit but it has to uncompress the whole file so it's not very practical except for small images. XZ (lzma) is much better here -- nbdkit has an xz filter already -- but you have to prepare the XZ file with special flags. Eventually zstd may support a seekable format (https://github.com/facebook/zstd/issues/395#issuecomment-535875379). I'm looking into making it easier to put nbdkit in front of qemu-nbd. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
Am 26.06.2020 um 20:47 hat Nir Soffer geschrieben:
> On Tue, Jun 23, 2020 at 1:47 PM Richard W.M. Jones wrote:
> >
> > On Tue, Jun 23, 2020 at 01:14:43PM +0300, Nir Soffer wrote:
> > > On Tue, Jun 23, 2020 at 12:47 PM Richard W.M. Jones
> > > wrote:
> > > > Here you go:
> > > >
> > > > https://github.com/libguestfs/nbdkit/commit/2d15e79f65764d9b0c68bea28ed6afbcbcc63467
> > >
> > > Nice!
> > >
> > > But using qemu-nbd directly is much simpler and will perform better.
> >
> > Not sure about simpler,
>
> These are the patches (in review) implementing this in imageio client:
>
> - https://gerrit.ovirt.org/c/109847
> - https://gerrit.ovirt.org/c/109848
>
> And this is example script for the engine SDK:
> https://gerrit.ovirt.org/c/109873/
>
> Work is not finished yet, we need to handle getting image virtual size
> and measuring
> required size to support upload to sparse disks on block storage:
> https://bugzilla.redhat.com/show_bug.cgi?id=1849981#c3
>
> Since we already run qemu-nbd to access images, this is just a
> modification of the
> 'json:{...}' filename, so we can also consume images inside tar file.
>
> And in the client, allow specifying image format and tar member name
> (like the tar plugin).
>
> Using tar plugin requires:
> - Adding support for NBD url - this is planned but at least the same
> amount of work as supporting
> json: with qcow2 drive over raw with range.
> - Integrating with nbdkit and qemu-nbd (since we must have qcow2 support)
> - packaging
> - handle missing dependencies
>
> Whats's missing in the current solution is supporting compressed raw
> disk. I don't think
> oVirt creates such ova files, so this should not be an issue. However
> if nbdkit tar
> plugin can support this (current implementation does not), this will
> be a good reason
> to integrate with it.
>
> In this case we will have:
>
> [ovf | gziped raw image | ...] -> nbdkit exposing uncompressed
> raw image -> imageio nbd client
>
> I wonder if it is possible to add gzip driver in qemu block layer?
gzip isn't block based, so if the file format isn't more complex than
simply gzipping the whole raw image at once, I think you can only read
things sequentially. This isn't a good fit for the QEMU block layer
which is based on random access operations.
Kevin
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
On Tue, Jun 23, 2020 at 01:14:43PM +0300, Nir Soffer wrote: > On Tue, Jun 23, 2020 at 12:47 PM Richard W.M. Jones wrote: > > > > > > Here you go: > > > > https://github.com/libguestfs/nbdkit/commit/2d15e79f65764d9b0c68bea28ed6afbcbcc63467 > > Nice! > > But using qemu-nbd directly is much simpler and will perform better. nbdkit-tar-plugin now rewritten in C: https://www.redhat.com/archives/libguestfs/2020-June/msg00119.html > Regardless, nbdit tar plugin is awesome. Is it possible to expose all > the disks from > a tar file so they are accessible using the export name? I still didn't implement this bit, left as an exercise for the reader. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
On Fri, Jun 26, 2020 at 11:34 PM Richard W.M. Jones wrote:
>
> On Fri, Jun 26, 2020 at 10:42:02PM +0300, Nir Soffer wrote:
> > Can we have better support in qemu-img/qemu-nbd for accessing images
> > in a tar file?
> >
> > Maybe something like:
> >
> > qemu-img info tar://vm.ova?member=fedora-32.qcow2
>
> Isn't this exactly a case where nbdkit-tar-plugin would work despite
> the performance problems with it being written in Python? Something like:
>
> $ tar tvf disk.ova
> -rw-r--r-- rjones/rjones 2031616 2020-06-26 21:32 disk.qcow2
>
> $ nbdkit -U - tar tar=disk.ova file=disk.qcow2 --run 'qemu-img info
> --output=json $nbd'
> {
> "virtual-size": 105923072,
> "filename": "nbd+unix://?socket=/tmp/nbdkitTjkeRd/socket",
> "cluster-size": 65536,
> "format": "qcow2",
> "format-specific": {
> "type": "qcow2",
> "data": {
> "compat": "1.1",
> "lazy-refcounts": false,
> "refcount-bits": 16,
> "corrupt": false
> }
> },
> "dirty-flag": false
> }
>
> qemu-img measure will work the same way.
Looks like format probing just works this way, I'll try this, thanks!
Nir
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
On Fri, Jun 26, 2020 at 10:42:02PM +0300, Nir Soffer wrote:
> Can we have better support in qemu-img/qemu-nbd for accessing images
> in a tar file?
>
> Maybe something like:
>
> qemu-img info tar://vm.ova?member=fedora-32.qcow2
Isn't this exactly a case where nbdkit-tar-plugin would work despite
the performance problems with it being written in Python? Something like:
$ tar tvf disk.ova
-rw-r--r-- rjones/rjones 2031616 2020-06-26 21:32 disk.qcow2
$ nbdkit -U - tar tar=disk.ova file=disk.qcow2 --run 'qemu-img info
--output=json $nbd'
{
"virtual-size": 105923072,
"filename": "nbd+unix://?socket=/tmp/nbdkitTjkeRd/socket",
"cluster-size": 65536,
"format": "qcow2",
"format-specific": {
"type": "qcow2",
"data": {
"compat": "1.1",
"lazy-refcounts": false,
"refcount-bits": 16,
"corrupt": false
}
},
"dirty-flag": false
}
qemu-img measure will work the same way.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
On Tue, Jun 23, 2020 at 1:21 AM Nir Soffer wrote:
>
> I'm trying to export qcow2 images from ova format using qemu-nbd.
>
> I create 2 compressed qcow2 images, with different data:
>
> $ qemu-img info disk1.qcow2
> image: disk1.qcow2
> file format: qcow2
> virtual size: 200 MiB (209715200 bytes)
> disk size: 384 KiB
> ...
>
> $ qemu-img info disk2.qcow2
> image: disk2.qcow2
> file format: qcow2
> virtual size: 200 MiB (209715200 bytes)
> disk size: 384 KiB
> ...
>
> And packed them in a tar file. This is not a valid ova but good enough
> for this test:
>
> $ tar tvf vm.ova
> -rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk1.qcow2
> -rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk2.qcow2
>
> To get info about the disks in ova file, we can use:
>
> $ python -c 'import tarfile; print(list({"name": m.name, "offset":
> m.offset_data, "size": m.size} for m in tarfile.open("vm.ova")))'
> [{'name': 'disk1.qcow2', 'offset': 512, 'size': 454144}, {'name':
> 'disk2.qcow2', 'offset': 455168, 'size': 454144}]
>
> First I tried the obvious:
>
> $ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only --offset=512 vm.ova
>
> And it works, but it exposes the qcow2 data. I want to raw data so I
> can upload the guest
> data to ovirt, where is may be converted to qcow2 format.
>
> $ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
> {
> "virtual-size": 209715200,
> "filename": "nbd+unix://?socket=/tmp/nbd.sock",
> "format": "qcow2",
> ...
> }
>
> Looking in qemu manual and qapi/block-core.json, I could construct this
> command:
>
> $ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only
> 'json:{"driver": "qcow2", "file": {"driver": "raw", "offset": 512,
> "size": 454144, "file": {"driver": "file", "filename": "vm.ova"}}}'
>
> And it works:
>
> $ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
> {
> "virtual-size": 209715200,
> "filename": "nbd+unix://?socket=/tmp/nbd.sock",
> "format": "raw"
> }
>
> $ qemu-img map --output json "nbd+unix://?socket=/tmp/nbd.sock"
> [{ "start": 0, "length": 104857600, "depth": 0, "zero": false, "data":
> true, "offset": 0},
> { "start": 104857600, "length": 104857600, "depth": 0, "zero": true,
> "data": false, "offset": 104857600}]
>
> $ qemu-img map --output json disk1.qcow2
> [{ "start": 0, "length": 104857600, "depth": 0, "zero": false, "data": true},
> { "start": 104857600, "length": 104857600, "depth": 0, "zero": true,
> "data": false}]
>
> $ qemu-img convert -f raw -O raw nbd+unix://?socket=/tmp/nbd.sock disk1.raw
>
> $ qemu-img info disk1.raw
> image: disk1.raw
> file format: raw
> virtual size: 200 MiB (209715200 bytes)
> disk size: 100 MiB
>
> $ qemu-img compare disk1.raw disk1.qcow2
> Images are identical.
>
> I wonder if this is the best way to stack a qcow2 driver on top of a
> raw driver exposing
> a range from a tar file.
>
> I found similar example for gluster in:
> docs/system/device-url-syntax.rst.inc
Other related challenges with this are:
1. probing image format
With standalone images, we probe image format using:
qemu-img info image
I know probing is considered dangerous, but I think this ok when user
run this code on
his machine, on an image they want to upload to oVirt. On a hypervisor
we use prlimit
to limit the resources used by qemu-img, so we can use the same
solution also when
running by a user if needed.
However not being able to probe image format is a usability issue. It
does not make sense
that qemu-img cannot probe image format safely, at least for qcow2 format.
I can get image info using:
$ qemu-img info 'json:{"driver": "qcow2", "file": {"driver": "raw",
"offset": 1536, "file": {"driver": "file", "filename":
"fedora-32.ova"}}}'
image: json:{"driver": "qcow2", "file": {"offset": 1536, "driver":
"raw", "file": {"driver": "file", "filename": "fedora-32.ova"}}}
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 645 MiB
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
refcount bits: 16
corrupt: false
But there is no way to probe the format, unless I try first with
qcow2, and consider the image as raw
otherwise.
We can parse the qcow2 header manually, as we already do in oVirt
engine UI in javascript:
https://github.com/oVirt/ovirt-engine/blob/9d48ea6274fdd1bef3fc8e309f9161be3b540890/frontend/webadmin/modules/uicommonweb/src/main/java/org/ovirt/engine/ui/uicommonweb/models/storage/ImageInfoModel.java#L103
We have used this code for 5 years and had no issues with it yet.
In the worst case, if we fail to detect, or let the user upload a
qcow2 files oVirt does not
support, the uload will fail at the end, in the verification step,
when we run check the
uploaded image using "qemu-img info". This is done using prlimit since
we treat this
image as untrusted.
I think it would be useful if the qemu project was publishing
libraries in C/python/javascript
supporting format probing for qcow2 format.
2. getting image
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
On Tue, Jun 23, 2020 at 1:47 PM Richard W.M. Jones wrote:
>
> On Tue, Jun 23, 2020 at 01:14:43PM +0300, Nir Soffer wrote:
> > On Tue, Jun 23, 2020 at 12:47 PM Richard W.M. Jones
> > wrote:
> > > Here you go:
> > >
> > > https://github.com/libguestfs/nbdkit/commit/2d15e79f65764d9b0c68bea28ed6afbcbcc63467
> >
> > Nice!
> >
> > But using qemu-nbd directly is much simpler and will perform better.
>
> Not sure about simpler,
These are the patches (in review) implementing this in imageio client:
- https://gerrit.ovirt.org/c/109847
- https://gerrit.ovirt.org/c/109848
And this is example script for the engine SDK:
https://gerrit.ovirt.org/c/109873/
Work is not finished yet, we need to handle getting image virtual size
and measuring
required size to support upload to sparse disks on block storage:
https://bugzilla.redhat.com/show_bug.cgi?id=1849981#c3
Since we already run qemu-nbd to access images, this is just a
modification of the
'json:{...}' filename, so we can also consume images inside tar file.
And in the client, allow specifying image format and tar member name
(like the tar plugin).
Using tar plugin requires:
- Adding support for NBD url - this is planned but at least the same
amount of work as supporting
json: with qcow2 drive over raw with range.
- Integrating with nbdkit and qemu-nbd (since we must have qcow2 support)
- packaging
- handle missing dependencies
Whats's missing in the current solution is supporting compressed raw
disk. I don't think
oVirt creates such ova files, so this should not be an issue. However
if nbdkit tar
plugin can support this (current implementation does not), this will
be a good reason
to integrate with it.
In this case we will have:
[ovf | gziped raw image | ...] -> nbdkit exposing uncompressed
raw image -> imageio nbd client
I wonder if it is possible to add gzip driver in qemu block layer?
> and you might want to verify the "perform better" claim
I did not test yet on real server and harder, but from initial tests I
get same performance
from uploading qcow2 compressed image and uploading qcow2 compressed
image inside
ova file.
> (it is likely to be true because writing a plugin in
> Python causes requests to be serialized, but it may not matter if
> you're reading linearly from a file).
It matters, because having serial reads also imply serial writes to the server
in imageio. We could change this but the current design we have multiple threads
reading from nbd server, and sending data or zero requests to imageio server.
This doubles the performance we had in ovirt 4.3.
You can see here results compared to qemu-img convert from local image
to block device:
https://github.com/oVirt/ovirt-imageio/commit/97f2e277458db579023ba54a4a4bd122b36f543e
The unexpected slow results with qemu-img are caused by the pre-zero
pessimisation
which should be removed soon:
https://lists.nongnu.org/archive/html/qemu-block/2020-06/msg01094.html
With the slow pre-zeroing removed, qemu-img convert gives similar performance.
> The tar plugin could be rewritten in C if performance was really a problem.
This may make the tar plugin more attractive, especially if we can
support multiple
readers. Maybe we can separate the tar parsing to a helper process, and use the
results (list of (member, offset, size)) in a file-like plugin in C?
> > Regardless, nbdit tar plugin is awesome. Is it possible to expose
> > all the disks from a tar file so they are accessible using the
> > export name?
>
> In theory yes, but it would require exposing the export name
> (nbdkit_export_name() -> nbdkit.export_name()) to Python plugins,
> which we don't do at the moment. See plugins/python/python.c:
> NbdkitMethods[]. That would also mean the plugin would require the
> latest nbdkit so you'd have to wait for patches to get backported to
> RHEL 8.
>
> You would also have to be cautious with security because the export
> name is supplied by the untrusted client.
>
> > For example:
> >
> > $ nbdkit tar file=vm.ova
> >
> > $ qemu-nbd --list
> > exports available: 2
> > export: 'disk1.qcow2'
> > size: 910848
> > flags: 0x48f ( readonly flush fua df cache )
> > min block: 512
> > opt block: 4096
> > max block: 33554432
> > available meta contexts: 1
> >base:allocation
> > export: 'disk2.qcow2'
> > size: 910848
> > flags: 0x48f ( readonly flush fua df cache )
> > min block: 512
> > opt block: 4096
> > max block: 33554432
> > available meta contexts: 1
> >base:allocation
> >
> > $ qemu-img convert -f qcow2 -O raw nbb://localhost/disk1.qcow2 disk1.raw
> >
> > $ qemu-img convert -f qcow2 -O raw nbb://localhost/disk2.qcow2 disk2.raw
>
> Yup, it'd be nice ...
>
> Also nbdkit doesn't support the extension for listing export names,
> and that's a bunch more work.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-p2v converts physical
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
On Tue, Jun 23, 2020 at 5:08 PM Richard W.M. Jones wrote: > > On Tue, Jun 23, 2020 at 08:47:52AM -0500, Eric Blake wrote: > > On 6/22/20 5:21 PM, Nir Soffer wrote: > > >And it works, but it exposes the qcow2 data. I want to raw data so I > > >can upload the guest > > >data to ovirt, where is may be converted to qcow2 format. > > Nir, can you use qemu-img convert and get a free conversion to your > choice of format? This works fine over NBD as long as you don't try > and write which I guess you don't want to do here. No, this code can run on remote host that does not have access to storage, or on a hypervisor as any user that cannot access storage. > > >Richard suggested to try nbdkit tar plugin, but the plugin is not > > >available on RHEL, > > >and this adds additional dependency, when we already use qemu-nbd. > > > > Rich just rewrote the tar plugin to use python instead of perl, > > which means it is that much easier for a future RHEL to pull it in. > > We still ought to consider having a tar filter, either in place of > > or in addition to, the tar plugin (similar to how we recently > > converted nbdkit's ext4 support from a plugin to a filter) - having > > a tar filter would allow you to read a compressed ova file (by > > combining the xz and tar filters to decompress then extract a file). > > But right now, nbdkit doesn't support non-C filters (and given that > > our tar plugin was written first in perl and now in python, that > > still means translation to yet another language if the filter > > requires it to be in C). > > The reason it was in Perl and is now in Python (and not C), and also > the reason it still a plugin, is that parsing tar files is very > complex because of historical compatibility. If we accept that we > cannot write a from-scratch tar file parser in C then we have to use > an existing tool or library (‘tar’ itself was used by Perl, now we're > using ‘tarfile.py’ from Python stdlib). Those tools require access to > an actual local file. So I'm afraid this rewrite is hard work :-) > > Unless we accept that we only parse files created by a narrow range of > tools, but the problem is that OVA files can be generated by a wide > variety of tools. > > If you can supply the offset by some other means then of course using > nbdkit-offset-filter or qemu's offset block layer is the solution. > > Rich. > > -- > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones > Read my programming and virtualization blog: http://rwmj.wordpress.com > virt-p2v converts physical machines to virtual machines. Boot with a > live CD or over the network (PXE) and turn machines into KVM guests. > http://libguestfs.org/virt-v2v >
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
On 2020-06-23 16:08, Richard W.M. Jones wrote: On Tue, Jun 23, 2020 at 08:47:52AM -0500, Eric Blake wrote: On 6/22/20 5:21 PM, Nir Soffer wrote: And it works, but it exposes the qcow2 data. I want to raw data so I can upload the guest data to ovirt, where is may be converted to qcow2 format. Nir, can you use qemu-img convert and get a free conversion to your choice of format? This works fine over NBD as long as you don't try and write which I guess you don't want to do here. Richard suggested to try nbdkit tar plugin, but the plugin is not available on RHEL, and this adds additional dependency, when we already use qemu-nbd. Rich just rewrote the tar plugin to use python instead of perl, which means it is that much easier for a future RHEL to pull it in. We still ought to consider having a tar filter, either in place of or in addition to, the tar plugin (similar to how we recently converted nbdkit's ext4 support from a plugin to a filter) - having a tar filter would allow you to read a compressed ova file (by combining the xz and tar filters to decompress then extract a file). But right now, nbdkit doesn't support non-C filters (and given that our tar plugin was written first in perl and now in python, that still means translation to yet another language if the filter requires it to be in C). The reason it was in Perl and is now in Python (and not C), and also the reason it still a plugin, is that parsing tar files is very complex because of historical compatibility. If we accept that we cannot write a from-scratch tar file parser in C then we have to use an existing tool or library (‘tar’ itself was used by Perl, now we're using ‘tarfile.py’ from Python stdlib). Those tools require access to an actual local file. So I'm afraid this rewrite is hard work :-) Unless we accept that we only parse files created by a narrow range of tools, but the problem is that OVA files can be generated by a wide variety of tools. If you can supply the offset by some other means then of course using nbdkit-offset-filter or qemu's offset block layer is the solution. Note that the BSD implementation of tar (in C) is available as a library, which is also available on some Linux systems. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
On Tue, Jun 23, 2020 at 08:47:52AM -0500, Eric Blake wrote: > On 6/22/20 5:21 PM, Nir Soffer wrote: > >And it works, but it exposes the qcow2 data. I want to raw data so I > >can upload the guest > >data to ovirt, where is may be converted to qcow2 format. Nir, can you use qemu-img convert and get a free conversion to your choice of format? This works fine over NBD as long as you don't try and write which I guess you don't want to do here. > >Richard suggested to try nbdkit tar plugin, but the plugin is not > >available on RHEL, > >and this adds additional dependency, when we already use qemu-nbd. > > Rich just rewrote the tar plugin to use python instead of perl, > which means it is that much easier for a future RHEL to pull it in. > We still ought to consider having a tar filter, either in place of > or in addition to, the tar plugin (similar to how we recently > converted nbdkit's ext4 support from a plugin to a filter) - having > a tar filter would allow you to read a compressed ova file (by > combining the xz and tar filters to decompress then extract a file). > But right now, nbdkit doesn't support non-C filters (and given that > our tar plugin was written first in perl and now in python, that > still means translation to yet another language if the filter > requires it to be in C). The reason it was in Perl and is now in Python (and not C), and also the reason it still a plugin, is that parsing tar files is very complex because of historical compatibility. If we accept that we cannot write a from-scratch tar file parser in C then we have to use an existing tool or library (‘tar’ itself was used by Perl, now we're using ‘tarfile.py’ from Python stdlib). Those tools require access to an actual local file. So I'm afraid this rewrite is hard work :-) Unless we accept that we only parse files created by a narrow range of tools, but the problem is that OVA files can be generated by a wide variety of tools. If you can supply the offset by some other means then of course using nbdkit-offset-filter or qemu's offset block layer is the solution. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
On 6/22/20 5:21 PM, Nir Soffer wrote:
I'm trying to export qcow2 images from ova format using qemu-nbd.
I create 2 compressed qcow2 images, with different data:
$ qemu-img info disk1.qcow2
image: disk1.qcow2
file format: qcow2
virtual size: 200 MiB (209715200 bytes)
disk size: 384 KiB
...
$ qemu-img info disk2.qcow2
image: disk2.qcow2
file format: qcow2
virtual size: 200 MiB (209715200 bytes)
disk size: 384 KiB
...
And packed them in a tar file. This is not a valid ova but good enough
for this test:
$ tar tvf vm.ova
-rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk1.qcow2
-rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk2.qcow2
Thanks for the reproduction steps, this helps me reproduce locally.
To get info about the disks in ova file, we can use:
$ python -c 'import tarfile; print(list({"name": m.name, "offset":
m.offset_data, "size": m.size} for m in tarfile.open("vm.ova")))'
[{'name': 'disk1.qcow2', 'offset': 512, 'size': 454144}, {'name':
'disk2.qcow2', 'offset': 455168, 'size': 454144}]
First I tried the obvious:
$ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only --offset=512 vm.ova
And it works, but it exposes the qcow2 data. I want to raw data so I
can upload the guest
data to ovirt, where is may be converted to qcow2 format.
So you definitely need two layers of interpretation: one that reads from
an offset within the tar file, another that reads guest-visible data
from the qcow2 contents at that offset. Both qemu and nbdkit can
perform the first layer, but at the moment, nbdkit cannot perform the
second (our thoughts along that line have been that if qemu can read
qcow2 files, there's no need to duplicate that effort in nbdkit). So we
either need to come up with a single qemu-nbd command line that can do
it all, or else a process chain where nbdkit serves the offset over a
Unix socket, then qemu-nbd consumes that socket to server the
guest-visible data of the qcow2 format.
$ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
{
"virtual-size": 209715200,
"filename": "nbd+unix://?socket=/tmp/nbd.sock",
"format": "qcow2",
...
}
Looking in qemu manual and qapi/block-core.json, I could construct this command:
$ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only
'json:{"driver": "qcow2", "file": {"driver": "raw", "offset": 512,
"size": 454144, "file": {"driver": "file", "filename": "vm.ova"}}}'
Yep, that's getting qemu to perform both layers. The pseudo-format
json:{...} is a bit of a pain to write but accurate; it is also possible
to write the same command using --image-opts:
qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only --image-opts \
driver=qcow2,file.driver=raw,file.offset=512,file.size=454144,\
file.file.driver=file,file.file.filename=vm.ova
And it works:
$ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
{
"virtual-size": 209715200,
"filename": "nbd+unix://?socket=/tmp/nbd.sock",
"format": "raw"
}
I wonder if this is the best way to stack a qcow2 driver on top of a
raw driver exposing
a range from a tar file.
Yes, that looks best to me. As I said above, the only other option I can
think of is to mix qemu-nbd and nbdkit, as in:
nbdkit -r -U - --filter=offset file vm.ova offset=512 range=454144 \
--run 'qemu-nbd -k /tmp/nbd.sock -r -f qcow2 $uri'
which gives the same contents over /tmp/nbd.sock with a shorter command
line but with more processes (and thus more computation spent per I/O)
involved.
I found similar example for gluster in:
docs/system/device-url-syntax.rst.inc
Richard suggested to try nbdkit tar plugin, but the plugin is not
available on RHEL,
and this adds additional dependency, when we already use qemu-nbd.
Rich just rewrote the tar plugin to use python instead of perl, which
means it is that much easier for a future RHEL to pull it in. We still
ought to consider having a tar filter, either in place of or in addition
to, the tar plugin (similar to how we recently converted nbdkit's ext4
support from a plugin to a filter) - having a tar filter would allow you
to read a compressed ova file (by combining the xz and tar filters to
decompress then extract a file). But right now, nbdkit doesn't support
non-C filters (and given that our tar plugin was written first in perl
and now in python, that still means translation to yet another language
if the filter requires it to be in C).
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
On Tue, Jun 23, 2020 at 11:47:19AM +0100, Richard W.M. Jones wrote: > In theory yes, but it would require exposing the export name > (nbdkit_export_name() -> nbdkit.export_name()) to Python plugins, > which we don't do at the moment. See plugins/python/python.c: > NbdkitMethods[]. That would also mean the plugin would require the > latest nbdkit so you'd have to wait for patches to get backported to > RHEL 8. I added a binding to nbdkit_export_name() upstream. Modifying the new tar plugin to support it is left as an exercise for the reader (but isn't very hard). Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
On Tue, Jun 23, 2020 at 01:14:43PM +0300, Nir Soffer wrote: > On Tue, Jun 23, 2020 at 12:47 PM Richard W.M. Jones wrote: > > Here you go: > > > > https://github.com/libguestfs/nbdkit/commit/2d15e79f65764d9b0c68bea28ed6afbcbcc63467 > > Nice! > > But using qemu-nbd directly is much simpler and will perform better. Not sure about simpler, and you might want to verify the "perform better" claim (it is likely to be true because writing a plugin in Python causes requests to be serialized, but it may not matter if you're reading linearly from a file). The tar plugin could be rewritten in C if performance was really a problem. > Regardless, nbdit tar plugin is awesome. Is it possible to expose > all the disks from a tar file so they are accessible using the > export name? In theory yes, but it would require exposing the export name (nbdkit_export_name() -> nbdkit.export_name()) to Python plugins, which we don't do at the moment. See plugins/python/python.c: NbdkitMethods[]. That would also mean the plugin would require the latest nbdkit so you'd have to wait for patches to get backported to RHEL 8. You would also have to be cautious with security because the export name is supplied by the untrusted client. > For example: > > $ nbdkit tar file=vm.ova > > $ qemu-nbd --list > exports available: 2 > export: 'disk1.qcow2' > size: 910848 > flags: 0x48f ( readonly flush fua df cache ) > min block: 512 > opt block: 4096 > max block: 33554432 > available meta contexts: 1 >base:allocation > export: 'disk2.qcow2' > size: 910848 > flags: 0x48f ( readonly flush fua df cache ) > min block: 512 > opt block: 4096 > max block: 33554432 > available meta contexts: 1 >base:allocation > > $ qemu-img convert -f qcow2 -O raw nbb://localhost/disk1.qcow2 disk1.raw > > $ qemu-img convert -f qcow2 -O raw nbb://localhost/disk2.qcow2 disk2.raw Yup, it'd be nice ... Also nbdkit doesn't support the extension for listing export names, and that's a bunch more work. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
On Tue, Jun 23, 2020 at 12:47 PM Richard W.M. Jones wrote: > > > Here you go: > > https://github.com/libguestfs/nbdkit/commit/2d15e79f65764d9b0c68bea28ed6afbcbcc63467 Nice! But using qemu-nbd directly is much simpler and will perform better. Regardless, nbdit tar plugin is awesome. Is it possible to expose all the disks from a tar file so they are accessible using the export name? For example: $ nbdkit tar file=vm.ova $ qemu-nbd --list exports available: 2 export: 'disk1.qcow2' size: 910848 flags: 0x48f ( readonly flush fua df cache ) min block: 512 opt block: 4096 max block: 33554432 available meta contexts: 1 base:allocation export: 'disk2.qcow2' size: 910848 flags: 0x48f ( readonly flush fua df cache ) min block: 512 opt block: 4096 max block: 33554432 available meta contexts: 1 base:allocation $ qemu-img convert -f qcow2 -O raw nbb://localhost/disk1.qcow2 disk1.raw $ qemu-img convert -f qcow2 -O raw nbb://localhost/disk2.qcow2 disk2.raw > Rich. > > -- > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones > Read my programming and virtualization blog: http://rwmj.wordpress.com > virt-top is 'top' for virtual machines. Tiny program with many > powerful monitoring features, net stats, disk stats, logging, etc. > http://people.redhat.com/~rjones/virt-top >
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
Here you go: https://github.com/libguestfs/nbdkit/commit/2d15e79f65764d9b0c68bea28ed6afbcbcc63467 Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
On Tue, Jun 23, 2020 at 01:21:21AM +0300, Nir Soffer wrote:
> $ python -c 'import tarfile; print(list({"name": m.name, "offset":
> m.offset_data, "size": m.size} for m in tarfile.open("vm.ova")))'
> [{'name': 'disk1.qcow2', 'offset': 512, 'size': 454144}, {'name':
> 'disk2.qcow2', 'offset': 455168, 'size': 454144}]
I didn't know about this library, but it swings my thinking towards
making a Python-based version of the tar plugin. This library is part
of python-libs and present in RHEL >= 7.
...
> Richard suggested to try nbdkit tar plugin, but the plugin is not
> available on RHEL,
> and this adds additional dependency, when we already use qemu-nbd.
I mean, nbdkit (with Python) is already in RHEL >= 7 and isn't going
away because we need it for virt-v2v, so a Python version of the tar
plugin seems like the easiest thing here. I'll see how easy that is.
You wouldn't need to even wait for an upgrade, you could ship the
Python-based tar plugin yourself until it becomes a standard plugin
since it'll only be a few dozen lines of code.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
On Tue, Jun 23, 2020, 03:37 Jakob Bohm wrote:
> Why not use qemu-img convert directly, it doesn't expose the disk
> content to any
> interface except the disk image file(s) created.
>
The context is uploading disks to a remote oVirt setup. You don't have
access to the target image.
> On 2020-06-23 00:21, Nir Soffer wrote:
> > I'm trying to export qcow2 images from ova format using qemu-nbd.
> >
> > I create 2 compressed qcow2 images, with different data:
> >
> > $ qemu-img info disk1.qcow2
> > image: disk1.qcow2
> > file format: qcow2
> > virtual size: 200 MiB (209715200 bytes)
> > disk size: 384 KiB
> > ...
> >
> > $ qemu-img info disk2.qcow2
> > image: disk2.qcow2
> > file format: qcow2
> > virtual size: 200 MiB (209715200 bytes)
> > disk size: 384 KiB
> > ...
> >
> > And packed them in a tar file. This is not a valid ova but good enough
> > for this test:
> >
> > $ tar tvf vm.ova
> > -rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk1.qcow2
> > -rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk2.qcow2
> >
> > To get info about the disks in ova file, we can use:
> >
> > $ python -c 'import tarfile; print(list({"name": m.name, "offset":
> > m.offset_data, "size": m.size} for m in tarfile.open("vm.ova")))'
> > [{'name': 'disk1.qcow2', 'offset': 512, 'size': 454144}, {'name':
> > 'disk2.qcow2', 'offset': 455168, 'size': 454144}]
> >
> > First I tried the obvious:
> >
> > $ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only --offset=512
> vm.ova
> >
> > And it works, but it exposes the qcow2 data. I want to raw data so I
> > can upload the guest
> > data to ovirt, where is may be converted to qcow2 format.
> >
> > $ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
> > {
> > "virtual-size": 209715200,
> > "filename": "nbd+unix://?socket=/tmp/nbd.sock",
> > "format": "qcow2",
> > ...
> > }
> >
> > Looking in qemu manual and qapi/block-core.json, I could construct this
> command:
> >
> > $ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only
> > 'json:{"driver": "qcow2", "file": {"driver": "raw", "offset": 512,
> > "size": 454144, "file": {"driver": "file", "filename": "vm.ova"}}}'
> >
> > And it works:
> >
> > $ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
> > {
> > "virtual-size": 209715200,
> > "filename": "nbd+unix://?socket=/tmp/nbd.sock",
> > "format": "raw"
> > }
> >
> > $ qemu-img map --output json "nbd+unix://?socket=/tmp/nbd.sock"
> > [{ "start": 0, "length": 104857600, "depth": 0, "zero": false, "data":
> > true, "offset": 0},
> > { "start": 104857600, "length": 104857600, "depth": 0, "zero": true,
> > "data": false, "offset": 104857600}]
> >
> > $ qemu-img map --output json disk1.qcow2
> > [{ "start": 0, "length": 104857600, "depth": 0, "zero": false, "data":
> true},
> > { "start": 104857600, "length": 104857600, "depth": 0, "zero": true,
> > "data": false}]
> >
> > $ qemu-img convert -f raw -O raw nbd+unix://?socket=/tmp/nbd.sock
> disk1.raw
> >
> > $ qemu-img info disk1.raw
> > image: disk1.raw
> > file format: raw
> > virtual size: 200 MiB (209715200 bytes)
> > disk size: 100 MiB
> >
> > $ qemu-img compare disk1.raw disk1.qcow2
> > Images are identical.
> >
> > I wonder if this is the best way to stack a qcow2 driver on top of a
> > raw driver exposing
> > a range from a tar file.
> >
> > I found similar example for gluster in:
> > docs/system/device-url-syntax.rst.inc
> >
> > Richard suggested to try nbdkit tar plugin, but the plugin is not
> > available on RHEL,
> > and this adds additional dependency, when we already use qemu-nbd.
> >
> > Nir
> >
> >
>
>
> Enjoy
>
> Jakob
> --
> Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
> Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
> This public discussion message is non-binding and may contain errors.
> WiseMo - Remote Service Management for PCs, Phones and Embedded
>
>
>
Re: Exporting qcow2 images as raw data from ova file with qemu-nbd
Why not use qemu-img convert directly, it doesn't expose the disk
content to any
interface except the disk image file(s) created.
On 2020-06-23 00:21, Nir Soffer wrote:
I'm trying to export qcow2 images from ova format using qemu-nbd.
I create 2 compressed qcow2 images, with different data:
$ qemu-img info disk1.qcow2
image: disk1.qcow2
file format: qcow2
virtual size: 200 MiB (209715200 bytes)
disk size: 384 KiB
...
$ qemu-img info disk2.qcow2
image: disk2.qcow2
file format: qcow2
virtual size: 200 MiB (209715200 bytes)
disk size: 384 KiB
...
And packed them in a tar file. This is not a valid ova but good enough
for this test:
$ tar tvf vm.ova
-rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk1.qcow2
-rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk2.qcow2
To get info about the disks in ova file, we can use:
$ python -c 'import tarfile; print(list({"name": m.name, "offset":
m.offset_data, "size": m.size} for m in tarfile.open("vm.ova")))'
[{'name': 'disk1.qcow2', 'offset': 512, 'size': 454144}, {'name':
'disk2.qcow2', 'offset': 455168, 'size': 454144}]
First I tried the obvious:
$ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only --offset=512 vm.ova
And it works, but it exposes the qcow2 data. I want to raw data so I
can upload the guest
data to ovirt, where is may be converted to qcow2 format.
$ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
{
"virtual-size": 209715200,
"filename": "nbd+unix://?socket=/tmp/nbd.sock",
"format": "qcow2",
...
}
Looking in qemu manual and qapi/block-core.json, I could construct this command:
$ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only
'json:{"driver": "qcow2", "file": {"driver": "raw", "offset": 512,
"size": 454144, "file": {"driver": "file", "filename": "vm.ova"}}}'
And it works:
$ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
{
"virtual-size": 209715200,
"filename": "nbd+unix://?socket=/tmp/nbd.sock",
"format": "raw"
}
$ qemu-img map --output json "nbd+unix://?socket=/tmp/nbd.sock"
[{ "start": 0, "length": 104857600, "depth": 0, "zero": false, "data":
true, "offset": 0},
{ "start": 104857600, "length": 104857600, "depth": 0, "zero": true,
"data": false, "offset": 104857600}]
$ qemu-img map --output json disk1.qcow2
[{ "start": 0, "length": 104857600, "depth": 0, "zero": false, "data": true},
{ "start": 104857600, "length": 104857600, "depth": 0, "zero": true,
"data": false}]
$ qemu-img convert -f raw -O raw nbd+unix://?socket=/tmp/nbd.sock disk1.raw
$ qemu-img info disk1.raw
image: disk1.raw
file format: raw
virtual size: 200 MiB (209715200 bytes)
disk size: 100 MiB
$ qemu-img compare disk1.raw disk1.qcow2
Images are identical.
I wonder if this is the best way to stack a qcow2 driver on top of a
raw driver exposing
a range from a tar file.
I found similar example for gluster in:
docs/system/device-url-syntax.rst.inc
Richard suggested to try nbdkit tar plugin, but the plugin is not
available on RHEL,
and this adds additional dependency, when we already use qemu-nbd.
Nir
Enjoy
Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded
Exporting qcow2 images as raw data from ova file with qemu-nbd
I'm trying to export qcow2 images from ova format using qemu-nbd.
I create 2 compressed qcow2 images, with different data:
$ qemu-img info disk1.qcow2
image: disk1.qcow2
file format: qcow2
virtual size: 200 MiB (209715200 bytes)
disk size: 384 KiB
...
$ qemu-img info disk2.qcow2
image: disk2.qcow2
file format: qcow2
virtual size: 200 MiB (209715200 bytes)
disk size: 384 KiB
...
And packed them in a tar file. This is not a valid ova but good enough
for this test:
$ tar tvf vm.ova
-rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk1.qcow2
-rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk2.qcow2
To get info about the disks in ova file, we can use:
$ python -c 'import tarfile; print(list({"name": m.name, "offset":
m.offset_data, "size": m.size} for m in tarfile.open("vm.ova")))'
[{'name': 'disk1.qcow2', 'offset': 512, 'size': 454144}, {'name':
'disk2.qcow2', 'offset': 455168, 'size': 454144}]
First I tried the obvious:
$ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only --offset=512 vm.ova
And it works, but it exposes the qcow2 data. I want to raw data so I
can upload the guest
data to ovirt, where is may be converted to qcow2 format.
$ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
{
"virtual-size": 209715200,
"filename": "nbd+unix://?socket=/tmp/nbd.sock",
"format": "qcow2",
...
}
Looking in qemu manual and qapi/block-core.json, I could construct this command:
$ qemu-nbd --persistent --socket=/tmp/nbd.sock --read-only
'json:{"driver": "qcow2", "file": {"driver": "raw", "offset": 512,
"size": 454144, "file": {"driver": "file", "filename": "vm.ova"}}}'
And it works:
$ qemu-img info --output json "nbd+unix://?socket=/tmp/nbd.sock"
{
"virtual-size": 209715200,
"filename": "nbd+unix://?socket=/tmp/nbd.sock",
"format": "raw"
}
$ qemu-img map --output json "nbd+unix://?socket=/tmp/nbd.sock"
[{ "start": 0, "length": 104857600, "depth": 0, "zero": false, "data":
true, "offset": 0},
{ "start": 104857600, "length": 104857600, "depth": 0, "zero": true,
"data": false, "offset": 104857600}]
$ qemu-img map --output json disk1.qcow2
[{ "start": 0, "length": 104857600, "depth": 0, "zero": false, "data": true},
{ "start": 104857600, "length": 104857600, "depth": 0, "zero": true,
"data": false}]
$ qemu-img convert -f raw -O raw nbd+unix://?socket=/tmp/nbd.sock disk1.raw
$ qemu-img info disk1.raw
image: disk1.raw
file format: raw
virtual size: 200 MiB (209715200 bytes)
disk size: 100 MiB
$ qemu-img compare disk1.raw disk1.qcow2
Images are identical.
I wonder if this is the best way to stack a qcow2 driver on top of a
raw driver exposing
a range from a tar file.
I found similar example for gluster in:
docs/system/device-url-syntax.rst.inc
Richard suggested to try nbdkit tar plugin, but the plugin is not
available on RHEL,
and this adds additional dependency, when we already use qemu-nbd.
Nir
