Adding some kubevirt developers to the thread. Thanks guys for the information! I think this could work perfectly for on the fly conversion of qcow2 images to raw format on our PVCs.
On Mon, Jul 9, 2018 at 9:47 PM Nir Soffer <nsof...@redhat.com> wrote: > On Mon, Jul 9, 2018 at 8:06 PM Richard W.M. Jones <rjo...@redhat.com> > wrote: > >> On Mon, Jul 09, 2018 at 07:02:50PM +0200, Kevin Wolf wrote: >> > Am 09.07.2018 um 18:52 hat Richard W.M. Jones geschrieben: >> > > On Mon, Jul 09, 2018 at 07:38:05PM +0300, Nir Soffer wrote: >> > > > We are discussing importing VM images to KubVirt. The goal is to be >> > > > able to import an existing qcow2 disk, probably some appliance >> stored >> > > > on http server, and and convert it to raw format for writing to >> storage. >> > > > >> > > > This can be also useful for for oVirt for importing OVA, since we >> like to >> > > > pack >> > > > disks in qcow2 format inside OVA, but the user may like to use raw >> disks, or >> > > > for uploading existing disks. >> > > > >> > > > Of course converting the image using qemu-img is easy, but requires >> > > > downloading the image to temporary disk. We would like to avoid >> temporary >> > > > disks, or telling users to convert the image. >> > > > >> > > > Base on the discussion we had here: >> > > > >> https://lists.ovirt.org/archives/list/us...@ovirt.org/thread/GNAVJ253FP65QUSOONES5XZGRIDX5ABC/#YMLSEGU7PN3MX5MUORGEGGAQLLSL4KKJ >> > > > >> > > > I think this is impossible since qcow2 is not built for streaming. >> But both >> > > > Richard and Eric suggested some solutions. >> > > > >> > > > The flow is: >> > > > >> > > > qcow2 image -- http --> importer -> raw file >> > > > >> > > > Is it possible to implement the importer using qemu-img and >> qemu-nbd, >> > > > or maybe nbdkit? >> > > >> > > Strictly speaking streaming qcow2 to raw is not possible. However >> > > placing an overlay on top of the original remote image will allow >> > > streaming to raw with only a modest amount of local storage consumed. >> > > >> > > You can demonstrate this fairly easily: >> > > >> > > $ qemu-img create -f qcow2 -b 'json: { "file.driver": "https", >> "file.url": " >> https://uk-mirrors.evowise.com/fedora/releases/28/Cloud/x86_64/images/Fedora-Cloud-Base-28-1.1.x86_64.qcow2", >> "file.timeout": 10000 }' /var/tmp/overlay.qcow2 >> > > $ qemu-img convert -p -f qcow2 -O raw overlay.qcow2 fedora.img >> > >> > This overlay stays empty, so it's pretty pointless and you could just >> > directly point 'qemu-img convert' to https and the real image. >> >> Right, indeed. I was copying what virt-v2v does without thinking >> about it enough. virt-v2v needs the overlay because it actually wants >> to write into it, and it does copy-on-read for the first phase (not >> the final ‘qemu-img convert’). >> > > Thanks, I just tested the simple: > > qemu-img convert -p -f qcow2 -O raw http://localhost/orig.qcow2 > converted.raw > > And it just works :-) > > I got timeouts trying to download from > > https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-28-20180625.1/AtomicHost/x86_64/images/Fedora-AtomicHost-28-20180625.1.x86_64.qcow2 > > I guess we need to use > 'json: { "file.driver": "http", "file.url": "url...", "file.timeout": > 10000 }' > To change timeout? Where is these and other options documented? > > I did also some timings, using sever on local network with 1g nic. > > $ time wget > http://local.server/Fedora-AtomicHost-28-20180625.1.x86_64.qcow2 > ... > Length: 638043136 (608M) [application/octet-stream]Saving to: > ‘Fedora-AtomicHost-28-20180625.1.x86_64.qcow2’ > > Fedora-AtomicHost-28-20180625.1.x86_64.qcow2 > 100%[=====================================================================================================>] > 608.49M 107MB/s in 5.6s > > 2018-07-09 21:38:39 (108 MB/s) - > ‘Fedora-AtomicHost-28-20180625.1.x86_64.qcow2’ saved [638043136/638043136] > > > real 0m5.941s > user 0m0.183s > sys 0m1.185s > > $ time qemu-img convert -p -f qcow2 -O raw > http://local.server/Fedora-AtomicHost-28-20180625.1.x86_64.qcow2 > converted.raw > (100.00/100%) > > real 0m14.217s > user 0m5.235s > sys 0m2.343s > > $ time qemu-img convert -p -f qcow2 -O raw > Fedora-AtomicHost-28-20180625.1.x86_64.qcow2 converted.raw > (100.00/100%) > > real 0m11.909s > user 0m4.728s > sys 0m1.595s > > So converting on the fly is even little faster then downloading to > temporary file and converting. > > Nir > -- Adam Litke