Re: [PATCH 0/3] qemu-img: Add convert --bitmaps
(adding Markus for a CLI question, look for [*])
On 4/16/20 1:20 PM, Nir Soffer wrote:
On Thu, Apr 16, 2020 at 5:51 PM Eric Blake wrote:
Without this series, the process for copying one qcow2 image to
another including all of its bitmaps involves running qemu and doing
the copying by hand with a series of QMP commands. This makes the
process a bit more convenient.
This seems good for copying an image chain from one storage to another,
but I think we need a similar --bitmaps option to qemu-img measure to make
this really useful.
Here is example use case showing how qemu-img measure is related:
Source chain:
/dev/vg1/base
/dev/vg1/top
Destination chain:
/dev/vg2/base
/dev/vg2/top
We create empty lvs with the same name on destination storage (/dev/vg2).
We measure the base lv using qemu-img measure for creating the target lv:
qemu-img measure -f qcow2 -O qcow2 /dev/vg1/base
lvcreate -L required_size /dev/vg2/base
qemu-img create -f qcow2 /dev/vg2/base 10g
For the top lv we use the current size of the source lv - I think we
should measure it instead but
I'm not sure if qemu-img measure supports measuring a single image in a chain
(maybe -o backing_file?).
qemu-measure --image-opts should be able to measure a single image by
specifying image opts that purposefully treat the image as standalone
rather than with its normal backing file included. Let's see if I can
whip up an example:
$ qemu-img create -f qcow2 img.base 100M
Formatting 'img.base', fmt=qcow2 size=104857600 cluster_size=65536
lazy_refcounts=off refcount_bits=16
$ qemu-io -f qcow2 -c 'w 0 25m' img.base
wrote 26214400/26214400 bytes at offset 0
25 MiB, 1 ops; 00.24 sec (103.405 MiB/sec and 4.1362 ops/sec)
$ qemu-img create -f qcow2 -F qcow2 -b img.base img.top
Formatting 'img.top', fmt=qcow2 size=104857600 backing_file=img.base
backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
$ qemu-io -f qcow2 -c 'w 25m 25m' img.top
wrote 26214400/26214400 bytes at offset 26214400
25 MiB, 1 ops; 00.24 sec (103.116 MiB/sec and 4.1247 ops/sec)
$ qemu-img measure -f qcow2 -O qcow2 img.base
required size: 26542080
fully allocated size: 105185280
required size: 52756480
fully allocated size: 105185280
Okay, I can reproduce what you are seeing - measuring the top image
defaults to measuring the full allocation of the entire chain, rather
than the allocation of just the top image. And now with --image-opts to
the rescue:
$ qemu-img measure --image-opts -O qcow2
driver=qcow2,backing=,file.driver=file,file.filename=img.top
qemu-img: warning: Use of "backing": "" is deprecated; use "backing":
null instead
required size: 26542080
fully allocated size: 105185280
There you go - by forcing qemu to treat the overlay image as though it
had no backing, you can then measure that image in isolation.
(*) Hmm - that warning about backing="" being deprecated is annoying,
but I don't know any other way to use dotted command line syntax and
still express that we want a QMP null. I tried to see if I could inject
an alternative backing driver, such as null-co, but was met with errors:
$ ./qemu-img measure --image-opts -O qcow2
driver=qcow2,backing.driver=null-co,file.driver=file,file.filename=img.top
qemu-img: Could not open
'driver=qcow2,backing.driver=null-co,file.driver=file,file.filename=img.top':
Could not open backing file: The only allowed filename for this driver
is 'null-co://'
$ ./qemu-img measure --image-opts -O qcow2
driver=qcow2,backing.driver=null-co,backing.file=null-co://,file.driver=file,file.filename=img.top
qemu-img: Could not open
'driver=qcow2,backing.driver=null-co,backing.file=null-co://,file.driver=file,file.filename=img.top':
Could not open backing file: The only allowed filename for this driver
is 'null-co://'
$ ./qemu-img measure --image-opts -O qcow2
driver=qcow2,backing.driver=null-co,backing.file.filename=null-co://,file.driver=file,file.filename=img.top
qemu-img: Could not open
'driver=qcow2,backing.driver=null-co,backing.file.filename=null-co://,file.driver=file,file.filename=img.top':
Could not open backing file: Block protocol 'null-co' doesn't support
the option 'file.filename'
We don't want to support "" in the QMP syntax forever, but if the CLI
syntax has to handle the empty string specially in order to get null
passed to the QMP code, then so be it.
I also tried, but failed, to use JSON syntax. I don't know why we
haven't wired up --image-opts to use JSON syntax yet.
$ qemu-img measure --image-opts -O qcow2 '{"driver":"qcow2", "backing":null,
"file":{"driver":"file", "filename":"img.top"}}'
qemu-img: Could not open '{"driver":"qcow2", "backing":null,
"file":{"driver":"file", "filename":"img.top"}}': Cannot find
device={"driver":"qcow2" nor node_name={"driver":"qcow2"
I guess there's always the pseudo-json protocol:
$ qemu-img measure -O qcow2 'json:{"driver":"qcow2", "backing":null,
"file":{"driver":"file", "filename":"img.top"}}'
require
Re: [PATCH 0/3] qemu-img: Add convert --bitmaps
On Thu, Apr 16, 2020 at 5:51 PM Eric Blake wrote: > > Without this series, the process for copying one qcow2 image to > another including all of its bitmaps involves running qemu and doing > the copying by hand with a series of QMP commands. This makes the > process a bit more convenient. This seems good for copying an image chain from one storage to another, but I think we need a similar --bitmaps option to qemu-img measure to make this really useful. Here is example use case showing how qemu-img measure is related: Source chain: /dev/vg1/base /dev/vg1/top Destination chain: /dev/vg2/base /dev/vg2/top We create empty lvs with the same name on destination storage (/dev/vg2). We measure the base lv using qemu-img measure for creating the target lv: qemu-img measure -f qcow2 -O qcow2 /dev/vg1/base lvcreate -L required_size /dev/vg2/base qemu-img create -f qcow2 /dev/vg2/base 10g For the top lv we use the current size of the source lv - I think we should measure it instead but I'm not sure if qemu-img measure supports measuring a single image in a chain (maybe -o backing_file?). lvcreate -L current_size /dev/vg2/top qemu-img create -f qcow2 -b /dev/vg2/base -F qcow2 /dev/vg2/top 10g And then convert the lvs one by one: qemu-img convert -f qcow2 -O qcow2 -n --bitmaps /dev/vg1/base /dev/vg2/base qemu-img convert -f qcow2 -O qcow2 -n --bitmaps -B /dev/vg2/base /dev/vg1/top /dev/vg2/top The first copy may fail with ENOSPC since qemu-img measure of the base does not consider the bitmaps in the required size. So I think we need to add a similar --bitmaps option to qemu-img measure, hopefully reusing the same code to find and estimate the size of the bitmaps. Maybe we can estimate the size using qemu-img info --bitmaps, but I think the right way to do this is in qemu-img measure. We have also another use case when we collapsed an image chain to single image: Source chain: /dev/vg1/base /dev/vg1/top Destination: /dev/vg2/collapsed In this case we measure the size of the entire chain (/dev/vg1/base <- /dev/vg1/top) and create /dev/vg2/collapsed in the correct size, and then we convert the chain using: qemu-img convert /dev/vg1/top /dev/vg2/collapsed Currently we use this for exporting images, for example when creating templates, or as a simple backup. In this case we don't need to copy the bitmaps in the target image - this is a new image not used by any VM. Copying the bitmaps may also be non-trivial since we may have the bitmaps with the same names in several layers (e.g. result of live snapshot). So I think using --bitmaps should be disabled when doing this kind of convert. We can handle this on our side easily, but I think this should fail or log a warning on qemu-img, or require merging of bitmaps with same names during the copy. I did not check if you already handle this. Finally we also have a use case when we copy the chain as is to new or same storage, but we create a new vm. In this case I don't think the backup history makes sense for the new vm, so we don't need to copy the bitmaps. I will review the rest of the patches next week and can maybe give this some testing. Nir > I still think that someday we will need a 'qemu-img bitmap' with > various subcommands for manipulating bitmaps within an offline image, > but in the meantime, this seems like a useful addition on its own. > > Series can also be downloaded at: > https://repo.or.cz/qemu/ericb.git/shortlog/refs/tags/qemu-img-bitmaps-v1 > > Eric Blake (3): > blockdev: Split off basic bitmap operations for qemu-img > qemu-img: Add convert --bitmaps option > iotests: Add test 291 to for qemu-img convert --bitmaps > > docs/tools/qemu-img.rst| 6 +- > Makefile.objs | 2 +- > include/sysemu/blockdev.h | 10 ++ > blockbitmaps.c | 217 + > blockdev.c | 184 --- > qemu-img.c | 81 +- > MAINTAINERS| 1 + > qemu-img-cmds.hx | 4 +- > tests/qemu-iotests/291 | 143 > tests/qemu-iotests/291.out | 56 ++ > tests/qemu-iotests/group | 1 + > 11 files changed, 514 insertions(+), 191 deletions(-) > create mode 100644 blockbitmaps.c > create mode 100755 tests/qemu-iotests/291 > create mode 100644 tests/qemu-iotests/291.out > > -- > 2.26.0 >
[PATCH 0/3] qemu-img: Add convert --bitmaps
Without this series, the process for copying one qcow2 image to another including all of its bitmaps involves running qemu and doing the copying by hand with a series of QMP commands. This makes the process a bit more convenient. I still think that someday we will need a 'qemu-img bitmap' with various subcommands for manipulating bitmaps within an offline image, but in the meantime, this seems like a useful addition on its own. Series can also be downloaded at: https://repo.or.cz/qemu/ericb.git/shortlog/refs/tags/qemu-img-bitmaps-v1 Eric Blake (3): blockdev: Split off basic bitmap operations for qemu-img qemu-img: Add convert --bitmaps option iotests: Add test 291 to for qemu-img convert --bitmaps docs/tools/qemu-img.rst| 6 +- Makefile.objs | 2 +- include/sysemu/blockdev.h | 10 ++ blockbitmaps.c | 217 + blockdev.c | 184 --- qemu-img.c | 81 +- MAINTAINERS| 1 + qemu-img-cmds.hx | 4 +- tests/qemu-iotests/291 | 143 tests/qemu-iotests/291.out | 56 ++ tests/qemu-iotests/group | 1 + 11 files changed, 514 insertions(+), 191 deletions(-) create mode 100644 blockbitmaps.c create mode 100755 tests/qemu-iotests/291 create mode 100644 tests/qemu-iotests/291.out -- 2.26.0
