Re: [Qemu-devel] [PATCH v3 0/8] qemu-img: add measure sub-command
On Mon, Mar 27, 2017 at 06:02:57PM +0300, Maor Lipchuk wrote:
> On Thu, Mar 23, 2017 at 8:01 PM, Stefan Hajnoczi wrote:
> > On Wed, Mar 22, 2017 at 07:29:07PM +, Nir Soffer wrote:
> >> On Wed, Mar 22, 2017 at 2:28 PM Stefan Hajnoczi
> wrote:
> >>
> >> > v3:
> >> > * Drop RFC, this is ready to go for QEMU 2.10
> >> > * Use "required size" instead of "required bytes" in qemu-img output
> for
> >> >consistency [Nir]
> >> > * Clarify BlockMeasureInfo semantics [Max]
> >> > * Clarify bdrv_measure() opts argument and error handling [Nir]
> >> > * Handle -o backing_file= for qcow2 [Max]
> >> > * Handle snapshot options in qemu-img measure
> >> > * Probe input image for allocated data clusters for qcow2. Didn't
> >> > centralize
> >> >this because there are format-specific aspects such as the
> >> > cluster_size. It
> >> >may make sense to centralize it later (with a bit more complexity)
> if
> >> >support is added to more formats.
> >> > * Add qemu-img(1) man page section for 'measure' sub-command [Max]
> >> > * Extend test case to cover additional scenarios [Nir]
> >> >
> >> > RFCv2:
> >> > * Publishing RFC again to discuss the new user-visible interfaces.
> Code
> >> > has
> >> >changed quite a bit, I have not kept any Reviewed-by tags.
> >> > * Rename qemu-img sub-command "measure" and API bdrv_measure() [Nir]
> >> > * Report both "required bytes" and "fully allocated bytes" to handle
> the
> >> > empty
> >> >image file and prealloc use cases [Nir and Dan]
> >> > * Use bdrv_getlength() instead of bdrv_nb_sectors() [Berto]
> >> > * Rename "err" label "out" in qemu-img-cmds.c [Nir]
> >> > * Add basic qcow2 support, doesn't support qemu-img convert from
> existing
> >> > files yet
> >> >
> >> > RFCv1:
> >> > * Publishing patch series with just raw support, no qcow2 yet. Please
> >> > review
> >> >the command-line interface and let me know if you are happy with
> this
> >> >approach.
> >> >
> >> > Users and management tools sometimes need to know the size required
> for a
> >> > new
> >> > disk image so that an LVM volume, SAN LUN, etc can be allocated ahead
> of
> >> > time.
> >> > Image formats like qcow2 have non-trivial metadata that makes it hard
> to
> >> > estimate the exact size without knowledge of file format internals.
> >> >
> >> > This patch series introduces a new qemu-img sub-command that
> calculates the
> >> > required size for both image creation and conversion scenarios.
> >> >
> >> > The conversion scenario is:
> >> >
> >> > $ qemu-img measure -f raw -O qcow2 input.img
> >> > required size: 1327680
> >> > fully allocated size: 1074069504
> >> >
> >> > Here an existing image file is taken and the output includes the space
> >> > required
> >> > for data from the input image file.
> >> >
> >> > The creation scenario is:
> >> >
> >> > $ qemu-img measure -O qcow2 --size 5G
> >> > required size: 327680
> >> > fully allocated size: 1074069504
> >> >
> >> > Stefan Hajnoczi (8):
> >> > block: add bdrv_measure() API
> >> > raw-format: add bdrv_measure() support
> >> > qcow2: extract preallocation calculation function
> >> > qcow2: extract image creation option parsing
> >> > qcow2: add bdrv_measure() support
> >> > qemu-img: add measure subcommand
> >> > qemu-iotests: support per-format golden output files
> >> > iotests: add test 178 for qemu-img measure
> >> >
> >> > qapi/block-core.json | 25 +++
> >> > include/block/block.h| 4 +
> >> > include/block/block_int.h| 2 +
> >> > block.c | 35
> >> > block/qcow2.c| 362
> >> > +--
> >> > block/raw-format.c | 22 +++
> >> > qemu-img.c | 227
> >> > qemu-img-cmds.hx | 6 +
> >> > qemu-img.texi| 25 +++
> >> > tests/qemu-iotests/178 | 144
> >> > tests/qemu-iotests/178.out.qcow2 | 242 ++
> >> > tests/qemu-iotests/178.out.raw | 130 ++
> >> > tests/qemu-iotests/check | 5 +
> >> > tests/qemu-iotests/group | 1 +
> >> > 14 files changed, 1136 insertions(+), 94 deletions(-)
> >> > create mode 100755 tests/qemu-iotests/178
> >> > create mode 100644 tests/qemu-iotests/178.out.qcow2
> >> > create mode 100644 tests/qemu-iotests/178.out.raw
> >> >
> >> > --
> >> > 2.9.3
> >> >
> >>
> >> Stefan, is this available in some public git repo?
> >>
> >> I would like to test it, and downloading and applying 8 patches is lot
> of
> >> work.
> >> (I'm probably not using the right tools for this).
> >
> > I've pushed the branch here:
> > https://github.com/stefanha/qemu/tree/qemu-img-query-max-size
> >
> > In mutt you can tag multiple emails ('t') and then pipe them to git am
> > in one go (';|git am'). Many other email clients offer no support for
> > processing patches by default
Re: [Qemu-devel] [PATCH v3 0/8] qemu-img: add measure sub-command
On Thu, Mar 23, 2017 at 8:01 PM, Stefan Hajnoczi wrote:
> On Wed, Mar 22, 2017 at 07:29:07PM +, Nir Soffer wrote:
>> On Wed, Mar 22, 2017 at 2:28 PM Stefan Hajnoczi
wrote:
>>
>> > v3:
>> > * Drop RFC, this is ready to go for QEMU 2.10
>> > * Use "required size" instead of "required bytes" in qemu-img output
for
>> >consistency [Nir]
>> > * Clarify BlockMeasureInfo semantics [Max]
>> > * Clarify bdrv_measure() opts argument and error handling [Nir]
>> > * Handle -o backing_file= for qcow2 [Max]
>> > * Handle snapshot options in qemu-img measure
>> > * Probe input image for allocated data clusters for qcow2. Didn't
>> > centralize
>> >this because there are format-specific aspects such as the
>> > cluster_size. It
>> >may make sense to centralize it later (with a bit more complexity)
if
>> >support is added to more formats.
>> > * Add qemu-img(1) man page section for 'measure' sub-command [Max]
>> > * Extend test case to cover additional scenarios [Nir]
>> >
>> > RFCv2:
>> > * Publishing RFC again to discuss the new user-visible interfaces.
Code
>> > has
>> >changed quite a bit, I have not kept any Reviewed-by tags.
>> > * Rename qemu-img sub-command "measure" and API bdrv_measure() [Nir]
>> > * Report both "required bytes" and "fully allocated bytes" to handle
the
>> > empty
>> >image file and prealloc use cases [Nir and Dan]
>> > * Use bdrv_getlength() instead of bdrv_nb_sectors() [Berto]
>> > * Rename "err" label "out" in qemu-img-cmds.c [Nir]
>> > * Add basic qcow2 support, doesn't support qemu-img convert from
existing
>> > files yet
>> >
>> > RFCv1:
>> > * Publishing patch series with just raw support, no qcow2 yet. Please
>> > review
>> >the command-line interface and let me know if you are happy with
this
>> >approach.
>> >
>> > Users and management tools sometimes need to know the size required
for a
>> > new
>> > disk image so that an LVM volume, SAN LUN, etc can be allocated ahead
of
>> > time.
>> > Image formats like qcow2 have non-trivial metadata that makes it hard
to
>> > estimate the exact size without knowledge of file format internals.
>> >
>> > This patch series introduces a new qemu-img sub-command that
calculates the
>> > required size for both image creation and conversion scenarios.
>> >
>> > The conversion scenario is:
>> >
>> > $ qemu-img measure -f raw -O qcow2 input.img
>> > required size: 1327680
>> > fully allocated size: 1074069504
>> >
>> > Here an existing image file is taken and the output includes the space
>> > required
>> > for data from the input image file.
>> >
>> > The creation scenario is:
>> >
>> > $ qemu-img measure -O qcow2 --size 5G
>> > required size: 327680
>> > fully allocated size: 1074069504
>> >
>> > Stefan Hajnoczi (8):
>> > block: add bdrv_measure() API
>> > raw-format: add bdrv_measure() support
>> > qcow2: extract preallocation calculation function
>> > qcow2: extract image creation option parsing
>> > qcow2: add bdrv_measure() support
>> > qemu-img: add measure subcommand
>> > qemu-iotests: support per-format golden output files
>> > iotests: add test 178 for qemu-img measure
>> >
>> > qapi/block-core.json | 25 +++
>> > include/block/block.h| 4 +
>> > include/block/block_int.h| 2 +
>> > block.c | 35
>> > block/qcow2.c| 362
>> > +--
>> > block/raw-format.c | 22 +++
>> > qemu-img.c | 227
>> > qemu-img-cmds.hx | 6 +
>> > qemu-img.texi| 25 +++
>> > tests/qemu-iotests/178 | 144
>> > tests/qemu-iotests/178.out.qcow2 | 242 ++
>> > tests/qemu-iotests/178.out.raw | 130 ++
>> > tests/qemu-iotests/check | 5 +
>> > tests/qemu-iotests/group | 1 +
>> > 14 files changed, 1136 insertions(+), 94 deletions(-)
>> > create mode 100755 tests/qemu-iotests/178
>> > create mode 100644 tests/qemu-iotests/178.out.qcow2
>> > create mode 100644 tests/qemu-iotests/178.out.raw
>> >
>> > --
>> > 2.9.3
>> >
>>
>> Stefan, is this available in some public git repo?
>>
>> I would like to test it, and downloading and applying 8 patches is lot
of
>> work.
>> (I'm probably not using the right tools for this).
>
> I've pushed the branch here:
> https://github.com/stefanha/qemu/tree/qemu-img-query-max-size
>
> In mutt you can tag multiple emails ('t') and then pipe them to git am
> in one go (';|git am'). Many other email clients offer no support for
> processing patches by default.
>
> Stefan
Hi Stefan,
Thank you for the branch, it was much helpful to verify and test the code
that way :)
I ran a test and created several files using python.
Some of the files that were converted to qcow2 seems to have larger size
than the measure size.
In the following example the converted file had
Re: [Qemu-devel] [PATCH v3 0/8] qemu-img: add measure sub-command
On Wed, Mar 22, 2017 at 07:29:07PM +, Nir Soffer wrote:
> On Wed, Mar 22, 2017 at 2:28 PM Stefan Hajnoczi wrote:
>
> > v3:
> > * Drop RFC, this is ready to go for QEMU 2.10
> > * Use "required size" instead of "required bytes" in qemu-img output for
> >consistency [Nir]
> > * Clarify BlockMeasureInfo semantics [Max]
> > * Clarify bdrv_measure() opts argument and error handling [Nir]
> > * Handle -o backing_file= for qcow2 [Max]
> > * Handle snapshot options in qemu-img measure
> > * Probe input image for allocated data clusters for qcow2. Didn't
> > centralize
> >this because there are format-specific aspects such as the
> > cluster_size. It
> >may make sense to centralize it later (with a bit more complexity) if
> >support is added to more formats.
> > * Add qemu-img(1) man page section for 'measure' sub-command [Max]
> > * Extend test case to cover additional scenarios [Nir]
> >
> > RFCv2:
> > * Publishing RFC again to discuss the new user-visible interfaces. Code
> > has
> >changed quite a bit, I have not kept any Reviewed-by tags.
> > * Rename qemu-img sub-command "measure" and API bdrv_measure() [Nir]
> > * Report both "required bytes" and "fully allocated bytes" to handle the
> > empty
> >image file and prealloc use cases [Nir and Dan]
> > * Use bdrv_getlength() instead of bdrv_nb_sectors() [Berto]
> > * Rename "err" label "out" in qemu-img-cmds.c [Nir]
> > * Add basic qcow2 support, doesn't support qemu-img convert from existing
> > files yet
> >
> > RFCv1:
> > * Publishing patch series with just raw support, no qcow2 yet. Please
> > review
> >the command-line interface and let me know if you are happy with this
> >approach.
> >
> > Users and management tools sometimes need to know the size required for a
> > new
> > disk image so that an LVM volume, SAN LUN, etc can be allocated ahead of
> > time.
> > Image formats like qcow2 have non-trivial metadata that makes it hard to
> > estimate the exact size without knowledge of file format internals.
> >
> > This patch series introduces a new qemu-img sub-command that calculates the
> > required size for both image creation and conversion scenarios.
> >
> > The conversion scenario is:
> >
> > $ qemu-img measure -f raw -O qcow2 input.img
> > required size: 1327680
> > fully allocated size: 1074069504
> >
> > Here an existing image file is taken and the output includes the space
> > required
> > for data from the input image file.
> >
> > The creation scenario is:
> >
> > $ qemu-img measure -O qcow2 --size 5G
> > required size: 327680
> > fully allocated size: 1074069504
> >
> > Stefan Hajnoczi (8):
> > block: add bdrv_measure() API
> > raw-format: add bdrv_measure() support
> > qcow2: extract preallocation calculation function
> > qcow2: extract image creation option parsing
> > qcow2: add bdrv_measure() support
> > qemu-img: add measure subcommand
> > qemu-iotests: support per-format golden output files
> > iotests: add test 178 for qemu-img measure
> >
> > qapi/block-core.json | 25 +++
> > include/block/block.h| 4 +
> > include/block/block_int.h| 2 +
> > block.c | 35
> > block/qcow2.c| 362
> > +--
> > block/raw-format.c | 22 +++
> > qemu-img.c | 227
> > qemu-img-cmds.hx | 6 +
> > qemu-img.texi| 25 +++
> > tests/qemu-iotests/178 | 144
> > tests/qemu-iotests/178.out.qcow2 | 242 ++
> > tests/qemu-iotests/178.out.raw | 130 ++
> > tests/qemu-iotests/check | 5 +
> > tests/qemu-iotests/group | 1 +
> > 14 files changed, 1136 insertions(+), 94 deletions(-)
> > create mode 100755 tests/qemu-iotests/178
> > create mode 100644 tests/qemu-iotests/178.out.qcow2
> > create mode 100644 tests/qemu-iotests/178.out.raw
> >
> > --
> > 2.9.3
> >
>
> Stefan, is this available in some public git repo?
>
> I would like to test it, and downloading and applying 8 patches is lot of
> work.
> (I'm probably not using the right tools for this).
I've pushed the branch here:
https://github.com/stefanha/qemu/tree/qemu-img-query-max-size
In mutt you can tag multiple emails ('t') and then pipe them to git am
in one go (';|git am'). Many other email clients offer no support for
processing patches by default.
Stefan
signature.asc
Description: PGP signature
Re: [Qemu-devel] [PATCH v3 0/8] qemu-img: add measure sub-command
On Wed, Mar 22, 2017 at 2:28 PM Stefan Hajnoczi wrote: > v3: > * Drop RFC, this is ready to go for QEMU 2.10 > * Use "required size" instead of "required bytes" in qemu-img output for >consistency [Nir] > * Clarify BlockMeasureInfo semantics [Max] > * Clarify bdrv_measure() opts argument and error handling [Nir] > * Handle -o backing_file= for qcow2 [Max] > * Handle snapshot options in qemu-img measure > * Probe input image for allocated data clusters for qcow2. Didn't > centralize >this because there are format-specific aspects such as the > cluster_size. It >may make sense to centralize it later (with a bit more complexity) if >support is added to more formats. > * Add qemu-img(1) man page section for 'measure' sub-command [Max] > * Extend test case to cover additional scenarios [Nir] > > RFCv2: > * Publishing RFC again to discuss the new user-visible interfaces. Code > has >changed quite a bit, I have not kept any Reviewed-by tags. > * Rename qemu-img sub-command "measure" and API bdrv_measure() [Nir] > * Report both "required bytes" and "fully allocated bytes" to handle the > empty >image file and prealloc use cases [Nir and Dan] > * Use bdrv_getlength() instead of bdrv_nb_sectors() [Berto] > * Rename "err" label "out" in qemu-img-cmds.c [Nir] > * Add basic qcow2 support, doesn't support qemu-img convert from existing > files yet > > RFCv1: > * Publishing patch series with just raw support, no qcow2 yet. Please > review >the command-line interface and let me know if you are happy with this >approach. > > Users and management tools sometimes need to know the size required for a > new > disk image so that an LVM volume, SAN LUN, etc can be allocated ahead of > time. > Image formats like qcow2 have non-trivial metadata that makes it hard to > estimate the exact size without knowledge of file format internals. > > This patch series introduces a new qemu-img sub-command that calculates the > required size for both image creation and conversion scenarios. > > The conversion scenario is: > > $ qemu-img measure -f raw -O qcow2 input.img > required size: 1327680 > fully allocated size: 1074069504 > > Here an existing image file is taken and the output includes the space > required > for data from the input image file. > > The creation scenario is: > > $ qemu-img measure -O qcow2 --size 5G > required size: 327680 > fully allocated size: 1074069504 > > Stefan Hajnoczi (8): > block: add bdrv_measure() API > raw-format: add bdrv_measure() support > qcow2: extract preallocation calculation function > qcow2: extract image creation option parsing > qcow2: add bdrv_measure() support > qemu-img: add measure subcommand > qemu-iotests: support per-format golden output files > iotests: add test 178 for qemu-img measure > > qapi/block-core.json | 25 +++ > include/block/block.h| 4 + > include/block/block_int.h| 2 + > block.c | 35 > block/qcow2.c| 362 > +-- > block/raw-format.c | 22 +++ > qemu-img.c | 227 > qemu-img-cmds.hx | 6 + > qemu-img.texi| 25 +++ > tests/qemu-iotests/178 | 144 > tests/qemu-iotests/178.out.qcow2 | 242 ++ > tests/qemu-iotests/178.out.raw | 130 ++ > tests/qemu-iotests/check | 5 + > tests/qemu-iotests/group | 1 + > 14 files changed, 1136 insertions(+), 94 deletions(-) > create mode 100755 tests/qemu-iotests/178 > create mode 100644 tests/qemu-iotests/178.out.qcow2 > create mode 100644 tests/qemu-iotests/178.out.raw > > -- > 2.9.3 > Stefan, is this available in some public git repo? I would like to test it, and downloading and applying 8 patches is lot of work. (I'm probably not using the right tools for this). Nir
[Qemu-devel] [PATCH v3 0/8] qemu-img: add measure sub-command
v3: * Drop RFC, this is ready to go for QEMU 2.10 * Use "required size" instead of "required bytes" in qemu-img output for consistency [Nir] * Clarify BlockMeasureInfo semantics [Max] * Clarify bdrv_measure() opts argument and error handling [Nir] * Handle -o backing_file= for qcow2 [Max] * Handle snapshot options in qemu-img measure * Probe input image for allocated data clusters for qcow2. Didn't centralize this because there are format-specific aspects such as the cluster_size. It may make sense to centralize it later (with a bit more complexity) if support is added to more formats. * Add qemu-img(1) man page section for 'measure' sub-command [Max] * Extend test case to cover additional scenarios [Nir] RFCv2: * Publishing RFC again to discuss the new user-visible interfaces. Code has changed quite a bit, I have not kept any Reviewed-by tags. * Rename qemu-img sub-command "measure" and API bdrv_measure() [Nir] * Report both "required bytes" and "fully allocated bytes" to handle the empty image file and prealloc use cases [Nir and Dan] * Use bdrv_getlength() instead of bdrv_nb_sectors() [Berto] * Rename "err" label "out" in qemu-img-cmds.c [Nir] * Add basic qcow2 support, doesn't support qemu-img convert from existing files yet RFCv1: * Publishing patch series with just raw support, no qcow2 yet. Please review the command-line interface and let me know if you are happy with this approach. Users and management tools sometimes need to know the size required for a new disk image so that an LVM volume, SAN LUN, etc can be allocated ahead of time. Image formats like qcow2 have non-trivial metadata that makes it hard to estimate the exact size without knowledge of file format internals. This patch series introduces a new qemu-img sub-command that calculates the required size for both image creation and conversion scenarios. The conversion scenario is: $ qemu-img measure -f raw -O qcow2 input.img required size: 1327680 fully allocated size: 1074069504 Here an existing image file is taken and the output includes the space required for data from the input image file. The creation scenario is: $ qemu-img measure -O qcow2 --size 5G required size: 327680 fully allocated size: 1074069504 Stefan Hajnoczi (8): block: add bdrv_measure() API raw-format: add bdrv_measure() support qcow2: extract preallocation calculation function qcow2: extract image creation option parsing qcow2: add bdrv_measure() support qemu-img: add measure subcommand qemu-iotests: support per-format golden output files iotests: add test 178 for qemu-img measure qapi/block-core.json | 25 +++ include/block/block.h| 4 + include/block/block_int.h| 2 + block.c | 35 block/qcow2.c| 362 +-- block/raw-format.c | 22 +++ qemu-img.c | 227 qemu-img-cmds.hx | 6 + qemu-img.texi| 25 +++ tests/qemu-iotests/178 | 144 tests/qemu-iotests/178.out.qcow2 | 242 ++ tests/qemu-iotests/178.out.raw | 130 ++ tests/qemu-iotests/check | 5 + tests/qemu-iotests/group | 1 + 14 files changed, 1136 insertions(+), 94 deletions(-) create mode 100755 tests/qemu-iotests/178 create mode 100644 tests/qemu-iotests/178.out.qcow2 create mode 100644 tests/qemu-iotests/178.out.raw -- 2.9.3
