Re: [Qemu-devel] [PATCH v10] spec: add qcow2 bitmaps extension specification

[Fam Zheng]

On Fri, 02/05 11:58, Vladimir Sementsov-Ogievskiy wrote:
> The new feature for qcow2: storing bitmaps.
> 
> This patch adds new header extension to qcow2 - Bitmaps Extension. It
> provides an ability to store virtual disk related bitmaps in a qcow2
> image. For now there is only one type of such bitmaps: Dirty Tracking
> Bitmap, which just tracks virtual disk changes from some moment.
> 
> Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
> should be stored in this qcow2 file. The size of each bitmap
> (considering its granularity) is equal to virtual disk size.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 
> ---
> 
> v10
> - rewordings, thanks to Fam
> - fix:
> Given an offset byte_nr into the virtual disk and the bitmap's granularity, 
> the
> -bit offset into the bitmap can be calculated like this:
> +bit offset into the image file to the corresponding bit of the bitmap can be
> +calculated like this:
> - remove last sentence about 'auto' flasg consistency as superfluous.
> 
> 
> v9
> - rewordings, thanks to Max
> 
> v8
> - rewordings
> - bitmap_directory_size: 4b -> 8b
> - add more descriptive description in == Bitmaps == section
> - add paragraph "Dirty tracking bitmaps"
> 
>   Bitmap directory entry:
> - extra data should not allocate additional clusters
> - padding must be all-bytes-zero
> - add extra_data_compatible flag (now behavior in case of unknown
>   extra data is defined by this flag)
> 
> v7:
> 
> - Rewordings, grammar.
>   Max, Eric, John, thank you very much.
> 
> - add last paragraph: remaining bits in bitmap data clusters must be
>   zero.
> 
> - s/Bitmap Directory/bitmap directory/ and other names like this at
>   the request of Max.
> 
> v6:
> 
> - reword bitmap_directory_size description
> - bitmap type: make 0 reserved
> - extra_data_size: resize to 4bytes
>   Also, I've marked this field as "must be zero". We can always change
>   it, if we decide allowing managing app to specify any extra data, by
>   defining some magic value as a top of user extra data.. So, for now
>   non zeor extra_data_size should be considered as an error.
> - swap name and extra_data to give good alignment to extra_data.
> 
> 
> v5:
> 
> - 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of
>   bitmaps.
> - rewordings
> - move upper bounds to "Notes about Qemu limits"
> - s/should/must somewhere. (but not everywhere)
> - move name_size field closer to name itself in bitmap header
> - add extra data area to bitmap header
> - move bitmap data description to separate section
> 
>  docs/specs/qcow2.txt | 221 
> ++-
>  1 file changed, 220 insertions(+), 1 deletion(-)
> 
> diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
> index f236d8c..80cdfd0 100644
> --- a/docs/specs/qcow2.txt
> +++ b/docs/specs/qcow2.txt
> @@ -103,7 +103,18 @@ in the description of a field.
>  write to an image with unknown auto-clear features if it
>  clears the respective bits from this field first.
>  
> -Bits 0-63:  Reserved (set to 0)
> +Bit 0:  Bitmaps extension bit
> +This bit indicates consistency for the 
> bitmaps
> +extension data.
> +
> +It is an error if this bit is set without the
> +bitmaps extension present.
> +
> +If the bitmaps extension is present but this
> +bit is unset, the bitmaps extension data 
> must be
> +considered inconsistent.
> +
> +Bits 1-63:  Reserved (set to 0)
>  
>   96 -  99:  refcount_order
>  Describes the width of a reference count block entry 
> (width
> @@ -123,6 +134,7 @@ be stored. Each extension has a structure like the 
> following:
>  0x - End of the header extension area
>  0xE2792ACA - Backing file format name
>  0x6803f857 - Feature name table
> +0x23852875 - Bitmaps extension
>  other  - Unknown header extension, can be safely
>   ignored
>  
> @@ -166,6 +178,36 @@ the header extension data. Each entry look like this:
>  terminated if it has full length)
>  
>  
> +== Bitmaps extension ==
> +
> +The bitmaps extension is an optional header extension. It provides the 
> ability
> +to store bitmaps related to a virtual disk. For now, there is only one bitmap
> +type: the dirty tracking bitmap, which tracks virtual disk changes from some
> +point in time.
> +
> +The data of the extension should be considered consistent only if the
> +corresponding auto-clear feature bit is set, see 

Re: [Qemu-devel] [PATCH v10] spec: add qcow2 bitmaps extension specification

[Kevin Wolf]

Am 05.02.2016 um 09:58 hat Vladimir Sementsov-Ogievskiy geschrieben:
> The new feature for qcow2: storing bitmaps.
> 
> This patch adds new header extension to qcow2 - Bitmaps Extension. It
> provides an ability to store virtual disk related bitmaps in a qcow2
> image. For now there is only one type of such bitmaps: Dirty Tracking
> Bitmap, which just tracks virtual disk changes from some moment.
> 
> Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
> should be stored in this qcow2 file. The size of each bitmap
> (considering its granularity) is equal to virtual disk size.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 

Thanks, applied to the block branch.

Kevin

Re: [Qemu-devel] [PATCH v10] spec: add qcow2 bitmaps extension specification

[Vladimir Sementsov-Ogievskiy]

On 12.02.2016 17:49, Kevin Wolf wrote:

Am 05.02.2016 um 09:58 hat Vladimir Sementsov-Ogievskiy geschrieben:

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It
provides an ability to store virtual disk related bitmaps in a qcow2
image. For now there is only one type of such bitmaps: Dirty Tracking
Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
should be stored in this qcow2 file. The size of each bitmap
(considering its granularity) is equal to virtual disk size.

Signed-off-by: Vladimir Sementsov-Ogievskiy 

Thanks, applied to the block branch.

Kevin


Cheers! Thank you! Thanks to everyone, who spent time on reviewing and 
discussing!



--
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH v10] spec: add qcow2 bitmaps extension specification

[Denis V. Lunev]

On 02/05/2016 11:58 AM, Vladimir Sementsov-Ogievskiy wrote:

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It
provides an ability to store virtual disk related bitmaps in a qcow2
image. For now there is only one type of such bitmaps: Dirty Tracking
Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
should be stored in this qcow2 file. The size of each bitmap
(considering its granularity) is equal to virtual disk size.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---

v10
 - rewordings, thanks to Fam
 - fix:
Given an offset byte_nr into the virtual disk and the bitmap's granularity, the
-bit offset into the bitmap can be calculated like this:
+bit offset into the image file to the corresponding bit of the bitmap can be
+calculated like this:
 - remove last sentence about 'auto' flasg consistency as superfluous.


v9
 - rewordings, thanks to Max

v8
 - rewordings
 - bitmap_directory_size: 4b -> 8b
 - add more descriptive description in == Bitmaps == section
 - add paragraph "Dirty tracking bitmaps"

   Bitmap directory entry:
 - extra data should not allocate additional clusters
 - padding must be all-bytes-zero
 - add extra_data_compatible flag (now behavior in case of unknown
   extra data is defined by this flag)

v7:

- Rewordings, grammar.
   Max, Eric, John, thank you very much.

- add last paragraph: remaining bits in bitmap data clusters must be
   zero.

- s/Bitmap Directory/bitmap directory/ and other names like this at
   the request of Max.

v6:

- reword bitmap_directory_size description
- bitmap type: make 0 reserved
- extra_data_size: resize to 4bytes
   Also, I've marked this field as "must be zero". We can always change
   it, if we decide allowing managing app to specify any extra data, by
   defining some magic value as a top of user extra data.. So, for now
   non zeor extra_data_size should be considered as an error.
- swap name and extra_data to give good alignment to extra_data.


v5:

- 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of
   bitmaps.
- rewordings
- move upper bounds to "Notes about Qemu limits"
- s/should/must somewhere. (but not everywhere)
- move name_size field closer to name itself in bitmap header
- add extra data area to bitmap header
- move bitmap data description to separate section

  docs/specs/qcow2.txt | 221 ++-
  1 file changed, 220 insertions(+), 1 deletion(-)

diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
index f236d8c..80cdfd0 100644
--- a/docs/specs/qcow2.txt
+++ b/docs/specs/qcow2.txt
@@ -103,7 +103,18 @@ in the description of a field.
  write to an image with unknown auto-clear features if it
  clears the respective bits from this field first.
  
-Bits 0-63:  Reserved (set to 0)

+Bit 0:  Bitmaps extension bit
+This bit indicates consistency for the bitmaps
+extension data.
+
+It is an error if this bit is set without the
+bitmaps extension present.
+
+If the bitmaps extension is present but this
+bit is unset, the bitmaps extension data must 
be
+considered inconsistent.
+
+Bits 1-63:  Reserved (set to 0)
  
   96 -  99:  refcount_order

  Describes the width of a reference count block entry 
(width
@@ -123,6 +134,7 @@ be stored. Each extension has a structure like the 
following:
  0x - End of the header extension area
  0xE2792ACA - Backing file format name
  0x6803f857 - Feature name table
+0x23852875 - Bitmaps extension
  other  - Unknown header extension, can be safely
   ignored
  
@@ -166,6 +178,36 @@ the header extension data. Each entry look like this:

  terminated if it has full length)
  
  
+== Bitmaps extension ==

+
+The bitmaps extension is an optional header extension. It provides the ability
+to store bitmaps related to a virtual disk. For now, there is only one bitmap
+type: the dirty tracking bitmap, which tracks virtual disk changes from some
+point in time.
+
+The data of the extension should be considered consistent only if the
+corresponding auto-clear feature bit is set, see autoclear_features above.
+
+The fields of the bitmaps extension are:
+
+Byte  0 -  3:  nb_bitmaps
+   The number of bitmaps contained in the image. Must be
+   greater than or equal 

Re: [Qemu-devel] [PATCH v10] spec: add qcow2 bitmaps extension specification

[John Snow]

On 02/05/2016 03:58 AM, Vladimir Sementsov-Ogievskiy wrote:
> The new feature for qcow2: storing bitmaps.
> 
> This patch adds new header extension to qcow2 - Bitmaps Extension. It
> provides an ability to store virtual disk related bitmaps in a qcow2
> image. For now there is only one type of such bitmaps: Dirty Tracking
> Bitmap, which just tracks virtual disk changes from some moment.
> 
> Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
> should be stored in this qcow2 file. The size of each bitmap
> (considering its granularity) is equal to virtual disk size.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 

Reviewed-by: John Snow 

> ---
> 
> v10
> - rewordings, thanks to Fam
> - fix:
> Given an offset byte_nr into the virtual disk and the bitmap's granularity, 
> the
> -bit offset into the bitmap can be calculated like this:
> +bit offset into the image file to the corresponding bit of the bitmap can be
> +calculated like this:
> - remove last sentence about 'auto' flasg consistency as superfluous.
> 
> 
> v9
> - rewordings, thanks to Max
> 
> v8
> - rewordings
> - bitmap_directory_size: 4b -> 8b
> - add more descriptive description in == Bitmaps == section
> - add paragraph "Dirty tracking bitmaps"
> 
>   Bitmap directory entry:
> - extra data should not allocate additional clusters
> - padding must be all-bytes-zero
> - add extra_data_compatible flag (now behavior in case of unknown
>   extra data is defined by this flag)
> 
> v7:
> 
> - Rewordings, grammar.
>   Max, Eric, John, thank you very much.
> 
> - add last paragraph: remaining bits in bitmap data clusters must be
>   zero.
> 
> - s/Bitmap Directory/bitmap directory/ and other names like this at
>   the request of Max.
> 
> v6:
> 
> - reword bitmap_directory_size description
> - bitmap type: make 0 reserved
> - extra_data_size: resize to 4bytes
>   Also, I've marked this field as "must be zero". We can always change
>   it, if we decide allowing managing app to specify any extra data, by
>   defining some magic value as a top of user extra data.. So, for now
>   non zeor extra_data_size should be considered as an error.
> - swap name and extra_data to give good alignment to extra_data.
> 
> 
> v5:
> 
> - 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of
>   bitmaps.
> - rewordings
> - move upper bounds to "Notes about Qemu limits"
> - s/should/must somewhere. (but not everywhere)
> - move name_size field closer to name itself in bitmap header
> - add extra data area to bitmap header
> - move bitmap data description to separate section
> 
>  docs/specs/qcow2.txt | 221 
> ++-
>  1 file changed, 220 insertions(+), 1 deletion(-)
> 
> diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
> index f236d8c..80cdfd0 100644
> --- a/docs/specs/qcow2.txt
> +++ b/docs/specs/qcow2.txt
> @@ -103,7 +103,18 @@ in the description of a field.
>  write to an image with unknown auto-clear features if it
>  clears the respective bits from this field first.
>  
> -Bits 0-63:  Reserved (set to 0)
> +Bit 0:  Bitmaps extension bit
> +This bit indicates consistency for the 
> bitmaps
> +extension data.
> +
> +It is an error if this bit is set without the
> +bitmaps extension present.
> +
> +If the bitmaps extension is present but this
> +bit is unset, the bitmaps extension data 
> must be
> +considered inconsistent.
> +
> +Bits 1-63:  Reserved (set to 0)
>  
>   96 -  99:  refcount_order
>  Describes the width of a reference count block entry 
> (width
> @@ -123,6 +134,7 @@ be stored. Each extension has a structure like the 
> following:
>  0x - End of the header extension area
>  0xE2792ACA - Backing file format name
>  0x6803f857 - Feature name table
> +0x23852875 - Bitmaps extension
>  other  - Unknown header extension, can be safely
>   ignored
>  
> @@ -166,6 +178,36 @@ the header extension data. Each entry look like this:
>  terminated if it has full length)
>  
>  
> +== Bitmaps extension ==
> +
> +The bitmaps extension is an optional header extension. It provides the 
> ability
> +to store bitmaps related to a virtual disk. For now, there is only one bitmap
> +type: the dirty tracking bitmap, which tracks virtual disk changes from some
> +point in time.
> +
> +The data of the extension should be considered consistent only if the
> 

Re: [Qemu-devel] [PATCH v10] spec: add qcow2 bitmaps extension specification

[Max Reitz]

On 05.02.2016 09:58, Vladimir Sementsov-Ogievskiy wrote:
> The new feature for qcow2: storing bitmaps.
> 
> This patch adds new header extension to qcow2 - Bitmaps Extension. It
> provides an ability to store virtual disk related bitmaps in a qcow2
> image. For now there is only one type of such bitmaps: Dirty Tracking
> Bitmap, which just tracks virtual disk changes from some moment.
> 
> Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
> should be stored in this qcow2 file. The size of each bitmap
> (considering its granularity) is equal to virtual disk size.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 
> ---

[...]

>  docs/specs/qcow2.txt | 221 
> ++-
>  1 file changed, 220 insertions(+), 1 deletion(-)

Reviewed-by: Max Reitz 



signature.asc
Description: OpenPGP digital signature

[Qemu-devel] [PATCH v10] spec: add qcow2 bitmaps extension specification

[Vladimir Sementsov-Ogievskiy]

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It
provides an ability to store virtual disk related bitmaps in a qcow2
image. For now there is only one type of such bitmaps: Dirty Tracking
Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
should be stored in this qcow2 file. The size of each bitmap
(considering its granularity) is equal to virtual disk size.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---

v10
- rewordings, thanks to Fam
- fix:
Given an offset byte_nr into the virtual disk and the bitmap's granularity, the
-bit offset into the bitmap can be calculated like this:
+bit offset into the image file to the corresponding bit of the bitmap can be
+calculated like this:
- remove last sentence about 'auto' flasg consistency as superfluous.


v9
- rewordings, thanks to Max

v8
- rewordings
- bitmap_directory_size: 4b -> 8b
- add more descriptive description in == Bitmaps == section
- add paragraph "Dirty tracking bitmaps"

  Bitmap directory entry:
- extra data should not allocate additional clusters
- padding must be all-bytes-zero
- add extra_data_compatible flag (now behavior in case of unknown
  extra data is defined by this flag)

v7:

- Rewordings, grammar.
  Max, Eric, John, thank you very much.

- add last paragraph: remaining bits in bitmap data clusters must be
  zero.

- s/Bitmap Directory/bitmap directory/ and other names like this at
  the request of Max.

v6:

- reword bitmap_directory_size description
- bitmap type: make 0 reserved
- extra_data_size: resize to 4bytes
  Also, I've marked this field as "must be zero". We can always change
  it, if we decide allowing managing app to specify any extra data, by
  defining some magic value as a top of user extra data.. So, for now
  non zeor extra_data_size should be considered as an error.
- swap name and extra_data to give good alignment to extra_data.


v5:

- 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of
  bitmaps.
- rewordings
- move upper bounds to "Notes about Qemu limits"
- s/should/must somewhere. (but not everywhere)
- move name_size field closer to name itself in bitmap header
- add extra data area to bitmap header
- move bitmap data description to separate section

 docs/specs/qcow2.txt | 221 ++-
 1 file changed, 220 insertions(+), 1 deletion(-)

diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
index f236d8c..80cdfd0 100644
--- a/docs/specs/qcow2.txt
+++ b/docs/specs/qcow2.txt
@@ -103,7 +103,18 @@ in the description of a field.
 write to an image with unknown auto-clear features if it
 clears the respective bits from this field first.
 
-Bits 0-63:  Reserved (set to 0)
+Bit 0:  Bitmaps extension bit
+This bit indicates consistency for the bitmaps
+extension data.
+
+It is an error if this bit is set without the
+bitmaps extension present.
+
+If the bitmaps extension is present but this
+bit is unset, the bitmaps extension data must 
be
+considered inconsistent.
+
+Bits 1-63:  Reserved (set to 0)
 
  96 -  99:  refcount_order
 Describes the width of a reference count block entry (width
@@ -123,6 +134,7 @@ be stored. Each extension has a structure like the 
following:
 0x - End of the header extension area
 0xE2792ACA - Backing file format name
 0x6803f857 - Feature name table
+0x23852875 - Bitmaps extension
 other  - Unknown header extension, can be safely
  ignored
 
@@ -166,6 +178,36 @@ the header extension data. Each entry look like this:
 terminated if it has full length)
 
 
+== Bitmaps extension ==
+
+The bitmaps extension is an optional header extension. It provides the ability
+to store bitmaps related to a virtual disk. For now, there is only one bitmap
+type: the dirty tracking bitmap, which tracks virtual disk changes from some
+point in time.
+
+The data of the extension should be considered consistent only if the
+corresponding auto-clear feature bit is set, see autoclear_features above.
+
+The fields of the bitmaps extension are:
+
+Byte  0 -  3:  nb_bitmaps
+   The number of bitmaps contained in the image. Must be
+   greater than or equal to 1.
+
+   Note: Qemu currently only supports up to 65535 bitmaps per
+