Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[Vladimir Sementsov-Ogievskiy]

On 25.01.2016 14:09, Kevin Wolf wrote:

Am 25.01.2016 um 11:15 hat Vladimir Sementsov-Ogievskiy geschrieben:

On 19.01.2016 20:29, Kevin Wolf wrote:

Am 19.01.2016 um 09:57 hat Vladimir Sementsov-Ogievskiy geschrieben:

On 19.01.2016 00:16, Eric Blake wrote:

preserving semantics of those extra_data bytes).  We
have enough room for future extension, and that's good e

Ok, so, what should go to the spec? Current wording is ok? Just
delete "Type-specific":

+
+20 - 23:extra_data_size
+Size of type-specific extra data.
+
+For now, as no extra data is defined, extra_data_size is
+reserved and must be zero.
+
+variable:   Extra data for the bitmap.

Please be explicit that if extra_data_size is non-zero, the bitmap must
not be used (i.e. specify the incompatible-feature-bit-like behaviour).

It is not enough. If there are some unknown extra data, then just
ignoring this bitmap may lead to its inconsistency. So, if it is
non-zero, the whole image should not be written. (real
incompatible-feature-bit behavior).

Don't we generally ignore all bitmaps until a user actively tries to
make use of it? Of course, with the 'auto' flag set, just ignoring the
bitmap isn't possible, but I think in all other cases it should be.

If we ever add another type of bitmaps that doesn't have the 'auto' flag
set, but is still automatically used, so that the image as a whole must
become read-only without the bitmap, we can still add a normal
incompatible feature flag. But I think it's more likely that we add
extra_data that doesn't prevent use of the image, so we should have a
way to express that.

Kevin


let's add one flag:

   2: extra_data_compatible
   This flag is meaningful when extra data is unknown for the software.
   If it is set, the bitmap may be used as usual, extra data must 
be left as is.
   If it is unset, the bitmap must not be used, but left as is with 
extra data.


Ok? And, if there will be incompatible features we will use normal 
incompatible feature flag.


--
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[Vladimir Sementsov-Ogievskiy]

On 19.01.2016 20:29, Kevin Wolf wrote:

Am 19.01.2016 um 09:57 hat Vladimir Sementsov-Ogievskiy geschrieben:

On 19.01.2016 00:16, Eric Blake wrote:

preserving semantics of those extra_data bytes).  We
have enough room for future extension, and that's good e

Ok, so, what should go to the spec? Current wording is ok? Just
delete "Type-specific":

+
+20 - 23:extra_data_size
+Size of type-specific extra data.
+
+For now, as no extra data is defined, extra_data_size is
+reserved and must be zero.
+
+variable:   Extra data for the bitmap.

Please be explicit that if extra_data_size is non-zero, the bitmap must
not be used (i.e. specify the incompatible-feature-bit-like behaviour).


It is not enough. If there are some unknown extra data, then just 
ignoring this bitmap may lead to its inconsistency. So, if it is 
non-zero, the whole image should not be written. (real 
incompatible-feature-bit behavior).




Kevin



--
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[Vladimir Sementsov-Ogievskiy]

On 19.01.2016 20:27, Kevin Wolf wrote:

Am 18.01.2016 um 22:16 hat Eric Blake geschrieben:

On 01/18/2016 09:54 AM, John Snow wrote:


Please, let's decide finally about extra data, than I'll reroll it and,
I hope, it will be committed, to make it possible to continue work on
persistence series. About extra data, I'm ready to accept any variant,
strictly defining, what software should do with unknown extra data.



I discussed this with Eric Blake on IRC briefly, and I mentioned I was
concerned that we didn't specify a format at all for the extra data.
Eric felt that it was not unusual to leave a space for future expansion
and that as we haven't used it yet, we don't need to solidify it.

He also felt it would be unusual to stipulate the format of data that we
don't even intend to use yet.

In short, I'm being too proactive.

A commit message mention that, should anyone wish to expand the
type-specific data in the future that adding a 2-byte version as the
first field in extra data would probably be sufficient, and we can worry
about the spec wording later. It is fine to assume for now that if
extra_data_size is 0 that the version/format of the data is "v0" and
that does not limit our future expansion.

Or put another way:

I'm just fine if our initial implementation provides sufficient
information for us to completely parse the file even when the file is
generated by a newer qemu (we have a length, so we know how far to skip
to find the next entry), while at the same time throwing up our hands if
the length is non-zero (we won't read the bitmap at all, because we
don't know if the non-zero extra_data contains instructions that would
change how to interpret the data) or even prevent writes (if the bitmap
entry is marked automatic, we must refuse any write that would requiring
an update to the bitmap because we don't know how to write to a bitmap
while correctly preserving semantics of those extra_data bytes).

Can we assume that the extra_data doesn't contain references to
clusters? Otherwise we need to forbid 'qemu-img check -r leaks' when
there is unknown extra_data.

FWIW, this assumption is already made for snapshots, so it seems okay to
make it here as well. But we could be explicit about it.


ok, will do.



Kevin



--
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[Kevin Wolf]

Am 25.01.2016 um 11:15 hat Vladimir Sementsov-Ogievskiy geschrieben:
> On 19.01.2016 20:29, Kevin Wolf wrote:
> >Am 19.01.2016 um 09:57 hat Vladimir Sementsov-Ogievskiy geschrieben:
> >>On 19.01.2016 00:16, Eric Blake wrote:
> >>>preserving semantics of those extra_data bytes).  We
> >>>have enough room for future extension, and that's good e
> >>Ok, so, what should go to the spec? Current wording is ok? Just
> >>delete "Type-specific":
> >>
> >>+
> >>+20 - 23:extra_data_size
> >>+Size of type-specific extra data.
> >>+
> >>+For now, as no extra data is defined, extra_data_size 
> >>is
> >>+reserved and must be zero.
> >>+
> >>+variable:   Extra data for the bitmap.
> >Please be explicit that if extra_data_size is non-zero, the bitmap must
> >not be used (i.e. specify the incompatible-feature-bit-like behaviour).
> 
> It is not enough. If there are some unknown extra data, then just
> ignoring this bitmap may lead to its inconsistency. So, if it is
> non-zero, the whole image should not be written. (real
> incompatible-feature-bit behavior).

Don't we generally ignore all bitmaps until a user actively tries to
make use of it? Of course, with the 'auto' flag set, just ignoring the
bitmap isn't possible, but I think in all other cases it should be.

If we ever add another type of bitmaps that doesn't have the 'auto' flag
set, but is still automatically used, so that the image as a whole must
become read-only without the bitmap, we can still add a normal
incompatible feature flag. But I think it's more likely that we add
extra_data that doesn't prevent use of the image, so we should have a
way to express that.

Kevin

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[Kevin Wolf]

Am 21.01.2016 um 09:22 hat Vladimir Sementsov-Ogievskiy geschrieben:
> On 21.01.2016 00:22, John Snow wrote:
> >
> >On 01/20/2016 07:34 AM, Vladimir Sementsov-Ogievskiy wrote:
> >>On 19.01.2016 20:48, Kevin Wolf wrote:
> >>>Am 11.01.2016 um 14:05 hat Vladimir Sementsov-Ogievskiy geschrieben:
> The new feature for qcow2: storing bitmaps.
> 
> This patch adds new header extension to qcow2 - Bitmaps Extension. It
> provides an ability to store virtual disk related bitmaps in a qcow2
> image. For now there is only one type of such bitmaps: Dirty Tracking
> Bitmap, which just tracks virtual disk changes from some moment.
> 
> Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
> should be stored in this qcow2 file. The size of each bitmap
> (considering its granularity) is equal to virtual disk size.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 
> ---
> 
> v7:
> 
> - Rewordings, grammar.
> Max, Eric, John, thank you very much.
> 
> - add last paragraph: remaining bits in bitmap data clusters must be
> zero.
> 
> - s/Bitmap Directory/bitmap directory/ and other names like this at
> the request of Max.
> 
> v6:
> 
> - reword bitmap_directory_size description
> - bitmap type: make 0 reserved
> - extra_data_size: resize to 4bytes
> Also, I've marked this field as "must be zero". We can always change
> it, if we decide allowing managing app to specify any extra data, by
> defining some magic value as a top of user extra data.. So, for now
> non zeor extra_data_size should be considered as an error.
> - swap name and extra_data to give good alignment to extra_data.
> 
> 
> v5:
> 
> - 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of
> bitmaps.
> - rewordings
> - move upper bounds to "Notes about Qemu limits"
> - s/should/must somewhere. (but not everywhere)
> - move name_size field closer to name itself in bitmap header
> - add extra data area to bitmap header
> - move bitmap data description to separate section
> 
>    docs/specs/qcow2.txt | 172
> ++-
>    1 file changed, 171 insertions(+), 1 deletion(-)
> 
> diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
> index 121dfc8..997239d 100644
> --- a/docs/specs/qcow2.txt
> +++ b/docs/specs/qcow2.txt
> @@ -103,7 +103,18 @@ in the description of a field.
>    write to an image with unknown auto-clear
> features if it
>    clears the respective bits from this field first.
>    -Bits 0-63:  Reserved (set to 0)
> +Bit 0:  Bitmaps extension bit
> +This bit indicates consistency for
> the bitmaps
> +extension data.
> +
> +It is an error if this bit is set
> without the
> +bitmaps extension present.
> +
> +If the bitmaps extension is present
> but this
> +bit is unset, the bitmaps extension
> data is
> +inconsistent.
> >>>It may as well be consistent, but we don't know.
> >>>
> >>>Perhaps something like "must be considered inconsistent" or "is
> >>>potentially inconsistent".
> >>>
> +
> +Bits 1-63:  Reserved (set to 0)
>   96 -  99:  refcount_order
>    Describes the width of a reference count block
> entry (width
> @@ -123,6 +134,7 @@ be stored. Each extension has a structure like
> the following:
>    0x - End of the header extension area
>    0xE2792ACA - Backing file format name
>    0x6803f857 - Feature name table
> +0x23852875 - Bitmaps extension
>    other  - Unknown header extension, can
> be safely
> ignored
>    @@ -166,6 +178,34 @@ the header extension data. Each entry look
> like this:
>    terminated if it has full length)
>  +== Bitmaps extension ==
> +
> +The bitmaps extension is an optional header extension. It provides
> the ability
> +to store bitmaps related to a virtual disk. For now, there is only
> one bitmap
> +type: the dirty tracking bitmap, which tracks virtual disk changes
> from some
> +point in time.
> >>>I have one major problem with this patch, and it starts here.
> >>>
> >>>The spec talks about dirty tracking bitmaps all the way, but it never
> 

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[Vladimir Sementsov-Ogievskiy]

On 21.01.2016 12:53, Kevin Wolf wrote:

Am 21.01.2016 um 09:22 hat Vladimir Sementsov-Ogievskiy geschrieben:

On 21.01.2016 00:22, John Snow wrote:

On 01/20/2016 07:34 AM, Vladimir Sementsov-Ogievskiy wrote:

On 19.01.2016 20:48, Kevin Wolf wrote:

Am 11.01.2016 um 14:05 hat Vladimir Sementsov-Ogievskiy geschrieben:

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It
provides an ability to store virtual disk related bitmaps in a qcow2
image. For now there is only one type of such bitmaps: Dirty Tracking
Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
should be stored in this qcow2 file. The size of each bitmap
(considering its granularity) is equal to virtual disk size.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---

v7:

- Rewordings, grammar.
Max, Eric, John, thank you very much.

- add last paragraph: remaining bits in bitmap data clusters must be
zero.

- s/Bitmap Directory/bitmap directory/ and other names like this at
the request of Max.

v6:

- reword bitmap_directory_size description
- bitmap type: make 0 reserved
- extra_data_size: resize to 4bytes
Also, I've marked this field as "must be zero". We can always change
it, if we decide allowing managing app to specify any extra data, by
defining some magic value as a top of user extra data.. So, for now
non zeor extra_data_size should be considered as an error.
- swap name and extra_data to give good alignment to extra_data.


v5:

- 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of
bitmaps.
- rewordings
- move upper bounds to "Notes about Qemu limits"
- s/should/must somewhere. (but not everywhere)
- move name_size field closer to name itself in bitmap header
- add extra data area to bitmap header
- move bitmap data description to separate section

   docs/specs/qcow2.txt | 172
++-
   1 file changed, 171 insertions(+), 1 deletion(-)

diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
index 121dfc8..997239d 100644
--- a/docs/specs/qcow2.txt
+++ b/docs/specs/qcow2.txt
@@ -103,7 +103,18 @@ in the description of a field.
   write to an image with unknown auto-clear
features if it
   clears the respective bits from this field first.
   -Bits 0-63:  Reserved (set to 0)
+Bit 0:  Bitmaps extension bit
+This bit indicates consistency for
the bitmaps
+extension data.
+
+It is an error if this bit is set
without the
+bitmaps extension present.
+
+If the bitmaps extension is present
but this
+bit is unset, the bitmaps extension
data is
+inconsistent.

It may as well be consistent, but we don't know.

Perhaps something like "must be considered inconsistent" or "is
potentially inconsistent".


+
+Bits 1-63:  Reserved (set to 0)
  96 -  99:  refcount_order
   Describes the width of a reference count block
entry (width
@@ -123,6 +134,7 @@ be stored. Each extension has a structure like
the following:
   0x - End of the header extension area
   0xE2792ACA - Backing file format name
   0x6803f857 - Feature name table
+0x23852875 - Bitmaps extension
   other  - Unknown header extension, can
be safely
ignored
   @@ -166,6 +178,34 @@ the header extension data. Each entry look
like this:
   terminated if it has full length)
 +== Bitmaps extension ==
+
+The bitmaps extension is an optional header extension. It provides
the ability
+to store bitmaps related to a virtual disk. For now, there is only
one bitmap
+type: the dirty tracking bitmap, which tracks virtual disk changes

>from some

+point in time.

I have one major problem with this patch, and it starts here.

The spec talks about dirty tracking bitmaps all the way, but it never
really defines what a dirty tracking bitmap even contains. It has a few
hints here and there, but they aren't consistent.

Here's the first hint: They track "virtual disk changes", which implies
they track guest clusters rather than host clusters.


+The data of the extension should be considered consistent only if the
+corresponding auto-clear feature bit is set, see autoclear_features
above.
+
+The fields of the bitmaps extension are:
+
+  0 -  3:  nb_bitmaps
+   The number of bitmaps contained in the image.
Must be
+   greater than or equal to 1.
+
+   

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[Vladimir Sementsov-Ogievskiy]

On 21.01.2016 00:22, John Snow wrote:


On 01/20/2016 07:34 AM, Vladimir Sementsov-Ogievskiy wrote:

On 19.01.2016 20:48, Kevin Wolf wrote:

Am 11.01.2016 um 14:05 hat Vladimir Sementsov-Ogievskiy geschrieben:

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It
provides an ability to store virtual disk related bitmaps in a qcow2
image. For now there is only one type of such bitmaps: Dirty Tracking
Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
should be stored in this qcow2 file. The size of each bitmap
(considering its granularity) is equal to virtual disk size.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---

v7:

- Rewordings, grammar.
Max, Eric, John, thank you very much.

- add last paragraph: remaining bits in bitmap data clusters must be
zero.

- s/Bitmap Directory/bitmap directory/ and other names like this at
the request of Max.

v6:

- reword bitmap_directory_size description
- bitmap type: make 0 reserved
- extra_data_size: resize to 4bytes
Also, I've marked this field as "must be zero". We can always change
it, if we decide allowing managing app to specify any extra data, by
defining some magic value as a top of user extra data.. So, for now
non zeor extra_data_size should be considered as an error.
- swap name and extra_data to give good alignment to extra_data.


v5:

- 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of
bitmaps.
- rewordings
- move upper bounds to "Notes about Qemu limits"
- s/should/must somewhere. (but not everywhere)
- move name_size field closer to name itself in bitmap header
- add extra data area to bitmap header
- move bitmap data description to separate section

   docs/specs/qcow2.txt | 172
++-
   1 file changed, 171 insertions(+), 1 deletion(-)

diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
index 121dfc8..997239d 100644
--- a/docs/specs/qcow2.txt
+++ b/docs/specs/qcow2.txt
@@ -103,7 +103,18 @@ in the description of a field.
   write to an image with unknown auto-clear
features if it
   clears the respective bits from this field first.
   -Bits 0-63:  Reserved (set to 0)
+Bit 0:  Bitmaps extension bit
+This bit indicates consistency for
the bitmaps
+extension data.
+
+It is an error if this bit is set
without the
+bitmaps extension present.
+
+If the bitmaps extension is present
but this
+bit is unset, the bitmaps extension
data is
+inconsistent.

It may as well be consistent, but we don't know.

Perhaps something like "must be considered inconsistent" or "is
potentially inconsistent".


+
+Bits 1-63:  Reserved (set to 0)
  96 -  99:  refcount_order
   Describes the width of a reference count block
entry (width
@@ -123,6 +134,7 @@ be stored. Each extension has a structure like
the following:
   0x - End of the header extension area
   0xE2792ACA - Backing file format name
   0x6803f857 - Feature name table
+0x23852875 - Bitmaps extension
   other  - Unknown header extension, can
be safely
ignored
   @@ -166,6 +178,34 @@ the header extension data. Each entry look
like this:
   terminated if it has full length)
 +== Bitmaps extension ==
+
+The bitmaps extension is an optional header extension. It provides
the ability
+to store bitmaps related to a virtual disk. For now, there is only
one bitmap
+type: the dirty tracking bitmap, which tracks virtual disk changes
from some
+point in time.

I have one major problem with this patch, and it starts here.

The spec talks about dirty tracking bitmaps all the way, but it never
really defines what a dirty tracking bitmap even contains. It has a few
hints here and there, but they aren't consistent.

Here's the first hint: They track "virtual disk changes", which implies
they track guest clusters rather than host clusters.


+The data of the extension should be considered consistent only if the
+corresponding auto-clear feature bit is set, see autoclear_features
above.
+
+The fields of the bitmaps extension are:
+
+  0 -  3:  nb_bitmaps
+   The number of bitmaps contained in the image.
Must be
+   greater than or equal to 1.
+
+   Note: Qemu currently only supports up to 65535
bitmaps per
+   image.
+
+  4 -  7:  

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[John Snow]

On 01/20/2016 07:34 AM, Vladimir Sementsov-Ogievskiy wrote:
> On 19.01.2016 20:48, Kevin Wolf wrote:
>> Am 11.01.2016 um 14:05 hat Vladimir Sementsov-Ogievskiy geschrieben:
>>> The new feature for qcow2: storing bitmaps.
>>>
>>> This patch adds new header extension to qcow2 - Bitmaps Extension. It
>>> provides an ability to store virtual disk related bitmaps in a qcow2
>>> image. For now there is only one type of such bitmaps: Dirty Tracking
>>> Bitmap, which just tracks virtual disk changes from some moment.
>>>
>>> Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
>>> should be stored in this qcow2 file. The size of each bitmap
>>> (considering its granularity) is equal to virtual disk size.
>>>
>>> Signed-off-by: Vladimir Sementsov-Ogievskiy 
>>> ---
>>>
>>> v7:
>>>
>>> - Rewordings, grammar.
>>>Max, Eric, John, thank you very much.
>>>
>>> - add last paragraph: remaining bits in bitmap data clusters must be
>>>zero.
>>>
>>> - s/Bitmap Directory/bitmap directory/ and other names like this at
>>>the request of Max.
>>>
>>> v6:
>>>
>>> - reword bitmap_directory_size description
>>> - bitmap type: make 0 reserved
>>> - extra_data_size: resize to 4bytes
>>>Also, I've marked this field as "must be zero". We can always change
>>>it, if we decide allowing managing app to specify any extra data, by
>>>defining some magic value as a top of user extra data.. So, for now
>>>non zeor extra_data_size should be considered as an error.
>>> - swap name and extra_data to give good alignment to extra_data.
>>>
>>>
>>> v5:
>>>
>>> - 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of
>>>bitmaps.
>>> - rewordings
>>> - move upper bounds to "Notes about Qemu limits"
>>> - s/should/must somewhere. (but not everywhere)
>>> - move name_size field closer to name itself in bitmap header
>>> - add extra data area to bitmap header
>>> - move bitmap data description to separate section
>>>
>>>   docs/specs/qcow2.txt | 172
>>> ++-
>>>   1 file changed, 171 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
>>> index 121dfc8..997239d 100644
>>> --- a/docs/specs/qcow2.txt
>>> +++ b/docs/specs/qcow2.txt
>>> @@ -103,7 +103,18 @@ in the description of a field.
>>>   write to an image with unknown auto-clear
>>> features if it
>>>   clears the respective bits from this field first.
>>>   -Bits 0-63:  Reserved (set to 0)
>>> +Bit 0:  Bitmaps extension bit
>>> +This bit indicates consistency for
>>> the bitmaps
>>> +extension data.
>>> +
>>> +It is an error if this bit is set
>>> without the
>>> +bitmaps extension present.
>>> +
>>> +If the bitmaps extension is present
>>> but this
>>> +bit is unset, the bitmaps extension
>>> data is
>>> +inconsistent.
>> It may as well be consistent, but we don't know.
>>
>> Perhaps something like "must be considered inconsistent" or "is
>> potentially inconsistent".
>>
>>> +
>>> +Bits 1-63:  Reserved (set to 0)
>>>  96 -  99:  refcount_order
>>>   Describes the width of a reference count block
>>> entry (width
>>> @@ -123,6 +134,7 @@ be stored. Each extension has a structure like
>>> the following:
>>>   0x - End of the header extension area
>>>   0xE2792ACA - Backing file format name
>>>   0x6803f857 - Feature name table
>>> +0x23852875 - Bitmaps extension
>>>   other  - Unknown header extension, can
>>> be safely
>>>ignored
>>>   @@ -166,6 +178,34 @@ the header extension data. Each entry look
>>> like this:
>>>   terminated if it has full length)
>>> +== Bitmaps extension ==
>>> +
>>> +The bitmaps extension is an optional header extension. It provides
>>> the ability
>>> +to store bitmaps related to a virtual disk. For now, there is only
>>> one bitmap
>>> +type: the dirty tracking bitmap, which tracks virtual disk changes
>>> from some
>>> +point in time.
>> I have one major problem with this patch, and it starts here.
>>
>> The spec talks about dirty tracking bitmaps all the way, but it never
>> really defines what a dirty tracking bitmap even contains. It has a few
>> hints here and there, but they aren't consistent.
>>
>> Here's the first hint: They track "virtual disk changes", which implies
>> they track guest clusters rather than host clusters.
>>
>>> +The data of the extension should be considered consistent only if the
>>> +corresponding auto-clear feature bit 

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[Vladimir Sementsov-Ogievskiy]

On 19.01.2016 20:48, Kevin Wolf wrote:

Am 11.01.2016 um 14:05 hat Vladimir Sementsov-Ogievskiy geschrieben:

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It
provides an ability to store virtual disk related bitmaps in a qcow2
image. For now there is only one type of such bitmaps: Dirty Tracking
Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
should be stored in this qcow2 file. The size of each bitmap
(considering its granularity) is equal to virtual disk size.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---

v7:

- Rewordings, grammar.
   Max, Eric, John, thank you very much.

- add last paragraph: remaining bits in bitmap data clusters must be
   zero.

- s/Bitmap Directory/bitmap directory/ and other names like this at
   the request of Max.

v6:

- reword bitmap_directory_size description
- bitmap type: make 0 reserved
- extra_data_size: resize to 4bytes
   Also, I've marked this field as "must be zero". We can always change
   it, if we decide allowing managing app to specify any extra data, by
   defining some magic value as a top of user extra data.. So, for now
   non zeor extra_data_size should be considered as an error.
- swap name and extra_data to give good alignment to extra_data.


v5:

- 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of
   bitmaps.
- rewordings
- move upper bounds to "Notes about Qemu limits"
- s/should/must somewhere. (but not everywhere)
- move name_size field closer to name itself in bitmap header
- add extra data area to bitmap header
- move bitmap data description to separate section

  docs/specs/qcow2.txt | 172 ++-
  1 file changed, 171 insertions(+), 1 deletion(-)

diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
index 121dfc8..997239d 100644
--- a/docs/specs/qcow2.txt
+++ b/docs/specs/qcow2.txt
@@ -103,7 +103,18 @@ in the description of a field.
  write to an image with unknown auto-clear features if it
  clears the respective bits from this field first.
  
-Bits 0-63:  Reserved (set to 0)

+Bit 0:  Bitmaps extension bit
+This bit indicates consistency for the bitmaps
+extension data.
+
+It is an error if this bit is set without the
+bitmaps extension present.
+
+If the bitmaps extension is present but this
+bit is unset, the bitmaps extension data is
+inconsistent.

It may as well be consistent, but we don't know.

Perhaps something like "must be considered inconsistent" or "is
potentially inconsistent".


+
+Bits 1-63:  Reserved (set to 0)
  
   96 -  99:  refcount_order

  Describes the width of a reference count block entry 
(width
@@ -123,6 +134,7 @@ be stored. Each extension has a structure like the 
following:
  0x - End of the header extension area
  0xE2792ACA - Backing file format name
  0x6803f857 - Feature name table
+0x23852875 - Bitmaps extension
  other  - Unknown header extension, can be safely
   ignored
  
@@ -166,6 +178,34 @@ the header extension data. Each entry look like this:

  terminated if it has full length)
  
  
+== Bitmaps extension ==

+
+The bitmaps extension is an optional header extension. It provides the ability
+to store bitmaps related to a virtual disk. For now, there is only one bitmap
+type: the dirty tracking bitmap, which tracks virtual disk changes from some
+point in time.

I have one major problem with this patch, and it starts here.

The spec talks about dirty tracking bitmaps all the way, but it never
really defines what a dirty tracking bitmap even contains. It has a few
hints here and there, but they aren't consistent.

Here's the first hint: They track "virtual disk changes", which implies
they track guest clusters rather than host clusters.


+The data of the extension should be considered consistent only if the
+corresponding auto-clear feature bit is set, see autoclear_features above.
+
+The fields of the bitmaps extension are:
+
+  0 -  3:  nb_bitmaps
+   The number of bitmaps contained in the image. Must be
+   greater than or equal to 1.
+
+   Note: Qemu currently only supports up to 65535 bitmaps per
+   image.
+
+  4 -  7:  bitmap_directory_size
+   Size of the bitmap directory in bytes. It is the cumulative
+   

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[Vladimir Sementsov-Ogievskiy]

On 19.01.2016 00:16, Eric Blake wrote:

preserving semantics of those extra_data bytes).  We
have enough room for future extension, and that's good e


Ok, so, what should go to the spec? Current wording is ok? Just delete 
"Type-specific":


+
+20 - 23:extra_data_size
+Size of type-specific extra data.
+
+For now, as no extra data is defined, extra_data_size is
+reserved and must be zero.
+
+variable:   Extra data for the bitmap.
+




--
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[Kevin Wolf]

Am 11.01.2016 um 14:05 hat Vladimir Sementsov-Ogievskiy geschrieben:
> The new feature for qcow2: storing bitmaps.
> 
> This patch adds new header extension to qcow2 - Bitmaps Extension. It
> provides an ability to store virtual disk related bitmaps in a qcow2
> image. For now there is only one type of such bitmaps: Dirty Tracking
> Bitmap, which just tracks virtual disk changes from some moment.
> 
> Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
> should be stored in this qcow2 file. The size of each bitmap
> (considering its granularity) is equal to virtual disk size.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 
> ---
> 
> v7:
> 
> - Rewordings, grammar.
>   Max, Eric, John, thank you very much.
> 
> - add last paragraph: remaining bits in bitmap data clusters must be
>   zero.
> 
> - s/Bitmap Directory/bitmap directory/ and other names like this at
>   the request of Max.
> 
> v6:
> 
> - reword bitmap_directory_size description
> - bitmap type: make 0 reserved
> - extra_data_size: resize to 4bytes
>   Also, I've marked this field as "must be zero". We can always change
>   it, if we decide allowing managing app to specify any extra data, by
>   defining some magic value as a top of user extra data.. So, for now
>   non zeor extra_data_size should be considered as an error.
> - swap name and extra_data to give good alignment to extra_data.
> 
> 
> v5:
> 
> - 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of
>   bitmaps.
> - rewordings
> - move upper bounds to "Notes about Qemu limits"
> - s/should/must somewhere. (but not everywhere)
> - move name_size field closer to name itself in bitmap header
> - add extra data area to bitmap header
> - move bitmap data description to separate section
> 
>  docs/specs/qcow2.txt | 172 
> ++-
>  1 file changed, 171 insertions(+), 1 deletion(-)
> 
> diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
> index 121dfc8..997239d 100644
> --- a/docs/specs/qcow2.txt
> +++ b/docs/specs/qcow2.txt
> @@ -103,7 +103,18 @@ in the description of a field.
>  write to an image with unknown auto-clear features if it
>  clears the respective bits from this field first.
>  
> -Bits 0-63:  Reserved (set to 0)
> +Bit 0:  Bitmaps extension bit
> +This bit indicates consistency for the 
> bitmaps
> +extension data.
> +
> +It is an error if this bit is set without the
> +bitmaps extension present.
> +
> +If the bitmaps extension is present but this
> +bit is unset, the bitmaps extension data is
> +inconsistent.

It may as well be consistent, but we don't know.

Perhaps something like "must be considered inconsistent" or "is
potentially inconsistent".

> +
> +Bits 1-63:  Reserved (set to 0)
>  
>   96 -  99:  refcount_order
>  Describes the width of a reference count block entry 
> (width
> @@ -123,6 +134,7 @@ be stored. Each extension has a structure like the 
> following:
>  0x - End of the header extension area
>  0xE2792ACA - Backing file format name
>  0x6803f857 - Feature name table
> +0x23852875 - Bitmaps extension
>  other  - Unknown header extension, can be safely
>   ignored
>  
> @@ -166,6 +178,34 @@ the header extension data. Each entry look like this:
>  terminated if it has full length)
>  
>  
> +== Bitmaps extension ==
> +
> +The bitmaps extension is an optional header extension. It provides the 
> ability
> +to store bitmaps related to a virtual disk. For now, there is only one bitmap
> +type: the dirty tracking bitmap, which tracks virtual disk changes from some
> +point in time.

I have one major problem with this patch, and it starts here.

The spec talks about dirty tracking bitmaps all the way, but it never
really defines what a dirty tracking bitmap even contains. It has a few
hints here and there, but they aren't consistent.

Here's the first hint: They track "virtual disk changes", which implies
they track guest clusters rather than host clusters.

> +The data of the extension should be considered consistent only if the
> +corresponding auto-clear feature bit is set, see autoclear_features above.
> +
> +The fields of the bitmaps extension are:
> +
> +  0 -  3:  nb_bitmaps
> +   The number of bitmaps contained in the image. Must be
> +   greater than or equal to 1.
> +
> +   Note: Qemu currently only supports up to 65535 bitmaps per
> +  

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[Kevin Wolf]

Am 18.01.2016 um 22:16 hat Eric Blake geschrieben:
> On 01/18/2016 09:54 AM, John Snow wrote:
> 
> >> Please, let's decide finally about extra data, than I'll reroll it and,
> >> I hope, it will be committed, to make it possible to continue work on
> >> persistence series. About extra data, I'm ready to accept any variant,
> >> strictly defining, what software should do with unknown extra data.
> >>
> >>
> > 
> > I discussed this with Eric Blake on IRC briefly, and I mentioned I was
> > concerned that we didn't specify a format at all for the extra data.
> > Eric felt that it was not unusual to leave a space for future expansion
> > and that as we haven't used it yet, we don't need to solidify it.
> > 
> > He also felt it would be unusual to stipulate the format of data that we
> > don't even intend to use yet.
> > 
> > In short, I'm being too proactive.
> > 
> > A commit message mention that, should anyone wish to expand the
> > type-specific data in the future that adding a 2-byte version as the
> > first field in extra data would probably be sufficient, and we can worry
> > about the spec wording later. It is fine to assume for now that if
> > extra_data_size is 0 that the version/format of the data is "v0" and
> > that does not limit our future expansion.
> 
> Or put another way:
> 
> I'm just fine if our initial implementation provides sufficient
> information for us to completely parse the file even when the file is
> generated by a newer qemu (we have a length, so we know how far to skip
> to find the next entry), while at the same time throwing up our hands if
> the length is non-zero (we won't read the bitmap at all, because we
> don't know if the non-zero extra_data contains instructions that would
> change how to interpret the data) or even prevent writes (if the bitmap
> entry is marked automatic, we must refuse any write that would requiring
> an update to the bitmap because we don't know how to write to a bitmap
> while correctly preserving semantics of those extra_data bytes).

Can we assume that the extra_data doesn't contain references to
clusters? Otherwise we need to forbid 'qemu-img check -r leaks' when
there is unknown extra_data.

FWIW, this assumption is already made for snapshots, so it seems okay to
make it here as well. But we could be explicit about it.

Kevin


pgpHaCqQsZMzl.pgp
Description: PGP signature

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[Kevin Wolf]

Am 19.01.2016 um 09:57 hat Vladimir Sementsov-Ogievskiy geschrieben:
> On 19.01.2016 00:16, Eric Blake wrote:
> >preserving semantics of those extra_data bytes).  We
> >have enough room for future extension, and that's good e
> 
> Ok, so, what should go to the spec? Current wording is ok? Just
> delete "Type-specific":
> 
> +
> +20 - 23:extra_data_size
> +Size of type-specific extra data.
> +
> +For now, as no extra data is defined, extra_data_size is
> +reserved and must be zero.
> +
> +variable:   Extra data for the bitmap.

Please be explicit that if extra_data_size is non-zero, the bitmap must
not be used (i.e. specify the incompatible-feature-bit-like behaviour).

Kevin

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[John Snow]

On 01/16/2016 09:06 AM, Vladimir Sementsov-Ogievskiy wrote:
> On 15.01.2016 02:26, John Snow wrote:
>>
>> On 01/14/2016 05:08 PM, Eric Blake wrote:
>>> On 01/11/2016 06:05 AM, Vladimir Sementsov-Ogievskiy wrote:
 The new feature for qcow2: storing bitmaps.

 This patch adds new header extension to qcow2 - Bitmaps Extension. It
 provides an ability to store virtual disk related bitmaps in a qcow2
 image. For now there is only one type of such bitmaps: Dirty Tracking
 Bitmap, which just tracks virtual disk changes from some moment.

 Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
 should be stored in this qcow2 file. The size of each bitmap
 (considering its granularity) is equal to virtual disk size.

 Signed-off-by: Vladimir Sementsov-Ogievskiy 
 ---

 @@ -166,6 +178,34 @@ the header extension data. Each entry look like
 this:
   terminated if it has full length)
 +== Bitmaps extension ==
 +  0 -  3:  nb_bitmaps
 +   The number of bitmaps contained in the image.
 Must be
 +   greater than or equal to 1.
 +
 +   Note: Qemu currently only supports up to 65535
 bitmaps per
 +   image.
 +
 +  4 -  7:  bitmap_directory_size
 +   Size of the bitmap directory in bytes. It is the
 cumulative
 +   size of all (nb_bitmaps) bitmap headers.
>>> Only 4 bytes - if we ever raise our 64k entry restriction (nb_bitmaps),
>>> could we run into an image that has so many directory entries as to make
>>> the directory itself spill past 4G?  But I don't think it is likely, so
>>> I can live with your choice.
>>>
>> "We'll never need this!"
>>
>> I hope someone in 2082 is reading this right now and is quite angry.
>>
>> (But really, I can't foresee needing this many per each drive -- and if
>> we do, we have external storage mechanisms in development to handle such
>> wild cases.)
> 
> Let's change it to 8 bytes, and add 4bytes reserved field to keep 8b
> alignment.
> 
>>
 +
 +== Bitmaps ==
 +
 +As mentioned above, the bitmaps extension provides the ability to
 store bitmaps
 +related a virtual disk. This section describes how these bitmaps
 are stored.
 +
 +Note: all bitmaps are related to the virtual disk stored in this
 image.
 +
 +=== Bitmap directory ===
 +
 +Each bitmap saved in the image is described in a bitmap directory
 entry. The
 +bitmap directory is a contiguous area in the image file, whose
 starting offset
 +and length are given by the header extension fields
 bitmap_directory_offset and
 +bitmap_directory_size. The entries of the bitmap directory have
 variable
 +length, depending on the length of the bitmap name and extra data.
 These
 +entries are also called bitmap headers.
 +
 +Structure of a bitmap directory entry:
 +
 +Byte 0 -  7:bitmap_table_offset
 +Offset into the image file at which the bitmap
 table
 +(described below) for the bitmap starts. Must
 be aligned to
 +a cluster boundary.
 +
 + 8 - 11:bitmap_table_size
 +Number of entries in the bitmap table of the
 bitmap.
>>> Should this be the size in bytes, instead of the number of entries? But
>> For what benefit? We can calculate either from the other, and this gives
>> us a better resolution.
>>
>>> at least the entries are fixed width of 8 bytes each, so this lets you
>>> get a bitmap table up to 32G bytes rather than just 4G in size.  (Let's
>>> see here - if we have 32G bytes in the bitmap table, that means 4G
>>> clusters occupied by the bitmap itself; in the worst case of 512-byte
>>> clusters and granularity 0, that is a maximum bitmap size of 2T
>>> describing 16T of virtual guest image; with larger cluster size and/or
>>> larger granularity, we cover a lot more virtual guest space with less
>>> bitmap size; so I guess we aren't too worried about running out of
>>> space?).
>>>
>> Yes, worst case of g=0 and cluster size of 512 bytes, we can get 2T
>> bitmaps describing 16T of virtual data.
>>
>> "default case" of 64K clusters and 64K granularity: 256TiB bitmaps
>> describing ... let's see ... if my math is right, 128EiB?
>>
>> We're probably fine :)
>>
>> (Cue future space-person from 2159 wondering how I could have ever been
>> so naive. Sorry, future space-person!)
> 
> it is just like l1_size from qcow2 header.
> 
>  36 - 39:   l1_size
> Number of entries in the active L1 table
> 
> 
>>
 +20 - 23:extra_data_size
 +Size of type-specific extra data.
 +
 +For now, as no extra data is defined,
 

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[Eric Blake]

On 01/18/2016 09:54 AM, John Snow wrote:

>> Please, let's decide finally about extra data, than I'll reroll it and,
>> I hope, it will be committed, to make it possible to continue work on
>> persistence series. About extra data, I'm ready to accept any variant,
>> strictly defining, what software should do with unknown extra data.
>>
>>
> 
> I discussed this with Eric Blake on IRC briefly, and I mentioned I was
> concerned that we didn't specify a format at all for the extra data.
> Eric felt that it was not unusual to leave a space for future expansion
> and that as we haven't used it yet, we don't need to solidify it.
> 
> He also felt it would be unusual to stipulate the format of data that we
> don't even intend to use yet.
> 
> In short, I'm being too proactive.
> 
> A commit message mention that, should anyone wish to expand the
> type-specific data in the future that adding a 2-byte version as the
> first field in extra data would probably be sufficient, and we can worry
> about the spec wording later. It is fine to assume for now that if
> extra_data_size is 0 that the version/format of the data is "v0" and
> that does not limit our future expansion.

Or put another way:

I'm just fine if our initial implementation provides sufficient
information for us to completely parse the file even when the file is
generated by a newer qemu (we have a length, so we know how far to skip
to find the next entry), while at the same time throwing up our hands if
the length is non-zero (we won't read the bitmap at all, because we
don't know if the non-zero extra_data contains instructions that would
change how to interpret the data) or even prevent writes (if the bitmap
entry is marked automatic, we must refuse any write that would requiring
an update to the bitmap because we don't know how to write to a bitmap
while correctly preserving semantics of those extra_data bytes).  We
have enough room for future extension, and that's good enough for now;
the commit message can document that we thought about the future,
without having to actually nail down in the spec what the future will
actually do.

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[Vladimir Sementsov-Ogievskiy]

On 15.01.2016 02:26, John Snow wrote:


On 01/14/2016 05:08 PM, Eric Blake wrote:

On 01/11/2016 06:05 AM, Vladimir Sementsov-Ogievskiy wrote:

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It
provides an ability to store virtual disk related bitmaps in a qcow2
image. For now there is only one type of such bitmaps: Dirty Tracking
Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
should be stored in this qcow2 file. The size of each bitmap
(considering its granularity) is equal to virtual disk size.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---

@@ -166,6 +178,34 @@ the header extension data. Each entry look like this:
  terminated if it has full length)
  
  
+== Bitmaps extension ==

+  0 -  3:  nb_bitmaps
+   The number of bitmaps contained in the image. Must be
+   greater than or equal to 1.
+
+   Note: Qemu currently only supports up to 65535 bitmaps per
+   image.
+
+  4 -  7:  bitmap_directory_size
+   Size of the bitmap directory in bytes. It is the cumulative
+   size of all (nb_bitmaps) bitmap headers.

Only 4 bytes - if we ever raise our 64k entry restriction (nb_bitmaps),
could we run into an image that has so many directory entries as to make
the directory itself spill past 4G?  But I don't think it is likely, so
I can live with your choice.


"We'll never need this!"

I hope someone in 2082 is reading this right now and is quite angry.

(But really, I can't foresee needing this many per each drive -- and if
we do, we have external storage mechanisms in development to handle such
wild cases.)


Let's change it to 8 bytes, and add 4bytes reserved field to keep 8b 
alignment.





+
+== Bitmaps ==
+
+As mentioned above, the bitmaps extension provides the ability to store bitmaps
+related a virtual disk. This section describes how these bitmaps are stored.
+
+Note: all bitmaps are related to the virtual disk stored in this image.
+
+=== Bitmap directory ===
+
+Each bitmap saved in the image is described in a bitmap directory entry. The
+bitmap directory is a contiguous area in the image file, whose starting offset
+and length are given by the header extension fields bitmap_directory_offset and
+bitmap_directory_size. The entries of the bitmap directory have variable
+length, depending on the length of the bitmap name and extra data. These
+entries are also called bitmap headers.
+
+Structure of a bitmap directory entry:
+
+Byte 0 -  7:bitmap_table_offset
+Offset into the image file at which the bitmap table
+(described below) for the bitmap starts. Must be aligned to
+a cluster boundary.
+
+ 8 - 11:bitmap_table_size
+Number of entries in the bitmap table of the bitmap.

Should this be the size in bytes, instead of the number of entries? But

For what benefit? We can calculate either from the other, and this gives
us a better resolution.


at least the entries are fixed width of 8 bytes each, so this lets you
get a bitmap table up to 32G bytes rather than just 4G in size.  (Let's
see here - if we have 32G bytes in the bitmap table, that means 4G
clusters occupied by the bitmap itself; in the worst case of 512-byte
clusters and granularity 0, that is a maximum bitmap size of 2T
describing 16T of virtual guest image; with larger cluster size and/or
larger granularity, we cover a lot more virtual guest space with less
bitmap size; so I guess we aren't too worried about running out of space?).


Yes, worst case of g=0 and cluster size of 512 bytes, we can get 2T
bitmaps describing 16T of virtual data.

"default case" of 64K clusters and 64K granularity: 256TiB bitmaps
describing ... let's see ... if my math is right, 128EiB?

We're probably fine :)

(Cue future space-person from 2159 wondering how I could have ever been
so naive. Sorry, future space-person!)


it is just like l1_size from qcow2 header.

 36 - 39:   l1_size
Number of entries in the active L1 table





+20 - 23:extra_data_size
+Size of type-specific extra data.
+
+For now, as no extra data is defined, extra_data_size is
+reserved and must be zero.
+
+variable:   Type-specific extra data for the bitmap.

I'd write this as:
variable:   extra_data
Type-specific extra data for the bitmap,
occupying extra_data_size bytes.


Ok.. then, similar should be done for the name.

About extra data and our discussion of versioning it. For now, we don't 
have concrete cases and requirements for this data, we are only foresee, 
that it may be needed in future, and waste time in 

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[Denis V. Lunev]

On 01/12/2016 03:30 AM, John Snow wrote:


On 01/11/2016 08:05 AM, Vladimir Sementsov-Ogievskiy wrote:

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It
provides an ability to store virtual disk related bitmaps in a qcow2
image. For now there is only one type of such bitmaps: Dirty Tracking
Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
should be stored in this qcow2 file. The size of each bitmap
(considering its granularity) is equal to virtual disk size.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---

v7:

- Rewordings, grammar.
   Max, Eric, John, thank you very much.

- add last paragraph: remaining bits in bitmap data clusters must be
   zero.

- s/Bitmap Directory/bitmap directory/ and other names like this at
   the request of Max.

v6:

- reword bitmap_directory_size description
- bitmap type: make 0 reserved
- extra_data_size: resize to 4bytes
   Also, I've marked this field as "must be zero". We can always change
   it, if we decide allowing managing app to specify any extra data, by
   defining some magic value as a top of user extra data.. So, for now
   non zeor extra_data_size should be considered as an error.
- swap name and extra_data to give good alignment to extra_data.


v5:

- 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of
   bitmaps.
- rewordings
- move upper bounds to "Notes about Qemu limits"
- s/should/must somewhere. (but not everywhere)
- move name_size field closer to name itself in bitmap header
- add extra data area to bitmap header
- move bitmap data description to separate section

  docs/specs/qcow2.txt | 172 ++-
  1 file changed, 171 insertions(+), 1 deletion(-)

diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
index 121dfc8..997239d 100644
--- a/docs/specs/qcow2.txt
+++ b/docs/specs/qcow2.txt
@@ -103,7 +103,18 @@ in the description of a field.
  write to an image with unknown auto-clear features if it
  clears the respective bits from this field first.
  
-Bits 0-63:  Reserved (set to 0)

+Bit 0:  Bitmaps extension bit
+This bit indicates consistency for the bitmaps
+extension data.
+
+It is an error if this bit is set without the
+bitmaps extension present.
+
+If the bitmaps extension is present but this
+bit is unset, the bitmaps extension data is
+inconsistent.
+
+Bits 1-63:  Reserved (set to 0)
  
   96 -  99:  refcount_order

  Describes the width of a reference count block entry 
(width
@@ -123,6 +134,7 @@ be stored. Each extension has a structure like the 
following:
  0x - End of the header extension area
  0xE2792ACA - Backing file format name
  0x6803f857 - Feature name table
+0x23852875 - Bitmaps extension
  other  - Unknown header extension, can be safely
   ignored
  
@@ -166,6 +178,34 @@ the header extension data. Each entry look like this:

  terminated if it has full length)
  
  
+== Bitmaps extension ==

+
+The bitmaps extension is an optional header extension. It provides the ability
+to store bitmaps related to a virtual disk. For now, there is only one bitmap
+type: the dirty tracking bitmap, which tracks virtual disk changes from some
+point in time.
+
+The data of the extension should be considered consistent only if the
+corresponding auto-clear feature bit is set, see autoclear_features above.
+
+The fields of the bitmaps extension are:
+
+  0 -  3:  nb_bitmaps
+   The number of bitmaps contained in the image. Must be
+   greater than or equal to 1.
+
+   Note: Qemu currently only supports up to 65535 bitmaps per
+   image.
+
+  4 -  7:  bitmap_directory_size
+   Size of the bitmap directory in bytes. It is the cumulative
+   size of all (nb_bitmaps) bitmap headers.
+
+  8 - 15:  bitmap_directory_offset
+   Offset into the image file at which the bitmap directory
+   starts. Must be aligned to a cluster boundary.
+
+
  == Host cluster management ==
  
  qcow2 manages the allocation of host clusters by maintaining a reference count

@@ -360,3 +400,133 @@ Snapshot table entry:
  
  variable:   Padding to round up the snapshot table entry size to the

  next multiple of 8.
+
+

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[John Snow]

On 01/14/2016 06:35 AM, Denis V. Lunev wrote:
> On 01/12/2016 03:30 AM, John Snow wrote:
>>
>> On 01/11/2016 08:05 AM, Vladimir Sementsov-Ogievskiy wrote:
>>> The new feature for qcow2: storing bitmaps.
>>>
>>> This patch adds new header extension to qcow2 - Bitmaps Extension. It
>>> provides an ability to store virtual disk related bitmaps in a qcow2
>>> image. For now there is only one type of such bitmaps: Dirty Tracking
>>> Bitmap, which just tracks virtual disk changes from some moment.
>>>
>>> Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
>>> should be stored in this qcow2 file. The size of each bitmap
>>> (considering its granularity) is equal to virtual disk size.
>>>
>>> Signed-off-by: Vladimir Sementsov-Ogievskiy 
>>> ---
>>>
>>> v7:
>>>
>>> - Rewordings, grammar.
>>>Max, Eric, John, thank you very much.
>>>
>>> - add last paragraph: remaining bits in bitmap data clusters must be
>>>zero.
>>>
>>> - s/Bitmap Directory/bitmap directory/ and other names like this at
>>>the request of Max.
>>>
>>> v6:
>>>
>>> - reword bitmap_directory_size description
>>> - bitmap type: make 0 reserved
>>> - extra_data_size: resize to 4bytes
>>>Also, I've marked this field as "must be zero". We can always change
>>>it, if we decide allowing managing app to specify any extra data, by
>>>defining some magic value as a top of user extra data.. So, for now
>>>non zeor extra_data_size should be considered as an error.
>>> - swap name and extra_data to give good alignment to extra_data.
>>>
>>>
>>> v5:
>>>
>>> - 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of
>>>bitmaps.
>>> - rewordings
>>> - move upper bounds to "Notes about Qemu limits"
>>> - s/should/must somewhere. (but not everywhere)
>>> - move name_size field closer to name itself in bitmap header
>>> - add extra data area to bitmap header
>>> - move bitmap data description to separate section
>>>
>>>   docs/specs/qcow2.txt | 172
>>> ++-
>>>   1 file changed, 171 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
>>> index 121dfc8..997239d 100644
>>> --- a/docs/specs/qcow2.txt
>>> +++ b/docs/specs/qcow2.txt
>>> @@ -103,7 +103,18 @@ in the description of a field.
>>>   write to an image with unknown auto-clear
>>> features if it
>>>   clears the respective bits from this field first.
>>>   -Bits 0-63:  Reserved (set to 0)
>>> +Bit 0:  Bitmaps extension bit
>>> +This bit indicates consistency for
>>> the bitmaps
>>> +extension data.
>>> +
>>> +It is an error if this bit is set
>>> without the
>>> +bitmaps extension present.
>>> +
>>> +If the bitmaps extension is present
>>> but this
>>> +bit is unset, the bitmaps extension
>>> data is
>>> +inconsistent.
>>> +
>>> +Bits 1-63:  Reserved (set to 0)
>>>  96 -  99:  refcount_order
>>>   Describes the width of a reference count block
>>> entry (width
>>> @@ -123,6 +134,7 @@ be stored. Each extension has a structure like
>>> the following:
>>>   0x - End of the header extension area
>>>   0xE2792ACA - Backing file format name
>>>   0x6803f857 - Feature name table
>>> +0x23852875 - Bitmaps extension
>>>   other  - Unknown header extension, can
>>> be safely
>>>ignored
>>>   @@ -166,6 +178,34 @@ the header extension data. Each entry look
>>> like this:
>>>   terminated if it has full length)
>>> +== Bitmaps extension ==
>>> +
>>> +The bitmaps extension is an optional header extension. It provides
>>> the ability
>>> +to store bitmaps related to a virtual disk. For now, there is only
>>> one bitmap
>>> +type: the dirty tracking bitmap, which tracks virtual disk changes
>>> from some
>>> +point in time.
>>> +
>>> +The data of the extension should be considered consistent only if the
>>> +corresponding auto-clear feature bit is set, see autoclear_features
>>> above.
>>> +
>>> +The fields of the bitmaps extension are:
>>> +
>>> +  0 -  3:  nb_bitmaps
>>> +   The number of bitmaps contained in the image.
>>> Must be
>>> +   greater than or equal to 1.
>>> +
>>> +   Note: Qemu currently only supports up to 65535
>>> bitmaps per
>>> +   image.
>>> +
>>> +  4 -  7:  bitmap_directory_size
>>> +   Size of the bitmap directory in bytes. It is the
>>> cumulative
>>> +   size of all 

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[John Snow]

On 01/14/2016 05:08 PM, Eric Blake wrote:
> On 01/11/2016 06:05 AM, Vladimir Sementsov-Ogievskiy wrote:
>> The new feature for qcow2: storing bitmaps.
>>
>> This patch adds new header extension to qcow2 - Bitmaps Extension. It
>> provides an ability to store virtual disk related bitmaps in a qcow2
>> image. For now there is only one type of such bitmaps: Dirty Tracking
>> Bitmap, which just tracks virtual disk changes from some moment.
>>
>> Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
>> should be stored in this qcow2 file. The size of each bitmap
>> (considering its granularity) is equal to virtual disk size.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy 
>> ---
>>
> 
>> @@ -166,6 +178,34 @@ the header extension data. Each entry look like this:
>>  terminated if it has full length)
>>  
>>  
>> +== Bitmaps extension ==
> 
>> +  0 -  3:  nb_bitmaps
>> +   The number of bitmaps contained in the image. Must be
>> +   greater than or equal to 1.
>> +
>> +   Note: Qemu currently only supports up to 65535 bitmaps 
>> per
>> +   image.
>> +
>> +  4 -  7:  bitmap_directory_size
>> +   Size of the bitmap directory in bytes. It is the 
>> cumulative
>> +   size of all (nb_bitmaps) bitmap headers.
> 
> Only 4 bytes - if we ever raise our 64k entry restriction (nb_bitmaps),
> could we run into an image that has so many directory entries as to make
> the directory itself spill past 4G?  But I don't think it is likely, so
> I can live with your choice.
> 

"We'll never need this!"

I hope someone in 2082 is reading this right now and is quite angry.

(But really, I can't foresee needing this many per each drive -- and if
we do, we have external storage mechanisms in development to handle such
wild cases.)

>> +
>> +== Bitmaps ==
>> +
>> +As mentioned above, the bitmaps extension provides the ability to store 
>> bitmaps
>> +related a virtual disk. This section describes how these bitmaps are stored.
>> +
>> +Note: all bitmaps are related to the virtual disk stored in this image.
>> +
>> +=== Bitmap directory ===
>> +
>> +Each bitmap saved in the image is described in a bitmap directory entry. The
>> +bitmap directory is a contiguous area in the image file, whose starting 
>> offset
>> +and length are given by the header extension fields bitmap_directory_offset 
>> and
>> +bitmap_directory_size. The entries of the bitmap directory have variable
>> +length, depending on the length of the bitmap name and extra data. These
>> +entries are also called bitmap headers.
>> +
>> +Structure of a bitmap directory entry:
>> +
>> +Byte 0 -  7:bitmap_table_offset
>> +Offset into the image file at which the bitmap table
>> +(described below) for the bitmap starts. Must be 
>> aligned to
>> +a cluster boundary.
>> +
>> + 8 - 11:bitmap_table_size
>> +Number of entries in the bitmap table of the bitmap.
> 
> Should this be the size in bytes, instead of the number of entries? But

For what benefit? We can calculate either from the other, and this gives
us a better resolution.

> at least the entries are fixed width of 8 bytes each, so this lets you
> get a bitmap table up to 32G bytes rather than just 4G in size.  (Let's
> see here - if we have 32G bytes in the bitmap table, that means 4G
> clusters occupied by the bitmap itself; in the worst case of 512-byte
> clusters and granularity 0, that is a maximum bitmap size of 2T
> describing 16T of virtual guest image; with larger cluster size and/or
> larger granularity, we cover a lot more virtual guest space with less
> bitmap size; so I guess we aren't too worried about running out of space?).
> 

Yes, worst case of g=0 and cluster size of 512 bytes, we can get 2T
bitmaps describing 16T of virtual data.

"default case" of 64K clusters and 64K granularity: 256TiB bitmaps
describing ... let's see ... if my math is right, 128EiB?

We're probably fine :)

(Cue future space-person from 2159 wondering how I could have ever been
so naive. Sorry, future space-person!)

>> +20 - 23:extra_data_size
>> +Size of type-specific extra data.
>> +
>> +For now, as no extra data is defined, extra_data_size is
>> +reserved and must be zero.
>> +
>> +variable:   Type-specific extra data for the bitmap.
> 
> I'd write this as:
>variable:   extra_data
>Type-specific extra data for the bitmap,
>occupying extra_data_size bytes.
> 
>> +
>> +variable:   The name of the bitmap (not null terminated). Must be
>> +unique among all bitmap names within the bitmaps 
>> extension.
>> +
>> +variable:   Padding to round up the bitmap directory entry size to 
>> the

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[Eric Blake]

On 01/11/2016 06:05 AM, Vladimir Sementsov-Ogievskiy wrote:
> The new feature for qcow2: storing bitmaps.
> 
> This patch adds new header extension to qcow2 - Bitmaps Extension. It
> provides an ability to store virtual disk related bitmaps in a qcow2
> image. For now there is only one type of such bitmaps: Dirty Tracking
> Bitmap, which just tracks virtual disk changes from some moment.
> 
> Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
> should be stored in this qcow2 file. The size of each bitmap
> (considering its granularity) is equal to virtual disk size.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 
> ---
> 

> @@ -166,6 +178,34 @@ the header extension data. Each entry look like this:
>  terminated if it has full length)
>  
>  
> +== Bitmaps extension ==

> +  0 -  3:  nb_bitmaps
> +   The number of bitmaps contained in the image. Must be
> +   greater than or equal to 1.
> +
> +   Note: Qemu currently only supports up to 65535 bitmaps per
> +   image.
> +
> +  4 -  7:  bitmap_directory_size
> +   Size of the bitmap directory in bytes. It is the 
> cumulative
> +   size of all (nb_bitmaps) bitmap headers.

Only 4 bytes - if we ever raise our 64k entry restriction (nb_bitmaps),
could we run into an image that has so many directory entries as to make
the directory itself spill past 4G?  But I don't think it is likely, so
I can live with your choice.

> +
> +== Bitmaps ==
> +
> +As mentioned above, the bitmaps extension provides the ability to store 
> bitmaps
> +related a virtual disk. This section describes how these bitmaps are stored.
> +
> +Note: all bitmaps are related to the virtual disk stored in this image.
> +
> +=== Bitmap directory ===
> +
> +Each bitmap saved in the image is described in a bitmap directory entry. The
> +bitmap directory is a contiguous area in the image file, whose starting 
> offset
> +and length are given by the header extension fields bitmap_directory_offset 
> and
> +bitmap_directory_size. The entries of the bitmap directory have variable
> +length, depending on the length of the bitmap name and extra data. These
> +entries are also called bitmap headers.
> +
> +Structure of a bitmap directory entry:
> +
> +Byte 0 -  7:bitmap_table_offset
> +Offset into the image file at which the bitmap table
> +(described below) for the bitmap starts. Must be aligned 
> to
> +a cluster boundary.
> +
> + 8 - 11:bitmap_table_size
> +Number of entries in the bitmap table of the bitmap.

Should this be the size in bytes, instead of the number of entries? But
at least the entries are fixed width of 8 bytes each, so this lets you
get a bitmap table up to 32G bytes rather than just 4G in size.  (Let's
see here - if we have 32G bytes in the bitmap table, that means 4G
clusters occupied by the bitmap itself; in the worst case of 512-byte
clusters and granularity 0, that is a maximum bitmap size of 2T
describing 16T of virtual guest image; with larger cluster size and/or
larger granularity, we cover a lot more virtual guest space with less
bitmap size; so I guess we aren't too worried about running out of space?).

> +20 - 23:extra_data_size
> +Size of type-specific extra data.
> +
> +For now, as no extra data is defined, extra_data_size is
> +reserved and must be zero.
> +
> +variable:   Type-specific extra data for the bitmap.

I'd write this as:
   variable:   extra_data
   Type-specific extra data for the bitmap,
   occupying extra_data_size bytes.

> +
> +variable:   The name of the bitmap (not null terminated). Must be
> +unique among all bitmap names within the bitmaps 
> extension.
> +
> +variable:   Padding to round up the bitmap directory entry size to 
> the
> +next multiple of 8.

Should we require the padding to be all NUL bytes?  (We aren't
consistent on whether we require that for other locations of padding in
the spec, so that could be a followup patch).

> +
> +=== Bitmap table ===
> +
> +Bitmaps are stored using a one-level structure (as opposed to two-level
> +structure like for refcounts and guest clusters mapping) for the mapping of
> +bitmap data to host clusters. This structure is called the bitmap table.

Possible wording tweak:
Bitmaps are stored using a one-level structure (as opposed to the
two-level structures for refcounts and guest cluster mapping), and are
used for the mapping of bitmap data to host clusters

> +
> +Each bitmap table has a variable size (stored in the bitmap directory Entry)

Does 'Entry' still need to be capitalized?

> +and may use multiple clusters, however, it must be 

Re: [Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[John Snow]

On 01/11/2016 08:05 AM, Vladimir Sementsov-Ogievskiy wrote:
> The new feature for qcow2: storing bitmaps.
> 
> This patch adds new header extension to qcow2 - Bitmaps Extension. It
> provides an ability to store virtual disk related bitmaps in a qcow2
> image. For now there is only one type of such bitmaps: Dirty Tracking
> Bitmap, which just tracks virtual disk changes from some moment.
> 
> Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
> should be stored in this qcow2 file. The size of each bitmap
> (considering its granularity) is equal to virtual disk size.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy 
> ---
> 
> v7:
> 
> - Rewordings, grammar.
>   Max, Eric, John, thank you very much.
> 
> - add last paragraph: remaining bits in bitmap data clusters must be
>   zero.
> 
> - s/Bitmap Directory/bitmap directory/ and other names like this at
>   the request of Max.
> 
> v6:
> 
> - reword bitmap_directory_size description
> - bitmap type: make 0 reserved
> - extra_data_size: resize to 4bytes
>   Also, I've marked this field as "must be zero". We can always change
>   it, if we decide allowing managing app to specify any extra data, by
>   defining some magic value as a top of user extra data.. So, for now
>   non zeor extra_data_size should be considered as an error.
> - swap name and extra_data to give good alignment to extra_data.
> 
> 
> v5:
> 
> - 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of
>   bitmaps.
> - rewordings
> - move upper bounds to "Notes about Qemu limits"
> - s/should/must somewhere. (but not everywhere)
> - move name_size field closer to name itself in bitmap header
> - add extra data area to bitmap header
> - move bitmap data description to separate section
> 
>  docs/specs/qcow2.txt | 172 
> ++-
>  1 file changed, 171 insertions(+), 1 deletion(-)
> 
> diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
> index 121dfc8..997239d 100644
> --- a/docs/specs/qcow2.txt
> +++ b/docs/specs/qcow2.txt
> @@ -103,7 +103,18 @@ in the description of a field.
>  write to an image with unknown auto-clear features if it
>  clears the respective bits from this field first.
>  
> -Bits 0-63:  Reserved (set to 0)
> +Bit 0:  Bitmaps extension bit
> +This bit indicates consistency for the 
> bitmaps
> +extension data.
> +
> +It is an error if this bit is set without the
> +bitmaps extension present.
> +
> +If the bitmaps extension is present but this
> +bit is unset, the bitmaps extension data is
> +inconsistent.
> +
> +Bits 1-63:  Reserved (set to 0)
>  
>   96 -  99:  refcount_order
>  Describes the width of a reference count block entry 
> (width
> @@ -123,6 +134,7 @@ be stored. Each extension has a structure like the 
> following:
>  0x - End of the header extension area
>  0xE2792ACA - Backing file format name
>  0x6803f857 - Feature name table
> +0x23852875 - Bitmaps extension
>  other  - Unknown header extension, can be safely
>   ignored
>  
> @@ -166,6 +178,34 @@ the header extension data. Each entry look like this:
>  terminated if it has full length)
>  
>  
> +== Bitmaps extension ==
> +
> +The bitmaps extension is an optional header extension. It provides the 
> ability
> +to store bitmaps related to a virtual disk. For now, there is only one bitmap
> +type: the dirty tracking bitmap, which tracks virtual disk changes from some
> +point in time.
> +
> +The data of the extension should be considered consistent only if the
> +corresponding auto-clear feature bit is set, see autoclear_features above.
> +
> +The fields of the bitmaps extension are:
> +
> +  0 -  3:  nb_bitmaps
> +   The number of bitmaps contained in the image. Must be
> +   greater than or equal to 1.
> +
> +   Note: Qemu currently only supports up to 65535 bitmaps per
> +   image.
> +
> +  4 -  7:  bitmap_directory_size
> +   Size of the bitmap directory in bytes. It is the 
> cumulative
> +   size of all (nb_bitmaps) bitmap headers.
> +
> +  8 - 15:  bitmap_directory_offset
> +   Offset into the image file at which the bitmap directory
> +   starts. Must be aligned to a cluster boundary.
> +
> +
>  == Host cluster management ==
>  
>  qcow2 manages the allocation of host clusters by maintaining a reference 
> 

[Qemu-devel] [PATCH v7] spec: add qcow2 bitmaps extension specification

[Vladimir Sementsov-Ogievskiy]

The new feature for qcow2: storing bitmaps.

This patch adds new header extension to qcow2 - Bitmaps Extension. It
provides an ability to store virtual disk related bitmaps in a qcow2
image. For now there is only one type of such bitmaps: Dirty Tracking
Bitmap, which just tracks virtual disk changes from some moment.

Note: Only bitmaps, relative to the virtual disk, stored in qcow2 file,
should be stored in this qcow2 file. The size of each bitmap
(considering its granularity) is equal to virtual disk size.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---

v7:

- Rewordings, grammar.
  Max, Eric, John, thank you very much.

- add last paragraph: remaining bits in bitmap data clusters must be
  zero.

- s/Bitmap Directory/bitmap directory/ and other names like this at
  the request of Max.

v6:

- reword bitmap_directory_size description
- bitmap type: make 0 reserved
- extra_data_size: resize to 4bytes
  Also, I've marked this field as "must be zero". We can always change
  it, if we decide allowing managing app to specify any extra data, by
  defining some magic value as a top of user extra data.. So, for now
  non zeor extra_data_size should be considered as an error.
- swap name and extra_data to give good alignment to extra_data.


v5:

- 'Dirty bitmaps' renamed to 'Bitmaps', as we may have several types of
  bitmaps.
- rewordings
- move upper bounds to "Notes about Qemu limits"
- s/should/must somewhere. (but not everywhere)
- move name_size field closer to name itself in bitmap header
- add extra data area to bitmap header
- move bitmap data description to separate section

 docs/specs/qcow2.txt | 172 ++-
 1 file changed, 171 insertions(+), 1 deletion(-)

diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
index 121dfc8..997239d 100644
--- a/docs/specs/qcow2.txt
+++ b/docs/specs/qcow2.txt
@@ -103,7 +103,18 @@ in the description of a field.
 write to an image with unknown auto-clear features if it
 clears the respective bits from this field first.
 
-Bits 0-63:  Reserved (set to 0)
+Bit 0:  Bitmaps extension bit
+This bit indicates consistency for the bitmaps
+extension data.
+
+It is an error if this bit is set without the
+bitmaps extension present.
+
+If the bitmaps extension is present but this
+bit is unset, the bitmaps extension data is
+inconsistent.
+
+Bits 1-63:  Reserved (set to 0)
 
  96 -  99:  refcount_order
 Describes the width of a reference count block entry (width
@@ -123,6 +134,7 @@ be stored. Each extension has a structure like the 
following:
 0x - End of the header extension area
 0xE2792ACA - Backing file format name
 0x6803f857 - Feature name table
+0x23852875 - Bitmaps extension
 other  - Unknown header extension, can be safely
  ignored
 
@@ -166,6 +178,34 @@ the header extension data. Each entry look like this:
 terminated if it has full length)
 
 
+== Bitmaps extension ==
+
+The bitmaps extension is an optional header extension. It provides the ability
+to store bitmaps related to a virtual disk. For now, there is only one bitmap
+type: the dirty tracking bitmap, which tracks virtual disk changes from some
+point in time.
+
+The data of the extension should be considered consistent only if the
+corresponding auto-clear feature bit is set, see autoclear_features above.
+
+The fields of the bitmaps extension are:
+
+  0 -  3:  nb_bitmaps
+   The number of bitmaps contained in the image. Must be
+   greater than or equal to 1.
+
+   Note: Qemu currently only supports up to 65535 bitmaps per
+   image.
+
+  4 -  7:  bitmap_directory_size
+   Size of the bitmap directory in bytes. It is the cumulative
+   size of all (nb_bitmaps) bitmap headers.
+
+  8 - 15:  bitmap_directory_offset
+   Offset into the image file at which the bitmap directory
+   starts. Must be aligned to a cluster boundary.
+
+
 == Host cluster management ==
 
 qcow2 manages the allocation of host clusters by maintaining a reference count
@@ -360,3 +400,133 @@ Snapshot table entry:
 
 variable:   Padding to round up the snapshot table entry size to the
 next multiple of 8.
+
+
+== Bitmaps ==
+
+As mentioned above, the bitmaps extension provides the ability to store bitmaps
+related a virtual disk. This section describes