On 15.09.2015 19:24, Eric Blake wrote:
On 09/05/2015 10:43 AM, Vladimir Sementsov-Ogievskiy wrote:
Persistent dirty bitmaps will be saved into qcow2 files. It may be used
as 'internal' bitmaps (for qcow2 drives) or as 'external' bitmaps for
other drives (there may be qcow2 file with zero disk size but with
several dirty bitmaps for other drives).
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsement...@virtuozzo.com>
---
docs/specs/qcow2.txt | 127 ++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 126 insertions(+), 1 deletion(-)
diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
index 121dfc8..5fc0365 100644
--- a/docs/specs/qcow2.txt
+++ b/docs/specs/qcow2.txt
@@ -103,7 +103,13 @@ in the description of a field.
write to an image with unknown auto-clear features if it
clears the respective bits from this field first.
- Bits 0-63: Reserved (set to 0)
+ Bit 0: Dirty bitmaps bit. If this bit is set then
+ there is a _consistent_ Dirty bitmaps extension
+ in the image. If it is not set, but there is a
+ Dirty bitmaps extension, its data should be
+ considered as inconsistent.
Thanks for documenting this. I don't know that we use underscore for
_emphasis_ anywhere else in the file, but I don't have any better
suggestions. Should you also require that it is an error if this bit is
set but no Dirty bitmap extension header is present?
There should be an error, if there are cmd flags which tries to load the
bitmap. If bitmap loading is not loading, would not a warning be enough?
+
+ Bits 1-63: Reserved (set to 0)
96 - 99: refcount_order
Describes the width of a reference count block entry
(width
@@ -123,6 +129,7 @@ be stored. Each extension has a structure like the
following:
0x00000000 - End of the header extension area
0xE2792ACA - Backing file format name
0x6803f857 - Feature name table
+ 0x23852875 - Dirty bitmaps
other - Unknown header extension, can be safely
ignored
@@ -166,6 +173,24 @@ the header extension data. Each entry look like this:
terminated if it has full length)
+== Dirty bitmaps ==
+
+Dirty bitmaps is an optional header extension. It provides an ability to store
+dirty bitmaps in a qcow2 image. The fields are:
Might not hurt to remind the reader about the auto-clear feature bit
mentioned earlier controlling whether this extension can be trusted as
consistent.
agree, will add.
+
+ 0 - 3: nb_dirty_bitmaps
+ The number of dirty bitmaps contained in the image. Valid
+ values: 0 - 65535.
+
+ 4 - 7: dirty_bitmap_directory_size
+ Size of the Dirty Bitmap Directory in bytes. Valid values:
+ 0 - 67108864 (= 1024 * nb_dirty_bitmaps).
Is it always going to be 1024 * nb_dirty_bitmaps? If so, why do we need
a redundant field? If not, then this wording needs help; from the rest
of this text, it looks like you want "at most 1024 * nb_dirty_bitmaps".
Also, while Dirty Bitmap Directory entries are variable length (and
thus a variable maximum), they do have a minimum size (so the minimum
value for dirty_bitmap_directory_size must be larger than 0 unless
nb_dirty_bitmaps is 0, in which case why would we have this header
extension)
Yes, strange mistake.
actually, it shoud be
<= (round_up_to_8byte_boundary(sizeof(dirty bitmap header) +
max_dirty_bitmap_name)) * nb_dirty_bitmaps
and
> (round_up_to_8byte_boundary(sizeof(dirty bitmap header) +
min_dirty_bitmap_name)) * nb_dirty_bitmaps
So, what is better to leave there, these formulas, or take max and min
of nb_dirty_bitmaps?
+
+ 8 - 15: dirty_bitmap_directory_offset
+ Offset into the image file at which the Dirty Bitmap
+ Directory starts. Must be aligned to a cluster boundary.
+
+
== Host cluster management ==
qcow2 manages the allocation of host clusters by maintaining a reference count
@@ -360,3 +385,103 @@ Snapshot table entry:
variable: Padding to round up the snapshot table entry size to the
next multiple of 8.
+
+
+== Dirty bitmaps ==
+
+The feature supports storing dirty bitmaps in a qcow2 image.
+
+=== Cluster mapping ===
+
+Dirty bitmaps are stored using a ONE-level structure for the mapping of
+bitmaps to host clusters. It is called Dirty Bitmap Table.
s/ONE/one/ (I didn't see the reason for the emphasis)
+
+The Dirty Bitmap Table has a variable size (stored in the Dirty Bitmap
s/The/Each/
+Directory Entry) and may use multiple clusters, however it must be contiguous
+in the image file.
+
+Given an offset (in bytes) into the bitmap, the offset into the image file can
+be obtained as follows:
+
+ byte_offset =
+ dirty_bitmap_table[offset / cluster_size] + (offset % cluster_size)
+
+Taking into accout the granularity of the bitmap, an offset in bits into the
s/accout/account/
+image file can be obtained like this:
+
+ bit_offset =
+ byte_offset(bit_nr / granularity / 8) * 8 + (bit_nr / granularity) % 8
+
+Here bit_nr is a number of "virtual" bit of the bitmap, which is covered by
+"physical" bit with number (bit_nr / granularity).
I got a bit lost on this sentence. Maybe an example would help? Is the
idea that every image has a certain number of clusters, one "virtual"
bit per cluster, and then the bitmap compresses multiple clusters into
one "physical" bit according to a compression ratio determined by the
bitmap granularity? That is, if I have an image with 64k clusters but
128k bitmap granularity, then each physical bit of the bitmap covers 2
clusters as being dirty?
You are right except that there are not clusters but sectors. For bdrv
dirty bitmaps, viratual bits are corresponding to sectors. But there I'm
trying to abstract from bdrv dirty bitmap and to store any dirty bitmap
with granularity. It may be better to not call it dirty, but just a
bitmap.. But I'm not sure.. JustABitmap is too general.
+
+Dirty Bitmap Table entry:
+
+ Bit 0 - 8: Reserved
s/Reserved/Reserved, must be 0/
+
+ 9 - 55: Bits 9-55 of host cluster offset. Must be aligned to a
+ cluster boundary. If the offset is 0, the cluster is
+ unallocated, and should be read as all zeros.
+
+ 56 - 63: Reserved
and again (specifying the user must write 0 for now leaves the door open
for extension)
+
+=== Dirty Bitmap Directory ===
+
+Each dirty bitmap, saved in the image is described in the Dirty Bitmap
s/bitmap,/bitmap/
s/in the Dirty/in a Dirty/
+Directory entry. Dirty Bitmap Directory is a contiguous area in the image file,
+whose starting offset and length are given by the header extension fields
+dirty_bitmap_directory_offset and dirty_bitmap_directory_size. The entries of
+the bitmap directory have variable length, depending on the length of the
+bitmap name.
+
+Dirty Bitmap Directory Entry:
+
+ Byte 0 - 7: dirty_bitmap_table_offset
+ Offset into the image file at which the Dirty Bitmap Table
+ for the bitmap starts. Must be aligned to a cluster
+ boundary.
+
+ 8 - 15: nb_virtual_bits
+ Number of "virtual" bits in the bitmap. Number of
+ "physical" bits would be:
+ (nb_virtual_bits + granularity - 1) / granularity
+
+ 16 - 19: dirty_bitmap_table_size
+ Number of entries in the Dirty Bitmap Table of the bitmap.
+ Valid values: 0 - 0x8000000.
+ Also, (dirty_bitmap_table_size * cluster_size) should not
+ be greater than 0x20000000 (512 MB)
+
+ 20 - 23: granularity_bits
+ Granularity bits. Valid values are: 0 - 63.
+
+ Granularity is calculated as
+ granularity = 1 << granularity_bits
63 seems like a rather high limit. Even 32 (1 bit covering 4 billion
clusters) is huge.
+
+ Granularity of the bitmap is how many "virtual" bits
+ accounts for one "physical" bit.
+
+ 24 - 27: flags
+ Bit
+ 0: in_use
+ The bitmap is in use and may be inconsistent.
+
+ 1: self
+ The bitmap is a dirty bitmap for the containing image.
+
+ 2: auto
+ The bitmap should be autoloaded as block dirty bitmap.
+ Only available if bit 1 (self) is set.
+
+ 3: read_only
+ The bitmap should not be rewritten.
+
+ Bits 4 - 31 are reserved.
+
+ 28 - 29: name_size
+ Size of the bitmap name. Valid values: 0 - 1023.
This limit is inconsistent with the limit above that a directory entry
will be at most 1024 bytes; since you have already burned 30 bytes on
essential information. Is 0 allowed (a bitmap with "" as its name), or
must the name_size be at least 1? Should you document constraints that
each bitmap name should be unique within the file (that is, no two
bitmaps in the directory have the same name)?
Ok. And it is a real question: should we allow zero-length names for
bitmaps?
+
+ variable: The name of the bitmap (not null terminated).
+
+ variable: Padding to round up the Dirty Bitmap Directory Entry size
to
+ the next multiple of 8.
--
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.
