Re: [Qemu-block] [Qemu-devel] [PATCH] block: modify top-id's comments

[wangweiwei]

Ok, I'wll wrap it.
On 09/29/2016 09:35 PM, Eric Blake wrote:

On 09/28/2016 05:31 AM, Stefan Hajnoczi wrote:

On Wed, Sep 28, 2016 at 01:30:39PM +0800, Wang WeiWei wrote:

Please send a proper commit message and commit description.  "modify
top-id's comments" says almost nothing and there is no commit
description.  I don't know why you are making this change so I can't
review and apply it.




+++ b/qapi/block-core.json
@@ -2184,7 +2184,7 @@
 # @mode: the replication mode
 #
 # @top-id: #optional In secondary mode, node name or device ID of the root
-#  node who owns the replication node chain. Ignored in primary mode.
+#  node who owns the replication node chain. Must not be given in 
primary mode.


That, and your line exceeds 80 columns; please wrap it.

[Qemu-block] [Qemu-devel] [PATCH] block: modify top-id's comments

[Wang WeiWei]

Kevin Wolf's advice:
http://lists.gnu.org/archive/html/qemu-devel/2016-09/msg02391.html

Signed-off-by: Wen Congyang 
Signed-off-by: Changlong Xie 
Signed-off-by: Wang WeiWei 
Signed-off-by: zhanghailiang 
Signed-off-by: Gonglei 
---
 qapi/block-core.json | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index ada3202..a605107 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -2184,7 +2184,8 @@
 # @mode: the replication mode
 #
 # @top-id: #optional In secondary mode, node name or device ID of the root
-#  node who owns the replication node chain. Ignored in primary mode.
+#  node who owns the replication node chain. Must not be given in
+#  primary mode.
 #
 # Since: 2.8
 ##
-- 
2.7.4

Re: [Qemu-block] [Qemu-devel] [PATCH] block: modify top-id's comments

[Kashyap Chamarthy]

On Fri, Sep 30, 2016 at 04:53:38PM +0800, Wang WeiWei wrote:
> Kevin Wolf's advice:
> http://lists.gnu.org/archive/html/qemu-devel/2016-09/msg02391.html

IIUC, what Stefan meant is to write a simple descriptive commit message
that explains why this change is necessary -- the above URL might have
context, but having it spelled out in the Git commit message will be
useful when someone is examining Git history offline.

I think Stefan was looking for something like (came up with this after
reading the above thread):

  QAPI: BlockdevOptionsReplication: Clarify 'top-id' parameter usage

  In primary mode, the replication driver will ignore 'top-id'
  parameter, thus it must not be supplied.  Explicitly spell that out in
  the QAPI schema for BlockdevOptionsReplication.

> Signed-off-by: Wen Congyang 
> Signed-off-by: Changlong Xie 
> Signed-off-by: Wang WeiWei 
> Signed-off-by: zhanghailiang 
> Signed-off-by: Gonglei 
> ---
>  qapi/block-core.json | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/qapi/block-core.json b/qapi/block-core.json
> index ada3202..a605107 100644
> --- a/qapi/block-core.json
> +++ b/qapi/block-core.json
> @@ -2184,7 +2184,8 @@
>  # @mode: the replication mode
>  #
>  # @top-id: #optional In secondary mode, node name or device ID of the root
> -#  node who owns the replication node chain. Ignored in primary mode.
> +#  node who owns the replication node chain. Must not be given in
> +#  primary mode.
>  #
>  # Since: 2.8
>  ##
> -- 
> 2.7.4
> 
> 
> 
> 

-- 
/kashyap

Re: [Qemu-block] [Qemu-devel] [PATCH v2] block: modify top-id's comments

[Kevin Wolf]

This is v2, should be included in the subject line.

Am 30.09.2016 um 10:53 hat Wang WeiWei geschrieben:
> Kevin Wolf's advice:
> http://lists.gnu.org/archive/html/qemu-devel/2016-09/msg02391.html
> 
> Signed-off-by: Wen Congyang 
> Signed-off-by: Changlong Xie 
> Signed-off-by: Wang WeiWei 
> Signed-off-by: zhanghailiang 
> Signed-off-by: Gonglei 
> ---
>  qapi/block-core.json | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/qapi/block-core.json b/qapi/block-core.json
> index ada3202..a605107 100644
> --- a/qapi/block-core.json
> +++ b/qapi/block-core.json
> @@ -2184,7 +2184,8 @@
>  # @mode: the replication mode
>  #
>  # @top-id: #optional In secondary mode, node name or device ID of the root
> -#  node who owns the replication node chain. Ignored in primary mode.
> +#  node who owns the replication node chain. Must not be given in
> +#  primary mode.
>  #
>  # Since: 2.8
>  ##

As I already wrote for v1:

The matching code in block/replication.c is missing to actually error
out if the option is given in primary mode.

Kevin

Re: [Qemu-block] [Qemu-devel] [PATCH v2] block: modify top-id's comments

[wangweiwei]

Ok,thanks.
On 09/30/2016 05:52 PM, Kevin Wolf wrote:

This is v2, should be included in the subject line.

Am 30.09.2016 um 10:53 hat Wang WeiWei geschrieben:

Kevin Wolf's advice:
http://lists.gnu.org/archive/html/qemu-devel/2016-09/msg02391.html

Signed-off-by: Wen Congyang 
Signed-off-by: Changlong Xie 
Signed-off-by: Wang WeiWei 
Signed-off-by: zhanghailiang 
Signed-off-by: Gonglei 
---
 qapi/block-core.json | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index ada3202..a605107 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -2184,7 +2184,8 @@
 # @mode: the replication mode
 #
 # @top-id: #optional In secondary mode, node name or device ID of the root
-#  node who owns the replication node chain. Ignored in primary mode.
+#  node who owns the replication node chain. Must not be given in
+#  primary mode.
 #
 # Since: 2.8
 ##


As I already wrote for v1:

The matching code in block/replication.c is missing to actually error
out if the option is given in primary mode.

Kevin


.

Re: [Qemu-block] [Qemu-devel] [PATCH] block: modify top-id's comments

[Stefan Hajnoczi]

On Fri, Sep 30, 2016 at 11:48:44AM +0200, Kashyap Chamarthy wrote:
> On Fri, Sep 30, 2016 at 04:53:38PM +0800, Wang WeiWei wrote:
> > Kevin Wolf's advice:
> > http://lists.gnu.org/archive/html/qemu-devel/2016-09/msg02391.html
> 
> IIUC, what Stefan meant is to write a simple descriptive commit message
> that explains why this change is necessary -- the above URL might have
> context, but having it spelled out in the Git commit message will be
> useful when someone is examining Git history offline.
> 
> I think Stefan was looking for something like (came up with this after
> reading the above thread):
> 
>   QAPI: BlockdevOptionsReplication: Clarify 'top-id' parameter usage
> 
>   In primary mode, the replication driver will ignore 'top-id'
>   parameter, thus it must not be supplied.  Explicitly spell that out in
>   the QAPI schema for BlockdevOptionsReplication.

Yes.  Posting just a URL as the commit description is not enough.

I tried to explain the purpose of commit messages/descriptions in my
previous reply.

You can read a lot more about it elsewhere too:

http://chris.beams.io/posts/git-commit/
(The key points are the introduction and "7. Use the body to explain
what and why vs. how")

Stefan


signature.asc
Description: PGP signature

[Qemu-block] [PATCH 02/22] tests: add hbitmap iter test

[Vladimir Sementsov-Ogievskiy]

Test that hbitmap iter is resistant to bitmap resetting.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Signed-off-by: Denis V. Lunev 
---
 tests/test-hbitmap.c | 19 +++
 1 file changed, 19 insertions(+)

diff --git a/tests/test-hbitmap.c b/tests/test-hbitmap.c
index 0ed918c..e24377f 100644
--- a/tests/test-hbitmap.c
+++ b/tests/test-hbitmap.c
@@ -881,6 +881,22 @@ static void hbitmap_test_add(const char *testpath,
hbitmap_test_teardown);
 }
 
+static void test_hbitmap_iter_and_reset(TestHBitmapData *data,
+const void *unused)
+{
+HBitmapIter hbi;
+
+hbitmap_test_init(data, L1 * 2, 0);
+hbitmap_set(data->hb, 0, data->size);
+
+hbitmap_iter_init(&hbi, data->hb, BITS_PER_LONG - 1);
+
+hbitmap_iter_next(&hbi);
+
+hbitmap_reset_all(data->hb);
+hbitmap_iter_next(&hbi);
+}
+
 int main(int argc, char **argv)
 {
 g_test_init(&argc, &argv, NULL);
@@ -938,6 +954,9 @@ int main(int argc, char **argv)
  test_hbitmap_serialize_part);
 hbitmap_test_add("/hbitmap/serialize/zeroes",
  test_hbitmap_serialize_zeroes);
+
+hbitmap_test_add("/hbitmap/iter/iter_and_reset",
+ test_hbitmap_iter_and_reset);
 g_test_run();
 
 return 0;
-- 
1.8.3.1

[Qemu-block] [PATCH 01/22] hbitmap: improve dirty iter

[Vladimir Sementsov-Ogievskiy]

Make dirty iter resistant to resetting bits in corresponding HBitmap.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 include/qemu/hbitmap.h | 24 ++--
 util/hbitmap.c | 23 ++-
 2 files changed, 24 insertions(+), 23 deletions(-)

diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h
index eb46475..9aa86c1 100644
--- a/include/qemu/hbitmap.h
+++ b/include/qemu/hbitmap.h
@@ -243,10 +243,7 @@ void hbitmap_free(HBitmap *hb);
  * the lowest-numbered bit that is set in @hb, starting at @first.
  *
  * Concurrent setting of bits is acceptable, and will at worst cause the
- * iteration to miss some of those bits.  Resetting bits before the current
- * position of the iterator is also okay.  However, concurrent resetting of
- * bits can lead to unexpected behavior if the iterator has not yet reached
- * those bits.
+ * iteration to miss some of those bits. Concurrent bits resetting is ok too.
  */
 void hbitmap_iter_init(HBitmapIter *hbi, const HBitmap *hb, uint64_t first);
 
@@ -285,24 +282,7 @@ void hbitmap_free_meta(HBitmap *hb);
  * Return the next bit that is set in @hbi's associated HBitmap,
  * or -1 if all remaining bits are zero.
  */
-static inline int64_t hbitmap_iter_next(HBitmapIter *hbi)
-{
-unsigned long cur = hbi->cur[HBITMAP_LEVELS - 1];
-int64_t item;
-
-if (cur == 0) {
-cur = hbitmap_iter_skip_words(hbi);
-if (cur == 0) {
-return -1;
-}
-}
-
-/* The next call will resume work from the next bit.  */
-hbi->cur[HBITMAP_LEVELS - 1] = cur & (cur - 1);
-item = ((uint64_t)hbi->pos << BITS_PER_LEVEL) + ctzl(cur);
-
-return item << hbi->granularity;
-}
+int64_t hbitmap_iter_next(HBitmapIter *hbi);
 
 /**
  * hbitmap_iter_next_word:
diff --git a/util/hbitmap.c b/util/hbitmap.c
index 6a13c12..4f5cf96 100644
--- a/util/hbitmap.c
+++ b/util/hbitmap.c
@@ -106,8 +106,9 @@ unsigned long hbitmap_iter_skip_words(HBitmapIter *hbi)
 
 unsigned long cur;
 do {
-cur = hbi->cur[--i];
+i--;
 pos >>= BITS_PER_LEVEL;
+cur = hbi->cur[i] & hb->levels[i][pos];
 } while (cur == 0);
 
 /* Check for end of iteration.  We always use fewer than BITS_PER_LONG
@@ -139,6 +140,26 @@ unsigned long hbitmap_iter_skip_words(HBitmapIter *hbi)
 return cur;
 }
 
+int64_t hbitmap_iter_next(HBitmapIter *hbi)
+{
+unsigned long cur = hbi->cur[HBITMAP_LEVELS - 1] &
+hbi->hb->levels[HBITMAP_LEVELS - 1][hbi->pos];
+int64_t item;
+
+if (cur == 0) {
+cur = hbitmap_iter_skip_words(hbi);
+if (cur == 0) {
+return -1;
+}
+}
+
+/* The next call will resume work from the next bit.  */
+hbi->cur[HBITMAP_LEVELS - 1] = cur & (cur - 1);
+item = ((uint64_t)hbi->pos << BITS_PER_LEVEL) + ctzl(cur);
+
+return item << hbi->granularity;
+}
+
 void hbitmap_iter_init(HBitmapIter *hbi, const HBitmap *hb, uint64_t first)
 {
 unsigned i, bit;
-- 
1.8.3.1

[Qemu-block] [PATCH 20/22] qcow2-dirty-bitmap: refcounts

[Vladimir Sementsov-Ogievskiy]

Calculate refcounts for dirty bitmaps.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2-bitmap.c   | 60 ++
 block/qcow2-refcount.c | 14 +++-
 block/qcow2.h  |  5 +
 3 files changed, 74 insertions(+), 5 deletions(-)

diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index 76f7e2b..6d9394a 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -901,3 +901,63 @@ out:
 g_free(new_dir);
 g_free(dir);
 }
+
+int qcow2_dirty_bitmaps_refcounts(BlockDriverState *bs,
+  BdrvCheckResult *res,
+  void **refcount_table,
+  int64_t *refcount_table_size)
+{
+BDRVQcow2State *s = bs->opaque;
+uint8_t *dir;
+Qcow2BitmapDirEntry *e;
+Error *local_err = NULL;
+
+int i;
+int ret = inc_refcounts(bs, res, refcount_table, refcount_table_size,
+s->bitmap_directory_offset,
+s->bitmap_directory_size);
+if (ret < 0) {
+return ret;
+}
+
+dir = directory_read(bs, s->bitmap_directory_offset,
+ s->bitmap_directory_size, &local_err);
+if (dir == NULL) {
+error_report_err(local_err);
+return -EINVAL;
+}
+
+for_each_bitmap_dir_entry(e, dir, s->bitmap_directory_size) {
+uint64_t *bitmap_table = NULL;
+
+ret = inc_refcounts(bs, res, refcount_table, refcount_table_size,
+e->bitmap_table_offset,
+e->bitmap_table_size);
+if (ret < 0) {
+return ret;
+}
+
+ret = bitmap_table_load(bs, e, &bitmap_table);
+if (ret < 0) {
+return ret;
+}
+
+for (i = 0; i < e->bitmap_table_size; ++i) {
+uint64_t off = be64_to_cpu(bitmap_table[i]);
+if (off <= 1) {
+continue;
+}
+
+ret = inc_refcounts(bs, res, refcount_table, refcount_table_size,
+off, s->cluster_size);
+if (ret < 0) {
+g_free(bitmap_table);
+return ret;
+}
+}
+
+g_free(bitmap_table);
+}
+
+return 0;
+}
diff --git a/block/qcow2-refcount.c b/block/qcow2-refcount.c
index cbfb3fe..05bcc57 100644
--- a/block/qcow2-refcount.c
+++ b/block/qcow2-refcount.c
@@ -1309,11 +1309,9 @@ static int realloc_refcount_array(BDRVQcow2State *s, 
void **array,
  *
  * Modifies the number of errors in res.
  */
-static int inc_refcounts(BlockDriverState *bs,
- BdrvCheckResult *res,
- void **refcount_table,
- int64_t *refcount_table_size,
- int64_t offset, int64_t size)
+int inc_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
+  void **refcount_table, int64_t *refcount_table_size,
+  int64_t offset, int64_t size)
 {
 BDRVQcow2State *s = bs->opaque;
 uint64_t start, last, cluster_offset, k, refcount;
@@ -1843,6 +1841,12 @@ static int calculate_refcounts(BlockDriverState *bs, 
BdrvCheckResult *res,
 return ret;
 }
 
+/* dirty bitmaps */
+ret = qcow2_dirty_bitmaps_refcounts(bs, res, refcount_table, nb_clusters);
+if (ret < 0) {
+return ret;
+}
+
 return check_refblocks(bs, res, fix, rebuild, refcount_table, nb_clusters);
 }
 
diff --git a/block/qcow2.h b/block/qcow2.h
index a5e7592..0a050ea 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -563,6 +563,9 @@ int qcow2_check_metadata_overlap(BlockDriverState *bs, int 
ign, int64_t offset,
  int64_t size);
 int qcow2_pre_write_overlap_check(BlockDriverState *bs, int ign, int64_t 
offset,
   int64_t size);
+int inc_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
+  void **refcount_table, int64_t *refcount_table_size,
+  int64_t offset, int64_t size);
 
 int qcow2_change_refcount_order(BlockDriverState *bs, int refcount_order,
 BlockDriverAmendStatusCB *status_cb,
@@ -616,6 +619,8 @@ int qcow2_read_snapshots(BlockDriverState *bs);
 int qcow2_read_bitmaps(BlockDriverState *bs, Error **errp);
 
 int qcow2_delete_bitmaps(BlockDriverState *bs);
+int qcow2_dirty_bitmaps_refcounts(BlockDriverState *bs, BdrvCheckResult *res,
+void **refcount_table, int64_t *refcount_table_size);
 
 /* qcow2-cache.c functions */
 Qcow2Cache *qcow2_cache_create(BlockDriverState *bs, int num_tables);
-- 
1.8.3.1

[Qemu-block] [PATCH 15/22] qcow2-bitmap: add autoclear bit

[Vladimir Sementsov-Ogievskiy]

Add autoclear bit for handling rewriting image by old qemu version.

If autoclear bit is not set, but bitmaps extension is found it
would not be loaded and warning will be generated.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2-bitmap.c |  4 
 block/qcow2.c| 12 ++--
 block/qcow2.h|  9 +
 3 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index 2642afe..76f7e2b 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -515,6 +515,10 @@ static int directory_update(BlockDriverState *bs, uint8_t 
*new_dir,
 if (ret < 0) {
 goto fail;
 }
+
+s->autoclear_features |= QCOW2_AUTOCLEAR_DIRTY_BITMAPS;
+} else {
+s->autoclear_features &= ~(uint64_t)QCOW2_AUTOCLEAR_DIRTY_BITMAPS;
 }
 s->bitmap_directory_offset = new_offset;
 s->bitmap_directory_size = new_size;
diff --git a/block/qcow2.c b/block/qcow2.c
index aa967ed..373cf7e 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -165,6 +165,13 @@ static int qcow2_read_extensions(BlockDriverState *bs, 
uint64_t start_offset,
 break;
 
 case QCOW2_EXT_MAGIC_DIRTY_BITMAPS:
+if (!(s->autoclear_features & QCOW2_AUTOCLEAR_DIRTY_BITMAPS)) {
+fprintf(stderr,
+"WARNING: bitmaps_ext: autoclear flag is not "
+"set, all bitmaps will be considered as inconsistent");
+break;
+}
+
 ret = bdrv_pread(bs->file, offset, &bitmaps_ext, ext.len);
 if (ret < 0) {
 error_setg_errno(errp, -ret, "ERROR: bitmaps_ext: "
@@ -1206,8 +1213,9 @@ static int qcow2_open(BlockDriverState *bs, QDict 
*options, int flags,
 }
 
 /* Clear unknown autoclear feature bits */
-if (!bs->read_only && !(flags & BDRV_O_INACTIVE) && s->autoclear_features) 
{
-s->autoclear_features = 0;
+if (!bs->read_only && !(flags & BDRV_O_INACTIVE) &&
+(s->autoclear_features & ~QCOW2_AUTOCLEAR_MASK)) {
+s->autoclear_features &= QCOW2_AUTOCLEAR_MASK;
 ret = qcow2_update_header(bs);
 if (ret < 0) {
 error_setg_errno(errp, -ret, "Could not update qcow2 header");
diff --git a/block/qcow2.h b/block/qcow2.h
index af18efc..a5e7592 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -215,6 +215,15 @@ enum {
 QCOW2_COMPAT_FEAT_MASK= QCOW2_COMPAT_LAZY_REFCOUNTS,
 };
 
+/* Autoclear feature bits */
+enum {
+QCOW2_AUTOCLEAR_DIRTY_BITMAPS_BITNR = 0,
+QCOW2_AUTOCLEAR_DIRTY_BITMAPS   =
+1 << QCOW2_AUTOCLEAR_DIRTY_BITMAPS_BITNR,
+
+QCOW2_AUTOCLEAR_MASK= QCOW2_AUTOCLEAR_DIRTY_BITMAPS,
+};
+
 enum qcow2_discard_type {
 QCOW2_DISCARD_NEVER = 0,
 QCOW2_DISCARD_ALWAYS,
-- 
1.8.3.1

[Qemu-block] [PATCH 10/22] block/dirty-bitmap: add bdrv_dirty_bitmap_next()

[Vladimir Sementsov-Ogievskiy]

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 block/dirty-bitmap.c | 7 +++
 include/block/dirty-bitmap.h | 3 +++
 2 files changed, 10 insertions(+)

diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 0314581..392d660 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -582,3 +582,10 @@ bool bdrv_has_persistent_bitmaps(BlockDriverState *bs)
 
 return false;
 }
+
+BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,
+BdrvDirtyBitmap *bitmap)
+{
+return bitmap == NULL ? QLIST_FIRST(&bs->dirty_bitmaps) :
+QLIST_NEXT(bitmap, list);
+}
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 8dbd16b..d71edc4 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -81,6 +81,9 @@ void bdrv_dirty_bitmap_set_persistance(BdrvDirtyBitmap 
*bitmap,
 bool persistent);
 bool bdrv_dirty_bitmap_get_persistance(BdrvDirtyBitmap *bitmap);
 
+BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,
+BdrvDirtyBitmap *bitmap);
+
 bool bdrv_has_persistent_bitmaps(BlockDriverState *bs);
 
 #endif
-- 
1.8.3.1

[Qemu-block] [PATCH 19/22] iotests: test qcow2 persistent dirty bitmap

[Vladimir Sementsov-Ogievskiy]

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 tests/qemu-iotests/165 | 87 ++
 tests/qemu-iotests/165.out |  5 +++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 93 insertions(+)
 create mode 100755 tests/qemu-iotests/165
 create mode 100644 tests/qemu-iotests/165.out

diff --git a/tests/qemu-iotests/165 b/tests/qemu-iotests/165
new file mode 100755
index 000..a69799c
--- /dev/null
+++ b/tests/qemu-iotests/165
@@ -0,0 +1,87 @@
+#!/usr/bin/env python
+#
+# Tests for persistent dirty bitmaps.
+#
+# Copyright: Vladimir Sementsov-Ogievskiy 2015-2016
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
+#
+
+import os
+import iotests
+from iotests import qemu_img
+
+disk = os.path.join(iotests.test_dir, 'disk')
+disk_size = 0x4000 # 1G
+
+# regions for qemu_io: (start, count) in bytes
+regions1 = ((0,0x10),
+(0x20, 0x10))
+
+regions2 = ((0x1000, 0x2),
+(0x3999, 0x1))
+
+class TestPersistentDirtyBitmap(iotests.QMPTestCase):
+
+def setUp(self):
+qemu_img('create', '-f', iotests.imgfmt, disk, str(disk_size))
+
+def tearDown(self):
+os.remove(disk)
+
+def mkVm(self):
+return iotests.VM().add_drive(disk)
+
+def getMd5(self):
+result = self.vm.qmp('query-block');
+return result['return'][0]['dirty-bitmaps'][0]['md5']
+
+def checkBitmap(self, md5):
+result = self.vm.qmp('query-block');
+self.assert_qmp(result, 'return[0]/dirty-bitmaps[0]/md5', md5);
+
+def writeRegions(self, regions):
+for r in regions:
+  self.vm.hmp_qemu_io('drive0',
+  'write %d %d' % r)
+
+def qmpAddBitmap(self):
+self.vm.qmp('block-dirty-bitmap-add', node='drive0',
+name='bitmap0', persistent=True, autoload=True)
+
+def test_persistent(self):
+self.vm = self.mkVm()
+self.vm.launch()
+self.qmpAddBitmap()
+
+self.writeRegions(regions1)
+md5 = self.getMd5()
+
+self.vm.shutdown()
+self.vm = self.mkVm()
+self.vm.launch()
+
+self.checkBitmap(md5)
+self.writeRegions(regions2)
+md5 = self.getMd5()
+
+self.vm.shutdown()
+self.vm.launch()
+
+self.checkBitmap(md5)
+
+self.vm.shutdown()
+
+if __name__ == '__main__':
+iotests.main(supported_fmts=['qcow2'])
diff --git a/tests/qemu-iotests/165.out b/tests/qemu-iotests/165.out
new file mode 100644
index 000..ae1213e
--- /dev/null
+++ b/tests/qemu-iotests/165.out
@@ -0,0 +1,5 @@
+.
+--
+Ran 1 tests
+
+OK
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 503eb27..ed14294 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -161,3 +161,4 @@
 159 rw auto quick
 160 rw auto quick
 162 auto quick
+165 rw auto quick
-- 
1.8.3.1

[Qemu-block] [PATCH 09/22] block: introduce persistent dirty bitmaps

[Vladimir Sementsov-Ogievskiy]

New field BdrvDirtyBitmap.persistent means, that bitmap should be saved
on bdrv_close, using format driver. Format driver should maintain bitmap
storing.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 block.c  | 30 ++
 block/dirty-bitmap.c | 27 +++
 block/qcow2-bitmap.c |  1 +
 include/block/block.h|  2 ++
 include/block/block_int.h|  2 ++
 include/block/dirty-bitmap.h |  6 ++
 6 files changed, 68 insertions(+)

diff --git a/block.c b/block.c
index 804e3d4..1cde03a 100644
--- a/block.c
+++ b/block.c
@@ -2196,6 +2196,7 @@ void bdrv_reopen_abort(BDRVReopenState *reopen_state)
 static void bdrv_close(BlockDriverState *bs)
 {
 BdrvAioNotifier *ban, *ban_next;
+Error *local_err = NULL;
 
 assert(!bs->job);
 assert(!bs->refcnt);
@@ -2204,6 +2205,10 @@ static void bdrv_close(BlockDriverState *bs)
 bdrv_flush(bs);
 bdrv_drain(bs); /* in case flush left pending I/O */
 
+bdrv_store_persistent_bitmaps(bs, &local_err);
+if (local_err != NULL) {
+error_report_err(local_err);
+}
 bdrv_release_named_dirty_bitmaps(bs);
 assert(QLIST_EMPTY(&bs->dirty_bitmaps));
 
@@ -3969,3 +3974,28 @@ void bdrv_del_child(BlockDriverState *parent_bs, 
BdrvChild *child, Error **errp)
 
 parent_bs->drv->bdrv_del_child(parent_bs, child, errp);
 }
+
+void bdrv_store_persistent_bitmaps(BlockDriverState *bs, Error **errp)
+{
+BlockDriver *drv = bs->drv;
+
+if (!bdrv_has_persistent_bitmaps(bs)) {
+return;
+}
+
+if (!drv) {
+error_setg_errno(errp, ENOMEDIUM,
+ "Can't store persistent bitmaps to %s",
+ bdrv_get_device_or_node_name(bs));
+return;
+}
+
+if (!drv->bdrv_store_persistent_bitmaps) {
+error_setg_errno(errp, ENOTSUP,
+ "Can't store persistent bitmaps to %s",
+ bdrv_get_device_or_node_name(bs));
+return;
+}
+
+drv->bdrv_store_persistent_bitmaps(bs, errp);
+}
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 623e1d1..0314581 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -44,6 +44,7 @@ struct BdrvDirtyBitmap {
 int64_t size;   /* Size of the bitmap (Number of sectors) */
 bool disabled;  /* Bitmap is read-only */
 int active_iterators;   /* How many iterators are active */
+bool persistent;/* bitmap must be saved to owner disk image */
 bool autoload;  /* bitmap must be autoloaded on image opening 
*/
 QLIST_ENTRY(BdrvDirtyBitmap) list;
 };
@@ -72,6 +73,7 @@ void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap)
 g_free(bitmap->name);
 bitmap->name = NULL;
 
+bitmap->persistent = false;
 bitmap->autoload = false;
 }
 
@@ -241,6 +243,8 @@ BdrvDirtyBitmap 
*bdrv_dirty_bitmap_abdicate(BlockDriverState *bs,
 bitmap->name = NULL;
 successor->name = name;
 bitmap->successor = NULL;
+successor->persistent = bitmap->persistent;
+bitmap->persistent = false;
 successor->autoload = bitmap->autoload;
 bitmap->autoload = false;
 bdrv_release_dirty_bitmap(bs, bitmap);
@@ -555,3 +559,26 @@ bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap 
*bitmap)
 {
 return bitmap->autoload;
 }
+
+void bdrv_dirty_bitmap_set_persistance(BdrvDirtyBitmap *bitmap,
+bool persistent)
+{
+bitmap->persistent = persistent;
+}
+
+bool bdrv_dirty_bitmap_get_persistance(BdrvDirtyBitmap *bitmap)
+{
+return bitmap->persistent;
+}
+
+bool bdrv_has_persistent_bitmaps(BlockDriverState *bs)
+{
+BdrvDirtyBitmap *bm;
+QLIST_FOREACH(bm, &bs->dirty_bitmaps, list) {
+if (bm->persistent) {
+return true;
+}
+}
+
+return false;
+}
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index c086436..81520cd 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -530,6 +530,7 @@ int qcow2_read_bitmaps(BlockDriverState *bs, Error **errp)
 goto out;
 }
 
+bdrv_dirty_bitmap_set_persistance(bitmap, true);
 bdrv_dirty_bitmap_set_autoload(bitmap, true);
 } else {
 /* leave bitmap in the image */
diff --git a/include/block/block.h b/include/block/block.h
index 7edce5c..8457b60 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -532,4 +532,6 @@ void bdrv_add_child(BlockDriverState *parent, 
BlockDriverState *child,
 Error **errp);
 void bdrv_del_child(BlockDriverState *parent, BdrvChild *child, Error **errp);
 
+void bdrv_store_persistent_bitmaps(BlockDriverState *bs, Error **errp);
+
 #endif
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 0ca6a78..a4a4816 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -320,6 +320,8 @@ struct BlockDriver {
 voi

[Qemu-block] [PATCH 22/22] specs/qcow2: do not use wording 'bitmap header'

[Vladimir Sementsov-Ogievskiy]

For now bitmap directory entry some times called 'bitmap header'. This
patch leaves only one name - 'bitmap directory entry'. The name 'bitmap
header' creates misunderstandings with 'qcow2 header' and 'qcow2 bitmap
header extension' (which is extension of qcow2 header)

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 docs/specs/qcow2.txt | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
index d3e292f..d5f88c1 100644
--- a/docs/specs/qcow2.txt
+++ b/docs/specs/qcow2.txt
@@ -201,7 +201,7 @@ The fields of the bitmaps extension are:
 
   8 - 15:  bitmap_directory_size
Size of the bitmap directory in bytes. It is the cumulative
-   size of all (nb_bitmaps) bitmap headers.
+   size of all (nb_bitmaps) bitmap directory entries.
 
  16 - 23:  bitmap_directory_offset
Offset into the image file at which the bitmap directory
@@ -426,8 +426,7 @@ Each bitmap saved in the image is described in a bitmap 
directory entry. The
 bitmap directory is a contiguous area in the image file, whose starting offset
 and length are given by the header extension fields bitmap_directory_offset and
 bitmap_directory_size. The entries of the bitmap directory have variable
-length, depending on the lengths of the bitmap name and extra data. These
-entries are also called bitmap headers.
+length, depending on the lengths of the bitmap name and extra data.
 
 Structure of a bitmap directory entry:
 
-- 
1.8.3.1

[Qemu-block] [PATCH 17/22] qmp: add autoload parameter to block-dirty-bitmap-add

[Vladimir Sementsov-Ogievskiy]

Optional. Default is false.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Signed-off-by: Denis V. Lunev 
---
 blockdev.c   | 22 --
 qapi/block-core.json |  7 ++-
 qmp-commands.hx  |  5 -
 3 files changed, 30 insertions(+), 4 deletions(-)

diff --git a/blockdev.c b/blockdev.c
index ec0ec75..00da7a1 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -1992,6 +1992,7 @@ static void block_dirty_bitmap_add_prepare(BlkActionState 
*common,
 qmp_block_dirty_bitmap_add(action->node, action->name,
action->has_granularity, action->granularity,
action->has_persistent, action->persistent,
+   action->has_autoload, action->autoload,
&local_err);
 
 if (!local_err) {
@@ -2696,6 +2697,7 @@ out:
 void qmp_block_dirty_bitmap_add(const char *node, const char *name,
 bool has_granularity, uint32_t granularity,
 bool has_persistent, bool persistent,
+bool has_autoload, bool autoload,
 Error **errp)
 {
 AioContext *aio_context;
@@ -2729,10 +2731,26 @@ void qmp_block_dirty_bitmap_add(const char *node, const 
char *name,
 if (!has_persistent) {
 persistent = false;
 }
+if (!has_autoload) {
+autoload = false;
+}
+
+if (autoload && !persistent) {
+error_setg(errp, "Autoload flag must be used only for persistent"
+ "bitmaps");
+goto out;
+}
 
 bitmap = bdrv_create_dirty_bitmap(bs, granularity, name, errp);
-if (bitmap != NULL) {
-bdrv_dirty_bitmap_set_persistance(bitmap, persistent);
+if (bitmap == NULL) {
+goto out;
+}
+
+if (persistent) {
+bdrv_dirty_bitmap_set_persistance(bitmap, true);
+if (autoload) {
+bdrv_dirty_bitmap_set_autoload(bitmap, true);
+}
 }
 
  out:
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 2bf56cd..087a681 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1239,11 +1239,16 @@
 #  corresponding block device on it's close. Default is false.
 #  For block-dirty-bitmap-add. (Since 2.8)
 #
+# @autoload: #optional the bitmap will be autoloaded on it's storage image
+#open. This flag is only for persistent bitmap and needed to inform
+#block driver that bitmap should be autoloaded on the next image
+#open. Default is false. For block-dirty-bitmap-add. (Since 2.8)
+#
 # Since 2.4
 ##
 { 'struct': 'BlockDirtyBitmapAdd',
   'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32',
-  '*persistent': 'bool' } }
+  '*persistent': 'bool', '*autoload': 'bool' } }
 
 ##
 # @block-dirty-bitmap-add
diff --git a/qmp-commands.hx b/qmp-commands.hx
index 434b418..8f4e841 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -1441,7 +1441,7 @@ EQMP
 
 {
 .name   = "block-dirty-bitmap-add",
-.args_type  = "node:B,name:s,granularity:i?,persistent:b?",
+.args_type  = "node:B,name:s,granularity:i?,persistent:b?,autoload:b?",
 .mhandler.cmd_new = qmp_marshal_block_dirty_bitmap_add,
 },
 
@@ -1461,6 +1461,9 @@ Arguments:
 - "persistent": bitmap will be saved to corresponding block device
 on it's close. Block driver should maintain persistent bitmaps
 (json-bool, optional, default false) (Since 2.8)
+- "autoload": only for persistent bitmaps. Bitmap will be autoloaded on it's
+  storage image open. (json-bool, optional, default false)
+  (Since 2.8)
 
 Example:
 
-- 
1.8.3.1

[Qemu-block] [PATCH 11/22] qcow2-bitmap: add qcow2_store_persistent_bitmaps()

[Vladimir Sementsov-Ogievskiy]

Realize block bitmap stroing interface, to allow qcow2 images store
persistent bitmaps.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2-bitmap.c | 241 +++
 block/qcow2.c|   2 +
 block/qcow2.h|   2 +
 3 files changed, 245 insertions(+)

diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index 81520cd..a5be25a 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -27,6 +27,7 @@
 
 #include "qemu/osdep.h"
 #include "qapi/error.h"
+#include "qemu/cutils.h"
 
 #include "block/block_int.h"
 #include "block/qcow2.h"
@@ -96,6 +97,15 @@ static inline void bitmap_table_to_cpu(uint64_t 
*bitmap_table, size_t size)
 }
 }
 
+static inline void bitmap_table_to_be(uint64_t *bitmap_table, size_t size)
+{
+size_t i;
+
+for (i = 0; i < size; ++i) {
+cpu_to_be64s(&bitmap_table[i]);
+}
+}
+
 static inline int calc_dir_entry_size(size_t name_size, size_t extra_data_size)
 {
 return align_offset(sizeof(Qcow2BitmapDirEntry) +
@@ -564,3 +574,234 @@ out:
 
 return ret;
 }
+
+/* store_bitmap_data()
+ * Store bitmap to image, filling bitamp table accordingly.
+ */
+static int store_bitmap_data(BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
+ uint64_t *bitmap_table, uint32_t 
bitmap_table_size)
+{
+int ret;
+BDRVQcow2State *s = bs->opaque;
+uint64_t sector, dsc;
+uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap);
+int cl_size = s->cluster_size;
+uint8_t *buf = NULL;
+uint32_t tb_size =
+size_to_clusters(s,
+bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_size));
+
+BdrvDirtyBitmapIter *dbi;
+
+if (tb_size != bitmap_table_size) {
+return -EINVAL;
+}
+
+memset(bitmap_table, 0, bitmap_table_size * sizeof(bitmap_table[0]));
+
+dbi = bdrv_dirty_iter_new(bitmap, 0);
+buf = g_malloc(cl_size);
+dsc = dirty_sectors_in_cluster(s, bitmap);
+
+while ((sector = bdrv_dirty_iter_next(dbi)) != -1) {
+uint64_t cluster = sector / dsc;
+sector = cluster * dsc;
+uint64_t end = MIN(bm_size, sector + dsc);
+uint64_t write_size =
+bdrv_dirty_bitmap_serialization_size(bitmap, sector, end - sector);
+
+int64_t off = qcow2_alloc_clusters(bs, cl_size);
+if (off < 0) {
+ret = off;
+goto finish;
+}
+bitmap_table[cluster] = off;
+
+bdrv_dirty_bitmap_serialize_part(bitmap, buf, sector, end);
+if (write_size < cl_size) {
+memset(buf + write_size, 0, cl_size - write_size);
+}
+
+ret = bdrv_pwrite(bs->file, off, buf, cl_size);
+if (ret < 0) {
+goto finish;
+}
+
+if (end >= bm_size) {
+break;
+}
+
+bdrv_set_dirty_iter(dbi, end);
+}
+ret = 0; /* writes */
+
+finish:
+if (ret < 0) {
+clear_bitmap_table(bs, bitmap_table, bitmap_table_size);
+}
+g_free(buf);
+bdrv_dirty_iter_free(dbi);
+
+return ret;
+}
+
+/* store_bitmap()
+ * Store bitmap to qcow2 and set bitmap_table. bitmap_table itself is not
+ * stored to qcow2.
+ */
+static int store_bitmap(BlockDriverState *bs,
+BdrvDirtyBitmap *bitmap,
+Qcow2BitmapDirEntry *entry)
+{
+int ret;
+BDRVQcow2State *s = bs->opaque;
+uint64_t bm_size = bdrv_dirty_bitmap_size(bitmap);
+const char *bm_name = bdrv_dirty_bitmap_name(bitmap);
+
+uint64_t *tb;
+int64_t tb_offset;
+uint32_t tb_size =
+size_to_clusters(s,
+bdrv_dirty_bitmap_serialization_size(bitmap, 0, bm_size));
+
+tb = g_try_new(uint64_t, tb_size);
+if (tb == NULL) {
+return -ENOMEM;
+}
+
+ret = store_bitmap_data(bs, bitmap, tb, tb_size);
+if (ret < 0) {
+g_free(tb);
+return ret;
+}
+
+tb_offset = qcow2_alloc_clusters(bs, tb_size * sizeof(tb[0]));
+if (tb_offset < 0) {
+ret = tb_offset;
+goto fail;
+}
+
+bitmap_table_to_be(tb, tb_size);
+ret = bdrv_pwrite(bs->file, tb_offset, tb, tb_size * sizeof(tb[0]));
+if (ret < 0) {
+goto fail;
+}
+
+g_free(tb);
+
+entry->bitmap_table_offset = tb_offset;
+entry->bitmap_table_size = tb_size;
+entry->flags = bdrv_dirty_bitmap_granularity(bitmap) ? BME_FLAG_AUTO : 0;
+entry->type = BT_DIRTY_TRACKING_BITMAP;
+entry->granularity_bits = ctz32(bdrv_dirty_bitmap_granularity(bitmap));
+entry->name_size = strlen(bm_name);
+entry->extra_data_size = 0;
+memcpy(entry + 1, bm_name, entry->name_size);
+
+return 0;
+
+fail:
+clear_bitmap_table(bs, tb, tb_size);
+
+if (tb_offset > 0) {
+qcow2_free_clusters(bs, tb_offset, tb_size, QCOW2_DISCARD_ALWAYS);
+}
+
+g_free(tb);
+
+return ret;
+}
+
+static Qcow2BitmapDirEntry *find_bitmap_by_name(uint8_t *bitmap_directory,
+ 

[Qemu-block] [PATCH 07/22] qcow2-bitmap: introduce auto-loading bitmaps

[Vladimir Sementsov-Ogievskiy]

Auto loading bitmaps are bitmaps in Qcow2, with AUTO flag set. They are
loaded at image open and becomes BdrvDirtyBitmap's for corresponding
drive. These bitmaps are deleted from Qcow2 image after loading to avoid
conflicts.

Extra data in bitmaps is not supported for now.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2-bitmap.c | 518 ++-
 block/qcow2.c|   5 +
 block/qcow2.h|   3 +
 3 files changed, 525 insertions(+), 1 deletion(-)

diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index cd18b07..760a047 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -25,6 +25,12 @@
  * THE SOFTWARE.
  */
 
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+
+#include "block/block_int.h"
+#include "block/qcow2.h"
+
 /* NOTICE: BME here means Bitmaps Extension and used as a namespace for
  * _internal_ constants. Please do not use this _internal_ abbreviation for
  * other needs and/or outside of this file. */
@@ -37,11 +43,521 @@
 #define BME_MAX_NAME_SIZE 1023
 
 /* Bitmap directory entry flags */
-#define BME_RESERVED_FLAGS 0x
+#define BME_RESERVED_FLAGS 0xfffd
+#define BME_FLAG_AUTO   (1U << 1)
 
 /* bits [1, 8] U [56, 63] are reserved */
 #define BME_TABLE_ENTRY_RESERVED_MASK 0xff0001fe
 
+#define for_each_bitmap_dir_entry_unsafe(entry, dir, size) \
+for (entry = (Qcow2BitmapDirEntry *)(dir); \
+ entry < (Qcow2BitmapDirEntry *)((uint8_t *)(dir) + size); \
+ entry = next_dir_entry(entry))
+
+#define for_each_bitmap_dir_entry(entry, dir, size) \
+for (entry = (Qcow2BitmapDirEntry *)(dir); \
+ assert(check_dir_iter(entry, (uint8_t *)(dir) + size)), \
+ entry < (Qcow2BitmapDirEntry *)((uint8_t *)(dir) + size); \
+ entry = next_dir_entry(entry))
+
 typedef enum BitmapType {
 BT_DIRTY_TRACKING_BITMAP = 1
 } BitmapType;
+
+static inline bool can_write(BlockDriverState *bs)
+{
+return !bdrv_is_read_only(bs) && !(bdrv_get_flags(bs) & BDRV_O_INACTIVE);
+}
+
+static inline void bitmap_dir_entry_to_cpu(Qcow2BitmapDirEntry *entry)
+{
+be64_to_cpus(&entry->bitmap_table_offset);
+be32_to_cpus(&entry->bitmap_table_size);
+be32_to_cpus(&entry->flags);
+be16_to_cpus(&entry->name_size);
+be32_to_cpus(&entry->extra_data_size);
+}
+
+static inline void bitmap_dir_entry_to_be(Qcow2BitmapDirEntry *entry)
+{
+cpu_to_be64s(&entry->bitmap_table_offset);
+cpu_to_be32s(&entry->bitmap_table_size);
+cpu_to_be32s(&entry->flags);
+cpu_to_be16s(&entry->name_size);
+cpu_to_be32s(&entry->extra_data_size);
+}
+
+static inline void bitmap_table_to_cpu(uint64_t *bitmap_table, size_t size)
+{
+size_t i;
+
+for (i = 0; i < size; ++i) {
+be64_to_cpus(&bitmap_table[i]);
+}
+}
+
+static inline int calc_dir_entry_size(size_t name_size, size_t extra_data_size)
+{
+return align_offset(sizeof(Qcow2BitmapDirEntry) +
+name_size + extra_data_size, 8);
+}
+
+static inline int dir_entry_size(Qcow2BitmapDirEntry *entry)
+{
+return calc_dir_entry_size(entry->name_size, entry->extra_data_size);
+}
+
+static inline const char *dir_entry_name_notcstr(Qcow2BitmapDirEntry *entry)
+{
+return (const char *)(entry + 1) + entry->extra_data_size;
+}
+
+static inline int copy_dir_entry(void *dest, Qcow2BitmapDirEntry *entry)
+{
+int sz = dir_entry_size(entry);
+memcpy(dest, entry, sz);
+return sz;
+}
+
+static inline Qcow2BitmapDirEntry *next_dir_entry(Qcow2BitmapDirEntry *entry)
+{
+return (Qcow2BitmapDirEntry *)((uint8_t *)entry + dir_entry_size(entry));
+}
+
+static inline bool check_dir_iter(Qcow2BitmapDirEntry *it, void *directory_end)
+{
+return ((void *)it == directory_end) ||
+   (((void *)(it + 1) <= directory_end) &&
+((void *)next_dir_entry(it) <= directory_end));
+}
+
+static inline void bitmap_directory_to_be(uint8_t *dir, size_t size)
+{
+uint8_t *end = dir + size;
+while (dir < end) {
+Qcow2BitmapDirEntry *e = (Qcow2BitmapDirEntry *)dir;
+dir += dir_entry_size(e);
+
+bitmap_dir_entry_to_be(e);
+}
+}
+
+static void clear_bitmap_table(BlockDriverState *bs, uint64_t *bitmap_table,
+   uint32_t bitmap_table_size)
+{
+BDRVQcow2State *s = bs->opaque;
+int cl_size = s->cluster_size;
+int i;
+
+for (i = 0; i < bitmap_table_size; ++i) {
+uint64_t addr = bitmap_table[i];
+if (addr <= 1) {
+continue;
+}
+
+qcow2_free_clusters(bs, addr, cl_size, QCOW2_DISCARD_ALWAYS);
+bitmap_table[i] = 0;
+}
+}
+
+static int bitmap_table_load(BlockDriverState *bs, Qcow2BitmapDirEntry *entry,
+ uint64_t **table)
+{
+int ret;
+
+*table = g_try_new(uint64_t, entry->bitmap_table_size);
+if (*table == NULL) {
+return -ENOMEM;
+}
+
+ret = bdrv_pread(bs->file, entry->bitmap_table_offset,
+

[Qemu-block] [PATCH v7 00/22] qcow2: persistent dirty bitmaps

[Vladimir Sementsov-Ogievskiy]

v7:
https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fqcow2-bitmap-v7
based on block-next (https://github.com/XanClic/qemu/commits/block-next)

- a lot of refactoring and reordering of patches.
- dead code removed (bdrv_dirty_bitmap_load, etc.)
- do not maintain extra data for now
- do not store dirty bitmap directory in memory
  (as we use it seldom, we can just reread if needed)

By Kevin's review:
01 - commit message changed: fix->improvement (as it was not a bug)
03 - r-b
04 - r-b
05 - add 21 patch to fix spec, also, removed all (I hope) mentions of
 "Bitmap Header", switch to one unified name for it - "Bitmap
 Directory Entry", to avoid misunderstanding with Qcow2 header.
 (also, add patch 22, to fix it in spec)
v6.06 - improve for_each_dir_entry loop, reorder patches, other small fixes
v6.07 ~> v7.09 - dead code removed, I've moved to one function 
.bdrv_store_persistent_bitmaps and have wrapper and callback in one
patch (with also some other staff. If it not ok, I can split them)
v6.08 - about keeping bitmap directory instead of bitmap list: no I don't keep
it at all.
v6.16 - old bdrv_ bitmap-storing related functions are removed. The new one is
bdrv_store_persistent_bitmaps.

v6:
https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fqcow2-bitmap-v6
based on block-next (https://github.com/XanClic/qemu/commits/block-next)

There are a lot of changes, reorderings and additions in comparement with v5.
One principal thing: now bitmaps are removed from image after loading instead
of marking them in_use. It is simpler and we do not need to store superfluous 
data.
Also, we are no more interested in command line interface to dirty bitmaps.
So it is dropped.  If someone needs it I can add it later.

Vladimir Sementsov-Ogievskiy (22):
  hbitmap: improve dirty iter
  tests: add hbitmap iter test
  block: fix bdrv_dirty_bitmap_granularity signature
  block/dirty-bitmap: add deserialize_ones func
  qcow2-bitmap: structs and consts
  qcow2: add dirty bitmaps extension
  qcow2-bitmap: introduce auto-loading bitmaps
  block/dirty-bitmap: add autoload field to BdrvDirtyBitmap
  block: introduce persistent dirty bitmaps
  block/dirty-bitmap: add bdrv_dirty_bitmap_next()
  qcow2-bitmap: add qcow2_store_persistent_bitmaps()
  qcow2-bitmap: add IN_USE flag
  qcow2-bitmap: check constraints
  qcow2: delete bitmaps on truncate
  qcow2-bitmap: add autoclear bit
  qmp: add persistent flag to block-dirty-bitmap-add
  qmp: add autoload parameter to block-dirty-bitmap-add
  qapi: add md5 checksum of last dirty bitmap level to query-block
  iotests: test qcow2 persistent dirty bitmap
  qcow2-dirty-bitmap: refcounts
  specs/qcow2: fix bitmap granularity qemu-specific note
  specs/qcow2: do not use wording 'bitmap header'

 block.c  |  30 ++
 block/Makefile.objs  |   2 +-
 block/dirty-bitmap.c |  59 ++-
 block/qcow2-bitmap.c | 963 +++
 block/qcow2-refcount.c   |  14 +-
 block/qcow2.c| 105 -
 block/qcow2.h|  54 +++
 blockdev.c   |  30 +-
 docs/specs/qcow2.txt |   8 +-
 include/block/block.h|   2 +
 include/block/block_int.h|   2 +
 include/block/dirty-bitmap.h |  16 +-
 include/qemu/hbitmap.h   |  47 ++-
 qapi/block-core.json |  17 +-
 qmp-commands.hx  |   8 +-
 tests/qemu-iotests/165   |  87 
 tests/qemu-iotests/165.out   |   5 +
 tests/qemu-iotests/group |   1 +
 tests/test-hbitmap.c |  19 +
 util/hbitmap.c   |  48 ++-
 20 files changed, 1475 insertions(+), 42 deletions(-)
 create mode 100644 block/qcow2-bitmap.c
 create mode 100755 tests/qemu-iotests/165
 create mode 100644 tests/qemu-iotests/165.out

-- 
1.8.3.1

[Qemu-block] [PATCH 12/22] qcow2-bitmap: add IN_USE flag

[Vladimir Sementsov-Ogievskiy]

This flag means that the bitmap is now in use by the software or was not
successfully saved. In any way, with this flag set the bitmap data must
be considered inconsistent and should not be loaded.

With current implementation this flag is never set, as we just remove
bitmaps from the image after loading. But it defined in qcow2 spec and
must be handled. Also, it can be used in future, if async schemes of
bitmap loading/saving are implemented.

We also remove in-use bitmaps from the image on open.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2-bitmap.c | 17 -
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index a5be25a..8cf40f0 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -28,6 +28,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "qemu/cutils.h"
+#include "exec/log.h"
 
 #include "block/block_int.h"
 #include "block/qcow2.h"
@@ -44,7 +45,8 @@
 #define BME_MAX_NAME_SIZE 1023
 
 /* Bitmap directory entry flags */
-#define BME_RESERVED_FLAGS 0xfffd
+#define BME_RESERVED_FLAGS 0xfffc
+#define BME_FLAG_IN_USE 1
 #define BME_FLAG_AUTO   (1U << 1)
 
 /* bits [1, 8] U [56, 63] are reserved */
@@ -287,6 +289,11 @@ static BdrvDirtyBitmap *load_bitmap(BlockDriverState *bs,
 BdrvDirtyBitmap *bitmap = NULL;
 char *name = g_strndup(dir_entry_name_notcstr(entry), entry->name_size);
 
+if (entry->flags & BME_FLAG_IN_USE) {
+error_setg(errp, "Bitmap '%s' is in use", name);
+goto fail;
+}
+
 ret = bitmap_table_load(bs, entry, &bitmap_table);
 if (ret < 0) {
 error_setg_errno(errp, -ret,
@@ -533,6 +540,14 @@ int qcow2_read_bitmaps(BlockDriverState *bs, Error **errp)
 for_each_bitmap_dir_entry(e, dir, dir_size) {
 uint32_t fl = e->flags;
 
+if (fl & BME_FLAG_IN_USE) {
+qemu_log("qcow2 warning: "
+ "removing in_use bitmap '%.*s' for image %s.\n",
+ e->name_size, (char *)(e + 1), 
bdrv_get_device_or_node_name(bs));
+
+continue;
+}
+
 if (fl & BME_FLAG_AUTO) {
 BdrvDirtyBitmap *bitmap = load_bitmap(bs, e, errp);
 if (bitmap == NULL) {
-- 
1.8.3.1

[Qemu-block] [PATCH 16/22] qmp: add persistent flag to block-dirty-bitmap-add

[Vladimir Sementsov-Ogievskiy]

Add optional 'persistent' flag to qmp command block-dirty-bitmap-add.
Default is false.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
Signed-off-by: Denis V. Lunev 
---
 blockdev.c   | 12 +++-
 qapi/block-core.json |  7 ++-
 qmp-commands.hx  |  5 -
 3 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/blockdev.c b/blockdev.c
index 97062e3..ec0ec75 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -1991,6 +1991,7 @@ static void block_dirty_bitmap_add_prepare(BlkActionState 
*common,
 /* AIO context taken and released within qmp_block_dirty_bitmap_add */
 qmp_block_dirty_bitmap_add(action->node, action->name,
action->has_granularity, action->granularity,
+   action->has_persistent, action->persistent,
&local_err);
 
 if (!local_err) {
@@ -2694,10 +2695,12 @@ out:
 
 void qmp_block_dirty_bitmap_add(const char *node, const char *name,
 bool has_granularity, uint32_t granularity,
+bool has_persistent, bool persistent,
 Error **errp)
 {
 AioContext *aio_context;
 BlockDriverState *bs;
+BdrvDirtyBitmap *bitmap;
 
 if (!name || name[0] == '\0') {
 error_setg(errp, "Bitmap name cannot be empty");
@@ -2723,7 +2726,14 @@ void qmp_block_dirty_bitmap_add(const char *node, const 
char *name,
 granularity = bdrv_get_default_bitmap_granularity(bs);
 }
 
-bdrv_create_dirty_bitmap(bs, granularity, name, errp);
+if (!has_persistent) {
+persistent = false;
+}
+
+bitmap = bdrv_create_dirty_bitmap(bs, granularity, name, errp);
+if (bitmap != NULL) {
+bdrv_dirty_bitmap_set_persistance(bitmap, persistent);
+}
 
  out:
 aio_context_release(aio_context);
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 31f9990..2bf56cd 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1235,10 +1235,15 @@
 # @granularity: #optional the bitmap granularity, default is 64k for
 #   block-dirty-bitmap-add
 #
+# @persistent: #optional the bitmap is persistent, i.e. it will be saved to
+#  corresponding block device on it's close. Default is false.
+#  For block-dirty-bitmap-add. (Since 2.8)
+#
 # Since 2.4
 ##
 { 'struct': 'BlockDirtyBitmapAdd',
-  'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32' } }
+  'data': { 'node': 'str', 'name': 'str', '*granularity': 'uint32',
+  '*persistent': 'bool' } }
 
 ##
 # @block-dirty-bitmap-add
diff --git a/qmp-commands.hx b/qmp-commands.hx
index ba2a916..434b418 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -1441,7 +1441,7 @@ EQMP
 
 {
 .name   = "block-dirty-bitmap-add",
-.args_type  = "node:B,name:s,granularity:i?",
+.args_type  = "node:B,name:s,granularity:i?,persistent:b?",
 .mhandler.cmd_new = qmp_marshal_block_dirty_bitmap_add,
 },
 
@@ -1458,6 +1458,9 @@ Arguments:
 - "node": device/node on which to create dirty bitmap (json-string)
 - "name": name of the new dirty bitmap (json-string)
 - "granularity": granularity to track writes with (int, optional)
+- "persistent": bitmap will be saved to corresponding block device
+on it's close. Block driver should maintain persistent bitmaps
+(json-bool, optional, default false) (Since 2.8)
 
 Example:
 
-- 
1.8.3.1

[Qemu-block] [PATCH 06/22] qcow2: add dirty bitmaps extension

[Vladimir Sementsov-Ogievskiy]

Add dirty bitmap extension as specified in docs/specs/qcow2.txt.
For now, just mirror extension header into Qcow2 state and check
constraints.

For now, disable image resize if it has bitmaps. It will be fixed later.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2.c | 83 +++
 block/qcow2.h |  4 +++
 2 files changed, 87 insertions(+)

diff --git a/block/qcow2.c b/block/qcow2.c
index c079aa8..08c4ef9 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -63,6 +63,7 @@ typedef struct {
 #define  QCOW2_EXT_MAGIC_END 0
 #define  QCOW2_EXT_MAGIC_BACKING_FORMAT 0xE2792ACA
 #define  QCOW2_EXT_MAGIC_FEATURE_TABLE 0x6803f857
+#define  QCOW2_EXT_MAGIC_DIRTY_BITMAPS 0x23852875
 
 static int qcow2_probe(const uint8_t *buf, int buf_size, const char *filename)
 {
@@ -92,6 +93,7 @@ static int qcow2_read_extensions(BlockDriverState *bs, 
uint64_t start_offset,
 QCowExtension ext;
 uint64_t offset;
 int ret;
+Qcow2BitmapHeaderExt bitmaps_ext;
 
 #ifdef DEBUG_EXT
 printf("qcow2_read_extensions: start=%ld end=%ld\n", start_offset, 
end_offset);
@@ -162,6 +164,62 @@ static int qcow2_read_extensions(BlockDriverState *bs, 
uint64_t start_offset,
 }
 break;
 
+case QCOW2_EXT_MAGIC_DIRTY_BITMAPS:
+ret = bdrv_pread(bs->file, offset, &bitmaps_ext, ext.len);
+if (ret < 0) {
+error_setg_errno(errp, -ret, "ERROR: bitmaps_ext: "
+ "Could not read ext header");
+return ret;
+}
+
+if (bitmaps_ext.reserved32 != 0) {
+error_setg_errno(errp, -ret, "ERROR: bitmaps_ext: "
+ "Reserved field is not zero.");
+return -EINVAL;
+}
+
+be32_to_cpus(&bitmaps_ext.nb_bitmaps);
+be64_to_cpus(&bitmaps_ext.bitmap_directory_size);
+be64_to_cpus(&bitmaps_ext.bitmap_directory_offset);
+
+if (bitmaps_ext.nb_bitmaps > QCOW_MAX_DIRTY_BITMAPS) {
+error_setg(errp, "ERROR: bitmaps_ext: "
+ "too many dirty bitmaps");
+return -EINVAL;
+}
+
+if (bitmaps_ext.nb_bitmaps == 0) {
+error_setg(errp, "ERROR: bitmaps_ext: "
+ "found bitmaps extension with zero bitmaps");
+return -EINVAL;
+}
+
+if (bitmaps_ext.bitmap_directory_offset & (s->cluster_size - 1)) {
+error_setg(errp, "ERROR: bitmaps_ext: "
+ "wrong dirty bitmap directory offset");
+return -EINVAL;
+}
+
+if (bitmaps_ext.bitmap_directory_size >
+QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE) {
+error_setg(errp, "ERROR: bitmaps_ext: "
+ "too large dirty bitmap directory");
+return -EINVAL;
+}
+
+s->nb_bitmaps = bitmaps_ext.nb_bitmaps;
+s->bitmap_directory_offset =
+bitmaps_ext.bitmap_directory_offset;
+s->bitmap_directory_size =
+bitmaps_ext.bitmap_directory_size;
+
+#ifdef DEBUG_EXT
+printf("Qcow2: Got dirty bitmaps extension:"
+   " offset=%" PRIu64 " nb_bitmaps=%" PRIu32 "\n",
+   s->bitmap_directory_offset, s->nb_bitmaps);
+#endif
+break;
+
 default:
 /* unknown magic - save it in case we need to rewrite the header */
 {
@@ -1939,6 +1997,24 @@ int qcow2_update_header(BlockDriverState *bs)
 buflen -= ret;
 }
 
+if (s->nb_bitmaps > 0) {
+Qcow2BitmapHeaderExt bitmaps_header = {
+.nb_bitmaps = cpu_to_be32(s->nb_bitmaps),
+.bitmap_directory_size =
+cpu_to_be64(s->bitmap_directory_size),
+.bitmap_directory_offset =
+cpu_to_be64(s->bitmap_directory_offset)
+};
+ret = header_ext_add(buf, QCOW2_EXT_MAGIC_DIRTY_BITMAPS,
+ &bitmaps_header, sizeof(bitmaps_header),
+ buflen);
+if (ret < 0) {
+goto fail;
+}
+buf += ret;
+buflen -= ret;
+}
+
 /* Keep unknown header extensions */
 QLIST_FOREACH(uext, &s->unknown_header_ext, next) {
 ret = header_ext_add(buf, uext->magic, uext->data, uext->len, buflen);
@@ -2509,6 +2585,13 @@ static int qcow2_truncate(BlockDriverState *bs, int64_t 
offset)
 return -ENOTSUP;
 }
 
+/* cannot proceed if image has bitmaps */
+if (s->nb_bitmaps) {
+/* FIXME */
+error_report("Can't resize an image which has bitmaps");
+return -ENOTSUP;
+}
+
 /* shrinking is currently not supported */
 if (offset < bs->total_sectors * 512) {
 error_report("qcow2 doesn't support s

[Qemu-block] [PATCH 14/22] qcow2: delete bitmaps on truncate

[Vladimir Sementsov-Ogievskiy]

This will not touch loaded bitmaps, as they are alredy removed from the
image.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2-bitmap.c | 38 ++
 block/qcow2.c|  9 ++---
 block/qcow2.h|  2 ++
 3 files changed, 46 insertions(+), 3 deletions(-)

diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index 1c3abea..2642afe 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -624,6 +624,44 @@ out:
 return ret;
 }
 
+
+int qcow2_delete_bitmaps(BlockDriverState *bs)
+{
+int ret;
+BDRVQcow2State *s = bs->opaque;
+uint8_t *dir;
+uint64_t dir_size;
+Qcow2BitmapDirEntry *e;
+
+if (s->nb_bitmaps == 0) {
+/* No bitmaps - nothing to do */
+return 0;
+}
+
+dir_size = s->bitmap_directory_size;
+dir = directory_read(bs, s->bitmap_directory_offset,
+ s->bitmap_directory_size, NULL);
+if (dir == NULL) {
+ret = -EINVAL;
+goto out;
+}
+
+ret = directory_update(bs, NULL, 0, 0);
+if (ret < 0) {
+goto out;
+}
+
+/* to be consistent, free bitmap only after successfull directory update */
+for_each_bitmap_dir_entry(e, dir, dir_size) {
+free_bitmap_clusters(bs, e);
+}
+
+out:
+g_free(dir);
+
+return ret;
+}
+
 /* store_bitmap_data()
  * Store bitmap to image, filling bitamp table accordingly.
  */
diff --git a/block/qcow2.c b/block/qcow2.c
index 8238205..aa967ed 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -2592,9 +2592,12 @@ static int qcow2_truncate(BlockDriverState *bs, int64_t 
offset)
 
 /* cannot proceed if image has bitmaps */
 if (s->nb_bitmaps) {
-/* FIXME */
-error_report("Can't resize an image which has bitmaps");
-return -ENOTSUP;
+/* FIXME: not loaded bitmaps will be lost */
+ret = qcow2_delete_bitmaps(bs);
+if (ret < 0) {
+error_report("Can't remove bitmaps from qcow2 on truncate");
+return ret;
+}
 }
 
 /* shrinking is currently not supported */
diff --git a/block/qcow2.h b/block/qcow2.h
index dfcf4c6..af18efc 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -606,6 +606,8 @@ int qcow2_read_snapshots(BlockDriverState *bs);
 /* qcow2-bitmap.c functions */
 int qcow2_read_bitmaps(BlockDriverState *bs, Error **errp);
 
+int qcow2_delete_bitmaps(BlockDriverState *bs);
+
 /* qcow2-cache.c functions */
 Qcow2Cache *qcow2_cache_create(BlockDriverState *bs, int num_tables);
 int qcow2_cache_destroy(BlockDriverState* bs, Qcow2Cache *c);
-- 
1.8.3.1

[Qemu-block] [PATCH 04/22] block/dirty-bitmap: add deserialize_ones func

[Vladimir Sementsov-Ogievskiy]

Add bdrv_dirty_bitmap_deserialize_ones() function, which is needed for
qcow2 bitmap loading, to handle unallocated bitmap parts, marked as
all-ones.

Reviewed-by: Kevin Wolf 
Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 block/dirty-bitmap.c |  7 +++
 include/block/dirty-bitmap.h |  3 +++
 include/qemu/hbitmap.h   | 15 +++
 util/hbitmap.c   | 17 +
 4 files changed, 42 insertions(+)

diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 186941c..90af372 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -499,6 +499,13 @@ void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap 
*bitmap,
 hbitmap_deserialize_zeroes(bitmap->bitmap, start, count, finish);
 }
 
+void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap *bitmap,
+uint64_t start, uint64_t count,
+bool finish)
+{
+hbitmap_deserialize_ones(bitmap->bitmap, start, count, finish);
+}
+
 void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap)
 {
 hbitmap_deserialize_finish(bitmap->bitmap);
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 7cbe623..1e17729 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -70,6 +70,9 @@ void bdrv_dirty_bitmap_deserialize_part(BdrvDirtyBitmap 
*bitmap,
 void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap,
   uint64_t start, uint64_t count,
   bool finish);
+void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap *bitmap,
+uint64_t start, uint64_t count,
+bool finish);
 void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);
 
 #endif
diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h
index 9aa86c1..64e5963 100644
--- a/include/qemu/hbitmap.h
+++ b/include/qemu/hbitmap.h
@@ -216,6 +216,21 @@ void hbitmap_deserialize_zeroes(HBitmap *hb, uint64_t 
start, uint64_t count,
 bool finish);
 
 /**
+ * hbitmap_deserialize_ones
+ * @hb: HBitmap to operate on.
+ * @start: First bit to restore.
+ * @count: Number of bits to restore.
+ * @finish: Whether to call hbitmap_deserialize_finish automatically.
+ *
+ * Fills the bitmap with ones.
+ *
+ * If @finish is false, caller must call hbitmap_serialize_finish before using
+ * the bitmap.
+ */
+void hbitmap_deserialize_ones(HBitmap *hb, uint64_t start, uint64_t count,
+  bool finish);
+
+/**
  * hbitmap_deserialize_finish
  * @hb: HBitmap to operate on.
  *
diff --git a/util/hbitmap.c b/util/hbitmap.c
index 4f5cf96..f49520f 100644
--- a/util/hbitmap.c
+++ b/util/hbitmap.c
@@ -531,6 +531,23 @@ void hbitmap_deserialize_zeroes(HBitmap *hb, uint64_t 
start, uint64_t count,
 }
 }
 
+void hbitmap_deserialize_ones(HBitmap *hb, uint64_t start, uint64_t count,
+  bool finish)
+{
+uint64_t el_count;
+unsigned long *first;
+
+if (!count) {
+return;
+}
+serialization_chunk(hb, start, count, &first, &el_count);
+
+memset(first, 0xff, el_count * sizeof(unsigned long));
+if (finish) {
+hbitmap_deserialize_finish(hb);
+}
+}
+
 void hbitmap_deserialize_finish(HBitmap *bitmap)
 {
 int64_t i, size, prev_size;
-- 
1.8.3.1

[Qemu-block] [PATCH 03/22] block: fix bdrv_dirty_bitmap_granularity signature

[Vladimir Sementsov-Ogievskiy]

Make getter signature const-correct. This allows other functions with
const dirty bitmap parameter use bdrv_dirty_bitmap_granularity().

Reviewed-by: Eric Blake 
Reviewed-by: John Snow 
Reviewed-by: Kevin Wolf 
Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 block/dirty-bitmap.c | 2 +-
 include/block/dirty-bitmap.h | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 519737c..186941c 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -388,7 +388,7 @@ uint32_t 
bdrv_get_default_bitmap_granularity(BlockDriverState *bs)
 return granularity;
 }
 
-uint32_t bdrv_dirty_bitmap_granularity(BdrvDirtyBitmap *bitmap)
+uint32_t bdrv_dirty_bitmap_granularity(const BdrvDirtyBitmap *bitmap)
 {
 return BDRV_SECTOR_SIZE << hbitmap_granularity(bitmap->bitmap);
 }
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 9dea14b..7cbe623 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -29,7 +29,7 @@ void bdrv_disable_dirty_bitmap(BdrvDirtyBitmap *bitmap);
 void bdrv_enable_dirty_bitmap(BdrvDirtyBitmap *bitmap);
 BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs);
 uint32_t bdrv_get_default_bitmap_granularity(BlockDriverState *bs);
-uint32_t bdrv_dirty_bitmap_granularity(BdrvDirtyBitmap *bitmap);
+uint32_t bdrv_dirty_bitmap_granularity(const BdrvDirtyBitmap *bitmap);
 uint32_t bdrv_dirty_bitmap_meta_granularity(BdrvDirtyBitmap *bitmap);
 bool bdrv_dirty_bitmap_enabled(BdrvDirtyBitmap *bitmap);
 bool bdrv_dirty_bitmap_frozen(BdrvDirtyBitmap *bitmap);
-- 
1.8.3.1

[Qemu-block] [PATCH 13/22] qcow2-bitmap: check constraints

[Vladimir Sementsov-Ogievskiy]

Check bitmap header constraints as specified in docs/specs/qcow2.txt

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 block/qcow2-bitmap.c | 39 +++
 1 file changed, 39 insertions(+)

diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index 8cf40f0..1c3abea 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -154,6 +154,34 @@ static inline void bitmap_directory_to_be(uint8_t *dir, 
size_t size)
 }
 }
 
+static int check_constraints(BlockDriverState *bs, Qcow2BitmapDirEntry *h)
+{
+BDRVQcow2State *s = bs->opaque;
+uint64_t phys_bitmap_bytes =
+(uint64_t)h->bitmap_table_size * s->cluster_size;
+uint64_t max_virtual_bits = (phys_bitmap_bytes * 8) << h->granularity_bits;
+int64_t nb_sectors = bdrv_nb_sectors(bs);
+
+if (nb_sectors < 0) {
+return nb_sectors;
+}
+
+int fail =
+((h->bitmap_table_size == 0) != (h->bitmap_table_offset == 0)) ||
+(h->bitmap_table_offset % s->cluster_size) ||
+(h->bitmap_table_size > BME_MAX_TABLE_SIZE) ||
+(phys_bitmap_bytes > BME_MAX_PHYS_SIZE) ||
+(h->bitmap_table_offset != 0 &&
+(nb_sectors << BDRV_SECTOR_BITS) > max_virtual_bits) ||
+(h->granularity_bits > BME_MAX_GRANULARITY_BITS) ||
+(h->granularity_bits < BME_MIN_GRANULARITY_BITS) ||
+(h->flags & BME_RESERVED_FLAGS) ||
+(h->name_size > BME_MAX_NAME_SIZE) ||
+(h->type != BT_DIRTY_TRACKING_BITMAP);
+
+return fail ? -EINVAL : 0;
+}
+
 static void clear_bitmap_table(BlockDriverState *bs, uint64_t *bitmap_table,
uint32_t bitmap_table_size)
 {
@@ -372,6 +400,12 @@ static uint8_t *directory_read(BlockDriverState *bs,
bdrv_get_device_or_node_name(bs));
 goto fail;
 }
+
+ret = check_constraints(bs, e);
+if (ret < 0) {
+error_setg(errp, "Bitmap doesn't satisfy the constraints.");
+goto fail;
+}
 }
 
 assert((uint8_t *)e == dir_end);
@@ -713,6 +747,11 @@ static int store_bitmap(BlockDriverState *bs,
 entry->extra_data_size = 0;
 memcpy(entry + 1, bm_name, entry->name_size);
 
+ret = check_constraints(bs, entry);
+if (ret < 0) {
+goto fail;
+}
+
 return 0;
 
 fail:
-- 
1.8.3.1

[Qemu-block] [PATCH 18/22] qapi: add md5 checksum of last dirty bitmap level to query-block

[Vladimir Sementsov-Ogievskiy]

Reviewed-by: John Snow 
Reviewed-by: Eric Blake 
Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 block/dirty-bitmap.c   | 1 +
 include/qemu/hbitmap.h | 8 
 qapi/block-core.json   | 5 -
 util/hbitmap.c | 8 
 4 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 392d660..d0fe382 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -359,6 +359,7 @@ BlockDirtyInfoList 
*bdrv_query_dirty_bitmaps(BlockDriverState *bs)
 info->has_name = !!bm->name;
 info->name = g_strdup(bm->name);
 info->status = bdrv_dirty_bitmap_status(bm);
+info->md5 = hbitmap_md5(bm->bitmap);
 entry->value = info;
 *plist = entry;
 plist = &entry->next;
diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h
index 64e5963..a47ce6a 100644
--- a/include/qemu/hbitmap.h
+++ b/include/qemu/hbitmap.h
@@ -240,6 +240,14 @@ void hbitmap_deserialize_ones(HBitmap *hb, uint64_t start, 
uint64_t count,
 void hbitmap_deserialize_finish(HBitmap *hb);
 
 /**
+ * hbitmap_md5:
+ * @bitmap: HBitmap to operate on.
+ *
+ * Returns md5 checksum of the last level.
+ */
+char *hbitmap_md5(const HBitmap *bitmap);
+
+/**
  * hbitmap_free:
  * @hb: HBitmap to operate on.
  *
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 087a681..8387e9c 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -419,11 +419,14 @@
 #
 # @status: current status of the dirty bitmap (since 2.4)
 #
+# @md5: md5 checksum (as a hexadecimal string) of the last bitmap level
+#   (since 2.8)
+#
 # Since: 1.3
 ##
 { 'struct': 'BlockDirtyInfo',
   'data': {'*name': 'str', 'count': 'int', 'granularity': 'uint32',
-   'status': 'DirtyBitmapStatus'} }
+   'status': 'DirtyBitmapStatus', 'md5': 'str'} }
 
 ##
 # @BlockInfo:
diff --git a/util/hbitmap.c b/util/hbitmap.c
index f49520f..1f0c417 100644
--- a/util/hbitmap.c
+++ b/util/hbitmap.c
@@ -707,3 +707,11 @@ void hbitmap_free_meta(HBitmap *hb)
 hbitmap_free(hb->meta);
 hb->meta = NULL;
 }
+
+char *hbitmap_md5(const HBitmap *bitmap)
+{
+uint64_t size =
+MAX((bitmap->size + BITS_PER_LONG - 1) >> BITS_PER_LEVEL, 1);
+const guchar *data = (const guchar *)bitmap->levels[HBITMAP_LEVELS - 1];
+return g_compute_checksum_for_data(G_CHECKSUM_MD5, data, size);
+}
-- 
1.8.3.1

[Qemu-block] [PATCH 05/22] qcow2-bitmap: structs and consts

[Vladimir Sementsov-Ogievskiy]

Create block/qcow2-bitmap.c
Add data structures and constraints accordingly to docs/specs/qcow2.txt

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 block/Makefile.objs  |  2 +-
 block/qcow2-bitmap.c | 47 +++
 block/qcow2.h| 29 +
 3 files changed, 77 insertions(+), 1 deletion(-)
 create mode 100644 block/qcow2-bitmap.c

diff --git a/block/Makefile.objs b/block/Makefile.objs
index fa4d8b8..0f661bb 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -1,5 +1,5 @@
 block-obj-y += raw_bsd.o qcow.o vdi.o vmdk.o cloop.o bochs.o vpc.o vvfat.o 
dmg.o
-block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o 
qcow2-cache.o
+block-obj-y += qcow2.o qcow2-refcount.o qcow2-cluster.o qcow2-snapshot.o 
qcow2-cache.o qcow2-bitmap.o
 block-obj-y += qed.o qed-gencb.o qed-l2-cache.o qed-table.o qed-cluster.o
 block-obj-y += qed-check.o
 block-obj-$(CONFIG_VHDX) += vhdx.o vhdx-endian.o vhdx-log.o
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
new file mode 100644
index 000..cd18b07
--- /dev/null
+++ b/block/qcow2-bitmap.c
@@ -0,0 +1,47 @@
+/*
+ * Bitmaps for the QCOW version 2 format
+ *
+ * Copyright (c) 2014-2016 Vladimir Sementsov-Ogievskiy
+ *
+ * This file is derived from qcow2-snapshot.c, original copyright:
+ * Copyright (c) 2004-2006 Fabrice Bellard
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to 
deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 
FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+/* NOTICE: BME here means Bitmaps Extension and used as a namespace for
+ * _internal_ constants. Please do not use this _internal_ abbreviation for
+ * other needs and/or outside of this file. */
+
+/* Bitmap directory entry constraints */
+#define BME_MAX_TABLE_SIZE 0x800
+#define BME_MAX_PHYS_SIZE 0x2000 /* 512 mb */
+#define BME_MAX_GRANULARITY_BITS 31
+#define BME_MIN_GRANULARITY_BITS 9
+#define BME_MAX_NAME_SIZE 1023
+
+/* Bitmap directory entry flags */
+#define BME_RESERVED_FLAGS 0x
+
+/* bits [1, 8] U [56, 63] are reserved */
+#define BME_TABLE_ENTRY_RESERVED_MASK 0xff0001fe
+
+typedef enum BitmapType {
+BT_DIRTY_TRACKING_BITMAP = 1
+} BitmapType;
diff --git a/block/qcow2.h b/block/qcow2.h
index b36a7bf..0480b8b 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -52,6 +52,10 @@
  * space for snapshot names and IDs */
 #define QCOW_MAX_SNAPSHOTS_SIZE (1024 * QCOW_MAX_SNAPSHOTS)
 
+/* Bitmap header extension constraints */
+#define QCOW_MAX_DIRTY_BITMAPS 65535
+#define QCOW_MAX_DIRTY_BITMAP_DIRECTORY_SIZE (1024 * QCOW_MAX_DIRTY_BITMAPS)
+
 /* indicate that the refcount of the referenced cluster is exactly one. */
 #define QCOW_OFLAG_COPIED (1ULL << 63)
 /* indicate that the cluster is compressed (they never have the copied flag) */
@@ -142,6 +146,22 @@ typedef struct QEMU_PACKED QCowSnapshotHeader {
 /* name follows  */
 } QCowSnapshotHeader;
 
+/* Qcow2BitmapDirEntry is actually a bitmap directory entry */
+typedef struct QEMU_PACKED Qcow2BitmapDirEntry {
+/* header is 8 byte aligned */
+uint64_t bitmap_table_offset;
+
+uint32_t bitmap_table_size;
+uint32_t flags;
+
+uint8_t type;
+uint8_t granularity_bits;
+uint16_t name_size;
+uint32_t extra_data_size;
+/* extra data follows  */
+/* name follows  */
+} Qcow2BitmapDirEntry;
+
 typedef struct QEMU_PACKED QCowSnapshotExtraData {
 uint64_t vm_state_size_large;
 uint64_t disk_size;
@@ -222,6 +242,15 @@ typedef uint64_t Qcow2GetRefcountFunc(const void 
*refcount_array,
 typedef void Qcow2SetRefcountFunc(void *refcount_array,
   uint64_t index, uint64_t value);
 
+/* Be careful, Qcow2BitmapHeaderExt is not an extension of 
Qcow2BitmapDirEntry, it
+ * is Qcow2 header extension */
+typedef struct Qcow2BitmapHeaderExt {
+uint32_t nb_bitmaps;
+uint32_t reserved32;
+uint64_t bitmap_directory_size;
+uint64_t bitmap_directory_offset;
+} QEMU_PACKED Qcow2BitmapHeaderExt;
+
 typedef struct BDRVQcow2S

Re: [Qemu-block] [PATCH 00/18] Dirty bitmaps postcopy migration

[Vladimir Sementsov-Ogievskiy]

ping

For now there are some notes mostly about accessory patches. What about 
migration itself?


On 16.08.2016 13:25, Vladimir Sementsov-Ogievskiy wrote:

v2:
some bugs fixed, iotests a bit changed and merged into one test.
based on block-next (https://github.com/XanClic/qemu/commits/block-next)
clone: tag postcopy-v2 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: 
https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fpostcopy-v2

v1:

These series are derived from my 'Dirty bitmaps migration' series. The
core idea is switch to postcopy migration and drop usage of meta
bitmaps.

These patches provide dirty bitmap postcopy migration feature. Only
named dirty bitmaps are to be migrated. Migration may be enabled using
migration capabilities.

The overall method (thanks to John Snow):

1. migrate bitmaps meta data in .save_live_setup
- create/find related bitmaps on target
- disable them
- create successors (anonimous children) only for enabled migrated
  bitmaps
2. do nothing in precopy stage
3. just before target vm start: enable successors, created in (1)
4. migrate bitmap data
5. reclaime bitmaps (merge successors to their parents)
6. enable bitmaps (only bitmaps, which was enabled in source)


Some patches are unchnaged from (v7) of 'Dirty bitmaps migration'
(DBMv7). I've left Reviewed-by's for them, if you don't like it, say me
and I'll drop them in the following version.

So, relatively to last DBMv7:

01-04: new patches, splitting common postcopy migration out of ram
postcopy migration
05: equal to DBMv7.05
06: new
07: equal to DBMv7.06
08: new
09: equal to DBMv7.07
10: new
11: derived from DBMv7.08, see below
12-15: equal to DBMv7.09-12
16: derived from DBMv7.13
- switch from fifo to socket, as postcopy don't work with fifo
  for now
- change parameters: size, granularity, regions
- add time.sleep, to wait for postcopy migration phase (bad
  temporary solution.
- drop Reviewed-by
17: new

11: the core patch of the series, it is derived from
[DBMv7.08: migration: add migration_block-dirty-bitmap.c]
There are a lot of changes related to switching from precopy to
postcopy, but functions related to migration stream itself
(structs, send/load sequences) are mostly unchnaged.

So, changes, to switch from precopy to postcopy:
- removed all staff related to meta bitmaps and dirty phase!!!
- add dirty_bitmap_mig_enable_successors, and call it before
  target vm start in loadvm_postcopy_handle_run
- add enabled_bitmaps list of bitmaps for
  dirty_bitmap_mig_enable_successors

- enabled flag is send with start bitmap chunk instead of
  completion chunk
- sectors_per_chunk is calculated directly from CHUNK_SIZE, not
  using meta bitmap granularity

- dirty_bitmap_save_iterate: remove dirty_phase, move bulk_phase
  to postcopy stage
- dirty_bitmap_save_pending: remove dirty phase related pending,
  switch pending to non-postcopyable
- dirty_bitmap_load_start: get enabled flag and prepare
  successors for enabled bitmaps, also add them to
  enabled_bitmaps list
- dirty_bitmap_load_complete: for enabled bitmaps: merge them
  with successors and enable

- savevm handlers:
  * remove separate savevm_dirty_bitmap_live_iterate_handlers state
(it was bad idea, any way), and move its save_live_iterate to
savevm_dirty_bitmap_handlers
  * add is_active_iterate savevm handler, which allows iterations
only in postcopy stage (after stopping source vm)
  * add has_postcopy savevm handler. (ofcourse, just returning true)
  * use save_live_complete_postcopy instead of
save_live_complete_precopy

Other changes:
- some debug output changed
- remove MIN_LIVE_SIZE, is_live_iterative and related staff (it
  was needed to omit iterations if bitmap data is small, possibly
  this should be reimplemented)

Vladimir Sementsov-Ogievskiy (18):
   migration: add has_postcopy savevm handler
   migration: fix ram_save_pending
   migration: split common postcopy out of ram postcopy
   migration: introduce postcopy-only pending
   block: add bdrv_next_dirty_bitmap()
   block: add bdrv_dirty_bitmap_enable_successor()
   qapi: add dirty-bitmaps migration capability
   block/dirty-bitmap: add bdrv_dirty_bitmap_release_successor
   migration: include migrate_dirty_bitmaps in migrate_postcopy
   migration/qemu-file: add qemu_put_counted_string()
   migration: add is_active_iterate handler
   migration: add postcopy migration of dirty bitmaps
   iotests: maintain several vms in test
   iotests: add add_incoming_migration to VM class
   qapi: add md5 checksum of last dirty bitmap level to query-block

[Qemu-block] [PATCH 21/22] specs/qcow2: fix bitmap granularity qemu-specific note

[Vladimir Sementsov-Ogievskiy]

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 docs/specs/qcow2.txt | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
index 80cdfd0..d3e292f 100644
--- a/docs/specs/qcow2.txt
+++ b/docs/specs/qcow2.txt
@@ -472,8 +472,7 @@ Structure of a bitmap directory entry:
  17:granularity_bits
 Granularity bits. Valid values: 0 - 63.
 
-Note: Qemu currently doesn't support granularity_bits
-greater than 31.
+Note: Qemu currently support only values 9 - 31.
 
 Granularity is calculated as
 granularity = 1 << granularity_bits
-- 
1.8.3.1

[Qemu-block] [PATCH 08/22] block/dirty-bitmap: add autoload field to BdrvDirtyBitmap

[Vladimir Sementsov-Ogievskiy]

Mirror AUTO flag from Qcow2 bitmap in BdrvDirtyBitmap. This will be
needed in future, to save this flag back to Qcow2 for persistent
bitmaps.

Signed-off-by: Vladimir Sementsov-Ogievskiy 
---
 block/dirty-bitmap.c | 15 +++
 block/qcow2-bitmap.c |  2 ++
 include/block/dirty-bitmap.h |  2 ++
 3 files changed, 19 insertions(+)

diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 90af372..623e1d1 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -44,6 +44,7 @@ struct BdrvDirtyBitmap {
 int64_t size;   /* Size of the bitmap (Number of sectors) */
 bool disabled;  /* Bitmap is read-only */
 int active_iterators;   /* How many iterators are active */
+bool autoload;  /* bitmap must be autoloaded on image opening 
*/
 QLIST_ENTRY(BdrvDirtyBitmap) list;
 };
 
@@ -70,6 +71,8 @@ void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap)
 assert(!bdrv_dirty_bitmap_frozen(bitmap));
 g_free(bitmap->name);
 bitmap->name = NULL;
+
+bitmap->autoload = false;
 }
 
 BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
@@ -238,6 +241,8 @@ BdrvDirtyBitmap 
*bdrv_dirty_bitmap_abdicate(BlockDriverState *bs,
 bitmap->name = NULL;
 successor->name = name;
 bitmap->successor = NULL;
+successor->autoload = bitmap->autoload;
+bitmap->autoload = false;
 bdrv_release_dirty_bitmap(bs, bitmap);
 
 return successor;
@@ -540,3 +545,13 @@ int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap)
 {
 return hbitmap_count(bitmap->meta);
 }
+
+void bdrv_dirty_bitmap_set_autoload(BdrvDirtyBitmap *bitmap, bool autoload)
+{
+bitmap->autoload = autoload;
+}
+
+bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap)
+{
+return bitmap->autoload;
+}
diff --git a/block/qcow2-bitmap.c b/block/qcow2-bitmap.c
index 760a047..c086436 100644
--- a/block/qcow2-bitmap.c
+++ b/block/qcow2-bitmap.c
@@ -529,6 +529,8 @@ int qcow2_read_bitmaps(BlockDriverState *bs, Error **errp)
 ret = -EINVAL;
 goto out;
 }
+
+bdrv_dirty_bitmap_set_autoload(bitmap, true);
 } else {
 /* leave bitmap in the image */
 new_pos += copy_dir_entry(new_pos, e);
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 1e17729..45a389a 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -75,4 +75,6 @@ void bdrv_dirty_bitmap_deserialize_ones(BdrvDirtyBitmap 
*bitmap,
 bool finish);
 void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);
 
+void bdrv_dirty_bitmap_set_autoload(BdrvDirtyBitmap *bitmap, bool autoload);
+bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap);
 #endif
-- 
1.8.3.1

[Qemu-block] [PATCH v8 00/36] block: Image locking series

[Fam Zheng]

Hi all,

I wanted to post something before the long holiday as promised, but I couldn't
refine or test enough due to limited time.  Please take this as an RFC and do a
high level review. Thanks.

v8: Move user interface option from block device to qdev. [Kevin]
Add "exclusive" back. [Kevin]

Note: limited by the qdev interface, until blk_lock_image() is called, the
images are not locked by block layer at open, because before device tells
us what to do, block layer cannot figure out the correct lock mode any
more.

TODO 1: Lock explicitly in utils (qemu-img, qemu-io, etc) after opening
image.

TODO 2: The image locking test case 153 patch is not updated thus broken
because of the moved option.

TODO 3: Are the open flags unnecessary because we already have
ImageLockMode? If so, how to converge them?

Fam Zheng (36):
  block: Add flag bits for image locking
  qapi: Add ImageLockMode
  block: Introduce image file locking
  osdep: Add qemu_lock_fd and qemu_unlock_fd
  raw-posix: Add image locking support
  qemu-io: Add "-L" option for BDRV_O_NO_LOCK
  qemu-img: Add "-L" option to sub commands
  qemu-img: Update documentation of "-L" option
  qemu-nbd: Add "--no-lock/-L" option
  block: Don't lock drive-backup target image in none mode
  block: Add blk_lock_image
  virtio-blk: Apply lock-mode when realize
  scsi-disk: Apply lock-mode when realize
  scsi-generic: Apply lock-mode when realize
  qdev: Add "lock-mode" to block device options
  ide: Apply lock-mode when initialize
  nvme: Apply lock-mode when initialize
  usb-storage: Apply lock-mode when realize
  pflash: Add "lock-mode" property
  qemu-iotests: 046: Move version detection out from verify_io
  qemu-iotests: 091: Prepare for image lock
  qemu-iotests: 030: Disable image locking when checking test image
  iotests: 087: Disable image locking in cases where file is shared
  iotests: 087: Disable image locking in cases where file is shared
  iotests: 130: Check image info locklessly
  iotests: Disable image locking in 085
  tests: Use null-co:// instead of /dev/null
  qemu-iotests: Add test case 153 for image locking
  ahci: Use shared lock for shared storage migration
  tests/postcopy: Use shared lock for images
  fdc: Add lock-mode qdev properties
  m25p80: Add 'lock-mode' property
  nand: Add 'lock-mode' property
  onenand: Add 'lock-mode' property
  spapr_nvram: Add 'lock-mode' property
  sd: Add 'lock-mode' property

 block.c|  52 +
 block/block-backend.c  |  18 ++
 block/raw-posix.c  | 318 +-
 blockdev.c |   5 +
 hw/block/fdc.c |  19 +-
 hw/block/m25p80.c  |   8 +
 hw/block/nand.c|   8 +
 hw/block/nvme.c|   6 +-
 hw/block/onenand.c |   7 +
 hw/block/pflash_cfi01.c|  10 +
 hw/block/pflash_cfi02.c|   9 +
 hw/block/virtio-blk.c  |   5 +
 hw/core/qdev-properties.c  |  10 +
 hw/ide/core.c  |  10 +-
 hw/ide/qdev.c  |   2 +-
 hw/nvram/spapr_nvram.c |   8 +
 hw/scsi/scsi-disk.c|   6 +
 hw/scsi/scsi-generic.c |   7 +
 hw/sd/sd.c |   8 +
 hw/usb/dev-storage.c   |   5 +
 include/block/block.h  |   8 +-
 include/block/block_int.h  |   5 +
 include/hw/block/block.h   |   3 +
 include/hw/ide/internal.h  |   3 +-
 include/hw/qdev-properties.h   |   3 +
 include/qemu/osdep.h   |   2 +
 include/sysemu/block-backend.h |   2 +
 qapi/block-core.json   |  18 ++
 qemu-img-cmds.hx   |  44 ++---
 qemu-img.c |  92 +++--
 qemu-img.texi  |   3 +
 qemu-io.c  |  24 ++-
 qemu-nbd.c |   7 +-
 qemu-nbd.texi  |   2 +
 tests/ahci-test.c  |  27 ++-
 tests/drive_del-test.c |   2 +-
 tests/nvme-test.c  |   2 +-
 tests/postcopy-test.c  |   9 +-
 tests/qemu-iotests/030 |   2 +-
 tests/qemu-iotests/046 |  22 ++-
 tests/qemu-iotests/085 |   6 +-
 tests/qemu-iotests/091 |   9 +-
 tests/qemu-iotests/091.out |   1 +
 tests/qemu-iotests/130 |   4 +-
 tests/qemu-iotests/130.out |   4 +-
 tests/qemu-iotests/153 | 197 +++
 tests/qemu-iotests/153.out | 426 +
 tests/qemu-iotests/group   |   1 +
 tests/usb-hcd-uhci-test.c  |   2 +-
 tests/usb-hcd-xhci-test.c  |   2 +-
 tests/virtio-blk-test.c|   2 +-
 tests/virtio-scsi-test.c   |   4 +-
 util/osdep.c   |  29 +++
 53 files changed, 1398 insertions(+), 90 deletions(-)
 create mode 100755 tests/qemu-iotests/153
 create mode 100644 tests/qemu-iotests/153.out

-- 
2.7.4

[Qemu-block] [PATCH v8 01/36] block: Add flag bits for image locking

[Fam Zheng]

Later the block layer will automatically lock the images to avoid unexpected
concurrent accesses to the same image, which will easily corrupt the metadata
or user data, unless in some very special cases, like migration.

The exceptional cases like shared storage migration and testing should
set BDRV_O_SHARED_LOCK or BDRV_O_NO_LOCK to advise an appropriate
locking mode.

Signed-off-by: Fam Zheng 
Reviewed-by: Max Reitz 
---
 include/block/block.h | 4 
 1 file changed, 4 insertions(+)

diff --git a/include/block/block.h b/include/block/block.h
index e18233a..cecab7a 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -98,7 +98,11 @@ typedef struct HDGeometry {
   select an appropriate protocol driver,
   ignoring the format layer */
 #define BDRV_O_NO_IO   0x1 /* don't initialize for I/O */
+#define BDRV_O_NO_LOCK 0x2 /* don't lock image file */
+#define BDRV_O_SHARED_LOCK 0x4 /* lock the image file in shared mode */
+#define BDRV_O_EXCLUSIVE_LOCK 0x8 /* lock the image file in exclusive mode 
*/
 
+#define BDRV_O_LOCK_MASK   (BDRV_O_NO_LOCK | BDRV_O_SHARED_LOCK | 
BDRV_O_EXCLUSIVE_LOCK)
 #define BDRV_O_CACHE_MASK  (BDRV_O_NOCACHE | BDRV_O_NO_FLUSH)
 
 
-- 
2.7.4

[Qemu-block] [PATCH v8 03/36] block: Introduce image file locking

[Fam Zheng]

Block drivers can implement this new operation .bdrv_lockf to actually lock the
image in the protocol specific way.

Signed-off-by: Fam Zheng 
---
 block.c   | 52 +++
 include/block/block.h |  4 +++-
 include/block/block_int.h |  5 +
 include/hw/block/block.h  |  2 ++
 4 files changed, 62 insertions(+), 1 deletion(-)

diff --git a/block.c b/block.c
index 493ecf3..9d600df 100644
--- a/block.c
+++ b/block.c
@@ -235,6 +235,7 @@ BlockDriverState *bdrv_new(void)
 notifier_with_return_list_init(&bs->before_write_notifiers);
 bs->refcnt = 1;
 bs->aio_context = qemu_get_aio_context();
+bs->cur_lock = IMAGE_LOCK_MODE__MAX;
 
 qemu_co_queue_init(&bs->flush_queue);
 
@@ -925,6 +926,48 @@ out:
 g_free(gen_node_name);
 }
 
+ImageLockMode bdrv_lock_mode_from_flags(int flags)
+{
+if (flags & BDRV_O_NO_LOCK) {
+return IMAGE_LOCK_MODE_NOLOCK;
+} else if (flags & BDRV_O_SHARED_LOCK) {
+return IMAGE_LOCK_MODE_SHARED;
+} else if (flags & BDRV_O_EXCLUSIVE_LOCK) {
+return IMAGE_LOCK_MODE_EXCLUSIVE;
+} else {
+return IMAGE_LOCK_MODE_AUTO;
+}
+}
+
+ImageLockMode bdrv_get_lock_mode(BlockDriverState *bs)
+{
+return bs->cur_lock;
+}
+
+int bdrv_set_lock_mode(BlockDriverState *bs, ImageLockMode mode)
+{
+int ret;
+
+if (bs->cur_lock == mode) {
+return 0;
+} else if (!bs->drv) {
+return -ENOMEDIUM;
+} else if (!bs->drv->bdrv_lockf) {
+if (bs->file) {
+return bdrv_set_lock_mode(bs->file->bs, mode);
+}
+return 0;
+}
+ret = bs->drv->bdrv_lockf(bs, mode);
+if (ret == -ENOTSUP) {
+/* Handle it the same way as !bs->drv->bdrv_lockf */
+ret = 0;
+} else if (ret == 0) {
+bs->cur_lock = mode;
+}
+return ret;
+}
+
 static QemuOptsList bdrv_runtime_opts = {
 .name = "bdrv_common",
 .head = QTAILQ_HEAD_INITIALIZER(bdrv_runtime_opts.head),
@@ -1076,6 +1119,10 @@ static int bdrv_open_common(BlockDriverState *bs, 
BdrvChild *file,
 goto free_and_fail;
 }
 
+if (open_flags & BDRV_O_INACTIVE) {
+open_flags = (open_flags & ~BDRV_O_LOCK_MASK) & BDRV_O_NO_LOCK;
+}
+
 ret = refresh_total_sectors(bs, bs->total_sectors);
 if (ret < 0) {
 error_setg_errno(errp, -ret, "Could not refresh total sector count");
@@ -2273,6 +2320,7 @@ static void bdrv_close(BlockDriverState *bs)
 if (bs->drv) {
 BdrvChild *child, *next;
 
+bdrv_set_lock_mode(bs, IMAGE_LOCK_MODE_NOLOCK);
 bs->drv->bdrv_close(bs);
 bs->drv = NULL;
 
@@ -3188,6 +3236,9 @@ void bdrv_invalidate_cache(BlockDriverState *bs, Error 
**errp)
 error_setg_errno(errp, -ret, "Could not refresh total sector count");
 return;
 }
+if (bs->cur_lock != IMAGE_LOCK_MODE__MAX) {
+bdrv_set_lock_mode(bs, bs->cur_lock);
+}
 }
 
 void bdrv_invalidate_cache_all(Error **errp)
@@ -3230,6 +3281,7 @@ static int bdrv_inactivate_recurse(BlockDriverState *bs,
 }
 
 if (setting_flag) {
+ret = bdrv_set_lock_mode(bs, IMAGE_LOCK_MODE_NOLOCK);
 bs->open_flags |= BDRV_O_INACTIVE;
 }
 return 0;
diff --git a/include/block/block.h b/include/block/block.h
index cecab7a..828ab14 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -272,7 +272,8 @@ int bdrv_drop_intermediate(BlockDriverState *active, 
BlockDriverState *top,
 BlockDriverState *bdrv_find_overlay(BlockDriverState *active,
 BlockDriverState *bs);
 BlockDriverState *bdrv_find_base(BlockDriverState *bs);
-
+ImageLockMode bdrv_lock_mode_from_flags(int flags);
+ImageLockMode bdrv_get_lock_mode(BlockDriverState *bs);
 
 typedef struct BdrvCheckResult {
 int corruptions;
@@ -529,5 +530,6 @@ void bdrv_drained_end(BlockDriverState *bs);
 void bdrv_add_child(BlockDriverState *parent, BlockDriverState *child,
 Error **errp);
 void bdrv_del_child(BlockDriverState *parent, BdrvChild *child, Error **errp);
+int bdrv_set_lock_mode(BlockDriverState *bs, ImageLockMode mode);
 
 #endif
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 3e79228..486e1ea 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -319,6 +319,10 @@ struct BlockDriver {
Error **errp);
 void (*bdrv_del_child)(BlockDriverState *parent, BdrvChild *child,
Error **errp);
+/**
+ * Lock/unlock the image.
+ */
+int (*bdrv_lockf)(BlockDriverState *bs, ImageLockMode mode);
 
 QLIST_ENTRY(BlockDriver) list;
 };
@@ -528,6 +532,7 @@ struct BlockDriverState {
 unsigned io_plug_disabled;
 
 int quiesce_counter;
+ImageLockMode cur_lock;
 };
 
 struct BlockBackendRootState {
diff --git a/include/hw/block/block.h b/include/hw/block/block.h
index df9d207..8c04469 100644
--- a/include/hw/block/block.h
+++ b/include/hw/block/bloc

[Qemu-block] [PATCH v8 04/36] osdep: Add qemu_lock_fd and qemu_unlock_fd

[Fam Zheng]

They are wrappers of POSIX fcntl "file private locking".

Signed-off-by: Fam Zheng 
---
 include/qemu/osdep.h |  2 ++
 util/osdep.c | 29 +
 2 files changed, 31 insertions(+)

diff --git a/include/qemu/osdep.h b/include/qemu/osdep.h
index 9e9fa61..f773f49 100644
--- a/include/qemu/osdep.h
+++ b/include/qemu/osdep.h
@@ -286,6 +286,8 @@ int qemu_close(int fd);
 #ifndef _WIN32
 int qemu_dup(int fd);
 #endif
+int qemu_lock_fd(int fd, int64_t start, int64_t len, bool exclusive);
+int qemu_unlock_fd(int fd, int64_t start, int64_t len);
 
 #if defined(__HAIKU__) && defined(__i386__)
 #define FMT_pid "%ld"
diff --git a/util/osdep.c b/util/osdep.c
index 06fb1cf..b85a490 100644
--- a/util/osdep.c
+++ b/util/osdep.c
@@ -140,6 +140,35 @@ static int qemu_parse_fdset(const char *param)
 {
 return qemu_parse_fd(param);
 }
+
+static int qemu_lock_fcntl(int fd, int64_t start, int64_t len, int fl_type)
+{
+#ifdef F_OFD_SETLK
+int ret;
+struct flock fl = {
+.l_whence = SEEK_SET,
+.l_start  = start,
+.l_len= len,
+.l_type   = fl_type,
+};
+do {
+ret = fcntl(fd, F_OFD_SETLK, &fl);
+} while (ret == -1 && errno == EINTR);
+return ret == -1 ? -errno : 0;
+#else
+return -ENOTSUP;
+#endif
+}
+
+int qemu_lock_fd(int fd, int64_t start, int64_t len, bool exclusive)
+{
+return qemu_lock_fcntl(fd, start, len, exclusive ? F_WRLCK : F_RDLCK);
+}
+
+int qemu_unlock_fd(int fd, int64_t start, int64_t len)
+{
+return qemu_lock_fcntl(fd, start, len, F_UNLCK);
+}
 #endif
 
 /*
-- 
2.7.4

[Qemu-block] [PATCH v8 02/36] qapi: Add ImageLockMode

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 qapi/block-core.json | 18 ++
 1 file changed, 18 insertions(+)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index 92193ab..22e8d04 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -2754,3 +2754,21 @@
   'data' : { 'parent': 'str',
  '*child': 'str',
  '*node': 'str' } }
+
+##
+# @ImageLockMode:
+#
+# @auto: defer to the block driver to use the least strict mode, based on
+#the nature of format and read-only flag, and the supported locking
+#operations of the protocol.
+#
+# @exclusive: always exclusively lock the image.
+#
+# @shared: use a shared lock mode.
+#
+# @nolock: don't lock the image.
+#
+# Since: 2.8
+##
+{ 'enum': 'ImageLockMode',
+  'data': ['auto', 'exclusive', 'shared', 'nolock'] }
-- 
2.7.4

[Qemu-block] [PATCH v8 13/36] scsi-disk: Apply lock-mode when realize

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 hw/scsi/scsi-disk.c | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/hw/scsi/scsi-disk.c b/hw/scsi/scsi-disk.c
index 88beaf4..fb406a0 100644
--- a/hw/scsi/scsi-disk.c
+++ b/hw/scsi/scsi-disk.c
@@ -2288,6 +2288,12 @@ static void scsi_realize(SCSIDevice *dev, Error **errp)
 return;
 }
 
+blk_lock_image(s->qdev.conf.blk, s->qdev.conf.lock_mode, &err);
+if (err) {
+error_propagate(errp, err);
+return;
+}
+
 if (!(s->features & (1 << SCSI_DISK_F_REMOVABLE)) &&
 !blk_is_inserted(s->qdev.conf.blk)) {
 error_setg(errp, "Device needs media, but drive is empty");
-- 
2.7.4

[Qemu-block] [PATCH v8 08/36] qemu-img: Update documentation of "-L" option

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 qemu-img-cmds.hx | 44 ++--
 qemu-img.c   |  1 +
 qemu-img.texi|  3 +++
 3 files changed, 26 insertions(+), 22 deletions(-)

diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx
index f054599..eaca454 100644
--- a/qemu-img-cmds.hx
+++ b/qemu-img-cmds.hx
@@ -16,33 +16,33 @@ STEXI
 ETEXI
 
 DEF("check", img_check,
-"check [-q] [--object objectdef] [--image-opts] [-f fmt] [--output=ofmt] 
[-r [leaks | all]] [-T src_cache] filename")
+"check [-q] [-L] [--object objectdef] [--image-opts] [-f fmt] 
[--output=ofmt] [-r [leaks | all]] [-T src_cache] filename")
 STEXI
-@item check [--object @var{objectdef}] [--image-opts] [-q] [-f @var{fmt}] 
[--output=@var{ofmt}] [-r [leaks | all]] [-T @var{src_cache}] @var{filename}
+@item check [--object @var{objectdef}] [--image-opts] [-q] [-L] [-f @var{fmt}] 
[--output=@var{ofmt}] [-r [leaks | all]] [-T @var{src_cache}] @var{filename}
 ETEXI
 
 DEF("create", img_create,
-"create [-q] [--object objectdef] [--image-opts] [-f fmt] [-o options] 
filename [size]")
+"create [-q] [-L] [--object objectdef] [--image-opts] [-f fmt] [-o 
options] filename [size]")
 STEXI
-@item create [--object @var{objectdef}] [--image-opts] [-q] [-f @var{fmt}] [-o 
@var{options}] @var{filename} [@var{size}]
+@item create [--object @var{objectdef}] [--image-opts] [-q] [-L] [-f 
@var{fmt}] [-o @var{options}] @var{filename} [@var{size}]
 ETEXI
 
 DEF("commit", img_commit,
-"commit [-q] [--object objectdef] [--image-opts] [-f fmt] [-t cache] [-b 
base] [-d] [-p] filename")
+"commit [-q] [-L] [--object objectdef] [--image-opts] [-f fmt] [-t cache] 
[-b base] [-d] [-p] filename")
 STEXI
-@item commit [--object @var{objectdef}] [--image-opts] [-q] [-f @var{fmt}] [-t 
@var{cache}] [-b @var{base}] [-d] [-p] @var{filename}
+@item commit [--object @var{objectdef}] [--image-opts] [-q] [-L] [-f 
@var{fmt}] [-t @var{cache}] [-b @var{base}] [-d] [-p] @var{filename}
 ETEXI
 
 DEF("compare", img_compare,
-"compare [--object objectdef] [--image-opts] [-f fmt] [-F fmt] [-T 
src_cache] [-p] [-q] [-s] filename1 filename2")
+"compare [--object objectdef] [--image-opts] [-f fmt] [-F fmt] [-T 
src_cache] [-p] [-q] [-L] [-s] filename1 filename2")
 STEXI
-@item compare [--object @var{objectdef}] [--image-opts] [-f @var{fmt}] [-F 
@var{fmt}] [-T @var{src_cache}] [-p] [-q] [-s] @var{filename1} @var{filename2}
+@item compare [--object @var{objectdef}] [--image-opts] [-f @var{fmt}] [-F 
@var{fmt}] [-T @var{src_cache}] [-p] [-q] [-L] [-s] @var{filename1} 
@var{filename2}
 ETEXI
 
 DEF("convert", img_convert,
-"convert [--object objectdef] [--image-opts] [-c] [-p] [-q] [-n] [-f fmt] 
[-t cache] [-T src_cache] [-O output_fmt] [-o options] [-s snapshot_id_or_name] 
[-l snapshot_param] [-S sparse_size] filename [filename2 [...]] 
output_filename")
+"convert [--object objectdef] [--image-opts] [-c] [-p] [-q] [-L] [-n] [-f 
fmt] [-t cache] [-T src_cache] [-O output_fmt] [-o options] [-s 
snapshot_id_or_name] [-l snapshot_param] [-S sparse_size] filename [filename2 
[...]] output_filename")
 STEXI
-@item convert [--object @var{objectdef}] [--image-opts] [-c] [-p] [-q] [-n] 
[-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] [-o 
@var{options}] [-s @var{snapshot_id_or_name}] [-l @var{snapshot_param}] [-S 
@var{sparse_size}] @var{filename} [@var{filename2} [...]] @var{output_filename}
+@item convert [--object @var{objectdef}] [--image-opts] [-c] [-p] [-q] [-L] 
[-n] [-f @var{fmt}] [-t @var{cache}] [-T @var{src_cache}] [-O @var{output_fmt}] 
[-o @var{options}] [-s @var{snapshot_id_or_name}] [-l @var{snapshot_param}] [-S 
@var{sparse_size}] @var{filename} [@var{filename2} [...]] @var{output_filename}
 ETEXI
 
 DEF("dd", img_dd,
@@ -52,38 +52,38 @@ STEXI
 ETEXI
 
 DEF("info", img_info,
-"info [--object objectdef] [--image-opts] [-f fmt] [--output=ofmt] 
[--backing-chain] filename")
+"info [--object objectdef] [--image-opts] [-f fmt] [-L] [--output=ofmt] 
[--backing-chain] filename")
 STEXI
-@item info [--object @var{objectdef}] [--image-opts] [-f @var{fmt}] 
[--output=@var{ofmt}] [--backing-chain] @var{filename}
+@item info [--object @var{objectdef}] [--image-opts] [-f @var{fmt}] [-L] 
[--output=@var{ofmt}] [--backing-chain] @var{filename}
 ETEXI
 
 DEF("map", img_map,
-"map [--object objectdef] [--image-opts] [-f fmt] [--output=ofmt] 
filename")
+"map [--object objectdef] [--image-opts] [-f fmt] [-L] [--output=ofmt] 
filename")
 STEXI
-@item map [--object @var{objectdef}] [--image-opts] [-f @var{fmt}] 
[--output=@var{ofmt}] @var{filename}
+@item map [--object @var{objectdef}] [--image-opts] [-f @var{fmt}] [-L] 
[--output=@var{ofmt}] @var{filename}
 ETEXI
 
 DEF("snapshot", img_snapshot,
-"snapshot [--object objectdef] [--image-opts] [-q] [-l | -a snapshot | -c 
snapshot | -d snapshot] filename")
+"snapshot [--object objectdef] [--image-opts] [-q] [-L] [-l | -a snapshot 
| -c snapshot 

[Qemu-block] [PATCH v8 06/36] qemu-io: Add "-L" option for BDRV_O_NO_LOCK

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 qemu-io.c | 24 +---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/qemu-io.c b/qemu-io.c
index db129ea..5c14eba 100644
--- a/qemu-io.c
+++ b/qemu-io.c
@@ -108,6 +108,7 @@ static void open_help(void)
 " -r, -- open file read-only\n"
 " -s, -- use snapshot file\n"
 " -n, -- disable host cache, short for -t none\n"
+" -L, -- disable image locking\n"
 " -k, -- use kernel AIO implementation (on Linux only)\n"
 " -t, -- use the given cache mode for the image\n"
 " -d, -- use the given discard mode for the image\n"
@@ -124,7 +125,7 @@ static const cmdinfo_t open_cmd = {
 .argmin = 1,
 .argmax = -1,
 .flags  = CMD_NOFILE_OK,
-.args   = "[-rsnk] [-t cache] [-d discard] [-o options] [path]",
+.args   = "[-rsnLk] [-t cache] [-d discard] [-o options] [path]",
 .oneline= "open the file specified by path",
 .help   = open_help,
 };
@@ -143,12 +144,13 @@ static int open_f(BlockBackend *blk, int argc, char 
**argv)
 {
 int flags = BDRV_O_UNMAP;
 int readonly = 0;
+bool nolock = false;
 bool writethrough = true;
 int c;
 QemuOpts *qopts;
 QDict *opts;
 
-while ((c = getopt(argc, argv, "snro:kt:d:")) != -1) {
+while ((c = getopt(argc, argv, "snrLo:kt:d:")) != -1) {
 switch (c) {
 case 's':
 flags |= BDRV_O_SNAPSHOT;
@@ -177,6 +179,9 @@ static int open_f(BlockBackend *blk, int argc, char **argv)
 return 0;
 }
 break;
+case 'L':
+nolock = true;
+break;
 case 'o':
 if (imageOpts) {
 printf("--image-opts and 'open -o' are mutually exclusive\n");
@@ -198,6 +203,10 @@ static int open_f(BlockBackend *blk, int argc, char **argv)
 flags |= BDRV_O_RDWR;
 }
 
+if (nolock) {
+flags |= BDRV_O_NO_LOCK;
+}
+
 if (imageOpts && (optind == argc - 1)) {
 if (!qemu_opts_parse_noisily(&empty_opts, argv[optind], false)) {
 qemu_opts_reset(&empty_opts);
@@ -436,13 +445,15 @@ static QemuOptsList file_opts = {
 int main(int argc, char **argv)
 {
 int readonly = 0;
-const char *sopt = "hVc:d:f:rsnmkt:T:";
+const char *sopt = "hVc:d:f:rLsnmkt:T:";
+bool nolock = false;
 const struct option lopt[] = {
 { "help", no_argument, NULL, 'h' },
 { "version", no_argument, NULL, 'V' },
 { "cmd", required_argument, NULL, 'c' },
 { "format", required_argument, NULL, 'f' },
 { "read-only", no_argument, NULL, 'r' },
+{ "no-lock", no_argument, NULL, 'L' },
 { "snapshot", no_argument, NULL, 's' },
 { "nocache", no_argument, NULL, 'n' },
 { "misalign", no_argument, NULL, 'm' },
@@ -501,6 +512,9 @@ int main(int argc, char **argv)
 case 'r':
 readonly = 1;
 break;
+case 'L':
+nolock = true;
+break;
 case 'm':
 qemuio_misalign = true;
 break;
@@ -586,6 +600,10 @@ int main(int argc, char **argv)
 flags |= BDRV_O_RDWR;
 }
 
+if (nolock) {
+flags |= BDRV_O_NO_LOCK;
+}
+
 if ((argc - optind) == 1) {
 if (imageOpts) {
 QemuOpts *qopts = NULL;
-- 
2.7.4

[Qemu-block] [PATCH v8 07/36] qemu-img: Add "-L" option to sub commands

[Fam Zheng]

If specified, BDRV_O_NO_LOCK flag will be set when opening the image.

Signed-off-by: Fam Zheng 
---
 qemu-img.c | 91 +-
 1 file changed, 72 insertions(+), 19 deletions(-)

diff --git a/qemu-img.c b/qemu-img.c
index ceffefe..e8d0e78 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -616,6 +616,7 @@ static int img_check(int argc, char **argv)
 ImageCheck *check;
 bool quiet = false;
 bool image_opts = false;
+bool nolock = false;
 
 fmt = NULL;
 output = NULL;
@@ -632,7 +633,7 @@ static int img_check(int argc, char **argv)
 {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS},
 {0, 0, 0, 0}
 };
-c = getopt_long(argc, argv, "hf:r:T:q",
+c = getopt_long(argc, argv, "hf:r:T:qL",
 long_options, &option_index);
 if (c == -1) {
 break;
@@ -666,6 +667,9 @@ static int img_check(int argc, char **argv)
 case 'q':
 quiet = true;
 break;
+case 'L':
+nolock = true;
+break;
 case OPTION_OBJECT: {
 QemuOpts *opts;
 opts = qemu_opts_parse_noisily(&qemu_object_opts,
@@ -699,6 +703,7 @@ static int img_check(int argc, char **argv)
 return 1;
 }
 
+flags |= nolock ? BDRV_O_NO_LOCK : 0;
 ret = bdrv_parse_cache_mode(cache, &flags, &writethrough);
 if (ret < 0) {
 error_report("Invalid source cache option: %s", cache);
@@ -819,6 +824,7 @@ static int img_commit(int argc, char **argv)
 Error *local_err = NULL;
 CommonBlockJobCBInfo cbi;
 bool image_opts = false;
+bool nolock = false;
 
 fmt = NULL;
 cache = BDRV_DEFAULT_CACHE;
@@ -830,7 +836,7 @@ static int img_commit(int argc, char **argv)
 {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS},
 {0, 0, 0, 0}
 };
-c = getopt_long(argc, argv, "f:ht:b:dpq",
+c = getopt_long(argc, argv, "f:ht:b:dpqL",
 long_options, NULL);
 if (c == -1) {
 break;
@@ -860,6 +866,9 @@ static int img_commit(int argc, char **argv)
 case 'q':
 quiet = true;
 break;
+case 'L':
+nolock = true;
+break;
 case OPTION_OBJECT: {
 QemuOpts *opts;
 opts = qemu_opts_parse_noisily(&qemu_object_opts,
@@ -891,6 +900,7 @@ static int img_commit(int argc, char **argv)
 }
 
 flags = BDRV_O_RDWR | BDRV_O_UNMAP;
+flags |= nolock ? BDRV_O_NO_LOCK : 0;
 ret = bdrv_parse_cache_mode(cache, &flags, &writethrough);
 if (ret < 0) {
 error_report("Invalid cache option: %s", cache);
@@ -1149,6 +1159,7 @@ static int img_compare(int argc, char **argv)
 int c, pnum;
 uint64_t progress_base;
 bool image_opts = false;
+bool nolock = false;
 
 cache = BDRV_DEFAULT_CACHE;
 for (;;) {
@@ -1158,7 +1169,7 @@ static int img_compare(int argc, char **argv)
 {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS},
 {0, 0, 0, 0}
 };
-c = getopt_long(argc, argv, "hf:F:T:pqs",
+c = getopt_long(argc, argv, "hf:F:T:pqsL",
 long_options, NULL);
 if (c == -1) {
 break;
@@ -1186,6 +1197,9 @@ static int img_compare(int argc, char **argv)
 case 's':
 strict = true;
 break;
+case 'L':
+nolock = true;
+break;
 case OPTION_OBJECT: {
 QemuOpts *opts;
 opts = qemu_opts_parse_noisily(&qemu_object_opts,
@@ -1223,7 +1237,7 @@ static int img_compare(int argc, char **argv)
 /* Initialize before goto out */
 qemu_progress_init(progress, 2.0);
 
-flags = 0;
+flags = nolock ? BDRV_O_NO_LOCK : 0;
 ret = bdrv_parse_cache_mode(cache, &flags, &writethrough);
 if (ret < 0) {
 error_report("Invalid source cache option: %s", cache);
@@ -1774,6 +1788,7 @@ static int img_convert(int argc, char **argv)
 QemuOpts *sn_opts = NULL;
 ImgConvertState state;
 bool image_opts = false;
+bool nolock = false;
 
 fmt = NULL;
 out_fmt = "raw";
@@ -1789,7 +1804,7 @@ static int img_convert(int argc, char **argv)
 {"image-opts", no_argument, 0, OPTION_IMAGE_OPTS},
 {0, 0, 0, 0}
 };
-c = getopt_long(argc, argv, "hf:O:B:ce6o:s:l:S:pt:T:qn",
+c = getopt_long(argc, argv, "hf:O:B:ce6o:s:l:S:pt:T:qnL",
 long_options, NULL);
 if (c == -1) {
 break;
@@ -1878,6 +1893,9 @@ static int img_convert(int argc, char **argv)
 case 'q':
 quiet = true;
 break;
+case 'L':
+nolock = true;
+break;
 case 'n':
 skip_create = 1;
 break;
@@ -1926,7 +1944,7 @@ static int img_convert(int argc, char **argv)
 goto out;
 }
 
-src_flags = 0;
+s

[Qemu-block] [PATCH v8 05/36] raw-posix: Add image locking support

[Fam Zheng]

virtlockd in libvirt locks the first byte, we lock byte 1 to avoid
the intervene.

Both file and host device protocols are covered.

The complication is with reopen. We have three different locking states,
namely "unlocked", "shared locked" and "exclusively locked".

When we reopen, the new fd may need a new locking mode. Moving away to or from
exclusive is a bit tricky because we cannot do it atomically. This patch solves
it by dup() s->fd to s->lock_fd and avoid close(), so that there isn't a racy
window where we drop the lock on one fd before acquiring the exclusive lock on
the other.

To make the logic easier to manage, and allow better reuse, the code is
internally organized by state transition table (old_lock -> new_lock).

Signed-off-by: Fam Zheng 
---
 block/raw-posix.c | 318 +-
 1 file changed, 317 insertions(+), 1 deletion(-)

diff --git a/block/raw-posix.c b/block/raw-posix.c
index 6ed7547..22de242 100644
--- a/block/raw-posix.c
+++ b/block/raw-posix.c
@@ -133,6 +133,7 @@ do { \
 
 typedef struct BDRVRawState {
 int fd;
+int lock_fd;
 int type;
 int open_flags;
 size_t buf_align;
@@ -149,6 +150,7 @@ typedef struct BDRVRawState {
 
 typedef struct BDRVRawReopenState {
 int fd;
+int lock_fd;
 int open_flags;
 } BDRVRawReopenState;
 
@@ -367,6 +369,43 @@ static void raw_parse_flags(int bdrv_flags, int 
*open_flags)
 }
 }
 
+static int raw_lock_fd(int fd, ImageLockMode mode)
+{
+assert(fd >= 0);
+/* Locking byte 1 avoids interfereing with virtlockd. */
+switch (mode) {
+case IMAGE_LOCK_MODE_EXCLUSIVE:
+return qemu_lock_fd(fd, 1, 1, true);
+case IMAGE_LOCK_MODE_SHARED:
+return qemu_lock_fd(fd, 1, 1, false);
+case IMAGE_LOCK_MODE_NOLOCK:
+return qemu_unlock_fd(fd, 1, 1);
+default:
+abort();
+}
+}
+
+static int raw_lockf(BlockDriverState *bs, ImageLockMode mode)
+{
+BDRVRawState *s = bs->opaque;
+
+if (s->lock_fd < 0) {
+if (mode == IMAGE_LOCK_MODE_NOLOCK) {
+return 0;
+}
+s->lock_fd = qemu_dup(s->fd);
+if (s->lock_fd < 0) {
+return s->lock_fd;
+}
+}
+if (mode == IMAGE_LOCK_MODE_AUTO) {
+mode = bdrv_get_flags(bs) & BDRV_O_RDWR ?
+   IMAGE_LOCK_MODE_EXCLUSIVE :
+   IMAGE_LOCK_MODE_SHARED;
+}
+return raw_lock_fd(s->lock_fd, mode);
+}
+
 #ifdef CONFIG_LINUX_AIO
 static bool raw_use_aio(int bdrv_flags)
 {
@@ -433,6 +472,7 @@ static int raw_open_common(BlockDriverState *bs, QDict 
*options,
 raw_parse_flags(bdrv_flags, &s->open_flags);
 
 s->fd = -1;
+s->lock_fd = -1;
 fd = qemu_open(filename, s->open_flags, 0644);
 if (fd < 0) {
 ret = -errno;
@@ -529,6 +569,268 @@ static int raw_open(BlockDriverState *bs, QDict *options, 
int flags,
 return raw_open_common(bs, options, flags, 0, errp);
 }
 
+typedef enum {
+RAW_REOPEN_PREPARE,
+RAW_REOPEN_COMMIT,
+RAW_REOPEN_ABORT
+} RawReopenOperation;
+
+typedef int (*RawReopenFunc)(BDRVReopenState *state,
+ RawReopenOperation op,
+ ImageLockMode old_lock,
+ ImageLockMode new_lock,
+ Error **errp);
+
+static int
+raw_reopen_identical(BDRVReopenState *state,
+ RawReopenOperation op,
+ ImageLockMode old_lock,
+ ImageLockMode new_lock,
+ Error **errp)
+{
+assert(old_lock == new_lock);
+return 0;
+}
+
+static int
+raw_reopen_from_unlock(BDRVReopenState *state,
+   RawReopenOperation op,
+   ImageLockMode old_lock,
+   ImageLockMode new_lock,
+   Error **errp)
+{
+BDRVRawReopenState *raw_s = state->opaque;
+int ret = 0;
+
+assert(old_lock != new_lock);
+assert(old_lock == IMAGE_LOCK_MODE_NOLOCK);
+switch (op) {
+case RAW_REOPEN_PREPARE:
+ret = raw_lock_fd(raw_s->lock_fd, new_lock);
+if (ret) {
+error_setg_errno(errp, -ret, "Failed to lock new fd %d", 
raw_s->lock_fd);
+}
+break;
+case RAW_REOPEN_COMMIT:
+case RAW_REOPEN_ABORT:
+break;
+}
+
+return ret;
+}
+
+static int
+raw_reopen_to_unlock(BDRVReopenState *state,
+ RawReopenOperation op,
+ ImageLockMode old_lock,
+ ImageLockMode new_lock,
+ Error **errp)
+{
+BDRVRawState *s = state->bs->opaque;
+int ret = 0;
+
+assert(old_lock != new_lock);
+assert(new_lock == IMAGE_LOCK_MODE_NOLOCK);
+switch (op) {
+case RAW_REOPEN_PREPARE:
+break;
+case RAW_REOPEN_COMMIT:
+if (s->lock_fd >= 0) {
+qemu_close(s->lock_fd);
+s->lock_fd = -1;
+}
+break;
+case RAW_REOPEN_ABORT:
+break;
+}
+
+ 

[Qemu-block] [PATCH v8 09/36] qemu-nbd: Add "--no-lock/-L" option

[Fam Zheng]

Signed-off-by: Fam Zheng 
Reviewed-by: Max Reitz 
---
 qemu-nbd.c| 7 ++-
 qemu-nbd.texi | 2 ++
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/qemu-nbd.c b/qemu-nbd.c
index 99297a5..6585b2c 100644
--- a/qemu-nbd.c
+++ b/qemu-nbd.c
@@ -102,6 +102,7 @@ static void usage(const char *name)
 "Block device options:\n"
 "  -f, --format=FORMAT   set image format (raw, qcow2, ...)\n"
 "  -r, --read-only   export read-only\n"
+"  -L, --no-lock disable image locking\n"
 "  -s, --snapshotuse FILE as an external snapshot, create a 
temporary\n"
 "file with backing_file=FILE, redirect the write 
to\n"
 "the temporary one\n"
@@ -474,7 +475,7 @@ int main(int argc, char **argv)
 off_t fd_size;
 QemuOpts *sn_opts = NULL;
 const char *sn_id_or_name = NULL;
-const char *sopt = "hVb:o:p:rsnP:c:dvk:e:f:tl:x:T:";
+const char *sopt = "hVb:o:p:rsnLP:c:dvk:e:f:tl:x:T:";
 struct option lopt[] = {
 { "help", no_argument, NULL, 'h' },
 { "version", no_argument, NULL, 'V' },
@@ -483,6 +484,7 @@ int main(int argc, char **argv)
 { "socket", required_argument, NULL, 'k' },
 { "offset", required_argument, NULL, 'o' },
 { "read-only", no_argument, NULL, 'r' },
+{ "no-lock", no_argument, NULL, 'L' },
 { "partition", required_argument, NULL, 'P' },
 { "connect", required_argument, NULL, 'c' },
 { "disconnect", no_argument, NULL, 'd' },
@@ -638,6 +640,9 @@ int main(int argc, char **argv)
 nbdflags |= NBD_FLAG_READ_ONLY;
 flags &= ~BDRV_O_RDWR;
 break;
+case 'L':
+flags |= BDRV_O_NO_LOCK;
+break;
 case 'P':
 partition = strtol(optarg, &end, 0);
 if (*end) {
diff --git a/qemu-nbd.texi b/qemu-nbd.texi
index 91ebf04..5936b37 100644
--- a/qemu-nbd.texi
+++ b/qemu-nbd.texi
@@ -43,6 +43,8 @@ Force the use of the block driver for format @var{fmt} 
instead of
 auto-detecting
 @item -r, --read-only
 Export the disk as read-only
+@item -L, --no-lock
+Disable image locking
 @item -P, --partition=@var{num}
 Only expose partition @var{num}
 @item -s, --snapshot
-- 
2.7.4

[Qemu-block] [PATCH v8 10/36] block: Don't lock drive-backup target image in none mode

[Fam Zheng]

As a very special case, in sync=none mode, the source is the backing image of
the target, which will be RO opened again. This won't work with image locking
because the first open could be exclusive.

Signed-off-by: Fam Zheng 
Reviewed-by: Max Reitz 
---
 blockdev.c | 5 +
 1 file changed, 5 insertions(+)

diff --git a/blockdev.c b/blockdev.c
index 29c6561..a9a7dee 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -3211,6 +3211,11 @@ static void do_drive_backup(DriveBackup *backup, 
BlockJobTxn *txn, Error **errp)
 }
 }
 if (backup->sync == MIRROR_SYNC_MODE_NONE) {
+/* XXX: bs will be open second time as the backing file of target,
+ * disable image locking. Once block layer allows sharing backing BDS,
+ * change below to BDRV_O_NO_BACKING and assign it after bdrv_open().
+ **/
+flags |= BDRV_O_NO_LOCK;
 source = bs;
 }
 
-- 
2.7.4

[Qemu-block] [PATCH v8 14/36] scsi-generic: Apply lock-mode when realize

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 hw/scsi/scsi-generic.c | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/hw/scsi/scsi-generic.c b/hw/scsi/scsi-generic.c
index 7a588a7..4f131e8 100644
--- a/hw/scsi/scsi-generic.c
+++ b/hw/scsi/scsi-generic.c
@@ -502,12 +502,19 @@ static void scsi_generic_realize(SCSIDevice *s, Error 
**errp)
 int rc;
 int sg_version;
 struct sg_scsi_id scsiid;
+Error *err = NULL;
 
 if (!s->conf.blk) {
 error_setg(errp, "drive property not set");
 return;
 }
 
+blk_lock_image(s->conf.blk, s->conf.lock_mode, &err);
+if (err) {
+error_propagate(errp, err);
+return;
+}
+
 if (blk_get_on_error(s->conf.blk, 0) != BLOCKDEV_ON_ERROR_ENOSPC) {
 error_setg(errp, "Device doesn't support drive option werror");
 return;
-- 
2.7.4

[Qemu-block] [PATCH v8 18/36] usb-storage: Apply lock-mode when realize

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 hw/usb/dev-storage.c | 5 +
 1 file changed, 5 insertions(+)

diff --git a/hw/usb/dev-storage.c b/hw/usb/dev-storage.c
index c607f76..6d0c00e 100644
--- a/hw/usb/dev-storage.c
+++ b/hw/usb/dev-storage.c
@@ -600,6 +600,11 @@ static void usb_msd_realize_storage(USBDevice *dev, Error 
**errp)
 error_setg(errp, "drive property not set");
 return;
 }
+blk_lock_image(blk, s->conf.lock_mode, &err);
+if (err) {
+error_propagate(errp, err);
+return;
+}
 
 blkconf_serial(&s->conf, &dev->serial);
 blkconf_blocksizes(&s->conf);
-- 
2.7.4

[Qemu-block] [PATCH v8 21/36] qemu-iotests: 091: Prepare for image lock

[Fam Zheng]

We should wait for the QEMU process to terminate and close the image
before we check the data.

Also use shared lock in migration source and target.

Signed-off-by: Fam Zheng 
---
 tests/qemu-iotests/091 | 9 +++--
 tests/qemu-iotests/091.out | 1 +
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/091 b/tests/qemu-iotests/091
index 32bbd56..d54ab73 100755
--- a/tests/qemu-iotests/091
+++ b/tests/qemu-iotests/091
@@ -61,13 +61,15 @@ echo === Starting QEMU VM1 ===
 echo
 
 qemu_comm_method="monitor"
-_launch_qemu -drive file="${TEST_IMG}",cache=${CACHEMODE},id=disk
+_launch_qemu -drive file="${TEST_IMG}",cache=${CACHEMODE},id=disk,if=none \
+ -device virtio-blk,drive=disk,lock-mode=shared
 h1=$QEMU_HANDLE
 
 echo
 echo === Starting QEMU VM2 ===
 echo
-_launch_qemu -drive file="${TEST_IMG}",cache=${CACHEMODE},id=disk \
+_launch_qemu -drive file="${TEST_IMG}",cache=${CACHEMODE},id=disk,if=none \
+ -device virtio-blk,drive=disk,lock-mode=shared \
  -incoming "exec: cat '${MIG_FIFO}'"
 h2=$QEMU_HANDLE
 
@@ -95,6 +97,9 @@ echo "vm2: qemu process running successfully"
 echo "vm2: flush io, and quit"
 _send_qemu_cmd $h2 'qemu-io disk flush' "(qemu)"
 _send_qemu_cmd $h2 'quit' ""
+echo "vm1: quit"
+_send_qemu_cmd $h1 'quit' ""
+wait
 
 echo "Check image pattern"
 ${QEMU_IO} -c "read -P 0x22 0 4M" "${TEST_IMG}" | _filter_testdir | 
_filter_qemu_io
diff --git a/tests/qemu-iotests/091.out b/tests/qemu-iotests/091.out
index 5017f8c..6658ca8 100644
--- a/tests/qemu-iotests/091.out
+++ b/tests/qemu-iotests/091.out
@@ -18,6 +18,7 @@ vm1: live migration completed
 vm2: qemu-io disk write complete
 vm2: qemu process running successfully
 vm2: flush io, and quit
+vm1: quit
 Check image pattern
 read 4194304/4194304 bytes at offset 0
 4 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-- 
2.7.4

[Qemu-block] [PATCH v8 24/36] iotests: 087: Disable image locking in cases where file is shared

[Fam Zheng]

Otherwise the error handling we are expecting will be masked by the
preceding image locking check, and is going to be indistinguishable
because the error messages are all the same.

Signed-off-by: Fam Zheng 
---
 tests/qemu-iotests/087 | 2 --
 1 file changed, 2 deletions(-)

diff --git a/tests/qemu-iotests/087 b/tests/qemu-iotests/087
index 91e4c71..5c04577 100755
--- a/tests/qemu-iotests/087
+++ b/tests/qemu-iotests/087
@@ -84,7 +84,6 @@ run_qemu -drive 
driver=$IMGFMT,id=disk,node-name=test-node,file="$TEST_IMG" <

[Qemu-block] [PATCH v8 11/36] block: Add blk_lock_image

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 block/block-backend.c  | 18 ++
 include/sysemu/block-backend.h |  2 ++
 2 files changed, 20 insertions(+)

diff --git a/block/block-backend.c b/block/block-backend.c
index 0bd19ab..dc1ad36 100644
--- a/block/block-backend.c
+++ b/block/block-backend.c
@@ -61,6 +61,7 @@ struct BlockBackend {
 bool allow_write_beyond_eof;
 
 NotifierList remove_bs_notifiers, insert_bs_notifiers;
+ImageLockMode lock_mode;
 };
 
 typedef struct BlockBackendAIOCB {
@@ -1720,3 +1721,20 @@ static void blk_root_drained_end(BdrvChild *child)
 assert(blk->public.io_limits_disabled);
 --blk->public.io_limits_disabled;
 }
+
+void blk_lock_image(BlockBackend *blk, ImageLockMode mode, Error **errp)
+{
+int r;
+BlockDriverState *bs = blk_bs(blk);
+
+if (!bs) {
+blk->lock_mode = mode;
+return;
+}
+r = bdrv_set_lock_mode(bs, mode);
+if (r) {
+error_setg(errp, "Failed to lock image");
+} else {
+blk->lock_mode = mode;
+}
+}
diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h
index 3b29317..78f9da5 100644
--- a/include/sysemu/block-backend.h
+++ b/include/sysemu/block-backend.h
@@ -224,4 +224,6 @@ void blk_io_limits_disable(BlockBackend *blk);
 void blk_io_limits_enable(BlockBackend *blk, const char *group);
 void blk_io_limits_update_group(BlockBackend *blk, const char *group);
 
+void blk_lock_image(BlockBackend *blk, ImageLockMode mode, Error **errp);
+
 #endif
-- 
2.7.4

[Qemu-block] [PATCH v8 12/36] virtio-blk: Apply lock-mode when realize

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 hw/block/virtio-blk.c | 5 +
 1 file changed, 5 insertions(+)

diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index 3a6112f..ce65615 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -896,6 +896,11 @@ static void virtio_blk_device_realize(DeviceState *dev, 
Error **errp)
 error_setg(errp, "num-queues property must be larger than 0");
 return;
 }
+blk_lock_image(conf->conf.blk, conf->conf.lock_mode, &err);
+if (err) {
+error_propagate(errp, err);
+return;
+}
 
 blkconf_serial(&conf->conf, &conf->serial);
 blkconf_apply_backend_options(&conf->conf);
-- 
2.7.4

[Qemu-block] [PATCH v8 16/36] ide: Apply lock-mode when initialize

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 hw/ide/core.c | 10 +-
 hw/ide/qdev.c |  2 +-
 include/hw/ide/internal.h |  3 ++-
 3 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/hw/ide/core.c b/hw/ide/core.c
index b0e42a6..a426baf 100644
--- a/hw/ide/core.c
+++ b/hw/ide/core.c
@@ -28,6 +28,7 @@
 #include "hw/pci/pci.h"
 #include "hw/isa/isa.h"
 #include "qemu/error-report.h"
+#include "qapi/error.h"
 #include "qemu/timer.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/dma.h"
@@ -2394,17 +2395,24 @@ static const BlockDevOps ide_hd_block_ops = {
 .resize_cb = ide_resize_cb,
 };
 
-int ide_init_drive(IDEState *s, BlockBackend *blk, IDEDriveKind kind,
+int ide_init_drive(IDEState *s, BlockBackend *blk, ImageLockMode lock_mode,
+   IDEDriveKind kind,
const char *version, const char *serial, const char *model,
uint64_t wwn,
uint32_t cylinders, uint32_t heads, uint32_t secs,
int chs_trans)
 {
+Error *local_err = NULL;
 uint64_t nb_sectors;
 
 s->blk = blk;
 s->drive_kind = kind;
 
+blk_lock_image(blk, lock_mode, &local_err);
+if (local_err) {
+error_report_err(local_err);
+return -1;
+}
 blk_get_geometry(blk, &nb_sectors);
 s->cylinders = cylinders;
 s->heads = heads;
diff --git a/hw/ide/qdev.c b/hw/ide/qdev.c
index 2eb055a..c5b88fb 100644
--- a/hw/ide/qdev.c
+++ b/hw/ide/qdev.c
@@ -188,7 +188,7 @@ static int ide_dev_initfn(IDEDevice *dev, IDEDriveKind kind)
 }
 blkconf_apply_backend_options(&dev->conf);
 
-if (ide_init_drive(s, dev->conf.blk, kind,
+if (ide_init_drive(s, dev->conf.blk, dev->conf.lock_mode, kind,
dev->version, dev->serial, dev->model, dev->wwn,
dev->conf.cyls, dev->conf.heads, dev->conf.secs,
dev->chs_trans) < 0) {
diff --git a/include/hw/ide/internal.h b/include/hw/ide/internal.h
index a6dd2c3..828f88c 100644
--- a/include/hw/ide/internal.h
+++ b/include/hw/ide/internal.h
@@ -600,7 +600,8 @@ uint32_t ide_data_readw(void *opaque, uint32_t addr);
 void ide_data_writel(void *opaque, uint32_t addr, uint32_t val);
 uint32_t ide_data_readl(void *opaque, uint32_t addr);
 
-int ide_init_drive(IDEState *s, BlockBackend *blk, IDEDriveKind kind,
+int ide_init_drive(IDEState *s, BlockBackend *blk, ImageLockMode lock_mode,
+   IDEDriveKind kind,
const char *version, const char *serial, const char *model,
uint64_t wwn,
uint32_t cylinders, uint32_t heads, uint32_t secs,
-- 
2.7.4

[Qemu-block] [PATCH v8 17/36] nvme: Apply lock-mode when initialize

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 hw/block/nvme.c | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/hw/block/nvme.c b/hw/block/nvme.c
index cef3bb4..318dc94 100644
--- a/hw/block/nvme.c
+++ b/hw/block/nvme.c
@@ -829,7 +829,7 @@ static int nvme_init(PCIDevice *pci_dev)
 {
 NvmeCtrl *n = NVME(pci_dev);
 NvmeIdCtrl *id = &n->id_ctrl;
-
+Error *local_err = NULL;
 int i;
 int64_t bs_size;
 uint8_t *pci_conf;
@@ -837,6 +837,10 @@ static int nvme_init(PCIDevice *pci_dev)
 if (!n->conf.blk) {
 return -1;
 }
+blk_lock_image(n->conf.blk, n->conf.lock_mode, &local_err);
+if (local_err) {
+return -1;
+}
 
 bs_size = blk_getlength(n->conf.blk);
 if (bs_size < 0) {
-- 
2.7.4

[Qemu-block] [PATCH v8 15/36] qdev: Add "lock-mode" to block device options

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 hw/core/qdev-properties.c| 10 ++
 include/hw/block/block.h |  1 +
 include/hw/qdev-properties.h |  3 +++
 3 files changed, 14 insertions(+)

diff --git a/hw/core/qdev-properties.c b/hw/core/qdev-properties.c
index 311af6d..829b143 100644
--- a/hw/core/qdev-properties.c
+++ b/hw/core/qdev-properties.c
@@ -552,6 +552,16 @@ PropertyInfo qdev_prop_blockdev_on_error = {
 .set = set_enum,
 };
 
+/* --- Block image lock mode --- */
+
+PropertyInfo qdev_prop_lock_mode = {
+.name  = "ImageLockMode",
+.description = "Image lock mode for drive (auto/shared/exclusive/nolock)",
+.enum_table = ImageLockMode_lookup,
+.get = get_enum,
+.set = set_enum,
+};
+
 /* --- BIOS CHS translation */
 
 QEMU_BUILD_BUG_ON(sizeof(BiosAtaTranslation) != sizeof(int));
diff --git a/include/hw/block/block.h b/include/hw/block/block.h
index 8c04469..520d142 100644
--- a/include/hw/block/block.h
+++ b/include/hw/block/block.h
@@ -47,6 +47,7 @@ static inline unsigned int get_physical_block_exp(BlockConf 
*conf)
 
 #define DEFINE_BLOCK_PROPERTIES(_state, _conf)  \
 DEFINE_PROP_DRIVE("drive", _state, _conf.blk),  \
+DEFINE_PROP_LOCK_MODE("lock-mode", _state, _conf.lock_mode),\
 DEFINE_PROP_BLOCKSIZE("logical_block_size", _state, \
   _conf.logical_block_size),\
 DEFINE_PROP_BLOCKSIZE("physical_block_size", _state,\
diff --git a/include/hw/qdev-properties.h b/include/hw/qdev-properties.h
index 2a9d2f9..1bce8ee 100644
--- a/include/hw/qdev-properties.h
+++ b/include/hw/qdev-properties.h
@@ -30,6 +30,7 @@ extern PropertyInfo qdev_prop_pci_devfn;
 extern PropertyInfo qdev_prop_blocksize;
 extern PropertyInfo qdev_prop_pci_host_devaddr;
 extern PropertyInfo qdev_prop_arraylen;
+extern PropertyInfo qdev_prop_lock_mode;
 
 #define DEFINE_PROP(_name, _state, _field, _prop, _type) { \
 .name  = (_name),\
@@ -155,6 +156,8 @@ extern PropertyInfo qdev_prop_arraylen;
 DEFINE_PROP(_n, _s, _f, qdev_prop_vlan, NICPeers)
 #define DEFINE_PROP_DRIVE(_n, _s, _f) \
 DEFINE_PROP(_n, _s, _f, qdev_prop_drive, BlockBackend *)
+#define DEFINE_PROP_LOCK_MODE(_n, _s, _f) \
+DEFINE_PROP(_n, _s, _f, qdev_prop_lock_mode, ImageLockMode)
 #define DEFINE_PROP_MACADDR(_n, _s, _f) \
 DEFINE_PROP(_n, _s, _f, qdev_prop_macaddr, MACAddr)
 #define DEFINE_PROP_ON_OFF_AUTO(_n, _s, _f, _d) \
-- 
2.7.4

[Qemu-block] [PATCH v8 20/36] qemu-iotests: 046: Move version detection out from verify_io

[Fam Zheng]

So the image lock won't complain.

Signed-off-by: Fam Zheng 
---
 tests/qemu-iotests/046 | 22 --
 1 file changed, 12 insertions(+), 10 deletions(-)

diff --git a/tests/qemu-iotests/046 b/tests/qemu-iotests/046
index e528b67..f15ccbf 100755
--- a/tests/qemu-iotests/046
+++ b/tests/qemu-iotests/046
@@ -192,15 +192,7 @@ echo "== Verify image content =="
 
 function verify_io()
 {
-if ($QEMU_IMG info -f "$IMGFMT" "$TEST_IMG" | grep "compat: 0.10" > 
/dev/null); then
-# For v2 images, discarded clusters are read from the backing file
-# Keep the variable empty so that the backing file value can be used as
-# the default below
-discarded=
-else
-# Discarded clusters are zeroed for v3 or later
-discarded=0
-fi
+discarded=$1
 
 echo read -P 0 0 0x1
 
@@ -261,7 +253,17 @@ function verify_io()
 echo read -P 17  0x11c000 0x4000
 }
 
-verify_io | $QEMU_IO "$TEST_IMG" | _filter_qemu_io
+if ($QEMU_IMG info -f "$IMGFMT" "$TEST_IMG" | grep "compat: 0.10" > 
/dev/null); then
+# For v2 images, discarded clusters are read from the backing file
+# Keep the variable empty so that the backing file value can be used as
+# the default below
+discarded=
+else
+# Discarded clusters are zeroed for v3 or later
+discarded=0
+fi
+
+verify_io $discarded | $QEMU_IO "$TEST_IMG" | _filter_qemu_io
 
 _check_test_img
 
-- 
2.7.4

[Qemu-block] [PATCH v8 19/36] pflash: Add "lock-mode" property

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 hw/block/pflash_cfi01.c | 10 ++
 hw/block/pflash_cfi02.c |  9 +
 2 files changed, 19 insertions(+)

diff --git a/hw/block/pflash_cfi01.c b/hw/block/pflash_cfi01.c
index 62d7a56..df82471 100644
--- a/hw/block/pflash_cfi01.c
+++ b/hw/block/pflash_cfi01.c
@@ -76,6 +76,7 @@ struct pflash_t {
 /*< public >*/
 
 BlockBackend *blk;
+ImageLockMode lock_mode;
 uint32_t nb_blocs;
 uint64_t sector_len;
 uint8_t bank_width;
@@ -738,6 +739,14 @@ static void pflash_cfi01_realize(DeviceState *dev, Error 
**errp)
 sysbus_init_mmio(SYS_BUS_DEVICE(dev), &pfl->mem);
 
 if (pfl->blk) {
+Error *local_err = NULL;
+
+blk_lock_image(pfl->blk, pfl->lock_mode, &local_err);
+if (local_err) {
+vmstate_unregister_ram(&pfl->mem, DEVICE(pfl));
+error_propagate(errp, local_err);
+return;
+}
 /* read the initial flash content */
 ret = blk_pread(pfl->blk, 0, pfl->storage, total_len);
 
@@ -852,6 +861,7 @@ static void pflash_cfi01_realize(DeviceState *dev, Error 
**errp)
 
 static Property pflash_cfi01_properties[] = {
 DEFINE_PROP_DRIVE("drive", struct pflash_t, blk),
+DEFINE_PROP_LOCK_MODE("lock-mode", struct pflash_t, lock_mode),
 /* num-blocks is the number of blocks actually visible to the guest,
  * ie the total size of the device divided by the sector length.
  * If we're emulating flash devices wired in parallel the actual
diff --git a/hw/block/pflash_cfi02.c b/hw/block/pflash_cfi02.c
index 4f6105c..ff6a9fb 100644
--- a/hw/block/pflash_cfi02.c
+++ b/hw/block/pflash_cfi02.c
@@ -620,6 +620,14 @@ static void pflash_cfi02_realize(DeviceState *dev, Error 
**errp)
 pfl->storage = memory_region_get_ram_ptr(&pfl->orig_mem);
 pfl->chip_len = chip_len;
 if (pfl->blk) {
+Error *local_err = NULL;
+
+blk_lock_image(pfl->blk, pfl->lock_mode, &local_err);
+if (local_err) {
+vmstate_unregister_ram(&pfl->mem, DEVICE(pfl));
+error_propagate(errp, local_err);
+return;
+}
 /* read the initial flash content */
 ret = blk_pread(pfl->blk, 0, pfl->storage, chip_len);
 if (ret < 0) {
@@ -724,6 +732,7 @@ static void pflash_cfi02_realize(DeviceState *dev, Error 
**errp)
 
 static Property pflash_cfi02_properties[] = {
 DEFINE_PROP_DRIVE("drive", struct pflash_t, blk),
+DEFINE_PROP_LOCK_MODE("lock-mode", struct pflash_t, lock_mode),
 DEFINE_PROP_UINT32("num-blocks", struct pflash_t, nb_blocs, 0),
 DEFINE_PROP_UINT32("sector-length", struct pflash_t, sector_len, 0),
 DEFINE_PROP_UINT8("width", struct pflash_t, width, 0),
-- 
2.7.4

[Qemu-block] [PATCH v8 22/36] qemu-iotests: 030: Disable image locking when checking test image

[Fam Zheng]

The VM is running, qemu-io would fail the lock acquisition.

Signed-off-by: Fam Zheng 
Reviewed-by: Max Reitz 
---
 tests/qemu-iotests/030 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030
index 107049b..acae67e 100755
--- a/tests/qemu-iotests/030
+++ b/tests/qemu-iotests/030
@@ -95,7 +95,7 @@ class TestSingleDrive(iotests.QMPTestCase):
 self.assert_no_active_block_jobs()
 
 # The image map is empty before the operation
-empty_map = qemu_io('-f', iotests.imgfmt, '-c', 'map', test_img)
+empty_map = qemu_io('-L', '-f', iotests.imgfmt, '-c', 'map', test_img)
 
 # This is a no-op: no data should ever be copied from the base image
 result = self.vm.qmp('block-stream', device='drive0', base=mid_img)
-- 
2.7.4

[Qemu-block] [PATCH v8 28/36] qemu-iotests: Add test case 153 for image locking

[Fam Zheng]

Signed-off-by: Fam Zheng 
Reviewed-by: Max Reitz 
---
 tests/qemu-iotests/153 | 197 +
 tests/qemu-iotests/153.out | 426 +
 tests/qemu-iotests/group   |   1 +
 3 files changed, 624 insertions(+)
 create mode 100755 tests/qemu-iotests/153
 create mode 100644 tests/qemu-iotests/153.out

diff --git a/tests/qemu-iotests/153 b/tests/qemu-iotests/153
new file mode 100755
index 000..68770ec
--- /dev/null
+++ b/tests/qemu-iotests/153
@@ -0,0 +1,197 @@
+#!/bin/bash
+#
+# Test image locking
+#
+# Copyright (C) 2016 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see .
+#
+
+# creator
+owner=f...@redhat.com
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+here="$PWD"
+tmp=/tmp/$$
+status=1   # failure is the default!
+
+_cleanup()
+{
+_cleanup_test_img
+rm -f "${TEST_IMG}.base"
+rm -f "${TEST_IMG}.convert"
+rm -f "${TEST_IMG}.a"
+rm -f "${TEST_IMG}.b"
+rm -f "${TEST_IMG}.lnk"
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.qemu
+
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+size=32M
+
+_run_cmd()
+{
+echo
+(echo "$@"; "$@" 2>&1 1>/dev/null) | _filter_testdir
+}
+
+function _do_run_qemu()
+{
+(
+if ! test -t 0; then
+while read cmd; do
+echo $cmd
+done
+fi
+echo quit
+) | $QEMU -nographic -monitor stdio -serial none "$@" 1>/dev/null
+}
+
+function _run_qemu_with_images()
+{
+_do_run_qemu \
+$(for i in $@; do echo "-drive if=none,file=$i"; done) 2>&1 \
+| _filter_testdir | _filter_qemu
+}
+
+for opts1 in "" "readonly=on" "lock-mode=off" "lock-mode=shared"; do
+echo
+echo "== Creating base image =="
+TEST_IMG="${TEST_IMG}.base" _make_test_img $size
+
+echo
+echo "== Creating test image =="
+$QEMU_IMG create -f $IMGFMT "${TEST_IMG}" -b ${TEST_IMG}.base | 
_filter_img_create
+
+echo
+echo "== Launching QEMU $opts1 =="
+_launch_qemu -drive file="${TEST_IMG}",if=none,$opts1
+h=$QEMU_HANDLE
+
+for opts2 in "" "lock-mode=auto" "lock-mode=shared" \
+ "lock-mode=off" "lock-mode=auto,readonly=on"; do
+echo
+echo "== Launching another QEMU $opts2 =="
+echo "quit" | \
+$QEMU -nographic -monitor stdio \
+-drive file="${TEST_IMG}",if=none,$opts2 2>&1 1>/dev/null | \
+_filter_testdir | _filter_qemu
+done
+
+for L in "" "-L"; do
+
+echo
+echo "== Running utility commands $(test -n "$L" && echo "(lock 
disabled)") =="
+_run_cmd $QEMU_IO $L -c "read 0 512" "${TEST_IMG}"
+_run_cmd $QEMU_IO $L -r -c "read 0 512" "${TEST_IMG}"
+_run_cmd $QEMU_IMG info$L "${TEST_IMG}"
+_run_cmd $QEMU_IMG check   $L "${TEST_IMG}"
+_run_cmd $QEMU_IMG compare $L "${TEST_IMG}" "${TEST_IMG}"
+_run_cmd $QEMU_IMG map $L "${TEST_IMG}"
+_run_cmd $QEMU_IMG amend -o "" $L "${TEST_IMG}"
+_run_cmd $QEMU_IMG commit  $L "${TEST_IMG}"
+_run_cmd $QEMU_IMG resize  $L "${TEST_IMG}" $size
+_run_cmd $QEMU_IMG rebase  $L "${TEST_IMG}" -b "${TEST_IMG}.base"
+_run_cmd $QEMU_IMG snapshot -l $L "${TEST_IMG}"
+_run_cmd $QEMU_IMG convert $L "${TEST_IMG}" "${TEST_IMG}.convert"
+done
+_send_qemu_cmd $h "{ 'execute': 'quit', }" ""
+echo
+echo "Round done"
+_cleanup_qemu
+done
+
+function _enum_opts()
+{
+echo lock-mode={shared,auto,off},readonly={on,off}
+}
+
+for opt1 in $(_enum_opts); do
+for opt2 in $(_enum_opts); do
+echo
+echo "== Two devices with the same image ($opt1 - $opt2) =="
+_run_qemu_with_images "${TEST_IMG},$opt1" "${TEST_IMG},$opt2"
+done
+done
+
+(
+$QEMU_IMG create -f qcow2 "${TEST_IMG}.a" -b "${TEST_IMG}"
+$QEMU_IMG create -f qcow2 "${TEST_IMG}.b" -b "${TEST_IMG}"
+$QEMU_IMG create -f qcow2 "${TEST_IMG}.c" -b "${TEST_IMG}.b"
+) | _filter_img_create
+
+echo
+echo "== Two devices sharing the same file in backing chain =="
+_run_qemu_with_images "${TEST_IMG}.a" "${TEST_IMG}.b"
+_run_qemu_with_images "${TEST_IMG}.a" "${TEST_IMG}.c"
+
+echo
+echo "== Backing image also as an active device =="
+_run_qemu_with_imag

[Qemu-block] [PATCH v8 26/36] iotests: Disable image locking in 085

[Fam Zheng]

The cases is about live snapshot features. Disable image locking because
otherwise a few tests are going to fail because we reuse the same images
at blockdev-add.

Signed-off-by: Fam Zheng 
---
 tests/qemu-iotests/085 | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/085 b/tests/qemu-iotests/085
index aa77eca..59bfd0e 100755
--- a/tests/qemu-iotests/085
+++ b/tests/qemu-iotests/085
@@ -102,6 +102,7 @@ function add_snapshot_image()
 cmd="{ 'execute': 'blockdev-add', 'arguments':
{ 'options':
  { 'driver': 'qcow2', 'node-name': 'snap_${1}', ${extra_params}
+   'read-only': true,
'file':
{ 'driver': 'file', 'filename': '${snapshot_file}',
  'node-name': 'file_${1}' } } } }"
@@ -130,7 +131,10 @@ echo === Running QEMU ===
 echo
 
 qemu_comm_method="qmp"
-_launch_qemu -drive file="${TEST_IMG}.1",if=virtio -drive 
file="${TEST_IMG}.2",if=virtio
+_launch_qemu -drive file="${TEST_IMG}.1",if=none,id=virtio0 \
+ -device virtio-blk,drive=virtio0,lock-mode=nolock \
+ -drive file="${TEST_IMG}.2",if=none,id=virtio1 \
+ -device virtio-blk,drive=virtio1,lock-mode=nolock
 h=$QEMU_HANDLE
 
 echo
-- 
2.7.4

[Qemu-block] [PATCH v8 30/36] tests/postcopy: Use shared lock for images

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 tests/postcopy-test.c | 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/tests/postcopy-test.c b/tests/postcopy-test.c
index 41ed1a9..3ec4d01 100644
--- a/tests/postcopy-test.c
+++ b/tests/postcopy-test.c
@@ -371,12 +371,14 @@ static void test_migrate(void)
 cmd_src = g_strdup_printf("-machine accel=kvm:tcg -m 150M"
   " -name pcsource,debug-threads=on"
   " -serial file:%s/src_serial"
-  " -drive file=%s,format=raw",
+  " -drive 
file=%s,format=raw,if=none,id=drive1"
+  " -device 
ide-hd,drive=drive1,lock-mode=shared",
   tmpfs, bootpath);
 cmd_dst = g_strdup_printf("-machine accel=kvm:tcg -m 150M"
   " -name pcdest,debug-threads=on"
   " -serial file:%s/dest_serial"
-  " -drive file=%s,format=raw"
+  " -drive 
file=%s,format=raw,if=none,id=drive2"
+  " -device 
ide-hd,drive=drive2,lock-mode=shared"
   " -incoming %s",
   tmpfs, bootpath, uri);
 } else if (strcmp(arch, "ppc64") == 0) {
@@ -384,7 +386,8 @@ static void test_migrate(void)
 cmd_src = g_strdup_printf("-machine accel=kvm:tcg -m 256M"
   " -name pcsource,debug-threads=on"
   " -serial file:%s/src_serial"
-  " -drive file=%s,if=pflash,format=raw",
+  " -drive 
file=%s,if=none,format=raw,id=drive3"
+  " -device 
pflash,drive=drive3,lock-mode=shared",
   tmpfs, bootpath);
 cmd_dst = g_strdup_printf("-machine accel=kvm:tcg -m 256M"
   " -name pcdest,debug-threads=on"
-- 
2.7.4

[Qemu-block] [PATCH v8 33/36] nand: Add 'lock-mode' property

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 hw/block/nand.c | 8 
 1 file changed, 8 insertions(+)

diff --git a/hw/block/nand.c b/hw/block/nand.c
index c69e675..75b5a68 100644
--- a/hw/block/nand.c
+++ b/hw/block/nand.c
@@ -64,6 +64,7 @@ struct NANDFlashState {
 int page_shift, oob_shift, erase_shift, addr_shift;
 uint8_t *storage;
 BlockBackend *blk;
+ImageLockMode lock_mode;
 int mem_oob;
 
 uint8_t cle, ale, ce, wp, gnd;
@@ -373,6 +374,7 @@ static void nand_realize(DeviceState *dev, Error **errp)
 {
 int pagesize;
 NANDFlashState *s = NAND(dev);
+Error *local_err = NULL;
 
 s->buswidth = nand_flash_ids[s->chip_id].width >> 3;
 s->size = nand_flash_ids[s->chip_id].size << 20;
@@ -407,6 +409,11 @@ static void nand_realize(DeviceState *dev, Error **errp)
 error_setg(errp, "Can't use a read-only drive");
 return;
 }
+blk_lock_image(s->blk, s->lock_mode, &local_err);
+if (local_err) {
+error_propagate(errp, local_err);
+return;
+}
 if (blk_getlength(s->blk) >=
 (s->pages << s->page_shift) + (s->pages << s->oob_shift)) {
 pagesize = 0;
@@ -427,6 +434,7 @@ static Property nand_properties[] = {
 DEFINE_PROP_UINT8("manufacturer_id", NANDFlashState, manf_id, 0),
 DEFINE_PROP_UINT8("chip_id", NANDFlashState, chip_id, 0),
 DEFINE_PROP_DRIVE("drive", NANDFlashState, blk),
+DEFINE_PROP_LOCK_MODE("lock-mode", NANDFlashState, lock_mode),
 DEFINE_PROP_END_OF_LIST(),
 };
 
-- 
2.7.4

[Qemu-block] [PATCH v8 29/36] ahci: Use shared lock for shared storage migration

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 tests/ahci-test.c | 27 +--
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/tests/ahci-test.c b/tests/ahci-test.c
index 9c0adce..838ff45 100644
--- a/tests/ahci-test.c
+++ b/tests/ahci-test.c
@@ -1132,10 +1132,14 @@ static void test_migrate_sanity(void)
 char *uri = g_strdup_printf("unix:%s", mig_socket);
 
 src = ahci_boot("-m 1024 -M q35 "
-"-drive if=ide,file=%s,format=%s ", tmp_path, imgfmt);
+"-drive if=none,id=drive0,file=%s,format=%s "
+"-device ide-hd,drive=drive0,lock-mode=shared ",
+tmp_path, imgfmt);
 dst = ahci_boot("-m 1024 -M q35 "
-"-drive if=ide,file=%s,format=%s "
-"-incoming %s", tmp_path, imgfmt, uri);
+"-drive if=none,id=drive0,file=%s,format=%s "
+"-device ide-hd,drive=drive0,lock-mode=shared "
+"-incoming %s",
+tmp_path, imgfmt, uri);
 
 ahci_migrate(src, dst, uri);
 
@@ -1157,11 +1161,14 @@ static void ahci_migrate_simple(uint8_t cmd_read, 
uint8_t cmd_write)
 char *uri = g_strdup_printf("unix:%s", mig_socket);
 
 src = ahci_boot_and_enable("-m 1024 -M q35 "
-   "-drive if=ide,format=%s,file=%s ",
+   "-drive if=none,format=%s,file=%s,id=drive0 "
+   "-device ide-hd,drive=drive0,lock-mode=shared ",
imgfmt, tmp_path);
 dst = ahci_boot("-m 1024 -M q35 "
-"-drive if=ide,format=%s,file=%s "
-"-incoming %s", imgfmt, tmp_path, uri);
+"-drive if=none,format=%s,file=%s,id=drive0 "
+"-device ide-hd,drive=drive0,lock-mode=shared "
+"-incoming %s",
+imgfmt, tmp_path, uri);
 
 set_context(src->parent);
 
@@ -1286,7 +1293,7 @@ static void ahci_migrate_halted_io(uint8_t cmd_read, 
uint8_t cmd_write)
"format=%s,cache=writeback,"
"rerror=stop,werror=stop "
"-M q35 "
-   "-device ide-hd,drive=drive0 ",
+   "-device ide-hd,drive=drive0,lock-mode=shared ",
debug_path,
tmp_path, imgfmt);
 
@@ -1294,7 +1301,7 @@ static void ahci_migrate_halted_io(uint8_t cmd_read, 
uint8_t cmd_write)
 "format=%s,cache=writeback,"
 "rerror=stop,werror=stop "
 "-M q35 "
-"-device ide-hd,drive=drive0 "
+"-device ide-hd,drive=drive0,lock-mode=shared "
 "-incoming %s",
 tmp_path, imgfmt, uri);
 
@@ -1358,13 +1365,13 @@ static void test_flush_migrate(void)
"cache=writeback,rerror=stop,werror=stop,"
"format=%s "
"-M q35 "
-   "-device ide-hd,drive=drive0 ",
+   "-device ide-hd,drive=drive0,lock-mode=shared ",
debug_path, tmp_path, imgfmt);
 dst = ahci_boot("-drive file=%s,if=none,id=drive0,"
 "cache=writeback,rerror=stop,werror=stop,"
 "format=%s "
 "-M q35 "
-"-device ide-hd,drive=drive0 "
+"-device ide-hd,drive=drive0,lock-mode=shared "
 "-incoming %s", tmp_path, imgfmt, uri);
 
 set_context(src->parent);
-- 
2.7.4

Re: [Qemu-block] [Qemu-devel] [PATCH] qcow2: Support BDRV_REQ_MAY_UNMAP

[Kevin Wolf]

Am 30.09.2016 um 04:04 hat Fam Zheng geschrieben:
> On Thu, 09/29 12:39, Kevin Wolf wrote:
> > Am 29.09.2016 um 11:55 hat Fam Zheng geschrieben:
> > > On Thu, 09/29 11:29, Kevin Wolf wrote:
> > > > Am 28.09.2016 um 09:04 hat Fam Zheng geschrieben:
> > > > > Handling this is similar to what is done to the L2 entry in the case 
> > > > > of
> > > > > compressed clusters.
> > > > > 
> > > > > Signed-off-by: Fam Zheng 
> > > > > ---
> > > > >  block/qcow2-cluster.c | 9 +
> > > > >  block/qcow2.c | 3 ++-
> > > > >  block/qcow2.h | 3 ++-
> > > > >  3 files changed, 9 insertions(+), 6 deletions(-)
> > > > > 
> > > > > diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
> > > > > index 61d1ffd..928c1e2 100644
> > > > > --- a/block/qcow2-cluster.c
> > > > > +++ b/block/qcow2-cluster.c
> > > > > @@ -1558,7 +1558,7 @@ fail:
> > > > >   * clusters.
> > > > >   */
> > > > >  static int zero_single_l2(BlockDriverState *bs, uint64_t offset,
> > > > > -  uint64_t nb_clusters)
> > > > > +  uint64_t nb_clusters, int flags)
> > > > >  {
> > > > >  BDRVQcow2State *s = bs->opaque;
> > > > >  uint64_t *l2_table;
> > > > > @@ -1582,7 +1582,7 @@ static int zero_single_l2(BlockDriverState *bs, 
> > > > > uint64_t offset,
> > > > >  
> > > > >  /* Update L2 entries */
> > > > >  qcow2_cache_entry_mark_dirty(bs, s->l2_table_cache, 
> > > > > l2_table);
> > > > > -if (old_offset & QCOW_OFLAG_COMPRESSED) {
> > > > > +if (old_offset & QCOW_OFLAG_COMPRESSED || flags & 
> > > > > BDRV_REQ_MAY_UNMAP) {
> > > > >  l2_table[l2_index + i] = cpu_to_be64(QCOW_OFLAG_ZERO);
> > > > >  qcow2_free_any_clusters(bs, old_offset, 1, 
> > > > > QCOW2_DISCARD_REQUEST);
> > > > >  } else {
> > > > 
> > > > I don't think we should always do this for BDRV_REQ_MAY_UNMAP. The user
> > > > may want to keep the image fully allocated even if the guest (or block
> > > > job etc.) thinks this can be discarded.
> > > > 
> > > > When we discussed this in another email thread (?), I think I suggested
> > > > checking whether the image was opened with pass-discard-request=on. If
> > > > so, go ahead and deallocate the cluster, otherwise keep it allocated.
> > > > 
> > > > In those cases where pass-discard-request=off, it doesn't make a lot of
> > > > sense to deallocate the cluster anyway because it won't shrink the file
> > > > size.
> > > 
> > > I think this patch does what you mean:
> > > 
> > > $ strace -f -e fallocate qemu-io \
> > > -c 'open -o pass-discard-request=on /var/tmp/test' \
> > > -c 'write 0 1M' \
> > > -c 'write -u -z 0 1M' \
> > > 2>&1 >/dev/null | \
> > > grep fallocate
> > > [pid 17548] fallocate(17, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 
> > > 327680, 1048576) = 0
> > > $ strace -f -e fallocate qemu-io \
> > > -c 'open -o pass-discard-request=off /var/tmp/test' \
> > > -c 'write 0 1M' \
> > > -c 'write -u -z 0 1M' \
> > > 2>&1 >/dev/null | \
> > > grep fallocate
> > > 
> > > Because there is another check of pass-discard-request value in
> > > update_refcount:
> > > 
> > > if (refcount == 0 && s->discard_passthrough[type]) {
> > > update_refcount_discard(bs, cluster_offset, s->cluster_size);
> > > }
> > 
> > What I mean is that in the second case, you're still uselessly
> > deallocating the cluster on the qcow2 level while you can't reclaim it
> > on the filesystem level. So it would be better to leave it allocated in
> > qcow2, too, so that you don't get an expensive reallocation the next
> > time you write to it.
> > 
> 
> Like this?

That's what I had in mind, but actually I think Paolo is right and the
BDRV_O_UNMAP check in bdrv_co_pwrite_zeroes() already does all that is
needed. So the approach of the original patch should be okay.

Kevin

> ---
> 
> diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
> index 928c1e2..3d3cea4 100644
> --- a/block/qcow2-cluster.c
> +++ b/block/qcow2-cluster.c
> @@ -1582,7 +1582,9 @@ static int zero_single_l2(BlockDriverState *bs, 
> uint64_t offset,
> 
>  /* Update L2 entries */
>  qcow2_cache_entry_mark_dirty(bs, s->l2_table_cache, l2_table);
> -if (old_offset & QCOW_OFLAG_COMPRESSED || flags & 
> BDRV_REQ_MAY_UNMAP) {
> +if (old_offset & QCOW_OFLAG_COMPRESSED ||
> +((flags & BDRV_REQ_MAY_UNMAP) &&
> + s->discard_passthrough[QCOW2_DISCARD_REQUEST])) {
>  l2_table[l2_index + i] = cpu_to_be64(QCOW_OFLAG_ZERO);
>  qcow2_free_any_clusters(bs, old_offset, 1, 
> QCOW2_DISCARD_REQUEST);
>  } else {

[Qemu-block] [PATCH v8 23/36] iotests: 087: Disable image locking in cases where file is shared

[Fam Zheng]

Otherwise the error handling we are expecting will be masked by the
preceding image locking check, and is going to be indistinguishable
because the error messages are all the same.

Signed-off-by: Fam Zheng 
Reviewed-by: Max Reitz 
---
 tests/qemu-iotests/087 | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tests/qemu-iotests/087 b/tests/qemu-iotests/087
index 5c04577..91e4c71 100755
--- a/tests/qemu-iotests/087
+++ b/tests/qemu-iotests/087
@@ -84,6 +84,7 @@ run_qemu -drive 
driver=$IMGFMT,id=disk,node-name=test-node,file="$TEST_IMG" <

[Qemu-block] [PATCH v8 25/36] iotests: 130: Check image info locklessly

[Fam Zheng]

By the time _img_info is run, QEMU process's resources may still be on
its way being cleaned up, asynchronously, even though the process itself
is already gone after the "kill -KILL" and "wait" commands in
_cleanup_qemu.

Change the last HMP command to 'q' to ensure the locks are released.

Signed-off-by: Fam Zheng 
---
 tests/qemu-iotests/130 | 4 ++--
 tests/qemu-iotests/130.out | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/tests/qemu-iotests/130 b/tests/qemu-iotests/130
index ecc8a5b..f14a04f 100755
--- a/tests/qemu-iotests/130
+++ b/tests/qemu-iotests/130
@@ -60,7 +60,7 @@ echo
 # Test that a backing file isn't written
 _launch_qemu -drive 
id=testdisk,file="$TEST_IMG",backing.file.filename="$TEST_IMG.base"
 _send_qemu_cmd $QEMU_HANDLE "commit testdisk" "(qemu)"
-_send_qemu_cmd $QEMU_HANDLE '' '(qemu)'
+_send_qemu_cmd $QEMU_HANDLE 'q' '(qemu)'
 _cleanup_qemu
 _img_info | _filter_img_info
 
@@ -69,7 +69,7 @@ _img_info | _filter_img_info
 _make_test_img -F raw -b "$TEST_IMG.orig" 64M
 _launch_qemu -drive 
id=testdisk,file="$TEST_IMG",backing.file.filename="$TEST_IMG.base",backing.driver=$IMGFMT
 _send_qemu_cmd $QEMU_HANDLE "commit testdisk" "(qemu)"
-_send_qemu_cmd $QEMU_HANDLE '' '(qemu)'
+_send_qemu_cmd $QEMU_HANDLE 'q' '(qemu)'
 _cleanup_qemu
 _img_info | _filter_img_info
 
diff --git a/tests/qemu-iotests/130.out b/tests/qemu-iotests/130.out
index ae95b50..2422329 100644
--- a/tests/qemu-iotests/130.out
+++ b/tests/qemu-iotests/130.out
@@ -10,14 +10,14 @@ virtual size: 64M (67108864 bytes)
 
 QEMU X.Y.Z monitor - type 'help' for more information
 (qemu) 
ccocomcommcommicommitcommit
 commit tcommit 
tecommit 
tescommit 
testcommit 
testdcommit 
testdicommit 
testdiscommit testdisk
-(qemu) 
+(qemu) q
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 64M (67108864 bytes)
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 
backing_file=TEST_DIR/t.IMGFMT.orig backing_fmt=raw
 QEMU X.Y.Z monitor - type 'help' for more information
 (qemu) 
ccocomcommcommicommitcommit
 commit tcommit 
tecommit 
tescommit 
testcommit 
testdcommit 
testdicommit 
testdiscommit testdisk
-(qemu) 
+(qemu) q
 image: TEST_DIR/t.IMGFMT
 file format: IMGFMT
 virtual size: 64M (67108864 bytes)
-- 
2.7.4

[Qemu-block] [PATCH v8 31/36] fdc: Add lock-mode qdev properties

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 hw/block/fdc.c | 19 ++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/hw/block/fdc.c b/hw/block/fdc.c
index b79873a..20684b8 100644
--- a/hw/block/fdc.c
+++ b/hw/block/fdc.c
@@ -158,6 +158,7 @@ typedef enum FDiskFlags {
 typedef struct FDrive {
 FDCtrl *fdctrl;
 BlockBackend *blk;
+ImageLockMode lock_mode;
 /* Drive status */
 FloppyDriveType drive;/* CMOS drive type*/
 uint8_t perpendicular;/* 2.88 MB access mode*/
@@ -2446,11 +2447,23 @@ static void fdctrl_realize_common(FDCtrl *fdctrl, Error 
**errp)
 {
 int i, j;
 static int command_tables_inited = 0;
+Error *local_err = NULL;
 
 if (fdctrl->fallback == FLOPPY_DRIVE_TYPE_AUTO) {
 error_setg(errp, "Cannot choose a fallback FDrive type of 'auto'");
 }
 
+for (i = 0; i < 2; ++i) {
+if (fdctrl->drives[i].blk) {
+blk_lock_image(fdctrl->drives[i].blk, fdctrl->drives[i].lock_mode,
+   &local_err);
+if (local_err) {
+error_propagate(errp, local_err);
+return;
+}
+}
+}
+
 /* Fill 'command_to_handler' lookup table */
 if (!command_tables_inited) {
 command_tables_inited = 1;
@@ -2495,7 +2508,6 @@ static void isabus_fdc_realize(DeviceState *dev, Error 
**errp)
 FDCtrlISABus *isa = ISA_FDC(dev);
 FDCtrl *fdctrl = &isa->state;
 Error *err = NULL;
-
 isa_register_portio_list(isadev, &fdctrl->portio_list,
  isa->iobase, fdc_portio_list, fdctrl,
  "fdc");
@@ -2608,6 +2620,8 @@ static Property isa_fdc_properties[] = {
 DEFINE_PROP_UINT32("dma", FDCtrlISABus, dma, 2),
 DEFINE_PROP_DRIVE("driveA", FDCtrlISABus, state.drives[0].blk),
 DEFINE_PROP_DRIVE("driveB", FDCtrlISABus, state.drives[1].blk),
+DEFINE_PROP_LOCK_MODE("lock-modeA", FDCtrlISABus, 
state.drives[0].lock_mode),
+DEFINE_PROP_LOCK_MODE("lock-modeB", FDCtrlISABus, 
state.drives[1].lock_mode),
 DEFINE_PROP_BIT("check_media_rate", FDCtrlISABus, state.check_media_rate,
 0, true),
 DEFINE_PROP_DEFAULT("fdtypeA", FDCtrlISABus, state.drives[0].drive,
@@ -2667,6 +2681,8 @@ static const VMStateDescription vmstate_sysbus_fdc ={
 static Property sysbus_fdc_properties[] = {
 DEFINE_PROP_DRIVE("driveA", FDCtrlSysBus, state.drives[0].blk),
 DEFINE_PROP_DRIVE("driveB", FDCtrlSysBus, state.drives[1].blk),
+DEFINE_PROP_LOCK_MODE("lock-modeA", FDCtrlISABus, 
state.drives[0].lock_mode),
+DEFINE_PROP_LOCK_MODE("lock-modeB", FDCtrlISABus, 
state.drives[1].lock_mode),
 DEFINE_PROP_DEFAULT("fdtypeA", FDCtrlSysBus, state.drives[0].drive,
 FLOPPY_DRIVE_TYPE_AUTO, qdev_prop_fdc_drive_type,
 FloppyDriveType),
@@ -2696,6 +2712,7 @@ static const TypeInfo sysbus_fdc_info = {
 
 static Property sun4m_fdc_properties[] = {
 DEFINE_PROP_DRIVE("drive", FDCtrlSysBus, state.drives[0].blk),
+DEFINE_PROP_LOCK_MODE("lock-mode", FDCtrlISABus, 
state.drives[0].lock_mode),
 DEFINE_PROP_DEFAULT("fdtype", FDCtrlSysBus, state.drives[0].drive,
 FLOPPY_DRIVE_TYPE_AUTO, qdev_prop_fdc_drive_type,
 FloppyDriveType),
-- 
2.7.4

[Qemu-block] [PATCH v8 27/36] tests: Use null-co:// instead of /dev/null

[Fam Zheng]

With image locking, opening /dev/null can fail when multiple tests run
in parallel (make -j2, for example). Use null-co:// as the null protocol
doesn't do image locking.

While it's arguable we could special-case /dev/null, /dev/zero,
/dev/urandom etc in raw-posix driver, it is not really necessary because
user can always specify lock-mode=off when it is appropriate. So let's
write sensible testing code too.

Signed-off-by: Fam Zheng 
Reviewed-by: Max Reitz 
---
 tests/drive_del-test.c| 2 +-
 tests/nvme-test.c | 2 +-
 tests/usb-hcd-uhci-test.c | 2 +-
 tests/usb-hcd-xhci-test.c | 2 +-
 tests/virtio-blk-test.c   | 2 +-
 tests/virtio-scsi-test.c  | 4 ++--
 6 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/tests/drive_del-test.c b/tests/drive_del-test.c
index 121b9c9..2175139 100644
--- a/tests/drive_del-test.c
+++ b/tests/drive_del-test.c
@@ -92,7 +92,7 @@ static void test_after_failed_device_add(void)
 static void test_drive_del_device_del(void)
 {
 /* Start with a drive used by a device that unplugs instantaneously */
-qtest_start("-drive if=none,id=drive0,file=/dev/null,format=raw"
+qtest_start("-drive if=none,id=drive0,file=null-co://,format=raw"
 " -device virtio-scsi-pci"
 " -device scsi-hd,drive=drive0,id=dev0");
 
diff --git a/tests/nvme-test.c b/tests/nvme-test.c
index c8bece4..7674a44 100644
--- a/tests/nvme-test.c
+++ b/tests/nvme-test.c
@@ -22,7 +22,7 @@ int main(int argc, char **argv)
 g_test_init(&argc, &argv, NULL);
 qtest_add_func("/nvme/nop", nop);
 
-qtest_start("-drive id=drv0,if=none,file=/dev/null,format=raw "
+qtest_start("-drive id=drv0,if=none,file=null-co://,format=raw "
 "-device nvme,drive=drv0,serial=foo");
 ret = g_test_run();
 
diff --git a/tests/usb-hcd-uhci-test.c b/tests/usb-hcd-uhci-test.c
index 5cd59ad..3684503 100644
--- a/tests/usb-hcd-uhci-test.c
+++ b/tests/usb-hcd-uhci-test.c
@@ -85,7 +85,7 @@ int main(int argc, char **argv)
 qtest_add_func("/uhci/pci/hotplug/usb-storage", test_usb_storage_hotplug);
 
 qtest_start("-device piix3-usb-uhci,id=uhci,addr=1d.0"
-" -drive id=drive0,if=none,file=/dev/null,format=raw"
+" -drive id=drive0,if=none,file=null-co://,format=raw"
 " -device usb-tablet,bus=uhci.0,port=1");
 ret = g_test_run();
 qtest_end();
diff --git a/tests/usb-hcd-xhci-test.c b/tests/usb-hcd-xhci-test.c
index 22513e9..031764d 100644
--- a/tests/usb-hcd-xhci-test.c
+++ b/tests/usb-hcd-xhci-test.c
@@ -89,7 +89,7 @@ int main(int argc, char **argv)
 qtest_add_func("/xhci/pci/hotplug/usb-uas", test_usb_uas_hotplug);
 
 qtest_start("-device nec-usb-xhci,id=xhci"
-" -drive id=drive0,if=none,file=/dev/null,format=raw");
+" -drive id=drive0,if=none,file=null-co://,format=raw");
 ret = g_test_run();
 qtest_end();
 
diff --git a/tests/virtio-blk-test.c b/tests/virtio-blk-test.c
index 811cf75..5dba567 100644
--- a/tests/virtio-blk-test.c
+++ b/tests/virtio-blk-test.c
@@ -66,7 +66,7 @@ static QPCIBus *pci_test_start(void)
 tmp_path = drive_create();
 
 cmdline = g_strdup_printf("-drive if=none,id=drive0,file=%s,format=raw "
-"-drive if=none,id=drive1,file=/dev/null,format=raw "
+"-drive if=none,id=drive1,file=null-co://,format=raw "
 "-device virtio-blk-pci,id=drv0,drive=drive0,"
 "addr=%x.%x",
 tmp_path, PCI_SLOT, PCI_FN);
diff --git a/tests/virtio-scsi-test.c b/tests/virtio-scsi-test.c
index f1489e6..3c2f5df 100644
--- a/tests/virtio-scsi-test.c
+++ b/tests/virtio-scsi-test.c
@@ -40,7 +40,7 @@ static void qvirtio_scsi_start(const char *extra_opts)
 char *cmdline;
 
 cmdline = g_strdup_printf(
-"-drive id=drv0,if=none,file=/dev/null,format=raw "
+"-drive id=drv0,if=none,file=null-co://,format=raw "
 "-device virtio-scsi-pci,id=vs0 "
 "-device scsi-hd,bus=vs0.0,drive=drv0 %s",
 extra_opts ? : "");
@@ -192,7 +192,7 @@ static void hotplug(void)
 {
 QDict *response;
 
-qvirtio_scsi_start("-drive id=drv1,if=none,file=/dev/null,format=raw");
+qvirtio_scsi_start("-drive id=drv1,if=none,file=null-co://,format=raw");
 response = qmp("{\"execute\": \"device_add\","
" \"arguments\": {"
"   \"driver\": \"scsi-hd\","
-- 
2.7.4

[Qemu-block] [PATCH v8 32/36] m25p80: Add 'lock-mode' property

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 hw/block/m25p80.c | 8 
 1 file changed, 8 insertions(+)

diff --git a/hw/block/m25p80.c b/hw/block/m25p80.c
index d29ff4c..3c1765a 100644
--- a/hw/block/m25p80.c
+++ b/hw/block/m25p80.c
@@ -380,6 +380,7 @@ typedef struct Flash {
 SSISlave parent_obj;
 
 BlockBackend *blk;
+ImageLockMode lock_mode;
 
 uint8_t *storage;
 uint32_t size;
@@ -1144,6 +1145,7 @@ static void m25p80_realize(SSISlave *ss, Error **errp)
 {
 Flash *s = M25P80(ss);
 M25P80Class *mc = M25P80_GET_CLASS(s);
+Error *local_err = NULL;
 
 s->pi = mc->pi;
 
@@ -1151,6 +1153,11 @@ static void m25p80_realize(SSISlave *ss, Error **errp)
 s->dirty_page = -1;
 
 if (s->blk) {
+blk_lock_image(s->blk, s->lock_mode, &local_err);
+if (local_err) {
+error_propagate(errp, local_err);
+return;
+}
 DB_PRINT_L(0, "Binding to IF_MTD drive\n");
 s->storage = blk_blockalign(s->blk, s->size);
 
@@ -1185,6 +1192,7 @@ static Property m25p80_properties[] = {
 DEFINE_PROP_UINT8("spansion-cr3nv", Flash, spansion_cr3nv, 0x2),
 DEFINE_PROP_UINT8("spansion-cr4nv", Flash, spansion_cr4nv, 0x10),
 DEFINE_PROP_DRIVE("drive", Flash, blk),
+DEFINE_PROP_LOCK_MODE("lock-mode", Flash, lock_mode),
 DEFINE_PROP_END_OF_LIST(),
 };
 
-- 
2.7.4

[Qemu-block] [PATCH v8 34/36] onenand: Add 'lock-mode' property

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 hw/block/onenand.c | 7 +++
 1 file changed, 7 insertions(+)

diff --git a/hw/block/onenand.c b/hw/block/onenand.c
index 8d84227..9b058e8 100644
--- a/hw/block/onenand.c
+++ b/hw/block/onenand.c
@@ -778,6 +778,7 @@ static int onenand_initfn(SysBusDevice *sbd)
 OneNANDState *s = ONE_NAND(dev);
 uint32_t size = 1 << (24 + ((s->id.dev >> 4) & 7));
 void *ram;
+Error *local_err = NULL;
 
 s->base = (hwaddr)-1;
 s->rdy = NULL;
@@ -796,6 +797,11 @@ static int onenand_initfn(SysBusDevice *sbd)
 error_report("Can't use a read-only drive");
 return -1;
 }
+blk_lock_image(s->blk, s->lock_mode, &local_err);
+if (local_err) {
+error_report_err(local_err);
+return -1;
+}
 s->blk_cur = s->blk;
 }
 s->otp = memset(g_malloc((64 + 2) << PAGE_SHIFT),
@@ -828,6 +834,7 @@ static Property onenand_properties[] = {
 DEFINE_PROP_UINT16("version_id", OneNANDState, id.ver, 0),
 DEFINE_PROP_INT32("shift", OneNANDState, shift, 0),
 DEFINE_PROP_DRIVE("drive", OneNANDState, blk),
+DEFINE_PROP_LOCK_MODE("lock-mode", OneNANDState, lock_mode),
 DEFINE_PROP_END_OF_LIST(),
 };
 
-- 
2.7.4

[Qemu-block] [PATCH v8 36/36] sd: Add 'lock-mode' property

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 hw/sd/sd.c | 8 
 1 file changed, 8 insertions(+)

diff --git a/hw/sd/sd.c b/hw/sd/sd.c
index 8e88e83..6c18fb8 100644
--- a/hw/sd/sd.c
+++ b/hw/sd/sd.c
@@ -124,6 +124,7 @@ struct SDState {
 qemu_irq readonly_cb;
 qemu_irq inserted_cb;
 BlockBackend *blk;
+ImageLockMode lock_mode;
 
 bool enable;
 };
@@ -1886,6 +1887,7 @@ static void sd_instance_finalize(Object *obj)
 
 static void sd_realize(DeviceState *dev, Error **errp)
 {
+Error *local_err = NULL;
 SDState *sd = SD_CARD(dev);
 
 if (sd->blk && blk_is_read_only(sd->blk)) {
@@ -1894,12 +1896,18 @@ static void sd_realize(DeviceState *dev, Error **errp)
 }
 
 if (sd->blk) {
+blk_lock_image(sd->blk, sd->lock_mode, &local_err);
+if (local_err) {
+error_propagate(errp, local_err);
+return;
+}
 blk_set_dev_ops(sd->blk, &sd_block_ops, sd);
 }
 }
 
 static Property sd_properties[] = {
 DEFINE_PROP_DRIVE("drive", SDState, blk),
+DEFINE_PROP_LOCK_MODE("lock-mode", SDState, lock_mode),
 /* We do not model the chip select pin, so allow the board to select
  * whether card should be in SSI or MMC/SD mode.  It is also up to the
  * board to ensure that ssi transfers only occur when the chip select
-- 
2.7.4

[Qemu-block] [PATCH v8 35/36] spapr_nvram: Add 'lock-mode' property

[Fam Zheng]

Signed-off-by: Fam Zheng 
---
 hw/nvram/spapr_nvram.c | 8 
 1 file changed, 8 insertions(+)

diff --git a/hw/nvram/spapr_nvram.c b/hw/nvram/spapr_nvram.c
index 4de5f70..b679e0b 100644
--- a/hw/nvram/spapr_nvram.c
+++ b/hw/nvram/spapr_nvram.c
@@ -39,6 +39,7 @@ typedef struct sPAPRNVRAM {
 uint32_t size;
 uint8_t *buf;
 BlockBackend *blk;
+ImageLockMode lock_mode;
 VMChangeStateEntry *vmstate;
 } sPAPRNVRAM;
 
@@ -140,8 +141,14 @@ static void rtas_nvram_store(PowerPCCPU *cpu, 
sPAPRMachineState *spapr,
 static void spapr_nvram_realize(VIOsPAPRDevice *dev, Error **errp)
 {
 sPAPRNVRAM *nvram = VIO_SPAPR_NVRAM(dev);
+Error *local_err = NULL;
 
 if (nvram->blk) {
+blk_lock_image(nvram->blk, nvram->lock_mode, &local_err);
+if (local_err) {
+error_propagate(errp, local_err);
+return;
+}
 nvram->size = blk_getlength(nvram->blk);
 } else {
 nvram->size = DEFAULT_NVRAM_SIZE;
@@ -226,6 +233,7 @@ static const VMStateDescription vmstate_spapr_nvram = {
 static Property spapr_nvram_properties[] = {
 DEFINE_SPAPR_PROPERTIES(sPAPRNVRAM, sdev),
 DEFINE_PROP_DRIVE("drive", sPAPRNVRAM, blk),
+DEFINE_PROP_LOCK_MODE("lock-mode", sPAPRNVRAM, lock_mode),
 DEFINE_PROP_END_OF_LIST(),
 };
 
-- 
2.7.4

Re: [Qemu-block] [PATCH v3] block: Turn on "unmap" in active commit

[Stefan Hajnoczi]

On Fri, Sep 30, 2016 at 10:12:05AM +0800, Fam Zheng wrote:
> On Thu, 09/29 13:41, Stefan Hajnoczi wrote:
> > On Tue, Sep 27, 2016 at 07:14:52PM +0800, Fam Zheng wrote:
> > > We already specified BDRV_O_UNMAP when opening images in 'qemu-img
> > > commit', but didn't turn on the "unmap" in the active commit job. This
> > > patch fixes that so that zeroed clusters in top image can be discarded
> > > which is desired in the virt-sparsify use case, where a temporary
> > > overlay is created and fstrim'ed before commiting back, to free space in
> > > the original image.
> > > 
> > > This also enables it for block-commit.
> > > 
> > > Signed-off-by: Fam Zheng 
> > > ---
> > > v3: Change the right parameter.
> > > v2: Add "unmap" to block-commit as well. [Kevin]
> > > ---
> > >  block/mirror.c | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/block/mirror.c b/block/mirror.c
> > > index f9d1fec..8847ec5 100644
> > > --- a/block/mirror.c
> > > +++ b/block/mirror.c
> > > @@ -1042,7 +1042,7 @@ void commit_active_start(const char *job_id, 
> > > BlockDriverState *bs,
> > >  
> > >  mirror_start_job(job_id, bs, base, NULL, speed, 0, 0,
> > >   MIRROR_LEAVE_BACKING_CHAIN,
> > > - on_error, on_error, false, cb, opaque, &local_err,
> > > + on_error, on_error, true, cb, opaque, &local_err,
> > >   &commit_active_job_driver, false, base, 
> > > auto_complete);
> > >  if (local_err) {
> > >  error_propagate(errp, local_err);
> > 
> > Why is unmap an option at all?
> > 
> > What's wrong with using BDRV_REQ_MAY_UNMAP on all
> > blk_aio_pwrite_zeroes() calls?
> 
> Because unmap is an QMP option of drive-backup. I think in the drive-mirror
> context, it mitigates the limitation that we have no control over target's
> BDRV_O_UNMAP (always inherited from source).

That doesn't explain why unmap is an option.

Is it because the user might not want to unmap the target and would
rather writes zeroes?

For example in qcow2 they might want to leave the data clusters
allocated and simply mark their contents zeroed:

"Standard Cluster Descriptor:

Bit   0:If set to 1, the cluster reads as all zeros. The host
cluster offset can be used to describe a preallocation,
but it won't be used for reading data from this cluster,
nor is data read from the backing file if the cluster is
unallocated."


signature.asc
Description: PGP signature

Re: [Qemu-block] [PATCH v3] block: Turn on "unmap" in active commit

[Kevin Wolf]

Am 30.09.2016 um 04:12 hat Fam Zheng geschrieben:
> On Thu, 09/29 13:41, Stefan Hajnoczi wrote:
> > On Tue, Sep 27, 2016 at 07:14:52PM +0800, Fam Zheng wrote:
> > > We already specified BDRV_O_UNMAP when opening images in 'qemu-img
> > > commit', but didn't turn on the "unmap" in the active commit job. This
> > > patch fixes that so that zeroed clusters in top image can be discarded
> > > which is desired in the virt-sparsify use case, where a temporary
> > > overlay is created and fstrim'ed before commiting back, to free space in
> > > the original image.
> > > 
> > > This also enables it for block-commit.
> > > 
> > > Signed-off-by: Fam Zheng 
> > > ---
> > > v3: Change the right parameter.
> > > v2: Add "unmap" to block-commit as well. [Kevin]
> > > ---
> > >  block/mirror.c | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/block/mirror.c b/block/mirror.c
> > > index f9d1fec..8847ec5 100644
> > > --- a/block/mirror.c
> > > +++ b/block/mirror.c
> > > @@ -1042,7 +1042,7 @@ void commit_active_start(const char *job_id, 
> > > BlockDriverState *bs,
> > >  
> > >  mirror_start_job(job_id, bs, base, NULL, speed, 0, 0,
> > >   MIRROR_LEAVE_BACKING_CHAIN,
> > > - on_error, on_error, false, cb, opaque, &local_err,
> > > + on_error, on_error, true, cb, opaque, &local_err,
> > >   &commit_active_job_driver, false, base, 
> > > auto_complete);
> > >  if (local_err) {
> > >  error_propagate(errp, local_err);
> > 
> > Why is unmap an option at all?
> > 
> > What's wrong with using BDRV_REQ_MAY_UNMAP on all
> > blk_aio_pwrite_zeroes() calls?
> 
> Because unmap is an QMP option of drive-backup. I think in the drive-mirror
> context, it mitigates the limitation that we have no control over target's
> BDRV_O_UNMAP (always inherited from source).

Wouldn't the more straightforward implementation then be if
qmp_drive_mirror() set BDRV_O_UNMAP for the target depending on the flag
rather than passing the flag down to the mirror job?

Hm... And should BDRV_O_UNMAP really be a BlockBackend option rather
than a BDS one? We already enable it unconditionally on non-root nodes
and it seems to make sense to me to allow discard e.g. from a block job,
but not from the guest.

Kevin

[Qemu-block] [PATCH v14 00/21] QAPI/QOM work for non-scalar object properties

[Daniel P. Berrange]

An update of a series previously posted

 v1: https://lists.gnu.org/archive/html/qemu-devel/2016-02/msg04618.html
 v2: https://lists.gnu.org/archive/html/qemu-devel/2016-03/msg01454.html
 v3: https://lists.gnu.org/archive/html/qemu-devel/2016-03/msg02498.html
 v4: https://lists.gnu.org/archive/html/qemu-devel/2016-05/msg01661.html
 v5: https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg00485.html
 v6: https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg03876.html
 v7: https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg00919.html
 v8: https://lists.gnu.org/archive/html/qemu-devel/2016-07/msg03115.html
 v9: https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg02653.html
 v10: https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg02694.html
 v11: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg00652.html
 v12: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg03559.html
 v13: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg04212.html
 v14: https://lists.gnu.org/archive/html/qemu-devel/2016-09/msg07037.html

This series provides the infrastructure to allow use of non-scalar
properties with the -object CLI arg, and object_add monitor commands.
eg a property which is a list of structs. The syntax used for this is
intentionally compatible with the syntax used by the block layer. This
will allow the qdict_crumple method to be used by the block layer to
convert from QemuOpts into structured QAPI block layer structs at a
future date. It is already used by one of Max's patch series, by
Kevin's -blockdev series, and by my own ACL series.

This patch series has grown a fair bit larger since  previous
postings (19^H^H21 patches instead of 6). This was to avoid trying
to squish too much into one patch, to make it easier to review.

It does not neccessarily need to be merged in one go. There are
three logical blocks of patches here.

Patches 1-8 are the core support for visiting QemuOpts using the
QObjectInputVisitor class. Merging this would unblock at least
Kevin's blockdev series, and probably Max's series too.

The patches 9-14 add enhancements to the code to support various
special cases / edge cases needed to provide fully semantic
replacement of the OptsVisitor

Patches 15-21 deal with converting code over to use the new visitor
logic.

Once this is merged, there is also scope to further enhance the
QObjectInputVisitor to deal with visiting single-properties,
which would let us delete StringInputVisitor too. This would
mean we have a single input visitor class, instead of the
current three.

Changed in v15:

 - Re-arranged patch series to allow easier incremental merge
   of core functionality
 - Extra error checking of some scenarios to be avoided (Eric)
 - Misc typos (Eric)
 - Rename QEMU_OPTS_REPEAT_POLICY_LIST to
   QEMU_OPTS_REPEAT_POLICY_ALL, and add QEMU_OPTS_REPEAT_POLICY_ERROR
   (Eric)

Changed in v14:

 - Extend qemu_opts_to_qdict to optionally handle case of
   repeated keys without dropping them.
 - Support parsing of integer ranges for "list of int" visits
   for back compat with OptsVisitor
 - Support visiting nested structs which are represented as
   a flat keyspace for back compat with OptsVisitor
 - Add trace events for visitor functions to facilitate
   debugging
 - Add a QObjectInputVisitor constructor which takes a
   QemuOpts object, to avoid repeated code patterns in callers
 - Greater test coverage of -object to validate correct handling
   of "list of ints" legacy OptsVisitor syntax
 - Convert all other use of OptsVisitor to QObjectInputVisitor
 - Delete OptsVisitor

Changed in v13:

 - Fix typos (Kevin)
 - Remove unneeded line breaks (Kevin)

Changed in v12:

 - More user friendly error message for mixing dict/list
   keys (Kevin)
 - Report error instead of assert for non-contiguous list
   keys (Kevin)
 - Fix tests for non-contiguous list keys (Kevin)
 - Add tests for escaping of '.' when crumpling (Kevin)
 - Fix remaining references to Qmp(In|Out)putVisitor (Markus)
 - Misc typos / whitespace fixes (Eric, Kevin)
 - Avoid touching 'ret' when parsing int64 fails (Eric)
 - Testing of more edge cases in QObjectInputVisitor (Eric)
 - Simplify API doc format (Markus)
 - Use parse_option_size instead of qemu_strtosz_suffix
   for consistency (Kevin)
 - Use safer qobject_to_qdict casts (Eric)

Changed in v11:

 - Split QAPI/QOM patches off from the access control patches

Changed in v10:

 - Fixed stupid build mistake

Changed in v9:

 - Rename QmpInputVisitor -> QObjectInputVisitor (Markus/Eric)
 - Rename QmpOutputVisitor -> QObjectOutputVisitor (Markus/Eric)
 - Drop "strict" param from qobject_string_visitor_new() (Marus)
 - Misc docs typos
 - Add a visitor able to use strict or string types (for Eric's
   netdev series)
 - Add a authorization API implementation that uses PAM

Changed in v8:

 - Rebase due to merge of Visitor API changes (Eric)

Changed in v7:

 - Misc typos in API docs (Marc-André)
 - Fix parsing of properties using type_size 

[Qemu-block] [PATCH v14 01/21] option: make parse_option_bool/number non-static

[Daniel P. Berrange]

The opts-visitor.c opts_type_bool() method has code for
parsing a string to set a bool value, as does the
qemu-option.c parse_option_bool() method, except it
handles fewer cases.

To enable consistency across the codebase, extend
parse_option_bool() to handle "yes", "no", "y" and
"n", and make it non-static. Convert the opts
visitor to call this method directly.

Also make parse_option_number() non-static to allow
for similar reuse later.

Reviewed-by: Kevin Wolf 
Reviewed-by: Marc-André Lureau 
Reviewed-by: Eric Blake 
Reviewed-by: Markus Armbruster 
Signed-off-by: Daniel P. Berrange 
---
 include/qemu/option.h |  4 
 qapi/opts-visitor.c   | 19 +--
 tests/qemu-iotests/051.out|  6 +++---
 tests/qemu-iotests/051.pc.out |  6 +++---
 tests/qemu-iotests/137.out|  4 ++--
 util/qemu-option.c| 27 ---
 6 files changed, 29 insertions(+), 37 deletions(-)

diff --git a/include/qemu/option.h b/include/qemu/option.h
index 1f9e3f9..2a5266f 100644
--- a/include/qemu/option.h
+++ b/include/qemu/option.h
@@ -37,6 +37,10 @@ int get_param_value(char *buf, int buf_size,
 const char *tag, const char *str);
 
 
+void parse_option_bool(const char *name, const char *value, bool *ret,
+   Error **errp);
+void parse_option_number(const char *name, const char *value,
+ uint64_t *ret, Error **errp);
 void parse_option_size(const char *name, const char *value,
uint64_t *ret, Error **errp);
 bool has_help_option(const char *param);
diff --git a/qapi/opts-visitor.c b/qapi/opts-visitor.c
index 1048bbc..084f7cc 100644
--- a/qapi/opts-visitor.c
+++ b/qapi/opts-visitor.c
@@ -334,7 +334,6 @@ opts_type_str(Visitor *v, const char *name, char **obj, 
Error **errp)
 }
 
 
-/* mimics qemu-option.c::parse_option_bool() */
 static void
 opts_type_bool(Visitor *v, const char *name, bool *obj, Error **errp)
 {
@@ -346,23 +345,7 @@ opts_type_bool(Visitor *v, const char *name, bool *obj, 
Error **errp)
 return;
 }
 
-if (opt->str) {
-if (strcmp(opt->str, "on") == 0 ||
-strcmp(opt->str, "yes") == 0 ||
-strcmp(opt->str, "y") == 0) {
-*obj = true;
-} else if (strcmp(opt->str, "off") == 0 ||
-strcmp(opt->str, "no") == 0 ||
-strcmp(opt->str, "n") == 0) {
-*obj = false;
-} else {
-error_setg(errp, QERR_INVALID_PARAMETER_VALUE, opt->name,
-   "on|yes|y|off|no|n");
-return;
-}
-} else {
-*obj = true;
-}
+parse_option_bool(opt->name, opt->str, obj, errp);
 
 processed(ov, name);
 }
diff --git a/tests/qemu-iotests/051.out b/tests/qemu-iotests/051.out
index 408d613..ffdc0b9 100644
--- a/tests/qemu-iotests/051.out
+++ b/tests/qemu-iotests/051.out
@@ -86,13 +86,13 @@ QEMU X.Y.Z monitor - type 'help' for more information
 (qemu) qququiquit
 
 Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=
-QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=: 
Parameter 'lazy-refcounts' expects 'on' or 'off'
+QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=: 
Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n'
 
 Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=42
-QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=42: 
Parameter 'lazy-refcounts' expects 'on' or 'off'
+QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=42: 
Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n'
 
 Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=foo
-QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=foo: 
Parameter 'lazy-refcounts' expects 'on' or 'off'
+QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=foo: 
Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n'
 
 
 === With version 2 images enabling lazy refcounts must fail ===
diff --git a/tests/qemu-iotests/051.pc.out b/tests/qemu-iotests/051.pc.out
index ec6d222..6abf1ab 100644
--- a/tests/qemu-iotests/051.pc.out
+++ b/tests/qemu-iotests/051.pc.out
@@ -86,13 +86,13 @@ QEMU X.Y.Z monitor - type 'help' for more information
 (qemu) qququiquit
 
 Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=
-QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=: 
Parameter 'lazy-refcounts' expects 'on' or 'off'
+QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=: 
Parameter 'lazy-refcounts' expects 'on', 'yes', 'y', 'off', 'no' or 'n'
 
 Testing: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=42
-QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcounts=42: 
Parameter 'lazy-refcounts' expects 'on' or 'off'
+QEMU_PROG: -drive file=TEST_DIR/t.qcow2,format=qcow2,lazy-refcount

[Qemu-block] [PATCH v14 04/21] qapi: rename QmpInputVisitor to QObjectInputVisitor

[Daniel P. Berrange]

The QmpInputVisitor has no direct dependency on QMP. It is
valid to use it anywhere that one has a QObject. Rename it
to better reflect its functionality as a generic QObject
to QAPI converter.

Reviewed-by: Kevin Wolf 
Reviewed-by: Eric Blake 
Signed-off-by: Daniel P. Berrange 
---
 docs/qapi-code-gen.txt |   2 +-
 ...qmp-input-visitor.h => qobject-input-visitor.h} |  10 +-
 include/qapi/visitor.h |   6 +-
 monitor.c  |   2 +-
 qapi/Makefile.objs |   2 +-
 ...qmp-input-visitor.c => qobject-input-visitor.c} | 171 +++--
 qmp.c  |   4 +-
 qom/qom-qobject.c  |   4 +-
 scripts/qapi-commands.py   |   4 +-
 target-s390x/cpu_models.c  |   4 +-
 tests/.gitignore   |   4 +-
 tests/Makefile.include |  12 +-
 tests/check-qnull.c|   4 +-
 tests/test-qmp-commands.c  |   4 +-
 ...-input-strict.c => test-qobject-input-strict.c} |   6 +-
 ...nput-visitor.c => test-qobject-input-visitor.c} |   6 +-
 tests/test-string-input-visitor.c  |   2 +-
 tests/test-visitor-serialization.c |   4 +-
 util/qemu-sockets.c|   2 +-
 19 files changed, 128 insertions(+), 125 deletions(-)
 rename include/qapi/{qmp-input-visitor.h => qobject-input-visitor.h} (63%)
 rename qapi/{qmp-input-visitor.c => qobject-input-visitor.c} (56%)
 rename tests/{test-qmp-input-strict.c => test-qobject-input-strict.c} (98%)
 rename tests/{test-qmp-input-visitor.c => test-qobject-input-visitor.c} (99%)

diff --git a/docs/qapi-code-gen.txt b/docs/qapi-code-gen.txt
index 5d4c2cd..d2604b6 100644
--- a/docs/qapi-code-gen.txt
+++ b/docs/qapi-code-gen.txt
@@ -1024,7 +1024,7 @@ Example:
 Visitor *v;
 UserDefOneList *arg1 = NULL;
 
-v = qmp_input_visitor_new(QOBJECT(args), true);
+v = qobject_input_visitor_new(QOBJECT(args), true);
 visit_start_struct(v, NULL, NULL, 0, &err);
 if (err) {
 goto out;
diff --git a/include/qapi/qmp-input-visitor.h 
b/include/qapi/qobject-input-visitor.h
similarity index 63%
rename from include/qapi/qmp-input-visitor.h
rename to include/qapi/qobject-input-visitor.h
index f3ff5f3..cde328d 100644
--- a/include/qapi/qmp-input-visitor.h
+++ b/include/qapi/qobject-input-visitor.h
@@ -11,20 +11,20 @@
  *
  */
 
-#ifndef QMP_INPUT_VISITOR_H
-#define QMP_INPUT_VISITOR_H
+#ifndef QOBJECT_INPUT_VISITOR_H
+#define QOBJECT_INPUT_VISITOR_H
 
 #include "qapi/visitor.h"
 #include "qapi/qmp/qobject.h"
 
-typedef struct QmpInputVisitor QmpInputVisitor;
+typedef struct QObjectInputVisitor QObjectInputVisitor;
 
 /*
- * Return a new input visitor that converts QMP to QAPI.
+ * Return a new input visitor that converts a QObject to a QAPI object.
  *
  * Set @strict to reject a parse that doesn't consume all keys of a
  * dictionary; otherwise excess input is ignored.
  */
-Visitor *qmp_input_visitor_new(QObject *obj, bool strict);
+Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
 
 #endif
diff --git a/include/qapi/visitor.h b/include/qapi/visitor.h
index 6c77a91..9bb6cba 100644
--- a/include/qapi/visitor.h
+++ b/include/qapi/visitor.h
@@ -25,14 +25,14 @@
  * for doing work at each node of a QAPI graph; it can also be used
  * for a virtual walk, where there is no actual QAPI C struct.
  *
- * There are four kinds of visitor classes: input visitors (QMP,
+ * There are four kinds of visitor classes: input visitors (QObject,
  * string, and QemuOpts) parse an external representation and build
- * the corresponding QAPI graph, output visitors (QMP and string) take
+ * the corresponding QAPI graph, output visitors (QObject and string) take
  * a completed QAPI graph and generate an external representation, the
  * dealloc visitor can take a QAPI graph (possibly partially
  * constructed) and recursively free its resources, and the clone
  * visitor performs a deep clone of one QAPI object to another.  While
- * the dealloc and QMP input/output visitors are general, the string,
+ * the dealloc and QObject input/output visitors are general, the string,
  * QemuOpts, and clone visitors have some implementation limitations;
  * see the documentation for each visitor for more details on what it
  * supports.  Also, see visitor-impl.h for the callback contracts
diff --git a/monitor.c b/monitor.c
index 83c4edf..14089cc 100644
--- a/monitor.c
+++ b/monitor.c
@@ -957,7 +957,7 @@ EventInfoList *qmp_query_events(Error **errp)
  * directly into QObject instead of first parsing it with
  * visit_type_SchemaInfoList() into a SchemaInfoList, then marshal it
  * to QObject with generated output marshallers, every time.  Instead,
- * we do it in test-qmp-input-visitor.c

[Qemu-block] [PATCH v14 03/21] qapi: add trace events for visitor

[Daniel P. Berrange]

Allow tracing of the operation of visitors

Signed-off-by: Daniel P. Berrange 
---
 Makefile.objs  |  1 +
 qapi/qapi-visit-core.c | 27 +++
 qapi/trace-events  | 33 +
 3 files changed, 61 insertions(+)
 create mode 100644 qapi/trace-events

diff --git a/Makefile.objs b/Makefile.objs
index a8e0224..b3e8aef 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -160,3 +160,4 @@ trace-events-y += target-s390x/trace-events
 trace-events-y += target-ppc/trace-events
 trace-events-y += qom/trace-events
 trace-events-y += linux-user/trace-events
+trace-events-y += qapi/trace-events
diff --git a/qapi/qapi-visit-core.c b/qapi/qapi-visit-core.c
index 55f5876..bfcb276 100644
--- a/qapi/qapi-visit-core.c
+++ b/qapi/qapi-visit-core.c
@@ -19,10 +19,12 @@
 #include "qapi/qmp/qerror.h"
 #include "qapi/visitor.h"
 #include "qapi/visitor-impl.h"
+#include "trace.h"
 
 void visit_complete(Visitor *v, void *opaque)
 {
 assert(v->type != VISITOR_OUTPUT || v->complete);
+trace_visit_complete(v, opaque);
 if (v->complete) {
 v->complete(v, opaque);
 }
@@ -30,6 +32,7 @@ void visit_complete(Visitor *v, void *opaque)
 
 void visit_free(Visitor *v)
 {
+trace_visit_free(v);
 if (v) {
 v->free(v);
 }
@@ -40,6 +43,7 @@ void visit_start_struct(Visitor *v, const char *name, void 
**obj,
 {
 Error *err = NULL;
 
+trace_visit_start_struct(v, name, obj, size);
 if (obj) {
 assert(size);
 assert(!(v->type & VISITOR_OUTPUT) || *obj);
@@ -53,6 +57,7 @@ void visit_start_struct(Visitor *v, const char *name, void 
**obj,
 
 void visit_check_struct(Visitor *v, Error **errp)
 {
+trace_visit_check_struct(v);
 if (v->check_struct) {
 v->check_struct(v, errp);
 }
@@ -60,6 +65,7 @@ void visit_check_struct(Visitor *v, Error **errp)
 
 void visit_end_struct(Visitor *v, void **obj)
 {
+trace_visit_end_struct(v, obj);
 v->end_struct(v, obj);
 }
 
@@ -69,6 +75,7 @@ void visit_start_list(Visitor *v, const char *name, 
GenericList **list,
 Error *err = NULL;
 
 assert(!list || size >= sizeof(GenericList));
+trace_visit_start_list(v, name, list, size);
 v->start_list(v, name, list, size, &err);
 if (list && (v->type & VISITOR_INPUT)) {
 assert(!(err && *list));
@@ -79,11 +86,13 @@ void visit_start_list(Visitor *v, const char *name, 
GenericList **list,
 GenericList *visit_next_list(Visitor *v, GenericList *tail, size_t size)
 {
 assert(tail && size >= sizeof(GenericList));
+trace_visit_next_list(v, tail, size);
 return v->next_list(v, tail, size);
 }
 
 void visit_end_list(Visitor *v, void **obj)
 {
+trace_visit_end_list(v, obj);
 v->end_list(v, obj);
 }
 
@@ -95,6 +104,7 @@ void visit_start_alternate(Visitor *v, const char *name,
 
 assert(obj && size >= sizeof(GenericAlternate));
 assert(!(v->type & VISITOR_OUTPUT) || *obj);
+trace_visit_start_alternate(v, name, obj, size, promote_int);
 if (v->start_alternate) {
 v->start_alternate(v, name, obj, size, promote_int, &err);
 }
@@ -106,6 +116,7 @@ void visit_start_alternate(Visitor *v, const char *name,
 
 void visit_end_alternate(Visitor *v, void **obj)
 {
+trace_visit_end_alternate(v, obj);
 if (v->end_alternate) {
 v->end_alternate(v, obj);
 }
@@ -113,6 +124,7 @@ void visit_end_alternate(Visitor *v, void **obj)
 
 bool visit_optional(Visitor *v, const char *name, bool *present)
 {
+trace_visit_optional(v, name, present);
 if (v->optional) {
 v->optional(v, name, present);
 }
@@ -127,6 +139,7 @@ bool visit_is_input(Visitor *v)
 void visit_type_int(Visitor *v, const char *name, int64_t *obj, Error **errp)
 {
 assert(obj);
+trace_visit_type_int(v, name, obj);
 v->type_int64(v, name, obj, errp);
 }
 
@@ -151,6 +164,7 @@ void visit_type_uint8(Visitor *v, const char *name, uint8_t 
*obj,
   Error **errp)
 {
 uint64_t value = *obj;
+trace_visit_type_uint8(v, name, obj);
 visit_type_uintN(v, &value, name, UINT8_MAX, "uint8_t", errp);
 *obj = value;
 }
@@ -159,6 +173,7 @@ void visit_type_uint16(Visitor *v, const char *name, 
uint16_t *obj,
Error **errp)
 {
 uint64_t value = *obj;
+trace_visit_type_uint16(v, name, obj);
 visit_type_uintN(v, &value, name, UINT16_MAX, "uint16_t", errp);
 *obj = value;
 }
@@ -167,6 +182,7 @@ void visit_type_uint32(Visitor *v, const char *name, 
uint32_t *obj,
Error **errp)
 {
 uint64_t value = *obj;
+trace_visit_type_uint32(v, name, obj);
 visit_type_uintN(v, &value, name, UINT32_MAX, "uint32_t", errp);
 *obj = value;
 }
@@ -175,6 +191,7 @@ void visit_type_uint64(Visitor *v, const char *name, 
uint64_t *obj,
Error **errp)
 {
 assert(obj);
+trace_visit_type_uint64(v, name, obj);
 v->type_uint64(v, name, obj, errp);
 }
 
@@ -199,6 +216,7 @@ static void vis

[Qemu-block] [PATCH v14 08/21] qapi: allow QObjectInputVisitor to be created with QemuOpts

[Daniel P. Berrange]

Instead of requiring all callers to go through the mutli-step
process of turning QemuOpts into a suitable QObject for visiting,
add a new constructor that encapsulates this logic. This will
allow QObjectInputVisitor to be a drop-in replacement for the
existing OptsVisitor with minimal code changes for callers.

NB, at this point it is only supporting opts syntax which
explicitly matches the QAPI schema structure, so is not yet
a true drop-in replacement for OptsVisitor. The patches that
follow will add the special cases requird for full backwards
compatibility with OptsVisitor.

Signed-off-by: Daniel P. Berrange 
---
 include/qapi/qobject-input-visitor.h | 15 +++
 include/qemu/option.h|  2 +-
 qapi/qobject-input-visitor.c | 25 +
 util/qemu-option.c   |  2 +-
 4 files changed, 42 insertions(+), 2 deletions(-)

diff --git a/include/qapi/qobject-input-visitor.h 
b/include/qapi/qobject-input-visitor.h
index 5022297..f134d90 100644
--- a/include/qapi/qobject-input-visitor.h
+++ b/include/qapi/qobject-input-visitor.h
@@ -51,4 +51,19 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool 
strict);
  */
 Visitor *qobject_input_visitor_new_autocast(QObject *obj);
 
+
+/**
+ * Create a new input visitor that converts @opts to a QAPI object.
+ *
+ * The QemuOpts will be converted into a QObject using the
+ * qdict_crumple() method to automatically create structs
+ * and lists. The resulting QDict will then be passed to the
+ * qobject_input_visitor_new_autocast() method. See the docs
+ * of that method for further details on processing behaviour.
+ *
+ * The returned input visitor should be released by calling
+ * visit_free() when no longer required.
+ */
+Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, Error **errp);
+
 #endif
diff --git a/include/qemu/option.h b/include/qemu/option.h
index 2a5266f..29f3f18 100644
--- a/include/qemu/option.h
+++ b/include/qemu/option.h
@@ -125,7 +125,7 @@ void qemu_opts_set_defaults(QemuOptsList *list, const char 
*params,
 int permit_abbrev);
 QemuOpts *qemu_opts_from_qdict(QemuOptsList *list, const QDict *qdict,
Error **errp);
-QDict *qemu_opts_to_qdict(QemuOpts *opts, QDict *qdict);
+QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict);
 void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp);
 
 typedef int (*qemu_opts_loopfunc)(void *opaque, QemuOpts *opts, Error **errp);
diff --git a/qapi/qobject-input-visitor.c b/qapi/qobject-input-visitor.c
index cf41df6..d9269c9 100644
--- a/qapi/qobject-input-visitor.c
+++ b/qapi/qobject-input-visitor.c
@@ -545,3 +545,28 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj)
 
 return &v->visitor;
 }
+
+
+Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
+Error **errp)
+{
+QDict *pdict;
+QObject *pobj = NULL;
+Visitor *v = NULL;
+
+pdict = qemu_opts_to_qdict(opts, NULL);
+if (!pdict) {
+goto cleanup;
+}
+
+pobj = qdict_crumple(pdict, true, errp);
+if (!pobj) {
+goto cleanup;
+}
+
+v = qobject_input_visitor_new_autocast(pobj);
+ cleanup:
+qobject_decref(pobj);
+QDECREF(pdict);
+return v;
+}
diff --git a/util/qemu-option.c b/util/qemu-option.c
index 41b356c..418cbb6 100644
--- a/util/qemu-option.c
+++ b/util/qemu-option.c
@@ -1058,7 +1058,7 @@ void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, 
Error **errp)
  * TODO We'll want to use types appropriate for opt->desc->type, but
  * this is enough for now.
  */
-QDict *qemu_opts_to_qdict(QemuOpts *opts, QDict *qdict)
+QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict)
 {
 QemuOpt *opt;
 QObject *val;
-- 
2.7.4

[Qemu-block] [PATCH v14 12/21] option: allow qemu_opts_to_qdict to merge repeated options

[Daniel P. Berrange]

If given an option string such as

  size=1024,nodes=10,nodes=4-5,nodes=1-2,policy=bind

the qemu_opts_to_qdict() method will currently overwrite
the values for repeated option keys, so only the last
value is in the returned dict:

size=QString("1024")
nodes=QString("1-2")
policy=QString("bind")

With this change the caller can optionally ask for all
the repeated values to be stored in a QList. In the
above example that would result in 'nodes' being a
QList, so the returned dict would contain

size=QString("1024")
nodes=QList([QString("10"),
 QString("4-5"),
 QString("1-2")])
policy=QString("bind")

Note that the conversion has no way of knowing whether
any given key is expected to be a list upfront - it can
only figure that out when seeing the first duplicated
key. Thus the caller has to be prepared to deal with the
fact that if a key 'foo' is a list, then the returned
qdict may contain either a QString or a QList for the
key 'foo'.

In a third mode, it is possible to ask for repeated
options to be reported as an error, rather than silently
dropping all but the last one.

All existing callers are all converted to explicitly
request the historical behaviour of only reporting the
last key. Later patches will make use of the new modes.

Signed-off-by: Daniel P. Berrange 
---
 blockdev.c   |   7 ++-
 include/qemu/option.h|  13 -
 monitor.c|   3 +-
 qapi/qobject-input-visitor.c |   4 +-
 qemu-img.c   |   4 +-
 qemu-io-cmds.c   |   3 +-
 qemu-io.c|   6 +-
 qemu-nbd.c   |   3 +-
 qom/object_interfaces.c  |   3 +-
 tests/test-qemu-opts.c   | 132 +++
 tests/test-replication.c |   9 ++-
 util/qemu-option.c   |  64 ++---
 12 files changed, 229 insertions(+), 22 deletions(-)

diff --git a/blockdev.c b/blockdev.c
index 814d49f..a999d46 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -914,7 +914,8 @@ DriveInfo *drive_new(QemuOpts *all_opts, BlockInterfaceType 
block_default_type)
 
 /* Get a QDict for processing the options */
 bs_opts = qdict_new();
-qemu_opts_to_qdict(all_opts, bs_opts);
+qemu_opts_to_qdict(all_opts, bs_opts, QEMU_OPTS_REPEAT_POLICY_LAST,
+   &error_abort);
 
 legacy_opts = qemu_opts_create(&qemu_legacy_drive_opts, NULL, 0,
&error_abort);
@@ -3804,8 +3805,8 @@ void hmp_drive_add_node(Monitor *mon, const char *optstr)
 return;
 }
 
-qdict = qemu_opts_to_qdict(opts, NULL);
-
+qdict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+   &error_abort);
 if (!qdict_get_try_str(qdict, "node-name")) {
 QDECREF(qdict);
 error_report("'node-name' needs to be specified");
diff --git a/include/qemu/option.h b/include/qemu/option.h
index 29f3f18..ffd841d 100644
--- a/include/qemu/option.h
+++ b/include/qemu/option.h
@@ -125,7 +125,18 @@ void qemu_opts_set_defaults(QemuOptsList *list, const char 
*params,
 int permit_abbrev);
 QemuOpts *qemu_opts_from_qdict(QemuOptsList *list, const QDict *qdict,
Error **errp);
-QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict);
+typedef enum {
+/* Last occurrence of a duplicate option silently replaces earlier */
+QEMU_OPTS_REPEAT_POLICY_LAST,
+/* Each occurrence of a duplicate option converts value to a QList */
+QEMU_OPTS_REPEAT_POLICY_ALL,
+/* First occurrence of a duplicate option causes an error */
+QEMU_OPTS_REPEAT_POLICY_ERROR,
+} QemuOptsRepeatPolicy;
+
+QDict *qemu_opts_to_qdict(const QemuOpts *opts, QDict *qdict,
+  QemuOptsRepeatPolicy repeatPolicy,
+  Error **errp);
 void qemu_opts_absorb_qdict(QemuOpts *opts, QDict *qdict, Error **errp);
 
 typedef int (*qemu_opts_loopfunc)(void *opaque, QemuOpts *opts, Error **errp);
diff --git a/monitor.c b/monitor.c
index 14089cc..84e79d4 100644
--- a/monitor.c
+++ b/monitor.c
@@ -2642,7 +2642,8 @@ static QDict *monitor_parse_arguments(Monitor *mon,
 if (!opts) {
 goto fail;
 }
-qemu_opts_to_qdict(opts, qdict);
+qemu_opts_to_qdict(opts, qdict, QEMU_OPTS_REPEAT_POLICY_LAST,
+   &error_abort);
 qemu_opts_del(opts);
 }
 break;
diff --git a/qapi/qobject-input-visitor.c b/qapi/qobject-input-visitor.c
index 6b3d0f2..2287d11 100644
--- a/qapi/qobject-input-visitor.c
+++ b/qapi/qobject-input-visitor.c
@@ -759,7 +759,9 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts 
*opts,
 QObject *pobj = NULL;
 Visitor *v = NULL;
 
-pdict = qemu_opts_to_qdict(opts, NULL);
+pdict = qemu_opts_to_qdict(opts, NULL,
+   Q

[Qemu-block] [PATCH v14 13/21] qdict: allow qdict_crumple to accept compound types as values

[Daniel P. Berrange]

Currently qdict_crumple requires a totally flat QDict as its
input. i.e. all values in the QDict must be scalar types.

In order to have backwards compatibility with the OptsVisitor,
qemu_opt_to_qdict() has a new mode where it may return a QList
for values in the QDict, if there was a repeated key. We thus
need to allow compound types to appear as values in the input
dict given to qdict_crumple().

To avoid confusion, we sanity check that the user has not mixed
the old and new syntax at the same time. e.g. these are allowed

   foo=hello,foo=world,foo=wibble
   foo.0=hello,foo.1=world,foo.2=wibble

but this is forbidden

   foo=hello,foo=world,foo.2=wibble

Signed-off-by: Daniel P. Berrange 
---
 qobject/qdict.c |  45 +-
 tests/check-qdict.c | 175 
 2 files changed, 204 insertions(+), 16 deletions(-)

diff --git a/qobject/qdict.c b/qobject/qdict.c
index c38e90e..83384b2 100644
--- a/qobject/qdict.c
+++ b/qobject/qdict.c
@@ -814,15 +814,16 @@ static int qdict_is_list(QDict *maybe_list, Error **errp)
 
 /**
  * qdict_crumple:
- * @src: the original flat dictionary (only scalar values) to crumple
+ * @src: the original dictionary to crumple
  * @recursive: true to recursively crumple nested dictionaries
  *
- * Takes a flat dictionary whose keys use '.' separator to indicate
- * nesting, and values are scalars, and crumples it into a nested
- * structure. If the @recursive parameter is false, then only the
- * first level of structure implied by the keys will be crumpled. If
- * @recursive is true, then the input will be recursively crumpled to
- * expand all levels of structure in the keys.
+ * Takes a dictionary whose keys use '.' separator to indicate
+ * nesting, crumples it into a nested structure. The values in
+ * @src are permitted to be scalars or compound typee. If the
+ * @recursive parameter is false, then only the first level of
+ * structure implied by the keys will be crumpled. If @recursive
+ * is true, then the input will be recursively crumpled to expand
+ * all levels of structure in the keys.
  *
  * To include a literal '.' in a key name, it must be escaped as '..'
  *
@@ -843,7 +844,10 @@ static int qdict_is_list(QDict *maybe_list, Error **errp)
  * The following scenarios in the input dict will result in an
  * error being returned:
  *
- *  - Any values in @src are non-scalar types
+ *  - If a key in @src has a value that is non-scalar, and
+ *a later key implies creation of a compound type. e.g.
+ *if 'foo' point to a 'QList' or 'QDict', then it is
+ *not permitted to also have 'foo.NNN' keys later.
  *  - If keys in @src imply that a particular level is both a
  *list and a dict. e.g., "foo.0.bar" and "foo.eek.bar".
  *  - If keys in @src imply that a particular level is a list,
@@ -864,21 +868,26 @@ QObject *qdict_crumple(const QDict *src, bool recursive, 
Error **errp)
 char *prefix = NULL;
 const char *suffix = NULL;
 int is_list;
+QDict *compound;
 
+compound = qdict_new();
 two_level = qdict_new();
 
 /* Step 1: split our totally flat dict into a two level dict */
 for (ent = qdict_first(src); ent != NULL; ent = qdict_next(src, ent)) {
-if (qobject_type(ent->value) == QTYPE_QDICT ||
-qobject_type(ent->value) == QTYPE_QLIST) {
-error_setg(errp, "Value %s is not a scalar",
-   ent->key);
-goto error;
-}
-
 qdict_split_flat_key(ent->key, &prefix, &suffix);
 
 child = qdict_get(two_level, prefix);
+
+if (qdict_haskey(compound, prefix)) {
+error_setg(errp, "Key %s is already set as a list/dict", prefix);
+goto error;
+}
+if (qobject_type(ent->value) == QTYPE_QLIST ||
+qobject_type(ent->value) == QTYPE_QDICT) {
+qobject_incref(ent->value);
+qdict_put_obj(compound, prefix, ent->value);
+}
 if (suffix) {
 if (child) {
 if (qobject_type(child) != QTYPE_QDICT) {
@@ -913,7 +922,8 @@ QObject *qdict_crumple(const QDict *src, bool recursive, 
Error **errp)
 for (ent = qdict_first(two_level); ent != NULL;
  ent = qdict_next(two_level, ent)) {
 
-if (qobject_type(ent->value) == QTYPE_QDICT) {
+if (qobject_type(ent->value) == QTYPE_QDICT &&
+!qdict_haskey(compound, ent->key)) {
 child = qdict_crumple(qobject_to_qdict(ent->value),
   recursive, errp);
 if (!child) {
@@ -961,10 +971,13 @@ QObject *qdict_crumple(const QDict *src, bool recursive, 
Error **errp)
 dst = QOBJECT(multi_level);
 }
 
+QDECREF(compound);
+
 return dst;
 
  error:
 g_free(prefix);
+QDECREF(compound);
 QDECREF(multi_level);
 QDECREF(two_level);
 qobject_decref(dst);
diff --git a/tests/check-qdict.c b/tests/check-qdict.c
index 64c33ab.

[Qemu-block] [PATCH v14 02/21] qdict: implement a qdict_crumple method for un-flattening a dict

[Daniel P. Berrange]

The qdict_flatten() method will take a dict whose elements are
further nested dicts/lists and flatten them by concatenating
keys.

The qdict_crumple() method aims to do the reverse, taking a flat
qdict, and turning it into a set of nested dicts/lists. It will
apply nesting based on the key name, with a '.' indicating a
new level in the hierarchy. If the keys in the nested structure
are all numeric, it will create a list, otherwise it will create
a dict.

If the keys are a mixture of numeric and non-numeric, or the
numeric keys are not in strictly ascending order, an error will
be reported.

As an example, a flat dict containing

 {
   'foo.0.bar': 'one',
   'foo.0.wizz': '1',
   'foo.1.bar': 'two',
   'foo.1.wizz': '2'
 }

will get turned into a dict with one element 'foo' whose
value is a list. The list elements will each in turn be
dicts.

 {
   'foo': [
 { 'bar': 'one', 'wizz': '1' },
 { 'bar': 'two', 'wizz': '2' }
   ],
 }

If the key is intended to contain a literal '.', then it must
be escaped as '..'. ie a flat dict

  {
 'foo..bar': 'wizz',
 'bar.foo..bar': 'eek',
 'bar.hello': 'world'
  }

Will end up as

  {
 'foo.bar': 'wizz',
 'bar': {
'foo.bar': 'eek',
'hello': 'world'
 }
  }

The intent of this function is that it allows a set of QemuOpts
to be turned into a nested data structure that mirrors the nesting
used when the same object is defined over QMP.

Reviewed-by: Eric Blake 
Reviewed-by: Kevin Wolf 
Reviewed-by: Marc-André Lureau 
Signed-off-by: Daniel P. Berrange 
---
 include/qapi/qmp/qdict.h |   1 +
 qobject/qdict.c  | 289 +++
 tests/check-qdict.c  | 261 ++
 3 files changed, 551 insertions(+)

diff --git a/include/qapi/qmp/qdict.h b/include/qapi/qmp/qdict.h
index 71b8eb0..e0d24e1 100644
--- a/include/qapi/qmp/qdict.h
+++ b/include/qapi/qmp/qdict.h
@@ -73,6 +73,7 @@ void qdict_flatten(QDict *qdict);
 void qdict_extract_subqdict(QDict *src, QDict **dst, const char *start);
 void qdict_array_split(QDict *src, QList **dst);
 int qdict_array_entries(QDict *src, const char *subqdict);
+QObject *qdict_crumple(const QDict *src, bool recursive, Error **errp);
 
 void qdict_join(QDict *dest, QDict *src, bool overwrite);
 
diff --git a/qobject/qdict.c b/qobject/qdict.c
index 60f158c..c38e90e 100644
--- a/qobject/qdict.c
+++ b/qobject/qdict.c
@@ -17,6 +17,7 @@
 #include "qapi/qmp/qbool.h"
 #include "qapi/qmp/qstring.h"
 #include "qapi/qmp/qobject.h"
+#include "qapi/error.h"
 #include "qemu/queue.h"
 #include "qemu-common.h"
 #include "qemu/cutils.h"
@@ -683,6 +684,294 @@ void qdict_array_split(QDict *src, QList **dst)
 }
 }
 
+
+/**
+ * qdict_split_flat_key:
+ * @key: the key string to split
+ * @prefix: non-NULL pointer to hold extracted prefix
+ * @suffix: non-NULL pointer to remaining suffix
+ *
+ * Given a flattened key such as 'foo.0.bar', split it into two parts
+ * at the first '.' separator. Allows double dot ('..') to escape the
+ * normal separator.
+ *
+ * e.g.
+ *'foo.0.bar' -> prefix='foo' and suffix='0.bar'
+ *'foo..0.bar' -> prefix='foo.0' and suffix='bar'
+ *
+ * The '..' sequence will be unescaped in the returned 'prefix'
+ * string. The 'suffix' string will be left in escaped format, so it
+ * can be fed back into the qdict_split_flat_key() key as the input
+ * later.
+ *
+ * The caller is responsible for freeing the string returned in @prefix
+ * using g_free().
+ */
+static void qdict_split_flat_key(const char *key, char **prefix,
+ const char **suffix)
+{
+const char *separator;
+size_t i, j;
+
+/* Find first '.' separator, but if there is a pair '..'
+ * that acts as an escape, so skip over '..' */
+separator = NULL;
+do {
+if (separator) {
+separator += 2;
+} else {
+separator = key;
+}
+separator = strchr(separator, '.');
+} while (separator && separator[1] == '.');
+
+if (separator) {
+*prefix = g_strndup(key, separator - key);
+*suffix = separator + 1;
+} else {
+*prefix = g_strdup(key);
+*suffix = NULL;
+}
+
+/* Unescape the '..' sequence into '.' */
+for (i = 0, j = 0; (*prefix)[i] != '\0'; i++, j++) {
+if ((*prefix)[i] == '.') {
+assert((*prefix)[i + 1] == '.');
+i++;
+}
+(*prefix)[j] = (*prefix)[i];
+}
+(*prefix)[j] = '\0';
+}
+
+
+/**
+ * qdict_is_list:
+ * @maybe_list: dict to check if keys represent list elements.
+ *
+ * Determine whether all keys in @maybe_list are valid list elements.
+ * If @maybe_list is non-zero in length and all the keys look like
+ * valid list indexes, this will return 1. If @maybe_list is zero
+ * length or all keys are non-numeric then it will return 0 to indicate
+ * it is a normal qdict. If there is a mix of numeric and non-numeric
+ * keys, or the list indexes are 

[Qemu-block] [PATCH v14 07/21] qapi: permit scalar type conversions in QObjectInputVisitor

[Daniel P. Berrange]

Currently the QObjectInputVisitor assumes that all scalar
values are directly represented as the final types declared
by the thing being visited. ie it assumes an 'int' is using
QInt, and a 'bool' is using QBool, etc.  This is good when
QObjectInputVisitor is fed a QObject that came from a JSON
document on the QMP monitor, as it will strictly validate
correctness.

To allow QObjectInputVisitor to be reused for visiting
a QObject originating from QemuOpts, an alternative mode
is needed where all the scalars types are represented as
QString and converted on the fly to the final desired
type.

Reviewed-by: Kevin Wolf 
Reviewed-by: Marc-André Lureau 
Signed-off-by: Daniel P. Berrange 
---
 include/qapi/qobject-input-visitor.h |  32 +-
 qapi/qobject-input-visitor.c | 132 
 tests/test-qobject-input-visitor.c   | 194 ++-
 3 files changed, 350 insertions(+), 8 deletions(-)

diff --git a/include/qapi/qobject-input-visitor.h 
b/include/qapi/qobject-input-visitor.h
index cde328d..5022297 100644
--- a/include/qapi/qobject-input-visitor.h
+++ b/include/qapi/qobject-input-visitor.h
@@ -19,12 +19,36 @@
 
 typedef struct QObjectInputVisitor QObjectInputVisitor;
 
-/*
- * Return a new input visitor that converts a QObject to a QAPI object.
+/**
+ * Create a new input visitor that converts @obj to a QAPI object.
+ *
+ * Any scalar values in the @obj input data structure should be in the
+ * required type already. i.e. if visiting a bool, the value should
+ * already be a QBool instance.
  *
- * Set @strict to reject a parse that doesn't consume all keys of a
- * dictionary; otherwise excess input is ignored.
+ * If @strict is set to true, then an error will be reported if any
+ * dict keys are not consumed during visitation. If @strict is false
+ * then extra dict keys are silently ignored.
+ *
+ * The returned input visitor should be released by calling
+ * visit_free() when no longer required.
  */
 Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
 
+/**
+ * Create a new input visitor that converts @obj to a QAPI object.
+ *
+ * Any scalar values in the @obj input data structure should always be
+ * represented as strings. i.e. if visiting a boolean, the value should
+ * be a QString whose contents represent a valid boolean.
+ *
+ * The visitor always operates in strict mode, requiring all dict keys
+ * to be consumed during visitation. An error will be reported if this
+ * does not happen.
+ *
+ * The returned input visitor should be released by calling
+ * visit_free() when no longer required.
+ */
+Visitor *qobject_input_visitor_new_autocast(QObject *obj);
+
 #endif
diff --git a/qapi/qobject-input-visitor.c b/qapi/qobject-input-visitor.c
index 5ff3db3..cf41df6 100644
--- a/qapi/qobject-input-visitor.c
+++ b/qapi/qobject-input-visitor.c
@@ -20,6 +20,7 @@
 #include "qemu-common.h"
 #include "qapi/qmp/types.h"
 #include "qapi/qmp/qerror.h"
+#include "qemu/cutils.h"
 
 #define QIV_STACK_SIZE 1024
 
@@ -263,6 +264,28 @@ static void qobject_input_type_int64(Visitor *v, const 
char *name, int64_t *obj,
 *obj = qint_get_int(qint);
 }
 
+
+static void qobject_input_type_int64_autocast(Visitor *v, const char *name,
+  int64_t *obj, Error **errp)
+{
+QObjectInputVisitor *qiv = to_qiv(v);
+QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
+true));
+int64_t ret;
+
+if (!qstr || !qstr->string) {
+error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+   "string");
+return;
+}
+
+if (qemu_strtoll(qstr->string, NULL, 0, &ret) < 0) {
+error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");
+return;
+}
+*obj = ret;
+}
+
 static void qobject_input_type_uint64(Visitor *v, const char *name,
   uint64_t *obj, Error **errp)
 {
@@ -279,6 +302,27 @@ static void qobject_input_type_uint64(Visitor *v, const 
char *name,
 *obj = qint_get_int(qint);
 }
 
+static void qobject_input_type_uint64_autocast(Visitor *v, const char *name,
+   uint64_t *obj, Error **errp)
+{
+QObjectInputVisitor *qiv = to_qiv(v);
+QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
+true));
+unsigned long long ret;
+
+if (!qstr || !qstr->string) {
+error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name ? name : "null",
+   "string");
+return;
+}
+
+if (parse_uint_full(qstr->string, &ret, 0) < 0) {
+error_setg(errp, QERR_INVALID_PARAMETER_VALUE, name, "a number");
+return;
+}
+*obj = ret;
+}
+
 static void qobject_input_type_bool(Visitor *v, const char *name, bool *obj,
 Error **errp)
 {

[Qemu-block] [PATCH v14 10/21] qapi: permit auto-creating nested structs

[Daniel P. Berrange]

Some of the historical command line opts that had their
keys in in a completely flat namespace are now represented
by QAPI schemas that use a nested structs. When converting
the QemuOpts to QObject, there is no information about
compound types available, so the QObject will be completely
flat, even after the qdict_crumple() call. So when starting
a struct, we may not have a QDict available in the input
data, so we auto-create a QDict containing all the currently
unvisited input data keys. Not all historical command line
opts require this, so the behaviour is opt-in, by specifying
how many levels of structs are permitted to be auto-created.

Note that this only works if the child struct is the last
field to the visited in the parent struct. This is always
the case for currently existing legacy command line options.

The example is the NetLegacy type which has 3 levels of
structs. The modern way to represent this in QemuOpts would
be the dot-separated component approach

  -net vlan=1,id=foo,name=bar,opts.type=tap,\
   opts.data.fd=3,opts.data.script=ifup

The legacy syntax will just be presenting

  -net vlan=1,id=foo,name=bar,type=tap,fd=3,script=ifup

So we need to auto-create 3 levels of struct when visiting.

The implementation here will enable visiting in both the
modern and legacy syntax, compared to OptsVisitor which
only allows the legacy syntax.

Signed-off-by: Daniel P. Berrange 
---
 include/qapi/qobject-input-visitor.h |  22 +-
 qapi/qobject-input-visitor.c |  59 ++--
 tests/test-qobject-input-visitor.c   | 130 ---
 3 files changed, 194 insertions(+), 17 deletions(-)

diff --git a/include/qapi/qobject-input-visitor.h 
b/include/qapi/qobject-input-visitor.h
index 1809f48..94051f3 100644
--- a/include/qapi/qobject-input-visitor.h
+++ b/include/qapi/qobject-input-visitor.h
@@ -45,7 +45,7 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  * If @autocreate_list is true, then as an alternative to a normal QList,
  * list values can be stored as a QString or QDict instead, which will
  * be interpreted as representing single element lists. This should only
- * by used if compatibility is required with the OptsVisitor which allowed
+ * be used if compatibility is required with the OptsVisitor which allowed
  * repeated keys, without list indexes, to represent lists. e.g. set this
  * to true if you have compatibility requirements for
  *
@@ -55,6 +55,22 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool 
strict);
  *
  *   -arg foo.0=hello,foo.1=world
  *
+ * If @autocreate_struct_levels is non-zero, then when visiting structs,
+ * if the corresponding QDict is not found, it will automatically create
+ * a QDict containing all remaining unvisited options. This should only
+ * be used if compatibility is required with the OptsVisitor which flatten
+ * structs so that all keys were at the same level. e.g. set this to a
+ * non-zero number if you compatibility requirements for
+ *
+ *   -arg type=person,surname=blogs,forename=fred
+ *
+ * to be treated as equivalent to the perferred syntax
+ *
+ *   -arg type=person,data.surname=blogs,data.forename=fred
+ *
+ * The value given determines how many levels of structs are allowed to
+ * be flattened in this way.
+ *
  * The visitor always operates in strict mode, requiring all dict keys
  * to be consumed during visitation. An error will be reported if this
  * does not happen.
@@ -63,7 +79,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  * visit_free() when no longer required.
  */
 Visitor *qobject_input_visitor_new_autocast(QObject *obj,
-bool autocreate_list);
+bool autocreate_list,
+size_t autocreate_struct_levels);
 
 
 /**
@@ -80,6 +97,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj,
  */
 Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
 bool autocreate_list,
+size_t autocreate_struct_levels,
 Error **errp);
 
 #endif
diff --git a/qapi/qobject-input-visitor.c b/qapi/qobject-input-visitor.c
index d88e9f9..1be4865 100644
--- a/qapi/qobject-input-visitor.c
+++ b/qapi/qobject-input-visitor.c
@@ -52,6 +52,14 @@ struct QObjectInputVisitor
 /* Whether we can auto-create single element lists when
  * encountering a non-QList type */
 bool autocreate_list;
+
+/* Current depth of recursion into structs */
+size_t struct_level;
+
+/* Numbers of levels at which we will
+ * consider auto-creating a struct containing
+ * remaining unvisited items */
+size_t autocreate_struct_levels;
 };
 
 static QObjectInputVisitor *to_qiv(Visitor *v)
@@ -79,7 +87,12 @@ static QObject *qobject_input_get_object(QObjectInputVisitor 
*qiv,
 
 if (qobject_type(qobj) == Q

[Qemu-block] [PATCH v14 11/21] qapi: add integer range support for QObjectInputVisitor

[Daniel P. Berrange]

The traditional CLI arg syntax allows two ways to specify
integer lists, either one value per key, or a range of
values per key. eg the following are identical:

  -arg foo=5,foo=6,foo=7
  -arg foo=5-7

This extends the QObjectInputVisitor so that it is able
to parse ranges and turn them into distinct list entries.

This means that

  -arg foo=5-7

is treated as equivalent to

  -arg foo.0=5,foo.1=6,foo.2=7

Edge case tests are copied from test-opts-visitor to
ensure identical behaviour when parsing.

Signed-off-by: Daniel P. Berrange 
---
 include/qapi/qobject-input-visitor.h |  23 -
 qapi/qobject-input-visitor.c | 158 ++--
 tests/test-qobject-input-visitor.c   | 195 +--
 3 files changed, 360 insertions(+), 16 deletions(-)

diff --git a/include/qapi/qobject-input-visitor.h 
b/include/qapi/qobject-input-visitor.h
index 94051f3..242b767 100644
--- a/include/qapi/qobject-input-visitor.h
+++ b/include/qapi/qobject-input-visitor.h
@@ -19,6 +19,12 @@
 
 typedef struct QObjectInputVisitor QObjectInputVisitor;
 
+/* Inclusive upper bound on the size of any flattened range. This is a safety
+ * (= anti-annoyance) measure; wrong ranges should not cause long startup
+ * delays nor exhaust virtual memory.
+ */
+#define QIV_RANGE_MAX 65536
+
 /**
  * Create a new input visitor that converts @obj to a QAPI object.
  *
@@ -71,6 +77,19 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool 
strict);
  * The value given determines how many levels of structs are allowed to
  * be flattened in this way.
  *
+ * If @permit_int_ranges is true, then when visiting a list of integers,
+ * the integer value strings may encode ranges eg a single element
+ * containing "5-7" is treated as if there were three elements "5", "6",
+ * "7". This should only be used if compatibility is required with the
+ * OptsVisitor which would allow integer ranges. e.g. set this to true
+ * if you have compatibility requirements for
+ *
+ *   -arg val=5-8
+ *
+ * to be treated as equivalent to the preferred syntax:
+ *
+ *   -arg val.0=5,val.1=6,val.2=7,val.3=8
+ *
  * The visitor always operates in strict mode, requiring all dict keys
  * to be consumed during visitation. An error will be reported if this
  * does not happen.
@@ -80,7 +99,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  */
 Visitor *qobject_input_visitor_new_autocast(QObject *obj,
 bool autocreate_list,
-size_t autocreate_struct_levels);
+size_t autocreate_struct_levels,
+bool permit_int_ranges);
 
 
 /**
@@ -98,6 +118,7 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj,
 Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
 bool autocreate_list,
 size_t autocreate_struct_levels,
+bool permit_int_ranges,
 Error **errp);
 
 #endif
diff --git a/qapi/qobject-input-visitor.c b/qapi/qobject-input-visitor.c
index 1be4865..6b3d0f2 100644
--- a/qapi/qobject-input-visitor.c
+++ b/qapi/qobject-input-visitor.c
@@ -31,6 +31,8 @@ typedef struct StackObject
 
 GHashTable *h;   /* If obj is dict: unvisited keys */
 const QListEntry *entry; /* If obj is list: unvisited tail */
+uint64_t range_val;
+uint64_t range_limit;
 
 QSLIST_ENTRY(StackObject) node;
 } StackObject;
@@ -60,6 +62,10 @@ struct QObjectInputVisitor
  * consider auto-creating a struct containing
  * remaining unvisited items */
 size_t autocreate_struct_levels;
+
+/* Whether int lists can have single values representing
+ * ranges of values */
+bool permit_int_ranges;
 };
 
 static QObjectInputVisitor *to_qiv(Visitor *v)
@@ -282,7 +288,7 @@ static GenericList *qobject_input_next_list(Visitor *v, 
GenericList *tail,
 QObjectInputVisitor *qiv = to_qiv(v);
 StackObject *so = QSLIST_FIRST(&qiv->stack);
 
-if (!so->entry) {
+if ((so->range_val == so->range_limit) && !so->entry) {
 return NULL;
 }
 tail->next = g_malloc0(size);
@@ -329,21 +335,87 @@ static void qobject_input_type_int64_autocast(Visitor *v, 
const char *name,
   int64_t *obj, Error **errp)
 {
 QObjectInputVisitor *qiv = to_qiv(v);
-QString *qstr = qobject_to_qstring(qobject_input_get_object(qiv, name,
-true));
+QString *qstr;
 int64_t ret;
+const char *end = NULL;
+StackObject *tos;
+bool inlist = false;
+
+/* Preferentially generate values from a range, before
+ * trying to consume another QList element */
+tos = QSLIST_FIRST(&qiv->stack);
+if (tos) {
+if ((int64_t)tos->range_val < (int64_t)tos-

[Qemu-block] [PATCH v14 16/21] hmp: support non-scalar properties with object_add

[Daniel P. Berrange]

The current object_add HMP syntax only allows for
creation of objects with scalar properties, or a list
with a fixed scalar element type. Objects which have
properties that are represented as structs in the QAPI
schema cannot be created using -object.

This is a design limitation of the way the OptsVisitor
is written. It simply iterates over the QemuOpts values
as a flat list. The support for lists is enabled by
allowing the same key to be repeated in the opts string.

The QObjectInputVisitor now has functionality that allows
it to replace OptsVisitor, maintaining full backwards
compatibility for previous CLI syntax, while also allowing
use of new syntax for structs.

Thus -object can now support non-scalar properties,
for example the QMP object:

  {
"execute": "object-add",
"arguments": {
  "qom-type": "demo",
  "id": "demo0",
  "parameters": {
"foo": [
  { "bar": "one", "wizz": "1" },
  { "bar": "two", "wizz": "2" }
]
  }
}
  }

Would be creatable via the HMP now using

   object_add demo,id=demo0,\
  foo.0.bar=one,foo.0.wizz=1,\
  foo.1.bar=two,foo.1.wizz=2

Notice that this syntax is intentionally compatible
with that currently used by block drivers. NB the
indentation seen here after line continuations should
not be used in reality, it is just for clarity in this
commit message.

Signed-off-by: Daniel P. Berrange 
---
 hmp.c | 25 +
 1 file changed, 17 insertions(+), 8 deletions(-)

diff --git a/hmp.c b/hmp.c
index 336e7bf..b32d8c8 100644
--- a/hmp.c
+++ b/hmp.c
@@ -25,7 +25,7 @@
 #include "qemu/sockets.h"
 #include "monitor/monitor.h"
 #include "monitor/qdev.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "qapi/qmp/qerror.h"
 #include "qapi/string-output-visitor.h"
 #include "qapi/util.h"
@@ -1696,21 +1696,30 @@ void hmp_netdev_del(Monitor *mon, const QDict *qdict)
 void hmp_object_add(Monitor *mon, const QDict *qdict)
 {
 Error *err = NULL;
-QemuOpts *opts;
 Visitor *v;
 Object *obj = NULL;
+QObject *pobj;
 
-opts = qemu_opts_from_qdict(qemu_find_opts("object"), qdict, &err);
+pobj = qdict_crumple(qdict, true, &err);
 if (err) {
-hmp_handle_error(mon, &err);
-return;
+goto cleanup;
 }
 
-v = opts_visitor_new(opts);
-obj = user_creatable_add(qdict, v, &err);
+/*
+ * We need autocreate_list=true + permit_int_ranges=true
+ * in order to maintain compat with OptsVisitor creation
+ * of the 'numa' object which had an int16List property.
+ *
+ * We need autocreate_structs=1, because at the CLI level
+ * we have the object type + properties in the same flat
+ * struct, even though at the QMP level they are nested.
+ */
+v = qobject_input_visitor_new_autocast(pobj, true, 1, true);
+obj = user_creatable_add(qobject_to_qdict(pobj), v, &err);
 visit_free(v);
-qemu_opts_del(opts);
 
+ cleanup:
+qobject_decref(pobj);
 if (err) {
 hmp_handle_error(mon, &err);
 }
-- 
2.7.4

[Qemu-block] [PATCH v14 05/21] qapi: rename QmpOutputVisitor to QObjectOutputVisitor

[Daniel P. Berrange]

The QmpOutputVisitor has no direct dependency on QMP. It is
valid to use it anywhere that one wants a QObject. Rename it
to better reflect its functionality as a generic QAPI
to QObject converter.

Reviewed-by: Kevin Wolf 
Reviewed-by: Eric Blake 
Signed-off-by: Daniel P. Berrange 
---
 block/qapi.c   |   4 +-
 blockdev.c |   4 +-
 docs/qapi-code-gen.txt |   2 +-
 ...p-output-visitor.h => qobject-output-visitor.h} |  10 +-
 qapi/Makefile.objs |   2 +-
 qapi/qapi-clone-visitor.c  |   2 +-
 qapi/qmp-output-visitor.c  | 256 -
 qapi/qobject-output-visitor.c  | 254 
 qemu-img.c |   8 +-
 qom/object_interfaces.c|   2 +-
 qom/qom-qobject.c  |   4 +-
 scripts/qapi-commands.py   |   4 +-
 scripts/qapi-event.py  |   4 +-
 tests/.gitignore   |   2 +-
 tests/Makefile.include |   8 +-
 tests/check-qnull.c|   4 +-
 ...put-visitor.c => test-qobject-output-visitor.c} |   6 +-
 tests/test-string-output-visitor.c |   2 +-
 tests/test-visitor-serialization.c |   4 +-
 util/qemu-sockets.c|   2 +-
 20 files changed, 291 insertions(+), 293 deletions(-)
 rename include/qapi/{qmp-output-visitor.h => qobject-output-visitor.h} (66%)
 delete mode 100644 qapi/qmp-output-visitor.c
 create mode 100644 qapi/qobject-output-visitor.c
 rename tests/{test-qmp-output-visitor.c => test-qobject-output-visitor.c} (99%)

diff --git a/block/qapi.c b/block/qapi.c
index 6f947e3..f35c89f 100644
--- a/block/qapi.c
+++ b/block/qapi.c
@@ -29,7 +29,7 @@
 #include "block/write-threshold.h"
 #include "qmp-commands.h"
 #include "qapi-visit.h"
-#include "qapi/qmp-output-visitor.h"
+#include "qapi/qobject-output-visitor.h"
 #include "qapi/qmp/types.h"
 #include "sysemu/block-backend.h"
 #include "qemu/cutils.h"
@@ -691,7 +691,7 @@ void bdrv_image_info_specific_dump(fprintf_function 
func_fprintf, void *f,
ImageInfoSpecific *info_spec)
 {
 QObject *obj, *data;
-Visitor *v = qmp_output_visitor_new(&obj);
+Visitor *v = qobject_output_visitor_new(&obj);
 
 visit_type_ImageInfoSpecific(v, NULL, &info_spec, &error_abort);
 visit_complete(v, &obj);
diff --git a/blockdev.c b/blockdev.c
index 29c6561..814d49f 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -43,7 +43,7 @@
 #include "qapi/qmp/types.h"
 #include "qapi-visit.h"
 #include "qapi/qmp/qerror.h"
-#include "qapi/qmp-output-visitor.h"
+#include "qapi/qobject-output-visitor.h"
 #include "qapi/util.h"
 #include "sysemu/sysemu.h"
 #include "block/block_int.h"
@@ -3828,7 +3828,7 @@ void qmp_blockdev_add(BlockdevOptions *options, Error 
**errp)
 {
 BlockDriverState *bs;
 QObject *obj;
-Visitor *v = qmp_output_visitor_new(&obj);
+Visitor *v = qobject_output_visitor_new(&obj);
 QDict *qdict;
 Error *local_err = NULL;
 
diff --git a/docs/qapi-code-gen.txt b/docs/qapi-code-gen.txt
index d2604b6..2841c51 100644
--- a/docs/qapi-code-gen.txt
+++ b/docs/qapi-code-gen.txt
@@ -1005,7 +1005,7 @@ Example:
 Error *err = NULL;
 Visitor *v;
 
-v = qmp_output_visitor_new(ret_out);
+v = qobject_output_visitor_new(ret_out);
 visit_type_UserDefOne(v, "unused", &ret_in, &err);
 if (!err) {
 visit_complete(v, ret_out);
diff --git a/include/qapi/qmp-output-visitor.h 
b/include/qapi/qobject-output-visitor.h
similarity index 66%
rename from include/qapi/qmp-output-visitor.h
rename to include/qapi/qobject-output-visitor.h
index 040fdda..358c959 100644
--- a/include/qapi/qmp-output-visitor.h
+++ b/include/qapi/qobject-output-visitor.h
@@ -11,20 +11,20 @@
  *
  */
 
-#ifndef QMP_OUTPUT_VISITOR_H
-#define QMP_OUTPUT_VISITOR_H
+#ifndef QOBJECT_OUTPUT_VISITOR_H
+#define QOBJECT_OUTPUT_VISITOR_H
 
 #include "qapi/visitor.h"
 #include "qapi/qmp/qobject.h"
 
-typedef struct QmpOutputVisitor QmpOutputVisitor;
+typedef struct QObjectOutputVisitor QObjectOutputVisitor;
 
 /*
- * Create a new QMP output visitor.
+ * Create a new QOBJECT output visitor.
  *
  * If everything else succeeds, pass @result to visit_complete() to
  * collect the result of the visit.
  */
-Visitor *qmp_output_visitor_new(QObject **result);
+Visitor *qobject_output_visitor_new(QObject **result);
 
 #endif
diff --git a/qapi/Makefile.objs b/qapi/Makefile.objs
index 6ec7bdc..33906ff 100644
--- a/qapi/Makefile.objs
+++ b/qapi/Makefile.objs
@@ -1,5 +1,5 @@
 util-obj-y = qapi-visit-core.o qapi-dealloc-visitor.o qobject-input-visitor.o
-util-obj-y += qmp-output-visitor.o qmp-registry.o qmp-dispatch.o
+util-ob

[Qemu-block] [PATCH v14 20/21] net: convert to QObjectInputVisitor for -net/-netdev parsing

[Daniel P. Berrange]

The -net/-netdev command line parsing code uses OptsVisitor
for parsing options to populate NetLegacy or NetDev struct
respectively. Although those structs have nesting, the
OptsVisitor flattens them, so we must enable compatibility
options to auto-create structs. This allows the legacy
syntax

  -net tap,id=net0,vlan=3,fd=3,script=/bin/ifup-qemu

to be treated as equivalent to the modern QAPI based
syntax

  -net 
id=net0,vlan=3,opts.type=tap,opts.data.fd=3,opts.data.script=/bin/ifup-qemu

Signed-off-by: Daniel P. Berrange 
---
 net/net.c | 18 --
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/net/net.c b/net/net.c
index ec984bf..de6bf8e 100644
--- a/net/net.c
+++ b/net/net.c
@@ -43,7 +43,7 @@
 #include "qemu/iov.h"
 #include "qemu/main-loop.h"
 #include "qapi-visit.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "sysemu/sysemu.h"
 #include "net/filter.h"
 #include "qapi/string-output-visitor.h"
@@ -1069,7 +1069,21 @@ int net_client_init(QemuOpts *opts, bool is_netdev, 
Error **errp)
 void *object = NULL;
 Error *err = NULL;
 int ret = -1;
-Visitor *v = opts_visitor_new(opts);
+/*
+ * Needs autocreate_lists=true in order support existing
+ * syntax for list options where the bare key is repeated
+ *
+ * Needs autocreate_struct_levels=3 in order to deal with
+ * 3 level nesting in NetLegacy option args, which was
+ * exposed as a flat namespace with OptVisitor
+ */
+Visitor *v = qobject_input_visitor_new_opts(opts, true, 3, false, true,
+&err);
+
+if (err) {
+error_propagate(errp, err);
+return -1;
+}
 
 {
 /* Parse convenience option format ip6-net=fec0::0[/64] */
-- 
2.7.4

[Qemu-block] [PATCH v14 06/21] qapi: don't pass two copies of TestInputVisitorData to tests

[Daniel P. Berrange]

The input_visitor_test_add() method was accepting an instance
of 'TestInputVisitorData' and passing it as the 'user_data'
parameter to test functions. The main 'TestInputVisitorData'
instance that was actually used, was meanwhile being allocated
automatically by the test framework fixture setup.

The 'user_data' parameter is going to be needed for tests
added in later patches, so getting rid of the current mistaken
usage now allows this.

Signed-off-by: Daniel P. Berrange 
---
 tests/test-qobject-input-visitor.c | 76 --
 1 file changed, 32 insertions(+), 44 deletions(-)

diff --git a/tests/test-qobject-input-visitor.c 
b/tests/test-qobject-input-visitor.c
index 0e65e63..26c5012 100644
--- a/tests/test-qobject-input-visitor.c
+++ b/tests/test-qobject-input-visitor.c
@@ -747,10 +747,11 @@ static void 
test_visitor_in_native_list_number(TestInputVisitorData *data,
 }
 
 static void input_visitor_test_add(const char *testpath,
-   TestInputVisitorData *data,
-   void (*test_func)(TestInputVisitorData 
*data, const void *user_data))
+   const void *user_data,
+   void (*test_func)(TestInputVisitorData 
*data,
+ const void *user_data))
 {
-g_test_add(testpath, TestInputVisitorData, data, NULL, test_func,
+g_test_add(testpath, TestInputVisitorData, user_data, NULL, test_func,
visitor_input_teardown);
 }
 
@@ -833,77 +834,64 @@ static void 
test_visitor_in_wrong_type(TestInputVisitorData *data,
 
 int main(int argc, char **argv)
 {
-TestInputVisitorData in_visitor_data;
-
 g_test_init(&argc, &argv, NULL);
 
 input_visitor_test_add("/visitor/input/int",
-   &in_visitor_data, test_visitor_in_int);
+   NULL, test_visitor_in_int);
 input_visitor_test_add("/visitor/input/int_overflow",
-   &in_visitor_data, test_visitor_in_int_overflow);
+   NULL, test_visitor_in_int_overflow);
 input_visitor_test_add("/visitor/input/bool",
-   &in_visitor_data, test_visitor_in_bool);
+   NULL, test_visitor_in_bool);
 input_visitor_test_add("/visitor/input/number",
-   &in_visitor_data, test_visitor_in_number);
+   NULL, test_visitor_in_number);
 input_visitor_test_add("/visitor/input/string",
-   &in_visitor_data, test_visitor_in_string);
+   NULL, test_visitor_in_string);
 input_visitor_test_add("/visitor/input/enum",
-   &in_visitor_data, test_visitor_in_enum);
+   NULL, test_visitor_in_enum);
 input_visitor_test_add("/visitor/input/struct",
-   &in_visitor_data, test_visitor_in_struct);
+   NULL, test_visitor_in_struct);
 input_visitor_test_add("/visitor/input/struct-nested",
-   &in_visitor_data, test_visitor_in_struct_nested);
+   NULL, test_visitor_in_struct_nested);
 input_visitor_test_add("/visitor/input/list",
-   &in_visitor_data, test_visitor_in_list);
+   NULL, test_visitor_in_list);
 input_visitor_test_add("/visitor/input/any",
-   &in_visitor_data, test_visitor_in_any);
+   NULL, test_visitor_in_any);
 input_visitor_test_add("/visitor/input/null",
-   &in_visitor_data, test_visitor_in_null);
+   NULL, test_visitor_in_null);
 input_visitor_test_add("/visitor/input/union-flat",
-   &in_visitor_data, test_visitor_in_union_flat);
+   NULL, test_visitor_in_union_flat);
 input_visitor_test_add("/visitor/input/alternate",
-   &in_visitor_data, test_visitor_in_alternate);
+   NULL, test_visitor_in_alternate);
 input_visitor_test_add("/visitor/input/errors",
-   &in_visitor_data, test_visitor_in_errors);
+   NULL, test_visitor_in_errors);
 input_visitor_test_add("/visitor/input/wrong-type",
-   &in_visitor_data, test_visitor_in_wrong_type);
+   NULL, test_visitor_in_wrong_type);
 input_visitor_test_add("/visitor/input/alternate-number",
-   &in_visitor_data, test_visitor_in_alternate_number);
+   NULL, test_visitor_in_alternate_number);
 input_visitor_test_add("/visitor/input/native_list/int",
-   &in_visitor_data,
-   test_visitor_in_native_list_int);
+   NULL, test_visitor_in_native_list_int);

[Qemu-block] [PATCH v14 09/21] qapi: permit auto-creating single element lists

[Daniel P. Berrange]

When converting QemuOpts to a QObject, there is no information
about compound types available, so when visiting a list, the
corresponding QObject is not guaranteed to be a QList. We
therefore need to be able to auto-create a single element QList
from whatever type we find.

This mode should only be enabled if you have compatibility
requirements for

   -arg foo=hello,foo=world

to be treated as equivalent to the preferred syntax:

   -arg foo.0=hello,foo.1=world

Signed-off-by: Daniel P. Berrange 
---
 include/qapi/qobject-input-visitor.h | 20 +++-
 qapi/qobject-input-visitor.c | 27 +--
 tests/test-qobject-input-visitor.c   | 88 +++-
 3 files changed, 117 insertions(+), 18 deletions(-)

diff --git a/include/qapi/qobject-input-visitor.h 
b/include/qapi/qobject-input-visitor.h
index f134d90..1809f48 100644
--- a/include/qapi/qobject-input-visitor.h
+++ b/include/qapi/qobject-input-visitor.h
@@ -42,6 +42,19 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool 
strict);
  * represented as strings. i.e. if visiting a boolean, the value should
  * be a QString whose contents represent a valid boolean.
  *
+ * If @autocreate_list is true, then as an alternative to a normal QList,
+ * list values can be stored as a QString or QDict instead, which will
+ * be interpreted as representing single element lists. This should only
+ * by used if compatibility is required with the OptsVisitor which allowed
+ * repeated keys, without list indexes, to represent lists. e.g. set this
+ * to true if you have compatibility requirements for
+ *
+ *   -arg foo=hello,foo=world
+ *
+ * to be treated as equivalent to the preferred syntax:
+ *
+ *   -arg foo.0=hello,foo.1=world
+ *
  * The visitor always operates in strict mode, requiring all dict keys
  * to be consumed during visitation. An error will be reported if this
  * does not happen.
@@ -49,7 +62,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool strict);
  * The returned input visitor should be released by calling
  * visit_free() when no longer required.
  */
-Visitor *qobject_input_visitor_new_autocast(QObject *obj);
+Visitor *qobject_input_visitor_new_autocast(QObject *obj,
+bool autocreate_list);
 
 
 /**
@@ -64,6 +78,8 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj);
  * The returned input visitor should be released by calling
  * visit_free() when no longer required.
  */
-Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts, Error **errp);
+Visitor *qobject_input_visitor_new_opts(const QemuOpts *opts,
+bool autocreate_list,
+Error **errp);
 
 #endif
diff --git a/qapi/qobject-input-visitor.c b/qapi/qobject-input-visitor.c
index d9269c9..d88e9f9 100644
--- a/qapi/qobject-input-visitor.c
+++ b/qapi/qobject-input-visitor.c
@@ -48,6 +48,10 @@ struct QObjectInputVisitor
 
 /* True to reject parse in visit_end_struct() if unvisited keys remain. */
 bool strict;
+
+/* Whether we can auto-create single element lists when
+ * encountering a non-QList type */
+bool autocreate_list;
 };
 
 static QObjectInputVisitor *to_qiv(Visitor *v)
@@ -108,6 +112,7 @@ static const QListEntry 
*qobject_input_push(QObjectInputVisitor *qiv,
 assert(obj);
 tos->obj = obj;
 tos->qapi = qapi;
+qobject_incref(obj);
 
 if (qiv->strict && qobject_type(obj) == QTYPE_QDICT) {
 h = g_hash_table_new(g_str_hash, g_str_equal);
@@ -147,6 +152,7 @@ static void qobject_input_stack_object_free(StackObject 
*tos)
 if (tos->h) {
 g_hash_table_unref(tos->h);
 }
+qobject_decref(tos->obj);
 
 g_free(tos);
 }
@@ -197,7 +203,7 @@ static void qobject_input_start_list(Visitor *v, const char 
*name,
 QObject *qobj = qobject_input_get_object(qiv, name, true);
 const QListEntry *entry;
 
-if (!qobj || qobject_type(qobj) != QTYPE_QLIST) {
+if (!qobj || (!qiv->autocreate_list && qobject_type(qobj) != QTYPE_QLIST)) 
{
 if (list) {
 *list = NULL;
 }
@@ -206,7 +212,16 @@ static void qobject_input_start_list(Visitor *v, const 
char *name,
 return;
 }
 
-entry = qobject_input_push(qiv, qobj, list, errp);
+if (qobject_type(qobj) != QTYPE_QLIST) {
+QList *tmplist = qlist_new();
+qlist_append_obj(tmplist, qobj);
+qobject_incref(qobj);
+entry = qobject_input_push(qiv, QOBJECT(tmplist), list, errp);
+QDECREF(tmplist);
+} else {
+entry = qobject_input_push(qiv, qobj, list, errp);
+}
+
 if (list) {
 if (entry) {
 *list = g_malloc0(size);
@@ -514,7 +529,8 @@ Visitor *qobject_input_visitor_new(QObject *obj, bool 
strict)
 return &v->visitor;
 }
 
-Visitor *qobject_input_visitor_new_autocast(QObject *obj)
+Visitor *qobject_input_visitor_new_autocast(QObject *obj,
+bool auto

[Qemu-block] [PATCH v14 14/21] qapi: allow repeated opts with qobject_input_visitor_new_opts

[Daniel P. Berrange]

The qobject_input_visitor_new_opts() method gains a new
parameter to control whether it allows repeated option
keys in the input QemuOpts or not.

Signed-off-by: Daniel P. Berrange 
---
 include/qapi/qobject-input-visitor.h | 6 ++
 qapi/qobject-input-visitor.c | 5 -
 2 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/include/qapi/qobject-input-visitor.h 
b/include/qapi/qobject-input-visitor.h
index 242b767..bc5062a 100644
--- a/include/qapi/qobject-input-visitor.h
+++ b/include/qapi/qobject-input-visitor.h
@@ -112,6 +112,11 @@ Visitor *qobject_input_visitor_new_autocast(QObject *obj,
  * qobject_input_visitor_new_autocast() method. See the docs
  * of that method for further details on processing behaviour.
  *
+ * If the @permit_repeated_opts parameter is true, then the input
+ * @opts is allowed to contain repeated keys and they will be
+ * turned into a QList. If it is false, then repeated keys will
+ * result in an error being reported.
+ *
  * The returned input visitor should be released by calling
  * visit_free() when no longer required.
  */
@@ -119,6 +124,7 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts 
*opts,
 bool autocreate_list,
 size_t autocreate_struct_levels,
 bool permit_int_ranges,
+bool permit_repeated_opts,
 Error **errp);
 
 #endif
diff --git a/qapi/qobject-input-visitor.c b/qapi/qobject-input-visitor.c
index 2287d11..5a3872c 100644
--- a/qapi/qobject-input-visitor.c
+++ b/qapi/qobject-input-visitor.c
@@ -753,6 +753,7 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts 
*opts,
 bool autocreate_list,
 size_t autocreate_struct_levels,
 bool permit_int_ranges,
+bool permit_repeated_opts,
 Error **errp)
 {
 QDict *pdict;
@@ -760,7 +761,9 @@ Visitor *qobject_input_visitor_new_opts(const QemuOpts 
*opts,
 Visitor *v = NULL;
 
 pdict = qemu_opts_to_qdict(opts, NULL,
-   QEMU_OPTS_REPEAT_POLICY_LAST,
+   permit_repeated_opts ?
+   QEMU_OPTS_REPEAT_POLICY_ALL :
+   QEMU_OPTS_REPEAT_POLICY_ERROR,
errp);
 if (!pdict) {
 goto cleanup;
-- 
2.7.4

[Qemu-block] [PATCH v14 15/21] qom: support non-scalar properties with -object

[Daniel P. Berrange]

The current -object command line syntax only allows for
creation of objects with scalar properties, or a list
with a fixed scalar element type. Objects which have
properties that are represented as structs in the QAPI
schema cannot be created using -object.

This is a design limitation of the way the OptsVisitor
is written. It simply iterates over the QemuOpts values
as a flat list. The support for lists is enabled by
allowing the same key to be repeated in the opts string.

The QObjectInputVisitor now has functionality that allows
it to replace OptsVisitor, maintaining full backwards
compatibility for previous CLI syntax, while also allowing
use of new syntax for structs.

Thus -object can now support non-scalar properties,
for example the QMP object:

  {
"execute": "object-add",
"arguments": {
  "qom-type": "demo",
  "id": "demo0",
  "parameters": {
"foo": [
  { "bar": "one", "wizz": "1" },
  { "bar": "two", "wizz": "2" }
]
  }
}
  }

Would be creatable via the CLI now using

$QEMU \
  -object demo,id=demo0,\
  foo.0.bar=one,foo.0.wizz=1,\
  foo.1.bar=two,foo.1.wizz=2

Notice that this syntax is intentionally compatible
with that currently used by block drivers. NB the
indentation seen here after line continuations should
not be used in reality, it is just for clarity in this
commit message.

Signed-off-by: Daniel P. Berrange 
---
 qapi/qobject-input-visitor.c |   2 +-
 qom/object_interfaces.c  |  37 -
 tests/check-qom-proplist.c   | 367 ++-
 3 files changed, 397 insertions(+), 9 deletions(-)

diff --git a/qapi/qobject-input-visitor.c b/qapi/qobject-input-visitor.c
index 5a3872c..f1030d5 100644
--- a/qapi/qobject-input-visitor.c
+++ b/qapi/qobject-input-visitor.c
@@ -204,7 +204,7 @@ static void qobject_input_start_struct(Visitor *v, const 
char *name, void **obj,
 *obj = NULL;
 }
 
-if (!qobj && (qiv->struct_level < qiv->autocreate_struct_levels)) {
+if (!qobj && (qiv->struct_level <= qiv->autocreate_struct_levels)) {
 /* Create a new dict that contains all the currently
  * unvisited items */
 if (tos) {
diff --git a/qom/object_interfaces.c b/qom/object_interfaces.c
index fdc406b..88a1e88 100644
--- a/qom/object_interfaces.c
+++ b/qom/object_interfaces.c
@@ -4,7 +4,8 @@
 #include "qemu/module.h"
 #include "qapi-visit.h"
 #include "qapi/qobject-output-visitor.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qobject-input-visitor.h"
+#include "qemu/option.h"
 
 void user_creatable_complete(Object *obj, Error **errp)
 {
@@ -63,12 +64,16 @@ Object *user_creatable_add(const QDict *qdict,
 if (local_err) {
 goto out_visit;
 }
-visit_check_struct(v, &local_err);
+
+obj = user_creatable_add_type(type, id, pdict, v, &local_err);
 if (local_err) {
 goto out_visit;
 }
 
-obj = user_creatable_add_type(type, id, pdict, v, &local_err);
+visit_check_struct(v, &local_err);
+if (local_err) {
+goto out_visit;
+}
 
 out_visit:
 visit_end_struct(v, NULL);
@@ -114,7 +119,7 @@ Object *user_creatable_add_type(const char *type, const 
char *id,
 
 assert(qdict);
 obj = object_new(type);
-visit_start_struct(v, NULL, NULL, 0, &local_err);
+visit_start_struct(v, "props", NULL, 0, &local_err);
 if (local_err) {
 goto out;
 }
@@ -158,14 +163,32 @@ Object *user_creatable_add_opts(QemuOpts *opts, Error 
**errp)
 {
 Visitor *v;
 QDict *pdict;
+QObject *pobj;
 Object *obj = NULL;
 
-v = opts_visitor_new(opts);
-pdict = qemu_opts_to_qdict(opts, NULL, QEMU_OPTS_REPEAT_POLICY_LAST,
+pdict = qemu_opts_to_qdict(opts, NULL,
+   QEMU_OPTS_REPEAT_POLICY_ALL,
&error_abort);
 
-obj = user_creatable_add(pdict, v, errp);
+pobj = qdict_crumple(pdict, true, errp);
+if (!pobj) {
+goto cleanup;
+}
+/*
+ * We need autocreate_list=true + permit_int_ranges=true
+ * in order to maintain compat with OptsVisitor creation
+ * of the 'numa' object which had an int16List property.
+ *
+ * We need autocreate_structs=1, because at the CLI level
+ * we have the object type + properties in the same flat
+ * struct, even though at the QMP level they are nested.
+ */
+v = qobject_input_visitor_new_autocast(pobj, true, 1, true);
+
+obj = user_creatable_add(qobject_to_qdict(pobj), v, errp);
 visit_free(v);
+qobject_decref(pobj);
+ cleanup:
 QDECREF(pdict);
 return obj;
 }
diff --git a/tests/check-qom-proplist.c b/tests/check-qom-proplist.c
index a16cefc..20eb1d1 100644
--- a/tests/check-qom-proplist.c
+++ b/tests/check-qom-proplist.c
@@ -22,7 +22,16 @@
 
 #include "qapi/error.h"
 #include "qom/object.h"
+#include "qom/object_interfaces.h"
 #include "qemu/module.h"
+#include "qapi/visitor.h"
+#include "qom/object_inter

[Qemu-block] [PATCH v14 17/21] numa: convert to use QObjectInputVisitor for -numa

[Daniel P. Berrange]

Switch away from using OptsVisitor to parse the -numa
argument processing. This enables use of the modern
list syntax for specifying CPUs. e.g. the old syntax

  -numa node,nodeid=0,cpus=0-3,cpus=8-11,mem=107

is equivalent to

  -numa node,nodeid=0,cpus.0=0,cpus.1=1,cpus.2=2,cpus.3=3,\
cpus.4=8,cpus.5=9,cpus.6=10,cpus.7=11,mem=107

Furthermore, the cli arg can now follow the QAPI schema
nesting, so the above is equivalent to the canonical
syntax:

  -numa type=node,data.nodeid=0,data.cpus.0=0,data.cpus.1=1,\
data.cpus.2=2,data.cpus.3=3,data.cpus.4=8,data.cpus.5=9,\
data.cpus.6=10,data.cpus.7=11,data.mem=107

A test case is added to cover argument parsing to validate
that both the old and new syntax is correctly handled.

Signed-off-by: Daniel P. Berrange 
---
 include/sysemu/numa_int.h |  11 +
 numa.c|  36 +-
 stubs/Makefile.objs   |   5 ++
 stubs/exec.c  |   6 +++
 stubs/hostmem.c   |  14 ++
 stubs/memory.c|  41 
 stubs/qdev.c  |   8 
 stubs/vl.c|   8 
 stubs/vmstate.c   |   4 ++
 tests/Makefile.include|   2 +
 tests/test-numa.c | 116 ++
 11 files changed, 240 insertions(+), 11 deletions(-)
 create mode 100644 include/sysemu/numa_int.h
 create mode 100644 stubs/exec.c
 create mode 100644 stubs/hostmem.c
 create mode 100644 stubs/memory.c
 create mode 100644 stubs/qdev.c
 create mode 100644 stubs/vl.c
 create mode 100644 tests/test-numa.c

diff --git a/include/sysemu/numa_int.h b/include/sysemu/numa_int.h
new file mode 100644
index 000..93160da
--- /dev/null
+++ b/include/sysemu/numa_int.h
@@ -0,0 +1,11 @@
+#ifndef SYSEMU_NUMA_INT_H
+#define SYSEMU_NUMA_INT_H
+
+#include "sysemu/numa.h"
+
+extern int have_memdevs;
+extern int max_numa_nodeid;
+
+int parse_numa(void *opaque, QemuOpts *opts, Error **errp);
+
+#endif
diff --git a/numa.c b/numa.c
index 6289f46..653ebf1 100644
--- a/numa.c
+++ b/numa.c
@@ -23,14 +23,14 @@
  */
 
 #include "qemu/osdep.h"
-#include "sysemu/numa.h"
+#include "sysemu/numa_int.h"
 #include "exec/cpu-common.h"
 #include "qemu/bitmap.h"
 #include "qom/cpu.h"
 #include "qemu/error-report.h"
 #include "include/exec/cpu-common.h" /* for RAM_ADDR_FMT */
 #include "qapi-visit.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "hw/boards.h"
 #include "sysemu/hostmem.h"
 #include "qmp-commands.h"
@@ -45,10 +45,10 @@ QemuOptsList qemu_numa_opts = {
 .desc = { { 0 } } /* validated with OptsVisitor */
 };
 
-static int have_memdevs = -1;
-static int max_numa_nodeid; /* Highest specified NUMA node ID, plus one.
- * For all nodes, nodeid < max_numa_nodeid
- */
+int have_memdevs = -1;
+int max_numa_nodeid; /* Highest specified NUMA node ID, plus one.
+  * For all nodes, nodeid < max_numa_nodeid
+  */
 int nb_numa_nodes;
 NodeInfo numa_info[MAX_NODES];
 
@@ -189,6 +189,9 @@ static void numa_node_parse(NumaNodeOptions *node, QemuOpts 
*opts, Error **errp)
 if (node->has_mem) {
 uint64_t mem_size = node->mem;
 const char *mem_str = qemu_opt_get(opts, "mem");
+if (!mem_str) {
+mem_str = qemu_opt_get(opts, "data.mem");
+}
 /* Fix up legacy suffix-less format */
 if (g_ascii_isdigit(mem_str[strlen(mem_str) - 1])) {
 mem_size <<= 20;
@@ -211,16 +214,27 @@ static void numa_node_parse(NumaNodeOptions *node, 
QemuOpts *opts, Error **errp)
 max_numa_nodeid = MAX(max_numa_nodeid, nodenr + 1);
 }
 
-static int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
+int parse_numa(void *opaque, QemuOpts *opts, Error **errp)
 {
 NumaOptions *object = NULL;
 Error *err = NULL;
+Visitor *v;
 
-{
-Visitor *v = opts_visitor_new(opts);
-visit_type_NumaOptions(v, NULL, &object, &err);
-visit_free(v);
+/*
+ * Needs autocreate_list=true, permit_int_ranges=true and
+ * permit_repeated_opts=true in order to support existing
+ * syntax for 'cpus' parameter which is an int list.
+ *
+ * Needs autocreate_struct_levels=1, because existing syntax
+ * used a nested struct in the QMP schema with a flat namespace
+ * in the CLI args.
+ */
+v = qobject_input_visitor_new_opts(opts, true, 1, true, true, &err);
+if (err) {
+goto end;
 }
+visit_type_NumaOptions(v, NULL, &object, &err);
+visit_free(v);
 
 if (err) {
 goto end;
diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
index c5850e8..661b48a 100644
--- a/stubs/Makefile.objs
+++ b/stubs/Makefile.objs
@@ -48,3 +48,8 @@ stub-obj-y += iohandler.o
 stub-obj-y += smbios_type_38.o
 stub-obj-y += ipmi.o
 stub-obj-y += pc_madt_cpu_entry.o
+stub-obj-y += vl.o
+stub-obj-y += exec.o
+stub-obj-y += memory.o
+stub-obj-y += hostmem.o
+stub-obj-y +

[Qemu-block] [PATCH v14 18/21] block: convert crypto driver to use QObjectInputVisitor

[Daniel P. Berrange]

The crypto block driver currently uses OptsVisitor to
convert from the block driver open/create options into
QCryptoBlockOpenOptions/QCryptoBlockCreateOptions. This
is easily replaced by use of QObjectInputVisitor with
no need to enable any compatibility options, since the
structs dealt with contain only scalars.

Signed-off-by: Daniel P. Berrange 
---
 block/crypto.c | 14 +++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/block/crypto.c b/block/crypto.c
index 7aa7eb5..12b0b9f 100644
--- a/block/crypto.c
+++ b/block/crypto.c
@@ -23,7 +23,7 @@
 #include "block/block_int.h"
 #include "sysemu/block-backend.h"
 #include "crypto/block.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "qapi-visit.h"
 #include "qapi/error.h"
 
@@ -206,7 +206,11 @@ block_crypto_open_opts_init(QCryptoBlockFormat format,
 ret = g_new0(QCryptoBlockOpenOptions, 1);
 ret->format = format;
 
-v = opts_visitor_new(opts);
+v = qobject_input_visitor_new_opts(opts, false, 0, false, false,
+   &local_err);
+if (local_err) {
+goto out;
+}
 
 visit_start_struct(v, NULL, NULL, 0, &local_err);
 if (local_err) {
@@ -252,7 +256,11 @@ block_crypto_create_opts_init(QCryptoBlockFormat format,
 ret = g_new0(QCryptoBlockCreateOptions, 1);
 ret->format = format;
 
-v = opts_visitor_new(opts);
+v = qobject_input_visitor_new_opts(opts, false, 0, false, false,
+   &local_err);
+if (local_err) {
+goto out;
+}
 
 visit_start_struct(v, NULL, NULL, 0, &local_err);
 if (local_err) {
-- 
2.7.4

[Qemu-block] [PATCH v14 19/21] acpi: convert to QObjectInputVisitor for -acpi parsing

[Daniel P. Berrange]

The -acpi command line option parsing uses the OptsVisitor
currently. This is easily replaced by the QObjectInputVisitor
instead. There is no need to enable any of the compatibility
options, since the AcpiTableOptions QAPI struct only contains
scalar properties.

Signed-off-by: Daniel P. Berrange 
---
 hw/acpi/core.c | 14 +++---
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/hw/acpi/core.c b/hw/acpi/core.c
index e890a5d..67c0abe 100644
--- a/hw/acpi/core.c
+++ b/hw/acpi/core.c
@@ -25,7 +25,7 @@
 #include "hw/acpi/acpi.h"
 #include "hw/nvram/fw_cfg.h"
 #include "qemu/config-file.h"
-#include "qapi/opts-visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "qapi-visit.h"
 #include "qapi-event.h"
 
@@ -237,14 +237,14 @@ void acpi_table_add(const QemuOpts *opts, Error **errp)
 char **cur;
 size_t bloblen = 0;
 char unsigned *blob = NULL;
+Visitor *v;
 
-{
-Visitor *v;
-
-v = opts_visitor_new(opts);
-visit_type_AcpiTableOptions(v, NULL, &hdrs, &err);
-visit_free(v);
+v = qobject_input_visitor_new_opts(opts, false, 0, false, false, &err);
+if (err) {
+goto out;
 }
+visit_type_AcpiTableOptions(v, NULL, &hdrs, &err);
+visit_free(v);
 
 if (err) {
 goto out;
-- 
2.7.4

[Qemu-block] [PATCH v14 21/21] qapi: delete unused OptsVisitor code

[Daniel P. Berrange]

Now that all code has been converted to QObjectInputVisitor's
QemuOpts compatibility mode, there is no longer any reason
to keep OptsVisitor alive.

Signed-off-by: Daniel P. Berrange 
---
 include/qapi/opts-visitor.h |  40 
 qapi/Makefile.objs  |   2 +-
 qapi/opts-visitor.c | 544 
 tests/Makefile.include  |   5 +-
 tests/test-opts-visitor.c   | 268 --
 vl.c|   1 -
 6 files changed, 2 insertions(+), 858 deletions(-)
 delete mode 100644 include/qapi/opts-visitor.h
 delete mode 100644 qapi/opts-visitor.c
 delete mode 100644 tests/test-opts-visitor.c

diff --git a/include/qapi/opts-visitor.h b/include/qapi/opts-visitor.h
deleted file mode 100644
index 6462c96..000
--- a/include/qapi/opts-visitor.h
+++ /dev/null
@@ -1,40 +0,0 @@
-/*
- * Options Visitor
- *
- * Copyright Red Hat, Inc. 2012
- *
- * Author: Laszlo Ersek 
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#ifndef OPTS_VISITOR_H
-#define OPTS_VISITOR_H
-
-#include "qapi/visitor.h"
-#include "qemu/option.h"
-
-/* Inclusive upper bound on the size of any flattened range. This is a safety
- * (= anti-annoyance) measure; wrong ranges should not cause long startup
- * delays nor exhaust virtual memory.
- */
-#define OPTS_VISITOR_RANGE_MAX 65536
-
-typedef struct OptsVisitor OptsVisitor;
-
-/* Contrarily to qemu-option.c::parse_option_number(), OptsVisitor's "int"
- * parser relies on strtoll() instead of strtoull(). Consequences:
- * - string representations of negative numbers yield negative values,
- * - values below INT64_MIN or LLONG_MIN are rejected,
- * - values above INT64_MAX or LLONG_MAX are rejected.
- *
- * The Opts input visitor does not implement support for visiting QAPI
- * alternates, numbers (other than integers), null, or arbitrary
- * QTypes.  It also requires a non-null list argument to
- * visit_start_list().
- */
-Visitor *opts_visitor_new(const QemuOpts *opts);
-
-#endif
diff --git a/qapi/Makefile.objs b/qapi/Makefile.objs
index 33906ff..4b820a7 100644
--- a/qapi/Makefile.objs
+++ b/qapi/Makefile.objs
@@ -1,6 +1,6 @@
 util-obj-y = qapi-visit-core.o qapi-dealloc-visitor.o qobject-input-visitor.o
 util-obj-y += qobject-output-visitor.o qmp-registry.o qmp-dispatch.o
 util-obj-y += string-input-visitor.o string-output-visitor.o
-util-obj-y += opts-visitor.o qapi-clone-visitor.o
+util-obj-y += qapi-clone-visitor.o
 util-obj-y += qmp-event.o
 util-obj-y += qapi-util.o
diff --git a/qapi/opts-visitor.c b/qapi/opts-visitor.c
deleted file mode 100644
index 084f7cc..000
--- a/qapi/opts-visitor.c
+++ /dev/null
@@ -1,544 +0,0 @@
-/*
- * Options Visitor
- *
- * Copyright Red Hat, Inc. 2012-2016
- *
- * Author: Laszlo Ersek 
- *
- * This work is licensed under the terms of the GNU LGPL, version 2.1 or later.
- * See the COPYING.LIB file in the top-level directory.
- *
- */
-
-#include "qemu/osdep.h"
-#include "qapi/error.h"
-#include "qemu/cutils.h"
-#include "qapi/qmp/qerror.h"
-#include "qapi/opts-visitor.h"
-#include "qemu/queue.h"
-#include "qemu/option_int.h"
-#include "qapi/visitor-impl.h"
-
-
-enum ListMode
-{
-LM_NONE, /* not traversing a list of repeated options */
-
-LM_IN_PROGRESS,  /* opts_next_list() ready to be called.
-  *
-  * Generating the next list link will consume the most
-  * recently parsed QemuOpt instance of the repeated
-  * option.
-  *
-  * Parsing a value into the list link will examine the
-  * next QemuOpt instance of the repeated option, and
-  * possibly enter LM_SIGNED_INTERVAL or
-  * LM_UNSIGNED_INTERVAL.
-  */
-
-LM_SIGNED_INTERVAL,  /* opts_next_list() has been called.
-  *
-  * Generating the next list link will consume the most
-  * recently stored element from the signed interval,
-  * parsed from the most recent QemuOpt instance of the
-  * repeated option. This may consume QemuOpt itself
-  * and return to LM_IN_PROGRESS.
-  *
-  * Parsing a value into the list link will store the
-  * next element of the signed interval.
-  */
-
-LM_UNSIGNED_INTERVAL /* Same as above, only for an unsigned interval. */
-};
-
-typedef enum ListMode ListMode;
-
-struct OptsVisitor
-{
-Visitor visitor;
-
-/* Ownership remains with opts_visitor_new()'s caller. */
-const QemuOpts *opts_root;
-
-unsigned depth;
-
-/* Non-null iff depth is positive. Each key is a QemuOpt name. Each valu

Re: [Qemu-block] [PATCH v5] qemu-img: change opening method for the output in dd

[Stefan Hajnoczi]

On Tue, Sep 13, 2016 at 4:35 PM, Stefan Hajnoczi  wrote:
> On Fri, Aug 26, 2016 at 09:19:40AM +0200, Reda Sallahi wrote:
>> -blk2 = img_open(image_opts, out.filename, out_fmt, BDRV_O_RDWR,
>> -false, false);
>> +ret = access(out_filename, F_OK); /* Check if file exists */
>
> The QEMU block layer does not use POSIX file I/O because it needs to
> support protocols like iSCSI and ssh.
>
> The only way to test for the existence of a disk image is by opening it
> with block layer APIs.
>
> Can you just move the img_open() call from the else statement up here
> instead?

Ping

Re: [Qemu-block] [Qemu-devel] [PATCH v14 00/21] QAPI/QOM work for non-scalar object properties

[no-reply]

Hi,

Your series failed automatic build test. Please find the testing commands and
their output below. If you have docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 1475246744-29302-1-git-send-email-berra...@redhat.com
Subject: [Qemu-devel] [PATCH v14 00/21] QAPI/QOM work for non-scalar object 
properties

=== TEST SCRIPT BEGIN ===
#!/bin/bash
set -e
git submodule update --init dtc
# Let docker tests dump environment info
export SHOW_ENV=1
make J=8 docker-test-quick@centos6
make J=8 docker-test-mingw@fedora
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 - [tag update]  
patchew/1475202478-20716-1-git-send-email-zhangchen.f...@cn.fujitsu.com -> 
patchew/1475202478-20716-1-git-send-email-zhangchen.f...@cn.fujitsu.com
Switched to a new branch 'test'
9b89b2d qapi: delete unused OptsVisitor code
7d7d027 net: convert to QObjectInputVisitor for -net/-netdev parsing
f7fd6c1 acpi: convert to QObjectInputVisitor for -acpi parsing
fecaabf block: convert crypto driver to use QObjectInputVisitor
7941b3f numa: convert to use QObjectInputVisitor for -numa
3c6762d hmp: support non-scalar properties with object_add
efb01de qom: support non-scalar properties with -object
3c90b78 qapi: allow repeated opts with qobject_input_visitor_new_opts
81f728f qdict: allow qdict_crumple to accept compound types as values
8353c22 option: allow qemu_opts_to_qdict to merge repeated options
0948fcd qapi: add integer range support for QObjectInputVisitor
6da7117 qapi: permit auto-creating nested structs
1170f4b qapi: permit auto-creating single element lists
2ec3eac qapi: allow QObjectInputVisitor to be created with QemuOpts
b56363f qapi: permit scalar type conversions in QObjectInputVisitor
6fa7850 qapi: don't pass two copies of TestInputVisitorData to tests
d1b24d3 qapi: rename QmpOutputVisitor to QObjectOutputVisitor
794fec6 qapi: rename QmpInputVisitor to QObjectInputVisitor
43b5ede qapi: add trace events for visitor
587d218 qdict: implement a qdict_crumple method for un-flattening a dict
67cbb95 option: make parse_option_bool/number non-static

=== OUTPUT BEGIN ===
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into 'dtc'...
Submodule path 'dtc': checked out '65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf'
  BUILD centos6
  ARCHIVE qemu.tgz
  ARCHIVE dtc.tgz
  COPY RUNNER
  RUN test-quick in centos6
Packages installed:
SDL-devel-1.2.14-7.el6_7.1.x86_64
ccache-3.1.6-2.el6.x86_64
epel-release-6-8.noarch
gcc-4.4.7-17.el6.x86_64
git-1.7.1-4.el6_7.1.x86_64
glib2-devel-2.28.8-5.el6.x86_64
libfdt-devel-1.4.0-1.el6.x86_64
make-3.81-23.el6.x86_64
package g++ is not installed
pixman-devel-0.32.8-1.el6.x86_64
tar-1.23-15.el6_8.x86_64
zlib-devel-1.2.3-29.el6.x86_64

Environment variables:
PACKAGES=libfdt-devel ccache tar git make gcc g++ zlib-devel 
glib2-devel SDL-devel pixman-devel epel-release
HOSTNAME=9320e4514cd6
TERM=xterm
MAKEFLAGS= -j8
HISTSIZE=1000
J=8
USER=root
CCACHE_DIR=/var/tmp/ccache
EXTRA_CONFIGURE_OPTS=
V=
SHOW_ENV=1
MAIL=/var/spool/mail/root
PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
LANG=en_US.UTF-8
TARGET_LIST=
HISTCONTROL=ignoredups
SHLVL=1
HOME=/root
TEST_DIR=/tmp/qemu-test
LOGNAME=root
LESSOPEN=||/usr/bin/lesspipe.sh %s
FEATURES= dtc
DEBUG=
G_BROKEN_FILENAMES=1
CCACHE_HASHDIR=
_=/usr/bin/env

Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu 
--prefix=/tmp/qemu-test/src/tests/docker/install
No C++ compiler available; disabling C++ specific optional code
Install prefix/tmp/qemu-test/src/tests/docker/install
BIOS directory/tmp/qemu-test/src/tests/docker/install/share/qemu
binary directory  /tmp/qemu-test/src/tests/docker/install/bin
library directory /tmp/qemu-test/src/tests/docker/install/lib
module directory  /tmp/qemu-test/src/tests/docker/install/lib/qemu
libexec directory /tmp/qemu-test/src/tests/docker/install/libexec
include directory /tmp/qemu-test/src/tests/docker/install/include
config directory  /tmp/qemu-test/src/tests/docker/install/etc
local state directory   /tmp/qemu-test/src/tests/docker/install/var
Manual directory  /tmp/qemu-test/src/tests/docker/install/share/man
ELF interp prefix /usr/gnemul/qemu-%M
Source path   /tmp/qemu-test/src
C compilercc
Host C compiler   cc
C++ compiler  
Objective-C compiler cc
ARFLAGS   rv
CFLAGS-O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g 
QEMU_CFLAGS   -I/usr/include/pixman-1-pthread -I/usr/include/glib-2.0 
-I/usr/lib64/glib-2.0/include   -fPIE -DPIE -m64 -D_GNU_SOURCE 
-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes 
-Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes 
-fno-strict-aliasing -fno-common -fwrapv  -Wendif-labels -Wmissing-include-dirs 
-Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self 
-Wignored-qualifiers -Wold-st

Re: [Qemu-block] [PATCH v14 03/21] qapi: add trace events for visitor

[Eric Blake]

On 09/30/2016 09:45 AM, Daniel P. Berrange wrote:
> Allow tracing of the operation of visitors
> 
> Signed-off-by: Daniel P. Berrange 
> ---
>  Makefile.objs  |  1 +
>  qapi/qapi-visit-core.c | 27 +++
>  qapi/trace-events  | 33 +
>  3 files changed, 61 insertions(+)
>  create mode 100644 qapi/trace-events

Reviewed-by: Eric Blake 

> +
> +visit_start_alternate(void *v, const char *name, void *obj, size_t size, 
> bool promote_int) "v=%p name=%s obj=%p size=%zu promote_int=%d"

Unrelated to your patch, but should the trace engines be taught how to
honor line wrapping in the trace files?

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

[Qemu-block] Transactions, Jobs, and Cancellation

[John Snow]

Hi Eric (as a proxy for Libvirt);

I want to make a change to transactions such that they do not actually 
start the jobs until the entire transaction is error-checked for validity.


This would be a change from the current setup where:

- Some jobs are started
- One job cannot start
- Existing jobs are cancelled, emitting job events
- QMP transaction returns failure.

to something more like:

- Some jobs are queued to start
- One job cannot start
- Existing queued jobs are un-created
- No events are emitted, but the QMP transaction fails.

If I understand correctly, it's possible for the transaction to fail 
even after it has started several jobs because they begin operating 
during the prepare phase.


Then, because the transaction preparation has failed, QEMU will cancel 
the transaction instead of proceeding, which will generate some 
BLOCK_JOB_CANCELLED events.


This may affect libvirt and others if I change these semantics.

Does that sound appropriate to you, or would you from a libvirt 
perspective RATHER get events for "uncreated" jobs like you do now? I 
could make it do either, but I'd rather prefer to simply not emit events 
for jobs that didn't truly never start.


I can also begin emitting events for jobs that *actually* start if it 
would help to disambiguate the cases between old and new transactions.


Note: This has nothing to do with the transactional-cancel property, 
which only impacts what happens when jobs created by a transaction fail 
AFTER a successful return from qmp_transaction.


--js

[Qemu-block] backup notifier fail policy

[Vladimir Sementsov-Ogievskiy]

Hi all!

Please, can somebody explain me, why we fail guest request in case of io 
error in write notifier? I think guest consistency is more important 
than success of unfinished backup. Or, what am I missing?


I'm saying about this code:

static int coroutine_fn backup_before_write_notify(
NotifierWithReturn *notifier,
void *opaque)
{
BackupBlockJob *job = container_of(notifier, BackupBlockJob, 
before_write);

BdrvTrackedRequest *req = opaque;
int64_t sector_num = req->offset >> BDRV_SECTOR_BITS;
int nb_sectors = req->bytes >> BDRV_SECTOR_BITS;

assert(req->bs == blk_bs(job->common.blk));
assert((req->offset & (BDRV_SECTOR_SIZE - 1)) == 0);
assert((req->bytes & (BDRV_SECTOR_SIZE - 1)) == 0);

return backup_do_cow(job, sector_num, nb_sectors, NULL, true);
}

So, what about something like

ret = backup_do_cow(job, ...
if (ret < 0 && job->notif_ret == 0) {
   job->notif_ret = ret;
}

return 0;

and fail block job if notif_ret < 0 in other places of backup code?

--
Best regards,
Vladimir

Re: [Qemu-block] [PATCH v4 01/12] block/nbd: Drop trailing "." in error messages

[Eric Blake]

On 09/28/2016 03:55 PM, Max Reitz wrote:
> Signed-off-by: Max Reitz 
> ---
>  block/nbd.c   | 4 ++--
>  tests/qemu-iotests/051.out| 4 ++--
>  tests/qemu-iotests/051.pc.out | 4 ++--
>  3 files changed, 6 insertions(+), 6 deletions(-)
> 

Reviewed-by: Eric Blake 

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-block] [PATCH v14 06/21] qapi: don't pass two copies of TestInputVisitorData to tests

[Eric Blake]

On 09/30/2016 09:45 AM, Daniel P. Berrange wrote:
> The input_visitor_test_add() method was accepting an instance
> of 'TestInputVisitorData' and passing it as the 'user_data'
> parameter to test functions. The main 'TestInputVisitorData'
> instance that was actually used, was meanwhile being allocated
> automatically by the test framework fixture setup.
> 
> The 'user_data' parameter is going to be needed for tests
> added in later patches, so getting rid of the current mistaken
> usage now allows this.
> 
> Signed-off-by: Daniel P. Berrange 
> ---
>  tests/test-qobject-input-visitor.c | 76 
> --
>  1 file changed, 32 insertions(+), 44 deletions(-)

Reviewed-by: Eric Blake 

The improved commit message makes the difference :)

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-block] [PATCH v14 07/21] qapi: permit scalar type conversions in QObjectInputVisitor

[Eric Blake]

On 09/30/2016 09:45 AM, Daniel P. Berrange wrote:
> Currently the QObjectInputVisitor assumes that all scalar
> values are directly represented as the final types declared
> by the thing being visited. ie it assumes an 'int' is using
> QInt, and a 'bool' is using QBool, etc.  This is good when
> QObjectInputVisitor is fed a QObject that came from a JSON
> document on the QMP monitor, as it will strictly validate
> correctness.
> 
> To allow QObjectInputVisitor to be reused for visiting
> a QObject originating from QemuOpts, an alternative mode
> is needed where all the scalars types are represented as
> QString and converted on the fly to the final desired
> type.
> 
> Reviewed-by: Kevin Wolf 
> Reviewed-by: Marc-André Lureau 
> Signed-off-by: Daniel P. Berrange 
> ---
>  include/qapi/qobject-input-visitor.h |  32 +-
>  qapi/qobject-input-visitor.c | 132 
>  tests/test-qobject-input-visitor.c   | 194 
> ++-
>  3 files changed, 350 insertions(+), 8 deletions(-)
> 

Reviewed-by: Eric Blake 

-- 
Eric Blake   eblake redhat com+1-919-301-3266
Libvirt virtualization library http://libvirt.org



signature.asc
Description: OpenPGP digital signature