Re: [Qemu-block] [PATCH v2 3/5] raw-posix: DPRINTF instead of DEBUG_BLOCK_PRINT

2015-05-11 Thread Kevin Wolf
Am 08.05.2015 um 19:47 hat Dimitris Aragiorgis geschrieben: Building the QEMU tools fails if we #define DEBUG_BLOCK inside block/raw-posix.c. Here instead of adding qemu-log.o in block-obj-y so that DEBUG_BLOCK_PRINT can be used, we substitute the latter with a simple DPRINTF().

Re: [Qemu-block] [PATCH v2 3/5] raw-posix: DPRINTF instead of DEBUG_BLOCK_PRINT

2015-05-11 Thread Paolo Bonzini
On 11/05/2015 12:16, Kevin Wolf wrote: Am 08.05.2015 um 19:47 hat Dimitris Aragiorgis geschrieben: Building the QEMU tools fails if we #define DEBUG_BLOCK inside block/raw-posix.c. Here instead of adding qemu-log.o in block-obj-y so that DEBUG_BLOCK_PRINT can be used, we substitute the

Re: [Qemu-block] [PATCH v8 00/10] qcow2: Allow refcount_bits amendment

2015-05-11 Thread Stefan Hajnoczi
On Fri, Apr 24, 2015 at 05:00:18PM +0200, Max Reitz wrote: On 18.03.2015 21:56, Max Reitz wrote: This series adds support to qemu for changing the refcount_bits option of an existing qcow2 file through the qemu-img amend command. Originally (up until v7), this series was called qcow2:

Re: [Qemu-block] [Qemu-devel] [PATCH v3 01/10] qapi: Add transaction support to block-dirty-bitmap operations

2015-05-11 Thread Stefan Hajnoczi
On Fri, May 08, 2015 at 08:29:09AM -0600, Eric Blake wrote: On 05/08/2015 07:14 AM, Stefan Hajnoczi wrote: No it doesn't. Actions have to appear atomic to the qmp_transaction caller. Both approaches achieve that so they are both correct in isolation. The ambiguity is whether commit

[Qemu-block] [PATCH 2/8] qcow2: simplify qcow2_cache_put() and qcow2_cache_entry_mark_dirty()

2015-05-11 Thread Alberto Garcia
Since all tables are now stored together, it is possible to obtain the position of a particular table directly from its address, so the operation becomes O(1). Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Stefan Hajnoczi stefa...@redhat.com Reviewed-by: Max Reitz mre...@redhat.com

[Qemu-block] [PATCH 8/8] docs: document how to configure the qcow2 L2/refcount caches

2015-05-11 Thread Alberto Garcia
QEMU has options to configure the size of the L2 and refcount caches for the qcow2 format. However, choosing the right sizes for a particular disk image is not a straightforward operation since the ratio between the cache size and the allocated disk space is not obvious and depends on the size of

[Qemu-block] [PATCH v3 0/8] qcow2 L2/refcount cache improvements

2015-05-11 Thread Alberto Garcia
New version of the qcow2 cache patches: v3: - Removed a dead comment in patch #3 - New document explaining how to configure the cache sizes v2: https://lists.nongnu.org/archive/html/qemu-devel/2015-05/msg00833.html - Don't do pointer arithmetic on void * - Rename table_addr() to

Re: [Qemu-block] [PATCH 8/8] docs: document how to configure the qcow2 L2/refcount caches

2015-05-11 Thread Alberto Garcia
On Mon 11 May 2015 03:23:25 PM CEST, Max Reitz mre...@redhat.com wrote: + disk_size = l2_cache_size * cluster_size / 8 + disk_size = refcount_cache_size * cluster_size / 2 Only with the default of refcount_bits=16. In the general case, it's refcount_cache_size * cluster_size * 8 /

Re: [Qemu-block] [Qemu-devel] [PATCH 01/34] qdict: Add qdict_array_entries()

2015-05-11 Thread Kevin Wolf
Am 08.05.2015 um 22:06 hat Eric Blake geschrieben: On 05/08/2015 11:21 AM, Kevin Wolf wrote: Signed-off-by: Kevin Wolf kw...@redhat.com --- Might want to include mention of what it will be used for in the commit body. You're right. This is the new commit message: This counts the

Re: [Qemu-block] [PATCH v5 2/3] block: Fix NULL deference for unaligned write if qiov is NULL

2015-05-11 Thread Stefan Hajnoczi
On Tue, May 05, 2015 at 10:51:14AM +0800, Fam Zheng wrote: This function is complex. I had to draw a diagram to remember the relationships between the variables. It would be nice to split it if that can be done in a way that makes the code nicer. @@ -1236,13 +1238,39 @@ static int

Re: [Qemu-block] [Qemu-devel] [PATCH v5 3/3] qemu-iotests: Test unaligned sub-block zero write

2015-05-11 Thread Stefan Hajnoczi
On Tue, May 05, 2015 at 10:51:15AM +0800, Fam Zheng wrote: Test zero write in byte range 512~1024 for 4k alignment. Signed-off-by: Fam Zheng f...@redhat.com --- tests/qemu-iotests/033 | 13 + tests/qemu-iotests/033.out | 30 ++ 2 files changed,

Re: [Qemu-block] [PATCH 05/34] block: Use macro for cache option names

2015-05-11 Thread Kevin Wolf
Am 11.05.2015 um 16:40 hat Max Reitz geschrieben: On 08.05.2015 19:21, Kevin Wolf wrote: Signed-off-by: Kevin Wolf kw...@redhat.com --- blockdev.c| 24 include/block/block.h | 8 2 files changed, 20 insertions(+), 12 deletions(-) Any

Re: [Qemu-block] [Qemu-devel] [PATCH 02/34] qdict: Add qdict_{set, copy}_default()

2015-05-11 Thread Kevin Wolf
Am 08.05.2015 um 23:30 hat Eric Blake geschrieben: On 05/08/2015 11:21 AM, Kevin Wolf wrote: In the block layer functions that determine options for a child block device, it's a common pattern to either copy options from the parent's options or to set a default string if the option isn't

Re: [Qemu-block] [PATCH 02/34] qdict: Add qdict_{set, copy}_default()

2015-05-11 Thread Max Reitz
On 08.05.2015 19:21, Kevin Wolf wrote: In the block layer functions that determine options for a child block device, it's a common pattern to either copy options from the parent's options or to set a default string if the option isn't explicitly set yet for the child. Provide convenience

Re: [Qemu-block] [PATCH 05/34] block: Use macro for cache option names

2015-05-11 Thread Max Reitz
On 08.05.2015 19:21, Kevin Wolf wrote: Signed-off-by: Kevin Wolf kw...@redhat.com --- blockdev.c| 24 include/block/block.h | 8 2 files changed, 20 insertions(+), 12 deletions(-) Any reason for not making it part of the BLOCK_OPT_* macros in

Re: [Qemu-block] [Qemu-devel] [PATCH 01/34] qdict: Add qdict_array_entries()

2015-05-11 Thread Eric Blake
On 05/11/2015 08:40 AM, Kevin Wolf wrote: +char indexstr[slen], prefix[slen]; And more dependence on a working C99 compiler, thanks to variable length array (VLA). +size_t snprintf_ret; + +snprintf_ret = snprintf(indexstr, slen, %s%u, subqdict, i); +

Re: [Qemu-block] [Qemu-devel] [PATCH 2/2] block: align bounce buffers to page

2015-05-11 Thread Eric Blake
On 05/11/2015 08:54 AM, Stefan Hajnoczi wrote: On Mon, May 04, 2015 at 04:42:24PM +0300, Denis V. Lunev wrote: @@ -726,7 +727,8 @@ static void raw_refresh_limits(BlockDriverState *bs, Error **errp) raw_probe_alignment(bs, s-fd, errp); bs-bl.min_mem_alignment = s-buf_align; -

Re: [Qemu-block] [PATCH 08/34] block: Add list of children to BlockDriverState

2015-05-11 Thread Max Reitz
On 08.05.2015 19:21, Kevin Wolf wrote: This allows iterating over all children of a given BDS, not only including bs-file and bs-backing_hd, but also driver-specific ones like VMDK extents or Quorum children. Signed-off-by: Kevin Wolf kw...@redhat.com --- block.c | 27

Re: [Qemu-block] [PATCH 09/34] block: Add BlockDriverState.inherits_from

2015-05-11 Thread Max Reitz
On 08.05.2015 19:21, Kevin Wolf wrote: Currently, the block layer assumes that any block node can have only one parent, and if it has a parent, that it inherits some options/flags from this parent. This is not true any more: With references used in block device creation, a single node can be

Re: [Qemu-block] [Qemu-devel] [PATCH v2 3/5] raw-posix: DPRINTF instead of DEBUG_BLOCK_PRINT

2015-05-11 Thread Eric Blake
On 05/08/2015 02:10 PM, Eric Blake wrote: On 05/08/2015 11:47 AM, Dimitris Aragiorgis wrote: Building the QEMU tools fails if we #define DEBUG_BLOCK inside block/raw-posix.c. Here instead of adding qemu-log.o in block-obj-y so that DEBUG_BLOCK_PRINT can be used, we substitute the latter with

Re: [Qemu-block] [Qemu-devel] [PATCH v2 2/6] block: Fix dirty bitmap in bdrv_co_discard

2015-05-11 Thread John Snow
On 05/06/2015 12:52 AM, Fam Zheng wrote: Unsetting dirty globally with discard is not very correct. The discard may zero out sectors (depending on can_write_zeroes_with_unmap), we should replicate this change to destinition side to make sure that the guest sees the same data. Calling

Re: [Qemu-block] [Qemu-devel] [PATCH v2 5/6] qemu-iotests: Add test case for mirror with unmap

2015-05-11 Thread John Snow
On 05/06/2015 12:52 AM, Fam Zheng wrote: This checks that the discard on mirror source that effectively zeroes data is also reflected by the data of target. Signed-off-by: Fam Zheng f...@redhat.com --- tests/qemu-iotests/131 | 59 ++

Re: [Qemu-block] [Qemu-devel] [PATCH v2 4/6] qemu-iotests: Make block job methods common

2015-05-11 Thread John Snow
On 05/06/2015 12:52 AM, Fam Zheng wrote: Signed-off-by: Fam Zheng f...@redhat.com --- tests/qemu-iotests/041| 66 ++- tests/qemu-iotests/iotests.py | 28 ++ 2 files changed, 43 insertions(+), 51 deletions(-) diff --git

Re: [Qemu-block] [Qemu-devel] [PATCH v2 6/6] iotests: Use event_wait in wait_ready

2015-05-11 Thread John Snow
On 05/06/2015 12:52 AM, Fam Zheng wrote: Only poll the specific type of event we are interested in, to avoid stealing events that should be consumed by someone else. Suggested-by: John Snow js...@redhat.com Signed-off-by: Fam Zheng f...@redhat.com --- tests/qemu-iotests/iotests.py | 9

[Qemu-block] [PATCH v4 05/11] block: add transactional callbacks feature

2015-05-11 Thread John Snow
The goal here is to add a new method to transactions that allows developers to specify a callback that will get invoked only once all jobs spawned by a transaction are completed, allowing developers the chance to perform actions conditionally pending complete success, partial failure, or complete

[Qemu-block] [PATCH v4 06/11] block: add refcount to Job object

2015-05-11 Thread John Snow
If we want to get at the job after the life of the job, we'll need a refcount for this object. This may occur for example if we wish to inspect the actions taken by a particular job after a transactional group of jobs runs, and further actions are required. Signed-off-by: John Snow

[Qemu-block] [PATCH v4 09/11] block: drive_backup transaction callback support

2015-05-11 Thread John Snow
This patch actually implements the transactional callback system for the drive_backup action. (1) We manually pick up a reference to the bitmap if present to allow its cleanup to be delayed until after all drive_backup jobs launched by the transaction have fully completed. (2) We create

[Qemu-block] [PATCH v4 01/11] qapi: Add transaction support to block-dirty-bitmap operations

2015-05-11 Thread John Snow
This adds two qmp commands to transactions. block-dirty-bitmap-add allows you to create a bitmap simultaneously alongside a new full backup to accomplish a clean synchronization point. block-dirty-bitmap-clear allows you to reset a bitmap back to as-if it were new, which can also be used

[Qemu-block] [PATCH v4 07/11] block: add delayed bitmap successor cleanup

2015-05-11 Thread John Snow
Allow bitmap successors to carry reference counts. We can in a later patch use this ability to clean up the dirty bitmap according to both the individual job's success and the success of all jobs in the transaction group. The code for cleaning up a bitmap is also moved from backup_run to

[Qemu-block] [PATCH v4 04/11] block: re-add BlkTransactionState

2015-05-11 Thread John Snow
Now that the structure formerly known as BlkTransactionState has been renamed to something sensible (BlkActionState), re-introduce an actual BlkTransactionState that actually manages state for the entire Transaction. In the process, convert the old QSIMPLEQ list of actions into a QTAILQ, to let

[Qemu-block] [PATCH v4 02/11] iotests: add transactional incremental backup test

2015-05-11 Thread John Snow
Test simple usage cases for using transactions to create and synchronize incremental backups. Signed-off-by: John Snow js...@redhat.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Stefan Hajnoczi stefa...@redhat.com --- tests/qemu-iotests/124 | 54

[Qemu-block] [PATCH v4 10/11] iotests: 124 - transactional failure test

2015-05-11 Thread John Snow
Use a transaction to request an incremental backup across two drives. Coerce one of the jobs to fail, and then re-run the transaction. Verify that no bitmap data was lost due to the partial transaction failure. Signed-off-by: John Snow js...@redhat.com Reviewed-by: Max Reitz mre...@redhat.com

[Qemu-block] [PATCH v4 11/11] qmp-commands.hx: Update the supported 'transaction' operations

2015-05-11 Thread John Snow
From: Kashyap Chamarthy kcham...@redhat.com Although the canonical source of reference for QMP commands is qapi-schema.json, for consistency's sake, update qmp-commands.hx to state the list of supported transactionable operations, namely: drive-backup blockdev-backup

[Qemu-block] [PATCH v4 08/11] qmp: Add an implementation wrapper for qmp_drive_backup

2015-05-11 Thread John Snow
We'd like to be able to specify the callback given to backup_start manually in the case of transactions, so split apart qmp_drive_backup into an implementation and a wrapper. Switch drive_backup_prepare to use the new wrapper, but don't overload the callback and closure yet. Signed-off-by: John

[Qemu-block] [PATCH 2/2] block: align bounce buffers to page

2015-05-11 Thread Denis V. Lunev
The following sequence int fd = open(argv[1], O_RDWR | O_CREAT | O_DIRECT, 0644); for (i = 0; i 10; i++) write(fd, buf, 4096); performs 5% better if buf is aligned to 4096 bytes. The difference is quite reliable. On the other hand we do not want at the moment to enforce

[Qemu-block] [PATCH 1/2] block: minimal bounce buffer alignment

2015-05-11 Thread Denis V. Lunev
The patch introduces new concept: minimal memory alignment for bounce buffers. Original so called optimal value is actually minimal required value for aligment. It should be used for validation that the IOVec is properly aligned and bounce buffer is not required. Though, from the performance

[Qemu-block] [PATCH v6 0/2] block: enforce minimal 4096 alignment in qemu_blockalign

2015-05-11 Thread Denis V. Lunev
I have used the following program to test #define _GNU_SOURCE #include stdio.h #include unistd.h #include fcntl.h #include sys/types.h #include malloc.h #include string.h int main(int argc, char *argv[]) { int fd = open(argv[1], O_RDWR | O_CREAT | O_DIRECT, 0644); void *buf; int i =

Re: [Qemu-block] [PATCH 11/34] block: Allow references for backing files

2015-05-11 Thread Max Reitz
On 08.05.2015 19:21, Kevin Wolf wrote: For bs-file, using references to existing BDSes has been possible for a while already. This patch enables the same for bs-backing_hd. Signed-off-by: Kevin Wolf kw...@redhat.com --- block.c | 42 --

Re: [Qemu-block] [PATCH 12/34] block: Allow specifying driver-specific options to reopen

2015-05-11 Thread Max Reitz
On 08.05.2015 19:21, Kevin Wolf wrote: Signed-off-by: Kevin Wolf kw...@redhat.com --- block.c | 42 +++--- block/commit.c| 4 ++-- include/block/block.h | 4 +++- 3 files changed, 44 insertions(+), 6 deletions(-) diff --git

Re: [Qemu-block] [PATCH v5 0/2] block: enforce minimal 4096 alignment in qemu_blockalign

2015-05-11 Thread Denis V. Lunev
On 11/05/15 18:08, Stefan Hajnoczi wrote: On Mon, May 04, 2015 at 04:42:22PM +0300, Denis V. Lunev wrote: The difference is quite reliable and the same 5%. qemu-io -n -c 'write -P 0xaa 0 1G' 1.img for image in qcow2 format is 1% faster. I looked a little at the qemu-io invocation but am not

Re: [Qemu-block] [PATCH v5 0/2] block: enforce minimal 4096 alignment in qemu_blockalign

2015-05-11 Thread Denis V. Lunev
On 11/05/15 19:07, Denis V. Lunev wrote: On 11/05/15 18:08, Stefan Hajnoczi wrote: On Mon, May 04, 2015 at 04:42:22PM +0300, Denis V. Lunev wrote: The difference is quite reliable and the same 5%. qemu-io -n -c 'write -P 0xaa 0 1G' 1.img for image in qcow2 format is 1% faster. I looked a