Re: [PATCH 1/2] iotests: Drop readarray from _do_filter_img_create

2020-07-10 Thread Alex Bennée
Max Reitz writes: > Some systems where we run tests on do not have a 4.x bash, so they do > not have readarray. While it looked a bit nicer than messing with > `head` and `tail`, we do not really need it, so we might as well not use > it. I've fixed the cirrus build failure by brew

Re: [PATCH v11 00/34] Add subcluster allocation to qcow2

2020-07-10 Thread Alberto Garcia
On Fri 10 Jul 2020 06:43:59 PM CEST, no-re...@patchew.org wrote: > /tmp/qemu-test/src/block/qcow2-cluster.c:473:19: error: 'expected_type' may > be used uninitialized in this function [-Werror=maybe-uninitialized] > } else if (type != expected_type) { >^ >

Re: [PATCH v7 17/47] block: Re-evaluate backing file handling in reopen

2020-07-10 Thread Andrey Shinkevich
On 25.06.2020 18:21, Max Reitz wrote: Reopening a node's backing child needs a bit of special handling because the "backing" child has different defaults than all other children (among other things). Adding filter support here is a bit more difficult than just using the child access functions.

[PATCH] vvfat: set status to odd fixes

2020-07-10 Thread P J P
From: Prasad J Pandit Virtual VFAT driver is quite old and rarely used. Set its status to Odd Fixes. Signed-off-by: Prasad J Pandit --- MAINTAINERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MAINTAINERS b/MAINTAINERS index 6aa54f7f8f..2a15a3987e 100644 ---

Re: [PATCH v5 5/5] vhost-user-blk: default num_queues to -smp N

2020-07-10 Thread Raphael Norwitz
On Fri, Jul 10, 2020 at 5:53 AM Stefan Hajnoczi wrote: > > On Thu, Jul 09, 2020 at 11:02:24AM -0700, Raphael Norwitz wrote: > > On Mon, Jul 6, 2020 at 7:00 AM Stefan Hajnoczi wrote: > > > diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c > > > index a00b854736..39aec42dae 100644

Re: [PATCH v7 14/47] stream: Deal with filters

2020-07-10 Thread Andrey Shinkevich
On 10.07.2020 18:24, Max Reitz wrote: On 09.07.20 16:52, Andrey Shinkevich wrote: On 25.06.2020 18:21, Max Reitz wrote: Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node

Re: [PATCH v11 00/34] Add subcluster allocation to qcow2

2020-07-10 Thread no-reply
Patchew URL: https://patchew.org/QEMU/cover.1594396418.git.be...@igalia.com/ Hi, This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally. === TEST SCRIPT BEGIN ===

[PATCH 1/2] block/block-backend: add converter from BlockAcctStats to BlockBackend

2020-07-10 Thread Denis V. Lunev
Right now BlockAcctStats is always reside on BlockBackend. This structure is not used in any other place. Thus we are able to create a converter from one pointer to another. Signed-off-by: Denis V. Lunev CC: Vladimir Sementsov-Ogievskiy CC: Kevin Wolf CC: Max Reitz --- block/block-backend.c

[PATCH 2/2] block: add logging facility for long standing IO requests

2020-07-10 Thread Denis V. Lunev
There are severe delays with IO requests processing if QEMU is running in virtual machine or over software defined storage. Such delays potentially results in unpredictable guest behavior. For example, guests over IDE or SATA drive could remount filesystem read-only if write is performed longer

[PATCH 0/2] block: add logging facility for long standing IO requests

2020-07-10 Thread Denis V. Lunev
There are severe delays with IO requests processing if QEMU is running in virtual machine or over software defined storage. Such delays potentially results in unpredictable guest behavior. For example, guests over IDE or SATA drive could remount filesystem read-only if write is performed longer

Re: [PATCH 0/2] iotests: More _filter_img_create fixes

2020-07-10 Thread John Snow
On 7/10/20 12:32 PM, Max Reitz wrote: > Hi, > > I’m sorry. > > John, could I ask you to test whether this series fixes the problems > you’re seeing? > This is based on kwolf/block, I see. By the time you return to reading work email, this link will have information for you:

Re: [PATCH v11 00/34] Add subcluster allocation to qcow2

2020-07-10 Thread no-reply
Patchew URL: https://patchew.org/QEMU/cover.1594396418.git.be...@igalia.com/ Hi, This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally. === TEST SCRIPT BEGIN ===

Re: [PATCH 2/2] iotests: Set LC_ALL=C for sort

2020-07-10 Thread Eric Blake
On 7/10/20 11:32 AM, Max Reitz wrote: Otherwise the result is basically unpredictable. (Note that the precise environment variable to control sorting order is LC_COLLATE, but LC_ALL overrides LC_COLLATE, and we do not want the sorting order to be messed up if LC_ALL is set in the environment.)

[PATCH 1/2] iotests: Drop readarray from _do_filter_img_create

2020-07-10 Thread Max Reitz
Some systems where we run tests on do not have a 4.x bash, so they do not have readarray. While it looked a bit nicer than messing with `head` and `tail`, we do not really need it, so we might as well not use it. Reported-by: Claudio Fontana Signed-off-by: Max Reitz ---

Re: [PATCH 1/2] iotests: Drop readarray from _do_filter_img_create

2020-07-10 Thread Eric Blake
On 7/10/20 11:32 AM, Max Reitz wrote: Some systems where we run tests on do not have a 4.x bash, so they do not have readarray. While it looked a bit nicer than messing with `head` and `tail`, we do not really need it, so we might as well not use it. Reported-by: Claudio Fontana

[PATCH] fixup! qemu-img: Deprecate use of -b without -F

2020-07-10 Thread Eric Blake
Port to BSD truncate. Signed-off-by: Eric Blake --- tests/qemu-iotests/114 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/qemu-iotests/114 b/tests/qemu-iotests/114 index 143683381334..d0609c499388 100755 --- a/tests/qemu-iotests/114 +++ b/tests/qemu-iotests/114 @@

[PATCH 2/2] iotests: Set LC_ALL=C for sort

2020-07-10 Thread Max Reitz
Otherwise the result is basically unpredictable. (Note that the precise environment variable to control sorting order is LC_COLLATE, but LC_ALL overrides LC_COLLATE, and we do not want the sorting order to be messed up if LC_ALL is set in the environment.) Reported-by: John Snow Signed-off-by:

[PATCH v11 31/34] qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit

2020-07-10 Thread Alberto Garcia
Now that the implementation of subclusters is complete we can finally add the necessary options to create and read images with this feature, which we call "extended L2 entries". Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- qapi/block-core.json |

[PATCH 0/2] iotests: More _filter_img_create fixes

2020-07-10 Thread Max Reitz
Hi, I’m sorry. John, could I ask you to test whether this series fixes the problems you’re seeing? Max Reitz (2): iotests: Drop readarray from _do_filter_img_create iotests: Set LC_ALL=C for sort tests/qemu-iotests/common.filter | 12 ++-- 1 file changed, 6 insertions(+), 6

[PATCH v11 21/34] qcow2: Add subcluster support to qcow2_get_host_offset()

2020-07-10 Thread Alberto Garcia
The logic of this function remains pretty much the same, except that it uses count_contiguous_subclusters(), which combines the logic of count_contiguous_clusters() / count_contiguous_clusters_unallocated() and checks individual subclusters. qcow2_cluster_to_subcluster_type() is not necessary as

[PATCH v11 34/34] iotests: Add tests for qcow2 images with extended L2 entries

2020-07-10 Thread Alberto Garcia
Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz --- tests/qemu-iotests/271 | 901 + tests/qemu-iotests/271.out | 726 ++ tests/qemu-iotests/group | 1 + 3 files changed, 1628 insertions(+) create mode 100755

[PATCH v11 28/34] qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()

2020-07-10 Thread Alberto Garcia
This works now at the subcluster level and pwrite_zeroes_alignment is updated accordingly. qcow2_cluster_zeroize() is turned into qcow2_subcluster_zeroize() with the following changes: - The request can now be subcluster-aligned. - The cluster-aligned body of the request is still zeroized

[PATCH v11 26/34] qcow2: Clear the L2 bitmap when allocating a compressed cluster

2020-07-10 Thread Alberto Garcia
Compressed clusters always have the bitmap part of the extended L2 entry set to 0. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz --- block/qcow2-cluster.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 5bdbb65e7b..6b5d0698d0

[PATCH v11 30/34] qcow2: Add prealloc field to QCowL2Meta

2020-07-10 Thread Alberto Garcia
This field allows us to indicate that the L2 metadata update does not come from a write request with actual data but from a preallocation request. For traditional images this does not make any difference, but for images with extended L2 entries this means that the clusters are allocated normally

[PATCH v11 33/34] qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters

2020-07-10 Thread Alberto Garcia
This function is only used by qcow2_expand_zero_clusters() to downgrade a qcow2 image to a previous version. This would require transforming all extended L2 entries into normal L2 entries but this is not a simple task and there are no plans to implement this at the moment. Signed-off-by: Alberto

[PATCH v11 17/34] qcow2: Add cluster type parameter to qcow2_get_host_offset()

2020-07-10 Thread Alberto Garcia
This function returns an integer that can be either an error code or a cluster type (a value from the QCow2ClusterType enum). We are going to start using subcluster types instead of cluster types in some functions so it's better to use the exact data types instead of integers for clarity and in

[PATCH v11 16/34] qcow2: Add qcow2_cluster_is_allocated()

2020-07-10 Thread Alberto Garcia
This helper function tells us if a cluster is allocated (that is, there is an associated host offset for it). Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block/qcow2.h | 6 ++ 1 file changed, 6 insertions(+) diff --git a/block/qcow2.h b/block/qcow2.h

Re: [PULL v2] Block layer patches

2020-07-10 Thread Eric Blake
On 7/9/20 7:17 AM, Kevin Wolf wrote: The following changes since commit 8796c64ecdfd34be394ea277a53df0c76996: Merge remote-tracking branch 'remotes/kraxel/tags/audio-20200706-pull-request' into staging (2020-07-08 16:33:59 +0100) are available in the Git repository at:

[PATCH v11 29/34] qcow2: Add subcluster support to qcow2_measure()

2020-07-10 Thread Alberto Garcia
Extended L2 entries are bigger than normal L2 entries so this has an impact on the amount of metadata needed for a qcow2 file. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz --- block/qcow2.c | 20 +--- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git

[PATCH v11 27/34] qcow2: Add subcluster support to handle_alloc_space()

2020-07-10 Thread Alberto Garcia
The bdrv_co_pwrite_zeroes() call here fills complete clusters with zeroes, but it can happen that some subclusters are not part of the write request or the copy-on-write. This patch makes sure that only the affected subclusters are overwritten. A potential improvement would be to also fill with

[PATCH v11 23/34] qcow2: Add subcluster support to discard_in_l2_slice()

2020-07-10 Thread Alberto Garcia
Two things need to be taken into account here: 1) With full_discard == true the L2 entry must be cleared completely. This also includes the L2 bitmap if the image has extended L2 entries. 2) With full_discard == false we have to make the discarded cluster read back as zeroes. With

[PATCH v11 13/34] qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()

2020-07-10 Thread Alberto Garcia
Extended L2 entries are 128-bit wide: 64 bits for the entry itself and 64 bits for the subcluster allocation bitmap. In order to support them correctly get/set_l2_entry() need to be updated so they take the entry width into account in order to calculate the correct offset. This patch also adds

[PATCH v11 25/34] qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()

2020-07-10 Thread Alberto Garcia
The L2 bitmap needs to be updated after each write to indicate what new subclusters are now allocated. This needs to happen even if the cluster was already allocated and the L2 entry was otherwise valid. In some cases however a write operation doesn't need change the L2 bitmap (because all

[PATCH v11 19/34] qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC

2020-07-10 Thread Alberto Garcia
When dealing with subcluster types there is a new value called QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC that has no equivalent in QCow2ClusterType. This patch handles that value in all places where subcluster types are processed. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz Reviewed-by:

[PATCH v11 04/34] qcow2: Split cluster_needs_cow() out of count_cow_clusters()

2020-07-10 Thread Alberto Garcia
We are going to need it in other places. Signed-off-by: Alberto Garcia Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- block/qcow2-cluster.c | 34 +++--- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/block/qcow2-cluster.c

[PATCH v11 12/34] qcow2: Add l2_entry_size()

2020-07-10 Thread Alberto Garcia
qcow2 images with subclusters have 128-bit L2 entries. The first 64 bits contain the same information as traditional images and the last 64 bits form a bitmap with the status of each individual subcluster. Because of that we cannot assume that L2 entries are sizeof(uint64_t) anymore. This

[PATCH v11 24/34] qcow2: Add subcluster support to check_refcounts_l2()

2020-07-10 Thread Alberto Garcia
The offset field of an uncompressed cluster's L2 entry must be aligned to the cluster size, otherwise it is invalid. If the cluster has no data then it means that the offset points to a preallocation, so we can clear the offset field without affecting the guest-visible data. This is what 'qemu-img

[PATCH v11 22/34] qcow2: Add subcluster support to zero_in_l2_slice()

2020-07-10 Thread Alberto Garcia
The QCOW_OFLAG_ZERO bit that indicates that a cluster reads as zeroes is only used in standard L2 entries. Extended L2 entries use individual 'all zeroes' bits for each subcluster. This must be taken into account when updating the L2 entry and also when deciding that an existing entry does not

[PATCH v11 32/34] qcow2: Allow preallocation and backing files if extended_l2 is set

2020-07-10 Thread Alberto Garcia
Traditional qcow2 images don't allow preallocation if a backing file is set. This is because once a cluster is allocated there is no way to tell that its data should be read from the backing file. Extended L2 entries have individual allocation bits for each subcluster, and therefore it is

[PATCH v11 09/34] qcow2: Add subcluster-related fields to BDRVQcow2State

2020-07-10 Thread Alberto Garcia
This patch adds the following new fields to BDRVQcow2State: - subclusters_per_cluster: Number of subclusters in a cluster - subcluster_size: The size of each subcluster, in bytes - subcluster_bits: No. of bits so 1 << subcluster_bits = subcluster_size Images without subclusters are treated as if

[PATCH v11 06/34] qcow2: Add get_l2_entry() and set_l2_entry()

2020-07-10 Thread Alberto Garcia
The size of an L2 entry is 64 bits, but if we want to have subclusters we need extended L2 entries. This means that we have to access L2 tables and slices differently depending on whether an image has extended L2 entries or not. This patch replaces all l2_slice[] accesses with calls to

[PATCH v11 05/34] qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()

2020-07-10 Thread Alberto Garcia
When writing to a qcow2 file there are two functions that take a virtual offset and return a host offset, possibly allocating new clusters if necessary: - handle_copied() looks for normal data clusters that are already allocated and have a reference count of 1. In those clusters we

[PATCH v11 07/34] qcow2: Document the Extended L2 Entries feature

2020-07-10 Thread Alberto Garcia
Subcluster allocation in qcow2 is implemented by extending the existing L2 table entries and adding additional information to indicate the allocation status of each subcluster. This patch documents the changes to the qcow2 format and how they affect the calculation of the L2 cache size.

[PATCH v11 18/34] qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*

2020-07-10 Thread Alberto Garcia
In order to support extended L2 entries some functions of the qcow2 driver need to start dealing with subclusters instead of clusters. qcow2_get_host_offset() is modified to return the subcluster type instead of the cluster type, and all callers are updated to replace all values of

[PATCH v11 11/34] qcow2: Add offset_into_subcluster() and size_to_subclusters()

2020-07-10 Thread Alberto Garcia
Like offset_into_cluster() and size_to_clusters(), but for subclusters. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block/qcow2.h | 10 ++ 1 file changed, 10 insertions(+) diff --git a/block/qcow2.h b/block/qcow2.h index 2503374677..4fe31adfd3

[PATCH v11 00/34] Add subcluster allocation to qcow2

2020-07-10 Thread Alberto Garcia
Hi, here's the new version of the patches to add subcluster allocation support to qcow2. Please refer to the cover letter of the first version for a full description of the patches: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html This version is rebased on top of the

[PATCH v11 10/34] qcow2: Add offset_to_sc_index()

2020-07-10 Thread Alberto Garcia
For a given offset, return the subcluster number within its cluster (i.e. with 32 subclusters per cluster it returns a number between 0 and 31). Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/qcow2.h | 5 + 1 file changed, 5

[PATCH v11 14/34] qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()

2020-07-10 Thread Alberto Garcia
This patch adds QCow2SubclusterType, which is the subcluster-level version of QCow2ClusterType. All QCOW2_SUBCLUSTER_* values have the the same meaning as their QCOW2_CLUSTER_* equivalents (when they exist). See below for details and caveats. In images without extended L2 entries clusters are

[PATCH v11 20/34] qcow2: Add subcluster support to calculate_l2_meta()

2020-07-10 Thread Alberto Garcia
If an image has subclusters then there are more copy-on-write scenarios that we need to consider. Let's say we have a write request from the middle of subcluster #3 until the end of the cluster: 1) If we are writing to a newly allocated cluster then we need copy-on-write. The previous contents

[PATCH v11 03/34] qcow2: Add calculate_l2_meta()

2020-07-10 Thread Alberto Garcia
handle_alloc() creates a QCowL2Meta structure in order to update the image metadata and perform the necessary copy-on-write operations. This patch moves that code to a separate function so it can be used from other places. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz ---

[PATCH v11 15/34] qcow2: Add qcow2_get_subcluster_range_type()

2020-07-10 Thread Alberto Garcia
There are situations in which we want to know how many contiguous subclusters of the same type there are in a given cluster. This can be done by simply iterating over the subclusters and repeatedly calling qcow2_get_subcluster_type() for each one of them. However once we determined the type of a

[PATCH v11 01/34] qcow2: Make Qcow2AioTask store the full host offset

2020-07-10 Thread Alberto Garcia
The file_cluster_offset field of Qcow2AioTask stores a cluster-aligned host offset. In practice this is not very useful because all users(*) of this structure need the final host offset into the cluster, which they calculate using host_offset = file_cluster_offset + offset_into_cluster(s,

[PATCH v11 08/34] qcow2: Add dummy has_subclusters() function

2020-07-10 Thread Alberto Garcia
This function will be used by the qcow2 code to check if an image has subclusters or not. At the moment this simply returns false. Once all patches needed for subcluster support are ready then QEMU will be able to create and read images with subclusters and this function will return the actual

[PATCH v11 02/34] qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()

2020-07-10 Thread Alberto Garcia
qcow2_get_cluster_offset() takes an (unaligned) guest offset and returns the (aligned) offset of the corresponding cluster in the qcow2 image. In practice none of the callers need to know where the cluster starts so this patch makes the function calculate and return the final host offset

Re: [PATCH for-5.1] file-posix: Mitigate file fragmentation with extent size hints

2020-07-10 Thread Max Reitz
On 07.07.20 18:17, Kevin Wolf wrote: > Am 07.07.2020 um 16:23 hat Kevin Wolf geschrieben: >> Espeically when O_DIRECT is used with image files so that the page cache >> indirection can't cause a merge of allocating requests, the file will >> fragment on the file system layer, with a potentially

Re: [PULL v2] Block layer patches

2020-07-10 Thread Eric Blake
On 7/10/20 10:41 AM, Peter Maydell wrote: qemu-img: Deprecate use of -b without -F (2020-07-09 14:14:55 +0200) Block layer patches: - file-posix: Mitigate file fragmentation with extent size hints - Tighten qemu-img rules on

Re: [PATCH] block: Raise an error when backing file parameter is an empty string

2020-07-10 Thread Connor Kuehl
Ping On 7/1/20 3:42 PM, Connor Kuehl wrote: Hi Kevin & Max, Just pinging this patch for your consideration. Thank you, Connor On 6/17/20 11:27 AM, Connor Kuehl wrote: Providing an empty string for the backing file parameter like so: qemu-img create -f qcow2 -b '' /tmp/foo allows the

Re: [PULL 03/31] iotests: Make _filter_img_create more active

2020-07-10 Thread John Snow
On 7/6/20 6:04 AM, Max Reitz wrote: > Right now, _filter_img_create just filters out everything that looks > format-dependent, and applies some filename filters. That means that we > have to add another filter line every time some format gets a new > creation option. This can be avoided by

Re: [PATCH v7 16/47] block: Use bdrv_cow_child() in bdrv_co_truncate()

2020-07-10 Thread Andrey Shinkevich
On 25.06.2020 18:21, Max Reitz wrote: The condition modified here is not about potentially filtered children, but only about COW sources (i.e. traditional backing files). Signed-off-by: Max Reitz --- block/io.c | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git

Re: [PULL v2] Block layer patches

2020-07-10 Thread Peter Maydell
On Thu, 9 Jul 2020 at 13:17, Kevin Wolf wrote: > > The following changes since commit 8796c64ecdfd34be394ea277a53df0c76996: > > Merge remote-tracking branch > 'remotes/kraxel/tags/audio-20200706-pull-request' into staging (2020-07-08 > 16:33:59 +0100) > > are available in the Git

Re: [PATCH v7 15/47] block: Use CAFs when working with backing chains

2020-07-10 Thread Andrey Shinkevich
On 25.06.2020 18:21, Max Reitz wrote: Use child access functions when iterating through backing chains so filters do not break the chain. In addition, bdrv_find_overlay() will now always return the actual overlay; that is, it will never return a filter node but only one with a COW backing file

Re: [PATCH v7 14/47] stream: Deal with filters

2020-07-10 Thread Max Reitz
On 09.07.20 17:13, Andrey Shinkevich wrote: > On 25.06.2020 18:21, Max Reitz wrote: >> Because of the (not so recent anymore) changes that make the stream job >> independent of the base node and instead track the node above it, we >> have to split that "bottom" node into two cases: The bottom COW

Re: [PATCH v7 14/47] stream: Deal with filters

2020-07-10 Thread Max Reitz
On 09.07.20 16:52, Andrey Shinkevich wrote: > On 25.06.2020 18:21, Max Reitz wrote: >> Because of the (not so recent anymore) changes that make the stream job >> independent of the base node and instead track the node above it, we >> have to split that "bottom" node into two cases: The bottom COW

Re: [PATCH for-5.1 0/2] qemu-img convert: Fix abort with unaligned image size

2020-07-10 Thread no-reply
Patchew URL: https://patchew.org/QEMU/20200710142149.40962-1-kw...@redhat.com/ Hi, This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally. === TEST SCRIPT BEGIN ===

Re: [PATCH 6/6] block/io: improve loadvm performance

2020-07-10 Thread Vladimir Sementsov-Ogievskiy
09.07.2020 16:26, Denis V. Lunev wrote: This patch creates intermediate buffer for reading from block driver state and performs read-ahead to this buffer. Snapshot code performs reads sequentially and thus we know what offsets will be required and when they will become not needed. Results are

Re: [PATCH for-5.1 2/2] file-posix: Allow byte-aligned O_DIRECT with NFS

2020-07-10 Thread Eric Blake
On 7/10/20 9:21 AM, Kevin Wolf wrote: Since commit a6b257a08e3 ('file-posix: Handle undetectable alignment'), we assume that if we open a file with O_DIRECT and alignment probing returns 1, we just couldn't find out the real alignment requirement because some filesystems make the requirement

Re: [PATCH for-5.1 1/2] block: Require aligned image size to avoid assertion failure

2020-07-10 Thread Eric Blake
On 7/10/20 9:21 AM, Kevin Wolf wrote: Unaligned requests will automatically be aligned to bl.request_alignment and we don't want to extend requests to access space beyond the end of the image, so it's required that the image size is aligned. Yep, that's what I've already done on nbd images.

[PATCH for-5.1 2/2] file-posix: Allow byte-aligned O_DIRECT with NFS

2020-07-10 Thread Kevin Wolf
Since commit a6b257a08e3 ('file-posix: Handle undetectable alignment'), we assume that if we open a file with O_DIRECT and alignment probing returns 1, we just couldn't find out the real alignment requirement because some filesystems make the requirement only for allocated blocks. In this case, a

[PATCH for-5.1 0/2] qemu-img convert: Fix abort with unaligned image size

2020-07-10 Thread Kevin Wolf
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1834646 Patch 1 fixes the assertion failure by failing gracefully when opening an image whose size isn't aligned to the required request alignment. Patch 2 relaxes the restrictions for NFS, which actually supports byte alignment, but incorrectly

[PATCH for-5.1 1/2] block: Require aligned image size to avoid assertion failure

2020-07-10 Thread Kevin Wolf
Unaligned requests will automatically be aligned to bl.request_alignment and we don't want to extend requests to access space beyond the end of the image, so it's required that the image size is aligned. With write requests, this could cause assertion failures like this if RESIZE permissions

[PATCH v1] qmp: don't hold ctx lock while querying blockstats

2020-07-10 Thread Zhenyu Ye
Because the QMP command runs in the main thread, and changes to the aio context of iothread will only be executed in the main thread (they will not be in parallel), so there is no need a lock protection while querying blockstats. If we hold the lock here, while the I/O pressure is high in vm and

Re: [PATCH v5 5/5] vhost-user-blk: default num_queues to -smp N

2020-07-10 Thread Stefan Hajnoczi
On Thu, Jul 09, 2020 at 11:02:24AM -0700, Raphael Norwitz wrote: > On Mon, Jul 6, 2020 at 7:00 AM Stefan Hajnoczi wrote: > > diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c > > index a00b854736..39aec42dae 100644 > > --- a/hw/block/vhost-user-blk.c > > +++

Re: [PATCH for-5.1] qemu-img resize: Require --shrink for shrinking all image formats

2020-07-10 Thread Kevin Wolf
Am 10.07.2020 um 14:33 hat Peter Maydell geschrieben: > On Fri, 10 Jul 2020 at 13:17, Kevin Wolf wrote: > > > > QEMU 2.11 introduced the --shrink option for qemu-img resize to avoid > > accidentally shrinking images (commit 4ffca8904a3). However, for > > compatibility reasons, it was not enforced

Re: [PATCH for-5.1] qemu-img resize: Require --shrink for shrinking all image formats

2020-07-10 Thread Peter Maydell
On Fri, 10 Jul 2020 at 13:17, Kevin Wolf wrote: > > QEMU 2.11 introduced the --shrink option for qemu-img resize to avoid > accidentally shrinking images (commit 4ffca8904a3). However, for > compatibility reasons, it was not enforced for raw images yet, but only > a deprecation warning was

Re: [PATCH 2/2] hw/sd/sdcard: Do not allow invalid SD card sizes

2020-07-10 Thread Peter Maydell
On Fri, 10 Jul 2020 at 13:07, Kevin Wolf wrote: > > Am 10.07.2020 um 11:59 hat Peter Maydell geschrieben: > > On Fri, 10 Jul 2020 at 10:58, Kevin Wolf wrote: > > > > > > Am 09.07.2020 um 16:15 hat Peter Maydell geschrieben: > > > > dd/truncate etc won't work if the image file is not raw (eg if >

Re: [PATCH for-5.1] qemu-img resize: Require --shrink for shrinking all image formats

2020-07-10 Thread Daniel P . Berrangé
On Fri, Jul 10, 2020 at 02:17:17PM +0200, Kevin Wolf wrote: > QEMU 2.11 introduced the --shrink option for qemu-img resize to avoid > accidentally shrinking images (commit 4ffca8904a3). However, for > compatibility reasons, it was not enforced for raw images yet, but only > a deprecation warning

[PATCH for-5.1] qemu-img resize: Require --shrink for shrinking all image formats

2020-07-10 Thread Kevin Wolf
QEMU 2.11 introduced the --shrink option for qemu-img resize to avoid accidentally shrinking images (commit 4ffca8904a3). However, for compatibility reasons, it was not enforced for raw images yet, but only a deprecation warning was printed. This warning has existed for long enough that we can now

Re: [PATCH 2/2] hw/sd/sdcard: Do not allow invalid SD card sizes

2020-07-10 Thread Kevin Wolf
Am 10.07.2020 um 11:59 hat Peter Maydell geschrieben: > On Fri, 10 Jul 2020 at 10:58, Kevin Wolf wrote: > > > > Am 09.07.2020 um 16:15 hat Peter Maydell geschrieben: > > > dd/truncate etc won't work if the image file is not raw (eg if > > > it's qcow2). The only chance you have of something

Re: [PATCH 2/2] hw/sd/sdcard: Do not allow invalid SD card sizes

2020-07-10 Thread Peter Maydell
On Fri, 10 Jul 2020 at 10:58, Kevin Wolf wrote: > > Am 09.07.2020 um 16:15 hat Peter Maydell geschrieben: > > dd/truncate etc won't work if the image file is not raw (eg if > > it's qcow2). The only chance you have of something that's actually > > generic would probably involve "qemu-img resize".

Re: [PATCH 2/2] hw/sd/sdcard: Do not allow invalid SD card sizes

2020-07-10 Thread Kevin Wolf
Am 09.07.2020 um 16:15 hat Peter Maydell geschrieben: > On Thu, 9 Jul 2020 at 14:56, Philippe Mathieu-Daudé wrote: > > > > On 7/7/20 10:29 PM, Niek Linnenbank wrote: > > > So I manually copy & pasted the change into hw/sd/sd.c to test it. > > > It looks like the check works, but my concern is

Re: [PULL 18/31] block/core: add generic infrastructure for x-blockdev-amend qmp command

2020-07-10 Thread Max Reitz
On 09.07.20 17:09, Peter Maydell wrote: > On Mon, 6 Jul 2020 at 11:05, Max Reitz wrote: >> >> From: Maxim Levitsky >> >> blockdev-amend will be used similiar to blockdev-create >> to allow on the fly changes of the structure of the format based block >> devices. >> >> Current plan is to first

Re: [PATCH 2/2] hw/sd/sdcard: Do not allow invalid SD card sizes

2020-07-10 Thread Peter Maydell
On Thu, 9 Jul 2020 at 17:27, Alistair Francis wrote: > > On Thu, Jul 9, 2020 at 7:35 AM Philippe Mathieu-Daudé wrote: > > > > On 7/9/20 4:15 PM, Peter Maydell wrote: > > > The only chance you have of something that's actually > > > generic would probably involve "qemu-img resize". But I'm a bit

[PATCH] block/amend: Check whether the node exists

2020-07-10 Thread Max Reitz
We should check whether the user-specified node-name actually refers to a node. The simplest way to do that is to use bdrv_lookup_bs() instead of bdrv_find_node() (the former wraps the latter, and produces an error message if necessary). Reported-by: Coverity (CID 1430268) Fixes:

Re: [PATCH v1] qmp: don't hold ctx lock while querying blockstats

2020-07-10 Thread Kevin Wolf
Am 10.07.2020 um 10:54 hat Zhenyu Ye geschrieben: > Because the QMP command runs in the main thread, and changes > to the aio context of iothread will only be executed in the > main thread (they will not be in parallel), so there is no > need a lock protection while querying blockstats. > > If we

Re: [PATCH v1] qmp: don't hold ctx lock while querying blockstats

2020-07-10 Thread no-reply
Patchew URL: https://patchew.org/QEMU/20200710085400.343-1-yezhen...@huawei.com/ Hi, This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally. === TEST SCRIPT BEGIN ===

Re: [PATCH] block: Avoid stale pointer dereference in blk_get_aio_context()

2020-07-10 Thread Kevin Wolf
Am 09.07.2020 um 15:50 hat Greg Kurz geschrieben: > It is possible for blk_remove_bs() to race with blk_drain_all(), causing > the latter to dereference a stale blk->root pointer: > > > blk_remove_bs(blk) >bdrv_root_unref_child(blk->root) > child_bs = blk->root->bs >

Re: [PULL 00/12] Block patches

2020-07-10 Thread Kevin Wolf
Am 09.07.2020 um 20:41 hat Eduardo Habkost geschrieben: > On Thu, Jul 09, 2020 at 05:02:06PM +0200, Kevin Wolf wrote: > > Am 08.07.2020 um 00:05 hat Eduardo Habkost geschrieben: > > > On Tue, Jul 07, 2020 at 05:28:21PM +0200, Philippe Mathieu-Daudé wrote: > > > > On 6/26/20 12:25 PM, Stefan

Re: [PATCH 09/13] nbd: add GUri-based URI parsing version

2020-07-10 Thread Daniel P . Berrangé
On Thu, Jul 09, 2020 at 11:42:30PM +0400, Marc-André Lureau wrote: > Signed-off-by: Marc-André Lureau > --- > block/nbd.c| 86 +++--- > util/Makefile.objs | 2 +- > 2 files changed, 66 insertions(+), 22 deletions(-) > > diff --git a/block/nbd.c