Re: Problems with c8bb23cbdbe3 on ppc64le

2019-10-24 Thread Max Reitz
On 10.10.19 17:17, Max Reitz wrote: > Hi everyone, > > (CCs just based on tags in the commit in question) > > I have two bug reports which claim problems of qcow2 on XFS on ppc64le > machines since qemu 4.1.0. One of those is about bad performance > (sorry, is isn’t public :-/), the other about

Re: [PATCH] blockdev: modify blockdev-change-medium to change non-removable device

2019-10-24 Thread Max Reitz
On 23.10.19 16:10, Vladimir Sementsov-Ogievskiy wrote: > 23.10.2019 16:56, Vladimir Sementsov-Ogievskiy wrote: >> 22.10.2019 14:05, Max Reitz wrote: >>> On 21.10.19 08:50, Denis Plotnikov wrote: On 18.10.2019 18:02, Max Reitz wrote: > On 18.10.19 14:09, Denis Plotnikov wrote: >> T

Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock

2019-10-24 Thread Denis Lunev
On 10/23/19 6:26 PM, Kevin Wolf wrote: > qcow2_cache_do_get() requires that s->lock is locked because it can > yield between picking a cache entry and actually taking ownership of it > by setting offset and increasing the reference count. > > Add an assertion to make sure the caller really holds th

Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock

2019-10-24 Thread Kevin Wolf
Am 24.10.2019 um 12:01 hat Denis Lunev geschrieben: > On 10/23/19 6:26 PM, Kevin Wolf wrote: > > qcow2_cache_do_get() requires that s->lock is locked because it can > > yield between picking a cache entry and actually taking ownership of it > > by setting offset and increasing the reference count.

Re: [PATCH v5 1/4] block: support compressed write at generic layer

2019-10-24 Thread Max Reitz
On 22.10.19 15:53, Andrey Shinkevich wrote: [...] > If the support of COW for compressed writes is found feasible, will it > make a sense to implement? Then this series will follow. Hm, what exactly do you mean by support of COW for compressed writes? > A comment for the option would warn a us

Re: [PATCH 3/3] qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()

2019-10-24 Thread Kevin Wolf
Am 24.10.2019 um 12:46 hat Vladimir Sementsov-Ogievskiy geschrieben: > 23.10.2019 18:26, Kevin Wolf wrote: > > qcow2_detect_metadata_preallocation() calls qcow2_get_refcount() which > > requires s->lock to be taken to protect its accesses to the refcount > > table and refcount blocks. However, noth

Re: [PATCH 1/3] coroutine: Add qemu_co_mutex_assert_locked()

2019-10-24 Thread Denis Lunev
On 10/23/19 6:26 PM, Kevin Wolf wrote: > Some functions require that the caller holds a certain CoMutex for them > to operate correctly. Add a function so that they can assert the lock is > really held. > > Cc: qemu-sta...@nongnu.org > Signed-off-by: Kevin Wolf > --- > include/qemu/coroutine.h |

Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock

2019-10-24 Thread Denis Lunev
On 10/24/19 1:57 PM, Kevin Wolf wrote: > Am 24.10.2019 um 12:01 hat Denis Lunev geschrieben: >> On 10/23/19 6:26 PM, Kevin Wolf wrote: >>> qcow2_cache_do_get() requires that s->lock is locked because it can >>> yield between picking a cache entry and actually taking ownership of it >>> by setting o

Re: [PATCH 3/3] qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()

2019-10-24 Thread Vladimir Sementsov-Ogievskiy
23.10.2019 18:26, Kevin Wolf wrote: > qcow2_detect_metadata_preallocation() calls qcow2_get_refcount() which > requires s->lock to be taken to protect its accesses to the refcount > table and refcount blocks. However, nothing in this code path actually > took the lock. This could cause the same cac

Re: [PATCH 1/3] coroutine: Add qemu_co_mutex_assert_locked()

2019-10-24 Thread Kevin Wolf
Am 24.10.2019 um 11:59 hat Denis Lunev geschrieben: > On 10/23/19 6:26 PM, Kevin Wolf wrote: > > Some functions require that the caller holds a certain CoMutex for them > > to operate correctly. Add a function so that they can assert the lock is > > really held. > > > > Cc: qemu-sta...@nongnu.org >

Re: [RFC PATCH 00/18] Add qemu-storage-daemon

2019-10-24 Thread Kevin Wolf
Am 17.10.2019 um 15:01 hat Kevin Wolf geschrieben: > This series adds a new tool 'qemu-storage-daemon', which can be used to > export and perform operations on block devices. There is some overlap > between qemu-img/qemu-nbd and the new qemu-storage-daemon, but there are > a few important differenc

Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock

2019-10-24 Thread Kevin Wolf
Am 24.10.2019 um 13:14 hat Denis Lunev geschrieben: > On 10/24/19 1:57 PM, Kevin Wolf wrote: > > Am 24.10.2019 um 12:01 hat Denis Lunev geschrieben: > >> On 10/23/19 6:26 PM, Kevin Wolf wrote: > >>> qcow2_cache_do_get() requires that s->lock is locked because it can > >>> yield between picking a ca

Re: [PATCH 1/3] coroutine: Add qemu_co_mutex_assert_locked()

2019-10-24 Thread Denis Lunev
On 10/24/19 1:54 PM, Kevin Wolf wrote: > Am 24.10.2019 um 11:59 hat Denis Lunev geschrieben: >> On 10/23/19 6:26 PM, Kevin Wolf wrote: >>> Some functions require that the caller holds a certain CoMutex for them >>> to operate correctly. Add a function so that they can assert the lock is >>> really

Re: [PATCH v3 5/6] iotests: Enable more tests in the 'auto' group to improve test coverage

2019-10-24 Thread Alex Bennée
Thomas Huth writes: > > According to Max, it would be good to have a test for iothreads and > migration. 127 and 256 seem to be good candidates for iothreads. For > migration, let's enable 091, 181, and 203 (which also tests iothreads). > @@ -112,7 +112,7 @@ > 088 rw quick > 089 rw auto qui

Re: [PATCH 3/3] qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()

2019-10-24 Thread Vladimir Sementsov-Ogievskiy
24.10.2019 14:17, Kevin Wolf wrote: > Am 24.10.2019 um 12:46 hat Vladimir Sementsov-Ogievskiy geschrieben: >> 23.10.2019 18:26, Kevin Wolf wrote: >>> qcow2_detect_metadata_preallocation() calls qcow2_get_refcount() which >>> requires s->lock to be taken to protect its accesses to the refcount >>> t

Re: [PATCH v5 1/4] block: support compressed write at generic layer

2019-10-24 Thread Andrey Shinkevich
On 24/10/2019 12:34, Max Reitz wrote: > On 22.10.19 15:53, Andrey Shinkevich wrote: > > [...] > >> If the support of COW for compressed writes is found feasible, will it >> make a sense to implement? Then this series will follow. > > Hm, what exactly do you mean by support of COW for compresse

Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock

2019-10-24 Thread Kevin Wolf
Am 24.10.2019 um 15:03 hat Vladimir Sementsov-Ogievskiy geschrieben: > 24.10.2019 13:57, Kevin Wolf wrote: > > Am 24.10.2019 um 12:01 hat Denis Lunev geschrieben: > >> On 10/23/19 6:26 PM, Kevin Wolf wrote: > >>> qcow2_cache_do_get() requires that s->lock is locked because it can > >>> yield betwee

Re: [RFC PATCH 01/18] qemu-storage-daemon: Add barebone tool

2019-10-24 Thread Eric Blake
On 10/17/19 8:01 AM, Kevin Wolf wrote: This adds a new binary qemu-storage-daemon that doesn't yet do more than some typical initialisation for tools and parsing the basic command options --version, --help and --trace. Signed-off-by: Kevin Wolf --- configure | 2 +- qemu-storag

Re: [RFC PATCH 00/18] Add qemu-storage-daemon

2019-10-24 Thread Vladimir Sementsov-Ogievskiy
Hi! This reflects our idea of using qemu binary instead of qemu-img for doing block-layer operations offline. What is the practical difference between qemu-storage-daemon and starting qemu binary in stopped state? 17.10.2019 16:01, Kevin Wolf wrote: > This series adds a new tool 'qemu-storage-da

Re: [PATCH 2/3] qcow2: Assert that qcow2_cache_get() callers hold s->lock

2019-10-24 Thread Vladimir Sementsov-Ogievskiy
24.10.2019 13:57, Kevin Wolf wrote: > Am 24.10.2019 um 12:01 hat Denis Lunev geschrieben: >> On 10/23/19 6:26 PM, Kevin Wolf wrote: >>> qcow2_cache_do_get() requires that s->lock is locked because it can >>> yield between picking a cache entry and actually taking ownership of it >>> by setting offs

Re: [PATCH v5 1/4] block: support compressed write at generic layer

2019-10-24 Thread Andrey Shinkevich
On 24/10/2019 16:48, Max Reitz wrote: > On 24.10.19 14:56, Andrey Shinkevich wrote: >> >> >> On 24/10/2019 12:34, Max Reitz wrote: >>> On 22.10.19 15:53, Andrey Shinkevich wrote: >>> >>> [...] >>> If the support of COW for compressed writes is found feasible, will it make a sense to imp

Re: [PATCH v5 1/4] block: support compressed write at generic layer

2019-10-24 Thread Max Reitz
On 24.10.19 14:56, Andrey Shinkevich wrote: > > > On 24/10/2019 12:34, Max Reitz wrote: >> On 22.10.19 15:53, Andrey Shinkevich wrote: >> >> [...] >> >>> If the support of COW for compressed writes is found feasible, will it >>> make a sense to implement? Then this series will follow. >> >> Hm, w

[PATCH v2 2/2] qcow2: Fix corruption bug in qcow2_detect_metadata_preallocation()

2019-10-24 Thread Kevin Wolf
qcow2_detect_metadata_preallocation() calls qcow2_get_refcount() which requires s->lock to be taken to protect its accesses to the refcount table and refcount blocks. However, nothing in this code path actually took the lock. This could cause the same cache entry to be used by two requests at the s

[PATCH v7 1/4] block/replication.c: Ignore requests after failover

2019-10-24 Thread Lukas Straub
After failover the Secondary side of replication shouldn't change state, because it now functions as our primary disk. In replication_start, replication_do_checkpoint, replication_stop, ignore the request if current state is BLOCK_REPLICATION_DONE (sucessful failover) or BLOCK_REPLICATION_FAILOVER

[PATCH v7 3/4] net/filter.c: Add Options to insert filters anywhere in the filter list

2019-10-24 Thread Lukas Straub
To switch the Secondary to Primary, we need to insert new filters before the filter-rewriter. Add the options insert= and position= to be able to insert filters anywhere in the filter list. position should be "head" or "tail" to insert at the head or tail of the filter list or it should be "id="

[PATCH v7 2/4] tests/test-replication.c: Add test for for secondary node continuing replication

2019-10-24 Thread Lukas Straub
This simulates the case that happens when we resume COLO after failover. Signed-off-by: Lukas Straub --- tests/test-replication.c | 52 1 file changed, 52 insertions(+) diff --git a/tests/test-replication.c b/tests/test-replication.c index f085d1993a..8e

Re: [PATCH v5 1/4] block: support compressed write at generic layer

2019-10-24 Thread Andrey Shinkevich
On 24/10/2019 16:48, Max Reitz wrote: > On 24.10.19 14:56, Andrey Shinkevich wrote: >> >> >> On 24/10/2019 12:34, Max Reitz wrote: >>> On 22.10.19 15:53, Andrey Shinkevich wrote: >>> >>> [...] >>> If the support of COW for compressed writes is found feasible, will it make a sense to imp

[PATCH v7 4/4] colo: Update Documentation for continuous replication

2019-10-24 Thread Lukas Straub
Document the qemu command-line and qmp commands for continuous replication Signed-off-by: Lukas Straub --- docs/COLO-FT.txt | 224 +++-- docs/block-replication.txt | 28 +++-- 2 files changed, 184 insertions(+), 68 deletions(-) diff --git a/docs/COLO-F

[PATCH v2 1/2] coroutine: Add qemu_co_mutex_assert_locked()

2019-10-24 Thread Kevin Wolf
Some functions require that the caller holds a certain CoMutex for them to operate correctly. Add a function so that they can assert the lock is really held. Cc: qemu-sta...@nongnu.org Signed-off-by: Kevin Wolf --- include/qemu/coroutine.h | 15 +++ 1 file changed, 15 insertions(+)

[PATCH v2 0/2] qcow2: Fix image corruption bug in 4.1

2019-10-24 Thread Kevin Wolf
This series fixes an image corruption bug that was introduced in commit 69f47505e ('block: avoid recursive block_status call if possible'), first contained in the QEMU 4.1.0 release. This bug was reported by Michael Weiser on Launchpad: https://bugs.launchpad.net/qemu/+bug/1846427 v2: - Dropped

Re: [PATCH v2 1/2] coroutine: Add qemu_co_mutex_assert_locked()

2019-10-24 Thread Vladimir Sementsov-Ogievskiy
24.10.2019 17:26, Kevin Wolf wrote: > Some functions require that the caller holds a certain CoMutex for them > to operate correctly. Add a function so that they can assert the lock is > really held. > > Cc: qemu-sta...@nongnu.org > Signed-off-by: Kevin Wolf Reviewed-by: Vladimir Sementsov-Ogiev

Re: [PATCH v2 1/2] coroutine: Add qemu_co_mutex_assert_locked()

2019-10-24 Thread Denis Lunev
On 10/24/19 6:35 PM, Vladimir Sementsov-Ogievskiy wrote: > 24.10.2019 17:26, Kevin Wolf wrote: >> Some functions require that the caller holds a certain CoMutex for them >> to operate correctly. Add a function so that they can assert the lock is >> really held. >> >> Cc: qemu-sta...@nongnu.org >> S

Re: [PATCH v5 1/4] block: support compressed write at generic layer

2019-10-24 Thread Max Reitz
On 24.10.19 16:07, Andrey Shinkevich wrote: > > > On 24/10/2019 16:48, Max Reitz wrote: >> On 24.10.19 14:56, Andrey Shinkevich wrote: >>> >>> >>> On 24/10/2019 12:34, Max Reitz wrote: On 22.10.19 15:53, Andrey Shinkevich wrote: [...] > If the support of COW for compressed

[PATCH v6 8/9] virtio: add some rcu comments

2019-10-24 Thread Eugenio Pérez
Signed-off-by: Eugenio Pérez --- hw/virtio/virtio.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c index 3cf12a62c0..9195b08da8 100644 --- a/hw/virtio/virtio.c +++ b/hw/virtio/virtio.c @@ -278,6 +278,7 @@ static void vring_packed_flag

[PATCH v6 1/9] virtio: basic structure for packed ring

2019-10-24 Thread Eugenio Pérez
From: Wei Xu Define packed ring structure according to Qemu nomenclature, field data(wrap counter, etc) are also included. Signed-off-by: Wei Xu Signed-off-by: Jason Wang Reviewed-by: Jens Freimann --- hw/virtio/virtio.c | 15 +++ 1 file changed, 15 insertions(+) diff --git a/hw

Re: [PATCH v2 0/4] apic: Fix migration breakage of >255 vcpus

2019-10-24 Thread John Snow
On 10/23/19 4:17 AM, Kevin Wolf wrote: > The important difference here is legacy IDE (which works) vs. AHCI > (which doesn't work). If you add a -device ahci to the -M pc case, it > starts failing, too. > > Not sure why AHCI fails, but I'll just CC John who is the lucky > maintainer of this dev

[PATCH v6 4/9] virtio: basic packed virtqueue support

2019-10-24 Thread Eugenio Pérez
From: Jason Wang This patch implements basic support for the packed virtqueue. Compare the split virtqueue which has three rings, packed virtqueue only have one which is supposed to have better cache utilization and more hardware friendly. Please refer virtio specification for more information.

[PATCH v6 7/9] virtio: add property to enable packed virtqueue

2019-10-24 Thread Eugenio Pérez
From: Jason Wang Signed-off-by: Jason Wang Reviewed-by: Jens Freimann --- include/hw/virtio/virtio.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h index d123d5b181..40ddeafadb 100644 --- a/include/hw/virtio/virtio

[PATCH v6 6/9] vhost_net: enable packed ring support

2019-10-24 Thread Eugenio Pérez
From: Jason Wang Signed-off-by: Jason Wang Reviewed-by: Jens Freimann --- hw/net/vhost_net.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c index e975700f95..6b82803fa7 100644 --- a/hw/net/vhost_net.c +++ b/hw/net/vhost_net.c @@ -49,6 +49,7 @@ sta

[PATCH v6 5/9] virtio: event suppression support for packed ring

2019-10-24 Thread Eugenio Pérez
From: Jason Wang This patch implements event suppression through device/driver area. Please refer virtio specification for more information. Signed-off-by: Wei Xu Signed-off-by: Jason Wang --- hw/virtio/virtio.c | 142 + 1 file changed, 133

[PATCH v6 9/9] virtio: Move vdev->broken check to dispatch drop_all

2019-10-24 Thread Eugenio Pérez
Previous commits did the same with others virtqueue_ functions, but this check was repeated in the split and the packed version. Signed-off-by: Eugenio Pérez --- hw/virtio/virtio.c | 12 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/hw/virtio/virtio.c b/hw/virtio/vir

[PATCH v6 0/9] Packed virtqueue for virtio

2019-10-24 Thread Eugenio Pérez
Hi: This is an updated version of packed virtqueue support based on Wei and Jason's V5, mainly solving the clang leak detector error CI gave. Please review. Changes from V5: - Fix qemu's CI asan error. - Move/copy rcu comments. - Merge duplicated vdev->broken check between split and packet versi

[PATCH v6 3/9] virtio: Free rng and blk virqueues

2019-10-24 Thread Eugenio Pérez
The function virtio_del_queue was not called at these devices unrealize() callbacks. This was detected due to add an allocated element on the next commit (used_elems) and running address sanitizer memory leak detector. Signed-off-by: Eugenio Pérez --- hw/block/virtio-blk.c | 5 + hw/virtio

[PATCH v6 2/9] virtio: device/driver area size calculation refactor for split ring

2019-10-24 Thread Eugenio Pérez
From: Wei Xu There is slight size difference between split/packed rings. This is the refactor of split ring as well as a helper to expanding device and driver area size calculation for packed ring. Signed-off-by: Wei Xu Signed-off-by: Jason Wang Reviewed-by: Jens Freimann --- hw/virtio/virt

Re: qemu crashing when attaching an ISO file to a virtio-scsi CD-ROM device through libvirt

2019-10-24 Thread Fernando Casas Schössow
Today I updated to qemu 4.0.1 since this was the latest version available for Alpine and I can confirm that I can repro the issue with this version as well. Not sure if relevant but I can also confirm that the problem happens with Windows Server 2012 R2 but also with Linux guests (it doesn't ma

Re: qemu crashing when attaching an ISO file to a virtio-scsi CD-ROM device through libvirt

2019-10-24 Thread Fernando Casas Schössow
BTW just to be clear, qemu is crashing in this scenario *only* if iothread is enabled for the guest. Without iothread enabled the operation is completed without any problems. On jue, oct 24, 2019 at 11:07 PM, Fernando Casas Schössow wrote: > Today I updated to qemu 4.0.1 since this was the lat

Re: [PATCH v2 0/4] apic: Fix migration breakage of >255 vcpus

2019-10-24 Thread Peter Xu
On Thu, Oct 24, 2019 at 01:49:11PM -0400, John Snow wrote: > > > On 10/23/19 4:17 AM, Kevin Wolf wrote: > > The important difference here is legacy IDE (which works) vs. AHCI > > (which doesn't work). If you add a -device ahci to the -M pc case, it > > starts failing, too. > > > > Not sure why A

Re: [PATCH v6 0/9] Packed virtqueue for virtio

2019-10-24 Thread Jason Wang
On 2019/10/25 上午1:13, Eugenio Pérez wrote: Hi: This is an updated version of packed virtqueue support based on Wei and Jason's V5, mainly solving the clang leak detector error CI gave. Please review. Changes from V5: - Fix qemu's CI asan error. - Move/copy rcu comments. - Merge duplicated vd