Re: [PATCH V6 0/6] hw/block/nvme: support multi-path for ctrl/ns

2021-01-26 Thread Minwoo Im
On 21-01-27 05:39:29, Keith Busch wrote: > This came out looking cleaner than I had initially expected. Thanks for > seeing this feature through! > > Reviewed-by: Keith Busch Thanks Keith for the review!

Re: [PATCH v2 8/9] tests/docker: Add dockerfile for Alpine Linux

2021-01-26 Thread John Snow
On 1/19/21 8:41 AM, Thomas Huth wrote: On 18/01/2021 11.33, Daniel P. Berrangé wrote: On Mon, Jan 18, 2021 at 02:38:07PM +0800, Jiaxun Yang wrote: Alpine Linux[1] is a security-oriented, lightweight Linux distribution based on musl libc and busybox. It it popular among Docker guests and

Re: [PATCH v5 03/10] iotests: Move try_remove to iotests.py

2021-01-26 Thread John Snow
On 1/18/21 5:57 AM, Max Reitz wrote: Signed-off-by: Max Reitz Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Willian Rampazzo --- tests/qemu-iotests/124| 8 +--- tests/qemu-iotests/iotests.py | 11 +++ 2 files changed, 8 insertions(+),

Re: [PATCH v5 02/10] iotests/297: Rewrite in Python and extend reach

2021-01-26 Thread John Snow
On 1/18/21 5:57 AM, Max Reitz wrote: Instead of checking iotests.py only, check all Python files in the qemu-iotests/ directory. Of course, most of them do not pass, so there is an extensive skip list for now. (The only files that do pass are 209, 254, 283, and iotests.py.) Chiming in to

Re: [PATCH v5 01/10] iotests.py: Assume a couple of variables as given

2021-01-26 Thread John Snow
On 1/18/21 5:57 AM, Max Reitz wrote: There are a couple of environment variables that we fetch with os.environ.get() without supplying a default. Clearly they are required and expected to be set by the ./check script (as evidenced by execute_setup_common(), which checks for test_dir and

Re: [PATCH V6 4/6] hw/block/nvme: support for multi-controller in subsystem

2021-01-26 Thread Minwoo Im
On 21-01-26 09:57:23, Keith Busch wrote: > On Tue, Jan 26, 2021 at 09:52:48AM +0900, Minwoo Im wrote: > > On 21-01-25 10:11:43, Keith Busch wrote: > > > On Mon, Jan 25, 2021 at 07:03:32PM +0100, Klaus Jensen wrote: > > > > On Jan 24 11:54, Minwoo Im wrote: > > > > > We have nvme-subsys and nvme

Re: [PATCH V6 0/6] hw/block/nvme: support multi-path for ctrl/ns

2021-01-26 Thread Keith Busch
This came out looking cleaner than I had initially expected. Thanks for seeing this feature through! Reviewed-by: Keith Busch

Re: [PATCH v2 1/2] tests/qtest: Only run fuzz-megasas-test if megasas device is available

2021-01-26 Thread Thomas Huth
On 26/01/2021 19.01, Alexander Bulekov wrote: On 210126 1851, Thomas Huth wrote: On 26/01/2021 12.16, Philippe Mathieu-Daudé wrote: This test fails when QEMU is built without the megasas device, restrict it to its availability. Signed-off-by: Philippe Mathieu-Daudé ---

Re: [PATCH v2 1/2] tests/qtest: Only run fuzz-megasas-test if megasas device is available

2021-01-26 Thread Alexander Bulekov
On 210126 1851, Thomas Huth wrote: > On 26/01/2021 12.16, Philippe Mathieu-Daudé wrote: > > This test fails when QEMU is built without the megasas device, > > restrict it to its availability. > > > > Signed-off-by: Philippe Mathieu-Daudé > > --- > > tests/qtest/fuzz-megasas-test.c | 49

Re: [PATCH V6 4/6] hw/block/nvme: support for multi-controller in subsystem

2021-01-26 Thread Keith Busch
On Tue, Jan 26, 2021 at 09:52:48AM +0900, Minwoo Im wrote: > On 21-01-25 10:11:43, Keith Busch wrote: > > On Mon, Jan 25, 2021 at 07:03:32PM +0100, Klaus Jensen wrote: > > > On Jan 24 11:54, Minwoo Im wrote: > > > > We have nvme-subsys and nvme devices mapped together. To support > > > >

Re: [PATCH v2 2/2] tests/qtest: Only run fuzz-virtio-scsi when virtio-scsi is available

2021-01-26 Thread Thomas Huth
On 26/01/2021 12.16, Philippe Mathieu-Daudé wrote: This test fails when QEMU is built without the virtio-scsi device, restrict it to its availability. Reviewed-by: Michael S. Tsirkin Signed-off-by: Philippe Mathieu-Daudé --- tests/qtest/fuzz-test.c | 51

Re: [PATCH v2 1/2] tests/qtest: Only run fuzz-megasas-test if megasas device is available

2021-01-26 Thread Thomas Huth
On 26/01/2021 12.16, Philippe Mathieu-Daudé wrote: This test fails when QEMU is built without the megasas device, restrict it to its availability. Signed-off-by: Philippe Mathieu-Daudé --- tests/qtest/fuzz-megasas-test.c | 49 + tests/qtest/fuzz-test.c

Re: [PATCH] hw/block/nvme: add broadcast nsid support flush command

2021-01-26 Thread Stefan Hajnoczi
On Mon, Jan 25, 2021 at 09:42:31PM +0100, Klaus Jensen wrote: > From: Gollu Appalanaidu > > Add support for using the broadcast nsid to issue a flush on all > namespaces through a single command. > > Signed-off-by: Gollu Appalanaidu > Reviewed-by: Klaus Jensen > --- > include/block/nvme.h |

Re: [PATCH v9 09/11] block: check availablity for preadv/pwritev on mac

2021-01-26 Thread Peter Maydell
On Tue, 26 Jan 2021 at 01:38, Joelle van Dyne wrote: > > macOS 11/iOS 14 added preadv/pwritev APIs. Due to weak linking, configure > will succeed with CONFIG_PREADV even when targeting a lower OS version. I just ran into this this afternoon. It turns out that all our OSX CI configs pass

Re: [PATCH v9 0/6] Rework iotests/check

2021-01-26 Thread Vladimir Sementsov-Ogievskiy
26.01.2021 18:36, Kevin Wolf wrote: Am 26.01.2021 um 16:15 hat Vladimir Sementsov-Ogievskiy geschrieben: OK, thanks for handling it! You're welcome. Only problem now: Max sent a conflicting pull request that touches 'group'. He suggested that we could split the deletion of 'group' from the

Re: [PATCH v9 0/6] Rework iotests/check

2021-01-26 Thread Kevin Wolf
Am 26.01.2021 um 16:15 hat Vladimir Sementsov-Ogievskiy geschrieben: > OK, thanks for handling it! You're welcome. Only problem now: Max sent a conflicting pull request that touches 'group'. He suggested that we could split the deletion of 'group' from the 'check' rewrite and merge it only later

Re: [PATCH v9 0/6] Rework iotests/check

2021-01-26 Thread Kevin Wolf
Am 26.01.2021 um 14:19 hat Vladimir Sementsov-Ogievskiy geschrieben: > 26.01.2021 15:53, Kevin Wolf wrote: > > Am 25.01.2021 um 19:50 hat Vladimir Sementsov-Ogievskiy geschrieben: > > > v9: > > > 01: new, one more whitespace-fixing patch > > > testenv: allow case when we don't have system-arch

Re: [PATCH v9 0/6] Rework iotests/check

2021-01-26 Thread Vladimir Sementsov-Ogievskiy
26.01.2021 18:07, Kevin Wolf wrote: Am 26.01.2021 um 14:19 hat Vladimir Sementsov-Ogievskiy geschrieben: 26.01.2021 15:53, Kevin Wolf wrote: Am 25.01.2021 um 19:50 hat Vladimir Sementsov-Ogievskiy geschrieben: v9: 01: new, one more whitespace-fixing patch testenv: allow case when we don't

[PULL 53/53] iotests/178: Pass value to invalid option

2021-01-26 Thread Max Reitz
ccd3b3b8112 has deprecated short-hand boolean options (i.e., options with values). All options without values are interpreted as boolean options, so this includes the invalid option "snapshot.foo" used in iotest 178. So after ccd3b3b8112, 178 fails with: +qemu-img: warning: short-form boolean

[PULL 50/53] block: report errno when flock fcntl fails

2021-01-26 Thread Max Reitz
From: David Edmondson When a call to fcntl(2) for the purpose of adding file locks fails with an error other than EAGAIN or EACCES, report the error returned by fcntl. EAGAIN or EACCES are elided as they are considered to be common failures, indicating that a conflicting lock is held by another

Re: [PATCH v9 03/11] configure: check for sys/disk.h

2021-01-26 Thread Warner Losh
On Mon, Jan 25, 2021 at 6:33 PM Joelle van Dyne wrote: > Some BSD platforms do not have this header. > > Signed-off-by: Joelle van Dyne > --- > meson.build| 1 + > block.c| 2 +- > block/file-posix.c | 2 +- > 3 files changed, 3 insertions(+), 2 deletions(-) > > diff --git

Re: [PATCH v9 03/11] configure: check for sys/disk.h

2021-01-26 Thread Warner Losh
On Tue, Jan 26, 2021 at 12:08 AM Philippe Mathieu-Daudé wrote: > On 1/26/21 6:55 AM, Joelle van Dyne wrote: > > Previously, the only case where sys/disk.h does not exist is on > > platforms that define __DragonFly__. However, iOS also does not have > > this header. Previously, I had it as > > >

[PULL 51/53] iotests: Add test for the regression fixed in c8bf9a9169

2021-01-26 Thread Max Reitz
From: Alberto Garcia Signed-off-by: Alberto Garcia Suggested-by: Maxim Levitsky Reviewed-by: Maxim Levitsky Message-Id: <20210112170540.2912-1-be...@igalia.com> [mreitz: Add "# group:" line] Signed-off-by: Max Reitz --- tests/qemu-iotests/313 | 104 +

[PULL 45/53] block/block-copy: drop unused block_copy_set_progress_callback()

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Drop unused code. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-20-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- include/block/block-copy.h | 6 -- block/block-copy.c | 15

Re: [PATCH v9 03/11] configure: check for sys/disk.h

2021-01-26 Thread Warner Losh
On Mon, Jan 25, 2021 at 10:55 PM Joelle van Dyne wrote: > Previously, the only case where sys/disk.h does not exist is on > platforms that define __DragonFly__. However, iOS also does not have > this header. Previously, I had it as > > #if defined(__DragonFly__) || defined(CONFIG_IOS) > > But

[PULL 52/53] iotests/118: Drop 'change' test

2021-01-26 Thread Max Reitz
Commit 0afec75734331 removed the 'change' QMP command, so we can no longer test it in 118. Fixes: 0afec75734331a0b52fa3aa4235220eda8c7846f ('qmp: remove deprecated "change" command') Signed-off-by: Max Reitz Message-Id: <20210126104833.57026-1-mre...@redhat.com> Reviewed-by: Philippe

[PULL 49/53] simplebench: add bench-backup.py

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Add script to benchmark new backup architecture. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20210116214705.822267-24-vsement...@virtuozzo.com> [mreitz: s/not unsupported/not supported/] Signed-off-by: Max Reitz ---

[PULL 46/53] block/block-copy: drop unused argument of block_copy()

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-21-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- include/block/block-copy.h | 2 +- block/backup-top.c | 2 +- block/block-copy.c

[PULL 37/53] iotests/129: Limit backup's max-chunk/max-workers

2021-01-26 Thread Max Reitz
Right now, this does not change anything, because backup ignores max-chunk and max-workers. However, as soon as backup is switched over to block-copy for the background copying process, we will need it to keep 129 passing. Signed-off-by: Max Reitz Message-Id:

[PULL 48/53] simplebench: bench_block_job: add cmd_options argument

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Add argument to allow additional block-job options. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-23-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- scripts/simplebench/bench-example.py | 2

[PULL 42/53] block/backup: drop extra gotos from backup_run()

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-17-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- block/backup.c | 12 +--- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git

[PULL 35/53] qapi: backup: add max-chunk and max-workers to x-perf struct

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Add new parameters to configure future backup features. The patch doesn't introduce aio backup requests (so we actually have only one worker) neither requests larger than one cluster. Still, formally we satisfy these maximums anyway, so add the parameters now,

[PULL 47/53] simplebench/bench_block_job: use correct shebang line with python3

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-22-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- scripts/simplebench/bench_block_job.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

[PULL 44/53] qapi: backup: disable copy_range by default

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Further commit will add a benchmark (scripts/simplebench/bench-backup.py), which will show that backup works better with async parallel requests (previous commit) and disabled copy_range. So, let's disable copy_range by default. Note: the option was added

[PULL 39/53] iotests: 219: prepare for backup over block-copy

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy The further change of moving backup to be a one block-copy call will make copying chunk-size and cluster-size two separate things. So, even with 64k cluster sized qcow2 image, default chunk would be 1M. Test 219 depends on specified chunk-size. Update it for

[PULL 36/53] iotests: 56: prepare for backup over block-copy

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy After introducing parallel async copy requests instead of plain cluster-by-cluster copying loop, we'll have to wait for paused status, as we need to wait for several parallel request. So, let's gently wait instead of just asserting that job already paused.

[PULL 34/53] job: call job_enter from job_pause

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy If main job coroutine called job_yield (while some background process is in progress), we should give it a chance to call job_pause_point(). It will be used in backup, when moved on async block-copy. Note, that job_user_pause is not enough: we want to handle

[PULL 31/53] block/block-copy: add ratelimit to block-copy

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy We are going to directly use one async block-copy operation for backup job, so we need rate limiter. We want to maintain current backup behavior: only background copying is limited and copy-before-write operations only participate in limit calculation.

[PULL 40/53] iotests: 257: prepare for backup over block-copy

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Iotest 257 dumps a lot of in-progress information of backup job, such as offset and bitmap dirtiness. Further commit will move backup to be one block-copy call, which will introduce async parallel requests instead of plain cluster-by-cluster copying. To keep

[PULL 43/53] backup: move to block-copy

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy This brings async request handling and block-status driven chunk sizes to backup out of the box, which improves backup performance. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id:

[PULL 41/53] block/block-copy: make progress_bytes_callback optional

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy We are going to stop use of this callback in the following commit. Still the callback handling code will be dropped in a separate commit. So, for now let's make it optional. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id:

[PULL 29/53] block/block-copy: add max_chunk and max_workers parameters

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy They will be used for backup. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-5-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- include/block/block-copy.h | 6 ++ block/block-copy.c

[PULL 26/53] qapi: backup: add perf.use-copy-range parameter

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Experiments show, that copy_range is not always making things faster. So, to make experimentation simpler, let's add a parameter. Some more perf parameters will be added soon, so here is a new struct. For now, add new backup qmp parameter with x- prefix for

[PULL 27/53] block/block-copy: More explicit call_state

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Refactor common path to use BlockCopyCallState pointer as parameter, to prepare it for use in asynchronous block-copy (at least, we'll need to run block-copy in a coroutine, passing the whole parameters as one pointer). Signed-off-by: Vladimir

[PULL 38/53] iotests: 185: prepare for backup over block-copy

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy The further change of moving backup to be a one block-copy call will make copying chunk-size and cluster-size two separate things. So, even with 64k cluster sized qcow2 image, default chunk would be 1M. 185 test however assumes, that with speed limited to 64K,

[PULL 19/53] iotests/129: Do not check @busy

2021-01-26 Thread Max Reitz
@busy is false when the job is paused, which happens all the time because that is how jobs yield (e.g. for mirror at least since commit 565ac01f8d3). Back when 129 was added (2015), perhaps there was no better way of checking whether the job was still actually running. Now we have the @status

[PULL 32/53] block/block-copy: add block_copy_cancel

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Add function to cancel running async block-copy call. It will be used in backup. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-8-vsement...@virtuozzo.com> Signed-off-by: Max Reitz ---

[PULL 28/53] block/block-copy: implement block_copy_async

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy We'll need async block-copy invocation to use in backup directly. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-4-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- include/block/block-copy.h | 29

[PULL 23/53] iotests/129: Clean up pylint and mypy complaints

2021-01-26 Thread Max Reitz
And consequentially drop it from 297's skip list. Signed-off-by: Max Reitz Reviewed-by: Willian Rampazzo Reviewed-by: Vladimir Sementsov-Ogievskiy Message-Id: <20210118105720.14824-10-mre...@redhat.com> --- tests/qemu-iotests/129 | 4 ++-- tests/qemu-iotests/297 | 2 +- 2 files changed, 3

[PULL 21/53] iotests/129: Actually test a commit job

2021-01-26 Thread Max Reitz
Before this patch, test_block_commit() performs an active commit, which under the hood is a mirror job. If we want to test various different block jobs, we should perhaps run an actual commit job instead. Doing so requires adding an overlay above the source node before the commit is done (and

[PULL 25/53] coroutine-sigaltstack: Add SIGUSR2 mutex

2021-01-26 Thread Max Reitz
Disposition (action) for any given signal is global for the process. When two threads run coroutine-sigaltstack's qemu_coroutine_new() concurrently, they may interfere with each other: One of them may revert the SIGUSR2 handler to SIG_DFL, between the other thread (a) setting up

[PULL 20/53] iotests/129: Use throttle node

2021-01-26 Thread Max Reitz
Throttling on the BB has not affected block jobs in a while, so it is possible that one of the jobs in 129 finishes before the VM is stopped. We can fix that by running the job from a throttle node. Signed-off-by: Max Reitz Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy

[PULL 33/53] blockjob: add set_speed to BlockJobDriver

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy We are going to use async block-copy call in backup, so we'll need to passthrough setting backup speed to block-copy call. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-9-vsement...@virtuozzo.com>

[PULL 15/53] iotests.py: Assume a couple of variables as given

2021-01-26 Thread Max Reitz
There are a couple of environment variables that we fetch with os.environ.get() without supplying a default. Clearly they are required and expected to be set by the ./check script (as evidenced by execute_setup_common(), which checks for test_dir and qemu_default_machine to be set, and aborts if

[PULL 30/53] block/block-copy: add list of all call-states

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy It simplifies debugging. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-6-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- block/block-copy.c | 11 ++- 1 file changed, 10 insertions(+), 1

[PULL 18/53] iotests/129: Remove test images in tearDown()

2021-01-26 Thread Max Reitz
Signed-off-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Reviewed-by: Willian Rampazzo Message-Id: <20210118105720.14824-5-mre...@redhat.com> --- tests/qemu-iotests/129 | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/qemu-iotests/129

[PULL 17/53] iotests: Move try_remove to iotests.py

2021-01-26 Thread Max Reitz
Signed-off-by: Max Reitz Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Willian Rampazzo Message-Id: <20210118105720.14824-4-mre...@redhat.com> --- tests/qemu-iotests/124| 8 +--- tests/qemu-iotests/iotests.py | 11 +++ 2 files changed, 8

[PULL 12/53] iotests: 30: prepare to COR filter insertion by stream job

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy test_stream_parallel run parallel stream jobs, intersecting so that top of one is base of another. It's OK now, but it would be a problem if insert the filter, as one job will want to use another job's filter as above_base node. Correct thing to do is move to

[PULL 24/53] iotests/300: Clean up pylint and mypy complaints

2021-01-26 Thread Max Reitz
And consequentially drop it from 297's skip list. Signed-off-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Willian Rampazzo Message-Id: <20210118105720.14824-11-mre...@redhat.com> --- tests/qemu-iotests/297 | 2 +- tests/qemu-iotests/300 | 18 +++--- 2

[PULL 13/53] block/stream: add s->target_bs

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Add a direct link to target bs for convenience and to simplify following commit which will insert COR filter above target bs. This is a part of original commit written by Andrey. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id:

[PULL 14/53] block: apply COR-filter to block-stream jobs

2021-01-26 Thread Max Reitz
From: Andrey Shinkevich This patch completes the series with the COR-filter applied to block-stream operations. Adding the filter makes it possible in future implement discarding copied regions in backing files during the block-stream job, to reduce the disk overuse (we need control on

[PULL 22/53] iotests/129: Limit mirror job's buffer size

2021-01-26 Thread Max Reitz
Issuing 'stop' on the VM drains all nodes. If the mirror job has many large requests in flight, this may lead to significant I/O that looks a bit like 'stop' would make the job try to complete (which is what 129 should verify not to happen). We can limit the I/O in flight by limiting the buffer

[PULL 08/53] block: include supported_read_flags into BDS structure

2021-01-26 Thread Max Reitz
From: Andrey Shinkevich Add the new member supported_read_flags to the BlockDriverState structure. It will control the flags set for copy-on-read operations. Make the block generic layer evaluate supported read flags before they go to a block driver. Suggested-by: Vladimir Sementsov-Ogievskiy

[PULL 06/53] qapi: copy-on-read filter: add 'bottom' option

2021-01-26 Thread Max Reitz
From: Andrey Shinkevich Add an option to limit copy-on-read operations to specified sub-chain of backing-chain, to make copy-on-read filter useful for block-stream job. Suggested-by: Max Reitz Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich Signed-off-by: Vladimir

[PULL 07/53] iotests: add #310 to test bottom node in COR driver

2021-01-26 Thread Max Reitz
From: Andrey Shinkevich The test case #310 is similar to #216 by Max Reitz. The difference is that the test #310 involves a bottom node to the COR filter driver. Signed-off-by: Andrey Shinkevich Signed-off-by: Vladimir Sementsov-Ogievskiy [vsementsov: detach backing to test reads from top,

[PULL 09/53] copy-on-read: skip non-guest reads if no copy needed

2021-01-26 Thread Max Reitz
From: Andrey Shinkevich If the flag BDRV_REQ_PREFETCH was set, skip idling read/write operations in COR-driver. It can be taken into account for the COR-algorithms optimization. That check is being made during the block stream job by the moment. Add the BDRV_REQ_PREFETCH flag to the

[PULL 05/53] qapi: add filter-node-name to block-stream

2021-01-26 Thread Max Reitz
From: Andrey Shinkevich Provide the possibility to pass the 'filter-node-name' parameter to the block-stream job as it is done for the commit block job. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy [vsementsov: comment indentation, s/Since: 5.2/Since: 6.0/]

[PULL 16/53] iotests/297: Rewrite in Python and extend reach

2021-01-26 Thread Max Reitz
Instead of checking iotests.py only, check all Python files in the qemu-iotests/ directory. Of course, most of them do not pass, so there is an extensive skip list for now. (The only files that do pass are 209, 254, 283, and iotests.py.) (Alternatively, we could have the opposite, i.e. an

[PULL 10/53] stream: rework backing-file changing

2021-01-26 Thread Max Reitz
From: Andrey Shinkevich Stream in stream_prepare calls bdrv_change_backing_file() to change backing-file in the metadata of bs. It may use either backing-file parameter given by user or just take filename of base on job start. Backing file format is determined by base on job finish. There are

[PULL 04/53] copy-on-read: add filter drop function

2021-01-26 Thread Max Reitz
From: Andrey Shinkevich Provide API for the COR-filter removal. Also, drop the filter child permissions for an inactive state when the filter node is being removed. To insert the filter, the block generic layer function bdrv_insert_node() can be used. The new function bdrv_cor_filter_drop() may

[PULL 03/53] block: add API function to insert a node

2021-01-26 Thread Max Reitz
From: Andrey Shinkevich Provide API for insertion a node to backing chain. Suggested-by: Max Reitz Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20201216061703.70908-3-vsement...@virtuozzo.com> Signed-off-by: Max Reitz ---

[PULL 11/53] qapi: block-stream: add "bottom" argument

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy The code already don't freeze base node and we try to make it prepared for the situation when base node is changed during the operation. In other words, block-stream doesn't own base node. Let's introduce a new interface which should replace the current one,

[PULL 02/53] copy-on-read: support preadv/pwritev_part functions

2021-01-26 Thread Max Reitz
From: Andrey Shinkevich Add support for the recently introduced functions bdrv_co_preadv_part() and bdrv_co_pwritev_part() to the COR-filter driver. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy Message-Id: <20201216061703.70908-2-vsement...@virtuozzo.com>

[PULL 01/53] iotests: fix _check_o_direct

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Unfortunately commit "iotests: handle tmpfs" breaks running iotests with -nbd -nocache, as _check_o_direct tries to create $TEST_IMG.test_o_direct, but in case of nbd TEST_IMG is something like nbd+unix:///... , and test fails with message qemu-img:

[PULL 00/53] Block patches

2021-01-26 Thread Max Reitz
The following changes since commit 31ee895047bdcf7387e3570cbd2a473c6f744b08: Merge remote-tracking branch 'remotes/jasowang/tags/net-pull-request' into staging (2021-01-25 15:56:13 +) are available in the Git repository at: https://github.com/XanClic/qemu.git tags/pull-block-2021-01-26

Re: [PATCH] iotests/178: Pass value to invalid option

2021-01-26 Thread Kevin Wolf
Am 26.01.2021 um 13:38 hat Max Reitz geschrieben: > ccd3b3b8112 has deprecated short-hand boolean options (i.e., options > with values). All options without values are interpreted as boolean > options, so this includes the invalid option "snapshot.foo" used in > iotest 178. > > So after

Re: [PATCH v9 0/6] Rework iotests/check

2021-01-26 Thread Kevin Wolf
Am 26.01.2021 um 14:19 hat Vladimir Sementsov-Ogievskiy geschrieben: > 26.01.2021 15:53, Kevin Wolf wrote: > > Am 25.01.2021 um 19:50 hat Vladimir Sementsov-Ogievskiy geschrieben: > > > v9: > > > 01: new, one more whitespace-fixing patch > > > testenv: allow case when we don't have system-arch

Re: [PATCH v9 0/6] Rework iotests/check

2021-01-26 Thread Vladimir Sementsov-Ogievskiy
26.01.2021 15:53, Kevin Wolf wrote: Am 25.01.2021 um 19:50 hat Vladimir Sementsov-Ogievskiy geschrieben: v9: 01: new, one more whitespace-fixing patch testenv: allow case when we don't have system-arch emulator, but have several for another architectures change direct os.access(...,

Re: [PATCH v9 0/6] Rework iotests/check

2021-01-26 Thread Kevin Wolf
Am 25.01.2021 um 19:50 hat Vladimir Sementsov-Ogievskiy geschrieben: > v9: > 01: new, one more whitespace-fixing patch > testenv: allow case when we don't have system-arch emulator, but have several > for another architectures > change direct os.access(..., os.X_OK) calls to new helper

[PATCH] iotests/178: Pass value to invalid option

2021-01-26 Thread Max Reitz
ccd3b3b8112 has deprecated short-hand boolean options (i.e., options with values). All options without values are interpreted as boolean options, so this includes the invalid option "snapshot.foo" used in iotest 178. So after ccd3b3b8112, 178 fails with: +qemu-img: warning: short-form boolean

Re: [PATCH] iotests/118: Drop 'change' test

2021-01-26 Thread Kevin Wolf
Am 26.01.2021 um 12:30 hat Paolo Bonzini geschrieben: > On 26/01/21 11:48, Max Reitz wrote: > > Commit 0afec75734331 removed the 'change' QMP command, so we can no > > longer test it in 118. > > > > Fixes: 0afec75734331a0b52fa3aa4235220eda8c7846f > > ('qmp: remove deprecated "change"

Re: [PATCH] iotests/118: Drop 'change' test

2021-01-26 Thread Max Reitz
On 26.01.21 12:18, Kevin Wolf wrote: Am 26.01.2021 um 11:48 hat Max Reitz geschrieben: Commit 0afec75734331 removed the 'change' QMP command, so we can no longer test it in 118. Fixes: 0afec75734331a0b52fa3aa4235220eda8c7846f ('qmp: remove deprecated "change" command') Signed-off-by:

[PATCH v2] hw/block/nvme: add missing mor/mar constraint checks

2021-01-26 Thread Klaus Jensen
From: Klaus Jensen Firstly, if zoned.max_active is non-zero, zoned.max_open must be less than or equal to zoned.max_active. Secondly, if only zones.max_active is set, we have to explicitly set zones.max_open or we end up with an invalid MAR/MOR configuration. This is an artifact of the

[PATCH] hw/block/nvme: add missing mor/mar constraint check

2021-01-26 Thread Klaus Jensen
From: Klaus Jensen The Maximum Open Resources field shall be less than or equal to the Maximum Active Resources field. Verify that constraint and remove the redundant check that the Maximum Open Resources field does not exceed the number of zones. Cc: Dmitry Fomichev Reported-by: Gollu

Re: [PATCH] iotests/118: Drop 'change' test

2021-01-26 Thread Paolo Bonzini
On 26/01/21 11:48, Max Reitz wrote: Commit 0afec75734331 removed the 'change' QMP command, so we can no longer test it in 118. Fixes: 0afec75734331a0b52fa3aa4235220eda8c7846f ('qmp: remove deprecated "change" command') Signed-off-by: Max Reitz --- tests/qemu-iotests/118 | 17

[PATCH V2 2/7] block/rbd: store object_size in BDRVRBDState

2021-01-26 Thread Peter Lieven
Signed-off-by: Peter Lieven --- block/rbd.c | 18 +++--- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/block/rbd.c b/block/rbd.c index a191c74619..1028596c68 100644 --- a/block/rbd.c +++ b/block/rbd.c @@ -90,6 +90,7 @@ typedef struct BDRVRBDState { char *snap;

[PATCH V2 4/7] block/rbd: add bdrv_attach_aio_context

2021-01-26 Thread Peter Lieven
Signed-off-by: Peter Lieven --- block/rbd.c | 15 +-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/block/rbd.c b/block/rbd.c index f68ebcf240..7abd0252c9 100644 --- a/block/rbd.c +++ b/block/rbd.c @@ -91,6 +91,7 @@ typedef struct BDRVRBDState { char *namespace;

[PATCH V2 5/7] block/rbd: migrate from aio to coroutines

2021-01-26 Thread Peter Lieven
Signed-off-by: Peter Lieven --- block/rbd.c | 253 ++-- 1 file changed, 86 insertions(+), 167 deletions(-) diff --git a/block/rbd.c b/block/rbd.c index 7abd0252c9..d11a3c6dd1 100644 --- a/block/rbd.c +++ b/block/rbd.c @@ -66,22 +66,6 @@ typedef

[PATCH V2 0/7] block/rbd: migrate to coroutines and add write zeroes support

2021-01-26 Thread Peter Lieven
this series migrates the qemu rbd driver from the old aio emulation to native coroutines and adds write zeroes support which is important for block operations. To achive this we first bump the librbd requirement to the already outdated luminous release of ceph to get rid of some wrappers and

[PATCH V2 6/7] block/rbd: add write zeroes support

2021-01-26 Thread Peter Lieven
Signed-off-by: Peter Lieven --- block/rbd.c | 36 +++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/block/rbd.c b/block/rbd.c index d11a3c6dd1..35dc1dc90e 100644 --- a/block/rbd.c +++ b/block/rbd.c @@ -63,7 +63,8 @@ typedef enum {

[PATCH V2 3/7] block/rbd: update s->image_size in qemu_rbd_getlength

2021-01-26 Thread Peter Lieven
in case the image size changed we should adjust our internally stored size as well. Signed-off-by: Peter Lieven --- block/rbd.c | 1 + 1 file changed, 1 insertion(+) diff --git a/block/rbd.c b/block/rbd.c index 1028596c68..f68ebcf240 100644 --- a/block/rbd.c +++ b/block/rbd.c @@ -964,6 +964,7

[PATCH V2 1/7] block/rbd: bump librbd requirement to luminous release

2021-01-26 Thread Peter Lieven
even luminous (version 12.2) is unmaintained for over 3 years now. Bump the requirement to get rid of the ifdef'ry in the code. Signed-off-by: Peter Lieven --- block/rbd.c | 120 meson.build | 13 -- 2 files changed, 17 insertions(+),

[PATCH V2 7/7] block/rbd: drop qemu_rbd_refresh_limits

2021-01-26 Thread Peter Lieven
librbd supports 1 byte alignment for all aio operations. Currently, there is no API call to query limits from the ceph backend. So drop the bdrv_refresh_limits completely until there is such an API call. Signed-off-by: Peter Lieven --- block/rbd.c | 9 - 1 file changed, 9 deletions(-)

Re: [PATCH v2 0/2] tests/qtest: Only run fuzz-tests when tested devices are available

2021-01-26 Thread Philippe Mathieu-Daudé
On 1/26/21 12:16 PM, Philippe Mathieu-Daudé wrote: > Some tests/qtest/fuzz-test fail when the device tested is > not available in the build. Fix this by only running the > test when devices are available. Forgot, since v1: - Do not make the testing generic, keep it restricted to x86 (thuth) >

Re: [PATCH] iotests/118: Drop 'change' test

2021-01-26 Thread Kevin Wolf
Am 26.01.2021 um 11:48 hat Max Reitz geschrieben: > Commit 0afec75734331 removed the 'change' QMP command, so we can no > longer test it in 118. > > Fixes: 0afec75734331a0b52fa3aa4235220eda8c7846f >('qmp: remove deprecated "change" command') > Signed-off-by: Max Reitz Thanks, applied to

[PATCH v2 2/2] tests/qtest: Only run fuzz-virtio-scsi when virtio-scsi is available

2021-01-26 Thread Philippe Mathieu-Daudé
This test fails when QEMU is built without the virtio-scsi device, restrict it to its availability. Reviewed-by: Michael S. Tsirkin Signed-off-by: Philippe Mathieu-Daudé --- tests/qtest/fuzz-test.c | 51 tests/qtest/fuzz-virtio-scsi-test.c | 75

[PATCH v2 1/2] tests/qtest: Only run fuzz-megasas-test if megasas device is available

2021-01-26 Thread Philippe Mathieu-Daudé
This test fails when QEMU is built without the megasas device, restrict it to its availability. Signed-off-by: Philippe Mathieu-Daudé --- tests/qtest/fuzz-megasas-test.c | 49 + tests/qtest/fuzz-test.c | 25 - MAINTAINERS

[PATCH v2 0/2] tests/qtest: Only run fuzz-tests when tested devices are available

2021-01-26 Thread Philippe Mathieu-Daudé
Some tests/qtest/fuzz-test fail when the device tested is not available in the build. Fix this by only running the test when devices are available. FWIW Alexander Bulekov suggested an improvement, putting each test in a directory named by the device tested. This series does not cover that.

Re: [PATCH 3/4] tests/qtest: Only run fuzz-megasas-test if megasas device is available

2021-01-26 Thread Philippe Mathieu-Daudé
On 1/15/21 11:39 PM, Alexander Bulekov wrote: > On 210115 1609, Philippe Mathieu-Daudé wrote: >> This test fails when QEMU is built without the megasas device, >> restrict it to its availability. > > Should we just make a separate directory for fuzzer tests and have a > separate source file for

Re: [PATCH 2/4] tests/qtest: Make fuzz-test generic to all targets

2021-01-26 Thread Philippe Mathieu-Daudé
On 1/15/21 11:21 PM, Thomas Huth wrote: > On 15/01/2021 16.09, Philippe Mathieu-Daudé wrote: >> Tests in fuzz-test's main() already check for the supported >> architecture before adding tests, therefore this test is not >> specific to the X86 target. Move it to the generic set. > > As long as it

  1   2   >