Re: [PATCH 0/7] qcow2: compressed write cache

2021-02-09 Thread Max Reitz
On 29.01.21 17:50, Vladimir Sementsov-Ogievskiy wrote: Hi all! I know, I have several series waiting for a resend, but I had to switch to another task spawned from our customer's bug. Original problem: we use O_DIRECT for all vm images in our product, it's the policy. The only exclusion is

Re: [PATCH 07/22] tests/acceptance/virtiofs_submounts.py: evaluate string not length

2021-02-09 Thread Max Reitz
On 09.02.21 12:24, Alex Bennée wrote: Max Reitz writes: On 04.02.21 14:23, Alex Bennée wrote: Cleber Rosa writes: If the vmlinuz variable is set to anything that evaluates to True, then the respective arguments should be set. If the variable contains an empty string, than

Re: [PATCH 07/22] tests/acceptance/virtiofs_submounts.py: evaluate string not length

2021-02-09 Thread Max Reitz
On 04.02.21 14:23, Alex Bennée wrote: Cleber Rosa writes: If the vmlinuz variable is set to anything that evaluates to True, then the respective arguments should be set. If the variable contains an empty string, than it will evaluate to False, and the extra arguments will not be set. This

Re: [PATCH] qemu-img: add seek and -n option to dd command

2021-02-05 Thread Max Reitz
On 05.02.21 10:16, Max Reitz wrote: On 05.02.21 09:47, Peter Lieven wrote: Am 05.02.21 um 09:18 schrieb Max Reitz: On 04.02.21 21:09, Peter Lieven wrote: Am 02.02.21 um 16:51 schrieb Eric Blake: On 1/28/21 8:07 AM, Peter Lieven wrote: Signed-off-by: Peter Lieven Your commit message says

Re: [PATCH] qemu-img: add seek and -n option to dd command

2021-02-05 Thread Max Reitz
On 05.02.21 09:47, Peter Lieven wrote: Am 05.02.21 um 09:18 schrieb Max Reitz: On 04.02.21 21:09, Peter Lieven wrote: Am 02.02.21 um 16:51 schrieb Eric Blake: On 1/28/21 8:07 AM, Peter Lieven wrote: Signed-off-by: Peter Lieven Your commit message says 'what', but not 'why'.  Generally

Re: [PATCH] qemu-img: add seek and -n option to dd command

2021-02-05 Thread Max Reitz
On 04.02.21 21:09, Peter Lieven wrote: Am 02.02.21 um 16:51 schrieb Eric Blake: On 1/28/21 8:07 AM, Peter Lieven wrote: Signed-off-by: Peter Lieven Your commit message says 'what', but not 'why'. Generally, the one-line 'what' works well as the subject line, but you want the commit body to

[Bug 1912224] Re: qemu may freeze during drive-mirroring on fragmented FS

2021-02-03 Thread Max Reitz
I’ve attached a patch to make file-posix cache the information about the last queried hole, does it help? ** Patch added: "0001-file-posix-Cache-next-hole.patch" https://bugs.launchpad.net/qemu/+bug/1912224/+attachment/5459438/+files/0001-file-posix-Cache-next-hole.patch -- You received

[Bug 1912224] Re: qemu may freeze during drive-mirroring on fragmented FS

2021-02-03 Thread Max Reitz
By the way, as a side note: I see you’re using raw, but qcow2 tries to avoid deferring the block-status call to file-posix (i.e., it tries to avoid SEEK_HOLE/DATA calls), because in most cases it knows from its own metadata which block are zero. So I would guess that with qcow2, the problem would

[Bug 1912224] Re: qemu may freeze during drive-mirroring on fragmented FS

2021-02-03 Thread Max Reitz
Hi, As I said on IRC, I’m not sure this additional block_status argument would be good, because the hole offset needs to be reset when the file is written to (at least on zero writes; if we additionally stored a data offset, then that would need to be reset on all writes). Technically, mirror

Re: [PULL 14/53] block: apply COR-filter to block-stream jobs

2021-01-29 Thread Max Reitz
On 29.01.21 06:26, Vladimir Sementsov-Ogievskiy wrote: 28.01.2021 21:38, Philippe Mathieu-Daudé wrote: Hi Andrey, On 1/26/21 3:19 PM, Max Reitz wrote: From: Andrey Shinkevich This patch completes the series with the COR-filter applied to block-stream operations. Adding the filter makes

Re: [PATCH] MAINTAINERS: suggest myself as co-maintainer for Block Jobs

2021-01-28 Thread Max Reitz
On 28.01.21 15:41, Vladimir Sementsov-Ogievskiy wrote: I'm developing Qemu backup for several years, and finally new backup architecture, including block-copy generic engine and backup-top filter landed upstream, great thanks to reviewers and especially to Max Reitz! I also have plans of moving

Re: [PATCH v5 02/10] iotests/297: Rewrite in Python and extend reach

2021-01-27 Thread Max Reitz
On 26.01.21 22:28, John Snow wrote: On 1/18/21 5:57 AM, Max Reitz wrote: Instead of checking iotests.py only, check all Python files in the qemu-iotests/ directory.  Of course, most of them do not pass, so there is an extensive skip list for now.  (The only files that do pass are 209, 254, 283

[PULL 46/53] block/block-copy: drop unused argument of block_copy()

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-21-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- include/block/block-copy.h | 2 +- block/backup-top.c | 2 +- block/block-

[PULL 53/53] iotests/178: Pass value to invalid option

2021-01-26 Thread Max Reitz
rt-form boolean options") Signed-off-by: Max Reitz Message-Id: <20210126123834.115915-1-mre...@redhat.com> --- tests/qemu-iotests/178 | 2 +- tests/qemu-iotests/178.out.qcow2 | 2 +- tests/qemu-iotests/178.out.raw | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-

[PULL 49/53] simplebench: add bench-backup.py

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Add script to benchmark new backup architecture. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20210116214705.822267-24-vsement...@virtuozzo.com> [mreitz: s/not unsupported/not supported/] Signed-off-by: Max Reitz --- scripts/simplebench

[PULL 52/53] iotests/118: Drop 'change' test

2021-01-26 Thread Max Reitz
Commit 0afec75734331 removed the 'change' QMP command, so we can no longer test it in 118. Fixes: 0afec75734331a0b52fa3aa4235220eda8c7846f ('qmp: remove deprecated "change" command') Signed-off-by: Max Reitz Message-Id: <20210126104833.57026-1-mre...@redhat.com> Revie

[PULL 42/53] block/backup: drop extra gotos from backup_run()

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-17-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- block/backup.c | 12 +--- 1 file changed, 5 insertions(+), 7 deletions(-) diff

[PULL 51/53] iotests: Add test for the regression fixed in c8bf9a9169

2021-01-26 Thread Max Reitz
From: Alberto Garcia Signed-off-by: Alberto Garcia Suggested-by: Maxim Levitsky Reviewed-by: Maxim Levitsky Message-Id: <20210112170540.2912-1-be...@igalia.com> [mreitz: Add "# group:" line] Signed-off-by: Max Reitz --- tests/qemu-iotes

[PULL 48/53] simplebench: bench_block_job: add cmd_options argument

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Add argument to allow additional block-job options. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-23-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- scripts/simplebench/bench-example.py

[PULL 50/53] block: report errno when flock fcntl fails

2021-01-26 Thread Max Reitz
process. No errors are elided when removing file locks. Signed-off-by: David Edmondson Message-Id: <20210113164447.2545785-1-david.edmond...@oracle.com> Reviewed-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Max Reitz --- block/file-posix.c | 38 ---

[PULL 36/53] iotests: 56: prepare for backup over block-copy

2021-01-26 Thread Max Reitz
. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-12-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- tests/qemu-iotests/056 | 9 +++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/056 b/test

[PULL 47/53] simplebench/bench_block_job: use correct shebang line with python3

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-22-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- scripts/simplebench/bench_block_job.py | 2 +- 1 file changed, 1 insertion(+), 1 de

[PULL 41/53] block/block-copy: make progress_bytes_callback optional

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy We are going to stop use of this callback in the following commit. Still the callback handling code will be dropped in a separate commit. So, for now let's make it optional. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id

[PULL 45/53] block/block-copy: drop unused block_copy_set_progress_callback()

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Drop unused code. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-20-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- include/block/block-copy.h | 6 -- block/block-copy.c

[PULL 35/53] qapi: backup: add max-chunk and max-workers to x-perf struct

2021-01-26 Thread Max Reitz
, to facilitate further patch which will really change backup job behavior. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-11-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- qapi/block-core.json | 13 - block/ba

[PULL 43/53] backup: move to block-copy

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy This brings async request handling and block-status driven chunk sizes to backup out of the box, which improves backup performance. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-18-vsem

[PULL 34/53] job: call job_enter from job_pause

2021-01-26 Thread Max Reitz
-by: Vladimir Sementsov-Ogievskiy Message-Id: <20210116214705.822267-10-vsement...@virtuozzo.com> Reviewed-by: Max Reitz Signed-off-by: Max Reitz --- job.c | 3 +++ tests/qemu-iotests/109.out | 24 2 files changed, 27 insertions(+) diff --git a

[PULL 44/53] qapi: backup: disable copy_range by default

2021-01-26 Thread Max Reitz
several commits ago with default to true, to follow old behavior (the feature was enabled unconditionally), and only now we are going to change the default behavior. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-19-vsement...@virtuozzo.

[PULL 29/53] block/block-copy: add max_chunk and max_workers parameters

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy They will be used for backup. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-5-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- include/block/block-copy.h | 6 ++ block/block-

[PULL 31/53] block/block-copy: add ratelimit to block-copy

2021-01-26 Thread Max Reitz
-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-7-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- include/block/block-copy.h | 5 - block/backup-top.c | 2 +- block/backup.c | 2 +- block/block-copy.c

[PULL 40/53] iotests: 257: prepare for backup over block-copy

2021-01-26 Thread Max Reitz
things deterministic, allow only one worker (only one copy request at a time) for this test. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-15-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- tests/qemu-iotests/257 | 1 +

[PULL 26/53] qapi: backup: add perf.use-copy-range parameter

2021-01-26 Thread Max Reitz
() new x-perf parameter handled in a way to make further options addition simpler. We add use-copy-range with default=true, and we'll change the default in further patch, after moving backup to use block-copy. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id

[PULL 39/53] iotests: 219: prepare for backup over block-copy

2021-01-26 Thread Max Reitz
it for explicit chunk-size for backup as for mirror. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-14-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- tests/qemu-iotests/219 | 13 +++-- 1 file changed, 7 insertions(+), 6 del

[PULL 30/53] block/block-copy: add list of all call-states

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy It simplifies debugging. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-6-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- block/block-copy.c | 11 ++- 1 file changed, 10 insertions

[PULL 38/53] iotests: 185: prepare for backup over block-copy

2021-01-26 Thread Max Reitz
will handle the situation and wouldn't start new workers when speed limit is obviously reached. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-13-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- tests/qemu-iotests/185

[PULL 33/53] blockjob: add set_speed to BlockJobDriver

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy We are going to use async block-copy call in backup, so we'll need to passthrough setting backup speed to block-copy call. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-9-vsement...@virtuozzo.

[PULL 21/53] iotests/129: Actually test a commit job

2021-01-26 Thread Max Reitz
(and then specifying the source node as the top node for the commit job). Signed-off-by: Max Reitz Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Willian Rampazzo Message-Id: <20210118105720.14824-8-mre...@redhat.com> --- tests/qemu-iotests/12

[PULL 37/53] iotests/129: Limit backup's max-chunk/max-workers

2021-01-26 Thread Max Reitz
Right now, this does not change anything, because backup ignores max-chunk and max-workers. However, as soon as backup is switched over to block-copy for the background copying process, we will need it to keep 129 passing. Signed-off-by: Max Reitz Message-Id: <20210120102043.28346-1-

[PULL 27/53] block/block-copy: More explicit call_state

2021-01-26 Thread Max Reitz
-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-3-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- block/block-copy.c | 51 ++ 1 file changed, 38 insertions(+), 13 deletions(-) diff --git a/block/block-copy.c b/block/block-

[PULL 32/53] block/block-copy: add block_copy_cancel

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Add function to cancel running async block-copy call. It will be used in backup. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-8-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- include

[PULL 19/53] iotests/129: Do not check @busy

2021-01-26 Thread Max Reitz
field (as of 58b295ba52c, i.e. 2018), which can give us exactly that information. Signed-off-by: Max Reitz Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Willian Rampazzo Message-Id: <20210118105720.14824-6-mre...@redhat.com> --- tests/qemu-iotests/129 | 2

[PULL 24/53] iotests/300: Clean up pylint and mypy complaints

2021-01-26 Thread Max Reitz
And consequentially drop it from 297's skip list. Signed-off-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Willian Rampazzo Message-Id: <20210118105720.14824-11-mre...@redhat.com> --- tests/qemu-iotests/297 | 2 +- tests/qemu-iotests/300 | 18 +++

[PULL 15/53] iotests.py: Assume a couple of variables as given

2021-01-26 Thread Max Reitz
it and move the abort code from execute_setup_common() there. Drop the 'assert iotests.sock_dir is not None' from iotest 300, because that sort of thing is precisely what this patch wants to prevent. Signed-off-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Willian Rampazzo

[PULL 28/53] block/block-copy: implement block_copy_async

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy We'll need async block-copy invocation to use in backup directly. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20210116214705.822267-4-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- include/block/block-copy.

[PULL 23/53] iotests/129: Clean up pylint and mypy complaints

2021-01-26 Thread Max Reitz
And consequentially drop it from 297's skip list. Signed-off-by: Max Reitz Reviewed-by: Willian Rampazzo Reviewed-by: Vladimir Sementsov-Ogievskiy Message-Id: <20210118105720.14824-10-mre...@redhat.com> --- tests/qemu-iotests/129 | 4 ++-- tests/qemu-iotests/297 | 2 +- 2 files chan

[PULL 20/53] iotests/129: Use throttle node

2021-01-26 Thread Max Reitz
Throttling on the BB has not affected block jobs in a while, so it is possible that one of the jobs in 129 finishes before the VM is stopped. We can fix that by running the job from a throttle node. Signed-off-by: Max Reitz Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy

[PULL 09/53] copy-on-read: skip non-guest reads if no copy needed

2021-01-26 Thread Max Reitz
to the supported_read_flags of the COR-filter. block: Modify the comment for the flag BDRV_REQ_PREFETCH as we are going to use it alone and pass it to the COR-filter driver for further processing. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id

[PULL 10/53] stream: rework backing-file changing

2021-01-26 Thread Max Reitz
Shinkevich Signed-off-by: Vladimir Sementsov-Ogievskiy [vsementsov: change commit subject, change logic in stream_prepare] Message-Id: <20201216061703.70908-10-vsement...@virtuozzo.com> Reviewed-by: Max Reitz Signed-off-by: Max Reitz --- block/stream.c | 9 + blockdev.c | 8 +

[PULL 25/53] coroutine-sigaltstack: Add SIGUSR2 mutex

2021-01-26 Thread Max Reitz
is more of a fallback implementation for platforms that do not support ucontext, that change may be a bit too invasive to be comfortable with it. The mutex proposed here may negatively impact performance, but the change is much simpler. Signed-off-by: Max Reitz Message-Id: <20210125120305.1952

[PULL 22/53] iotests/129: Limit mirror job's buffer size

2021-01-26 Thread Max Reitz
the backup job to only perform a single request on a very small chunk, too.) Signed-off-by: Max Reitz Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Willian Rampazzo Message-Id: <20210118105720.14824-9-mre...@redhat.com> --- tests/qemu-iotests/129 | 2 +-

[PULL 18/53] iotests/129: Remove test images in tearDown()

2021-01-26 Thread Max Reitz
Signed-off-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake Reviewed-by: Willian Rampazzo Message-Id: <20210118105720.14824-5-mre...@redhat.com> --- tests/qemu-iotests/129 | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/qemu-iotests/129 b/test

[PULL 13/53] block/stream: add s->target_bs

2021-01-26 Thread Max Reitz
From: Vladimir Sementsov-Ogievskiy Add a direct link to target bs for convenience and to simplify following commit which will insert COR filter above target bs. This is a part of original commit written by Andrey. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id

[PULL 08/53] block: include supported_read_flags into BDS structure

2021-01-26 Thread Max Reitz
Signed-off-by: Andrey Shinkevich Signed-off-by: Vladimir Sementsov-Ogievskiy [vsementsov: use assert instead of abort] Reviewed-by: Max Reitz Message-Id: <20201216061703.70908-8-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- include/block/block_int.h | 4 bloc

[PULL 16/53] iotests/297: Rewrite in Python and extend reach

2021-01-26 Thread Max Reitz
y's output is suppressed, because (A) it does not add useful information, and (B) it would leak information about the files having been tested to the reference output, which we decidedly do not want. Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Max Reitz Message-Id: <2021011810572

[PULL 17/53] iotests: Move try_remove to iotests.py

2021-01-26 Thread Max Reitz
Signed-off-by: Max Reitz Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Willian Rampazzo Message-Id: <20210118105720.14824-4-mre...@redhat.com> --- tests/qemu-iotests/124| 8 +--- tests/qemu-iotests/iotests.py | 11 +++ 2 files chan

[PULL 12/53] iotests: 30: prepare to COR filter insertion by stream job

2021-01-26 Thread Max Reitz
to new interface: "bottom" argument instead of base. This guarantees that jobs don't intersect by their actions. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20201216061703.70908-12-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- tests/qemu

[PULL 06/53] qapi: copy-on-read filter: add 'bottom' option

2021-01-26 Thread Max Reitz
From: Andrey Shinkevich Add an option to limit copy-on-read operations to specified sub-chain of backing-chain, to make copy-on-read filter useful for block-stream job. Suggested-by: Max Reitz Suggested-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Andrey Shinkevich Signed-off-by: Vladimir

[PULL 05/53] qapi: add filter-node-name to block-stream

2021-01-26 Thread Max Reitz
/] Reviewed-by: Max Reitz Message-Id: <20201216061703.70908-5-vsement...@virtuozzo.com> [mreitz: s/commit/stream/] Signed-off-by: Max Reitz --- qapi/block-core.json | 6 ++ include/block/block_int.h | 7 ++- block/monitor/block-hmp-cmds.c | 4 ++-- block/st

[PULL 14/53] block: apply COR-filter to block-stream jobs

2021-01-26 Thread Max Reitz
-by: Vladimir Sementsov-Ogievskiy Message-Id: <20201216061703.70908-14-vsement...@virtuozzo.com> Reviewed-by: Max Reitz Signed-off-by: Max Reitz --- block/stream.c | 105 ++--- tests/qemu-iotests/030 | 8 +-- tests/qemu-iotests/141.out | 2 +- test

[PULL 07/53] iotests: add #310 to test bottom node in COR driver

2021-01-26 Thread Max Reitz
From: Andrey Shinkevich The test case #310 is similar to #216 by Max Reitz. The difference is that the test #310 involves a bottom node to the COR filter driver. Signed-off-by: Andrey Shinkevich Signed-off-by: Vladimir Sementsov-Ogievskiy [vsementsov: detach backing to test reads from top

[PULL 03/53] block: add API function to insert a node

2021-01-26 Thread Max Reitz
From: Andrey Shinkevich Provide API for insertion a node to backing chain. Suggested-by: Max Reitz Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Message-Id: <20201216061703.70908-3-vsement...@virtuozzo.com> Signed-off-by: Max

[PULL 00/53] Block patches

2021-01-26 Thread Max Reitz
David Edmondson (1): block: report errno when flock fcntl fails Max Reitz (14): iotests.py: Assume a couple of variables as given iotests/297: Rewrite in Python and extend reach iotests: Move try_remove to iotests.py iotests/129: Remove test images in tearDown() iotests/129: Do

[PULL 01/53] iotests: fix _check_o_direct

2021-01-26 Thread Max Reitz
Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20201218182012.47607-1-vsement...@virtuozzo.com> Signed-off-by: Max Reitz --- tests/qemu-iotests/common.rc | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/tests/qemu-iotests/common.rc b/tests/qemu-iotests

[PULL 11/53] qapi: block-stream: add "bottom" argument

2021-01-26 Thread Max Reitz
interface. Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20201216061703.70908-11-vsement...@virtuozzo.com> Reviewed-by: Max Reitz Signed-off-by: Max Reitz --- qapi/block-core.json | 12 --- include/block/block_int.h | 1 + block/monitor/block-hmp-cmds.

[PULL 04/53] copy-on-read: add filter drop function

2021-01-26 Thread Max Reitz
be considered as an intermediate solution before the QEMU permission update system has overhauled. Then we are able to implement the API function bdrv_remove_node() on the block generic layer. Signed-off-by: Andrey Shinkevich Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz

[PULL 02/53] copy-on-read: support preadv/pwritev_part functions

2021-01-26 Thread Max Reitz
Signed-off-by: Max Reitz --- block/copy-on-read.c | 28 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/block/copy-on-read.c b/block/copy-on-read.c index 2816e61afe..cb03e0f2d3 100644 --- a/block/copy-on-read.c +++ b/block/copy-on-read.c @@ -74,21

Re: [PATCH] coroutine-sigaltstack: Add SIGUSR2 mutex

2021-01-26 Thread Max Reitz
On 26.01.21 13:44, Vladimir Sementsov-Ogievskiy wrote: 25.01.2021 15:03, Max Reitz wrote: Disposition (action) for any given signal is global for the process. When two threads run coroutine-sigaltstack's qemu_coroutine_new() concurrently, they may interfere with each other: One of them may

[PATCH] iotests/178: Pass value to invalid option

2021-01-26 Thread Max Reitz
rt-form boolean options") Signed-off-by: Max Reitz --- tests/qemu-iotests/178 | 2 +- tests/qemu-iotests/178.out.qcow2 | 2 +- tests/qemu-iotests/178.out.raw | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/qemu-iotests/178 b/tests/qemu-iotests/

Re: [PATCH] iotests/118: Drop 'change' test

2021-01-26 Thread Max Reitz
On 26.01.21 12:18, Kevin Wolf wrote: Am 26.01.2021 um 11:48 hat Max Reitz geschrieben: Commit 0afec75734331 removed the 'change' QMP command, so we can no longer test it in 118. Fixes: 0afec75734331a0b52fa3aa4235220eda8c7846f ('qmp: remove deprecated "change" command')

[PATCH] iotests/118: Drop 'change' test

2021-01-26 Thread Max Reitz
Commit 0afec75734331 removed the 'change' QMP command, so we can no longer test it in 118. Fixes: 0afec75734331a0b52fa3aa4235220eda8c7846f ('qmp: remove deprecated "change" command') Signed-off-by: Max Reitz --- tests/qemu-iotests/118 | 17 - tests/qemu-i

[PATCH] coroutine-sigaltstack: Add SIGUSR2 mutex

2021-01-25 Thread Max Reitz
is more of a fallback implementation for platforms that do not support ucontext, that change may be a bit too invasive to be comfortable with it. The mutex proposed here may negatively impact performance, but the change is much simpler. Signed-off-by: Max Reitz --- util/coroutine-sigaltstack.c | 9

Re: [PATCH] coroutine-sigaltstack: Keep SIGUSR2 handler up

2021-01-25 Thread Max Reitz
On 23.01.21 01:41, Laszlo Ersek wrote: On 01/22/21 22:26, Laszlo Ersek wrote: I'm drifting towards an overhaul of coroutine-sigaltstack, based on my personal understanding of POSIX, but given that I can absolutely not *test* coroutine-sigaltstack on the platforms where it actually matters, an

Re: [PATCH] coroutine-sigaltstack: Keep SIGUSR2 handler up

2021-01-22 Thread Max Reitz
On 22.01.21 18:09, Laszlo Ersek wrote: On 01/22/21 11:20, Max Reitz wrote: Modifying signal handlers is a process-global operation. When two threads run coroutine-sigaltstack's qemu_coroutine_new() concurrently, they may interfere with each other: One of them may revert the SIGUSR2 handler

Re: [PATCH] coroutine-sigaltstack: Keep SIGUSR2 handler up

2021-01-22 Thread Max Reitz
On 22.01.21 17:38, Laszlo Ersek wrote: On 01/22/21 11:20, Max Reitz wrote: Modifying signal handlers is a process-global operation. When two threads run coroutine-sigaltstack's qemu_coroutine_new() concurrently, they may interfere with each other: One of them may revert the SIGUSR2 handler

[PATCH] coroutine-sigaltstack: Keep SIGUSR2 handler up

2021-01-22 Thread Max Reitz
but SIGSEGV and SIGBUS, so if it registers a SIGUSR2 handler, sigaltstack coroutines will break from then on. However, we do not use coroutines for user-mode emulation, so that is fine. Suggested-by: Laszlo Ersek Signed-off-by: Max Reitz --- util/coroutine-sigaltstack.c | 56

Re: Thread safety of coroutine-sigaltstack

2021-01-22 Thread Max Reitz
On 22.01.21 11:14, Peter Maydell wrote: On Fri, 22 Jan 2021 at 08:50, Max Reitz wrote: On 20.01.21 18:25, Laszlo Ersek wrote: [...] A simple grep for SIGUSR2 seems to indicate that SIGUSR2 is not used by system emulation for anything else, in practice. Is it possible to dedicate SIGUSR2

Re: Thread safety of coroutine-sigaltstack

2021-01-22 Thread Max Reitz
On 20.01.21 18:25, Laszlo Ersek wrote: [...] A simple grep for SIGUSR2 seems to indicate that SIGUSR2 is not used by system emulation for anything else, in practice. Is it possible to dedicate SIGUSR2 explicitly to coroutine-sigaltstack, and set up the action beforehand, from some init

Re: Thread safety of coroutine-sigaltstack

2021-01-21 Thread Max Reitz
On 21.01.21 14:34, Laszlo Ersek wrote: On 01/21/21 10:27, Max Reitz wrote: On 20.01.21 18:25, Laszlo Ersek wrote: On 01/20/21 17:26, Max Reitz wrote: Hi, I’ve run into trouble with Vladimir’s async backup series on MacOS, namely that iotest 256 fails with qemu exiting because of a SIGUSR2

Re: Thread safety of coroutine-sigaltstack

2021-01-21 Thread Max Reitz
On 20.01.21 18:25, Laszlo Ersek wrote: On 01/20/21 17:26, Max Reitz wrote: Hi, I’ve run into trouble with Vladimir’s async backup series on MacOS, namely that iotest 256 fails with qemu exiting because of a SIGUSR2. Turns out this is because MacOS (-xcode) uses coroutine-sigaltstack, when I

Re: [PATCH v4 00/23] backup performance: block_status + async

2021-01-20 Thread Max Reitz
On 20.01.21 17:04, Daniel P. Berrangé wrote: On Wed, Jan 20, 2021 at 04:53:26PM +0100, Max Reitz wrote: On 20.01.21 15:44, Max Reitz wrote: On 20.01.21 15:34, Max Reitz wrote: [...]  From a glance, it looks to me like two coroutines are created simultaneously in two threads, and so one

Thread safety of coroutine-sigaltstack

2021-01-20 Thread Max Reitz
rom 08d4bb6a98fa731025683f20afe1381291d26031 Mon Sep 17 00:00:00 2001 From: Max Reitz Date: Wed, 20 Jan 2021 16:59:40 +0100 Subject: [RFC] coroutine-sigaltstack: Add SIGUSR2 mutex Modifying signal handlers or the signal handling stack is a process-global operation. When two threads run coroutine-sigaltstac

Re: [PATCH v4 00/23] backup performance: block_status + async

2021-01-20 Thread Max Reitz
On 20.01.21 16:53, Max Reitz wrote: On 20.01.21 15:44, Max Reitz wrote: On 20.01.21 15:34, Max Reitz wrote: [...]  From a glance, it looks to me like two coroutines are created simultaneously in two threads, and so one thread sets up a special SIGUSR2 action, then another reverts SIGUSR2

Re: [PATCH v4 00/23] backup performance: block_status + async

2021-01-20 Thread Max Reitz
On 20.01.21 15:44, Max Reitz wrote: On 20.01.21 15:34, Max Reitz wrote: [...]  From a glance, it looks to me like two coroutines are created simultaneously in two threads, and so one thread sets up a special SIGUSR2 action, then another reverts SIGUSR2 to the default, and then the first

Re: [PATCH v4 00/23] backup performance: block_status + async

2021-01-20 Thread Max Reitz
On 20.01.21 15:34, Max Reitz wrote: On 20.01.21 14:50, Max Reitz wrote: On 20.01.21 11:39, Max Reitz wrote: On 19.01.21 20:22, Vladimir Sementsov-Ogievskiy wrote: 19.01.2021 21:40, Max Reitz wrote: On 16.01.21 22:46, Vladimir Sementsov-Ogievskiy wrote: Hi Max! I applied my series onto yours

Re: [PATCH v4 00/23] backup performance: block_status + async

2021-01-20 Thread Max Reitz
On 20.01.21 14:50, Max Reitz wrote: On 20.01.21 11:39, Max Reitz wrote: On 19.01.21 20:22, Vladimir Sementsov-Ogievskiy wrote: 19.01.2021 21:40, Max Reitz wrote: On 16.01.21 22:46, Vladimir Sementsov-Ogievskiy wrote: Hi Max! I applied my series onto yours 129-fixing and found, that 129 fails

Re: [PATCH v4 00/23] backup performance: block_status + async

2021-01-20 Thread Max Reitz
On 20.01.21 11:39, Max Reitz wrote: On 19.01.21 20:22, Vladimir Sementsov-Ogievskiy wrote: 19.01.2021 21:40, Max Reitz wrote: On 16.01.21 22:46, Vladimir Sementsov-Ogievskiy wrote: Hi Max! I applied my series onto yours 129-fixing and found, that 129 fails for backup. And setting small max

Re: [PATCH v4 00/23] backup performance: block_status + async

2021-01-20 Thread Max Reitz
On 19.01.21 20:22, Vladimir Sementsov-Ogievskiy wrote: 19.01.2021 21:40, Max Reitz wrote: On 16.01.21 22:46, Vladimir Sementsov-Ogievskiy wrote: Hi Max! I applied my series onto yours 129-fixing and found, that 129 fails for backup. And setting small max-chunk and even max-workers to 1

[PATCH v4 11.5/23] iotests/129: Limit backup's max-chunk/max-workers

2021-01-20 Thread Max Reitz
Right now, this does not change anything, because backup ignores max-chunk and max-workers. However, as soon as backup is switched over to block-copy for the background copying process, we will need it to keep 129 passing. Signed-off-by: Max Reitz --- Hi Vladimir, would you be OK

Re: [PATCH v3 02/25] qapi: backup: add perf.use-copy-range parameter

2021-01-20 Thread Max Reitz
On 20.01.21 07:09, Vladimir Sementsov-Ogievskiy wrote: 19.01.2021 23:21, Eric Blake wrote: On 10/26/20 12:17 PM, Vladimir Sementsov-Ogievskiy wrote: Experiments show, that copy_range is not always making things faster. So, to make experimentation simpler, let's add a parameter. Some more perf

Re: [PATCH v4 00/23] backup performance: block_status + async

2021-01-19 Thread Max Reitz
On 16.01.21 22:46, Vladimir Sementsov-Ogievskiy wrote: Hi Max! I applied my series onto yours 129-fixing and found, that 129 fails for backup. And setting small max-chunk and even max-workers to 1 doesn't help! (setting speed like in v3 still helps). And I found, that the problem is that

Re: [PATCH 2/2] virtio-scsi-test: Test writing to scsi-cd device

2021-01-19 Thread Max Reitz
/qtest/virtio-scsi-test.c | 39 ++ 1 file changed, 39 insertions(+) Reviewed-by: Max Reitz

Re: [PATCH 1/2] block: Separate blk_is_writable() and blk_supports_write_perm()

2021-01-19 Thread Max Reitz
| 2 +- hw/scsi/scsi-disk.c| 10 +- hw/scsi/scsi-generic.c | 4 ++-- hw/sd/sd.c | 6 +++--- hw/usb/dev-storage.c | 4 ++-- 20 files changed, 57 insertions(+), 41 deletions(-) Reviewed-by: Max Reitz

Re: [PATCH v2] iotests: Add test for the regression fixed in c8bf9a9169

2021-01-19 Thread Max Reitz
On 12.01.21 18:05, Alberto Garcia wrote: Signed-off-by: Alberto Garcia Suggested-by: Maxim Levitsky Reviewed-by: Maxim Levitsky --- v2: Rebase on top of the latest master tests/qemu-iotests/313 | 103 + tests/qemu-iotests/313.out | 29 +++

Re: [PATCH v5] block: report errno when flock fcntl fails

2021-01-18 Thread Max Reitz
On 13.01.21 17:44, David Edmondson wrote: When a call to fcntl(2) for the purpose of adding file locks fails with an error other than EAGAIN or EACCES, report the error returned by fcntl. EAGAIN or EACCES are elided as they are considered to be common failures, indicating that a conflicting

Re: [PATCH v4 00/23] backup performance: block_status + async

2021-01-18 Thread Max Reitz
On 16.01.21 22:46, Vladimir Sementsov-Ogievskiy wrote: Hi Max! I applied my series onto yours 129-fixing and found, that 129 fails for backup. And setting small max-chunk and even max-workers to 1 doesn't help! (setting speed like in v3 still helps). And I found, that the problem is that

Re: [PATCH v4 09/23] job: call job_enter from job_pause

2021-01-18 Thread Max Reitz
On 18.01.21 14:45, Max Reitz wrote: On 16.01.21 22:46, Vladimir Sementsov-Ogievskiy wrote: If main job coroutine called job_yield (while some background process is in progress), we should give it a chance to call job_pause_point(). It will be used in backup, when moved on async block-copy

Re: [PATCH v4 23/23] simplebench: add bench-backup.py

2021-01-18 Thread Max Reitz
On 16.01.21 22:47, Vladimir Sementsov-Ogievskiy wrote: Add script to benchmark new backup architecture. Signed-off-by: Vladimir Sementsov-Ogievskiy --- scripts/simplebench/bench-backup.py | 167 1 file changed, 167 insertions(+) create mode 100755

Re: [PATCH v4 09/23] job: call job_enter from job_pause

2021-01-18 Thread Max Reitz
On 16.01.21 22:46, Vladimir Sementsov-Ogievskiy wrote: If main job coroutine called job_yield (while some background process is in progress), we should give it a chance to call job_pause_point(). It will be used in backup, when moved on async block-copy. Note, that job_user_pause is not enough:

[PATCH v5 10/10] iotests/300: Clean up pylint and mypy complaints

2021-01-18 Thread Max Reitz
And consequentially drop it from 297's skip list. Signed-off-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Willian Rampazzo --- tests/qemu-iotests/297 | 2 +- tests/qemu-iotests/300 | 18 +++--- 2 files changed, 16 insertions(+), 4 deletions(-) diff --git

[PATCH v5 07/10] iotests/129: Actually test a commit job

2021-01-18 Thread Max Reitz
(and then specifying the source node as the top node for the commit job). Signed-off-by: Max Reitz Reviewed-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Willian Rampazzo --- tests/qemu-iotests/129 | 27 +-- 1 file changed, 25 insertions(+), 2 deletions

<    1   2   3   4   5   6   7   8   9   10   >