Re: [PATCH 2/2] coroutine: Revert to constant batch size

2022-05-12 Thread 成川 弘樹
I'm not sure how much testing is expected for "Tested-by". If just checking my perspective is enough, yes. But I did not check that this patch fixes the problem of excessive resource consumption. On 2022/05/12 21:50, Philippe Mathieu-Daudé wrote: Hi Hiroki, On Thu, May 12, 2022 at 8:57 AM

[RFC PATCH 8/9] iotests: fix source directory location

2022-05-12 Thread John Snow
If you invoke the check script from outside of the tests/qemu-iotests directory, the directories initialized as source_iotests and build_iotests will be incorrect. We can use the location of the source file itself to be more accurate. (I don't know if this is actually *used*, but what was there

[RFC PATCH 7/9] tests: add check-venv to build-tcg-disabled CI recipe

2022-05-12 Thread John Snow
Signed-off-by: John Snow --- .gitlab-ci.d/buildtest.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml index 0aea7ab84c2..5c6201847f1 100644 --- a/.gitlab-ci.d/buildtest.yml +++ b/.gitlab-ci.d/buildtest.yml @@ -245,6 +245,7 @@

[RFC PATCH 4/9] tests: silence pip upgrade warnings during venv creation

2022-05-12 Thread John Snow
Turn off the nag warning coaxing us to upgrade pip. It's not really that interesting to see in CI logs, and as long as nothing is broken -- nothing is broken. Signed-off-by: John Snow --- tests/Makefile.include | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git

[RFC PATCH 3/9] tests: install "qemu" namespace package into venv

2022-05-12 Thread John Snow
This patch adds the "qemu" namespace package to the $build/tests/venv directory. It does so in "editable" mode, which means that changes to the source python directory will actively be reflected by the venv. This patch also then removes any sys.path hacking from the avocado test scripts directly.

[RFC PATCH 9/9] iotests: use tests/venv for running tests

2022-05-12 Thread John Snow
Essentially, this: (A) adjusts the python binary to be the one found in the venv (which is a symlink to the python binary chosen at configure time) (B) adds a new VIRTUAL_ENV export variable (C) changes PATH to front-load the venv binary directory. If the venv directory isn't found, raise a

[RFC PATCH 5/9] tests: use tests/venv to run basevm.py-based scripts

2022-05-12 Thread John Snow
This patch co-opts the virtual environment being used by avocado tests to also run the basevm.py tests. This is being done in preparation for for the qemu.qmp package being removed from qemu.git. As part of the change, remove any sys.path() hacks and treat "qemu" as a normal third-party import.

[RFC PATCH 6/9] tests: add check-venv as a dependency of check and check-block

2022-05-12 Thread John Snow
This patch is being front-loaded before iotests actually relies on the tests/venv being created in order to preserve bisectability. Problems I am aware of here (There are a lot, sorry): - I am not sure the right place to express this dependency, so I did it in tests/Makefile.include. It seems

[RFC PATCH 2/9] tests: add "TESTS_PYTHON" variable to Makefile

2022-05-12 Thread John Snow
This is a convenience feature: $(PYTHON) points to the Python executable we were instructed to use by the configure script. We use that Python to create a virtual environment with the "check-venv" target in tests/Makefile.include. $(TESTS_PYTHON) points to the Python executable belonging to the

[RFC PATCH 1/9] python: update for mypy 0.950

2022-05-12 Thread John Snow
typeshed (included in mypy) recently updated to improve the typing for WriteTransport objects. I was working around this, but now there's a version where I shouldn't work around it. Unfortunately this creates some minor ugliness if I want to support both pre- and post-0.950 versions. For now, for

[RFC PATCH 0/9] tests: run python tests under the build/tests/venv environment

2022-05-12 Thread John Snow
RFC: This is a very early, crude attempt at switching over to an external Python package dependency for QMP. This series does not actually make the switch in and of itself, but instead just switches to the paradigm of using a venv in general to install the QEMU python packages instead of using

[PATCH 7/7] block: Add bdrv_co_pwrite_sync()

2022-05-12 Thread Alberto Faria
Also convert bdrv_pwrite_sync() to being implemented using generated_co_wrapper. Signed-off-by: Alberto Faria --- block/io.c | 5 +++-- include/block/block-io.h | 8 ++-- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/block/io.c b/block/io.c index

[PATCH 5/7] block: Make 'bytes' param of bdrv_co_{pread, pwrite, preadv, pwritev}() an int64_t

2022-05-12 Thread Alberto Faria
For consistency with other I/O functions, and in preparation to implement bdrv_{pread,pwrite}() using generated_co_wrapper. unsigned int fits in int64_t, so all callers remain correct. Signed-off-by: Alberto Faria --- block/coroutines.h | 4 ++-- include/block/block_int-io.h | 4 ++--

[PATCH 4/7] block: Make bdrv_co_pwrite() take a const buffer

2022-05-12 Thread Alberto Faria
It does not mutate the buffer. Signed-off-by: Alberto Faria --- include/block/block_int-io.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/block/block_int-io.h b/include/block/block_int-io.h index bb454200e5..d4d3bed783 100644 --- a/include/block/block_int-io.h +++

[PATCH 6/7] block: Implement bdrv_{pread, pwrite, pwrite_zeroes}() using generated_co_wrapper

2022-05-12 Thread Alberto Faria
Signed-off-by: Alberto Faria --- block/io.c | 41 include/block/block-io.h | 15 +-- 2 files changed, 9 insertions(+), 47 deletions(-) diff --git a/block/io.c b/block/io.c index 78a289192e..ecd1c2a53c 100644 --- a/block/io.c +++

[PATCH 1/7] block: Add a 'flags' param to bdrv_{pread, pwrite, pwrite_sync}()

2022-05-12 Thread Alberto Faria
For consistency with other I/O functions, and in preparation to implement them using generated_co_wrapper. Callers were updated using this Coccinelle script: @@ expression child, offset, buf, bytes; @@ - bdrv_pread(child, offset, buf, bytes) + bdrv_pread(child, offset, buf, bytes, 0)

[PATCH 2/7] block: Change bdrv_{pread, pwrite, pwrite_sync}() param order

2022-05-12 Thread Alberto Faria
Swap 'buf' and 'bytes' around for consistency with bdrv_co_{pread,pwrite}(), and in preparation to implement these functions using generated_co_wrapper. Callers were updated using this Coccinelle script: @@ expression child, offset, buf, bytes, flags; @@ - bdrv_pread(child, offset, buf,

[PATCH 3/7] block: Make bdrv_{pread,pwrite}() return 0 on success

2022-05-12 Thread Alberto Faria
They currently return the value of their 'bytes' parameter on success. Make them return 0 instead, for consistency with other I/O functions and in preparation to implement them using generated_co_wrapper. This also makes it clear that short reads/writes are not possible. The few callers that

[PATCH 0/7] Implement bdrv_{pread, pwrite, pwrite_sync, pwrite_zeroes}() using generated_co_wrapper

2022-05-12 Thread Alberto Faria
Start by making the interfaces of analogous non-coroutine and coroutine functions consistent with each other, then implement the non-coroutine ones using generated_co_wrapper. For the bdrv_pwrite_sync() case, also add the missing bdrv_co_pwrite_sync() function. Alberto Faria (7): block: Add a

Re: [PATCH v2] hw: m25p80: allow write_enable latch get/set

2022-05-12 Thread Peter Delevoryas
> On May 11, 2022, at 10:25 PM, Cédric Le Goater wrote: > > Hello Iris, > > [ Fixing Thomas email ] > > On 5/12/22 02:54, Iris Chen via wrote: >> The write_enable latch property is not currently exposed. >> This commit makes it a modifiable property using get/set methods. >> Signed-off-by:

Re: [PULL 00/10] Block layer patches

2022-05-12 Thread Richard Henderson
On 5/12/22 08:33, Kevin Wolf wrote: The following changes since commit ec11dc41eec5142b4776db1296972c6323ba5847: Merge tag 'pull-misc-2022-05-11' of git://repo.or.cz/qemu/armbru into staging (2022-05-11 09:00:26 -0700) are available in the Git repository at:

Re: [PULL 06/13] nbd: remove peppering of nbd_client_connected

2022-05-12 Thread Peter Maydell
On Tue, 26 Apr 2022 at 21:21, Eric Blake wrote: > > From: Paolo Bonzini > > It is unnecessary to check nbd_client_connected() because every time > s->state is moved out of NBD_CLIENT_CONNECTED the socket is shut down > and all coroutines are resumed. Hi; Coverity points out (CID 1488362) that

Re: [PATCH v2] block/gluster: correctly set max_pdiscard

2022-05-12 Thread Stefano Garzarella
On Thu, May 12, 2022 at 05:44:13PM +0200, Stefano Garzarella wrote: On Thu, May 12, 2022 at 12:30:48PM +0200, Fabian Ebner wrote: On 64-bit platforms, SIZE_MAX is too large for max_pdiscard, which is The main problem is that SIZE_MAX for an int64_t is a negative value. int64_t, and the

Re: [PATCH v2] block/gluster: correctly set max_pdiscard

2022-05-12 Thread Stefano Garzarella
On Thu, May 12, 2022 at 12:30:48PM +0200, Fabian Ebner wrote: On 64-bit platforms, SIZE_MAX is too large for max_pdiscard, which is The main problem is that SIZE_MAX for an int64_t is a negative value. int64_t, and the following assertion would be triggered: qemu-system-x86_64:

[PULL 10/10] qemu-iotests: inline common.config into common.rc

2022-05-12 Thread Kevin Wolf
From: Paolo Bonzini common.rc has some complicated logic to find the common.config that dates back to xfstests and is completely unnecessary now. Just include the contents of the file. Signed-off-by: Paolo Bonzini Message-Id: <20220505094723.732116-1-pbonz...@redhat.com> Signed-off-by: Kevin

[PULL 08/10] qemu-nbd: Pass max connections to blockdev layer

2022-05-12 Thread Kevin Wolf
From: Eric Blake The next patch wants to adjust whether the NBD server code advertises MULTI_CONN based on whether it is known if the server limits to exactly one client. For a server started by QMP, this information is obtained through nbd_server_start (which can support more than one export);

[PULL 05/10] .gitlab-ci.d: export meson testlog.txt as an artifact

2022-05-12 Thread Kevin Wolf
From: Daniel P. Berrangé When running 'make check' we only get a summary of progress on the console. Fortunately meson/ninja have saved the raw test output to a logfile. Exposing this log will make it easier to debug failures that happen in CI. Signed-off-by: Daniel P. Berrangé Message-Id:

[PULL 09/10] nbd/server: Allow MULTI_CONN for shared writable exports

2022-05-12 Thread Kevin Wolf
From: Eric Blake According to the NBD spec, a server that advertises NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will not see any cache inconsistencies: when properly separated by a single flush, actions performed by one client will be visible to another client, regardless

[PULL 04/10] tests/qemu-iotests: print intent to run a test in TAP mode

2022-05-12 Thread Kevin Wolf
From: Daniel P. Berrangé When running I/O tests using TAP output mode, we get a single TAP test with a sub-test reported for each I/O test that is run. The output looks something like this: 1..123 ok qcow2 011 ok qcow2 012 ok qcow2 013 ok qcow2 217 ... If everything runs or fails

[PULL 06/10] hw/block/fdc: Prevent end-of-track overrun (CVE-2021-3507)

2022-05-12 Thread Kevin Wolf
From: Philippe Mathieu-Daudé Per the 82078 datasheet, if the end-of-track (EOT byte in the FIFO) is more than the number of sectors per side, the command is terminated unsuccessfully: * 5.2.5 DATA TRANSFER TERMINATION The 82078 supports terminal count explicitly through the TC pin and

[PULL 07/10] tests/qtest/fdc-test: Add a regression test for CVE-2021-3507

2022-05-12 Thread Kevin Wolf
From: Philippe Mathieu-Daudé Add the reproducer from https://gitlab.com/qemu-project/qemu/-/issues/339 Without the previous commit, when running 'make check-qtest-i386' with QEMU configured with '--enable-sanitizers' we get: ==4028352==ERROR: AddressSanitizer: heap-buffer-overflow on address

[PULL 03/10] iotests/testrunner: Flush after run_test()

2022-05-12 Thread Kevin Wolf
From: Hanna Reitz When stdout is not a terminal, the buffer may not be flushed at each end of line, so we should flush after each test is done. This is especially apparent when run by check-block, in two ways: First, when running make check-block -jX with X > 1, progress indication was

[PULL 02/10] coroutine: Revert to constant batch size

2022-05-12 Thread Kevin Wolf
Commit 4c41c69e changed the way the coroutine pool is sized because for virtio-blk devices with a large queue size and heavy I/O, it was just too small and caused coroutines to be deleted and reallocated soon afterwards. The change made the size dynamic based on the number of queues and the queue

[PULL 01/10] coroutine: Rename qemu_coroutine_inc/dec_pool_size()

2022-05-12 Thread Kevin Wolf
It's true that these functions currently affect the batch size in which coroutines are reused (i.e. moved from the global release pool to the allocation pool of a specific thread), but this is a bug and will be fixed in a separate patch. In fact, the comment in the header file already just

[PULL 00/10] Block layer patches

2022-05-12 Thread Kevin Wolf
The following changes since commit ec11dc41eec5142b4776db1296972c6323ba5847: Merge tag 'pull-misc-2022-05-11' of git://repo.or.cz/qemu/armbru into staging (2022-05-11 09:00:26 -0700) are available in the Git repository at: git://repo.or.cz/qemu/kevin.git tags/for-upstream for you to fetch

Re: [PATCH] qemu-iotests: inline common.config into common.rc

2022-05-12 Thread Kevin Wolf
Am 05.05.2022 um 11:47 hat Paolo Bonzini geschrieben: > common.rc has some complicated logic to find the common.config that > dates back to xfstests and is completely unnecessary now. Just include > the contents of the file. > > Signed-off-by: Paolo Bonzini Thanks, applied to the block branch.

Re: [PATCH 2/2] coroutine: Revert to constant batch size

2022-05-12 Thread Philippe Mathieu-Daudé via
Hi Hiroki, On Thu, May 12, 2022 at 8:57 AM 成川 弘樹 wrote: > > Thank you for your fix. > > I confirmed that after applying this patch, my intended performance > improvement by 4c41c69e is still kept in our environment. Is that equivalent to a formal Tested-by: Hiroki Narukawa tag? > On

Re: [PATCH v2 8/8] qapi: Stop using whitespace for alignment in comments

2022-05-12 Thread Markus Armbruster
Eric Blake writes: > On Tue, May 03, 2022 at 09:37:37AM +0200, Andrea Bolognani wrote: >> Perfectly aligned things look pretty, but keeping them that >> way as the schema evolves requires churn, and in some cases >> newly-added lines are not aligned properly. >> >> Overall, trying to align

Re: [PATCH v4 0/2] nbd: MULTI_CONN for shared writable exports

2022-05-12 Thread Kevin Wolf
Am 12.05.2022 um 02:49 hat Eric Blake geschrieben: > v3 was here: > https://lists.gnu.org/archive/html/qemu-devel/2022-03/msg03701.html > with additional review here: > https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg00166.html Thanks, applied to the block branch. Kevin

Re: [PATCH-for-6.2 0/2] hw/block/fdc: Fix CVE-2021-3507

2022-05-12 Thread Kevin Wolf
Am 03.05.2022 um 18:21 hat Jon Maloy geschrieben: > > > On 5/3/22 05:59, Kevin Wolf wrote: > > Am 23.03.2022 um 03:25 hat John Snow geschrieben: > > > On Fri, Mar 18, 2022 at 2:50 PM Thomas Huth wrote: > > > > On 10/03/2022 18.53, Jon Maloy wrote: > > > > > On 3/10/22 12:14, Thomas Huth wrote:

[PATCH v2] block/gluster: correctly set max_pdiscard

2022-05-12 Thread Fabian Ebner
On 64-bit platforms, SIZE_MAX is too large for max_pdiscard, which is int64_t, and the following assertion would be triggered: qemu-system-x86_64: ../block/io.c:3166: bdrv_co_pdiscard: Assertion `max_pdiscard >= bs->bl.request_alignment' failed. Fixes: 0c8022876f ("block: use int64_t instead of

Re: [PATCH 0/2] ci: improve debuggability of I/O tests

2022-05-12 Thread Kevin Wolf
Am 09.05.2022 um 14:41 hat Daniel P. Berrangé geschrieben: > Currently with the TAP harness we see essentially no useful information > about the I/O tests execution. To pick a random job: > > https://gitlab.com/qemu-project/qemu/-/jobs/2429330423 > > All that we get is this: > > 184/204

Re: [PATCH] iotests/testrunner: Flush after run_test()

2022-05-12 Thread Kevin Wolf
Am 06.05.2022 um 15:42 hat Hanna Reitz geschrieben: > When stdout is not a terminal, the buffer may not be flushed at each end > of line, so we should flush after each test is done. This is especially > apparent when run by check-block, in two ways: > > First, when running make check-block -jX

Re: [PATCH 2/2] coroutine: Revert to constant batch size

2022-05-12 Thread Kevin Wolf
Am 12.05.2022 um 08:56 hat 成川 弘樹 geschrieben: > Thank you for your fix. > > I confirmed that after applying this patch, my intended performance > improvement by 4c41c69e is still kept in our environment. This is good news. Thank you for testing the patch! Kevin

[PATCH] hw/nvme: clear aen mask on reset

2022-05-12 Thread Klaus Jensen
From: Klaus Jensen The internally maintained AEN mask is not cleared on reset. Fix this. Signed-off-by: Klaus Jensen --- hw/nvme/ctrl.c | 1 + 1 file changed, 1 insertion(+) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index 1e6e0fcad918..4c8200dfb859 100644 --- a/hw/nvme/ctrl.c +++

Re: [PATCH v2 0/5] hw/nvme: fix namespace identifiers

2022-05-12 Thread Klaus Jensen
On Apr 29 10:33, Klaus Jensen wrote: > From: Klaus Jensen > > The namespace identifiers reported by the controller is kind of a mess. > See [1,2]. > > This series should fix this for both the `-device nvme,drive=...` and > `-device nvme-ns,...` cases. > > [1]:

Re: [PATCH v2] hw: m25p80: allow write_enable latch get/set

2022-05-12 Thread Peter Delevoryas
> On May 11, 2022, at 6:38 PM, Peter Delevoryas wrote: > > > >> On May 11, 2022, at 5:54 PM, Iris Chen wrote: >> >> The write_enable latch property is not currently exposed. >> This commit makes it a modifiable property using get/set methods. >> >> Signed-off-by: Iris Chen >> --- >> Ran

Re: [PATCH v2] hw: m25p80: allow write_enable latch get/set

2022-05-12 Thread Peter Delevoryas
> On May 11, 2022, at 5:54 PM, Iris Chen wrote: > > The write_enable latch property is not currently exposed. > This commit makes it a modifiable property using get/set methods. > > Signed-off-by: Iris Chen > --- > Ran ./scripts/checkpatch.pl on the patch and added a description. Fixed >

[PATCH v2] hw: m25p80: allow write_enable latch get/set

2022-05-12 Thread Iris Chen via
The write_enable latch property is not currently exposed. This commit makes it a modifiable property using get/set methods. Signed-off-by: Iris Chen --- Ran ./scripts/checkpatch.pl on the patch and added a description. Fixed comments regarding DEFINE_PROP_BOOL. hw/block/m25p80.c |

Re: [PATCH 2/2] coroutine: Revert to constant batch size

2022-05-12 Thread 成川 弘樹
Thank you for your fix. I confirmed that after applying this patch, my intended performance improvement by 4c41c69e is still kept in our environment. On 2022/05/11 0:10, Kevin Wolf wrote: Commit 4c41c69e changed the way the coroutine pool is sized because for virtio-blk devices with a large