[PULL 0/3] hw/nvme updates

2021-09-23 Thread Klaus Jensen
From: Klaus Jensen Hi Peter, The following changes since commit 2c3e83f92d93fbab071b8a96b8ab769b01902475: Merge remote-tracking branch 'remotes/alistair23/tags/pull-riscv-to-apply-20210921' into staging (2021-09-21 10:57:48 -0700) are available in the Git repository at: git://git.infrad

[PULL 2/3] hw/nvme: fix verification of select field in namespace attachment

2021-09-23 Thread Klaus Jensen
From: Naveen Nagar Fix is added to check for reserved value in select field for namespace attachment CC: Minwoo Im Signed-off-by: Naveen Nagar Signed-off-by: Klaus Jensen --- hw/nvme/ctrl.c | 15 --- include/block/nvme.h | 5 + 2 files changed, 17 insertions(+), 3 dele

[PULL 3/3] hw/nvme: Return error for fused operations

2021-09-23 Thread Klaus Jensen
From: Pankaj Raghav Currently, FUSED operations are not supported by QEMU. As per the 1.4 SPEC, controller should abort the command that requested a fused operation with an INVALID FIELD error code if they are not supported. Changes from v1: Added FUSE flag check also to the admin cmd processing

[PULL 1/3] hw/nvme: fix validation of ASQ and ACQ

2021-09-23 Thread Klaus Jensen
From: Klaus Jensen Address 0x0 is a valid address. Fix the admin submission and completion queue address validation to not error out on this. Signed-off-by: Klaus Jensen Reviewed-by: Keith Busch --- hw/nvme/ctrl.c | 8 hw/nvme/trace-events | 2 -- 2 files changed, 10 deletions(

Re: [PATCH] hw/nvme: reattach subsystem namespaces on hotplug

2021-09-23 Thread Hannes Reinecke
On 9/23/21 10:09 PM, Klaus Jensen wrote: On Sep 9 13:37, Hannes Reinecke wrote: On 9/9/21 12:47 PM, Klaus Jensen wrote: On Sep 9 11:43, Hannes Reinecke wrote: With commit 5ffbaeed16 ("hw/nvme: fix controller hot unplugging") namespaces get moved from the controller to the subsystem if one is

Re: [PATCH] block: introduce max_hw_iov for use in scsi-generic

2021-09-23 Thread Paolo Bonzini
Yes, the question is whether it still exists... Paolo El jue., 23 sept. 2021 16:48, Christian Borntraeger escribió: > > > Am 23.09.21 um 15:04 schrieb Paolo Bonzini: > > Linux limits the size of iovecs to 1024 (UIO_MAXIOV in the kernel > > sources, IOV_MAX in POSIX). Because of this, on some h

Re: [PULL 00/12] jobs: mirror: Handle errors after READY cancel

2021-09-23 Thread Vladimir Sementsov-Ogievskiy
22.09.2021 22:19, Vladimir Sementsov-Ogievskiy wrote: 22.09.2021 19:05, Richard Henderson wrote: On 9/21/21 3:20 AM, Vladimir Sementsov-Ogievskiy wrote: The following changes since commit 326ff8dd09556fc2e257196c49f35009700794ac:    Merge remote-tracking branch 'remotes/jasowang/tags/net-pull-

Re: [PATCH v6 07/11] block: use int64_t instead of int in driver write_zeroes handlers

2021-09-23 Thread Vladimir Sementsov-Ogievskiy
23.09.2021 23:33, Eric Blake wrote: On Fri, Sep 03, 2021 at 01:28:03PM +0300, Vladimir Sementsov-Ogievskiy wrote: We are generally moving to int64_t for both offset and bytes parameters on all io paths. Main motivation is realization of 64-bit write_zeroes operation for fast zeroing large disk

Re: [PATCH v6 10/11] block: use int64_t instead of int in driver discard handlers

2021-09-23 Thread Eric Blake
On Fri, Sep 03, 2021 at 01:28:06PM +0300, Vladimir Sementsov-Ogievskiy wrote: > We are generally moving to int64_t for both offset and bytes parameters > on all io paths. > > Main motivation is realization of 64-bit write_zeroes operation for > fast zeroing large disk chunks, up to the whole disk.

Re: [PATCH v6 07/11] block: use int64_t instead of int in driver write_zeroes handlers

2021-09-23 Thread Eric Blake
On Thu, Sep 23, 2021 at 03:33:45PM -0500, Eric Blake wrote: > > +++ b/block/nbd.c > > @@ -1407,15 +1407,17 @@ static int nbd_client_co_pwritev(BlockDriverState > > *bs, int64_t offset, > > } > > > > static int nbd_client_co_pwrite_zeroes(BlockDriverState *bs, int64_t > > offset, > > -

Re: [PATCH v6 07/11] block: use int64_t instead of int in driver write_zeroes handlers

2021-09-23 Thread Eric Blake
On Fri, Sep 03, 2021 at 01:28:03PM +0300, Vladimir Sementsov-Ogievskiy wrote: > We are generally moving to int64_t for both offset and bytes parameters > on all io paths. > > Main motivation is realization of 64-bit write_zeroes operation for > fast zeroing large disk chunks, up to the whole disk.

Re: [RFC PATCH v2] hw/nvme:Adding Support for namespace management

2021-09-23 Thread Klaus Jensen
On Aug 19 18:39, Naveen Nagar wrote: > From: Naveen > > This patch supports namespace management : create and delete operations. > > Since v1: > - Modified and moved nvme_ns_identify_common in ns.c file > - Added check for CSI field in NS management > - Indentation fix in namespace create > >

Re: [PATCH v2 2/6] iotests: add warning for rogue 'qemu' packages

2021-09-23 Thread Vladimir Sementsov-Ogievskiy
23.09.2021 21:44, John Snow wrote: On Thu, Sep 23, 2021 at 2:32 PM Vladimir Sementsov-Ogievskiy mailto:vsement...@virtuozzo.com>> wrote: 23.09.2021 21:07, John Snow wrote: > Add a warning for when 'iotests' runs against a qemu namespace that > isn't the one in the source tree. Th

Re: [PATCH v2] hw/nvme: Return error for fused operations

2021-09-23 Thread Klaus Jensen
On Sep 15 17:43, Pankaj Raghav wrote: > Currently, FUSED operations are not supported by QEMU. As per the 1.4 SPEC, > controller should abort the command that requested a fused operation with > an INVALID FIELD error code if they are not supported. > > Changes from v1: > Added FUSE flag check als

Re: [PATCH] hw/nvme: reattach subsystem namespaces on hotplug

2021-09-23 Thread Klaus Jensen
On Sep 9 13:37, Hannes Reinecke wrote: > On 9/9/21 12:47 PM, Klaus Jensen wrote: > > On Sep 9 11:43, Hannes Reinecke wrote: > >> With commit 5ffbaeed16 ("hw/nvme: fix controller hot unplugging") > >> namespaces get moved from the controller to the subsystem if one > >> is specified. > >> That kee

Re: [PATCH v6 06/11] block: make BlockLimits::max_pwrite_zeroes 64bit

2021-09-23 Thread Eric Blake
On Fri, Sep 03, 2021 at 01:28:02PM +0300, Vladimir Sementsov-Ogievskiy wrote: > We are going to support 64 bit write-zeroes requests. Now update the > limit variable. It's absolutely safe. The variable is set in some > drivers, and used in bdrv_co_do_pwrite_zeroes(). > > Update also max_write_zero

Re: [PATCH v2 2/6] iotests: add warning for rogue 'qemu' packages

2021-09-23 Thread John Snow
On Thu, Sep 23, 2021 at 2:32 PM Vladimir Sementsov-Ogievskiy < vsement...@virtuozzo.com> wrote: > 23.09.2021 21:07, John Snow wrote: > > Add a warning for when 'iotests' runs against a qemu namespace that > > isn't the one in the source tree. This might occur if you have > > (accidentally) install

Re: [PATCH v2 2/6] iotests: add warning for rogue 'qemu' packages

2021-09-23 Thread Vladimir Sementsov-Ogievskiy
23.09.2021 21:07, John Snow wrote: Add a warning for when 'iotests' runs against a qemu namespace that isn't the one in the source tree. This might occur if you have (accidentally) installed the Python namespace package to your local packages. (I'm not going to say that this is because I bit mys

[PATCH v2 5/6] iotests/migrate-bitmaps-test: delint

2021-09-23 Thread John Snow
Mostly uninteresting stuff. Move the test injections under a function named main() so that the variables used during that process aren't in the global scope. Signed-off-by: John Snow Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Hanna Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed

[PATCH v2 6/6] iotests: Update for pylint 2.11.1

2021-09-23 Thread John Snow
1. Ignore the new f-strings warning, we're not interested in doing a full conversion at this time. 2. Just mute the unbalanced-tuple-unpacking warning, it's not a real error in this case and muting the dozens of callsites is just not worth it. 3. Add encodings to read_text(). Signed-off

[PATCH v2 4/6] iotests/mirror-top-perms: Adjust imports

2021-09-23 Thread John Snow
We need to import subpackages from the qemu namespace package; importing the namespace package alone doesn't bring the subpackages with it -- unless someone else (like iotests.py) imports them too. Adjust the imports. Signed-off-by: John Snow Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Han

[PATCH v2 1/6] iotests: add 'qemu' package location to PYTHONPATH in testenv

2021-09-23 Thread John Snow
We can drop the sys.path hacking in various places by doing this. Additionally, by doing it in one place right up top, we can print interesting warnings in case the environment does not look correct. (See next commit.) If we ever decide to change how the environment is crafted, all of the "help me

[PATCH v2 3/6] iotests/linters: check mypy files all at once

2021-09-23 Thread John Snow
We can circumvent the '__main__' redefinition problem by passing --scripts-are-modules. Take mypy out of the loop per-filename and check everything in one go: it's quite a bit faster. Signed-off-by: John Snow Reviewed-by: Hanna Reitz Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Vladimir Sem

[PATCH v2 0/6] iotests: update environment and linting configuration

2021-09-23 Thread John Snow
GitLab: https://gitlab.com/jsnow/qemu/-/commits/python-package-iotest-pt1 CI: https://gitlab.com/jsnow/qemu/-/pipelines/376236687 This series partially supersedes: [PATCH v3 00/16] python/iotests: Run iotest linters during Python CI' Howdy, this is good stuff we want even if we aren't yet in ag

[PATCH v2 2/6] iotests: add warning for rogue 'qemu' packages

2021-09-23 Thread John Snow
Add a warning for when 'iotests' runs against a qemu namespace that isn't the one in the source tree. This might occur if you have (accidentally) installed the Python namespace package to your local packages. (I'm not going to say that this is because I bit myself with this, but you can fill in th

Re: [PATCH] block: introduce max_hw_iov for use in scsi-generic

2021-09-23 Thread Halil Pasic
On Thu, 23 Sep 2021 16:28:11 +0200 Halil Pasic wrote: > Can't we use some of the established constants instead of hard coding a > qemu specific IOV_MAX? > > POSIX.1 seems to guarantee the availability of IOV_MAX in > according to: https://man7.org/linux/man-pages/man2/readv.2.html > and may ha

Re: [PATCH 5/6] iotests/migrate-bitmaps-test: delint

2021-09-23 Thread Vladimir Sementsov-Ogievskiy
23.09.2021 03:16, John Snow wrote: Mostly uninteresting stuff. Move the test injections under a function named main() so that the variables used during that process aren't in the global scope. Signed-off-by: John Snow Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Hanna Reitz Reviewed-by: Vl

Re: [PATCH 6/6] iotests: Update for pylint 2.11.1

2021-09-23 Thread Vladimir Sementsov-Ogievskiy
23.09.2021 03:16, John Snow wrote: 1. Ignore the new f-strings warning, we're not interested in doing a full conversion at this time. 2. Just mute the unbalanced-tuple-unpacking warning, it's not a real error in this case and muting the dozens of callsites is just not worth it. 3. A

Re: [PATCH 1/6] iotests: add 'qemu' package location to PYTHONPATH in testenv

2021-09-23 Thread John Snow
On Thu, Sep 23, 2021 at 11:20 AM Vladimir Sementsov-Ogievskiy < vsement...@virtuozzo.com> wrote: > 23.09.2021 03:16, John Snow wrote: > > We can drop the sys.path hacking in various places by doing > > this. Additionally, by doing it in one place right up top, we can print > > interesting warnings

Re: [PATCH 2/6] iotests: add warning for rogue 'qemu' packages

2021-09-23 Thread John Snow
On Thu, Sep 23, 2021 at 7:09 AM Daniel P. Berrangé wrote: > On Wed, Sep 22, 2021 at 08:16:21PM -0400, John Snow wrote: > > Add a warning for when 'iotests' runs against a qemu namespace that > > isn't the one in the source tree. This might occur if you have > > (accidentally) installed the Python

Re: [PATCH 3/6] iotests/linters: check mypy files all at once

2021-09-23 Thread Vladimir Sementsov-Ogievskiy
23.09.2021 03:16, John Snow wrote: We can circumvent the '__main__' redefinition problem by passing --scripts-are-modules. Take mypy out of the loop per-filename and check everything in one go: it's quite a bit faster. Signed-off-by: John Snow Reviewed-by: Hanna Reitz Reviewed-by: Vladimir Se

Re: [PATCH 4/6] iotests/mirror-top-perms: Adjust imports

2021-09-23 Thread Vladimir Sementsov-Ogievskiy
23.09.2021 03:16, John Snow wrote: We need to import things from the qemu namespace; importing the namespace alone doesn't bring the submodules with it -- unless someone else (like iotests.py) imports them too. Adjust the imports. Signed-off-by: John Snow Reviewed-by: Philippe Mathieu-Daudé Rev

Re: [PATCH 1/6] iotests: add 'qemu' package location to PYTHONPATH in testenv

2021-09-23 Thread Vladimir Sementsov-Ogievskiy
23.09.2021 03:16, John Snow wrote: We can drop the sys.path hacking in various places by doing this. Additionally, by doing it in one place right up top, we can print interesting warnings in case the environment does not look correct. If we ever decide to change how the environment is crafted, a

Re: [PATCH 2/6] iotests: add warning for rogue 'qemu' packages

2021-09-23 Thread John Snow
On Thu, Sep 23, 2021 at 7:17 AM Kevin Wolf wrote: > Am 23.09.2021 um 12:57 hat Kevin Wolf geschrieben: > > Am 23.09.2021 um 02:16 hat John Snow geschrieben: > > > Add a warning for when 'iotests' runs against a qemu namespace that > > > isn't the one in the source tree. This might occur if you ha

Re: [PATCH] block: introduce max_hw_iov for use in scsi-generic

2021-09-23 Thread Christian Borntraeger
Am 23.09.21 um 15:04 schrieb Paolo Bonzini: Linux limits the size of iovecs to 1024 (UIO_MAXIOV in the kernel sources, IOV_MAX in POSIX). Because of this, on some host adapters requests with many iovecs are rejected with -EINVAL by the io_submit() or readv()/writev() system calls. In fact, t

[PATCH 3/3] linux-aio: add `dev_max_batch` parameter to laio_io_unplug()

2021-09-23 Thread Stefano Garzarella
Between the submission of a request and the unplug, other devices with larger limits may have been queued new requests without flushing the batch. Using the new `dev_max_batch` parameter, laio_io_unplug() can check if the batch exceeds the device limit to flush the current batch. Signed-off-by: S

[PATCH 1/3] file-posix: add `aio-max-batch` option

2021-09-23 Thread Stefano Garzarella
Commit d7ddd0a161 ("linux-aio: limit the batch size using `aio-max-batch` parameter") added a way to limit the batch size of Linux AIO backend for the entire AIO context. The same AIO context can be shared by multiple devices, so latency-sensitive devices may want to limit the batch size even more

[PATCH 2/3] linux-aio: add `dev_max_batch` parameter to laio_co_submit()

2021-09-23 Thread Stefano Garzarella
This new parameter can be used by block devices to limit the Linux AIO batch size more than the limit set by the AIO context. file-posix backend supports this, passing its `aio-max-batch` option previously added. Add an helper function to calculate the maximum batch size. Signed-off-by: Stefano

[PATCH 0/3] linux-aio: allow block devices to limit aio-max-batch

2021-09-23 Thread Stefano Garzarella
Commit d7ddd0a161 ("linux-aio: limit the batch size using `aio-max-batch` parameter") added a way to limit the batch size of Linux AIO backend for the entire AIO context. The same AIO context can be shared by multiple devices, so latency-sensitive devices may want to limit the batch size even more

Re: [PATCH] block: introduce max_hw_iov for use in scsi-generic

2021-09-23 Thread Halil Pasic
On Thu, 23 Sep 2021 09:04:36 -0400 Paolo Bonzini wrote: > Linux limits the size of iovecs to 1024 (UIO_MAXIOV in the kernel > sources, IOV_MAX in POSIX). Because of this, on some host adapters > requests with many iovecs are rejected with -EINVAL by the > io_submit() or readv()/writev() system c

[PATCH] block: introduce max_hw_iov for use in scsi-generic

2021-09-23 Thread Paolo Bonzini
Linux limits the size of iovecs to 1024 (UIO_MAXIOV in the kernel sources, IOV_MAX in POSIX). Because of this, on some host adapters requests with many iovecs are rejected with -EINVAL by the io_submit() or readv()/writev() system calls. In fact, the same limit applies to SG_IO as well. To fix b

Re: [PULL 18/28] file-posix: try BLKSECTGET on block devices too, do not round to power of 2

2021-09-23 Thread Paolo Bonzini
On 23/09/21 14:13, Halil Pasic wrote: On Thu, 23 Sep 2021 12:57:38 +0200 Paolo Bonzini wrote: On 22/09/21 21:51, Halil Pasic wrote: We have figured out what is going on here. The problem seems to be specific to linux aio, which seems to limit the size of the iovec to 1024 (UIO_MAXIOV). Hi H

Re: [PATCH 0/6] iotests: update environment and linting configuration

2021-09-23 Thread Kevin Wolf
Am 23.09.2021 um 02:16 hat John Snow geschrieben: > GitLab: https://gitlab.com/jsnow/qemu/-/commits/python-package-iotest-pt1 > CI: https://gitlab.com/jsnow/qemu/-/pipelines/375630185 > > This series partially supersedes: > [PATCH v3 00/16] python/iotests: Run iotest linters during Python CI' >

Re: [PULL 18/28] file-posix: try BLKSECTGET on block devices too, do not round to power of 2

2021-09-23 Thread Halil Pasic
On Thu, 23 Sep 2021 12:57:38 +0200 Paolo Bonzini wrote: > On 22/09/21 21:51, Halil Pasic wrote: > > We have figured out what is going on here. The problem seems to be > > specific to linux aio, which seems to limit the size of the iovec to > > 1024 (UIO_MAXIOV). > > Hi Halil, > > I'll send a

Re: [PATCH 8/8] qapi: add blockdev-replace command

2021-09-23 Thread Vladimir Sementsov-Ogievskiy
23.09.2021 13:09, Markus Armbruster wrote: Vladimir Sementsov-Ogievskiy writes: Thanks a lot for reviewing! 20.09.2021 09:44, Markus Armbruster wrote: Vladimir Sementsov-Ogievskiy writes: Add command that can add and remove filters. Key points of functionality: What the command does is

Re: [PATCH 2/6] iotests: add warning for rogue 'qemu' packages

2021-09-23 Thread Kevin Wolf
Am 23.09.2021 um 12:57 hat Kevin Wolf geschrieben: > Am 23.09.2021 um 02:16 hat John Snow geschrieben: > > Add a warning for when 'iotests' runs against a qemu namespace that > > isn't the one in the source tree. This might occur if you have > > (accidentally) installed the Python namespace package

Re: [PATCH 2/6] iotests: add warning for rogue 'qemu' packages

2021-09-23 Thread Daniel P . Berrangé
On Wed, Sep 22, 2021 at 08:16:21PM -0400, John Snow wrote: > Add a warning for when 'iotests' runs against a qemu namespace that > isn't the one in the source tree. This might occur if you have > (accidentally) installed the Python namespace package to your local > packages. IIUC, it is/was a vali

Re: [PULL 18/28] file-posix: try BLKSECTGET on block devices too, do not round to power of 2

2021-09-23 Thread Paolo Bonzini
On 22/09/21 21:51, Halil Pasic wrote: We have figured out what is going on here. The problem seems to be specific to linux aio, which seems to limit the size of the iovec to 1024 (UIO_MAXIOV). Hi Halil, I'll send a patch shortly to fix this issue. Sorry about the delay as I was busy with KVM

Re: [PATCH 2/6] iotests: add warning for rogue 'qemu' packages

2021-09-23 Thread Kevin Wolf
Am 23.09.2021 um 02:16 hat John Snow geschrieben: > Add a warning for when 'iotests' runs against a qemu namespace that > isn't the one in the source tree. This might occur if you have > (accidentally) installed the Python namespace package to your local > packages. > > (I'm not going to say that

Re: [PATCH 1/6] iotests: add 'qemu' package location to PYTHONPATH in testenv

2021-09-23 Thread Philippe Mathieu-Daudé
On 9/23/21 02:16, John Snow wrote: We can drop the sys.path hacking in various places by doing this. Additionally, by doing it in one place right up top, we can print interesting warnings in case the environment does not look correct. If we ever decide to change how the environment is crafted, a

Re: [PATCH 3/6] iotests/linters: check mypy files all at once

2021-09-23 Thread Philippe Mathieu-Daudé
On 9/23/21 02:16, John Snow wrote: We can circumvent the '__main__' redefinition problem by passing --scripts-are-modules. Take mypy out of the loop per-filename and check everything in one go: it's quite a bit faster. Signed-off-by: John Snow Reviewed-by: Hanna Reitz --- tests/qemu-iotests/

Re: [PATCH 8/8] qapi: add blockdev-replace command

2021-09-23 Thread Markus Armbruster
Vladimir Sementsov-Ogievskiy writes: > Thanks a lot for reviewing! > > 20.09.2021 09:44, Markus Armbruster wrote: >> Vladimir Sementsov-Ogievskiy writes: >> >>> Add command that can add and remove filters. >>> >>> Key points of functionality: >>> >>> What the command does is simply replace some

Recent qemu patch results in aio failures with host DASD disks resulting in guest I/O errors

2021-09-23 Thread Christian Borntraeger
Am 22.09.21 um 21:51 schrieb Halil Pasic: On Mon, 6 Sep 2021 16:24:20 +0200 Halil Pasic wrote: On Fri, 25 Jun 2021 16:18:12 +0200 Paolo Bonzini wrote: bs->sg is only true for character devices, but block devices can also be used with scsi-block and scsi-generic. Unfortunately BLKSECTGET re