Re: [PATCH v2 25/25] python: rename qemu.aqmp to qemu.qmp

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
17.12.2021 00:10, John Snow wrote: On Thu, Dec 16, 2021 at 6:41 AM Vladimir Sementsov-Ogievskiy mailto:vsement...@virtuozzo.com>> wrote: 15.12.2021 22:39, John Snow wrote: > Now that we are fully switched over to the new QMP library, move it back > over the old namespace. This

[RFC PATCH 7/9] iotests/testenv: initialize an iotests venv

2021-12-16 Thread John Snow
Create a venv automatically when running iotests and install the QEMU namespace package to it, which will also pull in qemu.qmp from PyPI. The venv created will always use the same python binary as the one used to launch the check process. Assuming that care is taken to run 'check' with the

[RFC PATCH 9/9] (WIP) Python: update Pipfile

2021-12-16 Thread John Snow
With qemu.qmp gone, we'll need to install it to the Pipenv so that those tests pass. Important; if this still fails, check that you don't have any stale files in python/qemu/qmp/ which could interfere with this process. Signed-off-by: John Snow --- python/Pipfile.lock | 18 +-

[RFC PATCH 3/9] scripts/qmp: Update 'qmp-shell' forwarder stub

2021-12-16 Thread John Snow
The text here is slightly different than the text in other forwarder stubs in this directory, because the only dependency needed here is outside of the QEMU source tree entirely. Signed-off-by: John Snow --- scripts/qmp/qmp-shell | 39 ++- 1 file changed, 34

[RFC PATCH 5/9] scripts/qmp: delete qmp.py script stub

2021-12-16 Thread John Snow
It's been stubbed out for quite some time and nobody yelped, so out it goes. Signed-off-by: John Snow --- scripts/qmp/qmp | 11 --- 1 file changed, 11 deletions(-) delete mode 100755 scripts/qmp/qmp diff --git a/scripts/qmp/qmp b/scripts/qmp/qmp deleted file mode 100755 index

[RFC PATCH 0/9] Python: Switch to externally hosted qemu.qmp dependency

2021-12-16 Thread John Snow
Based-on: <20211215193939.3768033-1-js...@redhat.com> Hi, this series is part of an effort to publish the qemu.qmp package on PyPI. It is the last of three series to complete this work: (1) Switch the new Async QMP library in to python/qemu/qmp (2) Fork python/qemu/qmp out into its own

[RFC PATCH 6/9] scripts: remove sys.path hacks for qemu/qemu.qmp

2021-12-16 Thread John Snow
qemu.qmp can be found via the package index now. Direct the user to just install the package. When utilities from 'qemu' are needed, direct the user to install the package from that directory instead, which will pull in the 'qemu.qmp' package as a dependency. Signed-off-by: John Snow ---

[RFC PATCH 4/9] scripts/qmp: update remaining forwarder stubs

2021-12-16 Thread John Snow
The text for each forwarder is largely the same, though the qom-fuse stub instructs users to install the optional fuse dependency for the qemu package. Signed-off-by: John Snow --- scripts/qmp/qemu-ga-client | 40 - scripts/qmp/qom-fuse | 40

[RFC PATCH 1/9] Python: Update mypy dependency to >= 0.780

2021-12-16 Thread John Snow
A forthcoming commit makes use of the 'venv' package inside of iotests.py, which only has stubs available in the version of typeshed bundled with mypy 0.780 or greater, which increases our minimum dependency from 0.770. Signed-off-by: John Snow --- python/Pipfile.lock | 64

[RFC PATCH 2/9] Python: update isort dependency

2021-12-16 Thread John Snow
A forthcoming commit that deletes python/qemu/qmp and begins using the dependency from PyPI instead changes the way older versions of isort process import priorities. Sticking to isort versions newer than 5.6.0 keep us consistent with the bleeding edge of isort releases. Signed-off-by: John Snow

Re: [PATCH] iotests/testrunner.py: refactor test_field_width

2021-12-16 Thread John Snow
On Fri, Dec 10, 2021 at 3:15 PM Vladimir Sementsov-Ogievskiy < vsement...@virtuozzo.com> wrote: > A lot of Optional[] types doesn't make code beautiful. > test_field_width defaults to 8, but that is never used in the code. > > More over, if we want some default behavior for single call of >

Re: [PATCH 0/8] hw: Have DMA API take MemTxAttrs arg & propagate MemTxResult (part 2)

2021-12-16 Thread John Snow
Looks OK at a very quick glance. Very weak ACK from me. On Thu, Dec 16, 2021 at 7:36 AM Philippe Mathieu-Daudé wrote: > This is the continuation of part 1 (dma_memory API): > https://www.mail-archive.com/qemu-devel@nongnu.org/msg820359.html > > This series update the dma_buf API. > > Based on

Re: [PATCH v2 15/25] scripts/render-block-graph: switch to AQMP

2021-12-16 Thread John Snow
On Thu, Dec 16, 2021 at 5:58 AM Vladimir Sementsov-Ogievskiy < vsement...@virtuozzo.com> wrote: > 15.12.2021 22:39, John Snow wrote: > > Creating an instance of qemu.aqmp.ExecuteError is too involved here, so > > just drop the specificity down to a generic AQMPError. > > s/AQMPError/QMPError/ ?

Re: [PATCH v2 25/25] python: rename qemu.aqmp to qemu.qmp

2021-12-16 Thread John Snow
On Thu, Dec 16, 2021 at 6:41 AM Vladimir Sementsov-Ogievskiy < vsement...@virtuozzo.com> wrote: > 15.12.2021 22:39, John Snow wrote: > > Now that we are fully switched over to the new QMP library, move it back > > over the old namespace. This is being done primarily so that we may > > upload this

Re: [PATCH 0/2] hw/nvme: Fix CVE-2021-3929 (DMA re-entrancy exploitation)

2021-12-16 Thread Klaus Jensen
On Dec 16 20:13, Klaus Jensen wrote: > On Dec 16 18:55, Philippe Mathieu-Daudé wrote: > > Now that the DMA API allow passing MemTxAttrs argument and > > returning MemTxResult (with MEMTX_BUS_ERROR in particular), > > we can restrict the NVMe controller to memories (prohibitting > > accesses by the

Re: [PATCH 0/8] hw: Have DMA API take MemTxAttrs arg & propagate MemTxResult (part 2)

2021-12-16 Thread Klaus Jensen
On Dec 16 13:35, Philippe Mathieu-Daudé wrote: > This is the continuation of part 1 (dma_memory API): > https://www.mail-archive.com/qemu-devel@nongnu.org/msg820359.html > > This series update the dma_buf API. > > Based on "hw: Let the DMA API take a MemTxAttrs argument" > Based-on:

Re: [PATCH 0/2] hw/nvme: Fix CVE-2021-3929 (DMA re-entrancy exploitation)

2021-12-16 Thread Klaus Jensen
On Dec 16 18:55, Philippe Mathieu-Daudé wrote: > Now that the DMA API allow passing MemTxAttrs argument and > returning MemTxResult (with MEMTX_BUS_ERROR in particular), > we can restrict the NVMe controller to memories (prohibitting > accesses by the DMA engine to devices) and block yet another >

Re: [PATCH v3 02/28] glib-compat: Introduce g_memdup2() wrapper

2021-12-16 Thread Philippe Mathieu-Daudé
On 12/16/21 15:11, Alex Bennée wrote: > Philippe Mathieu-Daudé writes: > >> When experimenting raising GLIB_VERSION_MIN_REQUIRED to 2.68 >> (Fedora 34 provides GLib 2.68.1) we get: >> >> hw/virtio/virtio-crypto.c:245:24: error: 'g_memdup' is deprecated: Use >> 'g_memdup2' instead

Re: [PATCH v5 22/31] block_int-common.h: assertion in the callers of BlockDriver function pointers

2021-12-16 Thread Hanna Reitz
On 24.11.21 07:44, Emanuele Giuseppe Esposito wrote: Signed-off-by: Emanuele Giuseppe Esposito Reviewed-by: Stefan Hajnoczi --- block.c| 18 ++ block/create.c | 10 ++ 2 files changed, 28 insertions(+) [...] diff --git a/block/create.c b/block/create.c

Re: [PATCH 2/2] hw/nvme/ctrl: Prohibit DMA accesses to devices (CVE-2021-3929)

2021-12-16 Thread Mauro Matteo Cascella
On Thu, Dec 16, 2021 at 6:55 PM Philippe Mathieu-Daudé wrote: > > Fixes: CVE-2021-3929 RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=2020298 > Reported-by: Qiuhao Li > Signed-off-by: Philippe Mathieu-Daudé > --- > hw/nvme/ctrl.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > >

Re: [PATCH v2 02/25] python/aqmp: handle asyncio.TimeoutError on execute()

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
16.12.2021 20:22, John Snow wrote: On Thu, Dec 16, 2021 at 4:51 AM Vladimir Sementsov-Ogievskiy mailto:vsement...@virtuozzo.com>> wrote: 15.12.2021 22:39, John Snow wrote: > This exception can be injected into any await statement. If we are > canceled via timeout, we want to

Re: [PATCH 2/2] hw/nvme/ctrl: Prohibit DMA accesses to devices (CVE-2021-3929)

2021-12-16 Thread Keith Busch
On Thu, Dec 16, 2021 at 06:55:10PM +0100, Philippe Mathieu-Daudé wrote: > Async DMA requests might access MMIO regions and re-program the > NVMe controller internal registers while DMA requests are still > scheduled or in flight. Avoid that by prohibing the controller > to access non-memories

Re: [PATCH 1/2] hw/nvme/ctrl: Do not ignore DMA access errors

2021-12-16 Thread Keith Busch
On Thu, Dec 16, 2021 at 06:55:09PM +0100, Philippe Mathieu-Daudé wrote: > dma_buf_read/dma_buf_write() return a MemTxResult type. > Do not discard it, propagate the DMA error to the caller. > > Signed-off-by: Philippe Mathieu-Daudé Looks good. Reviewed-by: Keith Busch

[PATCH 2/2] hw/nvme/ctrl: Prohibit DMA accesses to devices (CVE-2021-3929)

2021-12-16 Thread Philippe Mathieu-Daudé
Async DMA requests might access MMIO regions and re-program the NVMe controller internal registers while DMA requests are still scheduled or in flight. Avoid that by prohibing the controller to access non-memories regions. The bug has been audited looking at the following report from Qiuhao Li:

[PATCH 1/2] hw/nvme/ctrl: Do not ignore DMA access errors

2021-12-16 Thread Philippe Mathieu-Daudé
dma_buf_read/dma_buf_write() return a MemTxResult type. Do not discard it, propagate the DMA error to the caller. Signed-off-by: Philippe Mathieu-Daudé --- hw/nvme/ctrl.c | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index

[PATCH 0/2] hw/nvme: Fix CVE-2021-3929 (DMA re-entrancy exploitation)

2021-12-16 Thread Philippe Mathieu-Daudé
Now that the DMA API allow passing MemTxAttrs argument and returning MemTxResult (with MEMTX_BUS_ERROR in particular), we can restrict the NVMe controller to memories (prohibitting accesses by the DMA engine to devices) and block yet another DMA re-entrancy attack. I'll will try to get a

Re: [PATCH v2 12/25] python/machine: permanently switch to AQMP

2021-12-16 Thread John Snow
On Thu, Dec 16, 2021 at 5:52 AM Vladimir Sementsov-Ogievskiy < vsement...@virtuozzo.com> wrote: > 15.12.2021 22:39, John Snow wrote: > > Remove the QEMU_PYTHON_LEGACY_QMP environment variable, making the > > switch permanent. Update Exceptions and import paths as necessary. > > > > Signed-off-by:

Re: [PATCH v2 07/25] python/qmp: switch qemu-ga-client to AQMP

2021-12-16 Thread John Snow
On Thu, Dec 16, 2021 at 5:31 AM Vladimir Sementsov-Ogievskiy < vsement...@virtuozzo.com> wrote: > 15.12.2021 22:39, John Snow wrote: > > Signed-off-by: John Snow > > Not simple to check, how much new behavior is equal to the old one.. And > impossible to check, is everything updated that should

Re: [PATCH v2 06/25] python/qemu-ga-client: update instructions to newer CLI syntax

2021-12-16 Thread John Snow
On Thu, Dec 16, 2021 at 5:14 AM Vladimir Sementsov-Ogievskiy < vsement...@virtuozzo.com> wrote: > I had to search a bit through the history to check this ) > > The commit ccd3b3b8112b670f "qemu-option: warn for short-form boolean > options" may be noted here. > > And may be subject changed to

Re: [PATCH v2 03/25] python/aqmp: copy type definitions from qmp

2021-12-16 Thread John Snow
On Thu, Dec 16, 2021 at 5:19 AM Daniel P. Berrangé wrote: > On Wed, Dec 15, 2021 at 02:39:17PM -0500, John Snow wrote: > > Copy the remaining type definitions from QMP into the qemu.aqmp.legacy > > module. Now, most users don't need to import anything else but > > qemu.aqmp.legacy. > > I'm

Re: [PATCH v2 02/25] python/aqmp: handle asyncio.TimeoutError on execute()

2021-12-16 Thread John Snow
On Thu, Dec 16, 2021 at 7:39 AM Beraldo Leal wrote: > On Wed, Dec 15, 2021 at 02:39:16PM -0500, John Snow wrote: > > This exception can be injected into any await statement. If we are > > canceled via timeout, we want to clear the pending execution record on > > our way out. > > > >

Re: [PATCH v2 02/25] python/aqmp: handle asyncio.TimeoutError on execute()

2021-12-16 Thread John Snow
On Thu, Dec 16, 2021 at 4:51 AM Vladimir Sementsov-Ogievskiy < vsement...@virtuozzo.com> wrote: > 15.12.2021 22:39, John Snow wrote: > > This exception can be injected into any await statement. If we are > > canceled via timeout, we want to clear the pending execution record on > > our way out. >

Re: [PATCH v2 01/25] python/aqmp: add __del__ method to legacy interface

2021-12-16 Thread John Snow
On Thu, Dec 16, 2021 at 4:31 AM Vladimir Sementsov-Ogievskiy < vsement...@virtuozzo.com> wrote: > 15.12.2021 22:39, John Snow wrote: > > asyncio can complain *very* loudly if you forget to back out of things > > gracefully before the garbage collector starts destroying objects that > > contain

Re: [RFC PATCH v2 03/14] job.h: define locked functions

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
16.12.2021 19:48, Stefan Hajnoczi wrote: On Thu, Nov 04, 2021 at 10:53:23AM -0400, Emanuele Giuseppe Esposito wrote: /** Returns whether the job is ready to be completed. */ bool job_is_ready(Job *job); +/** Same as job_is_ready(), but assumes job_lock is held. */ +bool

Re: [PATCH v2 05/25] python/aqmp: rename AQMPError to QMPError

2021-12-16 Thread John Snow
On Thu, Dec 16, 2021 at 5:09 AM Vladimir Sementsov-Ogievskiy < vsement...@virtuozzo.com> wrote: > 15.12.2021 22:39, John Snow wrote: > > This is in preparation for renaming qemu.aqmp to qemu.qmp. I should have > > done this from this from the very beginning, but it's a convenient time > > to make

Re: [RFC PATCH v2 06/14] job.c: make job_event_* functions static

2021-12-16 Thread Stefan Hajnoczi
On Thu, Nov 04, 2021 at 10:53:26AM -0400, Emanuele Giuseppe Esposito wrote: > job_event_* functions can all be static, as they are not used > outside job.c. > > Signed-off-by: Emanuele Giuseppe Esposito > --- > include/qemu/job.h | 6 -- > job.c | 12 ++-- > 2 files

Re: [RFC PATCH v2 03/14] job.h: define locked functions

2021-12-16 Thread Stefan Hajnoczi
On Thu, Nov 04, 2021 at 10:53:23AM -0400, Emanuele Giuseppe Esposito wrote: > /** Returns whether the job is ready to be completed. */ > bool job_is_ready(Job *job); > > +/** Same as job_is_ready(), but assumes job_lock is held. */ > +bool job_is_ready_locked(Job *job); What I see here is

Re: [RFC PATCH v2 04/14] job.h: define unlocked functions

2021-12-16 Thread Stefan Hajnoczi
On Thu, Nov 04, 2021 at 10:53:24AM -0400, Emanuele Giuseppe Esposito wrote: > All these functions assume that the lock is not held, and acquire > it internally. > > These functions will be useful when job_lock is globally applied, > as they will allow callers to access the job struct fields >

Re: [PATCH v5 13/31] block.c: add assertions to static functions

2021-12-16 Thread Emanuele Giuseppe Esposito
On 16/12/2021 17:08, Hanna Reitz wrote: On 24.11.21 07:44, Emanuele Giuseppe Esposito wrote: Following the assertion derived from the API split, propagate the assertion also in the static functions. Signed-off-by: Emanuele Giuseppe Esposito ---   block.c | 45

Re: [PATCH v2 00/25] Python: delete synchronous qemu.qmp package

2021-12-16 Thread John Snow
On Thu, Dec 16, 2021 at 5:51 AM Daniel P. Berrangé wrote: > On Wed, Dec 15, 2021 at 02:39:14PM -0500, John Snow wrote: > > GitLab: https://gitlab.com/jsnow/qemu/-/commits/python-qmp-legacy-switch > > CI: https://gitlab.com/jsnow/qemu/-/pipelines/430491195 > > > > Hi, this series is part of an

Re: [RFC PATCH v2 02/14] job.h: categorize fields in struct Job

2021-12-16 Thread Stefan Hajnoczi
On Thu, Nov 04, 2021 at 10:53:22AM -0400, Emanuele Giuseppe Esposito wrote: > Categorize the fields in struct Job to understand which ones > need to be protected by the job mutex and which don't. > > Signed-off-by: Emanuele Giuseppe Esposito > --- > include/qemu/job.h | 57

Re: [RFC PATCH v2 01/14] job.c: make job_lock/unlock public

2021-12-16 Thread Stefan Hajnoczi
On Thu, Nov 04, 2021 at 10:53:21AM -0400, Emanuele Giuseppe Esposito wrote: > job mutex will be used to protect the job struct elements and list, > replacing AioContext locks. > > Right now use a shared lock for all jobs, in order to keep things > simple. Once the AioContext lock is gone, we can

Re: [PATCH v5 10/31] block.c: modify .attach and .detach callbacks of child_of_bds

2021-12-16 Thread Hanna Reitz
On 16.12.21 17:05, Emanuele Giuseppe Esposito wrote: On 16/12/2021 15:57, Hanna Reitz wrote: On 24.11.21 07:43, Emanuele Giuseppe Esposito wrote: According to the assertions put in the previous patch, we should first drain and then modify the ->children list. In this way we prevent other

Re: [PATCH v5 13/31] block.c: add assertions to static functions

2021-12-16 Thread Hanna Reitz
On 24.11.21 07:44, Emanuele Giuseppe Esposito wrote: Following the assertion derived from the API split, propagate the assertion also in the static functions. Signed-off-by: Emanuele Giuseppe Esposito --- block.c | 45 - 1 file changed, 44

Re: [PATCH v5 10/31] block.c: modify .attach and .detach callbacks of child_of_bds

2021-12-16 Thread Emanuele Giuseppe Esposito
On 16/12/2021 15:57, Hanna Reitz wrote: On 24.11.21 07:43, Emanuele Giuseppe Esposito wrote: According to the assertions put in the previous patch, we should first drain and then modify the ->children list. In this way we prevent other iothreads to read the list while it is being updated.

Re: [PATCH v5 03/31] assertions for block global state API

2021-12-16 Thread Hanna Reitz
On 24.11.21 07:43, Emanuele Giuseppe Esposito wrote: All the global state (GS) API functions will check that qemu_in_main_thread() returns true. If not, it means that the safety of BQL cannot be guaranteed, and they need to be moved to I/O. Signed-off-by: Emanuele Giuseppe Esposito ---

Re: [PATCH v5 10/31] block.c: modify .attach and .detach callbacks of child_of_bds

2021-12-16 Thread Hanna Reitz
On 24.11.21 07:43, Emanuele Giuseppe Esposito wrote: According to the assertions put in the previous patch, we should first drain and then modify the ->children list. In this way we prevent other iothreads to read the list while it is being updated. In this case, moving the drain won't cause

Re: [PATCH v2 24/25] python: re-enable pylint duplicate-code warnings

2021-12-16 Thread Beraldo Leal
On Wed, Dec 15, 2021 at 02:39:38PM -0500, John Snow wrote: > With the old library gone, there's nothing duplicated in the tree, so > the warning suppression can be removed. > > Signed-off-by: John Snow > --- > python/setup.cfg | 1 - > 1 file changed, 1 deletion(-) > > diff --git

Re: [PATCH v3 02/28] glib-compat: Introduce g_memdup2() wrapper

2021-12-16 Thread Alex Bennée
Philippe Mathieu-Daudé writes: > When experimenting raising GLIB_VERSION_MIN_REQUIRED to 2.68 > (Fedora 34 provides GLib 2.68.1) we get: > > hw/virtio/virtio-crypto.c:245:24: error: 'g_memdup' is deprecated: Use > 'g_memdup2' instead [-Werror,-Wdeprecated-declarations] > ... > >

Re: [PATCH v2 23/25] python: remove the old QMP package

2021-12-16 Thread Beraldo Leal
On Wed, Dec 15, 2021 at 02:39:37PM -0500, John Snow wrote: > Thank you for your service! > > Signed-off-by: John Snow > --- > python/PACKAGE.rst | 4 +- > python/README.rst | 2 +- > python/qemu/qmp/README.rst | 9 - > python/qemu/qmp/__init__.py | 396

Re: [PATCH v2 21/25] python/aqmp: fully separate from qmp.QEMUMonitorProtocol

2021-12-16 Thread Beraldo Leal
On Wed, Dec 15, 2021 at 02:39:35PM -0500, John Snow wrote: > After this patch, qemu.aqmp.legacy.QEMUMonitorProtocol no longer > inherits from qemu.qmp.QEMUMonitorProtocol. To do this, several > inherited methods need to be explicitly re-defined. > > Signed-off-by: John Snow > --- >

Re: [PATCH v2 22/25] python/aqmp: copy qmp docstrings to qemu.aqmp.legacy

2021-12-16 Thread Beraldo Leal
On Wed, Dec 15, 2021 at 02:39:36PM -0500, John Snow wrote: > Copy the docstrings out of qemu.qmp, adjusting them as necessary to > more accurately reflect the current state of this class. > > Signed-off-by: John Snow > --- > python/qemu/aqmp/legacy.py | 110 ++---

Re: [PATCH v2 20/25] python/aqmp: take QMPBadPortError and parse_address from qemu.qmp

2021-12-16 Thread Beraldo Leal
On Wed, Dec 15, 2021 at 02:39:34PM -0500, John Snow wrote: > Shift these definitions over from the qmp package to the async qmp > package. > > Signed-off-by: John Snow > --- > python/qemu/aqmp/aqmp_tui.py | 2 +- > python/qemu/aqmp/legacy.py | 30 ++ >

Re: [PATCH v5 05/31] block-backend: special comments for blk_set/get_perm due to fuse

2021-12-16 Thread Hanna Reitz
On 15.12.21 11:13, Emanuele Giuseppe Esposito wrote: On 15/12/2021 09:57, Emanuele Giuseppe Esposito wrote: On 10/12/2021 15:38, Hanna Reitz wrote: On 24.11.21 07:43, Emanuele Giuseppe Esposito wrote: Fuse logic can be classified as I/O, so there is no BQL held during its execution. And

Re: [PATCH v5 09/31] block: introduce assert_bdrv_graph_writable

2021-12-16 Thread Hanna Reitz
On 14.12.21 20:48, Emanuele Giuseppe Esposito wrote: On 10/12/2021 18:43, Hanna Reitz wrote: On 24.11.21 07:43, Emanuele Giuseppe Esposito wrote: We want to be sure that the functions that write the child and parent list of a bs are under BQL and drain. BQL prevents from concurrent writings

Re: [PATCH v2 02/25] python/aqmp: handle asyncio.TimeoutError on execute()

2021-12-16 Thread Beraldo Leal
On Wed, Dec 15, 2021 at 02:39:16PM -0500, John Snow wrote: > This exception can be injected into any await statement. If we are > canceled via timeout, we want to clear the pending execution record on > our way out. > > Signed-off-by: John Snow > --- > python/qemu/aqmp/qmp_client.py | 8

Re: [PATCH v2 17/25] iotests/mirror-top-perms: switch to AQMP

2021-12-16 Thread Beraldo Leal
On Wed, Dec 15, 2021 at 02:39:31PM -0500, John Snow wrote: > Signed-off-by: John Snow > > --- > > Note: I still need to adjust the logging. The problem now is that the > logging messages include the PID of the test process, so they need to be > filtered out. I'll investigate that for a

Re: [PATCH v2 18/25] iotests: switch to AQMP

2021-12-16 Thread Beraldo Leal
On Wed, Dec 15, 2021 at 02:39:32PM -0500, John Snow wrote: > Simply import the type defition from the new location. > > Signed-off-by: John Snow > --- > tests/qemu-iotests/iotests.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/tests/qemu-iotests/iotests.py

Re: [PATCH v2 16/25] scripts/bench-block-job: switch to AQMP

2021-12-16 Thread Beraldo Leal
On Wed, Dec 15, 2021 at 02:39:30PM -0500, John Snow wrote: > For this commit, we only need to remove accommodations for the > synchronous QMP library. > > Signed-off-by: John Snow > --- > scripts/simplebench/bench_block_job.py | 3 +-- > 1 file changed, 1 insertion(+), 2 deletions(-) > > diff

Re: [PATCH v2 15/25] scripts/render-block-graph: switch to AQMP

2021-12-16 Thread Beraldo Leal
On Wed, Dec 15, 2021 at 02:39:29PM -0500, John Snow wrote: > Creating an instance of qemu.aqmp.ExecuteError is too involved here, so > just drop the specificity down to a generic AQMPError. > > Signed-off-by: John Snow > --- > scripts/render_block_graph.py | 8 +++- > 1 file changed, 3

Re: [PATCH v2 14/25] scripts/cpu-x86-uarch-abi: switch to AQMP

2021-12-16 Thread Beraldo Leal
On Wed, Dec 15, 2021 at 02:39:28PM -0500, John Snow wrote: > Signed-off-by: John Snow > --- > scripts/cpu-x86-uarch-abi.py | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/scripts/cpu-x86-uarch-abi.py b/scripts/cpu-x86-uarch-abi.py > index 8963d90f0b..c262d2f027

Re: [PATCH v2 11/25] python: move qmp-shell under the AQMP package

2021-12-16 Thread Beraldo Leal
On Wed, Dec 15, 2021 at 02:39:25PM -0500, John Snow wrote: > Signed-off-by: John Snow > --- > python/README.rst | 2 +- > python/qemu/{qmp => aqmp}/qmp_shell.py | 0 > python/setup.cfg | 2 +- > scripts/qmp/qmp-shell | 2 +- > 4 files

Re: [PATCH v2 08/25] python/qmp: switch qom tools to AQMP

2021-12-16 Thread Beraldo Leal
On Wed, Dec 15, 2021 at 02:39:22PM -0500, John Snow wrote: > Signed-off-by: John Snow > --- > python/qemu/qmp/qom.py| 5 +++-- > python/qemu/qmp/qom_common.py | 3 ++- > python/qemu/qmp/qom_fuse.py | 11 ++- > 3 files changed, 11 insertions(+), 8 deletions(-) > > diff --git

Re: [PATCH v2 12/25] python/machine: permanently switch to AQMP

2021-12-16 Thread Beraldo Leal
On Wed, Dec 15, 2021 at 02:39:26PM -0500, John Snow wrote: > Remove the QEMU_PYTHON_LEGACY_QMP environment variable, making the > switch permanent. Update Exceptions and import paths as necessary. > > Signed-off-by: John Snow > --- > python/qemu/machine/machine.py | 18 +++--- >

Re: [PATCH v2 07/25] python/qmp: switch qemu-ga-client to AQMP

2021-12-16 Thread Beraldo Leal
On Wed, Dec 15, 2021 at 02:39:21PM -0500, John Snow wrote: > Signed-off-by: John Snow > --- > python/qemu/qmp/qemu_ga_client.py | 22 +++--- > 1 file changed, 11 insertions(+), 11 deletions(-) > > diff --git a/python/qemu/qmp/qemu_ga_client.py >

Re: [PATCH v2 04/25] python/aqmp: add SocketAddrT to package root

2021-12-16 Thread Beraldo Leal
On Wed, Dec 15, 2021 at 02:39:18PM -0500, John Snow wrote: > It's a commonly needed definition, it can be re-exported by the root. > > Signed-off-by: John Snow > --- > python/qemu/aqmp/__init__.py | 10 +- > 1 file changed, 9 insertions(+), 1 deletion(-) > > diff --git

Re: [PATCH v2 01/25] python/aqmp: add __del__ method to legacy interface

2021-12-16 Thread Beraldo Leal
On Wed, Dec 15, 2021 at 02:39:15PM -0500, John Snow wrote: > asyncio can complain *very* loudly if you forget to back out of things > gracefully before the garbage collector starts destroying objects that > contain live references to asyncio Tasks. > > The usual fix is just to remember to call

[PATCH 6/8] dma: Let dma_buf_read() take MemTxAttrs argument

2021-12-16 Thread Philippe Mathieu-Daudé
Let devices specify transaction attributes when calling dma_buf_read(). Keep the default MEMTXATTRS_UNSPECIFIED in the few callers. Signed-off-by: Philippe Mathieu-Daudé --- include/sysemu/dma.h | 2 +- hw/ide/ahci.c | 4 ++-- hw/nvme/ctrl.c| 2 +- hw/scsi/megasas.c |

[PATCH 5/8] dma: Let dma_buf_write() take MemTxAttrs argument

2021-12-16 Thread Philippe Mathieu-Daudé
Let devices specify transaction attributes when calling dma_buf_write(). Keep the default MEMTXATTRS_UNSPECIFIED in the few callers. Signed-off-by: Philippe Mathieu-Daudé --- include/sysemu/dma.h | 2 +- hw/ide/ahci.c | 6 -- hw/nvme/ctrl.c| 3 ++- hw/scsi/megasas.c |

Re: [PATCH 8/8] dma: Let dma_buf_read() / dma_buf_write() propagate MemTxResult

2021-12-16 Thread Philippe Mathieu-Daudé
On 12/16/21 13:35, Philippe Mathieu-Daudé wrote: > Since the previous commit, dma_buf_rw() returns a MemTxResult > type. Do not discard it, return it to the caller. > > Since both dma_buf_read/dma_buf_write functions were previously > returning the QEMUSGList size not consumed, add an extra

[PATCH 4/8] dma: Let dma_buf_rw() take MemTxAttrs argument

2021-12-16 Thread Philippe Mathieu-Daudé
Let devices specify transaction attributes when calling dma_buf_rw(). Keep the default MEMTXATTRS_UNSPECIFIED in the 2 callers. Signed-off-by: Philippe Mathieu-Daudé --- softmmu/dma-helpers.c | 11 ++- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/softmmu/dma-helpers.c

[PATCH 0/8] hw: Have DMA API take MemTxAttrs arg & propagate MemTxResult (part 2)

2021-12-16 Thread Philippe Mathieu-Daudé
This is the continuation of part 1 (dma_memory API): https://www.mail-archive.com/qemu-devel@nongnu.org/msg820359.html This series update the dma_buf API. Based on "hw: Let the DMA API take a MemTxAttrs argument" Based-on: <20210702092439.989969-1-phi...@redhat.com> Philippe Mathieu-Daudé (8):

[PATCH 8/8] dma: Let dma_buf_read() / dma_buf_write() propagate MemTxResult

2021-12-16 Thread Philippe Mathieu-Daudé
Since the previous commit, dma_buf_rw() returns a MemTxResult type. Do not discard it, return it to the caller. Since both dma_buf_read/dma_buf_write functions were previously returning the QEMUSGList size not consumed, add an extra argument where the unconsummed size can be stored. Update the

[PATCH 7/8] dma: Let dma_buf_rw() propagate MemTxResult

2021-12-16 Thread Philippe Mathieu-Daudé
dma_memory_rw() returns a MemTxResult type. Do not discard it, return it to the caller. Since dma_buf_rw() was previously returning the QEMUSGList size not consumed, add an extra argument where this size can be stored. Update the 2 callers. Signed-off-by: Philippe Mathieu-Daudé ---

[PATCH 3/8] dma: Let pci_dma_rw() take MemTxAttrs argument

2021-12-16 Thread Philippe Mathieu-Daudé
Let devices specify transaction attributes when calling pci_dma_rw(). Keep the default MEMTXATTRS_UNSPECIFIED in the few callers. Signed-off-by: Philippe Mathieu-Daudé --- include/hw/pci/pci.h | 10 ++ hw/audio/intel-hda.c | 3 ++- hw/scsi/esp-pci.c| 2 +- 3 files changed, 9

[PATCH 2/8] dma: Have dma_buf_read() / dma_buf_write() take a void pointer

2021-12-16 Thread Philippe Mathieu-Daudé
DMA operations are run on any kind of buffer, not arrays of uint8_t. Convert dma_buf_read/dma_buf_write functions to take a void pointer argument and save us pointless casts to uint8_t *. Remove this pointless casts in the megasas device model. Signed-off-by: Philippe Mathieu-Daudé ---

[PATCH 1/8] dma: Have dma_buf_rw() take a void pointer

2021-12-16 Thread Philippe Mathieu-Daudé
DMA operations are run on any kind of buffer, not arrays of uint8_t. Convert dma_buf_rw() to take a void pointer argument to save us pointless casts to uint8_t *. Signed-off-by: Philippe Mathieu-Daudé --- softmmu/dma-helpers.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git

Re: [PATCH v3 0/3] block: minor refactoring in preparation to the block layer API split

2021-12-16 Thread Kevin Wolf
Am 15.12.2021 um 13:11 hat Emanuele Giuseppe Esposito geschrieben: > These patches are taken from my old patches and feedback of > my series "block layer: split block APIs in global state and I/O". > > The reason for a separate series is that the original one is > already too long, and these

Re: [PATCH v2 25/25] python: rename qemu.aqmp to qemu.qmp

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: Now that we are fully switched over to the new QMP library, move it back over the old namespace. This is being done primarily so that we may upload this package simply as "qemu.qmp" without introducing confusion over whether or not "aqmp" is a new protocol or

Re: [PATCH v2 23/25] python: remove the old QMP package

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: Thank you for your service! Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy -- Best regards, Vladimir

Re: [PATCH v2 24/25] python: re-enable pylint duplicate-code warnings

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: With the old library gone, there's nothing duplicated in the tree, so the warning suppression can be removed. Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy -- Best regards, Vladimir

Re: [PATCH v2 22/25] python/aqmp: copy qmp docstrings to qemu.aqmp.legacy

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: Copy the docstrings out of qemu.qmp, adjusting them as necessary to more accurately reflect the current state of this class. Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy -- Best regards, Vladimir

Re: [PATCH v2 21/25] python/aqmp: fully separate from qmp.QEMUMonitorProtocol

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: After this patch, qemu.aqmp.legacy.QEMUMonitorProtocol no longer inherits from qemu.qmp.QEMUMonitorProtocol. To do this, several inherited methods need to be explicitly re-defined. Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy -- Best

Re: [PATCH v2 20/25] python/aqmp: take QMPBadPortError and parse_address from qemu.qmp

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: Shift these definitions over from the qmp package to the async qmp package. Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy -- Best regards, Vladimir

Re: [PATCH v2 19/25] python: temporarily silence pylint duplicate-code warnings

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: The next several commits copy some code from qemu.qmp to qemu.aqmp, then delete qemu.qmp. In the interim, to prevent test failures, the duplicate code detection needs to be silenced to prevent bisect problems with CI testing. Signed-off-by: John Snow

Re: [PATCH v2 18/25] iotests: switch to AQMP

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: Simply import the type defition from the new location. Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/iotests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git

Re: [PATCH v2 17/25] iotests/mirror-top-perms: switch to AQMP

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy --- Note: I still need to adjust the logging. The problem now is that the logging messages include the PID of the test process, so they need to be filtered out. I'll investigate that

Re: [PATCH v2 16/25] scripts/bench-block-job: switch to AQMP

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: For this commit, we only need to remove accommodations for the synchronous QMP library. Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy -- Best regards, Vladimir

Re: [PATCH v2 15/25] scripts/render-block-graph: switch to AQMP

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: Creating an instance of qemu.aqmp.ExecuteError is too involved here, so just drop the specificity down to a generic AQMPError. s/AQMPError/QMPError/ ? Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy ---

Re: [PATCH v2 14/25] scripts/cpu-x86-uarch-abi: switch to AQMP

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy -- Best regards, Vladimir

Re: [PATCH v2 13/25] scripts/cpu-x86-uarch-abi: fix CLI parsing

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy -- Best regards, Vladimir

Re: [PATCH v2 00/25] Python: delete synchronous qemu.qmp package

2021-12-16 Thread Daniel P . Berrangé
On Wed, Dec 15, 2021 at 02:39:14PM -0500, John Snow wrote: > GitLab: https://gitlab.com/jsnow/qemu/-/commits/python-qmp-legacy-switch > CI: https://gitlab.com/jsnow/qemu/-/pipelines/430491195 > > Hi, this series is part of an effort to publish the qemu.qmp package on > PyPI. It is the first of

Re: [PATCH v2 12/25] python/machine: permanently switch to AQMP

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: Remove the QEMU_PYTHON_LEGACY_QMP environment variable, making the switch permanent. Update Exceptions and import paths as necessary. Signed-off-by: John Snow --- python/qemu/machine/machine.py | 18 +++--- python/qemu/machine/qtest.py | 2

Re: [PATCH v2 11/25] python: move qmp-shell under the AQMP package

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy -- Best regards, Vladimir

Re: [PATCH v2 10/25] python: move qmp utilities to python/qemu/utils

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: In order to upload a QMP package to PyPI, I want to remove any scripts that I am not 100% confident I want to support upstream, beyond our castle walls. Move most of our QMP utilities into the utils package so we can split them out from the PyPI upload.

Re: [PATCH v2 09/25] python/qmp: switch qmp-shell to AQMP

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: We have a replacement for async QMP, but it doesn't have feature parity yet. For now, then, port the old tool onto the new backend. Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy -- Best regards, Vladimir

Re: [PATCH v2 08/25] python/qmp: switch qom tools to AQMP

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy -- Best regards, Vladimir

Re: [RFC PATCH 0/6] Removal of Aiocontext lock and usage of subtree drains in aborted transactions

2021-12-16 Thread Kevin Wolf
Am 15.12.2021 um 13:34 hat Hanna Reitz geschrieben: > On 14.12.21 19:10, Emanuele Giuseppe Esposito wrote: > > > > > > On 13/12/2021 15:52, Stefan Hajnoczi wrote: > > > Off-topic: I don't understand the difference between the effects of > > > bdrv_drained_begin() and

Re: [PATCH v2 07/25] python/qmp: switch qemu-ga-client to AQMP

2021-12-16 Thread Vladimir Sementsov-Ogievskiy
15.12.2021 22:39, John Snow wrote: Signed-off-by: John Snow Not simple to check, how much new behavior is equal to the old one.. And impossible to check, is everything updated that should be ) --- python/qemu/qmp/qemu_ga_client.py | 22 +++--- 1 file changed, 11

Re: [PATCH v2 03/25] python/aqmp: copy type definitions from qmp

2021-12-16 Thread Daniel P . Berrangé
On Wed, Dec 15, 2021 at 02:39:17PM -0500, John Snow wrote: > Copy the remaining type definitions from QMP into the qemu.aqmp.legacy > module. Now, most users don't need to import anything else but > qemu.aqmp.legacy. I'm probably missing the historical discussion but it feels very wierd to be

  1   2   >