Re: [PULL 00/19] Memory API patches for 2022-01-18

2022-01-18 Thread Peter Maydell
t; > are available in the Git repository at: > > https://github.com/philmd/qemu.git tags/memory-api-20220118 > > for you to fetch changes up to 9d696cd50442327fd71ec7309e7b0c6fee693b1d: > > docs/devel: add some clarifying text

Re: [PATCH v3 09/19] block: introduce FleecingState class

2022-01-18 Thread Vladimir Sementsov-Ogievskiy
18.01.2022 19:37, Hanna Reitz wrote: On 22.12.21 18:40, Vladimir Sementsov-Ogievskiy wrote: FleecingState represents state shared between copy-before-write filter and upcoming fleecing block driver. Signed-off-by: Vladimir Sementsov-Ogievskiy ---   block/fleecing.h  | 135

Re: iotest 040, 041, intermittent failure in netbsd VM

2022-01-18 Thread John Snow
On Tue, Jan 18, 2022 at 7:13 AM Peter Maydell wrote: > > On Mon, 17 Jan 2022 at 20:35, John Snow wrote: > > I do expect this to print more information on failure than it > > currently is, though (bug somewhere in machine.py, I think). > > Can you please try applying this temporary patch and

Re: [PATCH v2 2/2] iotests/block-status-cache: New test

2022-01-18 Thread Nir Soffer
On Tue, Jan 18, 2022 at 7:12 PM Hanna Reitz wrote: > > Add a new test to verify that want_zero=false block-status calls do not > pollute the block-status cache for want_zero=true calls. > > We check want_zero=true calls and their results using `qemu-img map` > (over NBD), and want_zero=false

[PATCH v2 1/2] block/io: Update BSC only if want_zero is true

2022-01-18 Thread Hanna Reitz
We update the block-status cache whenever we get new information from a bdrv_co_block_status() call to the block driver. However, if we have passed want_zero=false to that call, it may flag areas containing zeroes as data, and so we would update the block-status cache with wrong information.

[PATCH v2 2/2] iotests/block-status-cache: New test

2022-01-18 Thread Hanna Reitz
Add a new test to verify that want_zero=false block-status calls do not pollute the block-status cache for want_zero=true calls. We check want_zero=true calls and their results using `qemu-img map` (over NBD), and want_zero=false calls also using `qemu-img map` over NBD, but using the

Re: [PATCH 2/2] iotests/block-status-cache: New test

2022-01-18 Thread Hanna Reitz
On 17.01.22 18:57, Nir Soffer wrote: On Mon, Jan 17, 2022 at 6:26 PM Hanna Reitz wrote: Add a new test to verify that want_zero=false block-status calls do not pollute the block-status cache for want_zero=true calls. We check want_zero=true calls and their results using `qemu-img map` (over

[PATCH 07/12] block/io.c: introduce bdrv_subtree_drained_{begin/end}_unlocked

2022-01-18 Thread Emanuele Giuseppe Esposito
Same as the locked version, but use BDRV_POLL_UNLOCKED Signed-off-by: Emanuele Giuseppe Esposito --- block/io.c | 50 +--- include/block/block-io.h | 2 ++ 2 files changed, 39 insertions(+), 13 deletions(-) diff --git a/block/io.c b/block/io.c

[PATCH v2 0/2] block/io: Update BSC only if want_zero is true

2022-01-18 Thread Hanna Reitz
Hi, Cover letter from v1: https://lists.nongnu.org/archive/html/qemu-block/2022-01/msg00314.html In v2, I’ve added a comment to the new test explaining why we need to pass --allocation-depth to qemu-nbd, as requested by Nir; and as he has implied, using the short options is kind of silly because

[PATCH 02/12] block/io.c: make bdrv_do_drained_begin_quiesce static and introduce bdrv_drained_begin_no_poll

2022-01-18 Thread Emanuele Giuseppe Esposito
Using bdrv_do_drained_begin_quiesce() in bdrv_child_cb_drained_end() is not a good idea: the callback might be called when running a drain in a coroutine, and bdrv_drained_begin_poll() does not handle that case, resulting in assertion failure. Instead, bdrv_do_drained_begin with no recursion and

[PATCH 11/12] block/io.c: fully enable assert_bdrv_graph_writable

2022-01-18 Thread Emanuele Giuseppe Esposito
As explained in the TODO, complete the function by checking that the node is also drained. In this way, we can ensure that modify the bs is thread safe, as the drain makes sure that no I/O concurrently reads the field, and all writes are under BQL. Signed-off-by: Emanuele Giuseppe Esposito ---

[PATCH 04/12] block.c: bdrv_replace_child_noperm: first call ->attach(), and then add child

2022-01-18 Thread Emanuele Giuseppe Esposito
Doing the opposite can make adding the child node to a non-drained node, as apply_subtree_drain is only done in ->attach() and thus make assert_bdrv_graph_writable fail. This can happen for example during a transaction rollback (test 245, test_io_with_graph_changes): 1. a node is removed from the

[PATCH 03/12] block.c: bdrv_replace_child_noperm: first remove the child, and then call ->detach()

2022-01-18 Thread Emanuele Giuseppe Esposito
Doing the opposite can make ->detach() (more precisely bdrv_unapply_subtree_drain() in bdrv_child_cb_detach) undo the subtree_drain just performed to protect the removal of the child from the graph, thus making the fully-enabled assert_bdrv_graph_writable fail. Note that

[PATCH 06/12] test-bdrv-drain.c: remove test_detach_by_parent_cb()

2022-01-18 Thread Emanuele Giuseppe Esposito
This test uses a callback of an I/O function (blk_aio_preadv) to modify the graph, using bdrv_attach_child. This is simply not allowed anymore. I/O cannot change the graph. Before "block/io.c: make bdrv_do_drained_begin_quiesce static and introduce bdrv_drained_begin_no_poll", the test would

[PATCH 12/12] block.c: additional assert qemu in main tread

2022-01-18 Thread Emanuele Giuseppe Esposito
Add some missing assertion in static functions of block.c Signed-off-by: Emanuele Giuseppe Esposito --- block.c | 2 ++ block/block-backend.c | 3 +++ 2 files changed, 5 insertions(+) diff --git a/block.c b/block.c index 6196c95aae..7961f5a984 100644 --- a/block.c +++ b/block.c

[PATCH 10/12] block.c: add subtree_drains where needed

2022-01-18 Thread Emanuele Giuseppe Esposito
Protect bdrv_replace_child_noperm, as it modifies the graph by adding/removing elements to .children and .parents list of a bs. Use the newly introduced bdrv_subtree_drained_{begin/end}_unlocked drains to achieve that and be free from the aiocontext lock. One important criteria to keep in mind is

[PATCH 08/12] reopen: add a transaction to drain_end nodes picked in bdrv_reopen_parse_file_or_backing

2022-01-18 Thread Emanuele Giuseppe Esposito
Depending on the options given to reopen_state, bdrv_reopen_parse_file_or_backing could pick another bs that could be from another graph, and thus not protected by subtree_drained_begin called by the callers of this function. We can't simply drain-undrain here, because of transactions. To

Re: [PATCH v3 09/19] block: introduce FleecingState class

2022-01-18 Thread Hanna Reitz
On 22.12.21 18:40, Vladimir Sementsov-Ogievskiy wrote: FleecingState represents state shared between copy-before-write filter and upcoming fleecing block driver. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/fleecing.h | 135 ++ block/fleecing.c |

[PATCH 05/12] test-bdrv-drain.c: adapt test to the coming subtree drains

2022-01-18 Thread Emanuele Giuseppe Esposito
There will be 2 problems in this test when we will add subtree drains in bdrv_replace_child_noperm: - First of all, inconsistency between block_job_create under aiocontext lock that internally calls blk_insert_bs and therefore bdrv_replace_child_noperm, and blk_insert_bs that is called two lines

[PATCH 09/12] jobs: ensure sleep in job_sleep_ns is fully performed

2022-01-18 Thread Emanuele Giuseppe Esposito
If a drain happens while a job is sleeping, the timeout gets cancelled and the job continues once the drain ends. This is especially bad for the sleep performed in commit and stream jobs, since that is dictated by ratelimit to maintain a certain speed. Basically the execution path is the

[PATCH 00/12] Removal of Aiocontext lock through drains: protect bdrv_replace_child_noperm.

2022-01-18 Thread Emanuele Giuseppe Esposito
This serie aims to ensure necessary protection to the BlockDriverState .children and .parents lists, modified in block.c:bdrv_replace_child_noperm(). Thanks to the assertion qemu_in_main_thread() introduced in "block layer: split block APIs in global state and I/O", we can verify that these lists

[PATCH 01/12] introduce BDRV_POLL_WHILE_UNLOCKED

2022-01-18 Thread Emanuele Giuseppe Esposito
Same as BDRV_POLL_WHILE, but uses AIO_WAIT_WHILE_UNLOCKED. Signed-off-by: Emanuele Giuseppe Esposito --- include/block/block-global-state.h | 5 + 1 file changed, 5 insertions(+) diff --git a/include/block/block-global-state.h b/include/block/block-global-state.h index

Re: iotest 040, 041, intermittent failure in netbsd VM

2022-01-18 Thread Li Zhang
On 1/18/22 12:08 AM, John Snow wrote: On Mon, Jan 17, 2022 at 3:49 PM Peter Maydell wrote: On Mon, 17 Jan 2022 at 20:35, John Snow wrote: Can you please try applying this temporary patch and running `./check -qcow2 040 041` until you see a breakage and show me the output from that? With

Re: [PATCH v3 07/19] block/dirty-bitmap: introduce bdrv_dirty_bitmap_status()

2022-01-18 Thread Hanna Reitz
On 22.12.21 18:40, Vladimir Sementsov-Ogievskiy wrote: Add a convenient function similar with bdrv_block_status() to get status of dirty bitmap. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/dirty-bitmap.h | 2 ++ include/qemu/hbitmap.h | 11 +++

Re: [PATCH v3 08/19] block/reqlist: add reqlist_wait_all()

2022-01-18 Thread Hanna Reitz
On 22.12.21 18:40, Vladimir Sementsov-Ogievskiy wrote: Add function to wait for all intersecting requests. To be used in the further commit. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/reqlist.h | 8 block/reqlist.c | 8 2 files changed, 16

Re: [RFC PATCH 3/3] tests/qtest/fuzz-sdcard-test: Add reproducer for OSS-Fuzz (Issue 29225)

2022-01-18 Thread Thomas Huth
On 15/12/2021 21.56, Philippe Mathieu-Daudé wrote: Include the qtest reproducer provided by Alexander Bulekov in https://gitlab.com/qemu-project/qemu/-/issues/451. Without the previous commit, we get: $ make check-qtest-i386 ... Running test qtest-i386/fuzz-sdcard-test

[PULL 15/19] hw/dma: Fix format string issues using dma_addr_t

2022-01-18 Thread Philippe Mathieu-Daudé via
Reviewed-by: Richard Henderson Reviewed-by: David Hildenbrand Signed-off-by: Philippe Mathieu-Daudé Message-Id: <2022084309.28637-10-f4...@amsat.org> --- hw/ide/ahci.c| 2 +- hw/rdma/trace-events | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/hw/ide/ahci.c

[PULL 17/19] hw/dma: Use dma_addr_t type definition when relevant

2022-01-18 Thread Philippe Mathieu-Daudé via
From: Philippe Mathieu-Daudé Update the obvious places where dma_addr_t should be used (instead of uint64_t, hwaddr, size_t, int32_t types). This allows to have _addr_t type portable on 32/64-bit hosts. Signed-off-by: Philippe Mathieu-Daudé Reviewed-by: David Hildenbrand Signed-off-by:

[PULL 12/19] hw/dma: Remove CONFIG_USER_ONLY check

2022-01-18 Thread Philippe Mathieu-Daudé via
From: Philippe Mathieu-Daudé DMA API should not be included in user-mode emulation. If so, build should fail. Remove the CONFIG_USER_ONLY check. Signed-off-by: Philippe Mathieu-Daudé Reviewed-by: Richard Henderson Reviewed-by: David Hildenbrand Message-Id:

[PULL 19/19] docs/devel: add some clarifying text for aliases

2022-01-18 Thread Philippe Mathieu-Daudé via
From: Alex Bennée We do mention the limitation of single parenthood for memory_region_add_subregion but lets also make it clear how aliases help solve that conundrum. Signed-off-by: Alex Bennée Reviewed-by: Philippe Mathieu-Daudé Message-Id: <20220110175104.2908956-7-alex.ben...@linaro.org>

Re: iotest 040, 041, intermittent failure in netbsd VM

2022-01-18 Thread Peter Maydell
On Mon, 17 Jan 2022 at 20:35, John Snow wrote: > I do expect this to print more information on failure than it > currently is, though (bug somewhere in machine.py, I think). > Can you please try applying this temporary patch and running `./check > -qcow2 040 041` until you see a breakage and show

[PULL 16/19] hw/dma: Move ScatterGatherEntry / QEMUSGList declarations around

2022-01-18 Thread Philippe Mathieu-Daudé via
In the next commit we will use the dma_addr_t type in the QEMUSGList structure. Since currently dma_addr_t is defined after QEMUSGList, move the declarations to have dma_addr_t defined first. This is a pure code-movement patch. Suggested-by: David Hildenbrand Signed-off-by: Philippe

[PULL 05/19] memory: Make memory_region_is_mapped() succeed when mapped via an alias

2022-01-18 Thread Philippe Mathieu-Daudé via
From: David Hildenbrand memory_region_is_mapped() currently does not return "true" when a memory region is mapped via an alias. Assuming we have: alias (A0) -> alias (A1) -> region (R0) Mapping A0 would currently only make memory_region_is_mapped() succeed on A0, but not on A1 and R0.

[PULL 11/19] hw/pci: Document pci_dma_map()

2022-01-18 Thread Philippe Mathieu-Daudé via
From: Philippe Mathieu-Daudé Signed-off-by: Philippe Mathieu-Daudé Reviewed-by: Michael S. Tsirkin Reviewed-by: Richard Henderson Message-Id: <2022084309.28637-5-f4...@amsat.org> Signed-off-by: Philippe Mathieu-Daudé --- include/hw/pci/pci.h | 12 1 file changed, 12

[PULL 10/19] hw/pci: Restrict pci-bus stub to sysemu

2022-01-18 Thread Philippe Mathieu-Daudé via
From: Philippe Mathieu-Daudé Neither tools nor user-mode emulation require the PCI bus stub. Signed-off-by: Philippe Mathieu-Daudé Reviewed-by: Michael S. Tsirkin Reviewed-by: Richard Henderson Reviewed-by: David Hildenbrand Message-Id: <2022084309.28637-4-f4...@amsat.org>

[PULL 07/19] memory: Fix incorrect calls of log_global_start/stop

2022-01-18 Thread Philippe Mathieu-Daudé via
From: Peter Xu We should only call the log_global_start/stop when the global dirty track bitmask changes from zero<->non-zero. No real issue reported for this yet probably because no immediate user to enable both dirty rate measurement and migration at the same time. However it'll be good to

[PULL 13/19] hw/rdma/rdma_utils: Rename rdma_pci_dma_map 'len' argument

2022-01-18 Thread Philippe Mathieu-Daudé via
From: Philippe Mathieu-Daudé Various APIs use 'pval' naming for 'pointer to val'. rdma_pci_dma_map() uses 'plen' for 'PCI length', but since 'PCI' is already explicit in the function name, simplify and rename the argument 'len'. No logical change. Signed-off-by: Philippe Mathieu-Daudé

[PULL 09/19] hw/nvram: Restrict fw_cfg QOM interface to sysemu and tools

2022-01-18 Thread Philippe Mathieu-Daudé via
From: Philippe Mathieu-Daudé fw_cfg QOM interface is required by system emulation and qemu-storage-daemon. User-mode emulation doesn't need it. Signed-off-by: Philippe Mathieu-Daudé Signed-off-by: Philippe Mathieu-Daudé Reviewed-by: David Hildenbrand Message-Id:

[PULL 14/19] hw/scsi: Rename SCSIRequest::resid as 'residual'

2022-01-18 Thread Philippe Mathieu-Daudé via
From: Philippe Mathieu-Daudé The 'resid' field is slightly confusing and could be interpreted as some ID. Rename it as 'residual' which is clearer to review. No logical change. Signed-off-by: Philippe Mathieu-Daudé Reviewed-by: Richard Henderson Reviewed-by: David Hildenbrand Message-Id:

[PULL 08/19] stubs: Restrict fw_cfg to system emulation

2022-01-18 Thread Philippe Mathieu-Daudé via
fw_cfg_arch_key_name() stub is only required for sysemu. Signed-off-by: Philippe Mathieu-Daudé Reviewed-by: David Hildenbrand Message-Id: <2022084309.28637-2-f4...@amsat.org> --- stubs/meson.build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stubs/meson.build

[PULL 06/19] memory: Update description of memory_region_is_mapped()

2022-01-18 Thread Philippe Mathieu-Daudé via
From: David Hildenbrand Let's update the documentation, making it clearer what the semantics of memory_region_is_mapped() actually are. Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Peter Xu Signed-off-by: David Hildenbrand Message-Id: <20211102164317.45658-4-da...@redhat.com>

[PULL 03/19] memory: Have 'info mtree' remove duplicated Address Space information

2022-01-18 Thread Philippe Mathieu-Daudé via
From: Philippe Mathieu-Daudé Per Peter Maydell [*]: 'info mtree' monitor command was designed on the assumption that there's really only one or two interesting address spaces, and with more recent developments that's just not the case any more. Similarly about how the FlatView are sorted

[PULL 02/19] memory: Split mtree_info() as mtree_info_flatview() + mtree_info_as()

2022-01-18 Thread Philippe Mathieu-Daudé via
From: Philippe Mathieu-Daudé While mtree_info() handles both ASes and flatviews cases, the two cases share basically no code. Split mtree_info() as mtree_info_flatview() + mtree_info_as() to simplify. Suggested-by: Peter Maydell Reviewed-by: David Hildenbrand Reviewed-by: Peter Xu

[PULL 18/19] hw/dma: Let dma_buf_read() / dma_buf_write() propagate MemTxResult

2022-01-18 Thread Philippe Mathieu-Daudé via
From: Philippe Mathieu-Daudé Since commit 292e13142d2, dma_buf_rw() returns a MemTxResult type. Do not discard it, return it to the caller. Pass the previously returned value (the QEMUSGList residual size, which was rarely used) as an optional argument. With this new API, SCSIRequest::residual

[PULL 00/19] Memory API patches for 2022-01-18

2022-01-18 Thread Philippe Mathieu-Daudé via
-20220118 for you to fetch changes up to 9d696cd50442327fd71ec7309e7b0c6fee693b1d: docs/devel: add some clarifying text for aliases (2022-01-18 12:56:29 +0100) Memory API patches - Directly dispatch MemoryRegion alias accesses - Remove

[PULL 01/19] memory: Directly dispatch alias accesses on origin memory region

2022-01-18 Thread Philippe Mathieu-Daudé via
Since commit 2cdfcf272d ("memory: assign MemoryRegionOps to all regions"), all newly created regions are assigned with unassigned_mem_ops (which might be then overwritten). When using aliased container regions, and there is no region mapped at address 0 in the container, the

[PULL 04/19] machine: Use host_memory_backend_is_mapped() in machine_consume_memdev()

2022-01-18 Thread Philippe Mathieu-Daudé via
From: David Hildenbrand memory_region_is_mapped() is the wrong check, we actually want to check whether the backend is already marked mapped. For example, memory regions mapped via an alias, such as NVDIMMs, currently don't make memory_region_is_mapped() return "true". As the machine is

Re: iotest 040, 041, intermittent failure in netbsd VM

2022-01-18 Thread Peter Maydell
On Mon, 17 Jan 2022 at 23:09, John Snow wrote: > Well, today I learned that: > > (1) vm-build-XXX targets use your host system's QEMU to run that VM > (2) my QMP library cannot talk to QEMU 2.11. Whoops, I hadn't realised I was running with that ancient a QEMU. For the scripted runs I set the