Re: [PATCH] iotests: Fix up python style in 300

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
16.02.2021 02:21, John Snow wrote: On 2/15/21 5:05 PM, Eric Blake wrote: Break some long lines, and relax our type hints to be more generic to any JSON, in order to more easily permit the additional JSON depth now possible in migration parameters.  Detected by iotest 297. Fixes: ca4bfec41d56  

Re: [PATCH] iotests: Fix up python style in 300

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
16.02.2021 01:05, Eric Blake wrote: Break some long lines, and relax our type hints to be more generic to any JSON, in order to more easily permit the additional JSON depth now possible in migration parameters. Detected by iotest 297. Fixes: ca4bfec41d56 (qemu-iotests: 300: Add test case for

[RFC PATCH 2/3] hw/block/pflash: Move code around

2021-02-25 Thread Philippe Mathieu-Daudé
First do the block checks, so we know if it is read-only or not. Then create the MemoryRegion. This will allow optimization in the next commit. Signed-off-by: Philippe Mathieu-Daudé --- hw/block/pflash_cfi01.c | 24 hw/block/pflash_cfi02.c | 18 +- 2 file

[RFC PATCH 1/3] exec/memory: Introduce memory_region_init_rom_device_from_file()

2021-02-25 Thread Philippe Mathieu-Daudé
Introduce memory_region_init_rom_device_from_file() which mmap the backing file of ROM devices. This allows to reduce QEMU memory footprint as the same file can be shared between multiple instances of QEMU. Signed-off-by: Philippe Mathieu-Daudé --- include/exec/memory.h | 85

[RFC PATCH 3/3] hw/block/pflash: use memory_region_init_rom_device_from_file()

2021-02-25 Thread Philippe Mathieu-Daudé
If the block drive is read-only we will model a "protected" flash device. We can thus use memory_region_init_rom_device_from_file() which mmap the backing file when creating the MemoryRegion. If the same backing file is used by multiple QEMU instances, this reduces the memory footprint (this is oft

[RFC PATCH 0/3] hw/block/pflash: Mmap read-only backend files with MAP_SHARED

2021-02-25 Thread Philippe Mathieu-Daudé
Hi, This series aims to reduce the memory footprint of flash devices when the backing file is read-only. When a backing file is read-only, the model considers the flash is in "protected" mode. No write are allowed, but the MMIO state machine is still usable. This series introduces a new memory r

Re: [PATCH v2 03/31] qapi/qom: Add ObjectOptions for iothread

2021-02-25 Thread Eric Blake
On 2/24/21 7:52 AM, Kevin Wolf wrote: > Add an ObjectOptions union that will eventually describe the options of > all user creatable object types. As unions can't exist without any > branches, also add the first object type. > > This adds a QAPI schema for the properties of the iothread object. >

Re: [PATCH v2 02/31] qapi/qom: Drop deprecated 'props' from object-add

2021-02-25 Thread Eric Blake
On 2/24/21 7:52 AM, Kevin Wolf wrote: > The option has been deprecated in QEMU 5.0, remove it. > > Signed-off-by: Kevin Wolf > --- > qapi/qom.json| 6 +- > docs/system/deprecated.rst | 5 - > docs/system/removed-features.rst | 5 + > qom/qom-qmp-cmds.c

Re: [PATCH v2 01/31] tests: Drop 'props' from object-add calls

2021-02-25 Thread Eric Blake
On 2/24/21 7:52 AM, Kevin Wolf wrote: > The 'props' option has been deprecated in 5.0 in favour of a flattened > object-add command. Time to change our test cases to drop the deprecated > option. > > Signed-off-by: Kevin Wolf > --- Reviewed-by: Eric Blake -- Eric Blake, Principal Software Eng

Re: [PATCH] nbd: server: Report holes for raw images

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
19.02.2021 19:58, Eric Blake wrote: On 2/19/21 10:42 AM, Eric Blake wrote: To me, data=false looks compatible with NBD_STATE_HOLE. From user point of view, getting same results from qemu-nbd and qemu-img is more important than being more correct about allocation status. More to the point, her

Re: [PATCH] nbd: server: Report holes for raw images

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
19.02.2021 19:07, Nir Soffer wrote: When querying image extents for raw image, qemu-nbd reports holes as zero: $ qemu-nbd -t -r -f raw empty-6g.raw $ qemu-img map --output json nbd://localhost [{ "start": 0, "length": 6442450944, "depth": 0, "zero": true, "data": true, "offset": 0}] $ qemu-im

Re: [PATCH 00/14] deprecations: remove many old deprecations

2021-02-25 Thread Jim Fehlig
Adding xen-devel and Ian to cc. On 2/24/21 6:11 AM, Daniel P. Berrangé wrote: The following features have been deprecated for well over the 2 release cycle we promise This reminded me of a bug report we received late last year when updating to 5.2.0. 'virsh setvcpus' suddenly stopped working

block/throttle and burst bucket

2021-02-25 Thread Peter Lieven
Hi, I was wondering if there is a way to check from outside (qmp etc.) if a throttled block device has exceeded the iops_max_length seconds of time bursting up to iops_max and is now hard limited to the iops limit that is supplied? Would it be also a good idea to exetend the accounting to ac

Re: [PATCH v5 0/9] block: Add retry for werror=/rerror= mechanism

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
23.02.2021 16:41, Eric Blake wrote: On 2/23/21 3:40 AM, Stefan Hajnoczi wrote: On Fri, Feb 05, 2021 at 06:13:06PM +0800, Jiahui Cen wrote: This patch series propose to extend the werror=/rerror= mechanism to add a 'retry' feature. It can automatically retry failed I/O requests on error without

Re: [RFC PATCH v2 3/4] block: Support multiple reopening with x-blockdev-reopen

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
24.02.2021 15:33, Kevin Wolf wrote: Am 09.02.2021 um 09:03 hat Vladimir Sementsov-Ogievskiy geschrieben: 08.02.2021 21:44, Alberto Garcia wrote: Signed-off-by: Alberto Garcia --- qapi/block-core.json | 2 +- include/block/block.h | 1 + block.c| 16 +

Re: [PATCH 2/5] block: Fix BDRV_BLOCK_RAW status to honor alignment

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
25.02.2021 19:03, Eric Blake wrote: On 2/25/21 8:55 AM, Vladimir Sementsov-Ogievskiy wrote: 18.02.2021 23:15, Eric Blake wrote: Previous patches mentioned how the blkdebug filter driver demonstrates a bug present in our NBD server (for example, commit b0245d64); the bug is also present with the

Re: [PATCH 1/5] iotests: Update 241 to expose backing layer fragmentation

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
25.02.2021 18:52, Eric Blake wrote: On 2/25/21 8:57 AM, Vladimir Sementsov-Ogievskiy wrote: 25.02.2021 16:50, Vladimir Sementsov-Ogievskiy wrote: 18.02.2021 23:15, Eric Blake wrote: Previous commits (such as 6e280648, 75d34eb9) have mentioned that our NBD server still sends unaligned fragments

Re: [PATCH 2/5] block: Fix BDRV_BLOCK_RAW status to honor alignment

2021-02-25 Thread Eric Blake
On 2/25/21 8:55 AM, Vladimir Sementsov-Ogievskiy wrote: > 18.02.2021 23:15, Eric Blake wrote: >> Previous patches mentioned how the blkdebug filter driver demonstrates >> a bug present in our NBD server (for example, commit b0245d64); the >> bug is also present with the raw format driver when probi

Re: [PATCH 13/14] block: remove 'dirty-bitmaps' field from 'BlockInfo' struct

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
24.02.2021 16:11, Daniel P. Berrangé wrote: The same data is available in the 'BlockDeviceInfo' struct. Signed-off-by: Daniel P. Berrangé Reviewed-by: Vladimir Sementsov-Ogievskiy -- Best regards, Vladimir

Re: [PATCH 1/5] iotests: Update 241 to expose backing layer fragmentation

2021-02-25 Thread Eric Blake
On 2/25/21 8:57 AM, Vladimir Sementsov-Ogievskiy wrote: > 25.02.2021 16:50, Vladimir Sementsov-Ogievskiy wrote: >> 18.02.2021 23:15, Eric Blake wrote: >>> Previous commits (such as 6e280648, 75d34eb9) have mentioned that our >>> NBD server still sends unaligned fragments when an active layer with >

Re: [PATCH 12/14] block: remove dirty bitmaps 'status' field

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
24.02.2021 16:11, Daniel P. Berrangé wrote: The same information is available via the 'recording' and 'busy' fields. Signed-off-by: Daniel P. Berrangé Reviewed-by: Vladimir Sementsov-Ogievskiy -- Best regards, Vladimir

Re: [PATCH 1/5] iotests: Update 241 to expose backing layer fragmentation

2021-02-25 Thread Eric Blake
On 2/25/21 7:50 AM, Vladimir Sementsov-Ogievskiy wrote: > 18.02.2021 23:15, Eric Blake wrote: >> Previous commits (such as 6e280648, 75d34eb9) have mentioned that our >> NBD server still sends unaligned fragments when an active layer with >> large advertised minimum block size is backed by another

Re: [PATCH 1/5] iotests: Update 241 to expose backing layer fragmentation

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
25.02.2021 16:50, Vladimir Sementsov-Ogievskiy wrote: 18.02.2021 23:15, Eric Blake wrote: Previous commits (such as 6e280648, 75d34eb9) have mentioned that our NBD server still sends unaligned fragments when an active layer with large advertised minimum block size is backed by another layer with

Re: [PATCH 2/5] block: Fix BDRV_BLOCK_RAW status to honor alignment

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
18.02.2021 23:15, Eric Blake wrote: Previous patches mentioned how the blkdebug filter driver demonstrates a bug present in our NBD server (for example, commit b0245d64); the bug is also present with the raw format driver when probing occurs. Basically, if we specify a particular alignment > 1, b

Re: [PATCH v2 01/22] block: add eMMC block device type

2021-02-25 Thread Cédric Le Goater
On 2/24/21 8:13 PM, Sai Pavan Boddu wrote: > Hi Cedric, > > >> -Original Message- >> From: Cédric Le Goater >> Sent: Wednesday, February 24, 2021 7:25 PM >> To: Stefan Hajnoczi ; Sai Pavan Boddu >> >> Cc: Philippe Mathieu-Daudé ; Markus Armbruster >> ; Kevin Wolf ; Max Reitz >> ; Vladim

Re: [PATCH 1/5] iotests: Update 241 to expose backing layer fragmentation

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
18.02.2021 23:15, Eric Blake wrote: Previous commits (such as 6e280648, 75d34eb9) have mentioned that our NBD server still sends unaligned fragments when an active layer with large advertised minimum block size is backed by another layer with a smaller block size. Expand the test to actually cove

Re: [PATCH 06/14] machine: remove 'query-cpus' QMP command

2021-02-25 Thread Wainer dos Santos Moschetta
Hi, On 2/24/21 10:11 AM, Daniel P. Berrangé wrote: The newer 'query-cpus-fast' command avoids side effects on the guest execution. Note that some of the field names are different in the 'query-cpus-fast' command. Signed-off-by: Daniel P. Berrangé --- docs/system/deprecated.rst

[PATCH v2 3/3] block/qcow2: introduce inflight writes counters: fix discard

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
There is a bug in qcow2: host cluster can be discarded (refcount becomes 0) and reused during data write. In this case data write may pollute another cluster (recently allocated) or even metadata. To fix the issue introduce rw-lock: take read-lock on data writing and write-lock on discard. Enable

[PATCH v2 1/3] qemu-io: add aio_discard

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
Add aio_discard command like existing aio_write. It will be used in further test. Signed-off-by: Vladimir Sementsov-Ogievskiy --- qemu-io-cmds.c | 117 + 1 file changed, 117 insertions(+) diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c index 97611969

[PATCH v2 2/3] iotests: add qcow2-discard-during-rewrite

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
Simple test: - start writing to allocated cluster A - discard this cluster - write to another unallocated cluster B (it's allocated in same place where A was allocated) - continue writing to A For now last action pollutes cluster B which is a bug fixed by the following commit. For now, add

[PATCH v2(RFC) 0/3] qcow2: fix parallel rewrite and discard

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
Hi all! It occurs that nothing prevents discarding and reallocating host cluster during data writing. This way data writing will pollute another newly allocated cluster of data or metadata. OK, v2 is a try to solve the problem with CoRwlock.. And it is marked RFC, because of a lot of iotest failur

Re: [PATCH 0/3] hw/block/nvme: mdts/zasl cleanup

2021-02-25 Thread Klaus Jensen
On Feb 23 06:00, Keith Busch wrote: > These look good. > > Reviewed-by: Keith Busch Thanks, applied to nvme-next. signature.asc Description: PGP signature

[PATCH v2] blockjob: report a better error message

2021-02-25 Thread Stefano Garzarella
When a block job fails, we report strerror(-job->job.ret) error message, also if the job set an error object. Let's report a better error message using error_get_pretty(job->job.err). If an error object was not set, strerror(-job->ret) is used as fallback, as explained in include/qemu/job.h: type

[PATCH v1 2/2] block/qcow2: introduce inflight writes counters: fix discard

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
There is a bug in qcow2: host cluster can be discarded (refcount becomes 0) and reused during data write. In this case data write may pollute another cluster (recently allocated) or even metadata. To fix the issue let's track inflight writes to host cluster in the hash table and consider new count

[PATCH v1 1/2] iotests: add qcow2-discard-during-rewrite

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
Simple test: - start writing to allocated cluster A - discard this cluster - write to another unallocated cluster B (it's allocated in same place where A was allocated) - continue writing to A For now last action pollutes cluster B which is a bug fixed by the following commit. For now, add

Re: [PATCH v4 0/8] hw/sh4: Kconfig cleanups

2021-02-25 Thread Paolo Bonzini
On 22/02/21 15:15, Philippe Mathieu-Daudé wrote: Missing review: 1 (license) Since v3: - Include full MIT license text (Peter) Since v2: - Added missing TC58128/SH_PCI Kconfig entries (Peter) Since v1: - Addressed Peter Maydell review comments from https://www.mail-archive.com/qemu-block@nongn

[PATCH v1 0/2] qcow2: fix parallel rewrite and discard

2021-02-25 Thread Vladimir Sementsov-Ogievskiy
Hi all! It occurs that nothing prevents discarding and reallocating host cluster during data writing. This way data writing will pollute another flash allocated cluster of data or metadata. Here is my suggestion to fix it basing on improved refcounts model. Look at 02 for details. I don't insist

Re: [PATCH v4] virtio-blk: Respect discard granularity

2021-02-25 Thread Stefano Garzarella
On Thu, Feb 25, 2021 at 09:12:39AM +0900, Akihiko Odaki wrote: Report the configured granularity for discard operation to the guest. If this is not set use the block size. Since until now we have ignored the configured discard granularity and always reported the block size, let's add 'report-dis