Re: [PATCH v5 07/10] block: introduce preallocate filter

2020-08-26 Thread Vladimir Sementsov-Ogievskiy
24.08.2020 20:52, Vladimir Sementsov-Ogievskiy wrote: 21.08.2020 17:11, Vladimir Sementsov-Ogievskiy wrote: It's intended to be inserted between format and protocol nodes to preallocate additional space (expanding protocol file) on writes crossing EOF. It improves performance for file-systems wi

[PULL v2 00/34] Block patches

2020-08-26 Thread Max Reitz
The following changes since commit 30aa19446d82358a30eac3b556b4d6641e00b7c1: Merge remote-tracking branch 'remotes/cschoenebeck/tags/pull-9p-20200812' into staging (2020-08-24 16:39:53 +0100) are available in the Git repository at: https://github.com/XanClic/qemu.git tags/pull-block-2020-08

Re: [PATCH v3 62/74] [automated] Use TYPE_INFO macro

2020-08-26 Thread Juan Quintela
Eduardo Habkost wrote: > Generated using: > $ ./scripts/codeconverter/converter.py -i --passes=2 \ > --pattern=TypeRegisterCall,TypeInitMacro $(git grep -l TypeInfo -- > '*.[ch]') > > One notable difference is that files declaring multiple types > will now have multiple separate __construto

Re: [PATCH v3 64/74] [automated] Move QOM typedefs and add missing includes

2020-08-26 Thread Juan Quintela
[dropped people from CC] Eduardo Habkost wrote: > Some typedefs and macros are defined after the type check macros. > This makes it difficult to automatically replace their > definitions with OBJECT_DECLARE_TYPE. > > Patch generated using: > > $ ./scripts/codeconverter/converter.py -i \ >-

Re: [PATCH v3 66/74] [automated] Use DECLARE_*CHECKER* macros

2020-08-26 Thread Juan Quintela
Eduardo Habkost wrote: > $ ./scripts/codeconverter/converter.py -i \ >--pattern=TypeCheckMacro $(git grep -l '' -- '*.[ch]') > > Reviewed-by: Daniel P. Berrangé > Signed-off-by: Eduardo Habkost > --- > Changes v2 -> v3: > * Removed hunks due to rebase conflicts: > hw/arm/integratorcp.c hw

Re: [PATCH for-4.2 v2 3/3] block/file-posix: Let post-EOF fallocate serialize

2020-08-26 Thread Vladimir Sementsov-Ogievskiy
22.08.2020 20:04, Vladimir Sementsov-Ogievskiy wrote: 22.08.2020 20:03, Vladimir Sementsov-Ogievskiy wrote: 01.11.2019 18:25, Max Reitz wrote: The XFS kernel driver has a bug that may cause data corruption for qcow2 images as of qemu commit c8bb23cbdbe32f.  We can work around it by treating pos

Re: [PATCH v5 06/10] block: introduce BDRV_REQ_NO_WAIT flag

2020-08-26 Thread Max Reitz
On 26.08.20 08:26, Vladimir Sementsov-Ogievskiy wrote: > 25.08.2020 16:10, Max Reitz wrote: >> On 21.08.20 16:11, Vladimir Sementsov-Ogievskiy wrote: >>> Add flag to make serialising request no wait: if there are conflicting >>> requests, just return error immediately. It's will be used in upcoming

Re: [PATCH v5 07/10] block: introduce preallocate filter

2020-08-26 Thread Max Reitz
On 26.08.20 08:49, Vladimir Sementsov-Ogievskiy wrote: > 25.08.2020 18:11, Max Reitz wrote: >> On 21.08.20 16:11, Vladimir Sementsov-Ogievskiy wrote: >>> It's intended to be inserted between format and protocol nodes to >>> preallocate additional space (expanding protocol file) on writes >>> crossi

Re: [PATCH v5 06/10] block: introduce BDRV_REQ_NO_WAIT flag

2020-08-26 Thread Vladimir Sementsov-Ogievskiy
26.08.2020 11:36, Max Reitz wrote: On 26.08.20 08:26, Vladimir Sementsov-Ogievskiy wrote: 25.08.2020 16:10, Max Reitz wrote: On 21.08.20 16:11, Vladimir Sementsov-Ogievskiy wrote: Add flag to make serialising request no wait: if there are conflicting requests, just return error immediately. It

Re: [PATCH v5 07/10] block: introduce preallocate filter

2020-08-26 Thread Vladimir Sementsov-Ogievskiy
26.08.2020 11:52, Max Reitz wrote: On 26.08.20 08:49, Vladimir Sementsov-Ogievskiy wrote: 25.08.2020 18:11, Max Reitz wrote: On 21.08.20 16:11, Vladimir Sementsov-Ogievskiy wrote: It's intended to be inserted between format and protocol nodes to preallocate additional space (expanding protocol

Re: [PATCH v5 07/10] block: introduce preallocate filter

2020-08-26 Thread Max Reitz
On 26.08.20 11:15, Vladimir Sementsov-Ogievskiy wrote: > 26.08.2020 11:52, Max Reitz wrote: >> On 26.08.20 08:49, Vladimir Sementsov-Ogievskiy wrote: >>> 25.08.2020 18:11, Max Reitz wrote: On 21.08.20 16:11, Vladimir Sementsov-Ogievskiy wrote: > It's intended to be inserted between format

Re: [PATCH v3 49/74] swim: Rename struct SWIM to Swim

2020-08-26 Thread Laurent Vivier
Le 25/08/2020 à 21:20, Eduardo Habkost a écrit : > Currently we have a SWIM typedef and a SWIM type checking macro, > but OBJECT_DECLARE* would transform the SWIM macro into a > function, and the function name would conflict with the SWIM > typedef name. > > Rename the struct and typedef to "Swim"

Re: [PATCH 10/12] block/file-posix: fix a possible undefined behavior

2020-08-26 Thread Li Qiang
Pan Nengyuan 于2020年8月14日周五 下午6:32写道: > > local_err is not initialized to NULL, it will cause a assert error as below: > qemu/util/error.c:59: error_setv: Assertion `*errp == NULL' failed. > > Fixes: c6447510690 > Reported-by: Euler Robot > Signed-off-by: Pan Nengyuan Reviewed-by: Li Qiang > -

Re: [PATCH 09/12] blockdev: Fix a memleak in drive_backup_prepare()

2020-08-26 Thread Li Qiang
Pan Nengyuan 于2020年8月14日周五 下午6:54写道: > > 'local_err' seems forgot to propagate in error path, it'll cause > a memleak. Fix it. > > Reported-by: Euler Robot > Signed-off-by: Pan Nengyuan Reviewed-by: Li Qiang > --- > Cc: Kevin Wolf > Cc: Max Reitz > Cc: Markus Armbruster > Cc: qemu-block@no

Re: [PATCH v4 3/6] util: add Error object for qemu_open_internal error reporting

2020-08-26 Thread Markus Armbruster
Daniel P. Berrangé writes: > On Tue, Aug 25, 2020 at 05:14:21PM +0200, Markus Armbruster wrote: >> Daniel P. Berrangé writes: >> >> > Instead of relying on the limited information from errno, we can now >> > also provide detailed error messages. >> >> The more detailed error messages are curre

Re: [PATCH v4 2/6] util: refactor qemu_open_old to split off variadic args handling

2020-08-26 Thread Markus Armbruster
Daniel P. Berrangé writes: > On Tue, Aug 25, 2020 at 04:56:40PM +0200, Markus Armbruster wrote: >> Daniel P. Berrangé writes: >> >> > This simple refactoring prepares for future patches. The variadic args >> > handling is split from the main bulk of the open logic. The duplicated >> > calls to

Re: [PATCH for-4.2 v2 3/3] block/file-posix: Let post-EOF fallocate serialize

2020-08-26 Thread Vladimir Sementsov-Ogievskiy
26.08.2020 11:23, Vladimir Sementsov-Ogievskiy wrote: 22.08.2020 20:04, Vladimir Sementsov-Ogievskiy wrote: 22.08.2020 20:03, Vladimir Sementsov-Ogievskiy wrote: 01.11.2019 18:25, Max Reitz wrote: The XFS kernel driver has a bug that may cause data corruption for qcow2 images as of qemu commit

Re: [PATCH v4 5/6] util: give a specific error message when O_DIRECT doesn't work

2020-08-26 Thread Markus Armbruster
Daniel P. Berrangé writes: > On Tue, Aug 25, 2020 at 05:19:53PM +0200, Markus Armbruster wrote: >> Daniel P. Berrangé writes: >> >> > A common error scenario is to tell QEMU to use O_DIRECT in combination >> > with a filesystem that doesn't support it. To aid users to diagnosing >> > their mist

Re: [PATCH v5 07/10] block: introduce preallocate filter

2020-08-26 Thread Vladimir Sementsov-Ogievskiy
26.08.2020 12:58, Max Reitz wrote: On 26.08.20 11:15, Vladimir Sementsov-Ogievskiy wrote: 26.08.2020 11:52, Max Reitz wrote: On 26.08.20 08:49, Vladimir Sementsov-Ogievskiy wrote: 25.08.2020 18:11, Max Reitz wrote: On 21.08.20 16:11, Vladimir Sementsov-Ogievskiy wrote: It's intended to be in

[PATCH 3/3] backup: initialize bcs bitmap on job create, not start

2020-08-26 Thread Stefan Reiter
After backup_init_bcs_bitmap the copy-before-write behaviour is active. This way, multiple backup jobs created at once but running in a sequential transaction will still represent the same point in time. Signed-off-by: Stefan Reiter --- I'd imagine this was done on job start for a purpose, so th

[PATCH 0/3] Add support for sequential backups

2020-08-26 Thread Stefan Reiter
Backups can already be done for multiple drives in a transaction. However, these jobs will start all at once, potentially hogging a lot of disk IO all at once. This problem is made worse, since IO throttling is only available on a per-job basis. Add a flag to QMP to support sequential transactions

[PATCH 2/3] blockdev: add sequential mode to *-backup transactions

2020-08-26 Thread Stefan Reiter
Only supported with completion-mode 'grouped', since it relies on a JobTxn to exist. This means that for now it is only available for {drive,blockdev}-backup transactions. Since only one job will be running at a time, bandwidth-limits can be applied effectively. It can also prevent overloading a h

[PATCH 1/3] job: add sequential transaction support

2020-08-26 Thread Stefan Reiter
Jobs in a sequential transaction should never be started with job_start manually. job_txn_start_seq and the sequentially called job_start will take care of it, 'assert'ing in case a job is already running or has finished. Signed-off-by: Stefan Reiter --- include/qemu/job.h | 12 job

Re: [PATCH] Simple & stupid coroutine-aware monitor_cur()

2020-08-26 Thread Markus Armbruster
Kevin Wolf writes: > Am 07.08.2020 um 15:27 hat Markus Armbruster geschrieben: >> This is just a sketch. It's incomplete, needs comments and a real >> commit message. >> >> Support for "[PATCH v6 09/12] hmp: Add support for coroutine command >> handlers" is missing. Marked FIXME. >> >> As is,

Re: [PATCH] Coroutine-aware monitor_cur() with coroutine-specific data

2020-08-26 Thread Markus Armbruster
Kevin Wolf writes: > Am 07.08.2020 um 15:29 hat Markus Armbruster geschrieben: >> This is just a sketch. It needs comments and a real commit message. >> >> As is, it goes on top of Kevin's series. It is meant to be squashed >> into PATCH 06. >> >> Signed-off-by: Markus Armbruster >> --- >>

Re: qcow2 overlay performance

2020-08-26 Thread Kevin Wolf
Am 26.08.2020 um 02:46 hat Yoonho Park geschrieben: > I have been measuring the performance of qcow2 overlays, and I am hoping to > get some help in understanding the data I collected. In my experiments, I > created a VM and attached a 16G qcow2 disk to it using "qemu-img create" > and "virsh attac

Re: [PATCH v3] block: Raise an error when backing file parameter is an empty string

2020-08-26 Thread Kevin Wolf
Am 13.08.2020 um 15:47 hat Connor Kuehl geschrieben: > Providing an empty string for the backing file parameter like so: > > qemu-img create -f qcow2 -b '' /tmp/foo > > allows the flow of control to reach and subsequently fail an assert > statement because passing an empty string to > >

Re: qcow2 overlay performance

2020-08-26 Thread Peter Krempa
On Wed, Aug 26, 2020 at 15:18:32 +0200, Kevin Wolf wrote: > Am 26.08.2020 um 02:46 hat Yoonho Park geschrieben: > > Another issue I hit is that I cannot set or change the cluster size of > > overlays. Is this possible with "virsh snapshot-create-as"? > > That's a libvirt question. Peter, can you h

[PATCH] block/mirror: fix core when using iothreads

2020-08-26 Thread Peng Liang
We found an issue when doing block-commit with iothreads, which tries to dereference a NULL pointer. | | mirror_start_job | 1. bdrv_ref(mirror_top_bs); | bdrv_drained_begin(bs);| bdrv_append(mirror_to

Re: [PATCH] Coroutine-aware monitor_cur() with coroutine-specific data

2020-08-26 Thread Kevin Wolf
Am 26.08.2020 um 14:40 hat Markus Armbruster geschrieben: > Kevin Wolf writes: > > > Am 07.08.2020 um 15:29 hat Markus Armbruster geschrieben: > >> This is just a sketch. It needs comments and a real commit message. > >> > >> As is, it goes on top of Kevin's series. It is meant to be squashed

Re: [PATCH v5 07/10] block: introduce preallocate filter

2020-08-26 Thread David Edmondson
On Tuesday, 2020-08-25 at 17:11:34 +02, Max Reitz wrote: > On 21.08.20 16:11, Vladimir Sementsov-Ogievskiy wrote: >> It's intended to be inserted between format and protocol nodes to >> preallocate additional space (expanding protocol file) on writes >> crossing EOF. It improves performance for fi

Re: [PULL v2 00/34] Block patches

2020-08-26 Thread Peter Maydell
On Wed, 26 Aug 2020 at 08:51, Max Reitz wrote: > > The following changes since commit 30aa19446d82358a30eac3b556b4d6641e00b7c1: > > Merge remote-tracking branch 'remotes/cschoenebeck/tags/pull-9p-20200812' > into staging (2020-08-24 16:39:53 +0100) > > are available in the Git repository at: >

Re: qcow2 overlay performance

2020-08-26 Thread Peter Krempa
On Wed, Aug 26, 2020 at 15:30:03 +0200, Peter Krempa wrote: > On Wed, Aug 26, 2020 at 15:18:32 +0200, Kevin Wolf wrote: > > Am 26.08.2020 um 02:46 hat Yoonho Park geschrieben: > > > Another issue I hit is that I cannot set or change the cluster size of > > > overlays. Is this possible with "virsh s

Re: [PATCH v2 (BROKEN) 0/6] migration: bring improved savevm/loadvm/delvm to QMP

2020-08-26 Thread Markus Armbruster
Sorry for taking so long to reply. Daniel P. Berrangé writes: > A followup to: > > v1: https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg00866.html > > When QMP was first introduced some 10+ years ago now, the snapshot > related commands (savevm/loadvm/delvm) were not converted. This was

Re: [PATCH v3 72/74] [automated] Remove redundant instance_size/class_size fields

2020-08-26 Thread Eduardo Habkost
On Tue, Aug 25, 2020 at 03:21:08PM -0400, Eduardo Habkost wrote: > This will remove instance_size/class_size fields from TypeInfo > variables when the value is exactly the same as the one in the > parent class. > > Generated by: > > $ ./scripts/codeconverter/converter.py -i \ >--pattern=Redu

Re: qcow2 overlay performance

2020-08-26 Thread Alberto Garcia
On Wed 26 Aug 2020 03:18:32 PM CEST, Kevin Wolf wrote: >> My understanding is that writing 4K blocks requires a >> read-modify-write because you must fetch a complete cluster from >> deeper in the overlay chain before writing to the active >> overlay. However, this does not explain the drop in perf

Re: qcow2 overlay performance

2020-08-26 Thread Yoonho Park
I create the attached disk with the following commands. The "qemu-img info" is to double check the cluster size. I am running the same experiments now with "--cache none" attached to the "virsh attach-disk". Is this sufficient to avoid the host page cache? qemu-img create -f qcow2 ${dir}/${disk}.q

Re: qcow2 overlay performance

2020-08-26 Thread Yoonho Park
Great. I will give your patches a try. Also, your workaround suggestion and a discussion with a colleague gave me another idea for an experiment. Is it possible that some of the overhead I am seeing is from the operations necessary to increase the size of the overlay? Would it make sense to use the

Re: qcow2 overlay performance

2020-08-26 Thread Yoonho Park
I used strace to collect the writes, and as far as I can tell they are aligned to the cluster size (64K). Below are some examples... 1520414 pwritev(35, [{iov_base="\5\277z\314\24\305\177\r\340\327\f:e\222\10\33\374\232Q;FuN\t\0\0\325\275\0\0\0\0"..., iov_len=4096}, {iov_base="v\24\324\337\347\364

Re: [PATCH v2 (BROKEN) 0/6] migration: bring improved savevm/loadvm/delvm to QMP

2020-08-26 Thread Daniel P . Berrangé
On Wed, Aug 26, 2020 at 05:52:06PM +0200, Markus Armbruster wrote: > Sorry for taking so long to reply. > > Daniel P. Berrangé writes: > > > A followup to: > > > > v1: https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg00866.html > > > > When QMP was first introduced some 10+ years ago no

Re: [PATCH 0/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-26 Thread Alberto Garcia
On Tue 25 Aug 2020 09:47:24 PM CEST, Brian Foster wrote: > My fio fallocates the entire file by default with this command. Is that > the intent of this particular test? I added --fallocate=none to my test > runs to incorporate the allocation cost in the I/Os. That wasn't intentional, you're right

Re: [PATCH v2 (BROKEN) 0/6] migration: bring improved savevm/loadvm/delvm to QMP

2020-08-26 Thread Daniel P . Berrangé
On Wed, Aug 26, 2020 at 07:28:24PM +0100, Daniel P. Berrangé wrote: > On Wed, Aug 26, 2020 at 05:52:06PM +0200, Markus Armbruster wrote: > > Open questions: > > > > * Do we want the QMP command to delete existing snapshots with > > conflicting tag / ID, like HMP savevm does? Or do we want it to