Re: RFC: Qemu backup interface plans

2021-05-21 Thread Vladimir Sementsov-Ogievskiy
17.05.2021 15:07, Vladimir Sementsov-Ogievskiy wrote: 3.1 and make it possible to modify the bitmap externally, so that consumer of fleecing can say to backup-top filter: I've already copied these blocks, don't bother with copying them to temp image". This is to solve [2]. Still, how consumer

Re: Qemu block filter insertion/removal API

2021-05-21 Thread Vladimir Sementsov-Ogievskiy
21.05.2021 21:32, Vladimir Sementsov-Ogievskiy wrote: 19.05.2021 17:14, Vladimir Sementsov-Ogievskiy wrote: 19.05.2021 16:02, Kevin Wolf wrote: Am 19.05.2021 um 14:19 hat Vladimir Sementsov-Ogievskiy geschrieben: 19.05.2021 14:44, Kevin Wolf wrote: Am 17.05.2021 um 14:44 hat Vladimir

Re: Qemu block filter insertion/removal API

2021-05-21 Thread Vladimir Sementsov-Ogievskiy
19.05.2021 17:14, Vladimir Sementsov-Ogievskiy wrote: 19.05.2021 16:02, Kevin Wolf wrote: Am 19.05.2021 um 14:19 hat Vladimir Sementsov-Ogievskiy geschrieben: 19.05.2021 14:44, Kevin Wolf wrote: Am 17.05.2021 um 14:44 hat Vladimir Sementsov-Ogievskiy geschrieben: Hi all! I'd like to be sure

Re: [PATCH v2 1/7] block-copy: streamline choice of copy_range vs. read/write

2021-05-21 Thread Vladimir Sementsov-Ogievskiy
21.05.2021 18:10, Paolo Bonzini wrote: Stats don't say everything.  Not having something like this:     s->copy_size =     MIN(MAX(s->cluster_size, BLOCK_COPY_MAX_COPY_RANGE),     QEMU_ALIGN_DOWN(block_copy_max_transfer(s->source,  

[PATCH 6/6] iotests/image-fleecing: test push backup with fleecing

2021-05-21 Thread Vladimir Sementsov-Ogievskiy
Add test for push backup with fleecing: - start fleecing with copy-before-write filter - start a backup job from temporary fleecing node to actual backup target Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/tests/image-fleecing | 104 +++-

[PATCH 4/6] block: blk_root(): return non-const pointer

2021-05-21 Thread Vladimir Sementsov-Ogievskiy
In the following patch we'll want to pass blk children to block-copy. Const pointers are not enough. So, return non const pointer from blk_root(). Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/sysemu/block-backend.h | 2 +- block/block-backend.c | 2 +- 2 files changed, 2

[PATCH 3/6] block: share writes on backing child of fleecing node

2021-05-21 Thread Vladimir Sementsov-Ogievskiy
By default, we share writes on backing child only if our parents share write permission on us. Still, with fleecing scheme we want to be able to unshare writes on fleecing node, which is a kind of immutable snapshot (copy-before-write operations are write-unchanged). So, let's detect fleecing

[PATCH 5/6] block/backup: don't insert filter if no writers

2021-05-21 Thread Vladimir Sementsov-Ogievskiy
If source is immutable and there no writers on it, we don't need to insert a filter, so let's detect it and use simple blk's for block-copy. Note, that it's possible, that user will try to add writers on source during backup. It will fail, as our source blk doesn't share write. In future we can

[PATCH 1/6] block/block-copy: use write-unchanged for fleecing scheme

2021-05-21 Thread Vladimir Sementsov-Ogievskiy
We are going to use fleecing scheme for push-backup, so that copy-before-write filter does copy before write operations to temporary image and backup job copies data from (immutable from backup's point of view) temporary image to actual backup target. For this to work properly, backup job should

[PATCH 2/6] block/copy-before-write: require BLK_PERM_WRITE_UNCHANGED for fleecing

2021-05-21 Thread Vladimir Sementsov-Ogievskiy
Now block-copy detects fleecing scheme and do write-unchanged operations if detected. So, let's require appropriate permissions. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/copy-before-write.c | 11 ++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git

[PATCH 0/6] push backup with fleecing

2021-05-21 Thread Vladimir Sementsov-Ogievskiy
Hi all! Here is push-backup with fleecing. What is it: 1. Make fleecing scheme guest blk | |root v copy-before-write filter ---> temp qcow2 | | |file | backing V | active disk

Re: [PATCH v6 03/25] python: create utils sub-package

2021-05-21 Thread John Snow
On 5/21/21 1:18 PM, Willian Rampazzo wrote: On Wed, May 12, 2021 at 8:13 PM John Snow wrote: Create a space for miscellaneous things that don't belong strictly in "qemu.machine" nor "qemu.qmp" packages. Signed-off-by: John Snow --- python/qemu/machine/__init__.py | 8

Re: [PATCH v3 2/2] virtio-blk: Convert QEMUBH callback to "bitops.h" API

2021-05-21 Thread Stefan Hajnoczi
On Mon, May 10, 2021 at 10:07:58PM +0200, Philippe Mathieu-Daudé wrote: > By directly using find_first_bit() and find_next_bit from the > "bitops.h" API to iterate over the bitmap, we can remove the > bitmap[] variable-length array copy on the stack and the complex > manual bit testing/clearing

Re: [PATCH v3 0/2] virtio-blk: Convert QEMUBH callback to "bitops.h" API

2021-05-21 Thread Stefan Hajnoczi
On Mon, May 10, 2021 at 10:07:56PM +0200, Philippe Mathieu-Daudé wrote: > This series follow a suggestion from Stefan to use the bitops > API in virtio-blk: > https://www.mail-archive.com/qemu-devel@nongnu.org/msg805139.html > > Since v2: > - clear bitmap to avoid spurious interrupts! (Stefan) >

Re: [PATCH v6 03/25] python: create utils sub-package

2021-05-21 Thread Willian Rampazzo
On Wed, May 12, 2021 at 8:13 PM John Snow wrote: > > Create a space for miscellaneous things that don't belong strictly in > "qemu.machine" nor "qemu.qmp" packages. > > Signed-off-by: John Snow > --- > python/qemu/machine/__init__.py | 8 > python/qemu/utils/__init__.py

Re: [PATCH v6 02/25] python: create qemu packages

2021-05-21 Thread Willian Rampazzo
On Wed, May 12, 2021 at 8:13 PM John Snow wrote: > > move python/qemu/*.py to python/qemu/[machine, qmp]/*.py and update import > directives across the tree. > > This is done to create a PEP420 namespace package, in which we may > create subpackages. To do this, the namespace directory ("qemu")

Re: [PATCH v2 7/7] block-copy: protect BlockCopyState .method fields

2021-05-21 Thread Vladimir Sementsov-Ogievskiy
18.05.2021 13:07, Emanuele Giuseppe Esposito wrote: With tasks and calls lock protecting all State fields, .method is the last BlockCopyState field left unprotected. Set it as atomic. Signed-off-by: Emanuele Giuseppe Esposito OK, in 06 some things are out of coroutine. Here could we just

Re: [PATCH v2 6/7] block-copy: atomic .cancelled and .finished fields in BlockCopyCallState

2021-05-21 Thread Paolo Bonzini
On 21/05/21 17:56, Vladimir Sementsov-Ogievskiy wrote: 21.05.2021 18:02, Paolo Bonzini wrote: On 20/05/21 17:34, Vladimir Sementsov-Ogievskiy wrote: By adding acquire/release pairs, we ensure that .ret and .error_is_read fields are written by block_copy_dirty_clusters before .finished is

Re: [PATCH v2 1/7] block-copy: streamline choice of copy_range vs. read/write

2021-05-21 Thread Paolo Bonzini
On 21/05/21 17:48, Vladimir Sementsov-Ogievskiy wrote: I keep it thinking about further moving qemu-img convert to block-copy. But I don't even have a plan when to start this work. So, if we want to do something around copy_range here to prepare for thread-safety, let's just drop it for now as

Re: [PATCH v2 6/7] block-copy: atomic .cancelled and .finished fields in BlockCopyCallState

2021-05-21 Thread Vladimir Sementsov-Ogievskiy
21.05.2021 18:02, Paolo Bonzini wrote: On 20/05/21 17:34, Vladimir Sementsov-Ogievskiy wrote: By adding acquire/release pairs, we ensure that .ret and .error_is_read fields are written by block_copy_dirty_clusters before .finished is true. As I already said, please, can we live with one

Re: [PATCH v2 6/7] block-copy: atomic .cancelled and .finished fields in BlockCopyCallState

2021-05-21 Thread Vladimir Sementsov-Ogievskiy
21.05.2021 18:02, Paolo Bonzini wrote: On 20/05/21 17:34, Vladimir Sementsov-Ogievskiy wrote: By adding acquire/release pairs, we ensure that .ret and .error_is_read fields are written by block_copy_dirty_clusters before .finished is true. As I already said, please, can we live with one

Re: [PATCH v6 04/25] python: add qemu package installer

2021-05-21 Thread John Snow
On 5/21/21 12:00 AM, Cleber Rosa wrote: On Wed, May 12, 2021 at 07:12:20PM -0400, John Snow wrote: Add setup.cfg and setup.py, necessary for installing a package via pip. Add a ReST document (PACKAGE.rst) explaining the basics of what this package is for and who to contact for more information.

Re: [PATCH v2 1/7] block-copy: streamline choice of copy_range vs. read/write

2021-05-21 Thread Vladimir Sementsov-Ogievskiy
21.05.2021 18:10, Paolo Bonzini wrote: On 20/05/21 16:42, Vladimir Sementsov-Ogievskiy wrote: 18.05.2021 13:07, Emanuele Giuseppe Esposito wrote: From: Paolo Bonzini Put the logic to determine the copy size in a separate function, so that there is a simple state machine for the possible

Re: [PATCH v2 1/7] block-copy: streamline choice of copy_range vs. read/write

2021-05-21 Thread Paolo Bonzini
On 20/05/21 16:42, Vladimir Sementsov-Ogievskiy wrote: 18.05.2021 13:07, Emanuele Giuseppe Esposito wrote: From: Paolo Bonzini Put the logic to determine the copy size in a separate function, so that there is a simple state machine for the possible methods of copying data from one

Re: [PATCH v2 5/7] block-copy: add QemuMutex lock for BlockCopyCallState list

2021-05-21 Thread Paolo Bonzini
On 20/05/21 17:30, Vladimir Sementsov-Ogievskiy wrote: 18.05.2021 13:07, Emanuele Giuseppe Esposito wrote: As for BlockCopyTask, add a lock to protect BlockCopyCallState ret and sleep_state fields. Also move ret, finished and cancelled in the OUT fields of BlockCopyCallState. Here a QemuMutex

Re: [PATCH v2 6/7] block-copy: atomic .cancelled and .finished fields in BlockCopyCallState

2021-05-21 Thread Paolo Bonzini
On 20/05/21 17:34, Vladimir Sementsov-Ogievskiy wrote: By adding acquire/release pairs, we ensure that .ret and .error_is_read fields are written by block_copy_dirty_clusters before .finished is true. As I already said, please, can we live with one mutex for now? finished, ret,

Re: [PATCH v2 0/6] coroutine: new sleep/wake API

2021-05-21 Thread Stefan Hajnoczi
On Mon, May 17, 2021 at 12:05:42PM +0200, Paolo Bonzini wrote: > This is a revamp of the qemu_co_sleep* API that makes it easier to > extend the API: the state that is needed to wake up a coroutine is now > part of the public API instead of hidden behind a pointer-to-pointer; > the API is made

Re: [PULL 0/9] scripts/simplebench patches

2021-05-21 Thread Vladimir Sementsov-Ogievskiy
21.05.2021 15:45, Peter Maydell wrote: On Tue, 4 May 2021 at 10:01, Vladimir Sementsov-Ogievskiy wrote: The following changes since commit 53c5433e84e8935abed8e91d4a2eb813168a0ecf: Merge remote-tracking branch 'remotes/rth-gitlab/tags/pull-tcg-20210501' into staging (2021-05-02 12:02:46

Re: [PULL 0/9] scripts/simplebench patches

2021-05-21 Thread Peter Maydell
On Tue, 4 May 2021 at 10:01, Vladimir Sementsov-Ogievskiy wrote: > > The following changes since commit 53c5433e84e8935abed8e91d4a2eb813168a0ecf: > > Merge remote-tracking branch 'remotes/rth-gitlab/tags/pull-tcg-20210501' > into staging (2021-05-02 12:02:46 +0100) > > are available in the Git

Re: making a qdev bus available from a (non-qtree?) device

2021-05-21 Thread Klaus Jensen
On May 21 09:33, Markus Armbruster wrote: I'm about to drop off for two weeks of much-needed vacation. I meant to study your explanation and give design advice before I leave, but I'm out of time. Regrettable. I hope Stefan can help you. Or perhaps Paolo. If you still have questions when

Re: making a qdev bus available from a (non-qtree?) device

2021-05-21 Thread Markus Armbruster
I'm about to drop off for two weeks of much-needed vacation. I meant to study your explanation and give design advice before I leave, but I'm out of time. Regrettable. I hope Stefan can help you. Or perhaps Paolo. If you still have questions when I'm back, feel free to contact me again.

Re: [PATCH] hw/nvme/ctrl: fix functions style

2021-05-21 Thread Klaus Jensen
On May 21 11:38, Gollu Appalanaidu wrote: Identify command related functions style fix. Signed-off-by: Gollu Appalanaidu --- hw/nvme/ctrl.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index 0bcaf7192f..40a7efcea9 100644 ---

Re: [PATCH] MAINTAINERS: update block/rbd.c maintainer

2021-05-21 Thread Stefano Garzarella
On Wed, May 19, 2021 at 01:25:13PM +0200, Ilya Dryomov wrote: Jason has moved on from working on RBD and Ceph. I'm taking over his role upstream. Signed-off-by: Ilya Dryomov --- MAINTAINERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MAINTAINERS b/MAINTAINERS index

[PATCH] hw/nvme/ctrl: fix functions style

2021-05-21 Thread Gollu Appalanaidu
Identify command related functions style fix. Signed-off-by: Gollu Appalanaidu --- hw/nvme/ctrl.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index 0bcaf7192f..40a7efcea9 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -4291,7