Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-25 Thread John Snow
On 04/25/2018 03:19 AM, Nikolay Shirokovskiy wrote: > > > On 24.04.2018 23:02, John Snow wrote: >> >> >> On 04/23/2018 06:38 AM, Nikolay Shirokovskiy wrote: >>> >>> >>> On 21.04.2018 00:26, Eric Blake wrote: On 04/20/2018 01:24 PM, John Snow wrote: >>> Why is option 3 unworkable,

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-25 Thread Nikolay Shirokovskiy
On 24.04.2018 23:02, John Snow wrote: > > > On 04/23/2018 06:38 AM, Nikolay Shirokovskiy wrote: >> >> >> On 21.04.2018 00:26, Eric Blake wrote: >>> On 04/20/2018 01:24 PM, John Snow wrote: >>> >> Why is option 3 unworkable, exactly?: >> >> (3) Checkpoints exist as structures only

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-24 Thread John Snow
On 04/23/2018 06:38 AM, Nikolay Shirokovskiy wrote: > > > On 21.04.2018 00:26, Eric Blake wrote: >> On 04/20/2018 01:24 PM, John Snow wrote: >> > Why is option 3 unworkable, exactly?: > > (3) Checkpoints exist as structures only with libvirt. They are saved > and remembered in

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-24 Thread John Snow
On 04/23/2018 05:31 AM, Vladimir Sementsov-Ogievskiy wrote: > 21.04.2018 00:26, Eric Blake wrote: >> On 04/20/2018 01:24 PM, John Snow wrote: >> > Why is option 3 unworkable, exactly?: > > (3) Checkpoints exist as structures only with libvirt. They are saved > and remembered in

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-24 Thread Nikolay Shirokovskiy
On 23.04.2018 16:50, Eric Blake wrote: > On 04/23/2018 05:38 AM, Nikolay Shirokovskiy wrote: > >>> I'd argue the converse. Libvirt already knows how to do atomic updates >>> of XML files that it tracks. If libvirtd crashes/restarts in the middle >>> of an API call, you already have

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-23 Thread Eric Blake
On 04/23/2018 04:31 AM, Vladimir Sementsov-Ogievskiy wrote: >> And if there is more than one bitmap on snap1, do we >> need to bring all of those bitmaps forward into snap2, or just the one >> that was currently active? > > Again, I think, to make snapshots unrelated, it's better to keep them >

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-23 Thread Eric Blake
On 04/23/2018 05:38 AM, Nikolay Shirokovskiy wrote: >> I'd argue the converse. Libvirt already knows how to do atomic updates >> of XML files that it tracks. If libvirtd crashes/restarts in the middle >> of an API call, you already have indeterminate results of whether the >> API worked or

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-23 Thread Nikolay Shirokovskiy
On 21.04.2018 00:26, Eric Blake wrote: > On 04/20/2018 01:24 PM, John Snow wrote: > Why is option 3 unworkable, exactly?: (3) Checkpoints exist as structures only with libvirt. They are saved and remembered in the XML entirely. Or put another way: Can you

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-23 Thread Vladimir Sementsov-Ogievskiy
21.04.2018 00:26, Eric Blake wrote: On 04/20/2018 01:24 PM, John Snow wrote: Why is option 3 unworkable, exactly?: (3) Checkpoints exist as structures only with libvirt. They are saved and remembered in the XML entirely. Or put another way: Can you explain to me why it's important for

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-20 Thread Eric Blake
On 04/20/2018 01:24 PM, John Snow wrote: >>> Why is option 3 unworkable, exactly?: >>> >>> (3) Checkpoints exist as structures only with libvirt. They are saved >>> and remembered in the XML entirely. >>> >>> Or put another way: >>> >>> Can you explain to me why it's important for libvirt to be

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-20 Thread John Snow
On 04/20/2018 08:22 AM, Nikolay Shirokovskiy wrote: > > > On 19.04.2018 20:28, John Snow wrote: >> >> >> On 04/16/2018 06:20 AM, Vladimir Sementsov-Ogievskiy wrote: >>> >>> So my point is: if we are going to implement something complicated, >>> let's implement entirely what we want, not a

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-20 Thread Nikolay Shirokovskiy
On 19.04.2018 20:28, John Snow wrote: > > > On 04/16/2018 06:20 AM, Vladimir Sementsov-Ogievskiy wrote: >> >> So my point is: if we are going to implement something complicated, >> let's implement entirely what we want, not a semi-solution. Otherwise, >> implement a minimal and simple thing,

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-19 Thread John Snow
On 04/16/2018 06:20 AM, Vladimir Sementsov-Ogievskiy wrote: > > So my point is: if we are going to implement something complicated, > let's implement entirely what we want, not a semi-solution. Otherwise, > implement a minimal and simple thing, to just make it all work (my > current solution).

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-18 Thread Eric Blake
On 04/11/2018 11:14 PM, John Snow wrote: >>> This API does not use existent disks snapshots. Instead it introduces >>> snapshots >>> provided by qemu's blockdev-backup command. The reason is we need >>> snapshotted >>> disk state only temporarily for

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-18 Thread Nikolay Shirokovskiy
On 18.04.2018 18:24, Eric Blake wrote: > On 04/13/2018 04:47 AM, Nikolay Shirokovskiy wrote: > > Earlier, you said that the new virDomainBlockSnapshotPtr are > independent, with no relations between them. But here, you are wanting > to keep incremental backups related to one

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-18 Thread Eric Blake
On 04/13/2018 03:02 PM, John Snow wrote: > What are the downsides to actually including a predecessor/successor* > pointer in QEMU? > > (1) We'd need to amend the bitmap persistence format Which I think is doable, since we have a size field. > (2) We'd need to amend some of the bitmap

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-18 Thread Eric Blake
On 04/13/2018 04:47 AM, Nikolay Shirokovskiy wrote: Earlier, you said that the new virDomainBlockSnapshotPtr are independent, with no relations between them. But here, you are wanting to keep incremental backups related to one another. >>> >>> Yes, but backups are not snapshots.

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-16 Thread Vladimir Sementsov-Ogievskiy
13.04.2018 23:02, John Snow wrote: On 04/12/2018 10:08 AM, Vladimir Sementsov-Ogievskiy wrote: It's not easier, as we'll have to implement either separate of bitmaps concept of checkpoints, which will be based on bitmaps, and we'll have to negotiate and implement storing these objects to qcow2

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-15 Thread John Snow
On 04/13/2018 05:47 AM, Nikolay Shirokovskiy wrote: >>> However as we use chain of disabled bitmaps with one active bitmap on tip >>> of the chain and qemu does not track their order we need to do it in >>> libvirt. >>> >> Well, you seem to be tracking it in *qemu*, by using the name field. >>

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-15 Thread Vladimir Sementsov-Ogievskiy
13.04.2018 21:02, John Snow wrote: On 04/13/2018 08:01 AM, Vladimir Sementsov-Ogievskiy wrote: 1. It looks unsafe to use nbd server + backup(sync=none) on same node, synchronization is needed, like in block/replication, which uses backup_wait_for_overlapping_requests, backup_cow_request_begin,

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-15 Thread John Snow
On 04/12/2018 10:08 AM, Vladimir Sementsov-Ogievskiy wrote: > It's not easier, as we'll have to implement either separate of bitmaps > concept of checkpoints, which will be based on bitmaps, and we'll have > to negotiate and implement storing these objects to qcow2 and migrate > them. Or we'll

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-15 Thread Vladimir Sementsov-Ogievskiy
13.04.2018 12:16, Nikolay Shirokovskiy wrote: On 13.04.2018 00:16, John Snow wrote: On 04/03/2018 08:01 AM, Nikolay Shirokovskiy wrote: Hi, all. This is another RFC on pull backup API. This API

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-15 Thread Nikolay Shirokovskiy
On 13.04.2018 12:07, Daniel P. Berrangé wrote: > On Tue, Apr 03, 2018 at 03:01:22PM +0300, Nikolay Shirokovskiy wrote: >> *Temporary snapshot API* >> >> In previous version it is called 'Fleece API' after qemu terms and I'll still >> use BlockSnapshot prefix for commands as in previous RFC

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-15 Thread Nikolay Shirokovskiy
On 13.04.2018 02:53, John Snow wrote: >> > First a few facts about qemu dirty bitmaps. > > Bitmap can be either in active or disable state. In disabled state it > does not > get changed on guest writes. And oppositely in active state it tracks > guest > writes.

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-15 Thread Nikolay Shirokovskiy
On 13.04.2018 03:04, John Snow wrote: > > > On 04/12/2018 10:08 AM, Vladimir Sementsov-Ogievskiy wrote: >> >> I propose, not to say that bitmap represents a checkpoint. It is simpler >> to say (and it reflects the reality) that bitmap is a difference between >> two consecutive checkpoints. And

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-15 Thread John Snow
On 04/12/2018 08:26 AM, Vladimir Sementsov-Ogievskiy wrote: > 1. It looks unsafe to use nbd server + backup(sync=none) on same node, > synchronization is needed, like in block/replication, which uses > backup_wait_for_overlapping_requests, backup_cow_request_begin, > backup_cow_request_end. We

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-15 Thread Nikolay Shirokovskiy
On 12.04.2018 07:14, John Snow wrote: > > > On 04/11/2018 12:32 PM, Eric Blake wrote: >> On 04/03/2018 07:01 AM, Nikolay Shirokovskiy wrote: >>> Hi, all. >>> >>>

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-15 Thread Nikolay Shirokovskiy
On 11.04.2018 19:32, Eric Blake wrote: > On 04/03/2018 07:01 AM, Nikolay Shirokovskiy wrote: >> Hi, all. >> >> >>

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-15 Thread Vladimir Sementsov-Ogievskiy
11.04.2018 16:56, Eric Blake wrote: On 04/03/2018 07:01 AM, Nikolay Shirokovskiy wrote: Hi, all. This is another RFC on pull backup API. This API provides means to read domain disks in a

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-15 Thread Nikolay Shirokovskiy
On 11.04.2018 17:16, Vladimir Sementsov-Ogievskiy wrote: > 11.04.2018 16:56, Eric Blake wrote: >> On 04/03/2018 07:01 AM, Nikolay Shirokovskiy wrote: >>> Hi, all. >>> >>>   This is another RFC on

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-13 Thread John Snow
On 04/13/2018 05:16 AM, Nikolay Shirokovskiy wrote: > > > On 13.04.2018 00:16, John Snow wrote: >> >> >> On 04/03/2018 08:01 AM, Nikolay Shirokovskiy wrote: >>> Hi, all. >>> >>>

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-13 Thread John Snow
On 04/13/2018 08:01 AM, Vladimir Sementsov-Ogievskiy wrote: >>> 1. It looks unsafe to use nbd server + backup(sync=none) on same node, >>> synchronization is needed, like in block/replication, which uses >>> backup_wait_for_overlapping_requests, backup_cow_request_begin, >>>

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-13 Thread Vladimir Sementsov-Ogievskiy
13.04.2018 18:05, Nikolay Shirokovskiy wrote: On 13.04.2018 14:41, Vladimir Sementsov-Ogievskiy wrote: 13.04.2018 11:51, Nikolay Shirokovskiy wrote: On 13.04.2018 03:04, John Snow wrote: On 04/12/2018 10:08 AM, Vladimir Sementsov-Ogievskiy wrote: I propose, not to say that bitmap represents

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-13 Thread Nikolay Shirokovskiy
On 13.04.2018 14:41, Vladimir Sementsov-Ogievskiy wrote: > 13.04.2018 11:51, Nikolay Shirokovskiy wrote: >> >> On 13.04.2018 03:04, John Snow wrote: >>> >>> On 04/12/2018 10:08 AM, Vladimir Sementsov-Ogievskiy wrote: I propose, not to say that bitmap represents a checkpoint. It is simpler

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-13 Thread Daniel P . Berrangé
On Fri, Apr 13, 2018 at 03:02:07PM +0300, Nikolay Shirokovskiy wrote: > > > On 13.04.2018 12:07, Daniel P. Berrangé wrote: > > On Tue, Apr 03, 2018 at 03:01:22PM +0300, Nikolay Shirokovskiy wrote: > >> *Temporary snapshot API* > >> > >> In previous version it is called 'Fleece API' after qemu

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-13 Thread Vladimir Sementsov-Ogievskiy
13.04.2018 00:35, John Snow wrote: On 04/12/2018 08:26 AM, Vladimir Sementsov-Ogievskiy wrote: 1. It looks unsafe to use nbd server + backup(sync=none) on same node, synchronization is needed, like in block/replication, which uses backup_wait_for_overlapping_requests, backup_cow_request_begin,

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-13 Thread Vladimir Sementsov-Ogievskiy
13.04.2018 11:51, Nikolay Shirokovskiy wrote: On 13.04.2018 03:04, John Snow wrote: On 04/12/2018 10:08 AM, Vladimir Sementsov-Ogievskiy wrote: I propose, not to say that bitmap represents a checkpoint. It is simpler to say (and it reflects the reality) that bitmap is a difference between

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-13 Thread Nikolay Shirokovskiy
On 13.04.2018 01:10, John Snow wrote: > > On 04/12/2018 04:58 AM, Nikolay Shirokovskiy wrote: >> On 11.04.2018 19:32, Eric Blake wrote: >>> On 04/03/2018 07:01 AM, Nikolay Shirokovskiy wrote: > > > [snip] > > >>> >>> I'm trying to figure out how BlockCheckpoint and BlockSnapshots relate.

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-13 Thread Nikolay Shirokovskiy
On 13.04.2018 00:16, John Snow wrote: > > > On 04/03/2018 08:01 AM, Nikolay Shirokovskiy wrote: >> Hi, all. >> >> >>

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-13 Thread Daniel P . Berrangé
On Tue, Apr 03, 2018 at 03:01:22PM +0300, Nikolay Shirokovskiy wrote: > *Temporary snapshot API* > > In previous version it is called 'Fleece API' after qemu terms and I'll still > use BlockSnapshot prefix for commands as in previous RFC instead of > TmpSnapshots which I inclined more now. > >

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-12 Thread John Snow
On 04/12/2018 10:08 AM, Vladimir Sementsov-Ogievskiy wrote: > > I propose, not to say that bitmap represents a checkpoint. It is simpler > to say (and it reflects the reality) that bitmap is a difference between > two consecutive checkpoints. And we can say, that active state is some > kind of

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-12 Thread John Snow
On 04/12/2018 08:57 AM, Nikolay Shirokovskiy wrote: > > > On 12.04.2018 07:14, John Snow wrote: >> >> >> On 04/11/2018 12:32 PM, Eric Blake wrote: >>> On 04/03/2018 07:01 AM, Nikolay Shirokovskiy wrote: Hi, all.

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-12 Thread John Snow
On 04/12/2018 04:58 AM, Nikolay Shirokovskiy wrote: > On 11.04.2018 19:32, Eric Blake wrote: >> On 04/03/2018 07:01 AM, Nikolay Shirokovskiy wrote: [snip] >> >> I'm trying to figure out how BlockCheckpoint and BlockSnapshots relate. >> Maybe it will be more clear when I read the

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-12 Thread John Snow
On 04/03/2018 08:01 AM, Nikolay Shirokovskiy wrote: > Hi, all. > > > > This is another RFC on pull backup

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-12 Thread Vladimir Sementsov-Ogievskiy
12.04.2018 15:57, Nikolay Shirokovskiy wrote: On 12.04.2018 07:14, John Snow wrote: On 04/11/2018 12:32 PM, Eric Blake wrote: On 04/03/2018 07:01 AM, Nikolay Shirokovskiy wrote: Hi, all. This

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-12 Thread Vladimir Sementsov-Ogievskiy
03.04.2018 15:01, Nikolay Shirokovskiy wrote: Hi, all. This is another RFC on pull backup API. This API provides means to read domain disks in a snapshotted state so that client can back them up

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-12 Thread Vladimir Sementsov-Ogievskiy
12.04.2018 11:58, Nikolay Shirokovskiy wrote: On 11.04.2018 19:32, Eric Blake wrote: On 04/03/2018 07:01 AM, Nikolay Shirokovskiy wrote: Hi, all. This is another RFC on pull backup API. This API

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-11 Thread John Snow
On 04/11/2018 12:32 PM, Eric Blake wrote: > On 04/03/2018 07:01 AM, Nikolay Shirokovskiy wrote: >> Hi, all. >> >> >>

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-11 Thread John Snow
On 04/11/2018 09:56 AM, Eric Blake wrote: > On 04/03/2018 07:01 AM, Nikolay Shirokovskiy wrote: >> Hi, all. >> >> >>

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-11 Thread Eric Blake
On 04/03/2018 07:01 AM, Nikolay Shirokovskiy wrote: > Hi, all. > > > > This is another RFC on pull backup

Re: [libvirt] [RFC v2] external (pull) backup API

2018-04-11 Thread Eric Blake
On 04/03/2018 07:01 AM, Nikolay Shirokovskiy wrote: > Hi, all. > > > > This is another RFC on pull backup

[libvirt] [RFC v2] external (pull) backup API

2018-04-03 Thread Nikolay Shirokovskiy
Hi, all. This is another RFC on pull backup API. This API provides means to read domain