Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-03-06 Thread Stefan Hajnoczi
On Mon, Feb 25, 2013 at 04:35:09PM +, Dietmar Maurer wrote: If we use nbd, how can we pass additional information to the other side, for example information about unallocated regions? You can either send trim commands, or just skip those regions and let the other side figure

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-03-06 Thread Dietmar Maurer
Or can we simply add a header to each write with additional information? Let's discuss it once we have a clearer picture of what that additional info would be. OK, just discovered such case. For example it would be nice to pass the size of the devices somehow? That is not

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-25 Thread Dietmar Maurer
So if I want to use the code from nbd.c, I need to write a specialized BlockDriver for the vma format (to pass that to nbd_export_new())? Yes. But I believe that would be a good thing to do anyway. For one thing, it gives you automatic coverage via qemu-iotests. But the only thing I

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-25 Thread Dietmar Maurer
unix sockets works with qemu nbd code? Sure. nbd+unix:///exportname?socket=path is the new URI syntax, I honestly forgot the old one. SCM_CREDENTIALS checks (qemu-nbd --pid or something like that) is not supported, but patches would be very welcome. Yes, this is better than my tcp

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-25 Thread Paolo Bonzini
Il 25/02/2013 13:59, Dietmar Maurer ha scritto: unix sockets works with qemu nbd code? Sure. nbd+unix:///exportname?socket=path is the new URI syntax, I honestly forgot the old one. SCM_CREDENTIALS checks (qemu-nbd --pid or something like that) is not supported, but patches would be very

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-25 Thread Stefan Hajnoczi
On Sat, Feb 23, 2013 at 09:05:37AM +, Dietmar Maurer wrote: Sure. nbd+unix:///exportname?socket=path is the new URI syntax, I honestly forgot the old one. SCM_CREDENTIALS checks (qemu-nbd --pid or something like that) is not supported, but patches would be very welcome.

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-25 Thread Dietmar Maurer
If we use nbd, how can we pass additional information to the other side, for example information about unallocated regions? You can either send trim commands, or just skip those regions and let the other side figure it out. But we are lost if we want to transfer something else? I

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-24 Thread Dietmar Maurer
-Original Message- From: Paolo Bonzini [mailto:pbonz...@redhat.com] Sent: Freitag, 22. Februar 2013 19:07 To: Dietmar Maurer Cc: Stefan Hajnoczi; Kevin Wolf; qemu-devel@nongnu.org; Wenchao Xia Subject: Re: [PATCH v4 0/6] Efficient VM backup for qemu Il 22/02/2013 18:57, Dietmar

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-24 Thread Paolo Bonzini
Sure. nbd+unix:///exportname?socket=path is the new URI syntax, I honestly forgot the old one. SCM_CREDENTIALS checks (qemu-nbd --pid or something like that) is not supported, but patches would be very welcome. Yes, this is better than my tcp suggestion. So if I want to use

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-23 Thread Dietmar Maurer
Sure. nbd+unix:///exportname?socket=path is the new URI syntax, I honestly forgot the old one. SCM_CREDENTIALS checks (qemu-nbd --pid or something like that) is not supported, but patches would be very welcome. Yes, this is better than my tcp suggestion. So if I want to use the code

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-22 Thread Stefan Hajnoczi
On Thu, Feb 21, 2013 at 03:56:15PM +, Dietmar Maurer wrote: On Wed, Feb 20, 2013 at 04:04:49PM +, Dietmar Maurer wrote: The backup writer abstraction is a special case interface somewhere between an image format and live migration. I'd prefer it if the backup block job stuck

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-22 Thread Kevin Wolf
Am 22.02.2013 um 11:31 hat Stefan Hajnoczi geschrieben: On Thu, Feb 21, 2013 at 03:56:15PM +, Dietmar Maurer wrote: On Wed, Feb 20, 2013 at 04:04:49PM +, Dietmar Maurer wrote: The backup writer abstraction is a special case interface somewhere between an image format and live

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-22 Thread Dietmar Maurer
Opening several NBD sockets 'nbd:127.0.0.1:1235' seems dangerous to me. One backup task and easily write to the wrong port - there is no protection? Or how can we guarantee that only one kvm process can write to that NBD device? transaction actions=[{'type': 'migrate-async',

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-22 Thread Paolo Bonzini
Il 22/02/2013 12:13, Dietmar Maurer ha scritto: Opening several NBD sockets 'nbd:127.0.0.1:1235' seems dangerous to me. One backup task and easily write to the wrong port - there is no protection? You can also use the named-export support. Something like --nbd=127.0.0.1:1235

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-22 Thread Dietmar Maurer
You can also use the named-export support. Something like --nbd=127.0.0.1:1235 --blockdev=virtio-drive0 --blockdev=virtio-drive1 --blockdev=ide0-hd0 etc. Then QEMU can migrate to nbd://127.0.0.1:1235/virtio-drive0. Or how can we guarantee that only one kvm process can write

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-22 Thread Paolo Bonzini
Il 22/02/2013 12:51, Dietmar Maurer ha scritto: You can also use the named-export support. Something like --nbd=127.0.0.1:1235 --blockdev=virtio-drive0 --blockdev=virtio-drive1 --blockdev=ide0-hd0 etc. Then QEMU can migrate to nbd://127.0.0.1:1235/virtio-drive0. Or

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-22 Thread Dietmar Maurer
Why is this needed? Security? I don't want that another process can write nonsense into my backup. They can already write nonsense to your iSCSI target, can't they? I am more concerned about software bugs. You need to find a free port, and then pass that port to kvm. If the original

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-22 Thread Paolo Bonzini
Il 22/02/2013 14:02, Dietmar Maurer ha scritto: Why is this needed? Security? I don't want that another process can write nonsense into my backup. They can already write nonsense to your iSCSI target, can't they? I am more concerned about software bugs. You need to find a free port, and

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-22 Thread Stefan Hajnoczi
On Fri, Feb 22, 2013 at 11:39 AM, Kevin Wolf kw...@redhat.com wrote: Am 22.02.2013 um 11:31 hat Stefan Hajnoczi geschrieben: On Thu, Feb 21, 2013 at 03:56:15PM +, Dietmar Maurer wrote: On Wed, Feb 20, 2013 at 04:04:49PM +, Dietmar Maurer wrote: I added the 'wait-for' attribute which

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-22 Thread Stefan Hajnoczi
On Fri, Feb 22, 2013 at 2:07 PM, Paolo Bonzini pbonz...@redhat.com wrote: Il 22/02/2013 14:02, Dietmar Maurer ha scritto: But you can always sandbox using SELinux, if you care about that, or use a Unix socket + SCM_CREDENTIALS. unix sockets works with qemu nbd code? Sure.

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-22 Thread Dietmar Maurer
On Fri, Feb 22, 2013 at 2:07 PM, Paolo Bonzini pbonz...@redhat.com wrote: Il 22/02/2013 14:02, Dietmar Maurer ha scritto: But you can always sandbox using SELinux, if you care about that, or use a Unix socket + SCM_CREDENTIALS. unix sockets works with qemu nbd code? Sure.

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-22 Thread Paolo Bonzini
Il 22/02/2013 18:49, Dietmar Maurer ha scritto: unix sockets works with qemu nbd code? Sure. nbd+unix:///exportname?socket=path is the new URI syntax, I honestly forgot the old one. SCM_CREDENTIALS checks (qemu-nbd --pid or something like that) is not supported, but patches would

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-22 Thread Dietmar Maurer
If we use nbd, how can we pass additional information to the other side, for example information about unallocated regions? You can either send trim commands, or just skip those regions and let the other side figure it out. But we are lost if we want to transfer something else? I am

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-22 Thread Paolo Bonzini
Il 22/02/2013 18:57, Dietmar Maurer ha scritto: If we use nbd, how can we pass additional information to the other side, for example information about unallocated regions? You can either send trim commands, or just skip those regions and let the other side figure it out. But we are lost

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-21 Thread Stefan Hajnoczi
On Thu, Feb 21, 2013 at 06:53:24AM +, Dietmar Maurer wrote: Interesting series, the backup block job makes sense to me. Regarding efficiency, I think incremental backup is a must, One can easily implement incremental backup on top of this patches. That is why I introduced the

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-21 Thread Stefan Hajnoczi
On Wed, Feb 20, 2013 at 04:04:49PM +, Dietmar Maurer wrote: Interesting series, the backup block job makes sense to me. Regarding efficiency, I think incremental backup is a must, otherwise regular backups using this approach won't really be a win over backing files. Incremental

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-21 Thread Kevin Wolf
On Thu, Feb 21, 2013 at 06:53:24AM +, Dietmar Maurer wrote: How would VMA fit in if the backup block job took a BlockDriverState destination and vmsave worked atomically with backup block jobs? You could have QEMU write to NBD server ports that a VMA writer process has open. VMA would

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-21 Thread Kevin Wolf
On Wed, Feb 20, 2013 at 04:04:49PM +, Dietmar Maurer wrote: The backup writer abstraction is a special case interface somewhere between an image format and live migration. I'd prefer it if the backup block job stuck to BlockDriverState as the destination for backup data. Then you

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-21 Thread Paolo Bonzini
Il 21/02/2013 16:56, Dietmar Maurer ha scritto: In _your_ use case. That means that we should support using non-seekable pipes, but it doesn't mean that any other use case is irrelevant. In fact I think backing up to a normal raw or qcow2 image file is the interesting case for the

[Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-20 Thread Dietmar Maurer
This series provides a way to efficiently backup VMs. * Backup to a single archive file * Backup contain all data to restore VM (full backup) * Do not depend on storage type or image format * Avoid use of temporary storage * store sparse images efficiently The file docs/backup.txt contains more

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-20 Thread Stefan Hajnoczi
On Wed, Feb 20, 2013 at 10:31:57AM +0100, Dietmar Maurer wrote: This series provides a way to efficiently backup VMs. * Backup to a single archive file * Backup contain all data to restore VM (full backup) * Do not depend on storage type or image format * Avoid use of temporary storage *

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-20 Thread Dietmar Maurer
Interesting series, the backup block job makes sense to me. Regarding efficiency, I think incremental backup is a must, otherwise regular backups using this approach won't really be a win over backing files. Incremental backup is something different, not touched by this code. You can add that

Re: [Qemu-devel] [PATCH v4 0/6] Efficient VM backup for qemu

2013-02-20 Thread Dietmar Maurer
Interesting series, the backup block job makes sense to me. Regarding efficiency, I think incremental backup is a must, One can easily implement incremental backup on top of this patches. That is why I introduced the BackupDriver abstraction. otherwise regular backups using this approach