Re: [libvirt] virsh domjobinfo during storage migration
On Fri, Nov 25, 2016 at 11:53 AM, Jiri Denemark wrote: >> How do you want to handle this, would you like me to open an issue in >> bugzilla? > > Yes please. Unless you want to provide the patches :-) I'd love to, but it is a bit outside of my area of expertise. I opened https://bugzilla.redhat.com/1398599 > > Jirka Thanks Jirka! -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] virsh domjobinfo during storage migration
On Fri, Nov 25, 2016 at 11:38:36 +0100, Ruben Kerkhof wrote: > Hi Jiri, > > On Fri, Nov 25, 2016 at 9:26 AM, Jiri Denemark wrote: > > I guess it should be possible to also check the progress of all running > > block jobs so that we can report statistics about ongoing storage > > migration even when NBD is used. > > That would be great, since I won't be able to upgrade to a newer > version of QEMU yet. > How do you want to handle this, would you like me to open an issue in > bugzilla? Yes please. Unless you want to provide the patches :-) Jirka -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] virsh domjobinfo during storage migration
Hi Jiri, On Fri, Nov 25, 2016 at 9:26 AM, Jiri Denemark wrote: > On Thu, Nov 24, 2016 at 22:54:10 +0100, Ruben Kerkhof wrote: >> Hi all, >> >> Running virsh domjobinfo on my CentOS 6 systems during a migration >> with --copy-storage-all, the output used to look like this: >> >> Job type: Unbounded >> Time elapsed: 1830632 ms >> Data processed: 37.212 GiB >> Data remaining: 1.025 GiB >> Data total: 16.016 GiB >> Memory processed: 37.212 GiB >> Memory remaining: 1.025 GiB >> Memory total: 16.016 GiB >> Memory bandwidth: 100.018 MiB/s >> Constant pages: 618279 >> Normal pages: 9734623 >> Normal data: 37.135 GiB >> Expected downtime: 1118 ms >> Setup time: 61 ms >> >> But when I run the same command on CentOS 7, libvirt-1.2.17-13 and >> qemu-kvm-1.5.3-105, I just get this: >> >> $ sudo virsh domjobinfo 58358fec-c35c-4a7a-a4dd-18ec2e1327bf >> Job type: Unbounded >> Time elapsed: 209616 ms >> >> I tried to debug this myself, but I'm a bit stuck. I'd expected >> libvirt to send query-migrate commands to qemu every 50 ms or so. To >> verify I attached gdb and put a breakpoint at >> qemuMonitorJSONGetMigrationStatus, but the breakpoint only hits after >> the complete disk is mirrored. > > That's expected since QEMU finally implemented a migration event, which > allowed us to stop polling every 50 ms. Libvirt will only call > query-migrate at the end of a migration or when virsh domjobinfo is > called. > > The reason why you don't see anything interesting in the output is NBD > storage migration. With new enough QEMU libvirt will first migrate > storage using QEMU's integrated NBD server and then "migrate" QMP > command will called. With old QEMU storage migration was done by the > "migrate" command itself. Thus calling query-migrate while storage is > migrated (i.e., before migrate was called) does not provide anything. That makes sense, thanks for the explanation. > I guess it should be possible to also check the progress of all running > block jobs so that we can report statistics about ongoing storage > migration even when NBD is used. That would be great, since I won't be able to upgrade to a newer version of QEMU yet. How do you want to handle this, would you like me to open an issue in bugzilla? > > Jirka Kind regards, Ruben Kerkhof -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] virsh domjobinfo during storage migration
On Thu, Nov 24, 2016 at 22:54:10 +0100, Ruben Kerkhof wrote: > Hi all, > > Running virsh domjobinfo on my CentOS 6 systems during a migration > with --copy-storage-all, the output used to look like this: > > Job type: Unbounded > Time elapsed: 1830632 ms > Data processed: 37.212 GiB > Data remaining: 1.025 GiB > Data total: 16.016 GiB > Memory processed: 37.212 GiB > Memory remaining: 1.025 GiB > Memory total: 16.016 GiB > Memory bandwidth: 100.018 MiB/s > Constant pages: 618279 > Normal pages: 9734623 > Normal data: 37.135 GiB > Expected downtime: 1118 ms > Setup time: 61 ms > > But when I run the same command on CentOS 7, libvirt-1.2.17-13 and > qemu-kvm-1.5.3-105, I just get this: > > $ sudo virsh domjobinfo 58358fec-c35c-4a7a-a4dd-18ec2e1327bf > Job type: Unbounded > Time elapsed: 209616 ms > > I tried to debug this myself, but I'm a bit stuck. I'd expected > libvirt to send query-migrate commands to qemu every 50 ms or so. To > verify I attached gdb and put a breakpoint at > qemuMonitorJSONGetMigrationStatus, but the breakpoint only hits after > the complete disk is mirrored. That's expected since QEMU finally implemented a migration event, which allowed us to stop polling every 50 ms. Libvirt will only call query-migrate at the end of a migration or when virsh domjobinfo is called. The reason why you don't see anything interesting in the output is NBD storage migration. With new enough QEMU libvirt will first migrate storage using QEMU's integrated NBD server and then "migrate" QMP command will called. With old QEMU storage migration was done by the "migrate" command itself. Thus calling query-migrate while storage is migrated (i.e., before migrate was called) does not provide anything. I guess it should be possible to also check the progress of all running block jobs so that we can report statistics about ongoing storage migration even when NBD is used. Jirka -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list