[Openstack] New instances booting time
Hello, guys, I have a question We now have OS Kilo + KVM+ Ubuntu 14.04 Nova-compute.conf: [libvirt] virt_type=kvm images_type = lvm images_volume_group =openstack-controller01-ky01-vg volume_clear = none the problem is when nova boots a new instance it's SUPER slow exactly this step: Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-img convert -O raw /var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e /dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b608c_disk Well, I understand what this step is doing - it's copying raw image to lvm. How can we speed it up? I don't wanna have instance from 100GB image booted for 4 hours Regards, IT engineer Farheap, Russia Ivan Derbenev ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] New instances booting time
bump Regards, IT engineer Farheap, Russia Ivan Derbenev From: Ivan Derbenev [mailto:ivan.derbe...@tech-corps.com] Sent: Wednesday, August 5, 2015 1:21 PM To: openstack@lists.openstack.org Subject: [Openstack] New instances booting time Hello, guys, I have a question We now have OS Kilo + KVM+ Ubuntu 14.04 Nova-compute.conf: [libvirt] virt_type=kvm images_type = lvm images_volume_group =openstack-controller01-ky01-vg volume_clear = none the problem is when nova boots a new instance it's SUPER slow exactly this step: Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-img convert -O raw /var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e /dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b608c_disk Well, I understand what this step is doing - it's copying raw image to lvm. How can we speed it up? I don't wanna have instance from 100GB image booted for 4 hours Regards, IT engineer Farheap, Russia Ivan Derbenev ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] New instances booting time
On 08/13/2015 11:37 PM, Ivan Derbenev wrote: *From:*Ivan Derbenev [mailto:ivan.derbe...@tech-corps.com] *Sent:* Wednesday, August 5, 2015 1:21 PM *To:* openstack@lists.openstack.org *Subject:* [Openstack] New instances booting time Hello, guys, I have a question We now have OS Kilo + KVM+ Ubuntu 14.04 Nova-compute.conf: [libvirt] virt_type=kvm images_type = lvm images_volume_group =openstack-controller01-ky01-vg volume_clear = none the problem is when nova boots a new instance it’s SUPER slow exactly this step: Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-img convert -O raw /var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e /dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b608c_disk Well, I understand what this step is doing – it’s copying raw image to lvm. How can we speed it up? I don’t wanna have instance from 100GB image booted for 4 hours Don't use base images that are 100G in size. Quite simple, really. Best, -jay ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] New instances booting time
On Fri, Aug 14, 2015 at 09:55:42AM -0400, Jay Pipes wrote: > On 08/13/2015 11:37 PM, Ivan Derbenev wrote: > >*From:*Ivan Derbenev [mailto:ivan.derbe...@tech-corps.com] > >*Sent:* Wednesday, August 5, 2015 1:21 PM > >*To:* openstack@lists.openstack.org > >*Subject:* [Openstack] New instances booting time > > > >Hello, guys, I have a question > > > >We now have OS Kilo + KVM+ Ubuntu 14.04 > > > >Nova-compute.conf: > > > >[libvirt] > > > >virt_type=kvm > > > >images_type = lvm > > > >images_volume_group =openstack-controller01-ky01-vg > > > >volume_clear = none > > > >the problem is when nova boots a new instance it’s SUPER slow > > > >exactly this step: > > > >Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf > >qemu-img convert -O raw > >/var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e > >/dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b608c_disk > > > >Well, I understand what this step is doing – it’s copying raw image to lvm. > > > >How can we speed it up? > > > >I don’t wanna have instance from 100GB image booted for 4 hours > > Don't use base images that are 100G in size. Quite simple, really. Presumably the actual image is only a few 100 MB in size, but the virtual qcow2 size is 100 GB ? I guess qemu-img probably has to still write out 100 GB of zeros as it can't assume that the LVM target is pre-zeroed. I do wonder though if there's a way to optimize this so that qemu-img only has to write out the 100 MB of actual data, and not al the zeros. Perhaps there's scope for a flag to qemu-img to tell it to seek in the target when it sees holes in the source image. If OpenStack then used sparse LVM images, there would be no need to rwite out 100's of GB of zeros. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] New instances booting time
On Fri, Aug 14, 2015 at 03:10:13PM +0100, Daniel P. Berrange wrote: > On Fri, Aug 14, 2015 at 09:55:42AM -0400, Jay Pipes wrote: > > On 08/13/2015 11:37 PM, Ivan Derbenev wrote: > > >*From:*Ivan Derbenev [mailto:ivan.derbe...@tech-corps.com] > > >*Sent:* Wednesday, August 5, 2015 1:21 PM > > >*To:* openstack@lists.openstack.org > > >*Subject:* [Openstack] New instances booting time > > > > > >Hello, guys, I have a question > > > > > >We now have OS Kilo + KVM+ Ubuntu 14.04 > > > > > >Nova-compute.conf: > > > > > >[libvirt] > > > > > >virt_type=kvm > > > > > >images_type = lvm > > > > > >images_volume_group =openstack-controller01-ky01-vg > > > > > >volume_clear = none > > > > > >the problem is when nova boots a new instance it’s SUPER slow > > > > > >exactly this step: > > > > > >Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf > > >qemu-img convert -O raw > > >/var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e > > >/dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b608c_disk > > > > > >Well, I understand what this step is doing – it’s copying raw image to lvm. > > > > > >How can we speed it up? > > > > > >I don’t wanna have instance from 100GB image booted for 4 hours > > > > Don't use base images that are 100G in size. Quite simple, really. > > Presumably the actual image is only a few 100 MB in size, but the > virtual qcow2 size is 100 GB ? > > I guess qemu-img probably has to still write out 100 GB of zeros > as it can't assume that the LVM target is pre-zeroed. I do wonder > though if there's a way to optimize this so that qemu-img only > has to write out the 100 MB of actual data, and not al the zeros. > > Perhaps there's scope for a flag to qemu-img to tell it to seek > in the target when it sees holes in the source image. If OpenStack > then used sparse LVM images, there would be no need to rwite out > 100's of GB of zeros. Speaking with the KVM block layer maintainers, the behaviour seen is not entirely expected. qemu-img wants to ensure no existing data is left behind as that could cause data corruption if the original qcow2 file had dropped empty sectors. So it has to make sure it can write out the 100 GB even if that is mostly zeros. That said, it has support for using discard, and it will use that in preference to writing zeros, which should be really fast. For it to work with block devices though, it seems that we need to disable the I/O cache by passing '-t none' to qemu-img when doing this conversion. This also assumes the underlying storage that the user has configured behind their LVM volume supports discard which may not always be the case. So it would be interesting to know if changing $ qemu-img convert -O raw \ /var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e \ /dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b608c_disk to $ qemu-img convert -O raw -t none \ /var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e \ /dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b608c_disk makes any significant difference to the speed. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] New instances booting time
On Fri, Aug 14, 2015 at 03:34:09PM +0100, Daniel P. Berrange wrote: > On Fri, Aug 14, 2015 at 03:10:13PM +0100, Daniel P. Berrange wrote: > > On Fri, Aug 14, 2015 at 09:55:42AM -0400, Jay Pipes wrote: > > > On 08/13/2015 11:37 PM, Ivan Derbenev wrote: > > > >*From:*Ivan Derbenev [mailto:ivan.derbe...@tech-corps.com] > > > >*Sent:* Wednesday, August 5, 2015 1:21 PM > > > >*To:* openstack@lists.openstack.org > > > >*Subject:* [Openstack] New instances booting time > > > > > > > >Hello, guys, I have a question > > > > > > > >We now have OS Kilo + KVM+ Ubuntu 14.04 > > > > > > > >Nova-compute.conf: > > > > > > > >[libvirt] > > > > > > > >virt_type=kvm > > > > > > > >images_type = lvm > > > > > > > >images_volume_group =openstack-controller01-ky01-vg > > > > > > > >volume_clear = none > > > > > > > >the problem is when nova boots a new instance it’s SUPER slow > > > > > > > >exactly this step: > > > > > > > >Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf > > > >qemu-img convert -O raw > > > >/var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e > > > >/dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b608c_disk > > > > > > > >Well, I understand what this step is doing – it’s copying raw image to > > > >lvm. > > > > > > > >How can we speed it up? > > > > > > > >I don’t wanna have instance from 100GB image booted for 4 hours > > > > > > Don't use base images that are 100G in size. Quite simple, really. > > > > Presumably the actual image is only a few 100 MB in size, but the > > virtual qcow2 size is 100 GB ? > > > > I guess qemu-img probably has to still write out 100 GB of zeros > > as it can't assume that the LVM target is pre-zeroed. I do wonder > > though if there's a way to optimize this so that qemu-img only > > has to write out the 100 MB of actual data, and not al the zeros. > > > > Perhaps there's scope for a flag to qemu-img to tell it to seek > > in the target when it sees holes in the source image. If OpenStack > > then used sparse LVM images, there would be no need to rwite out > > 100's of GB of zeros. > > Speaking with the KVM block layer maintainers, the behaviour seen > is not entirely expected. qemu-img wants to ensure no existing > data is left behind as that could cause data corruption if the > original qcow2 file had dropped empty sectors. So it has to make > sure it can write out the 100 GB even if that is mostly zeros. > > That said, it has support for using discard, and it will use that > in preference to writing zeros, which should be really fast. For > it to work with block devices though, it seems that we need to > disable the I/O cache by passing '-t none' to qemu-img when doing > this conversion. This also assumes the underlying storage that > the user has configured behind their LVM volume supports discard > which may not always be the case. > > So it would be interesting to know if changing > > $ qemu-img convert -O raw \ > /var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e \ > > /dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b608c_disk > > to > > $ qemu-img convert -O raw -t none \ > /var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e \ > > /dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b608c_disk > > makes any significant difference to the speed. I filed a bug to track this possible enhancement https://bugs.launchpad.net/nova/+bug/1484992 Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] New instances booting time
Well, thanks. Actually, wanted to know, if this behavior is ok for every installation, or I has misconfigured smthing Regards, IT engineer Farheap, Russia Ivan Derbenev -Original Message- From: Daniel P. Berrange [mailto:berra...@redhat.com] Sent: Friday, August 14, 2015 5:56 PM To: Jay Pipes Cc: openstack@lists.openstack.org Subject: Re: [Openstack] New instances booting time On Fri, Aug 14, 2015 at 03:34:09PM +0100, Daniel P. Berrange wrote: > On Fri, Aug 14, 2015 at 03:10:13PM +0100, Daniel P. Berrange wrote: > > On Fri, Aug 14, 2015 at 09:55:42AM -0400, Jay Pipes wrote: > > > On 08/13/2015 11:37 PM, Ivan Derbenev wrote: > > > >*From:*Ivan Derbenev [mailto:ivan.derbe...@tech-corps.com] > > > >*Sent:* Wednesday, August 5, 2015 1:21 PM > > > >*To:* openstack@lists.openstack.org > > > >*Subject:* [Openstack] New instances booting time > > > > > > > >Hello, guys, I have a question > > > > > > > >We now have OS Kilo + KVM+ Ubuntu 14.04 > > > > > > > >Nova-compute.conf: > > > > > > > >[libvirt] > > > > > > > >virt_type=kvm > > > > > > > >images_type = lvm > > > > > > > >images_volume_group =openstack-controller01-ky01-vg > > > > > > > >volume_clear = none > > > > > > > >the problem is when nova boots a new instance it’s SUPER slow > > > > > > > >exactly this step: > > > > > > > >Running cmd (subprocess): sudo nova-rootwrap > > > >/etc/nova/rootwrap.conf qemu-img convert -O raw > > > >/var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a1 > > > >9b42e > > > >/dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a6623 > > > >27b608c_disk > > > > > > > >Well, I understand what this step is doing – it’s copying raw image to > > > >lvm. > > > > > > > >How can we speed it up? > > > > > > > >I don’t wanna have instance from 100GB image booted for 4 hours > > > > > > Don't use base images that are 100G in size. Quite simple, really. > > > > Presumably the actual image is only a few 100 MB in size, but the > > virtual qcow2 size is 100 GB ? > > > > I guess qemu-img probably has to still write out 100 GB of zeros as > > it can't assume that the LVM target is pre-zeroed. I do wonder > > though if there's a way to optimize this so that qemu-img only has > > to write out the 100 MB of actual data, and not al the zeros. > > > > Perhaps there's scope for a flag to qemu-img to tell it to seek in > > the target when it sees holes in the source image. If OpenStack then > > used sparse LVM images, there would be no need to rwite out 100's of > > GB of zeros. > > Speaking with the KVM block layer maintainers, the behaviour seen is > not entirely expected. qemu-img wants to ensure no existing data is > left behind as that could cause data corruption if the original qcow2 > file had dropped empty sectors. So it has to make sure it can write > out the 100 GB even if that is mostly zeros. > > That said, it has support for using discard, and it will use that in > preference to writing zeros, which should be really fast. For it to > work with block devices though, it seems that we need to disable the > I/O cache by passing '-t none' to qemu-img when doing this conversion. > This also assumes the underlying storage that the user has configured > behind their LVM volume supports discard which may not always be the > case. > > So it would be interesting to know if changing > > $ qemu-img convert -O raw \ > /var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e \ > > /dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b60 > 8c_disk > > to > > $ qemu-img convert -O raw -t none \ > /var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e \ > > /dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b60 > 8c_disk > > makes any significant difference to the speed. I filed a bug to track this possible enhancement https://bugs.launchpad.net/nova/+bug/1484992 Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] New instances booting time
Well, I have 4gb qcow2 image But it contains a 100gb filesystem And the rest 96 gbs are still copied Regards, IT engineer Farheap, Russia Ivan Derbenev -Original Message- From: Jay Pipes [mailto:jaypi...@gmail.com] Sent: Friday, August 14, 2015 4:56 PM To: openstack@lists.openstack.org Subject: Re: [Openstack] New instances booting time On 08/13/2015 11:37 PM, Ivan Derbenev wrote: > *From:*Ivan Derbenev [mailto:ivan.derbe...@tech-corps.com] > *Sent:* Wednesday, August 5, 2015 1:21 PM > *To:* openstack@lists.openstack.org > *Subject:* [Openstack] New instances booting time > > Hello, guys, I have a question > > We now have OS Kilo + KVM+ Ubuntu 14.04 > > Nova-compute.conf: > > [libvirt] > > virt_type=kvm > > images_type = lvm > > images_volume_group =openstack-controller01-ky01-vg > > volume_clear = none > > the problem is when nova boots a new instance it's SUPER slow > > exactly this step: > > Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf > qemu-img convert -O raw > /var/lib/nova/instances/_base/999f7fff2521e4a7243c9e1d21599fd64a19b42e > /dev/openstack-controller01-ky01-vg/5f831046-435c-4636-8b71-a662327b60 > 8c_disk > > Well, I understand what this step is doing - it's copying raw image to lvm. > > How can we speed it up? > > I don't wanna have instance from 100GB image booted for 4 hours Don't use base images that are 100G in size. Quite simple, really. Best, -jay ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
