Re: [Openstack-operators] [nova] Live migration performance tests on 100 compute nodes
And you can use the nova option "live_migration_inbound_addr" [1] to avoid the impact of live migration at all, but that require additional network plane. [1]: https://github.com/openstack/nova/blob/master/nova/conf/libvirt.py#L243-L258 [2]: https://blueprints.launchpad.net/nova/+spec/split-network-plane-for-live-migration 2016-12-30 20:56 GMT+08:00 Koniszewski, Pawel : > Matt, > > > > This was a bandwidth issue. Nova kept being connected to the broker, but > it started to timeout on RPC messages, e.g., we lost some RPC messages > triggering post live migration steps which are there to update nova DB to > reflect new host of an instance. > > > > The good workaround for this issue is to slightly limit bandwidth used for > live migrations through nova.conf [1], live_migration_bandwidth config > option in [libvirt] section. By default it is set to 0, so, basically, it > is unlimited. Also please be aware that we changed default live migration > configuration in OpenStack Newton and tunneling is now off by default > (live_migration_tunnelled in [libvirt] section is set to False) due to huge > performance impact. > > > > [1] http://docs.openstack.org/newton/config-reference/ > compute/config-options.html > > > > Kind Regards, > > Pawel Koniszewski > > > > *From:* tadow...@gmail.com [mailto:tadow...@gmail.com] *On Behalf Of *Matt > Fischer > *Sent:* Friday, December 30, 2016 5:31 AM > *To:* Koniszewski, Pawel > *Cc:* openstack-operators@lists.openstack.org > *Subject:* Re: [Openstack-operators] [nova] Live migration performance > tests on 100 compute nodes > > > > On Wed, Dec 28, 2016 at 6:11 AM, Koniszewski, Pawel < > pawel.koniszew...@intel.com> wrote: > > Hello everyone, > > We made a research to see how live migration performance varies between > different configurations, especially we aimed to test tunneled vs > non-tunneled live migrations. To test live migration we simulated a case of > 0-day patching of 100 compute nodes (including reboot) with workloads that > are close to the real world workloads. All the results were published [1] > along with environment configuration and how we built test framework. Hope > you find this useful. > > [1] https://01.org/openstack/blogs/pkoniszewski/2016/ossc- > zero-day-patching > > Kind Regards, > Pawel Koniszewski > > > > Thanks for the write-up. I'm curious about your RabbitMQ connection > failures. Was it nova-compute failing to connect? Was it a bandwidth or > heartbeat issue? > > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > > ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [nova] Live migration performance tests on 100 compute nodes
Matt, This was a bandwidth issue. Nova kept being connected to the broker, but it started to timeout on RPC messages, e.g., we lost some RPC messages triggering post live migration steps which are there to update nova DB to reflect new host of an instance. The good workaround for this issue is to slightly limit bandwidth used for live migrations through nova.conf [1], live_migration_bandwidth config option in [libvirt] section. By default it is set to 0, so, basically, it is unlimited. Also please be aware that we changed default live migration configuration in OpenStack Newton and tunneling is now off by default (live_migration_tunnelled in [libvirt] section is set to False) due to huge performance impact. [1] http://docs.openstack.org/newton/config-reference/compute/config-options.html Kind Regards, Pawel Koniszewski From: tadow...@gmail.com [mailto:tadow...@gmail.com] On Behalf Of Matt Fischer Sent: Friday, December 30, 2016 5:31 AM To: Koniszewski, Pawel Cc: openstack-operators@lists.openstack.org Subject: Re: [Openstack-operators] [nova] Live migration performance tests on 100 compute nodes On Wed, Dec 28, 2016 at 6:11 AM, Koniszewski, Pawel mailto:pawel.koniszew...@intel.com>> wrote: Hello everyone, We made a research to see how live migration performance varies between different configurations, especially we aimed to test tunneled vs non-tunneled live migrations. To test live migration we simulated a case of 0-day patching of 100 compute nodes (including reboot) with workloads that are close to the real world workloads. All the results were published [1] along with environment configuration and how we built test framework. Hope you find this useful. [1] https://01.org/openstack/blogs/pkoniszewski/2016/ossc-zero-day-patching Kind Regards, Pawel Koniszewski Thanks for the write-up. I'm curious about your RabbitMQ connection failures. Was it nova-compute failing to connect? Was it a bandwidth or heartbeat issue? ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [nova] Live migration performance tests on 100 compute nodes
On Wed, Dec 28, 2016 at 6:11 AM, Koniszewski, Pawel < pawel.koniszew...@intel.com> wrote: > Hello everyone, > > We made a research to see how live migration performance varies between > different configurations, especially we aimed to test tunneled vs > non-tunneled live migrations. To test live migration we simulated a case of > 0-day patching of 100 compute nodes (including reboot) with workloads that > are close to the real world workloads. All the results were published [1] > along with environment configuration and how we built test framework. Hope > you find this useful. > > [1] https://01.org/openstack/blogs/pkoniszewski/2016/ossc- > zero-day-patching > > Kind Regards, > Pawel Koniszewski > Thanks for the write-up. I'm curious about your RabbitMQ connection failures. Was it nova-compute failing to connect? Was it a bandwidth or heartbeat issue? ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] [nova] Live migration performance tests on 100 compute nodes
Hello everyone, We made a research to see how live migration performance varies between different configurations, especially we aimed to test tunneled vs non-tunneled live migrations. To test live migration we simulated a case of 0-day patching of 100 compute nodes (including reboot) with workloads that are close to the real world workloads. All the results were published [1] along with environment configuration and how we built test framework. Hope you find this useful. [1] https://01.org/openstack/blogs/pkoniszewski/2016/ossc-zero-day-patching Kind Regards, Pawel Koniszewski ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
