[Bug 1361357] Re: metadata service performance regression ~8x
** Changed in: neutron (Ubuntu Utopic) Status: Confirmed = Fix Released ** Changed in: neutron (Ubuntu) Status: Confirmed = Fix Released ** Changed in: cloud-archive/icehouse Status: Confirmed = Fix Released ** Changed in: cloud-archive/juno Status: Confirmed = Fix Released ** Changed in: neutron (Ubuntu Trusty) Status: Confirmed = Fix Released -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
I've confirmed that non-caching performance on icehouse 2014.1.3 has been restored back to the 2014.1.1 range. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
** Changed in: neutron Assignee: Oleg Bondarev (obondarev) = (unassigned) ** Changed in: neutron Milestone: kilo-1 = None -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
As per a thread on the mailing list [1], this issue was already fixed [2] in Neutron in Juno and backported to Icehouse, so I'm going to remove Neutron as an affected project. 1: http://lists.openstack.org/pipermail/openstack-dev/2014-October/048916.html 2: https://bugs.launchpad.net/neutron/+bug/1276440 ** No longer affects: neutron -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
** Tags removed: icehouse-backport-potential juno-backport-potential -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
** Changed in: neutron Milestone: None = kilo-1 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
** Changed in: neutron Importance: Undecided = High ** Changed in: neutron Assignee: Corey Bryant (corey.bryant) = Oleg Bondarev (obondarev) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
I'm changing assignment to Oleg since he has submitted a patch to switch to eliminate the need for client initialization by using RPC. ** Changed in: neutron Importance: High = Medium ** Changed in: neutron Milestone: None = kilo-1 ** Tags added: juno-backport-potential -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
** Changed in: neutron Milestone: kilo-1 = None -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
** Tags added: icehouse-backport-potential -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
** Changed in: neutron (Ubuntu Trusty) Importance: Undecided = Medium ** Changed in: neutron (Ubuntu Utopic) Importance: Undecided = Medium -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
Fix proposed to branch: master Review: https://review.openstack.org/121649 ** Changed in: neutron Status: Confirmed = In Progress ** Changed in: neutron Assignee: (unassigned) = Corey Bryant (corey.bryant) -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
** Attachment added: ec2metadata-cached-with-fix.txt https://bugs.launchpad.net/cloud-archive/+bug/1361357/+attachment/4204660/+files/ec2metadata-cached-with-fix.txt -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
I just posted a patch for this. I'm attaching results for non-cached metadata performance before and after the fix, and I'm also attaching results for cached metadata performance before and after the fix (using cache_url = memory://?default_ttl=5). -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
** Attachment added: ec2metadata-nocached-with-fix.txt https://bugs.launchpad.net/cloud-archive/+bug/1361357/+attachment/4204658/+files/ec2metadata-nocached-with-fix.txt -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
** Attachment added: ec2metadata-cached-without-fix.txt https://bugs.launchpad.net/cloud-archive/+bug/1361357/+attachment/4204659/+files/ec2metadata-cached-without-fix.txt -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
** Attachment added: ec2metadata-nocached-without-fix.txt https://bugs.launchpad.net/cloud-archive/+bug/1361357/+attachment/4204657/+files/ec2metadata-nocached-without-fix.txt -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
** Summary changed: - metadata service performance regression ~100x + metadata service performance regression ~8x -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1361357] Re: metadata service performance regression ~8x
I updated the description with a better diagnosis of the problem, as the first was way off, and only would cause confusion. ** Description changed: - The fix for bug 1325128 [1] included changes to the metadata service - seen at: - trunk[2]: 4a60c6a655006b2882331844664fac5cf67c5f34 - icehouse [3]: 9f59ca751f1a392ef24d8ab73a7bf5ce9655017e + A change was made to the neutron portion of the nova metadata service to + disable caching. This ends up causing a all hits to the nova metadata + service to generate messages to retrieve data from neutron. Doing so + makes a crawl of the metadata service take at least 8x longer than it + did. cloud-init crawls the metadata service during boot. The end + result is that instances boot significantly slower than they did + previously. - The new code is around 100x slower. That slow down causes - excessive load on the metadata server if an instance crawls the metadata - service (which generates dozens of http requests). + The commits are marked as having fixed bug 1276440 [1]. + The commits are: + icehouse: d568fee34be36ca17a9124fe6539f62d702d6359 [2] + trunk: 3faea81c6029033c85cefd6e98d7a3e719e858f5 [3] - Both cloud-init and cirros's boot code crawl that meta-data service. The - end result is linux instances boot time is dramatically affected. A - typical Ubuntu boot time will be 10 seconds to executing user code, but - after this regression, I'm seeing boots in the range of 30 to 45 seconds. - - While the fix is exactly intended to slow things down to improve - security, the performance regression seems significant enough to - address. - - -- - [1] http://pad.lv/1325128 - [2] https://git.openstack.org/cgit/openstack/nova/commit/?id=4a60c6a655006b2882331844664fac5cf67c5f34 - [3] https://git.openstack.org/cgit/openstack/nova/commit/?id=4a60c6a655006b2882331844664fac5cf67c5f34 + [1] http://pad.lv/1276440 + [2] https://github.com/openstack/neutron/commit/d568fee34be36ca17a9124fe6539f62d702d6359 + [3] https://github.com/openstack/neutron/commit/423ca756af10e10398636d6d34a7594a4fd4bc87 -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to neutron in Ubuntu. https://bugs.launchpad.net/bugs/1361357 Title: metadata service performance regression ~8x To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1361357/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
