Public bug reported: We have been trying to analyze why under scale creating additional routers and associating floating IP address with a VM causes the operation to take longer completion times, and have found that the programming of iptables seems to be an issue. Particularly 4 functions (and their use) seem to degrade with large numbers of router instances.
We gathered this data with cprofile on the L3 agent. Tests were run to collect data on the first router instantiation and on the 40th router instantiation, with data also being collected at points between 1 and 40 routers. All the following functions showed an increasing trend: (_find_last_entry) (_weed_out_removes) (_weed_out_duplicate_chains) (_weed_out_duplicate_rules) ncalls tottime percall cumtime percall filename:lineno(function) For first router instantiation and Floating IP associate: ncalls tottime percall cumtime percall filename:lineno(function) 126 0.000 0.000 0.000 0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:504(_find_last_entry) 178 0.000 0.000 0.000 0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:633(_weed_out_removes) 178 0.000 0.000 0.000 0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:608(_weed_out_duplicate_chains) 178 0.000 0.000 0.000 0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:622(_weed_out_duplicate_rules) 40 run of creating router ncalls tottime percall cumtime percall filename:lineno(function) 313 0.001 0.000 0.004 0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:504(_find_last_entry) 371 0.000 0.000 0.004 0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:633(_weed_out_removes) 371 0.000 0.000 0.004 0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:608(_weed_out_duplicate_chains) 371 0.000 0.000 0.004 0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:622(_weed_out_duplicate_rules) In particular, the weed out routines seem to do multiple iterations of matching iptables rules with certain rules and then operate on them. The increasing iteration numbers the weed out routines seems to be degrading the performance. Does someone have information on whether this can be optimized? Thanks, -Uday ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1497379 Title: iptables programming: Adding router instances takes increasing amount of time Status in neutron: New Bug description: We have been trying to analyze why under scale creating additional routers and associating floating IP address with a VM causes the operation to take longer completion times, and have found that the programming of iptables seems to be an issue. Particularly 4 functions (and their use) seem to degrade with large numbers of router instances. We gathered this data with cprofile on the L3 agent. Tests were run to collect data on the first router instantiation and on the 40th router instantiation, with data also being collected at points between 1 and 40 routers. All the following functions showed an increasing trend: (_find_last_entry) (_weed_out_removes) (_weed_out_duplicate_chains) (_weed_out_duplicate_rules) ncalls tottime percall cumtime percall filename:lineno(function) For first router instantiation and Floating IP associate: ncalls tottime percall cumtime percall filename:lineno(function) 126 0.000 0.000 0.000 0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:504(_find_last_entry) 178 0.000 0.000 0.000 0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:633(_weed_out_removes) 178 0.000 0.000 0.000 0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:608(_weed_out_duplicate_chains) 178 0.000 0.000 0.000 0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:622(_weed_out_duplicate_rules) 40 run of creating router ncalls tottime percall cumtime percall filename:lineno(function) 313 0.001 0.000 0.004 0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:504(_find_last_entry) 371 0.000 0.000 0.004 0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:633(_weed_out_removes) 371 0.000 0.000 0.004 0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:608(_weed_out_duplicate_chains) 371 0.000 0.000 0.004 0.000 /usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:622(_weed_out_duplicate_rules) In particular, the weed out routines seem to do multiple iterations of matching iptables rules with certain rules and then operate on them. The increasing iteration numbers the weed out routines seems to be degrading the performance. Does someone have information on whether this can be optimized? Thanks, -Uday To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1497379/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp