Public bug reported:
We have been trying to analyze why under scale creating additional
routers and associating floating IP address with a VM causes the
operation to take longer completion times, and have found that the
programming of iptables seems to be an issue. Particularly 4 functions
(and their use) seem to degrade with large numbers of router instances.
We gathered this data with cprofile on the L3 agent. Tests were run to
collect data on the first router instantiation and on the 40th router
instantiation, with data also being collected at points between 1 and 40
routers. All the following functions showed an increasing trend:
(_find_last_entry)
(_weed_out_removes)
(_weed_out_duplicate_chains)
(_weed_out_duplicate_rules)
ncalls tottime percall cumtime percall filename:lineno(function)
For first router instantiation and Floating IP associate:
ncalls tottime percall cumtime percall filename:lineno(function)
126 0.000 0.000 0.000 0.000
/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:504(_find_last_entry)
178 0.000 0.000 0.000 0.000
/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:633(_weed_out_removes)
178 0.000 0.000 0.000 0.000
/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:608(_weed_out_duplicate_chains)
178 0.000 0.000 0.000 0.000
/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:622(_weed_out_duplicate_rules)
40 run of creating router
ncalls tottime percall cumtime percall filename:lineno(function)
313 0.001 0.000 0.004 0.000
/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:504(_find_last_entry)
371 0.000 0.000 0.004 0.000
/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:633(_weed_out_removes)
371 0.000 0.000 0.004 0.000
/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:608(_weed_out_duplicate_chains)
371 0.000 0.000 0.004 0.000
/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:622(_weed_out_duplicate_rules)
In particular, the weed out routines seem to do multiple iterations of matching
iptables rules with certain rules and then operate on them. The increasing
iteration numbers the weed out routines seems to be degrading the performance.
Does someone have information on whether this can be optimized?
Thanks,
-Uday
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1497379
Title:
iptables programming: Adding router instances takes increasing amount
of time
Status in neutron:
New
Bug description:
We have been trying to analyze why under scale creating additional
routers and associating floating IP address with a VM causes the
operation to take longer completion times, and have found that the
programming of iptables seems to be an issue. Particularly 4 functions
(and their use) seem to degrade with large numbers of router
instances.
We gathered this data with cprofile on the L3 agent. Tests were run to
collect data on the first router instantiation and on the 40th router
instantiation, with data also being collected at points between 1 and
40 routers. All the following functions showed an increasing trend:
(_find_last_entry)
(_weed_out_removes)
(_weed_out_duplicate_chains)
(_weed_out_duplicate_rules)
ncalls tottime percall cumtime percall filename:lineno(function)
For first router instantiation and Floating IP associate:
ncalls tottime percall cumtime percall filename:lineno(function)
126 0.000 0.000 0.000 0.000
/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:504(_find_last_entry)
178 0.000 0.000 0.000 0.000
/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:633(_weed_out_removes)
178 0.000 0.000 0.000 0.000
/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:608(_weed_out_duplicate_chains)
178 0.000 0.000 0.000 0.000
/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:622(_weed_out_duplicate_rules)
40 run of creating router
ncalls tottime percall cumtime percall filename:lineno(function)
313 0.001 0.000 0.004 0.000
/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:504(_find_last_entry)
371 0.000 0.000 0.004 0.000
/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:633(_weed_out_removes)
371 0.000 0.000 0.004 0.000
/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:608(_weed_out_duplicate_chains)
371 0.000 0.000 0.004 0.000
/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_manager.py:622(_weed_out_duplicate_rules)
In particular, the weed out routines seem to do multiple iterations of
matching iptables rules with certain rules and then operate on them. The
increasing iteration numbers the weed out routines seems to be degrading the
performance.
Does someone have information on whether this can be optimized?
Thanks,
-Uday
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1497379/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
