** Also affects: neutron Importance: Undecided Status: New ** No longer affects: openvswitch (Ubuntu)
** Summary changed: - ovsdb-client processes not getting cleaned up + neutron-rootwrap processes not getting cleaned up ** Description changed: - This can be recreated with the openstack charms using xenial-newton-staging. On newton deploys, neutron-gateway and nova-compute units will exhaust memory due to compounding ovsdb-client processes: + neutron-rootwrap processes aren't getting cleaned up on Newton. I'm + testing with Newton rc3. + + I was noticing memory exhaustion on my neutron gateway units, which turned out to be due to compounding neutron-rootwrap processes: sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport,external_ids --format=json $ top -n1 -b -o VIRT http://paste.ubuntu.com/23252407/ $ ps aux|grep ovsdb-client http://paste.ubuntu.com/23252658/ Restarting openvswitch cleans up the processes but they just start piling again up soon after: sudo systemctl restart openvswitch-switch + + At first I thought this was an openvswitch issue, however I reverted the + code in get_root_helper_child_pid() and neutron-rootwrap processes + started getting cleaned up. See corresponding commit at [1]. + + This can be recreated with the openstack charms using xenial-newton- + staging. On newton deploys, neutron-gateway and nova-compute units will + exhaust memory due to compounding ovsdb-client processes. + + [1] + commit fd93e19f2a415b3803700fc491749daba01a4390 + Author: Assaf Muller <amul...@redhat.com> + Date: Fri Mar 18 16:29:26 2016 -0400 + + Change get_root_helper_child_pid to stop when it finds cmd + + get_root_helper_child_pid recursively finds the child of pid, + until it can no longer find a child. However, the intention is + not to find the deepest child, but to strip away root helpers. + For example 'sudo neutron-rootwrap x' is supposed to find the + pid of x. However, in cases 'x' spawned quick lived children of + its own (For example: ip / brctl / ovs invocations), + get_root_helper_child_pid returned those pids if called in + the wrong time. + + Change-Id: I582aa5c931c8bfe57f49df6899445698270bb33e + Closes-Bug: #1558819 ** Description changed: neutron-rootwrap processes aren't getting cleaned up on Newton. I'm testing with Newton rc3. I was noticing memory exhaustion on my neutron gateway units, which turned out to be due to compounding neutron-rootwrap processes: sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport,external_ids --format=json $ top -n1 -b -o VIRT http://paste.ubuntu.com/23252407/ $ ps aux|grep ovsdb-client http://paste.ubuntu.com/23252658/ Restarting openvswitch cleans up the processes but they just start piling again up soon after: sudo systemctl restart openvswitch-switch At first I thought this was an openvswitch issue, however I reverted the code in get_root_helper_child_pid() and neutron-rootwrap processes - started getting cleaned up. See corresponding commit at [1]. + started getting cleaned up. See corresponding commit for code that + possibly introduced this at [1]. This can be recreated with the openstack charms using xenial-newton- staging. On newton deploys, neutron-gateway and nova-compute units will exhaust memory due to compounding ovsdb-client processes. [1] commit fd93e19f2a415b3803700fc491749daba01a4390 Author: Assaf Muller <amul...@redhat.com> Date: Fri Mar 18 16:29:26 2016 -0400 - Change get_root_helper_child_pid to stop when it finds cmd - - get_root_helper_child_pid recursively finds the child of pid, - until it can no longer find a child. However, the intention is - not to find the deepest child, but to strip away root helpers. - For example 'sudo neutron-rootwrap x' is supposed to find the - pid of x. However, in cases 'x' spawned quick lived children of - its own (For example: ip / brctl / ovs invocations), - get_root_helper_child_pid returned those pids if called in - the wrong time. - - Change-Id: I582aa5c931c8bfe57f49df6899445698270bb33e - Closes-Bug: #1558819 + Change get_root_helper_child_pid to stop when it finds cmd + + get_root_helper_child_pid recursively finds the child of pid, + until it can no longer find a child. However, the intention is + not to find the deepest child, but to strip away root helpers. + For example 'sudo neutron-rootwrap x' is supposed to find the + pid of x. However, in cases 'x' spawned quick lived children of + its own (For example: ip / brctl / ovs invocations), + get_root_helper_child_pid returned those pids if called in + the wrong time. + + Change-Id: I582aa5c931c8bfe57f49df6899445698270bb33e + Closes-Bug: #1558819 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1629097 Title: neutron-rootwrap processes not getting cleaned up Status in neutron: New Bug description: neutron-rootwrap processes aren't getting cleaned up on Newton. I'm testing with Newton rc3. I was noticing memory exhaustion on my neutron gateway units, which turned out to be due to compounding neutron-rootwrap processes: sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport,external_ids --format=json $ top -n1 -b -o VIRT http://paste.ubuntu.com/23252407/ $ ps aux|grep ovsdb-client http://paste.ubuntu.com/23252658/ Restarting openvswitch cleans up the processes but they just start piling again up soon after: sudo systemctl restart openvswitch-switch At first I thought this was an openvswitch issue, however I reverted the code in get_root_helper_child_pid() and neutron-rootwrap processes started getting cleaned up. See corresponding commit for code that possibly introduced this at [1]. This can be recreated with the openstack charms using xenial-newton- staging. On newton deploys, neutron-gateway and nova-compute units will exhaust memory due to compounding ovsdb-client processes. [1] commit fd93e19f2a415b3803700fc491749daba01a4390 Author: Assaf Muller <amul...@redhat.com> Date: Fri Mar 18 16:29:26 2016 -0400 Change get_root_helper_child_pid to stop when it finds cmd get_root_helper_child_pid recursively finds the child of pid, until it can no longer find a child. However, the intention is not to find the deepest child, but to strip away root helpers. For example 'sudo neutron-rootwrap x' is supposed to find the pid of x. However, in cases 'x' spawned quick lived children of its own (For example: ip / brctl / ovs invocations), get_root_helper_child_pid returned those pids if called in the wrong time. Change-Id: I582aa5c931c8bfe57f49df6899445698270bb33e Closes-Bug: #1558819 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1629097/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp