[Yahoo-eng-team] [Bug 1396677] Re: Heavy use of metering labels/rules cause memory leak in neutron server
[Expired for neutron because there has been no activity for 60 days.] ** Changed in: neutron Status: Incomplete => Expired -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1396677 Title: Heavy use of metering labels/rules cause memory leak in neutron server Status in Ceilometer: Invalid Status in neutron: Expired Bug description: We found that large amount of metering labels and rules cause memory leak in neutron server. This problem is multiplied by amount of workers (10 workers - 10x memory leak). In our case we have 657 metering-lables and 122399 metering-label- rules, If anyone query them, neutron-server (worker) picked request eats +400Mb of memory and keep it until restart. If more requests send, they come to different workers cause each of them to bloat up. Same problem happens if neutron-plugin-metering-agent running (it send requests to neutron-server with same effect). If neutron-server hit 100% CPU it starts to consume even more memory (in our case up to 1.4Gb per neutron-server worker). To manage notifications about this bug go to: https://bugs.launchpad.net/ceilometer/+bug/1396677/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1396677] Re: Heavy use of metering labels/rules cause memory leak in neutron server
** Changed in: ceilometer Status: Incomplete = Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1396677 Title: Heavy use of metering labels/rules cause memory leak in neutron server Status in Ceilometer: Invalid Status in neutron: New Bug description: We found that large amount of metering labels and rules cause memory leak in neutron server. This problem is multiplied by amount of workers (10 workers - 10x memory leak). In our case we have 657 metering-lables and 122399 metering-label- rules, If anyone query them, neutron-server (worker) picked request eats +400Mb of memory and keep it until restart. If more requests send, they come to different workers cause each of them to bloat up. Same problem happens if neutron-plugin-metering-agent running (it send requests to neutron-server with same effect). If neutron-server hit 100% CPU it starts to consume even more memory (in our case up to 1.4Gb per neutron-server worker). To manage notifications about this bug go to: https://bugs.launchpad.net/ceilometer/+bug/1396677/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1396677] Re: Heavy use of metering labels/rules cause memory leak in neutron server
Even if this isn't a memory leak and not something easily fixable, the behavior should at least be documented somewhere. This may be relevent: http://effbot.org/pyfaq/why-doesnt-python-release- the-memory-when-i-delete-a-large-object.htm ** Also affects: ceilometer Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1396677 Title: Heavy use of metering labels/rules cause memory leak in neutron server Status in OpenStack Telemetry (Ceilometer): New Status in OpenStack Neutron (virtual network service): New Bug description: We found that large amount of metering labels and rules cause memory leak in neutron server. This problem is multiplied by amount of workers (10 workers - 10x memory leak). In our case we have 657 metering-lables and 122399 metering-label- rules, If anyone query them, neutron-server (worker) picked request eats +400Mb of memory and keep it until restart. If more requests send, they come to different workers cause each of them to bloat up. Same problem happens if neutron-plugin-metering-agent running (it send requests to neutron-server with same effect). If neutron-server hit 100% CPU it starts to consume even more memory (in our case up to 1.4Gb per neutron-server worker). To manage notifications about this bug go to: https://bugs.launchpad.net/ceilometer/+bug/1396677/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
