Re: [openstack-dev] memory usage in devstack-gate (the oom-killer strikes again)

2014-09-09 Thread Chmouel Boudjnah
On Tue, Sep 9, 2014 at 12:24 AM, Joe Gordon joe.gord...@gmail.com wrote: 1) Should we explicitly set the number of workers that services use in devstack? Why have so many workers in a small all-in-one environment? What is the right balance here? This is what we do for Swift, without

Re: [openstack-dev] memory usage in devstack-gate (the oom-killer strikes again)

2014-09-09 Thread Mike Bayer
yes. guppy seems to have some nicer string formatting for this dump as well, but i was unable to figure out how to get this string format to write to a file, it seems like the tool is very geared towards interactive console use. We should pick a nice memory formatter we like, there’s a bunch

[openstack-dev] memory usage in devstack-gate (the oom-killer strikes again)

2014-09-08 Thread Joe Gordon
Hi All, We have recently started seeing assorted memory issues in the gate including the oom-killer [0] and libvirt throwing memory errors [1]. Luckily we run ps and dstat on every devstack run so we have some insight into why we are running out of memory. Based on the output from job taken at

Re: [openstack-dev] memory usage in devstack-gate (the oom-killer strikes again)

2014-09-08 Thread Mike Bayer
Hi All - Joe had me do some quick memory profiling on nova, just an FYI if anyone wants to play with this technique, I place a little bit of memory profiling code using Guppy into nova/api/__init__.py, or anywhere in your favorite app that will definitely get imported when the thing first

Re: [openstack-dev] memory usage in devstack-gate (the oom-killer strikes again)

2014-09-08 Thread Clint Byrum
Excerpts from Joe Gordon's message of 2014-09-08 15:24:29 -0700: Hi All, We have recently started seeing assorted memory issues in the gate including the oom-killer [0] and libvirt throwing memory errors [1]. Luckily we run ps and dstat on every devstack run so we have some insight into why