Excerpts from Joe Gordon's message of 2014-09-08 15:24:29 -0700: > Hi All, > > We have recently started seeing assorted memory issues in the gate > including the oom-killer [0] and libvirt throwing memory errors [1]. > Luckily we run ps and dstat on every devstack run so we have some insight > into why we are running out of memory. Based on the output from job taken > at random [2][3] a typical run consists of: > > * 68 openstack api processes alone > * the following services are running 8 processes (number of CPUs on test > nodes) > * nova-api (we actually run 24 of these, 8 compute, 8 EC2, 8 metadata) > * nova-conductor > * cinder-api > * glance-api > * trove-api > * glance-registry > * trove-conductor > * together nova-api, nova-conductor, cinder-api alone take over 45 %MEM > (note: some of that is memory usage is counted multiple times as RSS > includes shared libraries) > * based on dstat numbers, it looks like we don't use that much memory > before tempest runs, and after tempest runs we use a lot of memory. > > Based on this information I have two categories of questions: > > 1) Should we explicitly set the number of workers that services use in > devstack? Why have so many workers in a small all-in-one environment? What > is the right balance here?
I'm kind of wondering why we aren't pushing everything to go the same direction keystone did with apache. I may be crazy but apache gives us all kinds of tools to tune around process forking that we'll have to reinvent in our own daemon bits (like MaxRequestsPerChild to prevent leaky or slow GC from eating all our memory over time). Meanwhile, the idea on running api processes with ncpu is that we don't want to block an API request if there is a CPU available to it. Of course if we have enough cinder, nova, keystone, trove, etc. requests all at one time that we do need to block, we defer to the CPU scheduler of the box to do it, rather than queue things up at the event level. This can lead to quite ugly CPU starvation issues, and that is a lot easier to tune for if you have one tuning knob for apache + mod_wsgi instead of nservices. In production systems I'd hope that memory would be quite a bit more available than on the bazillions of cloud instances that run tests. So, while process-per-cpu-per-service is a large percentage of 8G, it is a very small percentage of 24G+, which is a pretty normal amount of memory to have on an all-in-one type of server that one might choose as a baremetal controller. For VMs that are handling production loads, It's a pretty easy trade-off to give them a little more RAM so they can take advantage of all the CPU's as needed. All this to say, since devstack is always expected to be run in a dev context, and not production, I think it would make sense to dial it back to 4 from ncpu. > > 2) Should we be worried that some OpenStack services such as nova-api, > nova-conductor and cinder-api take up so much memory? Does there memory > usage keep growing over time, does anyone have any numbers to answer this? > Why do these processes take up so much memory? Yes I do think we should be worried that they grow quite a bit. I've experienced this problem a few times in a few scripting languages, and almost every time it turned out to be too much data being read from the database or MQ. Moving to tighter messages, and tighter database interaction, nearly always results in less wasted RAM. I like the other suggestion to start graphing this. Since we have all that dstat data, I wonder if we can just process that directly into graphite. _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev