Kris and I also experimented with:
https://gist.github.com/harlowja/d2b5fc5282c76f10bc3f (which is like the
entrypoint to cProfile but instead using the eventlet profiler module
instead), that didn't appear to go so well...
So any thoughts would be great here (especially around how to profile
with eventlet, aka, with something that understands that greenthreads
are being used and takes that into account).
We also tried https://pypi.python.org/pypi/GreenletProfiler but that
didn't seem to go so well either.
Any success stories from people?
Kris G. Lindgren wrote:
Calling all Proflers!
Iam running into an issue with CPU usageon remote nova-conductor and Iam
trying to profile it to see where its consuming themost amount of cpu,
so that we can investigate further. The etherpad where we have been
working on this issue is located at:
https://etherpad.openstack.org/p/remote-conductor-performance
I tired running nova conductor under Cprofiler with 20 workers, however
it only saw the main thread. So I started conducotr with a single worker
and cProfiler was able to see more requests.
I ran the following:
* python -m cProfile -o conductor-1worker /usr/bin/nova-conductor
* ./gprof2dot/gprof2dot.py -f pstats conductor-1worker | dot -Tsvg -o
conductor-1worker.svg
* SVG output here:
http://tempsend.com/5340867576/F283/conductor1worker.svg
The SVG showed that most of the time was spent in multiple different
locations but AMQP seemed to be the highest consume of time. However, I
am reading that cProfiler can get confused under eventlet so I am
wondering if the cProfiler can be trusted. I tired using
greenletprofiler however, that started to causes errors to be thrown in
the conductor logs around connecting to rabbitmq.
So what I am asking is those of you who have experience debugging python
with eventlet, what tooling do you use?
___________________________________________________________________
Kris Lindgren
Senior Linux Systems Engineer
GoDaddy
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
