Aniruddha Marathe wrote:
I am trying to profile (get the call graph/call sequence of) Open MPI
communication routines using GNU Profiler (gprof) since the
communication calls are implemented using macros and it's harder to
trace them statically. In order to do that I compiled the OpenMPI
source code with following options supplied to 'configure' tool:
./configure CFLAGS=-pg CPPFLAGS=-pg --enable-debug
--prefix=/home/amarathe/mpi/svn_openmpi/install
When I recompiled my test MPI application that does MPI_Send and
MPI_Recv with the new library, it generated gmon.out file as expected
(I ran it as 'mpirun -np 2 send_recv'). However, running 'gprof' on
this file didn't provide any information such as the call graphs for
MPI_Send or MPI_Recv. Following is the only function call that I see
in the output:
$ gprof send_recv gmon.out
...
...
% cumulative self self total
time seconds seconds calls Ts/call Ts/call name
0.00 0.00 0.00 25 0.00 0.00 data_start
...
...
I would like to know if anyone has done something similar with gprof
or any other open source tool with OpenMPI code.
(I found a similar, fairly recent post on the mailing list, but it
seems to talk about profiling the MPI application itself and not the
OpenMPI library routines -
http://www.open-mpi.org/community/lists/users/2009/04/8999.php)
Open source tool or free download? That is, do you really need to be
able to see the tool's source code, or are you just interested in
avoiding license fees? In any case, since that post you mention, a FAQ
has appeared on performance tools. Check
http://www.open-mpi.org/faq/?category=perftools
You make an important distinction between profiling MPI applications
versus profiling the library itself, and many tools will help just with
applications. But I've used Sun Studio for profiling Open MPI.
Ideally, you should ./configure with -g among the compilation switches
so that you get symbolic information about the library, but that isn't
necessary. The use of macros and dynamically loaded objects makes
correlating profiles with source code hard, but it works. When you
bring the Analyzer up, I think you also have to unhide the symbols
within the MPI library, which as I remember are hidden by default.
Anyhow, it works and I've learned a lot doing things this way.
