Re: [OMPI devel] VampirTrace and MPI_Init_thread()

[Matthias Jurenz]

On Saturday 14 August 2010 00:10:49 Lisandro Dalcin wrote:
> On 13 August 2010 05:22, Matthias Jurenz  
wrote:
> > On Wednesday 11 August 2010 23:16:50 Lisandro Dalcin wrote:
> >> On 11 August 2010 03:12, Matthias Jurenz 
> >
> > wrote:
> >> > Hello Lisandro,
> >> >
> >> > this problem will be fixed in the next Open MPI release. There was an
> >> > obsolete preprocessor condition around the MPI_Init_thread wrapper, so
> >> > the source code could never be compiled :-(
> >> >
> >> > Thanks for the hint.
> >> >
> >> > Matthias
> >>
> >> OK. Many thanks for you clarification.
> >>
> >> BTW, I have and additional issue. I'm trying to build as shared
> >> library from libvt*.a using by passing -whole-archive to the linker.
> >> The idea behind this is to use that library with LD_PRELOAD to get MPI
> >> tracing of a binary compiled with plain mpicc (i.e, not mpicc-vt). For
> >> example, I managed to get this trick working with MPE. Moreover, I can
> >> enable MPI profiling at runtime in a Python script using mpi4pt by
> >> dlopen'ing the shared lib with profiling symbols before loading the
> >> mpi4py.MPI Python extension module. Being able to profile without a
> >> recompile is nice ;-)
> >>
> >> However, see this:
> >>
> >> $ pwd
> >> /usr/local/openmpi/1.4.2/lib
> >>
> >> $ ll libvt*
> >> -rw-r--r--. 1 root root 410784 2010-05-05 20:40 libvt.a
> >> -rw-r--r--. 1 root root 197618 2010-05-05 20:40 libvt.fmpi.a
> >> -rw-r--r--. 1 root root 569128 2010-05-05 20:40 libvt.mpi.a
> >> -rw-r--r--. 1 root root 503514 2010-05-05 20:40 libvt.omp.a
> >> -rw-r--r--. 1 root root 661466 2010-05-05 20:40 libvt.ompi.a
> >>
> >> $ nm libvt* | grep pomp_rd_table
> >>  U pomp_rd_table
> >>  U pomp_rd_table
> >>  U pomp_rd_table
> >>  U pomp_rd_table
> >>  U pomp_rd_table
> >>  U pomp_rd_table
> >>  U pomp_rd_table
> >>  U pomp_rd_table
> >>  U pomp_rd_table
> >>  U pomp_rd_table
> >>
> >> That symbol (and possibly others) are undefined and I cannot found
> >> them elsewhere. Is there any easy way to build a shared lib with the
> >> MPI_xxx symbols?
> >
> > Actually, the symbols above will be defined at compile/link time of the
> > application by the OpenMP instrumentor "OPARI".
> > However, while your application doesn't use OpenMP it should work to
> > define the missing symbols in a separate source file (see attachment)
> > when building the shared library:
> >
> > gcc -fPIC -I/vampirtrace -shared missing_syms.c -o
> > libvt.mpi.so -Wl,--whole-archive /libvt.mpi.a  > libdir>/libotf.a -Wl,--no-whole-archive -ldl -lz -L -lmpi
> 
> OK. Many thanks for the hint.
> 
> I was able to build a shared lib, dlopen() it at runtime and get MPI
> traces from Python scripts without need of recompiles with mpicc-vt.
> Sweet!
> 
> > FYI, the next Open MPI 1.5 will come with a newer VampirTrace which
> > provides shared libraries by default.
> 
> Nice! ... Perhaps Open MPI mpiexec's could gain a -vt flag to enable
> traces at runtime (should be easy to implement with LD_PRELOAD,
> right?)...

We plan to provide the script "vtpreload.sh" in future releases of VampirTrace 
which can be prepended to the application executable, like:

mpirun -np 2 vtpreload.sh 

In addition to preloading the VT library to get the MPI functions, the 
script will be capable to instrument almost each other function by using 
binary instrumentation. Furthermore, it will do some nm/ldd magic to determine 
the suitable VT library to preload.

An additional '-vt' flag for mpirun sounds good. This flags could also receive 
some options for the vtpreload script.

> 
> BTW, I understand Open MPI 1.5 VT will have the MPI_Init_thread()
> issue fixed. Any chance for v1.4 series?
> 

Yes, it will be fixed in Open MPI 1.4.3, too.

Matthias


smime.p7s
Description: S/MIME cryptographic signature

Re: [OMPI devel] VampirTrace and MPI_Init_thread()

[Lisandro Dalcin]

On 13 August 2010 05:22, Matthias Jurenz  wrote:
> On Wednesday 11 August 2010 23:16:50 Lisandro Dalcin wrote:
>> On 11 August 2010 03:12, Matthias Jurenz 
> wrote:
>> > Hello Lisandro,
>> >
>> > this problem will be fixed in the next Open MPI release. There was an
>> > obsolete preprocessor condition around the MPI_Init_thread wrapper, so
>> > the source code could never be compiled :-(
>> >
>> > Thanks for the hint.
>> >
>> > Matthias
>>
>> OK. Many thanks for you clarification.
>>
>> BTW, I have and additional issue. I'm trying to build as shared
>> library from libvt*.a using by passing -whole-archive to the linker.
>> The idea behind this is to use that library with LD_PRELOAD to get MPI
>> tracing of a binary compiled with plain mpicc (i.e, not mpicc-vt). For
>> example, I managed to get this trick working with MPE. Moreover, I can
>> enable MPI profiling at runtime in a Python script using mpi4pt by
>> dlopen'ing the shared lib with profiling symbols before loading the
>> mpi4py.MPI Python extension module. Being able to profile without a
>> recompile is nice ;-)
>>
>> However, see this:
>>
>> $ pwd
>> /usr/local/openmpi/1.4.2/lib
>>
>> $ ll libvt*
>> -rw-r--r--. 1 root root 410784 2010-05-05 20:40 libvt.a
>> -rw-r--r--. 1 root root 197618 2010-05-05 20:40 libvt.fmpi.a
>> -rw-r--r--. 1 root root 569128 2010-05-05 20:40 libvt.mpi.a
>> -rw-r--r--. 1 root root 503514 2010-05-05 20:40 libvt.omp.a
>> -rw-r--r--. 1 root root 661466 2010-05-05 20:40 libvt.ompi.a
>>
>> $ nm libvt* | grep pomp_rd_table
>>          U pomp_rd_table
>>          U pomp_rd_table
>>          U pomp_rd_table
>>          U pomp_rd_table
>>          U pomp_rd_table
>>          U pomp_rd_table
>>          U pomp_rd_table
>>          U pomp_rd_table
>>          U pomp_rd_table
>>          U pomp_rd_table
>>
>> That symbol (and possibly others) are undefined and I cannot found
>> them elsewhere. Is there any easy way to build a shared lib with the
>> MPI_xxx symbols?
>>
>
> Actually, the symbols above will be defined at compile/link time of the
> application by the OpenMP instrumentor "OPARI".
> However, while your application doesn't use OpenMP it should work to define 
> the
> missing symbols in a separate source file (see attachment) when building the
> shared library:
>
> gcc -fPIC -I/vampirtrace -shared missing_syms.c -o
> libvt.mpi.so -Wl,--whole-archive /libvt.mpi.a  libdir>/libotf.a -Wl,--no-whole-archive -ldl -lz -L -lmpi
>

OK. Many thanks for the hint.

I was able to build a shared lib, dlopen() it at runtime and get MPI
traces from Python scripts without need of recompiles with mpicc-vt.
Sweet!

> FYI, the next Open MPI 1.5 will come with a newer VampirTrace which provides
> shared libraries by default.
>

Nice! ... Perhaps Open MPI mpiexec's could gain a -vt flag to enable
traces at runtime (should be easy to implement with LD_PRELOAD,
right?)...

BTW, I understand Open MPI 1.5 VT will have the MPI_Init_thread()
issue fixed. Any chance for v1.4 series?


-- 
Lisandro Dalcin
---
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

Re: [OMPI devel] VampirTrace and MPI_Init_thread()

[Matthias Jurenz]

Hello Lisandro,

this problem will be fixed in the next Open MPI release. There was an obsolete 
preprocessor condition around the MPI_Init_thread wrapper, so the source code 
could never be compiled :-(

Thanks for the hint.

Matthias

On Wednesday 11 August 2010 05:34:33 Lisandro Dalcin wrote:
> On 10 August 2010 22:59, George Bosilca  wrote:
> > Lisandro,
> >
> > Thanks for the report. I quickly checked the Open MPI source code and the
> > .so library and both show the existence of both MPI_Init_thread and
> > PMPI_Init_thread symbols.
> >
> > 00031b60 T _MPI_Init_thread
> > 0005e7c0 T _PMPI_Init_thread
> >
> > I CC'ed the VT folks.
> 
> OK. Now I more confident that the problem is in VT:
> 
> nm /usr/local/openmpi/1.4.2/lib/libvt.mpi.a | grep MPI_Init
> 00ab T MPI_Init
>  U PMPI_Init
> 
> I would expect a "  T MPI_Init_thread" line to appear, but it is
> not the case.
> 
> Many thanks,
> 


smime.p7s
Description: S/MIME cryptographic signature

Re: [OMPI devel] VampirTrace and MPI_Init_thread()

[George Bosilca]

Lisandro,

Thanks for the report. I quickly checked the Open MPI source code and the .so 
library and both show the existence of both MPI_Init_thread and 
PMPI_Init_thread symbols.

00031b60 T _MPI_Init_thread
0005e7c0 T _PMPI_Init_thread

I CC'ed the VT folks.

  Thanks,
george.


On Aug 10, 2010, at 20:59 , Lisandro Dalcin wrote:

> Below you have C program that will MPI_Init_thread()
> 
> $ cat demo/helloworld.c
> #include 
> #include 
> 
> int main(int argc, char *argv[])
> {
>  int provided;
>  int size, rank, len;
>  char name[MPI_MAX_PROCESSOR_NAME];
> 
>  MPI_Init_thread(, , MPI_THREAD_MULTIPLE, );
> 
>  MPI_Comm_size(MPI_COMM_WORLD, );
>  MPI_Comm_rank(MPI_COMM_WORLD, );
>  MPI_Get_processor_name(name, );
> 
>  printf("Hello, World! I am process %d of %d on %s.\n", rank, size, name);
> 
>  MPI_Finalize();
>  return 0;
> }
> 
> 
> Now I build like this:
> 
> $ mpicc-vt demo/helloworld.c
> 
> and then try to run it:
> 
> $ ./a.out
> Hello, World! I am process 0 of 1 on trantor.
> [trantor:18854] *** An error occurred in MPI_Group_free
> [trantor:18854] *** on communicator MPI_COMM_WORLD
> [trantor:18854] *** MPI_ERR_GROUP: invalid group
> [trantor:18854] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
> 
> However, if MPI_Init() is used, it succeeds.
> 
> It seems the MPI_Init_thread() wrapper to PMPI_Init_thread() is
> missing, see this:
> 
> $ nm a.out | grep MPI_Init
> 0805c4ef T MPI_Init
> U MPI_Init_thread
> U PMPI_Init
> 
> 
> PS: Sorry if this is actually a VT bug. I'm not a VT user, I'm just
> reporting this issue (related to a mpi4py bug report that arrived at
> my inbox months ago).
> 
> -- 
> Lisandro Dalcin
> ---
> CIMEC (INTEC/CONICET-UNL)
> Predio CONICET-Santa Fe
> Colectora RN 168 Km 472, Paraje El Pozo
> Tel: +54-342-4511594 (ext 1011)
> Tel/Fax: +54-342-4511169
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

[OMPI devel] VampirTrace and MPI_Init_thread()

[Lisandro Dalcin]

Below you have C program that will MPI_Init_thread()

$ cat demo/helloworld.c
#include 
#include 

int main(int argc, char *argv[])
{
  int provided;
  int size, rank, len;
  char name[MPI_MAX_PROCESSOR_NAME];

  MPI_Init_thread(, , MPI_THREAD_MULTIPLE, );

  MPI_Comm_size(MPI_COMM_WORLD, );
  MPI_Comm_rank(MPI_COMM_WORLD, );
  MPI_Get_processor_name(name, );

  printf("Hello, World! I am process %d of %d on %s.\n", rank, size, name);

  MPI_Finalize();
  return 0;
}


Now I build like this:

$ mpicc-vt demo/helloworld.c

and then try to run it:

$ ./a.out
Hello, World! I am process 0 of 1 on trantor.
[trantor:18854] *** An error occurred in MPI_Group_free
[trantor:18854] *** on communicator MPI_COMM_WORLD
[trantor:18854] *** MPI_ERR_GROUP: invalid group
[trantor:18854] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)

However, if MPI_Init() is used, it succeeds.

It seems the MPI_Init_thread() wrapper to PMPI_Init_thread() is
missing, see this:

$ nm a.out | grep MPI_Init
0805c4ef T MPI_Init
 U MPI_Init_thread
 U PMPI_Init


PS: Sorry if this is actually a VT bug. I'm not a VT user, I'm just
reporting this issue (related to a mpi4py bug report that arrived at
my inbox months ago).

-- 
Lisandro Dalcin
---
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169