Re: [OMPI devel] gdb libmpi.dylib on Leopard

[Aurélien Bouteiller]

I filed the following bug report on Apple Developer Connexion. As a  
short summary, I suggest they get in touch with us and include the -- 
whole-archive mechanism in their ld.


Aurelien

19-Sep-2008 03:08 PM Aurelien Bouteiller:
Summary:
Because the Apple ld does not include the GNU's ld --whole-archive/-- 
no-whole-archive mechanism to allow loading of all members of  
selective archives, libtool (including gnu libtool) is forced to  
unpack all the members of a convenience library (and later delete  
them), and afterwards needs to run dsymutil. Unfortunately, because  
the archives are uncompressed to a temporary space before being  
included in the final library, dsymutil seams to get confused. As a  
consequence, it is impossible to debug a library with gdb, the .o  
files never being found, even if the library actually contains all the  
necessary debug symbols.


Steps to reproduce:
1. Download a svn Open MPI trunk release (or any libtool based  
project, I've experienced the same problems when compiling my own  
gcc4.3). Please note that you need autoconf 2.62 and automake 1.10 to  
compile Open MPI trunk.

2. configure Open MPI with the debug options (configure --enable-debug)
3. make install
4. find or create a sample MPI program, mpicc it.
5. mpirun -np 1 gdb mpi_sample_program
6. break MPI_Init, r, n.

Expected results:
6: you should step each line of the MPI_Init function

Actual results:
6. you see a large number of warnings
warning: Could not find object file "/Users/bouteill/ompi/debug.build/ 
opal/.libs/libopen-pal.lax/libmca_memchecker.a/memchecker_base_open.o"  
- no debug information available for "../../../../trunk/opal/mca/ 
memchecker/base/memchecker_base_open.c".


You are unable to step in MPI_Init. Instead the execution continues up  
to reach the "main" function.


Regression:
Used to work with Tiger.

Notes:
If you need some more details or want to cooperate with us, please  
register to the Open MPI devel mailing list. As a major open source  
project we have  been working on a fix for this issue for a while, but  
where unable to correct it without modifications to apple's ld.


We believe that the best workaround would be to include the --whole- 
archive/--no-whole-archive mechanism. Then there is no need anymore to  
unpack the convenience archives before building the .dylib, and as a  
friendly side effect compilation time should improve a lot.


Thanks,
--
* Dr. Aurélien Bouteiller
* Sr. Research Associate at Innovative Computing Laboratory
* University of Tennessee
* 1122 Volunteer Boulevard, suite 350
* Knoxville, TN 37996
* 865 974 6321
(on behalf of the Open MPI development community)



Le 19 sept. 08 à 17:22, Jeff Squyres a écrit :


Thanks for following up!

Aurelien, I'll leave this to you -- I rarely do OMPI development on  
my Mac...



On Sep 19, 2008, at 5:08 PM, Ralf Wildenhues wrote:


Hello,

I asked Peter O'Gorman about this issue, and he said

| I believe that running dsymutil on the generated lib would then  
create a

| libfoo.dSYM in the .libs directory conatining all the necessary
| debugging information, which could be used for debugging the  
library in
| the build tree (gdb should find it sitting there next to the  
original
| library and use the debug information  in the .dSYM).  
Libtool-2.2.6 does

| run dsymutil and create the .dSYM though...
|
| There should be a libmpi.dylib in a .libs directory and a
| libmpi.dylib.dSYM directory next to it.

Also, he said that it could help if you reported a bug at
, under the notion that the
more people file bugs with them, the more they will understand
what problems users have with the dsymutils issues.

Cheers,
Ralf

* Aurélien Bouteiller wrote on Fri, Sep 19, 2008 at 09:44:46PM CEST:

Ok,

I didn't forgot to rerun autogen.sh (I even erased the libltdl, and
various libtool wrappers that are generated at autogen/configure  
time). I
checked the link Ralf submitted to our attention. This is exactly  
the
same problem, or at least the same symptoms. The last version of  
libtool

runs dsymutil on the created .so/.dylib, but the bad thing is that
dsymutil returns similar warning message about missing ".lax" files.
Therefore, even running it manually on the .dsym does not help.

I upgraded (compiled my own copy) my gcc to 4.3.2 (you should do  
it too,
Jeff, the experimental have been giving me headaches in the past).  
Now, I
also have the same warning messages for internal libs of gcc than  
for
open MPI. This leads me to believe this is not an Open MPI bug,  
but more

probably a libtool/ld issue.

I'll switch to linux for my devel for now, but if you have any  
success

story...

Aurelien

Le 19 sept. 08 à 15:20, Jeff Squyres a écrit :


I get the same problem on my MBP with 10.5.5.  However, I'm running
the gcc from hpc.sf.net:

-
[15:16] rtp-jsquyres-8713:~/mpi % gcc --version
gcc (GCC) 4.3.0 20071026 (experimental)
...
-

Not the /usr/bin/gcc that ships with 

Re: [OMPI devel] gdb libmpi.dylib on Leopard

[Aurélien Bouteiller]

Ok,

I didn't forgot to rerun autogen.sh (I even erased the libltdl, and  
various libtool wrappers that are generated at autogen/configure  
time). I checked the link Ralf submitted to our attention. This is  
exactly the same problem, or at least the same symptoms. The last  
version of libtool runs dsymutil on the created .so/.dylib, but the  
bad thing is that dsymutil returns similar warning message about  
missing ".lax" files. Therefore, even running it manually on the .dsym  
does not help.


I upgraded (compiled my own copy) my gcc to 4.3.2 (you should do it  
too, Jeff, the experimental have been giving me headaches in the  
past). Now, I also have the same warning messages for internal libs of  
gcc than for open MPI. This leads me to believe this is not an Open  
MPI bug, but more probably a libtool/ld issue.


I'll switch to linux for my devel for now, but if you have any success  
story...


Aurelien

Le 19 sept. 08 à 15:20, Jeff Squyres a écrit :

I get the same problem on my MBP with 10.5.5.  However, I'm running  
the gcc from hpc.sf.net:


-
[15:16] rtp-jsquyres-8713:~/mpi % gcc --version
gcc (GCC) 4.3.0 20071026 (experimental)
...
-

Not the /usr/bin/gcc that ships with Leopard.  I don't know if that  
matters or not.


I'm using AC 2.63, AM 1.10.1, LT 2.2.6a with a fairly vanilla build  
of Open MPI:


./configure --prefix=/Users/jsquyres/bogus --disable-mpi-f77 -- 
enable-mpirun-prefix-by-default


Here's what happens -- I fire up an MPI program and it deadlocks.  I  
attach to an MPI process PID with gdb (I am using /usr/bin/gdb --  
the Leopard-shipped gdb).  I get oodles of messages like Aurelien's:


-
warning: Could not find object file "/data/jsquyres/svn/ompi/ 
ompi/.libs/libmpi.lax/libdatatype.a/convertor.o" - no debug  
information available for "convertor.c".
warning: Could not find object file "/data/jsquyres/svn/ompi/ 
ompi/.libs/libmpi.lax/libdatatype.a/copy_functions.o" - no debug  
information available for "copy_functions.c".
warning: Could not find object file "/data/jsquyres/svn/ompi/ 
ompi/.libs/libmpi.lax/libdatatype.a/copy_functions_heterogeneous.o"  
- no debug information available for "copy_functions_heterogeneous.c".

-


On Sep 19, 2008, at 2:31 PM, Ralf Wildenhues wrote:


* Aurélien Bouteiller wrote on Fri, Sep 19, 2008 at 08:02:40PM CEST:
Thanks Ralf for the support. I upgraded to libtool 2.2.6 and it  
didn't
solved the problem though. Still looking for somebody to confirm  
that

its working or not working on their Mac.


Did you rerun autogen.sh?  All I know is that your report looks  
really

similar to  and that
one is apparently solved with Libtool 2.2.6.

If yours is still broken, then some more details would be nice.

Cheers,
Ralf
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel



--
Jeff Squyres
Cisco Systems


___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel

Re: [OMPI devel] gdb libmpi.dylib on Leopard

[Jeff Squyres]

I get the same problem on my MBP with 10.5.5.  However, I'm running  
the gcc from hpc.sf.net:


-
[15:16] rtp-jsquyres-8713:~/mpi % gcc --version
gcc (GCC) 4.3.0 20071026 (experimental)
...
-

Not the /usr/bin/gcc that ships with Leopard.  I don't know if that  
matters or not.


I'm using AC 2.63, AM 1.10.1, LT 2.2.6a with a fairly vanilla build of  
Open MPI:


./configure --prefix=/Users/jsquyres/bogus --disable-mpi-f77 --enable- 
mpirun-prefix-by-default


Here's what happens -- I fire up an MPI program and it deadlocks.  I  
attach to an MPI process PID with gdb (I am using /usr/bin/gdb -- the  
Leopard-shipped gdb).  I get oodles of messages like Aurelien's:


-
warning: Could not find object file "/data/jsquyres/svn/ompi/ 
ompi/.libs/libmpi.lax/libdatatype.a/convertor.o" - no debug  
information available for "convertor.c".
warning: Could not find object file "/data/jsquyres/svn/ompi/ 
ompi/.libs/libmpi.lax/libdatatype.a/copy_functions.o" - no debug  
information available for "copy_functions.c".
warning: Could not find object file "/data/jsquyres/svn/ompi/ 
ompi/.libs/libmpi.lax/libdatatype.a/copy_functions_heterogeneous.o" -  
no debug information available for "copy_functions_heterogeneous.c".

-


On Sep 19, 2008, at 2:31 PM, Ralf Wildenhues wrote:


* Aurélien Bouteiller wrote on Fri, Sep 19, 2008 at 08:02:40PM CEST:
Thanks Ralf for the support. I upgraded to libtool 2.2.6 and it  
didn't

solved the problem though. Still looking for somebody to confirm that
its working or not working on their Mac.


Did you rerun autogen.sh?  All I know is that your report looks really
similar to  and that
one is apparently solved with Libtool 2.2.6.

If yours is still broken, then some more details would be nice.

Cheers,
Ralf
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel



--
Jeff Squyres
Cisco Systems

Re: [OMPI devel] gdb libmpi.dylib on Leopard

[Ralf Wildenhues]

* Aurélien Bouteiller wrote on Fri, Sep 19, 2008 at 08:02:40PM CEST:
> Thanks Ralf for the support. I upgraded to libtool 2.2.6 and it didn't  
> solved the problem though. Still looking for somebody to confirm that  
> its working or not working on their Mac.

Did you rerun autogen.sh?  All I know is that your report looks really
similar to  and that
one is apparently solved with Libtool 2.2.6.

If yours is still broken, then some more details would be nice.

Cheers,
Ralf

Re: [OMPI devel] gdb libmpi.dylib on Leopard

[Aurélien Bouteiller]

Thanks Ralf for the support. I upgraded to libtool 2.2.6 and it didn't  
solved the problem though. Still looking for somebody to confirm that  
its working or not working on their Mac.


Aurelien

Le 17 sept. 08 à 12:39, Ralf Wildenhues a écrit :


Hello Aurélien,

* Aurélien Bouteiller wrote on Wed, Sep 17, 2008 at 06:32:11PM CEST:
I have been facing a weird problem for several month now (I guess  
since I
upgraded from Tiger to Leopard). I am unable to debug Open MPI  
using gdb
on my mac. The problem comes from gdb not being able to load  
symbols from
the dynamic libraries of Open MPI. I receive a message "warning:  
Could

not find object file "/Users/bouteill/ompi/debug.build/
opal/.libs/libopen-pal.lax/libmca_memory.a/memory_base_close.o" - no
debug information available for "../../../../trunk/opal/mca/memory/
base/memory_base_close.c".". As you can see, the path to the object  
file
containing the symbols is not correct. It links to the temporary  
files

expanded during the final stage link. As those files do not exist
anymore, gdb gets confused.


I have a vague memory that this is fixed in Libtool 2.2.6.  If you're
using an older version, please retry bootstrapping OpenMPI with that
one.

Cheers,
Ralf
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel

Re: [OMPI devel] gdb libmpi.dylib on Leopard

[Ralf Wildenhues]

Hello Aurélien,

* Aurélien Bouteiller wrote on Wed, Sep 17, 2008 at 06:32:11PM CEST:
> I have been facing a weird problem for several month now (I guess since I 
> upgraded from Tiger to Leopard). I am unable to debug Open MPI using gdb 
> on my mac. The problem comes from gdb not being able to load symbols from 
> the dynamic libraries of Open MPI. I receive a message "warning: Could 
> not find object file "/Users/bouteill/ompi/debug.build/ 
> opal/.libs/libopen-pal.lax/libmca_memory.a/memory_base_close.o" - no  
> debug information available for "../../../../trunk/opal/mca/memory/ 
> base/memory_base_close.c".". As you can see, the path to the object file 
> containing the symbols is not correct. It links to the temporary files 
> expanded during the final stage link. As those files do not exist 
> anymore, gdb gets confused.

I have a vague memory that this is fixed in Libtool 2.2.6.  If you're
using an older version, please retry bootstrapping OpenMPI with that
one.

Cheers,
Ralf

[OMPI devel] gdb libmpi.dylib on Leopard

[Aurélien Bouteiller]

I have been facing a weird problem for several month now (I guess  
since I upgraded from Tiger to Leopard). I am unable to debug Open MPI  
using gdb on my mac. The problem comes from gdb not being able to load  
symbols from the dynamic libraries of Open MPI. I receive a message  
"warning: Could not find object file "/Users/bouteill/ompi/debug.build/ 
opal/.libs/libopen-pal.lax/libmca_memory.a/memory_base_close.o" - no  
debug information available for "../../../../trunk/opal/mca/memory/ 
base/memory_base_close.c".". As you can see, the path to the object  
file containing the symbols is not correct. It links to the temporary  
files expanded during the final stage link. As those files do not  
exist anymore, gdb gets confused.


supposedly, the rpath option of libtool should take care of this and  
correct the path to the symbols. Is anybody successful at debugging  
Open MPI on Leopard ? Is this a bug of Open MPI or a bug in libtool/ 
gdb ?  Any known fix ?


Aurelien

--
* Dr. Aurélien Bouteiller
* Sr. Research Associate at Innovative Computing Laboratory
* University of Tennessee
* 1122 Volunteer Boulevard, suite 350
* Knoxville, TN 37996
* 865 974 6321