Re: [O-MPI devel] Broken --enable-mpi-threads
On Tue, Aug 02, 2005 at 02:40:21PM -0500, Brian Barrett wrote:
> The tree now compiles with the --enable-mpi-threads problem. There
> is a bug in the event library that will cause deadlocks in orterun,
> so the tree isn't exactly useful right now. Tim Woodall is going to
> look into the problem.
ok - thanks!
A new problem arised after compiling and running my first test-program.
It simply spawns a separate thread on each rank and sends/receives 1
byte (MPI_BYTE) messages in this thread. There seems to be a race
condition, sometimes, all messages are received correctly, sometimes all
messages fail and the receiving rank eats up a lot of memory (>600MB)
and segfaults.
The backtrace is:
#0 0x0015d828 in ompi_convertor_unpack (pConv=0x83569e0, iov=0x479e798,
out_size=0x479e7bc, max_data=0x479e7b8, freeAfter=0x479e7b4)
at convertor.c:104
#1 0x00f76af4 in mca_ptl_tcp_recv_frag_progress (frag=0x8356980)
at ptl_tcp_recvfrag.h:166
#2 0x00f76124 in mca_ptl_tcp_matched (ptl=0x83321a8, frag=0x8356980)
at ptl_tcp.c:302
#3 0x0090d314 in mca_pml_teg_recv_frag_match (ptl=0x8320948, frag=0x8356980,
header=0x8356ab4) at pml_teg_recvfrag.c:82
#4 0x00f7bbdc in mca_ptl_tcp_recv_frag_handler (frag=0x8356a94, sd=12)
at ptl_tcp_recvfrag.c:107
#5 0x00f7a20f in mca_ptl_tcp_peer_recv_handler (sd=12, flags=2,
user=0x836b628) at ptl_tcp_peer.c:606
#6 0x002a8ff8 in opal_event_process_active () at event.c:453
#7 0x002a92e3 in opal_event_loop (flags=2) at event.c:543
#8 0x002b733b in opal_progress () at opal_progress.c:211
#9 0x00909295 in opal_condition_wait (c=0x23bc80, m=0x23bce0)
at condition.h:66
#10 0x00908a93 in mca_pml_teg_recv (addr=0x479ea94, count=1,
datatype=0x804a4a8, src=-1, tag=12, comm=0x804a5f0, status=0x8380108)
at pml_teg_irecv.c:100
#11 0x001bc50f in PMPI_Recv (buf=0x479ea94, count=1, type=0x804a4a8,
source=-1, tag=12, comm=0x804a5f0, status=0x8380108) at precv.c:66
#12 0x08048f66 in MPI_Barrier_start_worker_thread (param=0x83809f0)
at nbbarr.c:76
#13 0x0072fdec in pthread_create@@GLIBC_2.1 () from /lib/tls/libpthread.so.0
#14 0x0082519a in iswctype_l () from /lib/tls/libc.so.6
The zipped corefile can be found at:
http://gustav.informatik.tu-chemnitz.de/~htor/sec/core.23839.gz
Any Idea or should I try to debug it?
Thanks,
Torsten
--
bash$ :(){ :|:&};: - pgp: http://www.unixer.de/htor-key.asc -
An optimist believes we live in the best of all possible worlds.
A pessimist is sure of it!
[O-MPI devel] Fwd: Regarding MVAPI Component in Open MPI
From: "Sridhar Chirravuri \(schirrav\)" Date: August 4, 2005 2:07:04 AM EDT Cc: "Sridhar Chirravuri \(schirrav\)" Subject: Regarding MVAPI Component in OpenMPI All, I got OpenMPI tar ball and could configure and build on AMD x86_64 arch. In our case, we need to enable MVAPI and disable OpenIB. For this, I have moved .ompi_ignore file from mvapi directory to openib directory. I could see that OpenIB was disabled as the entire openib tree was skipped by the autogen.sh script. While running Pallas accross the nodes, I could see that data is passing over Gigbit ethernet and NOT over Infiniband. Does anyone has idea about why data is going through Gig and NOT over infiniband? Do I have to set any configuration options? Do I have to give any run-time options? I have tried with mpirun -mca btl mvapi but of no use. I could make out that TCP component is being used and in order to disable tcp, I have copied .ompi_ignore in to directories /ompi/orte/mca/oob/tcp and /ompi/ompi/mca/ptl/tcp. But this time program fails with segmentation fault error. These are the configure options that I have given while configuring OpenMPI. ./configure --prefix=/openmpi --with-btl-mvapi=/usr/local/topspin/ --with-btl-mvapi-libdir=/usr/local/topspin --with-mvapi Please help me in resolving this. Thanks -Sridhar
Re: [O-MPI devel] Fwd: Regarding MVAPI Component in Open MPI
On Aug 4, 2005, at 6:43 AM, Jeff Squyres wrote:
I got OpenMPI tar ball and could configure and build on AMD x86_64
arch.
Excellent. Note, however, that it's probably better to get a
Subversion checkout. As this is the current head of our development
tree, it's a constantly moving target -- having a Subversion checkout
will help you keep up with our progress.
In our case, we need to enable MVAPI and disable OpenIB. For this, I
have moved .ompi_ignore file from mvapi directory to openib directory.
I could see that OpenIB was disabled as the entire openib tree was
skipped by the autogen.sh script.
It depends on what version of the tarball you got -- in the version
that I have, the mvapi components (both btl and mpool) do not have
.ompi_ignore files (we recently removed them -- July 27th, r6613).
Additionally, you should not need to run autogen.sh in a tarball (in
fact, autogen.sh should warn you if you try to do this). autogen.sh is
only required in a Subversion checkout. Please see the top-level
HACKING file in a Subversion checkout (I don't think that it is
included in the tarball).
Finally, note that you'll need to give additional --with options to
configure to tell it where the MVAPI libraries and header files are
located -- more on this below.
While running Pallas accross the nodes, I could see that data is
passing over Gigbit ethernet and NOT over Infiniband. Does anyone has
idea about why data is going through Gig and NOT over infiniband? Do I
have to set any configuration options? Do I have to give any run-time
options? I have tried with mpirun -mca btl mvapi but of no use.
What is the output of the ompi_info command? This will tell you if the
mvapi component is compiled and installed (it sounds like it is not).
I could make out that TCP component is being used and in order to
disable tcp, I have copied .ompi_ignore in to directories
/ompi/orte/mca/oob/tcp and /ompi/ompi/mca/ptl/tcp. But this time
program fails with segmentation fault error.
Right now, IIRC, we don't have checks to ensure that there are valid
paths from one MPI process to another -- which is probably the seg
fault.
Also note that .ompi_ignore is an autogen mechanism. It is really
intended for developers who want to protect parts of the tree during
development when it is not ready for general use. It is not really
intended
These are the configure options that I have given while configuring
OpenMPI.
./configure --prefix=/openmpi --with-btl-mvapi=/usr/local/topspin/
--with-btl-mvapi-libdir=/usr/local/topspin --with-mvapi
Almost correct. Check out ./configure --help:
--with-btl-mvapi=MVAPI_DIR
Additional directory to search for MVAPI
installation
--with-btl-mvapi-libdir=IBLIBDIR
directory where the IB library can be found,
if it
is not in MVAPI_DIR/lib or MVAPI_DIR/lib64
The --with-btl-mvapi-libdir flag is only necessary if the MVAPI library
cannot be found the /usr/local/topspin/lib or /usr/local/topspin/lib64.
There is no --with-mvapi flag.
So it's quite possible that with the wrong value for
--with-btl-mvapi-libdir, it failed to compile the mvapi component
(i.e., I suspect it was looking for /usr/local/topspin/libmosal.* when
libmosal is most likely in /usr/local/topspin/lib or
/usr/local/topspin/lib64), which resulted in Open MPI falling back to
TCP/GigE.
After you install Open MPI, you can run the ompi_info command and it
will show a list of all the installed components. You should see the
mvapi component in both the btl and mpool frameworks if all went well.
If it didn't, then send us the output (stdout and stderr) of configure,
the top-level config.log file, and the output from "make all" (please
compress!) and we can have a look to see what went wrong.
Once you have the mvapi components built, you can choose to use them at
run-time via switches to mpirun. See the slides that we talked through
on the teleconference -- I provided some examples (you can set these
via command line arguments, environment variables, or files).
For one thing, you need to manually specify to use the 3rd generation
p2p stuff in Open MPI -- our 2nd generation is still currently the
default (that will likely change in the near future, but it hasn't been
done yet). For example:
mpirun --mca pml ob1 --mca btl mvapi,self -np 4 a.out
This tells the pml to use the "ob1" component (i.e., the 3rd generation
p2p stuff) and to use the mvapi and self btl components (self is
loopback -- one processing sending to itself).
Give that a whirl and let us know how it goes.
--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/
Re: [O-MPI devel] broken rmgr?
I can confirm this -- running a simple MPI "hello world" on one node
with the rsh pls, the MPI processes finish and exit, but then orterun
hangs:
(gdb) bt
#0 0xb7e8ef88 in poll () from /lib/libc.so.6
#1 0xb7f4f8a5 in poll_dispatch (arg=0xb7f6f080, tv=0xbfffe4f8) at
poll.c:196
#2 0xb7f4d72b in opal_event_loop (flags=1) at event.c:515
#3 0xb7f5ac6e in opal_progress () at opal_progress.c:211
#4 0xb7d6fca1 in opal_condition_wait (c=0xb7d7242c, m=0xb7d72418)
at condition.h:72
#5 0xb7d6f7f0 in orte_pls_rsh_finalize () at pls_rsh_module.c:833
#6 0xb7fb3ab6 in orte_pls_base_finalize () at pls_base_close.c:40
#7 0xb7d9092f in orte_rmgr_urm_finalize () at rmgr_urm.c:336
#8 0xb7fc14f7 in orte_rmgr_base_close () at rmgr_base_close.c:33
#9 0xb7fd3563 in orte_system_finalize () at orte_system_finalize.c:61
#10 0xb7fceca5 in orte_finalize () at orte_finalize.c:36
#11 0x0804a0d9 in main (argc=4, argv=0xbfffe6d4) at orterun.c:390
Am investigating...
On Aug 3, 2005, at 10:55 PM, Ralph H. Castain wrote:
Hmmm...it was running for me last night and (I thought) this morning,
but I'll test it again and see if I can reproduce the problem. Could
be something crept in there.
At 06:28 PM 8/3/2005, you wrote:
I just noticed that mpirun hangs forever inside the
orte_rmgr.finalize() routine. AFAIK this is new to today, and
confirmed
on PPC64, x86-64, x86-32.
Don't have the immediate time, at the moment, to dig deeper, but
thought I would throw that out there.
Cheers,
Josh
Josh Hursey
jjhur...@open-mpi.org
http://www.open-mpi.org/
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/
Re: [O-MPI devel] broken rmgr?
I think this is fixed now. There was a bug in the poll ops that was
uncovered yesterday when I fixed a deadlock issue in the event library.
Brian
On Aug 4, 2005, at 7:19 AM, Jeff Squyres wrote:
I can confirm this -- running a simple MPI "hello world" on one node
with the rsh pls, the MPI processes finish and exit, but then orterun
hangs:
(gdb) bt
#0 0xb7e8ef88 in poll () from /lib/libc.so.6
#1 0xb7f4f8a5 in poll_dispatch (arg=0xb7f6f080, tv=0xbfffe4f8) at
poll.c:196
#2 0xb7f4d72b in opal_event_loop (flags=1) at event.c:515
#3 0xb7f5ac6e in opal_progress () at opal_progress.c:211
#4 0xb7d6fca1 in opal_condition_wait (c=0xb7d7242c, m=0xb7d72418)
at condition.h:72
#5 0xb7d6f7f0 in orte_pls_rsh_finalize () at pls_rsh_module.c:833
#6 0xb7fb3ab6 in orte_pls_base_finalize () at pls_base_close.c:40
#7 0xb7d9092f in orte_rmgr_urm_finalize () at rmgr_urm.c:336
#8 0xb7fc14f7 in orte_rmgr_base_close () at rmgr_base_close.c:33
#9 0xb7fd3563 in orte_system_finalize () at orte_system_finalize.c:61
#10 0xb7fceca5 in orte_finalize () at orte_finalize.c:36
#11 0x0804a0d9 in main (argc=4, argv=0xbfffe6d4) at orterun.c:390
Am investigating...
On Aug 3, 2005, at 10:55 PM, Ralph H. Castain wrote:
Hmmm...it was running for me last night and (I thought) this morning,
but I'll test it again and see if I can reproduce the problem. Could
be something crept in there.
At 06:28 PM 8/3/2005, you wrote:
I just noticed that mpirun hangs forever inside the
orte_rmgr.finalize() routine. AFAIK this is new to today, and
confirmed
on PPC64, x86-64, x86-32.
Don't have the immediate time, at the moment, to dig deeper, but
thought I would throw that out there.
Cheers,
Josh
Josh Hursey
jjhur...@open-mpi.org
http://www.open-mpi.org/
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
--
Brian Barrett
Open MPI developer
http://www.open-mpi.org/
[O-MPI devel] rsh pls changes
In r6731, I added the first of three modifications to the rsh pls:
1. If your remote shell is sh (i.e., plain Bourne shell -- *not* bash),
the rsh pls will first invoke "! [ -e ./.profile ] || . ./.profile"
before invoking the orted (because "sh" does not invoke *any* shell
startup files on the remote node for non-interactive rsh/ssh logins --
see the LAM FAQ for details:
http://www.lam-mpi.org/faq/category4.php3#question8). This behavior
has a long-standing precedent in LAM/MPI (about a decade or so).
This gives the user the chance to setup PATH / LD_LIBRARY_PATH in their
.profile.
Two more changes will be coming soon:
2. Right now, the rsh pls assumes that your shell on the remote node is
the same as your shell on the local node. So we look at the shell on
the local node to determine if we need to source .profile on the remote
node. An enhancement to this is to allow the rsh pls to probe the
shell on the remote node to accurately determine what it is (i.e.,
execute "ssh node echo $SHELL" and look at the output). This also has
a long-standing precedent in LAM/MPI.
However, I think we can make the behavior described in #1 be the
default (assume the remote shell is the same as the local shell) --
this is certainly the common case. We can provide an MCA param for
users who need the "probe" behavior (which we'll probably need it for
multi-cell scenarios, for example).
3. Rainer and I started on a new command line flag to orterun last
week: --prefix . This allows the user to specify the installation
directory of Open MPI on the remote node, allowing us to set PATH and
LD_LIBRARY_PATH on the remote node as necessary. Hence, users don't
need to modify their shell startup files (.bashrc, .profile, .cshrc,
etc.) to specify the path to the Open MPI installation on the remote
side.
These last two features will take a bit of time to implement -- they're
a little complicated. #3 is actually more pressing than #2, and will
probably be implemented first.
--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/
Re: [O-MPI devel] os x build
Greg --
Did this get resolved? Sorry -- this slipped by me in inbox...
On Aug 1, 2005, at 5:56 PM, Greg Watson wrote:
Further this email. It appears that libtool now uses the shell
variable _S_ as a line separator instead of the ~ character.
Unfortunately _S_ never gets set on OS X, hence breaking libtool.
Greg
On Jul 31, 2005, at 9:10 PM, Greg Watson wrote:
Anyone having this problem with the latest build:
Making all in libltdl
make all-am
make[3]: Nothing to be done for `all-am'.
Making all in asm
/bin/sh ../../libtool --mode=link --tag=CC gcc -g -Wall -Wundef -Wno-
long-long -Wsign-compare -Wmissing-prototypes -Wstrict-prototypes -
Wcomment -pedantic -Wno-long-double -Werror-implicit-function-
declaration -fno-strict-aliasing -export-dynamic -o libasm.la
asm.lo atomic-asm.lo -lm
rm -fr .libs/libasm.a
ar cru .libs/libasm.a asm.o atomic-asm.o~ranlib .libs/libasm.a
ar: atomic-asm.o~ranlib: No such file or directory
make[2]: *** [libasm.la] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1
Something seems to have broken the libtool script.
Greg
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/
Re: [O-MPI devel] os x build
I just checked out a new version and it seems ok. I don't know what
the problem was.
Thanks,
Greg
On Aug 4, 2005, at 1:07 PM, Jeff Squyres wrote:
Greg --
Did this get resolved? Sorry -- this slipped by me in inbox...
On Aug 1, 2005, at 5:56 PM, Greg Watson wrote:
Further this email. It appears that libtool now uses the shell
variable _S_ as a line separator instead of the ~ character.
Unfortunately _S_ never gets set on OS X, hence breaking libtool.
Greg
On Jul 31, 2005, at 9:10 PM, Greg Watson wrote:
Anyone having this problem with the latest build:
Making all in libltdl
make all-am
make[3]: Nothing to be done for `all-am'.
Making all in asm
/bin/sh ../../libtool --mode=link --tag=CC gcc -g -Wall -Wundef -
Wno-
long-long -Wsign-compare -Wmissing-prototypes -Wstrict-prototypes -
Wcomment -pedantic -Wno-long-double -Werror-implicit-function-
declaration -fno-strict-aliasing -export-dynamic -o libasm.la
asm.lo atomic-asm.lo -lm
rm -fr .libs/libasm.a
ar cru .libs/libasm.a asm.o atomic-asm.o~ranlib .libs/libasm.a
ar: atomic-asm.o~ranlib: No such file or directory
make[2]: *** [libasm.la] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1
Something seems to have broken the libtool script.
Greg
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
