Sorry for the delay in replying - many of us were at SC last week. Admittedly, I'm looking at your code on a PDA, so I might be missing some things. But I have 2 q's:
1 your send routine doesn't seem to protect from sending to yourself. Correct? 2 you're not using nonblocking sends, which, if I understand your code right, can lead to deadlock. Right? Eg proc A sends to proc b and blocks until b receives. But b is blocking waiting for it's send completion, etc. I think with your random destinations (which may even be yourself, in which case the blocking send will never complete because you didn't prepost a nomblocking receive) and blocking sends, you can end up with deadlock. Sent from my PDA. No type good. On Nov 16, 2010, at 5:21 PM, Sébastien Boisvert <sebastien.boisver...@ulaval.ca> wrote: > Dear awesome community, > > > Over the last months, I closely followed the evolution of bug 2043, > entitled 'sm BTL hang with GCC 4.4.x'. > > https://svn.open-mpi.org/trac/ompi/ticket/2043 > > The reason is that I am developping an MPI-based software, and I use > Open-MPI as it is the only implementation I am aware of that send > messages eagerly (powerful feature, that is). > > http://denovoassembler.sourceforge.net/ > > I believe that this very pesky bug remains in Open-MPI 1.4.3, and > enclosed to this communication are scientific proofs of my claim, or at > least I think they are ;). > > > Each byte transfer layer has its default limit to send eagerly a > message. With shared memory (sm), the value is 4096 bytes. At least it > is according to ompi_info. > > > To verify this limit, I implemented a very simple test. The source code > is test4096.cpp, which basically just send a single message of 4096 > bytes from a rank to another (rank 1 to 0). > > The test was conclusive: the limit is 4096 bytes (see > mpirun-np-2-Simple.txt). > > > > Then, I implemented a simple program (103 lines) that makes Open-MPI > 1.4.3 hang. The code is in make-it-hang.cpp. At each iteration, each > rank send a message to a randomly-selected destination. A rank polls for > new messages with MPI_Iprobe. Each rank prints the current time at each > second during 30 seconds. Using this simple code, I ran 4 test cases, > each with a different outcome (use the Makefile if you want to reproduce > the bug). > > Before I describe these cases, I will describe the testing hardware. > > I use a computer with 32 x86_64 cores (see cat-proc-cpuinfo.txt.gz). > The computer has 128 GB of physical memory (see > cat-proc-meminfo.txt.gz). > It runs Fedora Core 11 with Linux 2.6.30.10-105.2.23.fc11.x86_64 (see > dmesg.txt.gz & uname.txt). > Default kernel parameters are utilized at runtime (see > sudo-sysctl-a.txt.gz). > > The C++ compiler is g++ (GCC) 4.4.1 20090725 (Red Hat 4.4.1-2) (see g > ++--version.txt). > > > I compiled Open-MPI 1.4.3 myself (see config.out.gz, make.out.gz, > make-install.out.gz). > Finally, I use Open-MPI 1.4.3 with defaults (see ompi_info.txt.gz). > > > > > Now I can describe the cases. > > > Case 1: 30 MPI ranks, message size is 4096 bytes > > File: mpirun-np-30-Program-4096.txt > Outcome: It hangs -- I killed the poor thing after 30 seconds or so. > > > > > Case 2: 30 MPI ranks, message size is 1 byte > > File: mpirun-np-30-Program-1.txt.gz > Outcome: It runs just fine. > > > > > Case 3: 2 MPI ranks, message size is 4096 bytes > > File: mpirun-np-2-Program-4096.txt > Outcome: It hangs -- I killed the poor thing after 30 seconds or so. > > > > > Case 4: 30 MPI ranks, message size if 4096 bytes, shared memory is > disabled > > File: mpirun-mca-btl-^sm-np-30-Program-4096.txt.gz > Outcome: It runs just fine. > > > > > > A backtrace of the processes in Case 1 is in gdb-bt.txt.gz. > > > > > Thank you ! > > <test4096.cpp> > <mpirun-np-2-Simple.txt> > <make-it-hang.cpp> > <Makefile> > <cat-proc-cpuinfo.txt.gz> > <cat-proc-meminfo.txt.gz> > <dmesg.txt.gz> > <uname.txt> > <sudo-sysctl-a.txt.gz> > <g++--version.txt> > <config.out.gz> > <make.out.gz> > <make-install.out.gz> > <ompi_info.txt.gz> > <mpirun-np-30-Program-4096.txt> > <mpirun-np-30-Program-1.txt.gz> > <mpirun-np-2-Program-4096.txt> > <mpirun-mca-btl-^sm-np-30-Program-4096.txt.gz> > <gdb-bt.txt.gz> > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
