Le mardi 23 novembre 2010 à 15:17 -0500, Jeff Squyres (jsquyres) a écrit : > Sorry for the delay in replying - many of us were at SC last week.
Nothing to be sorry for ! > > Admittedly, I'm looking at your code on a PDA, so I might be missing some > things. But I have 2 q's: You got it all right, I assure you ! > > 1 your send routine doesn't seem to protect from sending to yourself. Correct? > Correct. (my error !) The code is not compliant with MPI 2.2 -- I realized that afterward. see http://www.open-mpi.org/community/lists/devel/2010/11/8689.php Also, Mr. George Bosilca pointed that out too. see http://www.open-mpi.org/community/lists/devel/2010/11/8696.php > 2 you're not using nonblocking sends, which, if I understand your code right, > can lead to deadlock. Right? Eg proc A sends to proc b and blocks until b > receives. But b is blocking waiting for it's send completion, etc. Right. As Mr. George Bosilca underlined, since the same test case works for small messages, the problem is about congestion of the FIFOs which leads to resource locking, and as you wrote, deadlock. http://www.open-mpi.org/community/lists/devel/2010/11/8696.php I now use MPI_Isend, so the problem is no more. > > I think with your random destinations (which may even be yourself, in which > case the blocking send will never complete because you didn't prepost a > nomblocking receive) and blocking sends, you can end up with deadlock. Yes, you are right ! > > Sent from my PDA. No type good. Sent from my Ubuntu. Typing is good. ;) > > On Nov 16, 2010, at 5:21 PM, Sébastien Boisvert > <sebastien.boisver...@ulaval.ca> wrote: > > > Dear awesome community, > > > > > > Over the last months, I closely followed the evolution of bug 2043, > > entitled 'sm BTL hang with GCC 4.4.x'. > > > > https://svn.open-mpi.org/trac/ompi/ticket/2043 > > > > The reason is that I am developping an MPI-based software, and I use > > Open-MPI as it is the only implementation I am aware of that send > > messages eagerly (powerful feature, that is). > > > > http://denovoassembler.sourceforge.net/ > > > > I believe that this very pesky bug remains in Open-MPI 1.4.3, and > > enclosed to this communication are scientific proofs of my claim, or at > > least I think they are ;). > > > > > > Each byte transfer layer has its default limit to send eagerly a > > message. With shared memory (sm), the value is 4096 bytes. At least it > > is according to ompi_info. > > > > > > To verify this limit, I implemented a very simple test. The source code > > is test4096.cpp, which basically just send a single message of 4096 > > bytes from a rank to another (rank 1 to 0). > > > > The test was conclusive: the limit is 4096 bytes (see > > mpirun-np-2-Simple.txt). > > > > > > > > Then, I implemented a simple program (103 lines) that makes Open-MPI > > 1.4.3 hang. The code is in make-it-hang.cpp. At each iteration, each > > rank send a message to a randomly-selected destination. A rank polls for > > new messages with MPI_Iprobe. Each rank prints the current time at each > > second during 30 seconds. Using this simple code, I ran 4 test cases, > > each with a different outcome (use the Makefile if you want to reproduce > > the bug). > > > > Before I describe these cases, I will describe the testing hardware. > > > > I use a computer with 32 x86_64 cores (see cat-proc-cpuinfo.txt.gz). > > The computer has 128 GB of physical memory (see > > cat-proc-meminfo.txt.gz). > > It runs Fedora Core 11 with Linux 2.6.30.10-105.2.23.fc11.x86_64 (see > > dmesg.txt.gz & uname.txt). > > Default kernel parameters are utilized at runtime (see > > sudo-sysctl-a.txt.gz). > > > > The C++ compiler is g++ (GCC) 4.4.1 20090725 (Red Hat 4.4.1-2) (see g > > ++--version.txt). > > > > > > I compiled Open-MPI 1.4.3 myself (see config.out.gz, make.out.gz, > > make-install.out.gz). > > Finally, I use Open-MPI 1.4.3 with defaults (see ompi_info.txt.gz). > > > > > > > > > > Now I can describe the cases. > > > > > > Case 1: 30 MPI ranks, message size is 4096 bytes > > > > File: mpirun-np-30-Program-4096.txt > > Outcome: It hangs -- I killed the poor thing after 30 seconds or so. > > > > > > > > > > Case 2: 30 MPI ranks, message size is 1 byte > > > > File: mpirun-np-30-Program-1.txt.gz > > Outcome: It runs just fine. > > > > > > > > > > Case 3: 2 MPI ranks, message size is 4096 bytes > > > > File: mpirun-np-2-Program-4096.txt > > Outcome: It hangs -- I killed the poor thing after 30 seconds or so. > > > > > > > > > > Case 4: 30 MPI ranks, message size if 4096 bytes, shared memory is > > disabled > > > > File: mpirun-mca-btl-^sm-np-30-Program-4096.txt.gz > > Outcome: It runs just fine. > > > > > > > > > > > > A backtrace of the processes in Case 1 is in gdb-bt.txt.gz. > > > > > > > > > > Thank you ! > > > > <test4096.cpp> > > <mpirun-np-2-Simple.txt> > > <make-it-hang.cpp> > > <Makefile> > > <cat-proc-cpuinfo.txt.gz> > > <cat-proc-meminfo.txt.gz> > > <dmesg.txt.gz> > > <uname.txt> > > <sudo-sysctl-a.txt.gz> > > <g++--version.txt> > > <config.out.gz> > > <make.out.gz> > > <make-install.out.gz> > > <ompi_info.txt.gz> > > <mpirun-np-30-Program-4096.txt> > > <mpirun-np-30-Program-1.txt.gz> > > <mpirun-np-2-Program-4096.txt> > > <mpirun-mca-btl-^sm-np-30-Program-4096.txt.gz> > > <gdb-bt.txt.gz> > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel -- M. Sébastien Boisvert Étudiant au doctorat en physiologie-endocrinologie à l'Université Laval Boursier des Instituts de recherche en santé du Canada Équipe du Professeur Jacques Corbeil Centre de recherche en infectiologie de l'Université Laval Local R-61B 2705, boulevard Laurier Québec, Québec Canada G1V 4G2 Téléphone: 418 525 4444 46342 Courriel: s...@boisvert.info Web: http://boisvert.info "Innovation comes only from an assault on the unknown" -Sydney Brenner
