Le mardi 23 novembre 2010 à 17:38 -0500, George Bosilca a écrit : > The eager size reported by ompi_info includes the Open MPI internal headers. > They are anywhere between 20 and 64 bytes long (potentially more for some > particular networks), so what Eugene suggested was a safe boundary.
I see. > > Moreover, eager send can improve performance if and only if the matching > receives are already posted on the peer. If not, the data will become > unexpected, and there will be one additional memcpy. So it won't improve performance in my application (Ray, http://denovoassembler.sf.net) because I use MPI_Iprobe to check for incoming messages, which means any receive (MPI_Recv) is never posted before any send (MPI_Isend). Thanks, this thread is very informative for me ! > > george. > > On Nov 23, 2010, at 17:29 , Sébastien Boisvert wrote: > > > Le mardi 23 novembre 2010 à 16:07 -0500, Eugene Loh a écrit : > >> Sébastien Boisvert wrote: > >> > >>> Now I can describe the cases. > >>> > >>> > >> The test cases can all be explained by the test requiring eager messages > >> (something that test4096.cpp does not require). > >> > >>> Case 1: 30 MPI ranks, message size is 4096 bytes > >>> > >>> File: mpirun-np-30-Program-4096.txt > >>> Outcome: It hangs -- I killed the poor thing after 30 seconds or so. > >>> > >>> > >> 4096 is rendezvous. For eager, try 4000 or lower. > > > > According to ompi_info, the threshold is 4096, not 4000, right ? > > > > (Open-MPI 1.4.3) > > [sboisver12@colosse1 ~]$ ompi_info -a|less > > MCA btl: parameter "btl_sm_eager_limit" (current value: > > "4096", data source: default value) > > Maximum size (in bytes) of "short" messages > > (must be >= 1). > > > > > > "btl_sm_eager_limit: Below this size, messages are sent "eagerly" -- > > that is, a sender attempts to write its entire message to shared buffers > > without waiting for a receiver to be ready. Above this size, a sender > > will only write the first part of a message, then wait for the receiver > > to acknowledge its ready before continuing. Eager sends can improve > > performance by decoupling senders from receivers." > > > > > > > > source: > > http://www.open-mpi.org/faq/?category=sm#more-sm > > > > > > It should say "Below this size or equal to this size" instead of "Below > > this size" as ompi_info says. ;) > > > > > > > > > > As Mr. George Bosilca put it: > > > > "__should__ is not correct, __might__ is a better verb to describe the > > most "common" behavior for small messages. The problem comes from the > > fact that in each communicator the FIFO ordering is required by the MPI > > standard. As soon as there is any congestion, MPI_Send will block even > > for small messages (and this independent on the underlying network) > > until all he pending packets have been delivered." > > > > source: > > http://www.open-mpi.org/community/lists/devel/2010/11/8696.php > > > > > > > >> > >>> Case 2: 30 MPI ranks, message size is 1 byte > >>> > >>> File: mpirun-np-30-Program-1.txt.gz > >>> Outcome: It runs just fine. > >>> > >>> > >> 1 byte is eager. > > > > I agree. > > > >> > >>> Case 3: 2 MPI ranks, message size is 4096 bytes > >>> > >>> File: mpirun-np-2-Program-4096.txt > >>> Outcome: It hangs -- I killed the poor thing after 30 seconds or so. > >>> > >>> > >> Same as Case 1. > >> > >>> Case 4: 30 MPI ranks, message size if 4096 bytes, shared memory is > >>> disabled > >>> > >>> File: mpirun-mca-btl-^sm-np-30-Program-4096.txt.gz > >>> Outcome: It runs just fine. > >>> > >>> > >> Eager limit for TCP is 65536 (perhaps less some overhead). So, these > >> messages are eager. > > > > I agree. > > > >> > >> > >> _______________________________________________ > >> devel mailing list > >> de...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > > > > > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel -- M. Sébastien Boisvert Étudiant au doctorat en physiologie-endocrinologie à l'Université Laval Boursier des Instituts de recherche en santé du Canada Équipe du Professeur Jacques Corbeil Centre de recherche en infectiologie de l'Université Laval Local R-61B 2705, boulevard Laurier Québec, Québec Canada G1V 4G2 Téléphone: 418 525 4444 46342 Courriel: s...@boisvert.info Web: http://boisvert.info "Innovation comes only from an assault on the unknown" -Sydney Brenner
