On Sep 28, 2011, at 08:29 , Jeff Squyres wrote: > Why not use pre-posted non-blocking receives and MPI_WAIT_ANY?
That's not very scalable either… Might work for 256 processes, but that's about it. george. > If you effectively randomize the order of the requests in the array, you can > basically control the fairness. > > > On Sep 27, 2011, at 10:05 PM, Sébastien Boisvert wrote: > >> Hello/bonjour, >> >> >> I modified your patch so that in the worse case exactly comm->num_procs are >> probed (I just changed a < for a <= in the second for loop). >> >> The behavior of your patch was to probe a maximum of comm->num_procs - 1, >> thus hanging if comm->last_probed has to be probed. >> >> >> With the patch, I get a few good time slices, but most are not productive, >> see below. >> >> >> Therefore I strongly think that you were right to say that my program should >> not use MPI_ANY_SOURCE in the first place. >> >> >> I'll stick with my round-robin reception in which many calls to MPI_Iprobe >> says "no message". >> >> >> In my round-robin reception, I guess that the opal_progress() call in >> mca_pml_ob1_iprobe (ompi/mca/pml/ob1/pml_ob1_iprobe.c) for all the calls >> to MPI_Iprobe with no match is *really* what helps draining the messages and >> presumably contributing to the fairness of the whole thing. >> >> >> >> What do you think ? >> >> >> >> >> If I run Ray in gdb: >> >> (gdb) run >> Starting program: /home/sboisver12/git-clones/ray/Ray >> [Thread debugging using libthread_db enabled] >> [New Thread 0x40a00940 (LWP 16867)] >> [New Thread 0x41401940 (LWP 16869)] >> >> >> I see two new threads that are spawned. >> >> The draining of messages is also done in a "progress thread", right ? >> >> >> >> Thank you. >> >> >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.47 s Speed= 90354 Sent= 30 >> (processMessages: 30, processData: 0) Received= 30 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.57 s Speed= 93748 Sent= 26 >> (processMessages: 26, processData: 0) Received= 26 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.67 s Speed= 92086 Sent= 66 >> (processMessages: 66, processData: 0) Received= 66 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.77 s Speed= 93501 Sent= 104 >> (processMessages: 104, processData: 0) Received= 104 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.87 s Speed= 94512 Sent= 85 >> (processMessages: 85, processData: 0) Received= 85 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.97 s Speed= 93262 Sent= 138 >> (processMessages: 138, processData: 0) Received= 138 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.07 s Speed= 94944 Sent= 125 >> (processMessages: 125, processData: 0) Received= 125 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.17 s Speed= 95377 Sent= 119 >> (processMessages: 119, processData: 0) Received= 119 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.27 s Speed= 91206 Sent= 129 >> (processMessages: 129, processData: 0) Received= 129 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.37 s Speed= 92937 Sent= 83 >> (processMessages: 83, processData: 0) Received= 83 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.47 s Speed= 90803 Sent= 142 >> (processMessages: 142, processData: 0) Received= 142 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.57 s Speed= 94807 Sent= 87 >> (processMessages: 87, processData: 0) Received= 87 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.67 s Speed= 92527 Sent= 176 >> (processMessages: 176, processData: 0) Received= 176 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.77 s Speed= 93514 Sent= 151 >> (processMessages: 151, processData: 0) Received= 151 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.87 s Speed= 94102 Sent= 81 >> (processMessages: 81, processData: 0) Received= 81 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.97 s Speed= 95072 Sent= 104 >> (processMessages: 104, processData: 0) Received= 104 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.07 s Speed= 90934 Sent= 99 >> (processMessages: 99, processData: 0) Received= 99 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.17 s Speed= 92640 Sent= 117 >> (processMessages: 117, processData: 0) Received= 117 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.27 s Speed= 93260 Sent= 84 >> (processMessages: 84, processData: 0) Received= 84 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.37 s Speed= 94774 Sent= 134 >> (processMessages: 134, processData: 0) Received= 134 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.47 s Speed= 94714 Sent= 84 >> (processMessages: 84, processData: 0) Received= 84 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.57 s Speed= 94921 Sent= 55 >> (processMessages: 55, processData: 0) Received= 55 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.67 s Speed= 94461 Sent= 48 >> (processMessages: 48, processData: 0) Received= 48 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.77 s Speed= 93824 Sent= 58 >> (processMessages: 58, processData: 0) Received= 58 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.87 s Speed= 90685 Sent= 131 >> (processMessages: 131, processData: 0) Received= 131 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.97 s Speed= 92930 Sent= 65 >> (processMessages: 65, processData: 0) Received= 65 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.07 s Speed= 93235 Sent= 38 >> (processMessages: 38, processData: 0) Received= 38 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.17 s Speed= 92487 Sent= 56 >> (processMessages: 56, processData: 0) Received= 56 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.27 s Speed= 92559 Sent= 61 >> (processMessages: 61, processData: 0) Received= 61 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.37 s Speed= 91398 Sent= 47 >> (processMessages: 47, processData: 0) Received= 47 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.47 s Speed= 94528 Sent= 72 >> (processMessages: 72, processData: 0) Received= 72 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.57 s Speed= 93548 Sent= 112 >> (processMessages: 112, processData: 0) Received= 112 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.67 s Speed= 94701 Sent= 54 >> (processMessages: 54, processData: 0) Received= 54 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.77 s Speed= 94900 Sent= 50 >> (processMessages: 50, processData: 0) Received= 50 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.87 s Speed= 92868 Sent= 43 >> (processMessages: 43, processData: 0) Received= 43 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.97 s Speed= 92996 Sent= 170 >> (processMessages: 54, processData: 116) Received= 170 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.07 s Speed= 88456 Sent= 93 >> (processMessages: 77, processData: 16) Received= 93 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.17 s Speed= 88194 Sent= 874 >> (processMessages: 83, processData: 791) Received= 874 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.27 s Speed= 84966 Sent= 439 >> (processMessages: 79, processData: 360) Received= 439 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.37 s Speed= 83100 Sent= 406 >> (processMessages: 67, processData: 339) Received= 406 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.47 s Speed= 88243 Sent= 215 >> (processMessages: 54, processData: 161) Received= 215 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.57 s Speed= 87354 Sent= 550 >> (processMessages: 201, processData: 349) Received= 550 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.67 s Speed= 94519 Sent= 55 >> (processMessages: 55, processData: 0) Received= 55 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.77 s Speed= 90221 Sent= 79 >> (processMessages: 79, processData: 0) Received= 79 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.87 s Speed= 92418 Sent= 146 >> (processMessages: 146, processData: 0) Received= 146 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.97 s Speed= 94045 Sent= 68 >> (processMessages: 68, processData: 0) Received= 68 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.07 s Speed= 94432 Sent= 64 >> (processMessages: 64, processData: 0) Received= 64 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.17 s Speed= 90937 Sent= 68 >> (processMessages: 68, processData: 0) Received= 68 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.27 s Speed= 88159 Sent= 62 >> (processMessages: 62, processData: 0) Received= 62 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.37 s Speed= 94526 Sent= 84 >> (processMessages: 84, processData: 0) Received= 84 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.47 s Speed= 93362 Sent= 73 >> (processMessages: 73, processData: 0) Received= 73 Balance= 0 >> Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.57 s Speed= 90062 Sent= 45 >> (processMessages: 45, processData: 0) Received= 45 Balance= 0 >> >> >>> ________________________________________ >>> De : devel-boun...@open-mpi.org [devel-boun...@open-mpi.org] de la part de >>> George Bosilca [bosi...@eecs.utk.edu] >>> Date d'envoi : 27 septembre 2011 16:44 >>> À : Open MPI Developers >>> Objet : Re: [OMPI devel] RE : Implementation of MPI_Iprobe >>> >>> Sebastien, >>> >>> Please try the attached patch. It is made against the trunk, so you might >>> have to adapt it slightly. >>> >>> Let me know of the outcome, so we can decide if it's worth pushing it in >>> the next releases. >>> >>> Thanks, >>> george. >>> >>> >> >> >> >> >> Sébastien<iprobe_starvation_fixed.patch>_______________________________________________ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
