On Jan 7, 2020, at 11:48 AM, Pablo Goloboff via users <users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>> wrote:
Hi, I am trying to port an existing PVM-based application (developed over many years) onto MPI, trying to preserve as much of the organization as possible. This is my first contact with MPI, after years of working with PVM. In my PVM implementation, after doing sends (e.g. pvm_send, pvm_mcast) the master continues running (doing its own work), regardless of whether workers have actually downloaded the corresponding messages. I need to keep this model. When I do a send with MPI_Bcast, master can't do any more work until workers have downloaded the messages (i.e. there is no return from MPI_Bcast). Likewise with MPI_Send. In MPI_Bcast, since all processes in the communicator *must* participate in the operation, implementations are allowed to synchronize but are not required to do so. E.g., if the message is short enough, MPI_Bcast may send its data to its peers eagerly and return before any peers have received the payload yet. If the message isn't "short enough", then MPI_Bcast may block while the implementation pipelines or otherwise coordinates with the peers to transfer the message (which may have a side-effect of synchronizing / blocking the root until some/all of the processes have invoked MPI_Bcast). I think you probably figured this out, but I figured I'd just state this explicitly so that we're on the same sheet of music. When I do a send with MPI_Ibcast, master can continue doing work, as expected, BUT workers never receive their messages. I tried MPI_Ibcast on workers, which never succeeds (i.e. as written to last argument, MPI_Request *). I also tried MPI_Bcast on the workers to match the MPI_Ibcast from the parent, but this blocks and never returns. I have tried with MPI_Probe on the workers, but this doesn't ever work, either --message never arrives. Doing a repeated check (with MPI_Test) on master of whether workers downloaded message makes them receive it, but if I do this, master can't continue doing any work until workers got their messages! If you MPI_Ibcast on the root process, you must also MPI_Ibcast on the non-root processes. The MPI implementation may only have "weak" progress on actually performing the non-blocking broadcast behind the scenes. You can repeatedly call MPI_Test on the request to "prod" the MPI implementation to keep making progress in the background. When MPI_Test returns flag==true, then the operation on that request has completed. Technically, this shouldn't be necessary, but it's the reality of several different MPI implementations: call MPI_Test() to get background progress. And don't forget that you *must* call MPI_Test (and have it return flag==True) or MPI_Wait to complete a pending request. Meaning: if you start a non-blocking operation (like MPI_Isend, MPI_Irecv, MPI_Ibcast, ...etc.), you *must* eventually complete it via a call to one of the flavors of MPI_Test or MPI_Wait. -- Jeff Squyres jsquy...@cisco.com<mailto:jsquy...@cisco.com>
