Oops, I knew I forgot something!
I am using OpenMPI 3.1.1
I have tried loading in a OpenMPI 4.0.3 module but receive a repeating error at
runtime:
[tcn560.bullx:16698] pml_ucx.c:175 Error: Failed to receive UCX worker
address: Not found (-13)
[tcn560.bullx:16698] [[42671,6],29] ORTE_ERROR_LOG: Error in file dpm/dpm.c at
line 493
Which makes me think I am in fact on an IB cluster, I will configure my own
local OpenMPI 4.0.4 with the config options you suggest and test it out.
I will also set up a run with the mca parameters suggested by Gilles. The
cluster I work on will be down for the next two days so it will be a bit before
I can provide any updates! Thank you for your patience and your help, much
appreciated.
-Sean
On 7/25/20, 4:33 AM, "users on behalf of Joseph Schuchart via users"
<users-boun...@lists.open-mpi.org on behalf of users@lists.open-mpi.org> wrote:
External.
Hi Sean,
Thanks for the report! I have a few questions/suggestions:
1) What version of Open MPI are you using?
2) What is your network? It sounds like you are on an IB cluster using
btl/openib (which is essentially discontinued). Can you try the Open MPI
4.0.4 release with UCX instead of openib (configure with --without-verbs
and --with-ucx)?
3) If that does not help, can you boil your code down to a minimum
working example? That would make it easier for people to try to
reproduce what happens.
Cheers
Joseph
On 7/24/20 11:34 PM, Lewis,Sean via users wrote:
> Hi all,
>
> I am encountering a silent hang involving MPI_Ssend and MPI_Irecv. The
> subroutine in question is called by each processor and is structured
> similar to the pseudo code below. The subroutine is successfully called
> several thousand times before the silent hang behavior manifests and
> never resolves. The hang will occur in nearly (but not exactly) the same
> spot for bit-wise identical tests. During the hang, all MPI ranks will
> be at the Line 18 Barrier except for two. One will be waiting at Line
> 17, waiting for its Irecv to complete, and the other at one of the Ssend
> Line 9 or 14. This suggests that a MPI_Irecv never completes and a
> processor is indefinitely blocked in the Ssend unable to complete the
> transfer.
>
> I’ve found similar discussion of this kind of behavior on the OpenMPI
> mailing list:
>
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mail-archive.com%2Fusers%40lists.open-mpi.org%2Fmsg19227.html&data=02%7C01%7Cscl63%40drexel.edu%7C3b0c3aea09a64654a4c108d830756470%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637312628045924572&sdata=H2yKKppFOFeJo1SrwQSi2Lts%2BtPQBjpf%2FjLVSlKrCiY%3D&reserved=0
> ultimately resolving in setting the mca parameter btl_openib_flags to
> 304 or 305 (default 310):
>
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mail-archive.com%2Fusers%40lists.open-mpi.org%2Fmsg19277.html&data=02%7C01%7Cscl63%40drexel.edu%7C3b0c3aea09a64654a4c108d830756470%7C3664e6fa47bd45a696708c4f080f8ca6%7C0%7C0%7C637312628045924572&sdata=GulI2%2F%2F%2BnQskKGz7HVSmwVG053T0v4w66TSxrWMvTEc%3D&reserved=0.
I
> have seen some promising behavior by doing the same. As the mailer
> suggests, this implies a problem with the RDMA protocols in infiniband
> for large messages.
>
> I wanted to breathe life back into this conversation as the silent hang
> issue is particularly debilitating and confusing to me.
> Increasing/decreasing the number of processors used does not seem to
> alleviate the issue, using MPI_Send results in the same behavior,
> perhaps a message has exceeded a memory limit? I am running a test now
> that reports the individual message sizes but I previously implemented a
> switch to check for buffer size discrepancies which is not triggered. In
> the meantime, has anyone run into similar issues or have thoughts as to
> remedies for this behavior?
>
> 1: call MPI_BARRIER(…)
>
> 2: do i = 1,nprocs
>
> 3: if(commatrix_recv(i) .gt. 0) then ! Identify which procs to
> receive from via predefined matrix
>
> 4: call Mpi_Irecv(…)
>
> 5: endif
>
> 6: enddo
>
> 7: do j = mype+1,nproc
>
> 8: if(commatrix_send(j) .gt. 0) then ! Identify which procs to
> send to via predefined matrix
>
> 9: MPI_Ssend(…)
>
> 10: endif
>
> 11: enddo
>
> 12: do j = 1,mype
>
> 13: if(commatrix_send(j) .gt. 0) then ! Identify which procs to
> send to via predefined matrix
>
> 14: MPI_Ssend(…)
>
> 15: endif
>
> 16: enddo
>
> 17: call MPI_Waitall(…) ! Wait for all Irecv to complete
>
> 18: call MPI_Barrier(…)
>
> Cluster information:
>
> 30 processors
>
> Managed by slurm
>
> OS: Red Hat v. 7.7
>
> Thank you for help/advice you can provide,
>
> Sean
>
> *Sean C. Lewis*
>
> Doctoral Candidate
>
> Department of Physics
>
> Drexel University
>
