[OMPI users] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file util/show_help.c at line 501; error in device init Mesh created.

2023-05-19 Thread Rob Kudyba via users
RHEL 8 with OpenMPI 4.1.5a1 on a HPC cluster compute node Singularity version 3.7.1. I see the error in another issue mentioned at the Git page an on SO

Re: [OMPI users] Open MPI 4.0.3 outside as well as inside a SimpleFOAM container: step creation temporarily disabled, retrying Requested nodes are busy

2023-03-01 Thread Rob Kudyba via users
> > Do you invoke mpirun from **inside** the container? > > IIRC, mpirun is generally invoked from **outside** the container, could > you try this if not already the case? > > > The error message is from SLURM, so this is really a SLURM vs > singularity issue. > > What if you > > srun -N 2 -n 2 hos

[OMPI users] Open MPI 4.0.3 outside as well as inside a SimpleFOAM container: step creation temporarily disabled, retrying Requested nodes are busy

2023-02-28 Thread Rob Kudyba via users
Singularity 3.5.3 on RHEL 7 cluster w/ OpenMPI 4.0.3 lives inside a SimpleFOAM version 10 container. I've confirmed the OpenMPI versions are the same. Perhaps this is a question for Singularity users as well but how can I troubleshoot why mpirun just returns step creation temporarily disabled, retr

[OMPI users] --mca parameter explainer; mpirun WARNING: There was an error initializing an OpenFabrics device

2022-09-22 Thread Rob Kudyba via users
We're using OpenMPI 4.1.1, CUDA aware on RHEL 8 cluster that we load as a module with Infiniband controller Mellanox Technologies MT28908 Family ConnectX-6, we see this warning runnig mpirun without any MCA options/parameters: WARNING: There was an error initializing an OpenFabrics device. Local