I had something like this happen on a test cluster of Raspberry Pis several years ago, and in my case, I was able to isolate it to being an MPI issue unrelated to SLURM. If you can run directly on the nodes, that might be a useful distinction for you to try to make. (Running "directly" might mean manually doing "miprun -n <n>" inside an srun-dispatched shell, if you can''t bypass SLURM, which you shouldn't anyways.)
In my case, on the four-core RPi, running e.g. two two-way jobs just oversubscribed the first two cores, whether run from SLURM or directly. I found a work-around, was to use the "--map-by socket" argument to mpirun. I don't think I ever figured it out -- it was a short-lifetime test cluster that I was using to explore SLURM config options. I also don't recall which version of OpenMPI it was, but I'd guess it was the one that's packaged for Debian/Raspbian 11, which is 4.1.0. -- A. On Fri, Feb 23, 2024 at 5:06 PM Christopher Daley via users < users@lists.open-mpi.org> wrote: > Dear Support, > > I'm seeking clarification about the expected behavior of mpirun in Slurm > jobs. > > Our setup consists of using Slurm for resource allocation and OpenMPI > mpirun to launch MPI applications. We have found that when two Slurm jobs > have been allocated different cores on the same compute node that the MPI > ranks in Slurm job 1 map to the same cores as Slurm job 2. It appears that > OpenMPI mpirun is not considering the details of the Slurm allocation. We > get expected behavior when srun is employed as the MPI launcher instead of > mpirun, i.e. the MPI ranks in Slurm job 1 use different cores than the MPI > ranks in Slurm job 2. > > We have observed this with OpenMPI-4.1.6 and OpenMPI-5.0.2. Should we > expect that the mpirun in each job will only use the exact cores that were > allocated by Slurm? > > Thanks, > Chris > -- Andrew Reid / andrew.ce.r...@gmail.com
