Re: [OMPI users] WG: Bug in OpenMPI-1.8.3: storage limition in shared memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code
On Mon, Oct 27, 2014 at 02:15:45PM +, michael.rach...@dlr.de wrote: > Dear Gilles, > > This is the system response on the login node of cluster5: > > cluster5:~/dat> mpirun -np 1 df -h > Filesystem Size Used Avail Use% Mounted on > /dev/sda31 228G 5.6G 211G 3% / > udev 32G 232K 32G 1% /dev > tmpfs32G 0 32G 0% /dev/shm > /dev/sda11 291M 39M 237M 15% /boot > /dev/gpfs10 495T 280T 216T 57% /gpfs10 > /dev/loop1 3.2G 3.2G 0 100% /media > cluster5:~/dat> mpirun -np 1 df -hi > Filesystem Inodes IUsed IFree IUse% Mounted on > /dev/sda3115M 253K 15M2% / > udev0 0 0 - /dev > tmpfs7.9M 3 7.9M1% /dev/shm > /dev/sda1176K41 76K1% /boot > /dev/gpfs10 128M 67M 62M 53% /gpfs10 > /dev/loop1 0 0 0 - /media > cluster5:~/dat> > > > And this the system response on the compute node of cluster5: > > rachner@r5i5n13:~> mpirun -np 1 df -h > Filesystem Size Used Avail Use% Mounted on > tmpfs63G 1.4G 62G 3% / > udev 63G 92K 63G 1% /dev > tmpfs63G 0 63G 0% /dev/shm > tmpfs 150M 12M 139M 8% /tmp This is the problem right here. /tmp can only be used to back a total of 139M of shared memory. /dev/shm can back up to 63G so using that will solve your problem. Try setting adding -mca shmem_mmap_relocate_backing_file true to your mpirun line or add shmem_mmap_relocate_backing_file = true to your installation's /etc/openmpi-mca-params.conf -Nathan pgpOl0hwQ3Qey.pgp Description: PGP signature
Re: [OMPI users] WG: Bug in OpenMPI-1.8.3: storage limition in shared memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code
Dear Gilles, This is the system response on the login node of cluster5: cluster5:~/dat> mpirun -np 1 df -h Filesystem Size Used Avail Use% Mounted on /dev/sda31 228G 5.6G 211G 3% / udev 32G 232K 32G 1% /dev tmpfs32G 0 32G 0% /dev/shm /dev/sda11 291M 39M 237M 15% /boot /dev/gpfs10 495T 280T 216T 57% /gpfs10 /dev/loop1 3.2G 3.2G 0 100% /media cluster5:~/dat> mpirun -np 1 df -hi Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda3115M 253K 15M2% / udev0 0 0 - /dev tmpfs7.9M 3 7.9M1% /dev/shm /dev/sda1176K41 76K1% /boot /dev/gpfs10 128M 67M 62M 53% /gpfs10 /dev/loop1 0 0 0 - /media cluster5:~/dat> And this the system response on the compute node of cluster5: rachner@r5i5n13:~> mpirun -np 1 df -h Filesystem Size Used Avail Use% Mounted on tmpfs63G 1.4G 62G 3% / udev 63G 92K 63G 1% /dev tmpfs63G 0 63G 0% /dev/shm tmpfs 150M 12M 139M 8% /tmp /dev/gpfs10 495T 280T 216T 57% /gpfs10 rachner@r5i5n13:~> mpirun -np 1 df -hi Filesystem Inodes IUsed IFree IUse% Mounted on tmpfs 16M 63K 16M1% / udev0 0 0 - /dev tmpfs 16M 3 16M1% /dev/shm tmpfs 16M 183 16M1% /tmp /dev/gpfs10 128M 67M 62M 53% /gpfs10 rachner@r5i5n13:~> You wrote: "From the logs, the error message makes sense to me : there is not enough space in /tmp Since the compute nodes have a lot of memory, you might want to try using /dev/shm instead of /tmp for the backing files" I do not understand that system output. Is it required now to switch to /dev/shm ? And how can I do that? Or must our operators change something (the cluster is very new)? Greetings Michael Rachner -Ursprüngliche Nachricht- Von: users [mailto:users-boun...@open-mpi.org] Im Auftrag von Gilles Gouaillardet Gesendet: Montag, 27. Oktober 2014 14:49 An: Open MPI Users Betreff: Re: [OMPI users] WG: Bug in OpenMPI-1.8.3: storage limition in shared memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code Michael, Could you please run mpirun -np 1 df -h mpirun -np 1 df -hi on both compute and login nodes Thanks Gilles michael.rach...@dlr.de wrote: >Dear developers of OPENMPI, > >We have now installed and tested the bugfixed OPENMPI Nightly Tarball of >2014-10-24 (openmpi-dev-176-g9334abc.tar.gz) on Cluster5 . >As before (with OPENMPI-1.8.3 release version) the small Ftn-testprogram runs >correctly on the login-node. >As before the program aborts on the compute node, but now with a different >error message: > >The following message appears when launching the program with 2 processes: >mpiexec -np 2 -bind-to core -tag-output ./a.out > >[1,0]: on nodemaster: iwin= 685 : >[1,0]: total storage [MByte] alloc. in shared windows so far: >137. >[ [1,0]: === allocation of shared window no. iwin= 686 >[1,0]: starting now with idim_1= 5 >--- >-- It appears as if there is not enough space for >/tmp/openmpi-sessions-rachner@r5i5n13_0/48127/1/shared_window_688.r5i5n >13 (the shared-memory backing file). It is likely that your MPI job >will now either abort or experience performance degradation. > > Local host: r5i5n13 > Space Requested: 204256 B > Space Available: 208896 B >--- >--- [r5i5n13:26917] *** An error occurred in MPI_Win_allocate_shared >[r5i5n13:26917] *** reported by process [3154051073,140733193388032] >[r5i5n13:26917] *** on communicator MPI_COMM_WORLD [r5i5n13:26917] *** >MPI_ERR_INTERN: internal error [r5i5n13:26917] *** MPI_ERRORS_ARE_FATAL >(processes in this communicator will now abort, >[r5i5n13:26917] ***and potentially your MPI job) >rachner@r5i5n13:~/dat> >*** >* > > >When I repeat the run using 24 processes (on same compute node) the same kind >of abort message occurs, but earlier: > >[1,0]: on nodemaster: iwin= 231 : >[1,0]: total storage [MByte] alloc. in shared windows so far: >46.2 > [1,0]: === allocation of shared window no. iwin= 232 >[1,0]: starting now with idim_1= 5 >--- >-- It appears as if there is not enough space for >/tmp/openmpi-sessions-rachner@r5i5n13_0/48029/1/shared_window_234.r5i5n >13 (the shared-memory backing file). It is
Re: [OMPI users] WG: Bug in OpenMPI-1.8.3: storage limition in shared memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code
-Ursprüngliche Nachricht- Von: users [mailto:users-boun...@open-mpi.org] Im Auftrag von Gilles Gouaillardet Gesendet: Montag, 27. Oktober 2014 14:49 An: Open MPI Users Betreff: Re: [OMPI users] WG: Bug in OpenMPI-1.8.3: storage limition in shared memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code Michael, Could you please run mpirun -np 1 df -h mpirun -np 1 df -hi on both compute and login nodes Thanks Gilles michael.rach...@dlr.de wrote: >Dear developers of OPENMPI, > >We have now installed and tested the bugfixed OPENMPI Nightly Tarball of >2014-10-24 (openmpi-dev-176-g9334abc.tar.gz) on Cluster5 . >As before (with OPENMPI-1.8.3 release version) the small Ftn-testprogram runs >correctly on the login-node. >As before the program aborts on the compute node, but now with a different >error message: > >The following message appears when launching the program with 2 processes: >mpiexec -np 2 -bind-to core -tag-output ./a.out > >[1,0]: on nodemaster: iwin= 685 : >[1,0]: total storage [MByte] alloc. in shared windows so far: >137. >[ [1,0]: === allocation of shared window no. iwin= 686 >[1,0]: starting now with idim_1= 5 >--- >-- It appears as if there is not enough space for >/tmp/openmpi-sessions-rachner@r5i5n13_0/48127/1/shared_window_688.r5i5n >13 (the shared-memory backing file). It is likely that your MPI job >will now either abort or experience performance degradation. > > Local host: r5i5n13 > Space Requested: 204256 B > Space Available: 208896 B >--- >--- [r5i5n13:26917] *** An error occurred in MPI_Win_allocate_shared >[r5i5n13:26917] *** reported by process [3154051073,140733193388032] >[r5i5n13:26917] *** on communicator MPI_COMM_WORLD [r5i5n13:26917] *** >MPI_ERR_INTERN: internal error [r5i5n13:26917] *** MPI_ERRORS_ARE_FATAL >(processes in this communicator will now abort, >[r5i5n13:26917] ***and potentially your MPI job) >rachner@r5i5n13:~/dat> >*** >* > > >When I repeat the run using 24 processes (on same compute node) the same kind >of abort message occurs, but earlier: > >[1,0]: on nodemaster: iwin= 231 : >[1,0]: total storage [MByte] alloc. in shared windows so far: >46.2 > [1,0]: === allocation of shared window no. iwin= 232 >[1,0]: starting now with idim_1= 5 >--- >-- It appears as if there is not enough space for >/tmp/openmpi-sessions-rachner@r5i5n13_0/48029/1/shared_window_234.r5i5n >13 (the shared-memory backing file). It is likely that your MPI job >will now either abort or experience performance degradation. > > Local host: r5i5n13 > Space Requested: 204784 B > Space Available: 131072 B >--- >--- [r5i5n13:26947] *** An error occurred in MPI_Win_allocate_shared >[r5i5n13:26947] *** reported by process [3147628545,140733193388032] >[r5i5n13:26947] *** on communicator MPI_COMM_WORLD [r5i5n13:26947] *** >MPI_ERR_INTERN: internal error [r5i5n13:26947] *** MPI_ERRORS_ARE_FATAL >(processes in this communicator will now abort, >[r5i5n13:26947] ***and potentially your MPI job) >rachner@r5i5n13:~/dat> >*** >* > >So the problem is not yet resolved. > >Greetings > Michael Rachner > > > > > > >-Ursprüngliche Nachricht- >Von: Rachner, Michael >Gesendet: Montag, 27. Oktober 2014 11:49 >An: 'Open MPI Users' >Betreff: AW: [OMPI users] Bug in OpenMPI-1.8.3: storage limition in >shared memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code > >Dear Mr. Squyres. > >We will try to install your bug-fixed nigthly tarball of 2014-10-24 on >Cluster5 to see whether it works or not. >The installation however will take some time. I get back to you, if I know >more. > >Let me add the information that on the Laki each nodes has 16 GB of shared >memory (there it worked), the login-node on Cluster 5 has 64 GB (there it >worked too), whereas the compute nodes on Cluster5 have 128 GB (there it did >not work). >So possibly the bug might have something to do with the size of the physical >shared memory available on the node. > >Greetings >Michael Rachner > >-Ursprüngliche Nachricht- >Von: users [mailto:users-boun...@open-mpi.org] Im Auftrag von Jeff >Squyres (jsquyres) >Gesendet: Freitag, 24. Oktober 2014 22:45 >An: Open MPI User's List >Betreff: Re: [OMPI users] Bug in
Re: [OMPI users] WG: Bug in OpenMPI-1.8.3: storage limition in shared memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code
Michael, The available space must be greater than the requested size + 5% From the logs, the error message makes sense to me : there is not enough space in /tmp Since the compute nodes have a lot of memory, you might want to try using /dev/shm instead of /tmp for the backing files Cheers, Gilles michael.rach...@dlr.de wrote: >Dear developers of OPENMPI, > >We have now installed and tested the bugfixed OPENMPI Nightly Tarball of >2014-10-24 (openmpi-dev-176-g9334abc.tar.gz) on Cluster5 . >As before (with OPENMPI-1.8.3 release version) the small Ftn-testprogram runs >correctly on the login-node. >As before the program aborts on the compute node, but now with a different >error message: > >The following message appears when launching the program with 2 processes: >mpiexec -np 2 -bind-to core -tag-output ./a.out > >[1,0]: on nodemaster: iwin= 685 : >[1,0]: total storage [MByte] alloc. in shared windows so far: >137. >[ [1,0]: === allocation of shared window no. iwin= 686 >[1,0]: starting now with idim_1= 5 >- >It appears as if there is not enough space for >/tmp/openmpi-sessions-rachner@r5i5n13_0/48127/1/shared_window_688.r5i5n13 (the >shared-memory backing >file). It is likely that your MPI job will now either abort or experience >performance degradation. > > Local host: r5i5n13 > Space Requested: 204256 B > Space Available: 208896 B >-- >[r5i5n13:26917] *** An error occurred in MPI_Win_allocate_shared >[r5i5n13:26917] *** reported by process [3154051073,140733193388032] >[r5i5n13:26917] *** on communicator MPI_COMM_WORLD >[r5i5n13:26917] *** MPI_ERR_INTERN: internal error >[r5i5n13:26917] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will >now abort, >[r5i5n13:26917] ***and potentially your MPI job) >rachner@r5i5n13:~/dat> > > > >When I repeat the run using 24 processes (on same compute node) the same kind >of abort message occurs, but earlier: > >[1,0]: on nodemaster: iwin= 231 : >[1,0]: total storage [MByte] alloc. in shared windows so far: >46.2 > [1,0]: === allocation of shared window no. iwin= 232 >[1,0]: starting now with idim_1= 5 >- >It appears as if there is not enough space for >/tmp/openmpi-sessions-rachner@r5i5n13_0/48029/1/shared_window_234.r5i5n13 (the >shared-memory backing >file). It is likely that your MPI job will now either abort or experience >performance degradation. > > Local host: r5i5n13 > Space Requested: 204784 B > Space Available: 131072 B >-- >[r5i5n13:26947] *** An error occurred in MPI_Win_allocate_shared >[r5i5n13:26947] *** reported by process [3147628545,140733193388032] >[r5i5n13:26947] *** on communicator MPI_COMM_WORLD >[r5i5n13:26947] *** MPI_ERR_INTERN: internal error >[r5i5n13:26947] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will >now abort, >[r5i5n13:26947] ***and potentially your MPI job) >rachner@r5i5n13:~/dat> > > >So the problem is not yet resolved. > >Greetings > Michael Rachner > > > > > > >-Ursprüngliche Nachricht- >Von: Rachner, Michael >Gesendet: Montag, 27. Oktober 2014 11:49 >An: 'Open MPI Users' >Betreff: AW: [OMPI users] Bug in OpenMPI-1.8.3: storage limition in shared >memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code > >Dear Mr. Squyres. > >We will try to install your bug-fixed nigthly tarball of 2014-10-24 on >Cluster5 to see whether it works or not. >The installation however will take some time. I get back to you, if I know >more. > >Let me add the information that on the Laki each nodes has 16 GB of shared >memory (there it worked), the login-node on Cluster 5 has 64 GB (there it >worked too), whereas the compute nodes on Cluster5 have 128 GB (there it did >not work). >So possibly the bug might have something to do with the size of the physical >shared memory available on the node. > >Greetings >Michael Rachner > >-Ursprüngliche Nachricht- >Von: users [mailto:users-boun...@open-mpi.org] Im Auftrag von Jeff Squyres >(jsquyres) >Gesendet: Freitag, 24. Oktober 2014 22:45 >An: Open MPI User's List >Betreff: Re: [OMPI users] Bug in OpenMPI-1.8.3: storage limition in shared >memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code > >Nathan tells me that this may well be related to a
Re: [OMPI users] WG: Bug in OpenMPI-1.8.3: storage limition in shared memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code
Michael, Could you please run mpirun -np 1 df -h mpirun -np 1 df -hi on both compute and login nodes Thanks Gilles michael.rach...@dlr.de wrote: >Dear developers of OPENMPI, > >We have now installed and tested the bugfixed OPENMPI Nightly Tarball of >2014-10-24 (openmpi-dev-176-g9334abc.tar.gz) on Cluster5 . >As before (with OPENMPI-1.8.3 release version) the small Ftn-testprogram runs >correctly on the login-node. >As before the program aborts on the compute node, but now with a different >error message: > >The following message appears when launching the program with 2 processes: >mpiexec -np 2 -bind-to core -tag-output ./a.out > >[1,0]: on nodemaster: iwin= 685 : >[1,0]: total storage [MByte] alloc. in shared windows so far: >137. >[ [1,0]: === allocation of shared window no. iwin= 686 >[1,0]: starting now with idim_1= 5 >- >It appears as if there is not enough space for >/tmp/openmpi-sessions-rachner@r5i5n13_0/48127/1/shared_window_688.r5i5n13 (the >shared-memory backing >file). It is likely that your MPI job will now either abort or experience >performance degradation. > > Local host: r5i5n13 > Space Requested: 204256 B > Space Available: 208896 B >-- >[r5i5n13:26917] *** An error occurred in MPI_Win_allocate_shared >[r5i5n13:26917] *** reported by process [3154051073,140733193388032] >[r5i5n13:26917] *** on communicator MPI_COMM_WORLD >[r5i5n13:26917] *** MPI_ERR_INTERN: internal error >[r5i5n13:26917] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will >now abort, >[r5i5n13:26917] ***and potentially your MPI job) >rachner@r5i5n13:~/dat> > > > >When I repeat the run using 24 processes (on same compute node) the same kind >of abort message occurs, but earlier: > >[1,0]: on nodemaster: iwin= 231 : >[1,0]: total storage [MByte] alloc. in shared windows so far: >46.2 > [1,0]: === allocation of shared window no. iwin= 232 >[1,0]: starting now with idim_1= 5 >- >It appears as if there is not enough space for >/tmp/openmpi-sessions-rachner@r5i5n13_0/48029/1/shared_window_234.r5i5n13 (the >shared-memory backing >file). It is likely that your MPI job will now either abort or experience >performance degradation. > > Local host: r5i5n13 > Space Requested: 204784 B > Space Available: 131072 B >-- >[r5i5n13:26947] *** An error occurred in MPI_Win_allocate_shared >[r5i5n13:26947] *** reported by process [3147628545,140733193388032] >[r5i5n13:26947] *** on communicator MPI_COMM_WORLD >[r5i5n13:26947] *** MPI_ERR_INTERN: internal error >[r5i5n13:26947] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will >now abort, >[r5i5n13:26947] ***and potentially your MPI job) >rachner@r5i5n13:~/dat> > > >So the problem is not yet resolved. > >Greetings > Michael Rachner > > > > > > >-Ursprüngliche Nachricht- >Von: Rachner, Michael >Gesendet: Montag, 27. Oktober 2014 11:49 >An: 'Open MPI Users' >Betreff: AW: [OMPI users] Bug in OpenMPI-1.8.3: storage limition in shared >memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code > >Dear Mr. Squyres. > >We will try to install your bug-fixed nigthly tarball of 2014-10-24 on >Cluster5 to see whether it works or not. >The installation however will take some time. I get back to you, if I know >more. > >Let me add the information that on the Laki each nodes has 16 GB of shared >memory (there it worked), the login-node on Cluster 5 has 64 GB (there it >worked too), whereas the compute nodes on Cluster5 have 128 GB (there it did >not work). >So possibly the bug might have something to do with the size of the physical >shared memory available on the node. > >Greetings >Michael Rachner > >-Ursprüngliche Nachricht- >Von: users [mailto:users-boun...@open-mpi.org] Im Auftrag von Jeff Squyres >(jsquyres) >Gesendet: Freitag, 24. Oktober 2014 22:45 >An: Open MPI User's List >Betreff: Re: [OMPI users] Bug in OpenMPI-1.8.3: storage limition in shared >memory allocation (MPI_WIN_ALLOCATE_SHARED) in Ftn-code > >Nathan tells me that this may well be related to a fix that was literally just >pulled into the v1.8 branch today: > >https://github.com/open-mpi/ompi-release/pull/56 > >Would you mind testing any nightly tarball after tonight?
