Re: [OMPI users] New ib locked pages behavior?
On Oct 22, 2014, at 3:37 AM, r...@q-leap.de wrote: > I've commented in detail on this (non-)issue on 2014-08-20: > > http://www.open-mpi.org/community/lists/users/2014/08/25090.php > > A change in the FAQ and a fix in the code would really be nice > at this stage. Thanks for the reminder; I've pinged some folks to update the FAQ. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI users] New ib locked pages behavior?
> "Bill" == Bill Broadley writes: Hi Bill, >> I've commented in detail on this (non-)issue on 2014-08-20: >> http://www.open-mpi.org/community/lists/users/2014/08/25090.php Bill> I read that. It seems pretty clear what the problem is, but Bill> not so clear on what a user experiencing this problem should Bill> do about it. sorry for not to have made that clear enough. Bill> So for people who are using ubuntu 14.04 and openmpi-1.6.5 and Bill> 64 GB nodes. Bill> Should they: Bill> * bump log_mtts_per_seg from 3 to 4 (64GB) or 5 (128GB)? No. Bill> * ignore the error message because it doesn't apply? Fortunately yes :) Bill> * ditch ubuntu's packagedopenmpi 1.6.5 and all the packages Bill> that depends on it and install something newer than Bill> 1.8.2rc4? Not necessary. -- Roland --- http://www.q-leap.com / http://qlustar.com --- HPC / Storage / Cloud Linux Cluster OS ---
Re: [OMPI users] New ib locked pages behavior?
On 10/22/2014 12:37 AM, r...@q-leap.de wrote: >> "Bill" == Bill Broadley writes: > > It seems the half-life period of knowledge on the list has decayed to > two weeks on the list :) > > I've commented in detail on this (non-)issue on 2014-08-20: > > http://www.open-mpi.org/community/lists/users/2014/08/25090.php I read that. It seems pretty clear what the problem is, but not so clear on what a user experiencing this problem should do about it. So for people who are using ubuntu 14.04 and openmpi-1.6.5 and 64 GB nodes. Should they: * bump log_mtts_per_seg from 3 to 4 (64GB) or 5 (128GB)? * ignore the error message because it doesn't apply? * ditch ubuntu's packagedopenmpi 1.6.5 and all the packages that depends on it and install something newer than 1.8.2rc4? I also found: http://www.open-mpi.org/community/lists/users/2013/02/21430.php It was similarly vague as to if it was a real problem and exactly what the fix is.
Re: [OMPI users] New ib locked pages behavior?
> "Bill" == Bill Broadley writes: It seems the half-life period of knowledge on the list has decayed to two weeks on the list :) I've commented in detail on this (non-)issue on 2014-08-20: http://www.open-mpi.org/community/lists/users/2014/08/25090.php A change in the FAQ and a fix in the code would really be nice at this stage. Roland --- http://www.q-leap.com / http://qlustar.com --- HPC / Storage / Cloud Linux Cluster OS --- Bill> I've setup several clusters over the years with OpenMPI. I Bill> often get the below error: Bill>WARNING: It appears that your OpenFabrics subsystem is Bill>configured to only allow registering part of your physical Bill>memory. This can cause MPI jobs to run with erratic Bill>performance, hang, and/or crash. ... Bill>http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages Bill> Local host: c2-31 Registerable memory: 32768 MiB Total Bill> memory: 64398 MiB Bill> I'm well aware of the normal fixes, and have implemented them Bill> in puppet to ensure compute nodes get the changes. To be Bill> paranoid I've implemented all the changes, and they all worked Bill> under ubuntu 13.10. Bill> However with ubuntu 14.04 it seems like it's not working, thus Bill> the above message. Bill> As recommended by the faq's I've implemented: Bill> 1) ulimit -l unlimited in /etc/profile.d/slurm.sh Bill> 2) PropagateResourceLimitsExcept=MEMLOCK in slurm.conf Bill> 3) UsePAM=1 in slurm.conf Bill> 4) in /etc/security/limits.conf Bill>* hard memlock unlimited Bill>* soft memlock unlimited Bill>* hard stack unlimited Bill>* soft stack unlimited Bill> My changes seem to be working, of I submit this to slurm: Bill> #!/bin/bash -l Bill> ulimit -l hostname mpirun bash -c ulimit -l mpirun ./relay 1 Bill> 131072 Bill> I get: Bill>unlimited c2-31 unlimited unlimited unlimited unlimited Bill> Bill> Bill> Is there some new kernel parameter, ofed parameter, or similar Bill> that controls locked pages now? The kernel is 3.13.0-36 and Bill> the libopenmpi-dev package is 1.6.5. Bill> Since the ulimit -l is getting to both the slurm launched Bill> script and also to the mpirun launched binaries I'm pretty Bill> puzzled. Bill> Any suggestions? Bill> ___ users mailing Bill> list us...@open-mpi.org Subscription: Bill> http://www.open-mpi.org/mailman/listinfo.cgi/users Link to Bill> this post: Bill> http://www.open-mpi.org/community/lists/users/2014/10/25544.php
Re: [OMPI users] New ib locked pages behavior?
On 10/21/2014 05:38 PM, Gus Correa wrote: > Hi Bill > > I have 2.6.X CentOS stock kernel. Heh, wow, quite a blast from the past. > I set both parameters. > It works. Yes, for kernels that old I had it working fine. > Maybe the parameter names may changed in 3.X kernels? > (Which is really bad ...) > You could check if there is more information in: > /sys/module/mlx4_core/parameters/ $ ls /sys/module/mlx4_core/parameters/ debug_level log_mtts_per_segmsi_xuse_prio enable_64b_cqe_eqe log_num_mac num_vfs enable_qos log_num_mgm_entry_size port_type_array internal_err_reset log_num_vlanprobe_vf $ As expected there's a log_mtts_per_seg, but no log_num_mtt or num_mtt. > There seems to be a thread on the list about this (but apparently > no solution): > http://www.open-mpi.org/community/lists/users/2013/02/21430.php > > Maybe Mellanox has more information about this? I'm all ears. No idea what was behind the change to eliminate what sound like fairly important parameters in mlx4_core.
Re: [OMPI users] New ib locked pages behavior?
Hi Bill I have 2.6.X CentOS stock kernel. I set both parameters. It works. Maybe the parameter names may changed in 3.X kernels? (Which is really bad ...) You could check if there is more information in: /sys/module/mlx4_core/parameters/ There seems to be a thread on the list about this (but apparently no solution): http://www.open-mpi.org/community/lists/users/2013/02/21430.php Maybe Mellanox has more information about this? Gus Correa On 10/21/2014 08:15 PM, Bill Broadley wrote: On 10/21/2014 04:18 PM, Gus Correa wrote: Hi Bill Maybe you're missing these settings in /etc/modprobe.d/mlx4_core.conf ? http://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem Ah, that helped. Although: /lib/modules/3.13.0-36-generic/kernel/drivers/net/ethernet/mellanox/mlx4$ modinfo mlx4_core | grep "^parm" Lists some promising looking parameters: parm: log_mtts_per_seg:Log2 number of MTT entries per segment (1-7) (int) The FAQ recommends log_num_mtt or num_mtt and NOT log_mtts_per_seg, sadly: $ modinfo mlx4_core | grep "^parm" | grep mtt parm: log_mtts_per_seg:Log2 number of MTT entries per segment (1-7) (int) $ Looks like the best I can do is bump log_mtts_per_seg. I tried: $ cat /etc/modprobe.d/mlx4_core.conf options mlx4_core log_num_mtt=24 $ But: [6.691959] mlx4_core: unknown parameter 'log_num_mtt' ignored I ended up with: options mlx4_core log_mtts_per_seg=2 I'm hoping that doubles the registerable memory, although I did see a recommendation to raise it to double the system ram (in this case 64GB ram/128GB locakable. Maybe an update to the FAQ is needed?
Re: [OMPI users] New ib locked pages behavior?
On 10/21/2014 04:18 PM, Gus Correa wrote: > Hi Bill > > Maybe you're missing these settings in /etc/modprobe.d/mlx4_core.conf ? > > http://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem Ah, that helped. Although: /lib/modules/3.13.0-36-generic/kernel/drivers/net/ethernet/mellanox/mlx4$ modinfo mlx4_core | grep "^parm" Lists some promising looking parameters: parm: log_mtts_per_seg:Log2 number of MTT entries per segment (1-7) (int) The FAQ recommends log_num_mtt or num_mtt and NOT log_mtts_per_seg, sadly: $ modinfo mlx4_core | grep "^parm" | grep mtt parm: log_mtts_per_seg:Log2 number of MTT entries per segment (1-7) (int) $ Looks like the best I can do is bump log_mtts_per_seg. I tried: $ cat /etc/modprobe.d/mlx4_core.conf options mlx4_core log_num_mtt=24 $ But: [6.691959] mlx4_core: unknown parameter 'log_num_mtt' ignored I ended up with: options mlx4_core log_mtts_per_seg=2 I'm hoping that doubles the registerable memory, although I did see a recommendation to raise it to double the system ram (in this case 64GB ram/128GB locakable. Maybe an update to the FAQ is needed?
Re: [OMPI users] New ib locked pages behavior?
Hi Bill Maybe you're missing these settings in /etc/modprobe.d/mlx4_core.conf ? http://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem I hope this helps, Gus Correa On 10/21/2014 06:36 PM, Bill Broadley wrote: I've setup several clusters over the years with OpenMPI. I often get the below error: WARNING: It appears that your OpenFabrics subsystem is configured to only allow registering part of your physical memory. This can cause MPI jobs to run with erratic performance, hang, and/or crash. ... http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages Local host: c2-31 Registerable memory: 32768 MiB Total memory:64398 MiB I'm well aware of the normal fixes, and have implemented them in puppet to ensure compute nodes get the changes. To be paranoid I've implemented all the changes, and they all worked under ubuntu 13.10. However with ubuntu 14.04 it seems like it's not working, thus the above message. As recommended by the faq's I've implemented: 1) ulimit -l unlimited in /etc/profile.d/slurm.sh 2) PropagateResourceLimitsExcept=MEMLOCK in slurm.conf 3) UsePAM=1 in slurm.conf 4) in /etc/security/limits.conf * hard memlock unlimited * soft memlock unlimited * hard stack unlimited * soft stack unlimited My changes seem to be working, of I submit this to slurm: #!/bin/bash -l ulimit -l hostname mpirun bash -c ulimit -l mpirun ./relay 1 131072 I get: unlimited c2-31 unlimited unlimited unlimited unlimited Is there some new kernel parameter, ofed parameter, or similar that controls locked pages now? The kernel is 3.13.0-36 and the libopenmpi-dev package is 1.6.5. Since the ulimit -l is getting to both the slurm launched script and also to the mpirun launched binaries I'm pretty puzzled. Any suggestions? ___ users mailing list us...@open-mpi.org Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users Link to this post: http://www.open-mpi.org/community/lists/users/2014/10/25544.php
[OMPI users] New ib locked pages behavior?
I've setup several clusters over the years with OpenMPI. I often get the below error: WARNING: It appears that your OpenFabrics subsystem is configured to only allow registering part of your physical memory. This can cause MPI jobs to run with erratic performance, hang, and/or crash. ... http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages Local host: c2-31 Registerable memory: 32768 MiB Total memory:64398 MiB I'm well aware of the normal fixes, and have implemented them in puppet to ensure compute nodes get the changes. To be paranoid I've implemented all the changes, and they all worked under ubuntu 13.10. However with ubuntu 14.04 it seems like it's not working, thus the above message. As recommended by the faq's I've implemented: 1) ulimit -l unlimited in /etc/profile.d/slurm.sh 2) PropagateResourceLimitsExcept=MEMLOCK in slurm.conf 3) UsePAM=1 in slurm.conf 4) in /etc/security/limits.conf * hard memlock unlimited * soft memlock unlimited * hard stack unlimited * soft stack unlimited My changes seem to be working, of I submit this to slurm: #!/bin/bash -l ulimit -l hostname mpirun bash -c ulimit -l mpirun ./relay 1 131072 I get: unlimited c2-31 unlimited unlimited unlimited unlimited Is there some new kernel parameter, ofed parameter, or similar that controls locked pages now? The kernel is 3.13.0-36 and the libopenmpi-dev package is 1.6.5. Since the ulimit -l is getting to both the slurm launched script and also to the mpirun launched binaries I'm pretty puzzled. Any suggestions?
