On 10/21/2014 04:18 PM, Gus Correa wrote: > Hi Bill > > Maybe you're missing these settings in /etc/modprobe.d/mlx4_core.conf ? > > http://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem
Ah, that helped. Although: /lib/modules/3.13.0-36-generic/kernel/drivers/net/ethernet/mellanox/mlx4$ modinfo mlx4_core | grep "^parm" Lists some promising looking parameters: parm: log_mtts_per_seg:Log2 number of MTT entries per segment (1-7) (int) The FAQ recommends log_num_mtt or num_mtt and NOT log_mtts_per_seg, sadly: $ modinfo mlx4_core | grep "^parm" | grep mtt parm: log_mtts_per_seg:Log2 number of MTT entries per segment (1-7) (int) $ Looks like the best I can do is bump log_mtts_per_seg. I tried: $ cat /etc/modprobe.d/mlx4_core.conf options mlx4_core log_num_mtt=24 $ But: [ 6.691959] mlx4_core: unknown parameter 'log_num_mtt' ignored I ended up with: options mlx4_core log_mtts_per_seg=2 I'm hoping that doubles the registerable memory, although I did see a recommendation to raise it to double the system ram (in this case 64GB ram/128GB locakable. Maybe an update to the FAQ is needed?
