On Sep 25, 2013, at 2:56 PM, "Bryan, Clifton W ERDC-RDE-MSRC-MS Contractor"
<clifton.w.br...@erdc.dren.mil> wrote:
> I don’t understand how to respond to the threads..?
If you subscribe to the digest version of the list, it may be difficult to
reply to an individual message.
> Ralph,
> Yes, the job completes – it just gives the warning message. I checked all the
> available suggestions in the link – everything was already set.
>
> Is there any way to just disable the warning message?
No, because Open MPI isn't able to register enough memory. This means that
something still isn't configured right. This is the relevant FAQ entry in
particular:
http://www.open-mpi.org/faq/?category=openfabrics#ib-low-reg-mem
In particular, this is where OMPI is making that calculation:
https://svn.open-mpi.org/trac/ompi/browser/branches/v1.6/ompi/mca/btl/openib/btl_openib.c#L618
Sorry to throw code at you, but it's the most concise way to explain what OMPI
is doing. Basically, depending on what model of Mellanox IB card you have, it
reads some files out of /sys. I'm *guessing* you have new-ish Mellanox cards,
and OMPI is therefore reading /sys/module/mlx4_core/parameters -- in
particular, the log_mtts_per_seg and log_num_mtt files. OMPI then effectively
calculates:
max_reg = log_num_mtt * page_size * log_mtts_per_seg;
And if (max_reg < mem_total * 3 / 4), then OMPI displays the warning.
What you want to do is ensure that you are setting those kernel parameters
properly. The *only* way to know that they were set properly is to examine
those /sys values (i.e., /sys is exposing the values directly from the mlx4
driver; if the values are wrong in /sys, then the driver is not seeing the
values that you set). I seem to recall that there was some oddity in when/how
you could set those particular parameters due to how the mlx4 driver was
implemented; you'll have to check with Mellanox on the details.
--
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/
