On Sep 1, 2011, at 12:52 AM, Eugene Loh wrote: > On 8/31/2011 4:48 AM, Ralph Castain wrote: >> Perhaps it would help if you had clearly stated your concern. > Yeah. It would have helped had I clearly understood what was going on. Most > of all, that way I wouldn't have had to ask any questions! :^) >> From this description, I gather your concern is -not- that remote processes >> don't see the setting, but that the remote -orteds- don't see it. > Let's dumb this down a notch for my sake. Let's say I want to run a job with > the TCP BTL and have lots of processes. I hit a descriptor limit and so my > friend tells me to use the MCA parameter opal_set_max_sys_limits. So, is the > point that while most MCA parameters can be set on the mpirun command line > *OR* in param files *OR* with environment variables, in this case the > environment-variable setting should be avoided?
It's more general than that - IF you are using an rsh-like launcher, then you need to set any MCA params INTENDED FOR ORTEDS on the cmd line or in the param files. MCA params INTENDED FOR APPS can always be in the environment. The orteds don't care about BTL params, so params for those can be in the environment. Orteds do care about set_limits, so that param cannot go in the environ IF you are using an rsh-like launcher. I know it's somewhat confusing, but that's one of the consequences of selecting an rsh-like launcher. Fortunately, most of the time the system param file can be setup to avoid having users deal with orted-level params, so this isn't an issue. I suppose a more intelligent option is available. All of the MCA params are added to the app_context's env array. The rsh launcher could be smart and (a) harvest the MCA params from that array, (b) see if the cmd line is able to handle adding them, and if so then (c) pass them along - if not, then warn the user with a helpful message about adding them to the param file. A patch would always be welcome, if this would help resolve the concern. Or you can file it as a ticket and someone might tackle it during a quiet time someday. >> Yes, I'm aware of that issue for rsh-based launches. It stems from rsh not >> allowing one to extend the environment. If you place the param on the cmd >> line, then it gets propagated because we collect and extend the cmd line >> params. If you place it in the environment, then we don't - because (as we >> have repeatedly explained to people) we cannot pass all relevant envars on >> the cmd line due to length restrictions. We don't have this issue with cmd >> line params because (the thinking goes) it already fit on the cmd line. >> >> So for rsh-like launches, there is an unavoidable discrepancy. It's one >> reason why we have both system-level and personal-level MCA param files. > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
