unfortunately, there is no one size fits all here. mxm provides best performance for IB.
different application may require different OMPI, mxm, OS tunables and requires some performance engineering. On Mon, Sep 28, 2015 at 9:49 PM, Grigory Shamov <grigory.sha...@umanitoba.ca > wrote: > Hi Nathan, > Hi Mike, > > Thanks for the quick replies! > > My problem is I don't know what are my applications. I mean, I know them, > but we are a general purpose cluster, running in production for quite a > while, and there are everybody, from quantum chemists to machine learners > to bioinformatists. SO a system-wide change might harm some of them; and > doing per-app benchmarking/tuning looks a bit daunting. > > The default behaviour our users are used to was to have unlimited values > for all memory limits. We have set it so a few years ago, as a response > for some user complaints that applications won't start (we set the ulimits > in Torque). > > Is it known (I know every application is different ) how much costs, > performance-wise, to have MXM with good ulimits vs unlimited ulimits, vs > not using MXM at all? > > -- > Grigory Shamov > > Westgrid/ComputeCanada Site Lead > University of Manitoba > E2-588 EITC Building, > (204) 474-9625 > > > > > > > On 15-09-28 12:58 PM, "users on behalf of Nathan Hjelm" > <users-boun...@open-mpi.org on behalf of hje...@lanl.gov> wrote: > > > > >I would like to add that you may want to play with the value and see > >what works for your applications. Most applications should be using > >malloc or similar functions to allocate large memory regions in the heap > >and not on the stack. > > > >-Nathan > > > >On Mon, Sep 28, 2015 at 08:01:09PM +0300, Mike Dubman wrote: > >> Hello Grigory, > >> We observed ~10% performance degradation with heap size set to > >>unlimited > >> for CFD applications. > >> You can measure your application performance with default and > >>unlimited > >> "limits" and select the best setting. > >> Kind Regards. > >> M > >> On Mon, Sep 28, 2015 at 7:36 PM, Grigory Shamov > >> <grigory.sha...@umanitoba.ca> wrote: > >> > >> Hi All, > >> > >> We have built OpenMPI (1.8.8., 1.10.0) against Mellanox OFED 2.4 > >>and > >> corresponding MXM. When it runs now, it gives the following > >>warning, per > >> process: > >> > >> [1443457390.911053] [myhist:5891 :0] mxm.c:185 MXM WARN > >>The > >> 'ulimit -s' on the system is set to 'unlimited'. This may have > >>negative > >> performance implications. Please set the heap size to the default > >>value > >> (10240) > >> > >> We have ulimits for heap (as well as most of the other limits) set > >> unlimited because of applications that might possibly need a lot > >>of RAM. > >> > >> The question is if we should do as MXM wants, or ignore it? Has > >>anyone > >> an > >> experience running recent OpenMPI with MXM enabled, and what kind > >>of > >> ulimits do you have? Any suggestions/comments appreciated, thanks! > >> > >> -- > >> Grigory Shamov > >> > >> Westgrid/ComputeCanada Site Lead > >> University of Manitoba > >> E2-588 EITC Building, > >> (204) 474-9625 > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > >> Link to this post: > >> http://www.open-mpi.org/community/lists/users/2015/09/27697.php > >> > >> -- > >> Kind Regards, > >> M. > > > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > >> Link to this post: > >>http://www.open-mpi.org/community/lists/users/2015/09/27698.php > > > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/09/27701.php > -- Kind Regards, M.
