Hi Michael, When Open MPI run on Omni-Path it will choose the PSM2 MTL by default, to use the libpsm2.so. Strictly speaking, it has compatibility to run using the openib BTL. However, the performance so significantly impacted that it is, not only discouraged, but no tuning would make sense. Regarding the PSM2 MTL, currently it only supports two mca parameters ("mtl_psm2_connect_timeout" and "mtl_psm2_priority") which are not for what you are looking for. Instead, you can set values directly in the PSM2 library with environment variables. Further info in the Programmers Guide:
http://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/fabric-products/Intel_PSM2_PG_H76473_v3_0.pdf More docs: https://www-ssl.intel.com/content/www/us/en/support/network-and-i-o/fabric-products/000016242.html?wapkw=psm2 Now, for your parameters: btl = openib,vader,self -> Ignore this one btl_openib_eager_limit = 160000 -> I don’t clearly see the diff with the below parameter. However, they are set to the same value. Just in case PSM2_MQ_EAGER_SDMA_SZ changes PIO to SDMA, always in eager mode. btl_openib_rndv_eager_limit = 160000 -> PSM2_MQ_RNDV_HFI_THRESH btl_openib_max_send_size = 160000 -> does not apply to PSM2 btl_openib_receive_queues = P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,160000,1024,512,512 -> does not apply for PSM2. Thanks, Regards, _MAC BTW, should change the subject OMA -> OPA From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Xiaolong Cui Sent: Tuesday, August 09, 2016 2:22 PM To: users@lists.open-mpi.org<mailto:users@lists.open-mpi.org> Subject: [OMPI users] runtime performance tuning for Intel OMA interconnect I used to tune the performance of OpenMPI on InfiniBand by changing the OpenMPI MCA parameters for openib component (see https://www.open-mpi.org/faq/?category=openfabrics). Now I migrate to a new cluster that deploys Intel's omni-path interconnect, and my previous approach does not work any more. Does anyone know how to tune the performance for omni-path interconnect (what OpenMPI component to change) ? The version of OpenMPI is openmpi-1.10.2-hfi. I have included the output from opmi_info and openib parameters that I used to change. Thanks! Sincerely, Michael
_______________________________________________ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users